Failure to Build Features from BigQuery

I set up a project with a BigQuery data source. However, it fails to build features, which happens after a long time trying to do so. I have attached a screenshot of the build process.

I am running version 0.6.9 of Layer.

On my Layer dashboard, I am able to see the features SQL.


Screenshot from 2021-08-02 10-41-51

Screenshot from 2021-08-02 10-42-22

The git repo

Hey, Eric.

Thanks for reporting. I’ve checked the data source you’ve created for the big query and it looks like it is named layer-bigquery. However, in dataset/dataset.yml and features/dataset.yml you’re using layer. Changing it to layer-bigquery should fix the issue.

On the other hand, such error should be reported and there’s a bug on our side. We plan to release a fix this week.

Thank you.

I renamed the target, but now I run into an error without any clue as to what could be causing it. There is no indication of what is throwing the LayerClientException…

Hi @gitonga
We are still looking into it.
In the meantime, can you try the following:
First, layer 0.7.0 is now released. Please upgrade it;
After that, try to re-run your project and in case it fails attach the full console output(as before but also including run id that is printed in the beginning).

Lastly, can you run(either before or after the run) layer diagnose and add it’s output? It will help us look through backend logs because we will understand your timezone, etc.

Hi. I did try with the latest version, and still failed. Let me re-run it and attach the information you request.

Run session id: 3d9b8567-5e23-4604-a5e1-56c413553db8

Local diagnostics for Layer 0.7.0
OS: Linux 5.11.0-25-generic
Platform: uname_result(system='Linux', node='delton', release='5.11.0-25-generic', version='#27~20.04.1-Ubuntu SMP Tue Jul 13 17:41:23 UTC 2021', machine='x86_64', processor='x86_64')
Interpreter path: /home/gitonga/Develop/Data Science/Projects/Rabbit Holes/Writing/Mine/clients/layer/bin/python3
Interpreter version: 3.8.10 (default, Jun  2 2021, 10:49:15) 
[GCC 9.4.0]
Timezone: EAT
Environment variable keys: GJS_DEBUG_TOPICS LESSOPEN TMUX CONDA_PROMPT_MODIFIER USER LC_TIME SSH_AGENT_PID XDG_SESSION_TYPE SHLVL LIBGL_ALWAYS_SOFTWARE LD_LIBRARY_PATH HOME CONDA_SHLVL OLDPWD DESKTOP_SESSION GNOME_SHELL_SESSION_MODE GTK_MODULES PS1 LC_MONETARY MANAGERPID DBUS_SESSION_BUS_ADDRESS COLORTERM _CE_M IM_CONFIG_PHASE AIRFLOW_HOME LOGNAME JOURNAL_STREAM _ XDG_SESSION_CLASS USERNAME TERM GNOME_DESKTOP_SESSION_ID _CE_CONDA WINDOWPATH PATH SESSION_MANAGER INVOCATION_ID PAPERSIZE XDG_MENU_PREFIX LC_ADDRESS GNOME_TERMINAL_SCREEN XDG_RUNTIME_DIR DISPLAY LANG XDG_CURRENT_DESKTOP LC_TELEPHONE XMODIFIERS XDG_SESSION_DESKTOP XAUTHORITY LS_COLORS GNOME_TERMINAL_SERVICE SSH_AUTH_SOCK CONDA_PYTHON_EXE SHELL LC_NAME QT_ACCESSIBILITY GDMSESSION LESSCLOSE CONDA_DEFAULT_ENV LC_MEASUREMENT GPG_AGENT_INFO GJS_DEBUG_OUTPUT LC_IDENTIFICATION VIRTUAL_ENV QT_IM_MODULE PWD XDG_CONFIG_DIRS CONDA_EXE XDG_DATA_DIRS LC_NUMERIC CONDA_PREFIX LC_PAPER VTE_VERSION TMUX_PANE
Layer project found in the current directory. Printing the directory structure:
β”œβ”€β”€ models
β”‚   └── model
β”‚       β”œβ”€β”€ requirements.txt
β”‚       β”œβ”€β”€ model.yml
β”‚       └── model.py
β”œβ”€β”€ ld
β”œβ”€β”€ .layer
β”‚   └── project.yml
β”œβ”€β”€ data
β”‚   β”œβ”€β”€ features
β”‚   β”‚   β”œβ”€β”€ success.sql
β”‚   β”‚   β”œβ”€β”€ month.sql
β”‚   β”‚   β”œβ”€β”€ day.sql
β”‚   β”‚   β”œβ”€β”€ rated.sql
β”‚   β”‚   β”œβ”€β”€ dataset.yml
β”‚   β”‚   β”œβ”€β”€ number_of_directors.sql
β”‚   β”‚   β”œβ”€β”€ release_date.sql
β”‚   β”‚   └── runtime.sql
β”‚   └── dataset
β”‚       └── dataset.yaml
β”œβ”€β”€ .git
β”‚   β”œβ”€β”€ COMMIT_EDITMSG
β”‚   β”œβ”€β”€ hooks
β”‚   β”‚   β”œβ”€β”€ post-update.sample
β”‚   β”‚   β”œβ”€β”€ pre-receive.sample
β”‚   β”‚   β”œβ”€β”€ prepare-commit-msg.sample
β”‚   β”‚   β”œβ”€β”€ fsmonitor-watchman.sample
β”‚   β”‚   β”œβ”€β”€ pre-merge-commit.sample
β”‚   β”‚   β”œβ”€β”€ pre-commit.sample
β”‚   β”‚   β”œβ”€β”€ update.sample
β”‚   β”‚   β”œβ”€β”€ pre-rebase.sample
β”‚   β”‚   β”œβ”€β”€ pre-push.sample
β”‚   β”‚   β”œβ”€β”€ commit-msg.sample
β”‚   β”‚   β”œβ”€β”€ applypatch-msg.sample
β”‚   β”‚   └── pre-applypatch.sample
β”‚   β”œβ”€β”€ description
β”‚   β”œβ”€β”€ info
β”‚   β”‚   └── exclude
β”‚   β”œβ”€β”€ HEAD
β”‚   β”œβ”€β”€ logs
β”‚   β”‚   β”œβ”€β”€ HEAD
β”‚   β”‚   └── refs
β”‚   β”‚       β”œβ”€β”€ heads
β”‚   β”‚       β”‚   └── main
β”‚   β”‚       └── remotes
β”‚   β”‚           └── origin
β”‚   β”‚               └── main
β”‚   β”œβ”€β”€ branches
β”‚   β”œβ”€β”€ index
β”‚   β”œβ”€β”€ refs
β”‚   β”‚   β”œβ”€β”€ heads
β”‚   β”‚   β”‚   └── main
β”‚   β”‚   β”œβ”€β”€ remotes
β”‚   β”‚   β”‚   └── origin
β”‚   β”‚   β”‚       └── main
β”‚   β”‚   └── tags
β”‚   β”œβ”€β”€ config
β”‚   └── objects
β”‚       β”œβ”€β”€ 53
β”‚       β”‚   └── cfac1ef716416209f50a93c3a63fb1a2ae83b7
β”‚       β”œβ”€β”€ bb
β”‚       β”‚   β”œβ”€β”€ e8ee49901432b79b4a49d9731b22abb9ab96a5
β”‚       β”‚   └── 92ce7994bf8c97b4da0255a8ce073306846151
β”‚       β”œβ”€β”€ info
β”‚       β”œβ”€β”€ 1f
β”‚       β”‚   └── 2961e802b0539716a6af24079c8d5f75cc305a
β”‚       β”œβ”€β”€ 98
β”‚       β”‚   └── b1ad4f88eaed1203d778c1cac1aeb5c1c1b0e6
β”‚       β”œβ”€β”€ ef
β”‚       β”‚   └── f2d5263410988aec4371ff8e8e63f7533efa9e
β”‚       β”œβ”€β”€ f7
β”‚       β”‚   β”œβ”€β”€ 2660e2e479745aa1a1be8580ff10055d1f1a2b
β”‚       β”‚   └── 31b0b8061cdee80500ed1652d23f4013a43bb3
β”‚       β”œβ”€β”€ 9b
β”‚       β”‚   └── 90dc5340f9b5f18c9208e3e98420a51e636c01
β”‚       β”œβ”€β”€ 4a
β”‚       β”‚   └── 74672c3a3aa9680c8d0a6f1dabb0fdd40daad4
β”‚       β”œβ”€β”€ pack
β”‚       β”œβ”€β”€ 32
β”‚       β”‚   └── 0769546e7e7881ae6d29f7eb7c9c9e204164de
β”‚       β”œβ”€β”€ c2
β”‚       β”‚   └── 9365031579da64a2e984b78ffb10db84190d74
β”‚       β”œβ”€β”€ e7
β”‚       β”‚   └── cc5312927cdc5d73efecb3ba677ccd781fee06
β”‚       β”œβ”€β”€ d5
β”‚       β”‚   └── 5304d832f680e0cc833620e839315da52e3d0f
β”‚       β”œβ”€β”€ c9
β”‚       β”‚   └── 6918a753845b117876df729eb62b458e735660
β”‚       β”œβ”€β”€ 8a
β”‚       β”‚   β”œβ”€β”€ 0a7fe3a4e8919128bdb8c96218697f6a05f5cd
β”‚       β”‚   └── a18a00b6a543a55cc876c0196e9eaa658d3b20
β”‚       β”œβ”€β”€ 4b
β”‚       β”‚   β”œβ”€β”€ 825dc642cb6eb9a060e54bf8d69288fbee4904
β”‚       β”‚   └── c45fece64fa08c8fc68d0edbe658ebb7889a39
β”‚       β”œβ”€β”€ 69
β”‚       β”‚   └── cd92992aa63b2d3d75ace2ffd722553845abfe
β”‚       β”œβ”€β”€ e3
β”‚       β”‚   └── ac850747d0d7ed8b7fbf7d93c428164d35dad7
β”‚       β”œβ”€β”€ ec
β”‚       β”‚   └── 72c259c9e1520c9e897e6bdff0ef9fabb7ac13
β”‚       β”œβ”€β”€ 37
β”‚       β”‚   └── cd344d0f8d710768cb9e229d37c2806a1ea717
β”‚       β”œβ”€β”€ 1c
β”‚       β”‚   └── b3fa37dc92392a9d532f8a9d740200d64ea749
β”‚       β”œβ”€β”€ 9c
β”‚       β”‚   └── 3641b49076838ff7c648e6c8844813d2c9b611
β”‚       β”œβ”€β”€ e5
β”‚       β”‚   └── f3cfe55e31e1b1643ba5d9626f7b8eea8b4f20
β”‚       β”œβ”€β”€ 79
β”‚       β”‚   └── 142d54829a7e42c213ed4e037116675ba7145d
β”‚       β”œβ”€β”€ 08
β”‚       β”‚   └── 74e1576524342af4d03052be11a406d52123ec
β”‚       β”œβ”€β”€ c8
β”‚       β”‚   └── e611f2c333dbb3aae916cdb6d32946cc4860ae
β”‚       β”œβ”€β”€ 55
β”‚       β”‚   └── a34adc4fbee09bb4a7fc0e88ea69749b330090
β”‚       β”œβ”€β”€ 27
β”‚       β”‚   └── cc7988877a577ef8cf05ce3b3f657ffaef4b1c
β”‚       β”œβ”€β”€ 5d
β”‚       β”‚   └── 2701487f4106704acc04a07e14774e7017ce2d
β”‚       β”œβ”€β”€ 8e
β”‚       β”‚   └── a945cbd4af2e4a7f15bfe3302b13e20290c163
β”‚       β”œβ”€β”€ 1e
β”‚       β”‚   └── d0e42fa866dc68f4a86290c35561ec013a20c8
β”‚       β”œβ”€β”€ 7a
β”‚       β”‚   └── 9645b59e3096eaf36e27231e6e33948b4bf55d
β”‚       β”œβ”€β”€ 80
β”‚       β”‚   β”œβ”€β”€ 45106745ce7f244c8c98711fa63b503682ef33
β”‚       β”‚   └── a965239cffe5237323c676b254730666b96536
β”‚       β”œβ”€β”€ 3d
β”‚       β”‚   └── 74b660f0484c10b3508bd1c9ffcd796731fdfc
β”‚       β”œβ”€β”€ 1a
β”‚       β”‚   └── a8c41512b1c27e8605f867c3e9dd13aa6aeb58
β”‚       β”œβ”€β”€ b9
β”‚       β”‚   └── 733ef5423aaba3a9833ef266430acf89d99888
β”‚       β”œβ”€β”€ 92
β”‚       β”‚   β”œβ”€β”€ b5d612f0fa55ea818115f2888f2100af7505d6
β”‚       β”‚   └── 5a8397193fe649e18149cba0a2b8c948add59c
β”‚       β”œβ”€β”€ ab
β”‚       β”‚   β”œβ”€β”€ 3bc43670b49a65d315694678bad01e1a87ed69
β”‚       β”‚   └── 446a9de943c5c2a5c3ef8d30b963c5e89be966
β”‚       β”œβ”€β”€ 26
β”‚       β”‚   └── 41651495691377ce57a738fb148143fa453331
β”‚       β”œβ”€β”€ 5b
β”‚       β”‚   └── c5e1a3aaca1cca176e6607b2a5365501aa898f
β”‚       β”œβ”€β”€ 76
β”‚       β”‚   β”œβ”€β”€ 7be6d416e55f2cb098890c6ea1f4018640b310
β”‚       β”‚   └── 1bb3a90f210f5664dc0dc635cf546a8bbb2f6b
β”‚       β”œβ”€β”€ f4
β”‚       β”‚   └── 3905c2c388317b4a49e79b5f071d893dd063c4
β”‚       β”œβ”€β”€ 7e
β”‚       β”‚   └── bfb267a17c5c8b410fb2c5949298bbfb9000ba
β”‚       β”œβ”€β”€ a7
β”‚       β”‚   └── a7ac0b2acc5cc4fe857f69661b7d4b239a9a1c
β”‚       └── 13
β”‚           └── 473094fdb4365364a01135895168e75ed2807c
└── README.md
/home/gitonga/.layer is found. Contents:
└── config.json
Installed packages: Babel 2.9.1 Flask 2.0.1 GitPython 3.1.14 Jinja2 3.0.1 Mako 1.1.4 MarkupSafe 2.0.1 Pillow 8.3.1 PyJWT 1.7.1 PyQt5 5.12.3 PyQt5-sip 12.9.0 PyQtWebEngine 5.12.1 PyYAML 5.4.1 Pygments 2.9.0 QDarkStyle 3.0.2 QtAwesome 1.0.3 QtPy 1.9.0 Rtree 0.9.7 SQLAlchemy 1.4.20 SecretStorage 3.3.1 Send2Trash 1.7.1 Sphinx 4.0.3 Werkzeug 2.0.1 aiodns 3.0.0 aiodocker 0.19.1 aiohttp 3.7.4.post0 alabaster 0.7.12 alembic 1.4.1 altair 4.1.0 appdirs 1.4.4 argon2-cffi 20.1.0 arrow 1.1.1 astor 0.8.1 astroid 2.6.2 async-generator 1.10 async-timeout 3.0.1 atomicwrites 1.4.0 attrs 21.2.0 autopep8 1.5.5 backcall 0.2.0 base58 2.1.0 binaryornot 0.4.4 black 21.6b0 bleach 3.3.0 blinker 1.4 boto3 1.17.106 botocore 1.20.106 brotlipy 0.7.0 cachetools 4.2.2 cchardet 2.1.7 certifi 2021.5.30 cffi 1.14.5 chardet 4.0.0 click 8.0.1 cloudpickle 1.6.0 colorama 0.4.4 commonmark 0.9.1 cookiecutter 1.7.3 cryptography 3.4.7 databricks-cli 0.14.3 debugpy 1.3.0 decorator 4.4.2 defusedxml 0.7.1 diff-match-patch 20200713 docker 5.0.0 docutils 0.17.1 entrypoints 0.3 filelock 3.0.12 flake8 3.8.4 gitdb 4.0.7 greenlet 1.1.0 grpcio 1.39.0 grpcio-tools 1.39.0 gunicorn 20.1.0 huggingface-hub 0.0.12 idna 2.10 imagesize 1.2.0 importlib-metadata 4.6.1 inflection 0.5.1 intervaltree 3.1.0 ipykernel 6.0.1 ipython 7.25.0 ipython-genutils 0.2.0 ipywidgets 7.6.3 isort 5.9.1 itsdangerous 2.0.1 jedi 0.17.2 jeepney 0.6.0 jinja2-time 0.2.0 jmespath 0.10.0 joblib 1.0.1 jsonschema 3.2.0 jupyter-client 6.1.12 jupyter-core 4.7.1 jupyterlab-pygments 0.1.2 jupyterlab-widgets 1.0.0 keyring 23.0.1 layer-sdk 0.7.0 lazy-object-proxy 1.6.0 matplotlib-inline 0.1.2 mccabe 0.6.1 mistune 0.8.4 mlflow 1.18.0 multidict 5.1.0 mypy-extensions 0.4.3 nbclient 0.5.3 nbconvert 6.1.0 nbformat 5.1.3 nest-asyncio 1.5.1 networkx 2.5.1 notebook 6.4.0 numpy 1.21.0 numpydoc 1.1.0 packaging 21.0 pandas 1.3.0 pandocfilters 1.4.3 parso 0.7.0 pathspec 0.8.1 pexpect 4.8.0 pickleshare 0.7.5 pip 21.2.2 pkg-resources 0.0.0 pluggy 0.13.1 polling 0.3.2 poyo 0.5.0 prometheus-client 0.11.0 prometheus-flask-exporter 0.18.2 prompt-toolkit 3.0.19 protobuf 3.17.3 psutil 5.8.0 ptyprocess 0.7.0 py4j 0.10.9 pyarrow 4.0.1 pycares 4.0.0 pycodestyle 2.6.0 pycparser 2.20 pydeck 0.6.2 pydocstyle 6.1.1 pyflakes 2.2.0 pylint 2.9.3 pyls-black 0.4.7 pyls-spyder 0.3.2 pyparsing 2.4.7 pyrsistent 0.18.0 pyspark 3.1.2 python-dateutil 2.8.1 python-editor 1.0.4 python-jsonrpc-server 0.4.0 python-language-server 0.36.2 python-slugify 5.0.2 pytz 2021.1 pyxdg 0.27 pyzmq 22.1.0 qstylizer 0.2.0 qtconsole 5.1.1 querystring-parser 1.2.4 regex 2021.7.6 requests 2.25.1 rich 10.5.0 rope 0.19.0 s3transfer 0.4.2 sacremoses 0.0.45 scikit-learn 0.24.2 scipy 1.7.0 setuptools 44.0.0 six 1.16.0 sklearn 0.0 smmap 4.0.0 snowballstemmer 2.1.0 sortedcontainers 2.4.0 sphinxcontrib-applehelp 1.0.2 sphinxcontrib-devhelp 1.0.2 sphinxcontrib-htmlhelp 2.0.0 sphinxcontrib-jsmath 1.0.1 sphinxcontrib-qthelp 1.0.3 sphinxcontrib-serializinghtml 1.1.5 spyder 5.0.5 spyder-kernels 2.0.5 sqlparse 0.4.1 tabulate 0.8.9 terminado 0.10.1 testpath 0.5.0 text-unidecode 1.3 textdistance 4.2.1 threadpoolctl 2.1.0 three-merge 0.1.1 tinycss2 1.1.0 tokenizers 0.10.3 toml 0.10.2 toolz 0.11.1 tornado 6.1 tqdm 4.61.2 traitlets 5.0.5 transformers 4.8.2 typing-extensions 3.10.0.0 tzlocal 2.1 ujson 4.0.2 urllib3 1.26.6 validate-email 1.3 validators 0.18.2 watchdog 2.1.3 watermark 2.2.0 wcwidth 0.2.5 webencodings 0.5.1 websocket-client 1.1.0 wheel 0.34.2 widgetsnbextension 3.5.1 wrapt 1.12.1 wurlitzer 2.1.0 xgboost 1.4.2 yapf 0.31.0 yarl 1.6.3 zipp 3.5.0

Hi @gitonga

Apologies for delayed response

Can you please update materialization section of bq-movies/dataset.yml at main Β· ericgitonga/bq-movies Β· GitHub similar to bq-movies/dataset.yaml at main Β· ericgitonga/bq-movies Β· GitHub

Please let us know if it still fails.

I just tried that, and it still fails…

Any update on this issue?

Hi Eric,

Apologies for the delay here. We are trying to reproduce the issue.
Let me get back to you with an update.

kind regards, Gerard.

Hi Gerad,

Okay. Thank you.

Hi Eric,

Would you have a data sample or a link to the dataset you are using? I’d like to upload it to our BQ instance to reproduce this issue.

Thanks, Gerard.

Here it is:

Hi Eric,
Thanks for the data sample. Let me add it to our BQ instance. I’ll get back to you with any findings.

kr, Gerard.

Hi Eric,

A quick reply to let you know that we have found related issues with BigQuery and we are working on a fix for it.
I’ll keep you updated.

kind regards, Gerard.

1 Like

Thank you for the update Gerard.

Hey @gitonga, we’re still working on this bugfix, and will keep you updated as we make progress. Thanks!

Thank you for the update @volkan