BIgQuery Integration

Dataset ID: layer_anomaly_detection
error id: b'', details: gRPC header 'x-request-id' should be filled with valid UUID

I got the Dataset ID from GCP BigQuery Explore tab but I am still getting this error

Hi @CodeBrain ,

Can you please share the output of your layer diagnose command?

Thanks!

I don’t understand what you mean by " layer diagnose command?"

This is what I mean by layer diagnose: Layer SDK commands | Layer Documentation

Where/when exactly do you get the Dataset ID: layer_anomaly_detection error id: b'', details: gRPC header 'x-request-id' should be filled with valid UUID error by the way?

When I run my Layer project with BigQuery integration, it asks for Service Account Key Path, Project ID & Dataset ID.

I input the path of the service account key JSON file, Project ID, and Dataset ID values from GCP. But once I input my Dataset ID it outputs the above error

When I run Layer diagnose (I excluded my env file):

Local diagnostics for Layer 0.7.3
OS: Darwin 20.5.0
Platform: uname_result(system='Darwin', node='Brains-MacBook-Pro.local', release='20.5.0', version='Darwin Kernel Version 20.5.0: Sat May  8 05:10:31 PDT 2021; root:xnu-7195.121.3~9/RELEASE_ARM64_T8101', machine='x86_64', processor='i386')
Interpreter path: /Users/codebrain/opt/anaconda3/bin/python
Interpreter version: 3.8.8 (default, Apr 13 2021, 12:59:45)
[Clang 10.0.0 ]
Timezone: WAT
Environment variable keys: TERM_SESSION_ID SSH_AUTH_SOCK LC_TERMINAL_VERSION COLORFGBG ITERM_PROFILE XPC_FLAGS PWD SHELL __CFBundleIdentifier LC_CTYPE TERM_PROGRAM_VERSION TERM_PROGRAM PATH LC_TERMINAL COLORTERM COMMAND_MODE TERM HOME TMPDIR USER XPC_SERVICE_NAME LOGNAME ITERM_SESSION_ID __CF_USER_TEXT_ENCODING SHLVL OLDPWD HOMEBREW_PREFIX HOMEBREW_CELLAR HOMEBREW_REPOSITORY HOMEBREW_SHELLENV_PREFIX MANPATH INFOPATH XDG_CONFIG_HOME XDG_CACHE_HOME P9K_TTY _P9K_TTY ZSH PAGER LESS LSCOLORS P9K_SSH CONDA_EXE _CE_M _CE_CONDA CONDA_PYTHON_EXE CONDA_SHLVL CONDA_PREFIX CONDA_DEFAULT_ENV CONDA_PROMPT_MODIFIER _
Layer project found in the current directory. Printing the directory structure:
├── .DS_Store
├── service-key.json
├── .layer
│   ├── project.yml
│   └── .ipynb_checkpoints
├── models
│   ├── .DS_Store
│   └── fraud_detection_model
│       ├── requirements.txt
│       ├── model.py
│       └── model.yml
├── .env
├── test.txt
├── .ipynb_checkpoints
└── data
    ├── .DS_Store
    ├── transactions
    │   └── dataset.yaml
    ├── .ipynb_checkpoints
    ├── transaction_features
    │   ├── new_balance_dest.sql
    │   ├── old_balance_orig.sql
    │   ├── error_balance_dest.sql
    │   ├── is_fraud.sql
    │   ├── type.sql
    │   ├── new_balance_orig.sql
    │   ├── error_balance_orig.sql
    │   ├── old_balance_dest.sql
    │   └── dataset.yml
    └── test.ipynb
/Users/codebrain/.layer is found. Contents:
├── config.json
└── logs
    ├── 20210830T182637-session-110a6a50-6c31-4bb0-bc1f-82f57e6dbef0.log
    └── 20210830T185700-session-44e16c3f-e65c-408c-8f34-fded4d2d0f78.log
Installed packages: Babel 2.9.0 Bottleneck 1.3.2 Cython 0.29.23 Flask 1.1.2 GitPython 3.1.14 HeapDict 1.0.1 Jinja2 2.11.3 Mako 1.1.4 MarkupSafe 1.1.1 Pillow 8.2.0 PyJWT 1.7.1 PySocks 1.7.1 PyWavelets 1.1.1 PyYAML 5.4.1 Pygments 2.8.1 QDarkStyle 2.8.1 QtAwesome 1.0.2 QtPy 1.9.0 Rtree 0.9.7 SQLAlchemy 1.4.7 Send2Trash 1.5.0 Sphinx 4.0.1 Werkzeug 1.0.1 XlsxWriter 1.3.8 aiodns 3.0.0 aiodocker 0.21.0 aiohttp 3.7.4.post0 alabaster 0.7.12 alembic 1.4.1 anaconda-client 1.7.2 anaconda-navigator 2.0.3 anaconda-project 0.9.1 anyio 2.2.0 appdirs 1.4.4 applaunchservices 0.2.1 appnope 0.1.2 appscript 1.1.2 argh 0.26.2 argon2-cffi 20.1.0 asn1crypto 1.4.0 astroid 2.5 astropy 4.2.1 async-generator 1.10 async-timeout 3.0.1 atomicwrites 1.4.0 attrs 20.3.0 autopep8 1.5.6 backcall 0.2.0 backports.entry-points-selectable 1.1.0 backports.functools-lru-cache 1.6.4 backports.shutil-get-terminal-size 1.0.0 backports.tempfile 1.0 backports.weakref 1.0.post1 beautifulsoup4 4.9.3 bitarray 1.9.2 bkcharts 0.2 black 19.10b0 bleach 3.3.0 bokeh 2.3.2 boto 2.49.0 boto3 1.18.9 botocore 1.21.9 brotlipy 0.7.0 cchardet 2.1.7 certifi 2020.12.5 cffi 1.14.5 chardet 4.0.0 click 8.0.1 cloudpickle 1.6.0 clyent 1.2.2 colorama 0.4.4 commonmark 0.9.1 conda 4.10.3 conda-build 3.21.4 conda-content-trust 0.1.3 conda-package-handling 1.7.3 conda-repo-cli 1.0.4 conda-token 0.3.0 conda-verify 3.4.2 contextlib2 0.6.0.post1 cryptography 3.4.7 cycler 0.10.0 cytoolz 0.11.0 dask 2021.4.0 databricks-cli 0.14.3 decorator 5.0.6 defusedxml 0.7.1 diff-match-patch 20200713 distlib 0.3.2 distributed 2021.4.0 docker 5.0.0 docutils 0.17 entrypoints 0.3 et-xmlfile 1.0.1 fastcache 1.1.0 filelock 3.0.12 flake8 3.9.0 fsspec 0.9.0 future 0.18.2 gevent 21.1.2 gitdb 4.0.7 glob2 0.7 gmpy2 2.0.8 greenlet 1.0.0 grpcio 1.39.0 grpcio-tools 1.39.0 gunicorn 20.1.0 h5py 2.10.0 html5lib 1.1 huggingface-hub 0.0.12 idna 2.10 imageio 2.9.0 imagesize 1.2.0 importlib-metadata 3.10.0 iniconfig 1.1.1 intervaltree 3.1.0 ipykernel 5.3.4 ipython 7.22.0 ipython-genutils 0.2.0 ipywidgets 7.6.3 isort 5.8.0 itsdangerous 1.1.0 jdcal 1.4.1 jedi 0.17.2 jmespath 0.10.0 joblib 1.0.1 json5 0.9.5 jsonschema 3.2.0 jupyter 1.0.0 jupyter-client 6.1.12 jupyter-console 6.4.0 jupyter-core 4.7.1 jupyter-packaging 0.7.12 jupyter-server 1.4.1 jupyterlab 3.0.14 jupyterlab-pygments 0.1.2 jupyterlab-server 2.4.0 jupyterlab-widgets 1.0.0 keyring 22.3.0 kiwisolver 1.3.1 layer-sdk 0.7.3 lazy-object-proxy 1.6.0 libarchive-c 2.9 lightgbm 3.1.1 llvmlite 0.36.0 locket 0.2.1 lxml 4.6.3 matplotlib 3.3.4 mccabe 0.6.1 mistune 0.8.4 mkl-fft 1.3.0 mkl-random 1.2.1 mkl-service 2.3.0 mlflow 1.19.0 mock 4.0.3 more-itertools 8.7.0 mpmath 1.2.1 msgpack 1.0.2 multidict 5.1.0 multipledispatch 0.6.0 mypy-extensions 0.4.3 navigator-updater 0.2.1 nbclassic 0.2.6 nbclient 0.5.3 nbconvert 6.0.7 nbformat 5.1.3 nest-asyncio 1.5.1 networkx 2.5 nltk 3.6.1 nose 1.3.7 notebook 6.3.0 numba 0.53.1 numexpr 2.7.3 numpy 1.20.1 numpydoc 1.1.0 olefile 0.46 openpyxl 3.0.7 packaging 20.9 pandas 1.2.4 pandocfilters 1.4.3 parso 0.7.0 partd 1.2.0 path 15.1.2 pathlib2 2.3.5 pathspec 0.7.0 patsy 0.5.1 pep8 1.7.1 pexpect 4.8.0 pickleshare 0.7.5 pip 21.2.4 pkginfo 1.7.0 platformdirs 2.2.0 plotly 5.1.0 plotly-express 0.4.0 pluggy 0.13.1 ply 3.11 polling 0.3.2 prometheus-client 0.10.1 prometheus-flask-exporter 0.18.2 prompt-toolkit 3.0.17 protobuf 3.17.3 psutil 5.8.0 ptyprocess 0.7.0 py 1.10.0 py4j 0.10.9 pyOpenSSL 20.0.1 pyarrow 5.0.0 pycares 4.0.0 pycodestyle 2.6.0 pycosat 0.6.3 pycparser 2.20 pycurl 7.43.0.6 pydocstyle 6.0.0 pyerfa 1.7.3 pyflakes 2.2.0 pylint 2.7.4 pyls-black 0.4.6 pyls-spyder 0.3.2 pyodbc 4.0.0-unsupported pyparsing 2.4.7 pyrsistent 0.17.3 pyspark 3.1.2 pytest 6.2.3 python-dateutil 2.8.1 python-editor 1.0.4 python-jsonrpc-server 0.4.0 python-language-server 0.36.2 pytz 2021.1 pyzmq 20.0.0 qtconsole 5.0.3 querystring-parser 1.2.4 regex 2021.4.4 requests 2.25.1 rich 10.6.0 rope 0.18.0 ruamel-yaml-conda 0.15.100 s3transfer 0.5.0 sacremoses 0.0.45 scikit-image 0.18.1 scikit-learn 0.24.1 scipy 1.6.2 seaborn 0.11.1 setuptools 52.0.0.post20210125 simplegeneric 0.8.1 singledispatch 0.0.0 six 1.15.0 smmap 4.0.0 sniffio 1.2.0 snowballstemmer 2.1.0 sortedcollections 2.1.0 sortedcontainers 2.3.0 soupsieve 2.2.1 sphinxcontrib-applehelp 1.0.2 sphinxcontrib-devhelp 1.0.2 sphinxcontrib-htmlhelp 1.0.3 sphinxcontrib-jsmath 1.0.1 sphinxcontrib-qthelp 1.0.3 sphinxcontrib-serializinghtml 1.1.4 sphinxcontrib-websupport 1.2.4 spyder 4.2.5 spyder-kernels 1.10.2 sqlparse 0.4.1 statsmodels 0.12.2 sympy 1.8 tables 3.6.1 tabulate 0.8.9 tblib 1.7.0 tenacity 8.0.1 terminado 0.9.4 testpath 0.4.4 textdistance 4.2.1 threadpoolctl 2.1.0 three-merge 0.1.1 tifffile 2020.10.1 tokenizers 0.10.3 toml 0.10.2 toolz 0.11.1 tornado 6.1 tqdm 4.59.0 traitlets 5.0.5 transformers 4.9.1 typed-ast 1.4.2 typing-extensions 3.7.4.3 ujson 4.0.2 unicodecsv 0.14.1 urllib3 1.26.4 validate-email 1.3 validators 0.18.2 virtualenv 20.7.2 watchdog 1.0.2 wcwidth 0.2.5 webencodings 0.5.1 websocket-client 1.1.0 wheel 0.36.2 widgetsnbextension 3.5.1 wrapt 1.12.1 wurlitzer 2.1.0 xgboost 1.4.2 xlrd 2.0.1 xlwings 0.23.0 xlwt 1.3.0 xmltodict 0.12.0 yapf 0.31.0 yarl 1.6.3 zict 2.0.0 zipp 3.4.1 zope.event 4.5.0 zope.interface 5.3.0

I Just resolved the issue, I was using the wrong values in my dataset.yaml file.

materializations:
  - type: table
    target: anomaly-db
    table_name: "Anomaly detection table"

I changed the target value to the name of my integration for the default Layer and also updated my table_name to my table name in GCP BigQuery.

Now i am having a failed build error

LAYER RUN FAILED after 8489ms:
Failed to build featureset 'transaction_features': Unexpected exception reading PKCS#8 data

Hey @CodeBrain, thank you so much for your posts. Would you mind sharing the output of layer logs?

Hey @CodeBrain, are you still facing this issue?

Hey @CodeBrain, haven’t heard from you so am going to close this issue for now. Please don’t hesitate to create a new thread if you encounter any issues in future!