Failure to Build Features from Snowflake

I have problems building entities using a Snowflake integration I set up.

On executing layer run, I get errors as indicated in the portion of the log below.

{"timestamp_utc": 1629711465.402157, "method": "/api.FlowManagerAPI/GetRunHistoryById", "request": {"runId": {"value": "79cfb986-de6e-40e6-b79f-0f85661b54f7"}}, "response": {"runId": {"value": "79cfb986-de6e-40e6-b79f-0f85661b54f7"}, "events": [{"task": {"id": "features", "status": "STATUS_SCHEDULED", "type": "TYPE_FEATURE_BUILD", "statusTime": "2021-08-23T09:36:59Z"}}, {"task": {"id": "features", "status": "STATUS_RUNNING", "type": "TYPE_FEATURE_BUILD", "statusTime": "2021-08-23T09:36:59Z"}}, {"run": {"id": {"value": "79cfb986-de6e-40e6-b79f-0f85661b54f7"}, "runStatus": "STATUS_RUNNING", "statusTime": "2021-08-23T09:36:59Z"}}, {"task": {"id": "features", "status": "STATUS_FAILED", "type": "TYPE_FEATURE_BUILD", "info": "(DatasetBuildId(019102e6-23e6-4f38-8a1f-7dbb8e1141e4),org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 223.0 failed 1 times, most recent failure: Lost task 0.0 in stage 223.0 (TID 221, feature-engine-86c766c666-swn74, executor driver): java.lang.NullPointerException\n\tat net.snowflake.client.jdbc.telemetryOOB.TelemetryService$TELEMETRY_SERVER_DEPLOYMENT.access$000(TelemetryService.java:217)\n\tat net.snowflake.client.jdbc.telemetryOOB.TelemetryService.getServerDeploymentName(TelemetryService.java:256)\n\tat net.snowflake.client.jdbc.telemetryOOB.TelemetryEvent$Builder.<init>(TelemetryEvent.java:126)\n\tat net.snowflake.client.jdbc.telemetryOOB.TelemetryEvent$LogBuilder.<init>(TelemetryEvent.java:61)\n\tat net.snowflake.client.jdbc.SnowflakeSQLLoggedException.sendOutOfBandTelemetryMessage(SnowflakeSQLLoggedException.java:53)\n\tat net.snowflake.client.jdbc.SnowflakeSQLLoggedException.sendTelemetryData(SnowflakeSQLLoggedException.java:212)\n\tat net.snowflake.client.jdbc.SnowflakeSQLLoggedException.<init>(SnowflakeSQLLoggedException.java:237)\n\tat net.snowflake.client.jdbc.cloud.storage.SnowflakeGCSClient.uploadWithPresignedUrl(SnowflakeGCSClient.java:759)\n\tat net.snowflake.client.jdbc.cloud.storage.SnowflakeGCSClient.uploadWithPresignedUrlWithoutConnection(SnowflakeGCSClient.java:543)\n\tat net.snowflake.client.jdbc.SnowflakeFileTransferAgent.pushFileToRemoteStoreWithPresignedUrl(SnowflakeFileTransferAgent.java:2041)\n\tat net.snowflake.client.jdbc.SnowflakeFileTransferAgent.uploadWithoutConnection(SnowflakeFileTransferAgent.java:1959)\n\tat net.snowflake.spark.snowflake.io.CloudStorage.doUploadPartition(CloudStorageOperations.scala:705)\n\tat net.snowflake.spark.snowflake.io.CloudStorage.uploadPartition(CloudStorageOperations.scala:574)\n\tat net.snowflake.spark.snowflake.io.CloudStorage.uploadPartition$(CloudStorageOperations.scala:557)\n\tat net.snowflake.spark.snowflake.io.InternalGcsStorage.uploadPartition(CloudStorageOperations.scala:1645)\n\tat net.snowflake.spark.snowflake.io.InternalGcsStorage.$anonfun$upload$2(CloudStorageOperations.scala:1755)\n\tat net.snowflake.spark.snowflake.io.InternalGcsStorage.$anonfun$upload$2$adapted(CloudStorageOperations.scala:1739)\n\tat org.apache.spark.rdd.RDD.$anonfun$mapPartitionsWithIndex$2(RDD.scala:889)\n\tat org.apache.spark.rdd.RDD.$anonfun$mapPartitionsWithIndex$2$adapted(RDD.scala:889)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:313)\n\tat org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)\n\tat org.apache.spark.scheduler.Task.run(Task.scala:127)\n\tat org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)\n\tat org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:829)\n\nDriver stacktrace:)", "statusTime": "2021-08-23T09:37:44Z"}}, {"run": {"id": {"value": "79cfb986-de6e-40e6-b79f-0f85661b54f7"}, "runStatus": "STATUS_FAILED", "statusTime": "2021-08-23T09:37:44Z"}}]}, "metadata": {"x-request-id": "48fd76ea-44e3-47c4-84b7-6df2074c3a3f", "layer-sdk-version": "0.7.2", "layer-sdk-language": "python", "layer-sdk-language-version": "3.8.10"}}

Output of diagnose:


Local diagnostics for Layer 0.7.2
OS: Linux 5.11.0-25-generic
Platform: uname_result(system='Linux', node='delton', release='5.11.0-25-generic', version='#27~20.04.1-Ubuntu SMP Tue Jul 13 17:41:23 UTC 2021', machine='x86_64', processor='x86_64')
Interpreter path: /home/gitonga/Develop/layer/bin/python3
Interpreter version: 3.8.10 (default, Jun  2 2021, 10:49:15) 
[GCC 9.4.0]
Timezone: EAT
Environment variable keys: SHELL SESSION_MANAGER QT_ACCESSIBILITY COLORTERM XDG_CONFIG_DIRS XDG_MENU_PREFIX GNOME_DESKTOP_SESSION_ID CONDA_EXE _CE_M TMUX LC_ADDRESS GNOME_SHELL_SESSION_MODE LC_NAME SSH_AUTH_SOCK XMODIFIERS DESKTOP_SESSION LC_MONETARY SSH_AGENT_PID GTK_MODULES PWD XDG_SESSION_DESKTOP LOGNAME XDG_SESSION_TYPE CONDA_PREFIX GPG_AGENT_INFO XAUTHORITY GJS_DEBUG_TOPICS WINDOWPATH HOME USERNAME IM_CONFIG_PHASE LC_PAPER LANG LS_COLORS XDG_CURRENT_DESKTOP VIRTUAL_ENV AIRFLOW_HOME VTE_VERSION CONDA_PROMPT_MODIFIER GNOME_TERMINAL_SCREEN INVOCATION_ID MANAGERPID GJS_DEBUG_OUTPUT LESSCLOSE XDG_SESSION_CLASS TERM LC_IDENTIFICATION _CE_CONDA LESSOPEN USER TMUX_PANE GNOME_TERMINAL_SERVICE CONDA_SHLVL DISPLAY SHLVL LC_TELEPHONE QT_IM_MODULE LC_MEASUREMENT LIBGL_ALWAYS_SOFTWARE PAPERSIZE CONDA_PYTHON_EXE LD_LIBRARY_PATH XDG_RUNTIME_DIR PS1 CONDA_DEFAULT_ENV LC_TIME JOURNAL_STREAM XDG_DATA_DIRS PATH GDMSESSION DBUS_SESSION_BUS_ADDRESS LC_NUMERIC OLDPWD _
Layer project found in the current directory. Printing the directory structure:
β”œβ”€β”€ .gitignore
β”œβ”€β”€ models
β”‚   └── model
β”‚       β”œβ”€β”€ requirements.txt
β”‚       β”œβ”€β”€ model.yml
β”‚       └── model.py
β”œβ”€β”€ ld
β”œβ”€β”€ .layer
β”‚   └── project.yml
β”œβ”€β”€ data
β”‚   β”œβ”€β”€ features
β”‚   β”‚   β”œβ”€β”€ success.sql
β”‚   β”‚   β”œβ”€β”€ month.sql
β”‚   β”‚   β”œβ”€β”€ day.sql
β”‚   β”‚   β”œβ”€β”€ rated.sql
β”‚   β”‚   β”œβ”€β”€ dataset.yml
β”‚   β”‚   β”œβ”€β”€ number_of_directors.sql
β”‚   β”‚   β”œβ”€β”€ release_date.sql
β”‚   β”‚   └── runtime.sql
β”‚   └── movies
β”‚       └── dataset.yaml
β”œβ”€β”€ .git
β”‚   β”œβ”€β”€ COMMIT_EDITMSG
β”‚   β”œβ”€β”€ hooks
β”‚   β”‚   β”œβ”€β”€ post-update.sample
β”‚   β”‚   β”œβ”€β”€ pre-receive.sample
β”‚   β”‚   β”œβ”€β”€ prepare-commit-msg.sample
β”‚   β”‚   β”œβ”€β”€ fsmonitor-watchman.sample
β”‚   β”‚   β”œβ”€β”€ pre-merge-commit.sample
β”‚   β”‚   β”œβ”€β”€ pre-commit.sample
β”‚   β”‚   β”œβ”€β”€ update.sample
β”‚   β”‚   β”œβ”€β”€ pre-rebase.sample
β”‚   β”‚   β”œβ”€β”€ pre-push.sample
β”‚   β”‚   β”œβ”€β”€ commit-msg.sample
β”‚   β”‚   β”œβ”€β”€ applypatch-msg.sample
β”‚   β”‚   └── pre-applypatch.sample
β”‚   β”œβ”€β”€ description
β”‚   β”œβ”€β”€ info
β”‚   β”‚   └── exclude
β”‚   β”œβ”€β”€ HEAD
β”‚   β”œβ”€β”€ logs
β”‚   β”‚   β”œβ”€β”€ HEAD
β”‚   β”‚   └── refs
β”‚   β”‚       β”œβ”€β”€ heads
β”‚   β”‚       β”‚   └── main
β”‚   β”‚       └── remotes
β”‚   β”‚           └── origin
β”‚   β”‚               └── main
β”‚   β”œβ”€β”€ branches
β”‚   β”œβ”€β”€ index
β”‚   β”œβ”€β”€ refs
β”‚   β”‚   β”œβ”€β”€ heads
β”‚   β”‚   β”‚   └── main
β”‚   β”‚   β”œβ”€β”€ remotes
β”‚   β”‚   β”‚   └── origin
β”‚   β”‚   β”‚       └── main
β”‚   β”‚   └── tags
β”‚   β”œβ”€β”€ config
β”‚   └── objects
β”‚       β”œβ”€β”€ 53
β”‚       β”‚   └── 34a78de5ac7067dcb09f1775a53f5af9cd3757
β”‚       β”œβ”€β”€ bc
β”‚       β”‚   └── 5265f04e488e7c1006f4f67fd8d4eb4e876e32
β”‚       β”œβ”€β”€ info
β”‚       β”œβ”€β”€ 98
β”‚       β”‚   └── b1ad4f88eaed1203d778c1cac1aeb5c1c1b0e6
β”‚       β”œβ”€β”€ f7
β”‚       β”‚   └── 31b0b8061cdee80500ed1652d23f4013a43bb3
β”‚       β”œβ”€β”€ 97
β”‚       β”‚   └── 8f26417f802dc699adfb53faade15b4487b0c0
β”‚       β”œβ”€β”€ pack
β”‚       β”œβ”€β”€ 32
β”‚       β”‚   └── 0769546e7e7881ae6d29f7eb7c9c9e204164de
β”‚       β”œβ”€β”€ 8a
β”‚       β”‚   └── a18a00b6a543a55cc876c0196e9eaa658d3b20
β”‚       β”œβ”€β”€ d3
β”‚       β”‚   └── 60230b3d1dd27452476d4d56df3e1f889b0274
β”‚       β”œβ”€β”€ a6
β”‚       β”‚   └── d7ecd9ea8185459ab8c74b2e777994fa54de29
β”‚       β”œβ”€β”€ 1c
β”‚       β”‚   └── ab4ff92f9b6889b63746ba33ad613725b6a62e
β”‚       β”œβ”€β”€ f6
β”‚       β”‚   └── 13d5f969faf8fc0c4c4845db555a7bbb344c0c
β”‚       β”œβ”€β”€ 79
β”‚       β”‚   └── 142d54829a7e42c213ed4e037116675ba7145d
β”‚       β”œβ”€β”€ 8e
β”‚       β”‚   └── a945cbd4af2e4a7f15bfe3302b13e20290c163
β”‚       β”œβ”€β”€ 1e
β”‚       β”‚   └── d0e42fa866dc68f4a86290c35561ec013a20c8
β”‚       β”œβ”€β”€ cd
β”‚       β”‚   └── cb420e14947f6b22bf93b936b1fc12608914b0
β”‚       β”œβ”€β”€ 3d
β”‚       β”‚   └── 74b660f0484c10b3508bd1c9ffcd796731fdfc
β”‚       β”œβ”€β”€ 70
β”‚       β”‚   └── f292b4f34d5513d2d5e3e1c38e6b8301dbd595
β”‚       β”œβ”€β”€ 92
β”‚       β”‚   β”œβ”€β”€ b5d612f0fa55ea818115f2888f2100af7505d6
β”‚       β”‚   └── 5a8397193fe649e18149cba0a2b8c948add59c
β”‚       β”œβ”€β”€ 25
β”‚       β”‚   └── 43ce9180ffc3689671bc5829ed79ada199352c
β”‚       β”œβ”€β”€ f4
β”‚       β”‚   └── 3905c2c388317b4a49e79b5f071d893dd063c4
β”‚       β”œβ”€β”€ 7e
β”‚       β”‚   └── bfb267a17c5c8b410fb2c5949298bbfb9000ba
β”‚       └── 74
β”‚           └── d0d495db7b7fa23e905d06ffbc3dba269ae9cf
β”œβ”€β”€ README.md
└── temp
    β”œβ”€β”€ layer-movies.ipynb
    β”œβ”€β”€ log
    β”œβ”€β”€ error-features
    β”œβ”€β”€ drop
    β”œβ”€β”€ .ipynb_checkpoints
    β”œβ”€β”€ en-snowflake
    └── error-model
/home/gitonga/.layer is found. Contents:
β”œβ”€β”€ logs
β”‚   β”œβ”€β”€ 20210823T123652-session-48fd76ea-44e3-47c4-84b7-6df2074c3a3f.log
β”‚   β”œβ”€β”€ movie-fail.log
β”‚   β”œβ”€β”€ fraud-succeed.log
β”‚   β”œβ”€β”€ fail succeed.log
β”‚   └── 20210823T121342-session-a8739f9f-000a-4435-946e-008525dfa18b.log
β”œβ”€β”€ config.json
└── jars
    β”œβ”€β”€ hadoop-aws-3.2.0.jar
    └── aws-java-sdk-bundle-1.11.375.jar
Installed packages: Flask 2.0.1 GitPython 3.1.14 Jinja2 3.0.1 Mako 1.1.4 MarkupSafe 2.0.1 Pillow 8.3.1 PyJWT 1.7.1 PyYAML 5.4.1 Pygments 2.10.0 SQLAlchemy 1.4.23 Werkzeug 2.0.1 aiodns 3.0.0 aiodocker 0.21.0 aiohttp 3.7.4.post0 alembic 1.4.1 async-timeout 3.0.1 attrs 21.2.0 boto3 1.18.25 botocore 1.21.25 brotlipy 0.7.0 cchardet 2.1.7 certifi 2021.5.30 cffi 1.14.6 chardet 4.0.0 charset-normalizer 2.0.4 click 8.0.1 cloudpickle 1.6.0 colorama 0.4.4 commonmark 0.9.1 cryptography 3.4.7 cycler 0.10.0 databricks-cli 0.15.0 decorator 5.0.9 docker 5.0.0 entrypoints 0.3 filelock 3.0.12 gitdb 4.0.7 greenlet 1.1.1 grpcio 1.39.0 grpcio-tools 1.39.0 gunicorn 20.1.0 huggingface-hub 0.0.12 idna 2.10 itsdangerous 2.0.1 jmespath 0.10.0 joblib 1.0.1 kiwisolver 1.3.1 layer-sdk 0.7.2 matplotlib 3.4.3 mlflow 1.19.0 multidict 5.1.0 networkx 2.6.2 numpy 1.21.2 packaging 21.0 pandas 1.3.2 pip 21.2.4 pkg-resources 0.0.0 polling 0.3.2 prometheus-client 0.11.0 prometheus-flask-exporter 0.18.2 prompt-toolkit 3.0.20 protobuf 3.17.3 py4j 0.10.9 pyarrow 5.0.0 pycares 4.0.0 pycparser 2.20 pyparsing 2.4.7 pyspark 3.1.2 python-dateutil 2.8.2 python-editor 1.0.4 pytz 2021.1 querystring-parser 1.2.4 regex 2021.8.3 requests 2.26.0 rich 10.7.0 s3transfer 0.5.0 sacremoses 0.0.45 scipy 1.7.1 seaborn 0.11.2 setuptools 44.0.0 six 1.16.0 smmap 4.0.0 sqlparse 0.4.1 tabulate 0.8.9 tokenizers 0.10.3 tqdm 4.62.1 transformers 4.9.2 typing-extensions 3.10.0.0 urllib3 1.26.6 validate-email 1.3 validators 0.18.2 wcwidth 0.2.5 websocket-client 1.2.1 wheel 0.34.2 xgboost 1.4.2 yarl 1.6.3

The repository is at GitHub - ericgitonga/snowflake-movies

Thanks for your message, Eric,
This looks like an odd issue:

java.lang.NullPointerException
at net.snowflake.client.jdbc.telemetryOOB.TelemetryService$TELEMETRY_SERVER_DEPLOYMENT.access$000(TelemetryService.java:217)

Would you have a small data sample to try to run the project on our system?

Thanks, Gerard.

Using the same dataset as for BigQuery. But formatted to load to Snowflake.

Here’s the link: snowflake-movies/movies_snowflake.csv at main Β· ericgitonga/snowflake-movies Β· GitHub

Thanks Eric.
Another question: Which Snowflake instance is using the Integration layersn ?
Is it the one we offer or did you bring your own? The error above seems to be related to a SSL configuration issue.

kind regards, Gerard.

It’s my own. I am trying to map the entire process from I have data to I have my models in place, thus wanted to use my own instance.

1 Like

Hi Eric,

I successfully reproduced the snowflake project.

I presume that the issue has to do with some permission/configuration option. Let me pass this on to our infra team to assist in finding a solution.

kind regards, Gerard.

Interesting. Thanks for the update. Looking forward to hear what the infra team says.

Hi Gerard,

Any word from the infra team on this?

Hey Eric,

As far as we saw in the logs, there may be a permission issue (basically, GCP returns 403, but not sure if it’s GCP or Snowflake related), but I can’t really make more suggestions unfortunately.

We have not tested GCP <-> Snowflake yet, and is not something we officially support yet. We do support Snowflake <-> AWS and BigQuery <-> GCP.

Thanks,
Cihat

Thanks for the feedback.

I went ahead and set up a different Snowflake account based on AWS and tried the building again.

I ran into an error, but this time showing the model building failed. Attached are the log and diagnose output.

Local diagnostics for Layer 0.8.1
OS: Linux 5.11.0-27-generic
Platform: uname_result(system='Linux', node='delton', release='5.11.0-27-generic', version='#29~20.04.1-Ubuntu SMP Wed Aug 11 15:58:17 UTC 2021', machine='x86_64', processor='x86_64')
Interpreter path: /home/gitonga/Develop/layer/bin/python3
Interpreter version: 3.8.10 (default, Jun  2 2021, 10:49:15) 
[GCC 9.4.0]
Timezone: EAT
Environment variable keys: SHELL SESSION_MANAGER QT_ACCESSIBILITY COLORTERM XDG_CONFIG_DIRS XDG_MENU_PREFIX GNOME_DESKTOP_SESSION_ID CONDA_EXE _CE_M TMUX LC_ADDRESS GNOME_SHELL_SESSION_MODE LC_NAME SSH_AUTH_SOCK XMODIFIERS DESKTOP_SESSION LC_MONETARY SSH_AGENT_PID GTK_MODULES PWD XDG_SESSION_DESKTOP LOGNAME XDG_SESSION_TYPE CONDA_PREFIX GPG_AGENT_INFO XAUTHORITY GJS_DEBUG_TOPICS WINDOWPATH HOME USERNAME IM_CONFIG_PHASE LC_PAPER LANG LS_COLORS XDG_CURRENT_DESKTOP VIRTUAL_ENV AIRFLOW_HOME VTE_VERSION CONDA_PROMPT_MODIFIER GNOME_TERMINAL_SCREEN INVOCATION_ID MANAGERPID GJS_DEBUG_OUTPUT LESSCLOSE XDG_SESSION_CLASS TERM LC_IDENTIFICATION _CE_CONDA LESSOPEN USER TMUX_PANE GNOME_TERMINAL_SERVICE CONDA_SHLVL DISPLAY SHLVL LC_TELEPHONE QT_IM_MODULE LC_MEASUREMENT LIBGL_ALWAYS_SOFTWARE PAPERSIZE CONDA_PYTHON_EXE LD_LIBRARY_PATH XDG_RUNTIME_DIR PS1 CONDA_DEFAULT_ENV LC_TIME JOURNAL_STREAM XDG_DATA_DIRS PATH GDMSESSION DBUS_SESSION_BUS_ADDRESS LC_NUMERIC OLDPWD _
Layer project found in the current directory. Printing the directory structure:
β”œβ”€β”€ .gitignore
β”œβ”€β”€ models
β”‚   └── model
β”‚       β”œβ”€β”€ requirements.txt
β”‚       β”œβ”€β”€ model.yml
β”‚       └── model.py
β”œβ”€β”€ sndiagnose
β”œβ”€β”€ .layer
β”‚   └── project.yml
β”œβ”€β”€ data
β”‚   β”œβ”€β”€ features
β”‚   β”‚   β”œβ”€β”€ success.sql
β”‚   β”‚   β”œβ”€β”€ month.sql
β”‚   β”‚   β”œβ”€β”€ day.sql
β”‚   β”‚   β”œβ”€β”€ rated.sql
β”‚   β”‚   β”œβ”€β”€ dataset.yml
β”‚   β”‚   β”œβ”€β”€ number_of_directors.sql
β”‚   β”‚   β”œβ”€β”€ release_date.sql
β”‚   β”‚   └── runtime.sql
β”‚   └── movies
β”‚       └── dataset.yaml
β”œβ”€β”€ schema
β”œβ”€β”€ movies_snowflake.csv
β”œβ”€β”€ .git
β”‚   β”œβ”€β”€ COMMIT_EDITMSG
β”‚   β”œβ”€β”€ hooks
β”‚   β”‚   β”œβ”€β”€ post-update.sample
β”‚   β”‚   β”œβ”€β”€ pre-receive.sample
β”‚   β”‚   β”œβ”€β”€ prepare-commit-msg.sample
β”‚   β”‚   β”œβ”€β”€ fsmonitor-watchman.sample
β”‚   β”‚   β”œβ”€β”€ pre-merge-commit.sample
β”‚   β”‚   β”œβ”€β”€ pre-commit.sample
β”‚   β”‚   β”œβ”€β”€ update.sample
β”‚   β”‚   β”œβ”€β”€ pre-rebase.sample
β”‚   β”‚   β”œβ”€β”€ pre-push.sample
β”‚   β”‚   β”œβ”€β”€ commit-msg.sample
β”‚   β”‚   β”œβ”€β”€ applypatch-msg.sample
β”‚   β”‚   └── pre-applypatch.sample
β”‚   β”œβ”€β”€ description
β”‚   β”œβ”€β”€ info
β”‚   β”‚   └── exclude
β”‚   β”œβ”€β”€ HEAD
β”‚   β”œβ”€β”€ logs
β”‚   β”‚   β”œβ”€β”€ HEAD
β”‚   β”‚   └── refs
β”‚   β”‚       β”œβ”€β”€ heads
β”‚   β”‚       β”‚   └── main
β”‚   β”‚       └── remotes
β”‚   β”‚           └── origin
β”‚   β”‚               └── main
β”‚   β”œβ”€β”€ branches
β”‚   β”œβ”€β”€ index
β”‚   β”œβ”€β”€ refs
β”‚   β”‚   β”œβ”€β”€ heads
β”‚   β”‚   β”‚   └── main
β”‚   β”‚   β”œβ”€β”€ remotes
β”‚   β”‚   β”‚   └── origin
β”‚   β”‚   β”‚       └── main
β”‚   β”‚   └── tags
β”‚   β”œβ”€β”€ config
β”‚   └── objects
β”‚       β”œβ”€β”€ 53
β”‚       β”‚   └── 34a78de5ac7067dcb09f1775a53f5af9cd3757
β”‚       β”œβ”€β”€ bc
β”‚       β”‚   └── 5265f04e488e7c1006f4f67fd8d4eb4e876e32
β”‚       β”œβ”€β”€ info
β”‚       β”œβ”€β”€ 98
β”‚       β”‚   └── b1ad4f88eaed1203d778c1cac1aeb5c1c1b0e6
β”‚       β”œβ”€β”€ f7
β”‚       β”‚   └── 31b0b8061cdee80500ed1652d23f4013a43bb3
β”‚       β”œβ”€β”€ 97
β”‚       β”‚   └── 8f26417f802dc699adfb53faade15b4487b0c0
β”‚       β”œβ”€β”€ pack
β”‚       β”œβ”€β”€ 32
β”‚       β”‚   └── 0769546e7e7881ae6d29f7eb7c9c9e204164de
β”‚       β”œβ”€β”€ 8a
β”‚       β”‚   └── a18a00b6a543a55cc876c0196e9eaa658d3b20
β”‚       β”œβ”€β”€ d3
β”‚       β”‚   └── 60230b3d1dd27452476d4d56df3e1f889b0274
β”‚       β”œβ”€β”€ a6
β”‚       β”‚   └── d7ecd9ea8185459ab8c74b2e777994fa54de29
β”‚       β”œβ”€β”€ 1c
β”‚       β”‚   └── ab4ff92f9b6889b63746ba33ad613725b6a62e
β”‚       β”œβ”€β”€ f6
β”‚       β”‚   └── 13d5f969faf8fc0c4c4845db555a7bbb344c0c
β”‚       β”œβ”€β”€ 79
β”‚       β”‚   └── 142d54829a7e42c213ed4e037116675ba7145d
β”‚       β”œβ”€β”€ 93
β”‚       β”‚   └── 113899b2166c845c68a665e32c87689a0ebb3a
β”‚       β”œβ”€β”€ 8e
β”‚       β”‚   └── a945cbd4af2e4a7f15bfe3302b13e20290c163
β”‚       β”œβ”€β”€ 1e
β”‚       β”‚   └── d0e42fa866dc68f4a86290c35561ec013a20c8
β”‚       β”œβ”€β”€ cd
β”‚       β”‚   └── cb420e14947f6b22bf93b936b1fc12608914b0
β”‚       β”œβ”€β”€ 3d
β”‚       β”‚   └── 74b660f0484c10b3508bd1c9ffcd796731fdfc
β”‚       β”œβ”€β”€ ac
β”‚       β”‚   └── 8417053d8cd65bb8a97e91f59cafbd3a57d051
β”‚       β”œβ”€β”€ 70
β”‚       β”‚   └── f292b4f34d5513d2d5e3e1c38e6b8301dbd595
β”‚       β”œβ”€β”€ 92
β”‚       β”‚   β”œβ”€β”€ b5d612f0fa55ea818115f2888f2100af7505d6
β”‚       β”‚   └── 5a8397193fe649e18149cba0a2b8c948add59c
β”‚       β”œβ”€β”€ cf
β”‚       β”‚   └── 4827ed4263ffb32953a59d9c0495828b431e50
β”‚       β”œβ”€β”€ 25
β”‚       β”‚   └── 43ce9180ffc3689671bc5829ed79ada199352c
β”‚       β”œβ”€β”€ f4
β”‚       β”‚   └── 3905c2c388317b4a49e79b5f071d893dd063c4
β”‚       β”œβ”€β”€ 7e
β”‚       β”‚   └── bfb267a17c5c8b410fb2c5949298bbfb9000ba
β”‚       └── 74
β”‚           └── d0d495db7b7fa23e905d06ffbc3dba269ae9cf
β”œβ”€β”€ README.md
β”œβ”€β”€ .~lock.movies_snowflake.csv#
β”œβ”€β”€ snaws.log
└── temp
    β”œβ”€β”€ layer-movies.ipynb
    β”œβ”€β”€ sf
    β”œβ”€β”€ log
    β”œβ”€β”€ Screenshot from 2021-08-20 14-11-01.png
    β”œβ”€β”€ error-features
    β”œβ”€β”€ Screenshot from 2021-08-20 00-40-22.png
    β”œβ”€β”€ drop
    β”œβ”€β”€ Screenshot from 2021-08-19 10-47-36.png
    β”œβ”€β”€ .ipynb_checkpoints
    β”‚   └── layer-movies-checkpoint.ipynb
    β”œβ”€β”€ en-snowflake
    β”œβ”€β”€ Screenshot from 2021-08-24 00-16-03.png
    β”œβ”€β”€ error-model
    └── Screenshot from 2021-08-20 14-11-35.png
/home/gitonga/.layer is found. Contents:
β”œβ”€β”€ logs
β”‚   β”œβ”€β”€ 20210902T134124-session-53a9891c-f7a3-401a-be63-e8e64ad1bb98.log
β”‚   β”œβ”€β”€ 20210906T132852-session-248f49eb-35e9-4364-bc3f-20d21749843e.log
β”‚   β”œβ”€β”€ 20210906T132950-session-4b448116-1b41-4c7d-89fa-845188af51c2.log
β”‚   β”œβ”€β”€ 20210902T132257-session-e9f02451-e97a-43cd-96de-1a05742a5e0e.log
β”‚   β”œβ”€β”€ 20210906T132445-session-2ec53b11-ae8c-4c93-8514-b37ea765624f.log
β”‚   β”œβ”€β”€ 20210830T150450-session-975d4f90-1212-443c-a209-382f38d76894.log
β”‚   β”œβ”€β”€ 20210902T132148-session-a98cd63e-4acc-4433-8d4f-d084af45b559.log
β”‚   β”œβ”€β”€ 20210823T123652-session-48fd76ea-44e3-47c4-84b7-6df2074c3a3f.log
β”‚   β”œβ”€β”€ 20210824T004755-session-64f96888-d099-4e24-9494-3a78f95616ac.log
β”‚   β”œβ”€β”€ 20210906T133016-session-3c91088f-460b-4695-afc0-753e3e832649.log
β”‚   β”œβ”€β”€ 20210902T132344-session-df10dc50-7a3c-4a81-9408-54b7a8b5ab5c.log
β”‚   β”œβ”€β”€ 20210902T132404-session-c9ea0535-2523-4367-9c8c-cb2bd8e986a7.log
β”‚   β”œβ”€β”€ movie-fail.log
β”‚   β”œβ”€β”€ fraud-succeed.log
β”‚   β”œβ”€β”€ 20210830T150402-session-31efec5d-7698-432a-9082-d9c10b582db5.log
β”‚   β”œβ”€β”€ 20210824T001618-session-723a378d-aeb1-4e31-bb56-5248b166bc4b.log
β”‚   β”œβ”€β”€ 20210824T022323-session-ffe66fbd-2c94-4573-91e4-e6d1b374b52c.log
β”‚   β”œβ”€β”€ fail succeed.log
β”‚   β”œβ”€β”€ 20210904T142337-session-ca18f132-5ff7-43f7-a077-ccf908331be4.log
β”‚   β”œβ”€β”€ 20210823T121342-session-a8739f9f-000a-4435-946e-008525dfa18b.log
β”‚   β”œβ”€β”€ 20210824T005121-session-8473c1fb-008f-44fb-84e7-f4858171e3e4.log
β”‚   β”œβ”€β”€ 20210906T132315-session-165ed110-4110-4a83-a940-dbe461c64241.log
β”‚   β”œβ”€β”€ 20210824T005153-session-54603837-25c5-4490-8e7f-b99b92436fea.log
β”‚   β”œβ”€β”€ 20210824T005242-session-ff5d7f40-e0da-4ae9-b97e-5d619c80a40d.log
β”‚   β”œβ”€β”€ 20210906T132013-session-1d0297a5-cdda-4fdd-8e6d-411b348b1c4f.log
β”‚   └── 20210823T235343-session-776fbb66-3429-42a4-8064-31f853330874.log
β”œβ”€β”€ config.json
└── jars
    β”œβ”€β”€ hadoop-aws-3.2.0.jar
    └── aws-java-sdk-bundle-1.11.375.jar
Installed packages: Flask 2.0.1 GitPython 3.1.14 Jinja2 3.0.1 Mako 1.1.4 MarkupSafe 2.0.1 Pillow 8.3.1 PyJWT 1.7.1 PyYAML 5.4.1 Pygments 2.10.0 SQLAlchemy 1.4.23 Werkzeug 2.0.1 aiodns 3.0.0 aiodocker 0.21.0 aiohttp 3.7.4.post0 alembic 1.4.1 async-timeout 3.0.1 attrs 21.2.0 boto3 1.18.25 botocore 1.21.25 brotlipy 0.7.0 cchardet 2.1.7 certifi 2021.5.30 cffi 1.14.6 chardet 4.0.0 charset-normalizer 2.0.4 click 8.0.1 cloudpickle 1.6.0 colorama 0.4.4 commonmark 0.9.1 cryptography 3.4.7 cycler 0.10.0 databricks-cli 0.15.0 decorator 5.0.9 docker 5.0.0 entrypoints 0.3 filelock 3.0.12 gitdb 4.0.7 greenlet 1.1.1 grpcio 1.39.0 grpcio-tools 1.39.0 gunicorn 20.1.0 huggingface-hub 0.0.12 idna 2.10 itsdangerous 2.0.1 jmespath 0.10.0 joblib 1.0.1 jsonschema 3.2.0 kiwisolver 1.3.1 layer-sdk 0.8.1 matplotlib 3.4.3 mlflow 1.19.0 multidict 5.1.0 networkx 2.6.2 numpy 1.21.2 packaging 21.0 pandas 1.3.2 pip 21.2.4 pkg-resources 0.0.0 polling 0.3.2 prometheus-client 0.11.0 prometheus-flask-exporter 0.18.2 prompt-toolkit 3.0.20 protobuf 3.17.3 py4j 0.10.9 pyarrow 5.0.0 pycares 4.0.0 pycparser 2.20 pyparsing 2.4.7 pyrsistent 0.18.0 pyspark 3.1.2 python-dateutil 2.8.2 python-editor 1.0.4 pytz 2021.1 querystring-parser 1.2.4 regex 2021.8.3 requests 2.26.0 rich 10.7.0 s3transfer 0.5.0 sacremoses 0.0.45 scipy 1.7.1 seaborn 0.11.2 setuptools 44.0.0 six 1.16.0 smmap 4.0.0 sqlparse 0.4.1 tabulate 0.8.9 tokenizers 0.10.3 tqdm 4.62.1 transformers 4.9.2 typing-extensions 3.10.0.0 urllib3 1.26.6 validate-email 1.3 validators 0.18.2 wcwidth 0.2.5 websocket-client 1.2.1 wheel 0.34.2 xgboost 1.4.2 yarl 1.6.3
[2021-09-06 10:29:12, model, model-training-2b348696-5969-41dd-97d1-e573bea003b3-s6vpd] Starting job.
[2021-09-06 10:29:13, model, model-training-2b348696-5969-41dd-97d1-e573bea003b3-s6vpd] Using selector: EpollSelector
[2021-09-06 10:29:14, model, model-training-2b348696-5969-41dd-97d1-e573bea003b3-s6vpd] Using selector: EpollSelector
[2021-09-06 10:29:14, model, model-training-2b348696-5969-41dd-97d1-e573bea003b3-s6vpd] Successfully logged into https://beta.layer.co
[2021-09-06 10:29:14, model, model-training-2b348696-5969-41dd-97d1-e573bea003b3-s6vpd] Using selector: EpollSelector
[2021-09-06 10:29:14, model, model-training-2b348696-5969-41dd-97d1-e573bea003b3-s6vpd] Creating ~/source dir
[2021-09-06 10:29:14, model, model-training-2b348696-5969-41dd-97d1-e573bea003b3-s6vpd] Place __init__.py in ~/source
[2021-09-06 10:29:14, model, model-training-2b348696-5969-41dd-97d1-e573bea003b3-s6vpd] Download binary(16d8c5dd-70d5-4040-9112-023226dedd2f/22e4b351-a897-4016-bad1-1a21c9815a4a/movie_success_model_training.tgz) to temp directory
[2021-09-06 10:29:14, model, model-training-2b348696-5969-41dd-97d1-e573bea003b3-s6vpd] Binary archive movie_success_model_training.tgz downloaded and extracted successfully
[2021-09-06 10:29:14, model, model-training-2b348696-5969-41dd-97d1-e573bea003b3-s6vpd] Installing python dependencies
[2021-09-06 10:29:15, model, model-training-2b348696-5969-41dd-97d1-e573bea003b3-s6vpd] Requirement already satisfied: scikit-learn>=0.18 in /venv/lib/python3.8/site-packages (from -r /root/source/requirements.txt (line 1)) (0.24.2)
[2021-09-06 10:29:15, model, model-training-2b348696-5969-41dd-97d1-e573bea003b3-s6vpd] Requirement already satisfied: xgboost>=1.2.0 in /venv/lib/python3.8/site-packages (from -r /root/source/requirements.txt (line 2)) (1.3.3)
[2021-09-06 10:29:15, model, model-training-2b348696-5969-41dd-97d1-e573bea003b3-s6vpd] Requirement already satisfied: scipy>=0.19.1 in /venv/lib/python3.8/site-packages (from scikit-learn>=0.18->-r /root/source/requirements.txt (line 1)) (1.6.0)
[2021-09-06 10:29:15, model, model-training-2b348696-5969-41dd-97d1-e573bea003b3-s6vpd] Requirement already satisfied: threadpoolctl>=2.0.0 in /venv/lib/python3.8/site-packages (from scikit-learn>=0.18->-r /root/source/requirements.txt (line 1)) (2.2.0)
[2021-09-06 10:29:15, model, model-training-2b348696-5969-41dd-97d1-e573bea003b3-s6vpd] Requirement already satisfied: numpy>=1.13.3 in /venv/lib/python3.8/site-packages (from scikit-learn>=0.18->-r /root/source/requirements.txt (line 1)) (1.20.2)
[2021-09-06 10:29:15, model, model-training-2b348696-5969-41dd-97d1-e573bea003b3-s6vpd] Requirement already satisfied: joblib>=0.11 in /venv/lib/python3.8/site-packages (from scikit-learn>=0.18->-r /root/source/requirements.txt (line 1)) (1.0.1)
[2021-09-06 10:29:16, model, model-training-2b348696-5969-41dd-97d1-e573bea003b3-s6vpd] Python dependencies installed successfully
[2021-09-06 10:29:16, model, model-training-2b348696-5969-41dd-97d1-e573bea003b3-s6vpd] Importing user code(model.py) from /source
[2021-09-06 10:29:16, model, model-training-2b348696-5969-41dd-97d1-e573bea003b3-s6vpd] train_model function imported successfully
[2021-09-06 10:29:16, model, model-training-2b348696-5969-41dd-97d1-e573bea003b3-s6vpd] Injecting the dependencies
[2021-09-06 10:29:16, model, model-training-2b348696-5969-41dd-97d1-e573bea003b3-s6vpd] Annotations: {'train': <class 'layer.client.Train'>, 'tf': Featureset(name='features', datasource=DatasourceRef(name='', type=<DatasourceType.STORAGE: 'storage'>, id=UUID('ce43abaa-5ddf-41c2-bb90-e389e9278ad2')), description='', id=UUID('5324d4f7-5c2e-4512-a745-cc3b737fff59'), features=[], feature_names=[], dependencies=[]), 'return': typing.Any}
[2021-09-06 10:29:16, model, model-training-2b348696-5969-41dd-97d1-e573bea003b3-s6vpd] Entity dependencies: {'featuresets': {'tf': Featureset(name='features', datasource=DatasourceRef(name='', type=<DatasourceType.STORAGE: 'storage'>, id=UUID('ce43abaa-5ddf-41c2-bb90-e389e9278ad2')), description='', id=UUID('5324d4f7-5c2e-4512-a745-cc3b737fff59'), features=[], feature_names=[], dependencies=[])}, 'models': {}, 'datasets': {}, 'context': None, 'train': 'train'}
[2021-09-06 10:29:16, model, model-training-2b348696-5969-41dd-97d1-e573bea003b3-s6vpd] Injecting features featureset with individual features []
[2021-09-06 10:29:16, model, model-training-2b348696-5969-41dd-97d1-e573bea003b3-s6vpd] Using selector: EpollSelector
[2021-09-06 10:29:16, model, model-training-2b348696-5969-41dd-97d1-e573bea003b3-s6vpd] Injected dependencies successfully: {'tf': Featureset(name='features', datasource=DatasourceRef(name='', type=<DatasourceType.STORAGE: 'storage'>, id=UUID('ce43abaa-5ddf-41c2-bb90-e389e9278ad2')), description='', id=UUID('57b02882-02e7-4f26-aae8-e9667c2aa9c5'), features=[], feature_names=[], dependencies=[]), 'train': <layer.train.Train object at 0x7fc791d4c0a0>}
[2021-09-06 10:29:16, model, model-training-2b348696-5969-41dd-97d1-e573bea003b3-s6vpd] Executing the train_model
[2021-09-06 10:29:17, model, model-training-2b348696-5969-41dd-97d1-e573bea003b3-s6vpd] Caught exception stacktrace:
[2021-09-06 10:29:17, model, model-training-2b348696-5969-41dd-97d1-e573bea003b3-s6vpd] Failure during train_model execution LayerClientException('Internal error while getting dataset')
[2021-09-06 10:29:17, model, model-training-2b348696-5969-41dd-97d1-e573bea003b3-s6vpd] ['/venv/bin/python -X faulthandler -m pyruntime.model.train_executor' exited with 1]
[2021-09-06 10:29:17, model, model-training-2b348696-5969-41dd-97d1-e573bea003b3-s6vpd] Module execution process return code ->  1
[2021-09-06 10:29:17, model, model-training-2b348696-5969-41dd-97d1-e573bea003b3-s6vpd] Training job completed

Hey @gitonga – we’re about to improve error reporting so that you’ll get something more descriptive than β€œInternal error while getting dataset” soon

Are you sure about this? When I look at the internal logs, I see that it’s BigQuery related. I’d double-check the YAML configs.

This is the repository of the project I ran… GitHub - ericgitonga/snowflake-movies

@gitonga Just out of curiosity, can you please create a new Snowflake datasource, using a different name than layersn, and then try again? I wonder if it’s because you initially started with that name and a BigQuery datasource.

Okay. Let me try that just now.

Same issue…

Screenshot from 2021-09-07 12-05-05

[2021-09-07 09:03:06, model, model-training-9120294e-85ec-42e7-aa08-7339b9404e54-2pp6j] Starting job.
[2021-09-07 09:03:07, model, model-training-9120294e-85ec-42e7-aa08-7339b9404e54-2pp6j] Using selector: EpollSelector
[2021-09-07 09:03:08, model, model-training-9120294e-85ec-42e7-aa08-7339b9404e54-2pp6j] Using selector: EpollSelector
[2021-09-07 09:03:08, model, model-training-9120294e-85ec-42e7-aa08-7339b9404e54-2pp6j] Successfully logged into https://beta.layer.co
[2021-09-07 09:03:08, model, model-training-9120294e-85ec-42e7-aa08-7339b9404e54-2pp6j] Using selector: EpollSelector
[2021-09-07 09:03:08, model, model-training-9120294e-85ec-42e7-aa08-7339b9404e54-2pp6j] Creating ~/source dir
[2021-09-07 09:03:08, model, model-training-9120294e-85ec-42e7-aa08-7339b9404e54-2pp6j] Place __init__.py in ~/source
[2021-09-07 09:03:08, model, model-training-9120294e-85ec-42e7-aa08-7339b9404e54-2pp6j] Download binary(16d8c5dd-70d5-4040-9112-023226dedd2f/22e4b351-a897-4016-bad1-1a21c9815a4a/movie_success_model_training.tgz) to temp directory
[2021-09-07 09:03:08, model, model-training-9120294e-85ec-42e7-aa08-7339b9404e54-2pp6j] Binary archive movie_success_model_training.tgz downloaded and extracted successfully
[2021-09-07 09:03:08, model, model-training-9120294e-85ec-42e7-aa08-7339b9404e54-2pp6j] Installing python dependencies
[2021-09-07 09:03:09, model, model-training-9120294e-85ec-42e7-aa08-7339b9404e54-2pp6j] Requirement already satisfied: scikit-learn>=0.18 in /venv/lib/python3.8/site-packages (from -r /root/source/requirements.txt (line 1)) (0.24.2)
[2021-09-07 09:03:09, model, model-training-9120294e-85ec-42e7-aa08-7339b9404e54-2pp6j] Requirement already satisfied: xgboost>=1.2.0 in /venv/lib/python3.8/site-packages (from -r /root/source/requirements.txt (line 2)) (1.3.3)
[2021-09-07 09:03:09, model, model-training-9120294e-85ec-42e7-aa08-7339b9404e54-2pp6j] Requirement already satisfied: joblib>=0.11 in /venv/lib/python3.8/site-packages (from scikit-learn>=0.18->-r /root/source/requirements.txt (line 1)) (1.0.1)
[2021-09-07 09:03:09, model, model-training-9120294e-85ec-42e7-aa08-7339b9404e54-2pp6j] Requirement already satisfied: numpy>=1.13.3 in /venv/lib/python3.8/site-packages (from scikit-learn>=0.18->-r /root/source/requirements.txt (line 1)) (1.20.2)
[2021-09-07 09:03:09, model, model-training-9120294e-85ec-42e7-aa08-7339b9404e54-2pp6j] Requirement already satisfied: threadpoolctl>=2.0.0 in /venv/lib/python3.8/site-packages (from scikit-learn>=0.18->-r /root/source/requirements.txt (line 1)) (2.2.0)
[2021-09-07 09:03:09, model, model-training-9120294e-85ec-42e7-aa08-7339b9404e54-2pp6j] Requirement already satisfied: scipy>=0.19.1 in /venv/lib/python3.8/site-packages (from scikit-learn>=0.18->-r /root/source/requirements.txt (line 1)) (1.6.0)
[2021-09-07 09:03:10, model, model-training-9120294e-85ec-42e7-aa08-7339b9404e54-2pp6j] Python dependencies installed successfully
[2021-09-07 09:03:10, model, model-training-9120294e-85ec-42e7-aa08-7339b9404e54-2pp6j] Importing user code(model.py) from /source
[2021-09-07 09:03:10, model, model-training-9120294e-85ec-42e7-aa08-7339b9404e54-2pp6j] train_model function imported successfully
[2021-09-07 09:03:10, model, model-training-9120294e-85ec-42e7-aa08-7339b9404e54-2pp6j] Injecting the dependencies
[2021-09-07 09:03:10, model, model-training-9120294e-85ec-42e7-aa08-7339b9404e54-2pp6j] Annotations: {'train': <class 'layer.client.Train'>, 'tf': Featureset(name='features', datasource=DatasourceRef(name='', type=<DatasourceType.STORAGE: 'storage'>, id=UUID('bbbd012e-06f7-41bc-83e6-02f906b88634')), description='', id=UUID('c97bd5e1-b409-470d-8921-488e1bfc53a0'), features=[], feature_names=[], dependencies=[]), 'return': typing.Any}
[2021-09-07 09:03:10, model, model-training-9120294e-85ec-42e7-aa08-7339b9404e54-2pp6j] Entity dependencies: {'featuresets': {'tf': Featureset(name='features', datasource=DatasourceRef(name='', type=<DatasourceType.STORAGE: 'storage'>, id=UUID('bbbd012e-06f7-41bc-83e6-02f906b88634')), description='', id=UUID('c97bd5e1-b409-470d-8921-488e1bfc53a0'), features=[], feature_names=[], dependencies=[])}, 'models': {}, 'datasets': {}, 'context': None, 'train': 'train'}
[2021-09-07 09:03:10, model, model-training-9120294e-85ec-42e7-aa08-7339b9404e54-2pp6j] Injecting features featureset with individual features []
[2021-09-07 09:03:10, model, model-training-9120294e-85ec-42e7-aa08-7339b9404e54-2pp6j] Using selector: EpollSelector
[2021-09-07 09:03:10, model, model-training-9120294e-85ec-42e7-aa08-7339b9404e54-2pp6j] Injected dependencies successfully: {'tf': Featureset(name='features', datasource=DatasourceRef(name='', type=<DatasourceType.STORAGE: 'storage'>, id=UUID('bbbd012e-06f7-41bc-83e6-02f906b88634')), description='', id=UUID('d35a5ea5-d010-4b87-90a4-42c560f6f6eb'), features=[], feature_names=[], dependencies=[]), 'train': <layer.train.Train object at 0x7f45854c4e80>}
[2021-09-07 09:03:10, model, model-training-9120294e-85ec-42e7-aa08-7339b9404e54-2pp6j] Executing the train_model
[2021-09-07 09:03:11, model, model-training-9120294e-85ec-42e7-aa08-7339b9404e54-2pp6j] Caught exception stacktrace:
[2021-09-07 09:03:11, model, model-training-9120294e-85ec-42e7-aa08-7339b9404e54-2pp6j] Failure during train_model execution LayerClientException('Internal error while getting dataset')
[2021-09-07 09:03:11, model, model-training-9120294e-85ec-42e7-aa08-7339b9404e54-2pp6j] ['/venv/bin/python -X faulthandler -m pyruntime.model.train_executor' exited with 1]
[2021-09-07 09:03:11, model, model-training-9120294e-85ec-42e7-aa08-7339b9404e54-2pp6j] Module execution process return code ->  1
[2021-09-07 09:03:11, model, model-training-9120294e-85ec-42e7-aa08-7339b9404e54-2pp6j] Training job completed

This is really weird! We’re looking into it now, I’ll get back to you as soon as we have some findings.

Thanks. Looking forward to hearing from you.

@gitonga we have discovered a bug thanks to your report, and will fix in. In the meantime, can you please try changing the name of your featureset? That should fix the issue.

Thanks for the feedback. I had actually tried using a different name for the featureset, but then I got an error stating that featurset β€œfeatures” could not be found. I have changed it to movie_features as per the screenshots…
Screenshot from 2021-09-08 16-21-05
Screenshot from 2021-09-08 16-23-03
Screenshot from 2021-09-08 16-23-15