Error While Running layer run on MacOS

I have successfully installed Layer 0.7.0, after installing all the dependencies, I had the following error while running layer run:

(env) ➜  creditScoringLayer layer run            
Layer 0.7.0 using https://beta.layer.co
Loading the project under /Users/omarmhaimdat/Documents/creditScoringLayer/creditScoringLayer
Run session id: a95bdd17-d4f2-4f06-a7f3-31455a730eaf
No module named 'pyarrow._flight'

I did check if the pyarrow is installed:

(env) ➜  creditScoringLayer pip freeze | grep pyarrow
pyarrow==5.0.0

Hey Omar, we have taken a look and struggled to reproduce the same issue.

Could it be that you have a separate python (e.g. from an alias) that is being used by layer? This could cause it not to use the python env you are expecting it to use.

To help debug the issue, can you perhaps share the output of layer diagnose in the failing project?

As shared by Omar

(env) ➜  cs_training git:(main) layer diagnose
Local diagnostics for Layer 0.7.0
OS: Darwin 20.6.0
Platform: uname_result(system='Darwin', node='omars-mbp', release='20.6.0', version='Darwin Kernel Version 20.6.0: Wed Jun 23 00:26:27 PDT 2021; root:xnu-7195.141.2~5/RELEASE_ARM64_T8101', machine='arm64')
Interpreter path: /opt/homebrew/opt/python@3.9/bin/python3.9
Interpreter version: 3.9.5 (default, May  3 2021, 19:12:05)
[Clang 12.0.5 (clang-1205.0.22.9)]
Timezone: +01
Environment variable keys: TERM_SESSION_ID SSH_AUTH_SOCK LC_TERMINAL_VERSION COLORFGBG ITERM_PROFILE XPC_FLAGS PWD SHELL __CFBundleIdentifier LC_CTYPE TERM_PROGRAM_VERSION TERM_PROGRAM PATH LC_TERMINAL COLORTERM COMMAND_MODE TERM HOME TMPDIR USER XPC_SERVICE_NAME LOGNAME ITERM_SESSION_ID __CF_USER_TEXT_ENCODING SHLVL OLDPWD HOMEBREW_PREFIX HOMEBREW_CELLAR HOMEBREW_REPOSITORY MANPATH INFOPATH ZSH PAGER LESS LSCOLORS PIXICLI CONDA_EXE _CE_M _CE_CONDA CONDA_PYTHON_EXE CONDA_SHLVL CONDA_PREFIX CONDA_DEFAULT_ENV CONDA_PROMPT_MODIFIER CONDA_PREFIX_1 _
Layer project found in the current directory. Printing the directory structure:
├── .layer
│   └── project.yml
├── README.md
├── .git
│   ├── config
│   ├── objects
│   │   ├── pack
│   │   │   ├── pack-db790418fde31c83d8cea54af6fce6016ae995c0.idx
│   │   │   └── pack-db790418fde31c83d8cea54af6fce6016ae995c0.pack
│   │   └── info
│   ├── HEAD
│   ├── info
│   │   └── exclude
│   ├── logs
│   │   ├── HEAD
│   │   └── refs
│   │       ├── heads
│   │       │   └── main
│   │       └── remotes
│   │           └── origin
│   │               └── HEAD
│   ├── description
│   ├── hooks
│   │   ├── commit-msg.sample
│   │   ├── pre-rebase.sample
│   │   ├── pre-commit.sample
│   │   ├── applypatch-msg.sample
│   │   ├── fsmonitor-watchman.sample
│   │   ├── pre-receive.sample
│   │   ├── prepare-commit-msg.sample
│   │   ├── post-update.sample
│   │   ├── pre-merge-commit.sample
│   │   ├── pre-applypatch.sample
│   │   ├── pre-push.sample
│   │   ├── update.sample
│   │   └── push-to-checkout.sample
│   ├── refs
│   │   ├── heads
│   │   │   └── main
│   │   ├── tags
│   │   └── remotes
│   │       └── origin
│   │           └── HEAD
│   ├── index
│   └── packed-refs
└── data
    ├── features
    │   ├── ageband.sql
    │   └── dataset.yml
    └── cs_training
        └── dataset.yaml
/Users/omarmhaimdat/.layer is found. Contents:
└── config.json
Installed packages: Flask 2.0.1 GitPython 3.1.14 Jinja2 3.0.1 Mako 1.1.4 MarkupSafe 2.0.1 PyJWT 1.7.1 PyYAML 5.4.1 Pygments 2.9.0 SQLAlchemy 1.4.22 Werkzeug 2.0.1 aiodns 3.0.0 aiodocker 0.21.0 aiohttp 3.7.4.post0 alembic 1.4.1 async-timeout 3.0.1 attrs 21.2.0 boto3 1.18.17 botocore 1.21.17 brotlipy 0.7.0 cchardet 2.1.7 certifi 2021.5.30 cffi 1.14.6 chardet 4.0.0 charset-normalizer 2.0.4 click 8.0.1 cloudpickle 1.6.0 colorama 0.4.4 commonmark 0.9.1 cryptography 3.4.7 databricks-cli 0.15.0 decorator 5.0.9 docker 5.0.0 entrypoints 0.3 filelock 3.0.12 gitdb 4.0.7 greenlet 1.1.1 grpcio 1.39.0 grpcio-tools 1.39.0 gunicorn 20.1.0 huggingface-hub 0.0.12 idna 2.10 itsdangerous 2.0.1 jmespath 0.10.0 joblib 1.0.1 layer-sdk 0.7.0 mlflow 1.19.0 multidict 5.1.0 networkx 2.6.2 numpy 1.21.1 packaging 21.0 pandas 1.3.1 pip 21.2.3 polling 0.3.2 prometheus-client 0.11.0 prometheus-flask-exporter 0.18.2 prompt-toolkit 3.0.19 protobuf 3.17.3 py4j 0.10.9 pyarrow 5.0.0 pycares 4.0.0 pycparser 2.20 pyparsing 2.4.7 pyspark 3.1.2 python-dateutil 2.8.2 python-editor 1.0.4 pytz 2021.1 querystring-parser 1.2.4 regex 2021.8.3 requests 2.26.0 rich 10.7.0 s3transfer 0.5.0 sacremoses 0.0.45 setuptools 56.0.0 six 1.16.0 smmap 4.0.0 sqlparse 0.4.1 tabulate 0.8.9 tokenizers 0.10.3 tqdm 4.62.0 transformers 4.9.2 typing-extensions 3.10.0.0 urllib3 1.26.6 validate-email 1.3 validators 0.18.2 wcwidth 0.2.5 websocket-client 1.1.1 wheel 0.36.2 yarl 1.6.3


@omarmhaimdat , maybe something happened to your local pyarrow distribution. Can you check where it’s located first. For example, I do this with:
$ pip list -v | grep pyarrow
pyarrow 5.0.0 /home/yuranos/.local/lib/python3.8/site-packages pip

Then, proceed to that installation folder and see if you have precompiled arrow files in it. Specifically, on my machine I have:
_flight.cpython-38-x86_64-linux-gnu.so
_flight.pyx
If I, for example, remove _flight.cpython-38-x86_64-linux-gnu.so, I have exactly the same error you do.
In that case, you should try to force reinstall pyarrow. I’d do it with:
$ pip install --upgrade --force-reinstall pyarrow==5.0

It seems to be present but, I did process to force reinstall it anyway, and got the same error

Hey @omarmhaimdat , we’d like to look deeper into this but are unable to reproduce it. Would you mind giving us more details on your python environment?

  • Can you share the output of which python?
  • Can you share the output of which layer?
  • Do you have/use pyenv? Can you try if not?

Any other reproduction steps you can think of would help us debug your environment.

Thanks!

Hey @omarmhaimdat, haven’t heard back from you so going to close this topic. I hope the issue is resolve, if it is not then please just create a new topic and we’ll help you asap.