Failed To Build Featureset - Internal Error

I’m getting an error with the internal dataset store when I perform layer run. I believe this is an issue on the backend! - Image below

Project can be access via this Github Repository - Credit-Approval/credit-approval at main · kurtispykes/Credit-Approval · GitHub

Thank you Kurtis! We’re looking into this now.

1 Like

Hi Kurtis,

Thanks for reporting this issue.
I see that you are using an outdated version of the layer cli (v.0.6.9).
Could you upgrade to the latest version (v0.7.2) by pip install layer-sdk --upgrade and try again?
The new CLI reports a build-id that we can use to trace the logs and will help us finding what’s the problem with your build.

Please let us know if the problem persists.

kind regards, Gerard.

1 Like

I’ve upgraded and run the command. Hopefully, that provides more information.

Hi Kurtis,

Thanks for upgrading your client.
I did some further research and I found out that the issue is related to the OHE and the way we store features. It turns out that we currently don’t support uint8 types and OHE creates uint8 columns as the result of the encoding.
Could you try explicitly casting the uint8 to int. For example, on feature_one, we could do the following diff:

- df = pd.concat([df["ID"], features], axis=1)
+ df = pd.concat([df["ID"], features], axis=1).astype({'feature_one':'int'})

Likewise with the other features.
We have a ticket in our backlog to implement the uint8 support in the near future.

Let us know if this solves your issue.

kind regards, Gerard.

1 Like

That solved the issue. Thank you… Now, there’s a new error. screenshot attached below. I believe this is an error on my end, I’ll check out my scripts

Hi Kurtis,

That’s great news. Thanks for letting us know.
Now that you have the new CLI, you have an execution log file for each run.
if you grep for 'StartRunyou will find thex-request-id` related to your project run.
If you could provide me with that id, I could have a look at the python execution log.

Let us know if we can help with anything else.

kind regards, Gerard.

Sorry, you’ve lost me. Do you mean I should receive the details of the x-request-id in the Model Catalog?

Hi Kurtis,

I’m sorry for the confusion. I should have explained better.
The latest CLI creates a “session log” (highlighted in your screenshot)
If you open that log in an editor (or use ‘grep’) and look for the “StartRun” entry, it will contain a x-request-id attribute with an UUID as value. With that x-request-id we can better trace your execution in the backend.
So, if you are still having issues with the model train, could you do a fresh run and provide us with the x-request-id? That way, we will --more easily-- trace it to the containers that executed your code.

Let me know if this is helpful.

kind regards, Gerard.

1 Like

ooops, I forgot the screenshot I referred to.

“x-request-id”: “fe2d20d7-678a-4b46-a4c9-c80e3be8639a”

Thanks for the reply, Kurtis,

I searched the logs and found a hyper-parameter tuning job running but I didn’t see it finalizing.

  data: "{\"objective\":{\"name\":\"None\",\"is_maximization\":false},\"fixed_parameters\":{},\"strategy\":{\"max_jobs\":5,\"parameters\":{\"max_depth\":{\"type\":\"float\",\"min\":3.0,\"max\":20.0},\"n_estimators\":{\"type\":\"integer\",\"min\":100,\"max\":300}},\"categorical_parameters\":{},\"type\":\"random\"},\"max_parallel_jobs\":0,\"early_stop\":false}"
  status: STATUS_RUNNING

Is that what you are seeing? That the training phase is not ending? Or do you get an error (maybe the error is getting lost in our system?)

Let us know,

kind regards, Gerard.

That’s correct! It takes a very long time then after a while, training stops and it says error id: b'', details: keepalive watchdog timeout

Thanks for confirming that behavior. I’ll pass this info to the dev team in charge of hyper parameter tuning.
It might take us a few days to get back to you on that particular issue.I’m sorry for the inconvenience.

kind regards, Gerard

1 Like

Hey @kurtispykes, we’re still working on this bug and will keep you updated as we make progress. Thank you for your patience!

Hi @kurtispykes,

The issue with error id: b'', details: keepalive watchdog timeout should now be resolved with our newest sdk version.

Can you please upgrade your layer-sdk (by doing pip install --upgrade layer-sdk for example) and let us know if you still experience this issue?

Thank you very much,
Kind regards,

Adrien