Dynamic parameters

I'm trying to get the space tutorial to run in Kubeflow using my namespace's internal Minio service for data storage. When I run the

build_kubeflow_pipeline.py

script, there are many references to AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in the generated kfp yaml file. But I'm not using AWS at all. Why are these in there?

Hey I'm a newbie and experimenting with kedro. I follow the tutorial here - https://kedro.readthedocs.io/en/stable/extend_kedro/custom_datasets.html to create a custom dataset object to store the "sentence-transformer" model for my project. But when I run

kedro viz

to visualize the pipeline, I got this error as the screenshot shows. I configured the "catalog.yml" with the filepath pointing to the dataset class script. Am I doing anything wrong? Or is it a bug in "kedro-viz" that it can't handle custom dataset since it's looking for the specific installed package?

Another question, does kedro support huggingface "transformer" / "sentence-transformer" model class natively? I didn't see it the list - https://kedro.readthedocs.io/en/stable/kedro.extras.datasets.html

If you run the pipeline does it work?

No. It also gives the

DataSetError: An exception occurred when parsing config for DataSet
'name_match_model':
Class
'name_matching_v2.extras.datasets.transformer_dataset.SentenceTransformerModel' not
found or one of its dependencies has not been installed.

Okay so the error message is valid

Either the classpath is wrong or one of imports is bad

hmm I have the "__init__.py" file under the

{project_name}/extras

folder. Then I specify the

type

of the model artifact in the

catalog.yml

file as

{project_name}.extras.datasets.{custom_dataset.py}.{custom_dataset_class}

. Anything else I missed?

The import path is same as any python module, if you dataset definition file is in the init file then you don't need the .datasets prefix

Assume it is in project/extras then your path will be just project.extras.dataset_name

Same as you do import xxx. Xxx. Xxx

hmm but the tutorial says the custom dataset class script has to be created under "{project_name}/extras/datasets" and have two init files under "extras" and "datasets"

If you follow the same structure, then you have the right import path. It's a convention, But not a must to name your folder datasets

Yeah behind the scenes were just using importlib, nothing fancy

Maybe check your stacktrace and see if you missed some dependency there?

hmm how to inspect the stacktrace? I validated that I have all the "import" dependencies from that custom class script though.

Do I need to specify anything specific in the "__init__.py" file so kedro will recognize it as a module?

No it's just a standard python import

ya I think so - it's empty in the "init.py" file. I still can't figure out why the custom dataset is not working. Do you have any other pointers? Or some example I can refer to?

So something you can do

Jump into an ipython/Jupyter session at the root of the kedro project and try importing the class the standard way... Does that work?

Hmm please see the attached. I have to put the "init" file under the

src

folder and change the import to

src.{project_name}.extras.datasets.{custom_dataset_file}

to make it work in ipython though.

but when I updated the

catalog.yml

with

src.{project_name}.extras.datasets.{custom_dataset_file}

and execute

kedro run

in the root directory, I still got the error below

Why does the import work in the ipython env but not in kedro with the same path?

Well the prefixed src isn't correct

Are you in the name-matching directory when doing this?

Ah I think I fixed something when I tried the "import" in ipython and then I removed the

src

prefix again in the catalog.yml. Not it seems working (with a different error.) Thanks for your help!! good debugging tip.:)

What's the error?

oh it's just specifically related to my ML project. I think I should be able to figure out. Thanks!