https://kedro.org/ logo
#advanced-need-help
Title
# advanced-need-help
n

Nick Sieraad

07/28/2022, 7:43 AM
Hi all, I am using Kedro v0.18.2 and I have a question. When I want to config the catalog I got an error (Error.png). The error is in the globals.yml (globals yml.png) where I specified the dataset that is initialised in my folder 'base' (__init__ from base). When the file, where I config the catalog, is a folder level above 'base' it works, but when the file (nodes.py) is on a lower level (Folder structure) it doesn't work. I tried to specify the whole path in the globals.yml, but that doesn't work. I hope you can help me out!
n

noklam

07/28/2022, 11:01 AM
let's move it here.
n

Nick Sieraad

07/28/2022, 11:02 AM
Good one!
n

noklam

07/28/2022, 11:02 AM
Is the problem with your custom dataset only?
n

Nick Sieraad

07/28/2022, 11:03 AM
No also with the ImageDataSet
n

noklam

07/28/2022, 11:04 AM
How are you running the program to get the error that you get above?
n

Nick Sieraad

07/28/2022, 11:05 AM
python src/certifai/pipelines/data/nodes.py
I am trying to run nodes.py
So, when I hard code like this "jpg: kedro.extras.datasets.pillow.ImageDataSet" it is working. But I import it from base. From globals.yml the templated catalog is not reading it. The normal datasets work, but I still have to import the custom datasets from base.
n

noklam

07/28/2022, 12:21 PM
I am a bit confused, can you share your
nodes.py
? It's shouldn't import any datasets
n

Nick Sieraad

07/28/2022, 12:25 PM
Maybe I am confused. Where should I load catalogs?
n

noklam

07/28/2022, 12:39 PM
Without any context, I am just assuming you are trying to run a kedro pipeline with custom dataset implementation.
If so,
kedro run
is still the entrypoint, and you don't need to load up
catalog
in any of your code.
Assuming your screenshot above is under your package folder, which I call "x" Then your dataset type should be named:
x.base.custom_datasets.THE_CLASS_NAME
the
type
argument you put int
catalog.yml
is just the path to import any class, same as you do
import pandas
. For Kedro's built-in datasets, we provide a shortcut so instead of
kedro.extras.datasets.pandas.CSVDataSet
, you can do
pandas.CSVDataSet
.
n

Nick Sieraad

07/29/2022, 6:36 AM
Perfect! This worked! Thank you very much! It reads my custom_dataset now!
n

noklam

07/29/2022, 11:12 AM
Awesome! Glad it helps
2 Views