beginners-need-help
- n
noklam
09/14/2022, 10:41 PMThe import path is same as any python module, if you dataset definition file is in the init file then you don't need the .datasets prefix - n
noklam
09/14/2022, 10:41 PMAssume it is in project/extras then your path will be just project.extras.dataset_name - n
noklam
09/14/2022, 10:41 PMSame as you do import xxx. Xxx. Xxx - m
mrjpz99
09/14/2022, 10:45 PMhmm but the tutorial says the custom dataset class script has to be created under "{project_name}/extras/datasets" and have two init files under "extras" and "datasets" - m
mrjpz99
09/14/2022, 10:46 PMmessage has been deleted - n
noklam
09/14/2022, 10:48 PMIf you follow the same structure, then you have the right import path. It's a convention, But not a must to name your folder datasets - d
datajoely
09/14/2022, 10:49 PMYeah behind the scenes were just using importlib, nothing fancy - n
noklam
09/14/2022, 10:49 PMMaybe check your stacktrace and see if you missed some dependency there? - m
mrjpz99
09/14/2022, 10:58 PMhmm how to inspect the stacktrace? I validated that I have all the "import" dependencies from that custom class script though. - m
mrjpz99
09/15/2022, 8:30 AMDo I need to specify anything specific in the "__init__.py" file so kedro will recognize it as a module? - d
datajoely
09/15/2022, 8:35 AMNo it's just a standard python import - m
mrjpz99
09/15/2022, 8:37 AMya I think so - it's empty in the "init.py" file. I still can't figure out why the custom dataset is not working. Do you have any other pointers? Or some example I can refer to? - d
datajoely
09/15/2022, 8:37 AMSo something you can do - d
datajoely
09/15/2022, 8:38 AMJump into an ipython/Jupyter session at the root of the kedro project and try importing the class the standard way... Does that work? - m
mrjpz99
09/15/2022, 8:49 AMHmm please see the attached. I have to put the "init" file under the
folder and change the import tosrc
to make it work in ipython though.src.{project_name}.extras.datasets.{custom_dataset_file} - m
mrjpz99
09/15/2022, 8:52 AMbut when I updated the
withcatalog.yml
and executesrc.{project_name}.extras.datasets.{custom_dataset_file}
in the root directory, I still got the error belowkedro run - m
mrjpz99
09/15/2022, 8:55 AMWhy does the import work in the ipython env but not in kedro with the same path? - d
datajoely
09/15/2022, 8:56 AMWell the prefixed src isn't correct - d
datajoely
09/15/2022, 8:57 AMAre you in the name-matching directory when doing this? - m
mrjpz99
09/15/2022, 9:00 AMAh I think I fixed something when I tried the "import" in ipython and then I removed the
prefix again in the catalog.yml. Not it seems working (with a different error.) Thanks for your help!! good debugging tip.:)src - d
datajoely
09/15/2022, 9:00 AMWhat's the error? - m
mrjpz99
09/15/2022, 9:01 AMoh it's just specifically related to my ML project. I think I should be able to figure out. Thanks! - d
datajoely
09/15/2022, 9:02 AM🔥 - s
silvadenis
09/15/2022, 10:25 AMHey people, would like to know if someone already tried to use Spyder inside a Kedro repo, would be nice to have global variables just like a jupyter lab server. - d
datajoely
09/15/2022, 10:28 AMSpyder should work just fine, but most of our users use Vs code and pycharm which can give you the same sort of experience in the debugger mode - s
silvadenis
09/15/2022, 11:19 AMIs there a way from a plain python file to use the catalog.load("..") from kedro? We would like to have a python file for experiments on Scipy but we would like to use Kedro's catalog to reuse the data from the project 🤔 - d
datajoely
09/15/2022, 11:34 AMSo by design we don't want your nodes to be aware of io, it's much harder to debug if you stop your nodes being pure python functions - d
datajoely
09/15/2022, 11:34 AMYou can access the catalog live using kedro hooks - d
datajoely
09/15/2022, 11:35 AMWhich may allow you to do what you need - s
silvadenis
09/15/2022, 12:50 PMThanks, I will take a look!
Title
s
silvadenis
09/15/2022, 12:50 PMThanks, I will take a look!
View count: 1