https://kedro.org/ logo
Join the conversationJoin Discord
Channels
advanced-need-help
announcements
beginners-need-help
introductions
job-posting
plugins-integrations
random
resources
welcome
Powered by Linen
beginners-need-help
  • n

    noklam

    09/14/2022, 10:41 PM
    The import path is same as any python module, if you dataset definition file is in the init file then you don't need the .datasets prefix
  • n

    noklam

    09/14/2022, 10:41 PM
    Assume it is in project/extras then your path will be just project.extras.dataset_name
  • n

    noklam

    09/14/2022, 10:41 PM
    Same as you do import xxx. Xxx. Xxx
  • m

    mrjpz99

    09/14/2022, 10:45 PM
    hmm but the tutorial says the custom dataset class script has to be created under "{project_name}/extras/datasets" and have two init files under "extras" and "datasets"
  • m

    mrjpz99

    09/14/2022, 10:46 PM
    message has been deleted
  • n

    noklam

    09/14/2022, 10:48 PM
    If you follow the same structure, then you have the right import path. It's a convention, But not a must to name your folder datasets
  • d

    datajoely

    09/14/2022, 10:49 PM
    Yeah behind the scenes were just using importlib, nothing fancy
  • n

    noklam

    09/14/2022, 10:49 PM
    Maybe check your stacktrace and see if you missed some dependency there?
  • m

    mrjpz99

    09/14/2022, 10:58 PM
    hmm how to inspect the stacktrace? I validated that I have all the "import" dependencies from that custom class script though.
  • m

    mrjpz99

    09/15/2022, 8:30 AM
    Do I need to specify anything specific in the "__init__.py" file so kedro will recognize it as a module?
  • d

    datajoely

    09/15/2022, 8:35 AM
    No it's just a standard python import
  • m

    mrjpz99

    09/15/2022, 8:37 AM
    ya I think so - it's empty in the "init.py" file. I still can't figure out why the custom dataset is not working. Do you have any other pointers? Or some example I can refer to?
  • d

    datajoely

    09/15/2022, 8:37 AM
    So something you can do
  • d

    datajoely

    09/15/2022, 8:38 AM
    Jump into an ipython/Jupyter session at the root of the kedro project and try importing the class the standard way... Does that work?
  • m

    mrjpz99

    09/15/2022, 8:49 AM
    Hmm please see the attached. I have to put the "init" file under the
    src
    folder and change the import to
    src.{project_name}.extras.datasets.{custom_dataset_file}
    to make it work in ipython though.
  • m

    mrjpz99

    09/15/2022, 8:52 AM
    but when I updated the
    catalog.yml
    with
    src.{project_name}.extras.datasets.{custom_dataset_file}
    and execute
    kedro run
    in the root directory, I still got the error below
  • m

    mrjpz99

    09/15/2022, 8:55 AM
    Why does the import work in the ipython env but not in kedro with the same path?
  • d

    datajoely

    09/15/2022, 8:56 AM
    Well the prefixed src isn't correct
  • d

    datajoely

    09/15/2022, 8:57 AM
    Are you in the name-matching directory when doing this?
  • m

    mrjpz99

    09/15/2022, 9:00 AM
    Ah I think I fixed something when I tried the "import" in ipython and then I removed the
    src
    prefix again in the catalog.yml. Not it seems working (with a different error.) Thanks for your help!! good debugging tip.:)
  • d

    datajoely

    09/15/2022, 9:00 AM
    What's the error?
  • m

    mrjpz99

    09/15/2022, 9:01 AM
    oh it's just specifically related to my ML project. I think I should be able to figure out. Thanks!
  • d

    datajoely

    09/15/2022, 9:02 AM
    🔥
  • s

    silvadenis

    09/15/2022, 10:25 AM
    Hey people, would like to know if someone already tried to use Spyder inside a Kedro repo, would be nice to have global variables just like a jupyter lab server.
  • d

    datajoely

    09/15/2022, 10:28 AM
    Spyder should work just fine, but most of our users use Vs code and pycharm which can give you the same sort of experience in the debugger mode
  • s

    silvadenis

    09/15/2022, 11:19 AM
    Is there a way from a plain python file to use the catalog.load("..") from kedro? We would like to have a python file for experiments on Scipy but we would like to use Kedro's catalog to reuse the data from the project 🤔
  • d

    datajoely

    09/15/2022, 11:34 AM
    So by design we don't want your nodes to be aware of io, it's much harder to debug if you stop your nodes being pure python functions
  • d

    datajoely

    09/15/2022, 11:34 AM
    You can access the catalog live using kedro hooks
  • d

    datajoely

    09/15/2022, 11:35 AM
    Which may allow you to do what you need
  • s

    silvadenis

    09/15/2022, 12:50 PM
    Thanks, I will take a look!
Powered by Linen
Title
s

silvadenis

09/15/2022, 12:50 PM
Thanks, I will take a look!
View count: 1