https://kedro.org/ logo
Join the conversationJoin Discord
Channels
advanced-need-help
announcements
beginners-need-help
introductions
job-posting
plugins-integrations
random
resources
welcome
Powered by Linen
beginners-need-help
  • c

    Caz

    07/12/2021, 11:17 AM
    Awesome thanks
  • d

    datajoely

    07/12/2021, 12:40 PM
    @User can you DM me your full stack trace / error log?
  • d

    datajoely

    07/12/2021, 1:02 PM
    + steps to reproduce
  • u

    user

    07/14/2021, 2:18 PM
    guys.. I just notice that a custom data catalog I built this week, only works in my host machine but inside the docker the kedro cli cant find the project_root.extras.datasets.mycustom_dataset.MyCustomDataset I am extendind the AbstractDataSet with _load, _describe and _save In catalogs file, I set a path like this project_root.extras.datasets.mycustom_dataset.MyCustomDataset Am I missing something?
  • d

    datajoely

    07/14/2021, 2:19 PM
    It will be a working directory issue, the class path isn't accessible from wherever you are calling Kedro from. If you can run
    pwd
    from inside the container that may help
  • u

    user

    07/14/2021, 2:45 PM
    o dont get it.. pwd inside docker give the root of my project.. and kedro run in my host machine is on root of my project either :/
  • d

    datajoely

    07/14/2021, 2:47 PM
    behind the scenes all we're doing is
    importlib.import_module(project_root.extras.datasets.mycustom_dataset.MyCustomDataset)
    so that's all we need to get working outside of Kedro. Do you have
    __init__.py
    files in every directory?
  • u

    user

    07/14/2021, 2:48 PM
    yep
  • u

    user

    07/14/2021, 2:50 PM
    src does not
  • u

    user

    07/14/2021, 8:24 PM
    do i need to register the extras.datasets in somefile? i followed the doc, I watched the dataengineerone videos about custom data catalog, he is doing exactly the samething i am.. very confusing..
  • d

    datajoely

    07/14/2021, 8:26 PM
    Let me DM you
  • w

    waylonwalker

    07/15/2021, 2:20 PM
    @User can you start a repl in your docker container anf try importing it there. The solution may become more aparant, and easier to iterate on.
    python
    from project_root.extras.datasets.mycustom_dataset import MyCustomDataset
  • u

    user

    07/15/2021, 2:30 PM
    of course
  • u

    user

    07/15/2021, 2:30 PM
    give me a minute
  • u

    user

    07/15/2021, 2:30 PM
    i am entering a meeting right now
  • u

    user

    07/15/2021, 3:01 PM
    sorry
  • u

    user

    07/15/2021, 3:01 PM
    i am back.. let me test it
  • u

    user

    07/15/2021, 3:05 PM
    wow!! hahahaha.. i forgot to add trino dependency in my requirements.. and from repl o could see exactly this error! thank you @User @User i should be able to see that error in kedro run right? the error i was facing was so generic :/
  • d

    datajoely

    07/15/2021, 3:06 PM
    ^ Good point - let me look into that
  • d

    datajoely

    07/15/2021, 4:47 PM
    Thanks! Fix is in scope for next sprint
  • w

    WolVez

    07/19/2021, 10:53 PM
    Quick question, while I know its a bit different, is it possible to load data from the catalog into a pytest unit test?
  • d

    datajoely

    07/19/2021, 10:55 PM
    @WolVez very possible - however many people avoid using real data in unit tests
  • d

    datajoely

    07/19/2021, 10:55 PM
    The best way to do so is to either create a session or DataCatalog object as a fixture
  • j

    julien-e

    07/21/2021, 6:45 AM
    Hi everyone. I have a very basic question regarding the installation and starting a new Kedro project. Does
    pip install kedro
    needs to be executed for every new Kedro project within a dedicated Python virtual environment? In other words, for every new project, are the following steps correct to get started? 1 - Create a new Python virtual environment. For instance, with
    conda
    , with the command:
    conda create --name <environment_name> python=3.7 -y
    2 - Install Kedro (i.e. execute
    pip install kedro
    ) 3 - Create a new Kedro project (e.g. without starter, with the command
    kedro new
    ) I hope my question makes sense. Thank you in advance!
  • d

    datajoely

    07/21/2021, 8:02 AM
    Hi @julien-e yes this is the right way to do things. New venv new kedro - it makes things easier to manage long term when there are different python or kedro versions.
  • d

    datajoely

    07/21/2021, 8:03 AM
    So your process is correct
  • d

    datajoely

    07/21/2021, 8:03 AM
    Between steps 1 and 2 remember to activate your new env
  • j

    julien-e

    07/21/2021, 8:04 AM
    Brilliant, thank you so much for your super quick answer @User 🙇
  • w

    waylonwalker

    07/21/2021, 12:57 PM
    I need some public kedro project with a git history to try
    kedro-diff
    on anyone who can share one, it would be greatly appreciated.
  • d

    datajoely

    07/21/2021, 12:58 PM
    These are dependents from GitHub https://github.com/quantumblacklabs/kedro/network/dependents
Powered by Linen
Title
d

datajoely

07/21/2021, 12:58 PM
These are dependents from GitHub https://github.com/quantumblacklabs/kedro/network/dependents
View count: 1