https://kedro.org/ logo
Join the conversationJoin Discord
Channels
advanced-need-help
announcements
beginners-need-help
introductions
job-posting
plugins-integrations
random
resources
welcome
Powered by Linen
beginners-need-help
  • d

    datajoely

    07/27/2021, 10:58 AM
    Some other discussions here that may be relevant https://stackoverflow.com/questions/59768146/how-to-run-startup-commands-on-loading-python-interactive-window-in-vscode
  • j

    jcasanuevam

    07/27/2021, 11:00 AM
    I think this is good to me! thanks 🙂
  • w

    WolVez

    07/28/2021, 6:40 PM
    Hi, I am getting a "DLL Load failed while importing win32api" error when attempting to utilize a kedro jupyter notebook kernal. I currently have winpy32 301 installed from the kedro install process, is there an version that you all recommend using that may be more stable?
    d
    • 2
    • 9
  • w

    waylonwalker

    07/29/2021, 1:15 AM
    @User can you run jupyter outside of kedro? i.e.
    jupyter notebook
    or
    jupyter lab
  • w

    waylonwalker

    07/29/2021, 1:17 AM
    @User I modified our teams template to have a good __main__file such that running
    %run -m project_name
    gets you all the same things you get from running
    kedro ipython
    might be worth looking into that.
  • d

    datajoely

    07/29/2021, 8:34 AM
    Hi @User we're just looking into fixing issue and am unable to to reproduce. What version of Kedro were you running?
    • 1
    • 2
  • j

    jcasanuevam

    07/30/2021, 8:54 AM
    Hello guys! I got a question related to the kedro-airflow plugin but I'm not sure if this is the place to ask for...my company has Airflow installed in an internal server using Docker. Kedro docs says that we should install our kedro packaged project inside the airflow environment, but that means we should change the Airflow Docker Image every time we want to install a new package, module, etc. Is there a way to manage that scenario without modifying the Airflow DockerImage/Compose file??
    d
    • 2
    • 132
  • d

    datajoely

    07/30/2021, 11:18 AM
    Airflow question
  • w

    waylonwalker

    07/30/2021, 12:43 PM
    Is it possible to configure the default dataset used by kedro? Is there a way to make
    kedro catalog create
    generate something other than
    MemoryDataSets
    ?
  • w

    waylonwalker

    07/30/2021, 12:50 PM
    Found it, its hard coded. https://github.com/quantumblacklabs/kedro/blob/master/kedro/framework/cli/catalog.py#L196
  • d

    datajoely

    07/30/2021, 12:50 PM
    What would you like to generate instead?
  • w

    waylonwalker

    07/30/2021, 1:02 PM
    parquet
  • w

    waylonwalker

    07/30/2021, 1:02 PM
    I am making an issue, now if you want to join in. or maybe I can do a discord call.
  • w

    waylonwalker

    07/30/2021, 1:03 PM
    twitch.tv/waylonwalker
  • d

    datajoely

    07/30/2021, 1:03 PM
    I'm on a call already! so only able to respond here
  • d

    datajoely

    07/30/2021, 1:03 PM
    so I think we're unlikely to change the default CLI command for this, but this is a good example of where you could override it for your own purposes
  • w

    waylonwalker

    07/30/2021, 1:04 PM
    I don't want to change the cli command.
  • d

    datajoely

    07/30/2021, 1:05 PM
    ah please create a GH issue 🙂
  • w

    waylonwalker

    07/30/2021, 1:07 PM
    https://github.com/quantumblacklabs/kedro/blob/master/kedro/framework/cli/catalog.py#L196 I want to not hard code line 196, and pull defaults from settings.py. It would be a trivial change to switch the type over to something configurable, but being able to configure filepath per dataset is what is not trivial. I'll brainstorm a solution in the readme and you can tell me how far off base I am.
  • d

    datajoely

    07/30/2021, 1:13 PM
    I would personally think doing something like
    kedro catalog generate --type pandas.ParquetDataSet
    would be a neat way of doing this
  • d

    datajoely

    07/30/2021, 1:14 PM
    however, the core team are quite reluctant to make changes to the CLI overall
  • w

    waylonwalker

    07/30/2021, 1:36 PM
    https://github.com/quantumblacklabs/kedro/issues/849 hashed out a quick idea. Let me know what you think.
  • w

    waylonwalker

    07/30/2021, 1:36 PM
    No change to the cli at all
  • d

    datajoely

    07/30/2021, 1:37 PM
    Thanks!
  • w

    waylonwalker

    07/30/2021, 1:37 PM
    I think this would really improve the onramp of new users onto a kedro team, as the catalog could be mostly automated.
  • d

    datajoely

    07/30/2021, 1:39 PM
    So with my PM hat on this feels like a symptom of our YAML config overhead. This is a key research/development focus for the second half of the year so stay tuned
  • u

    user

    07/30/2021, 6:01 PM
    hey guys
  • u

    user

    07/30/2021, 6:01 PM
    where is a best place to put reusable personal content in kedro structure?
  • u

    user

    07/30/2021, 6:02 PM
    i am leaving in pipelines level or extras package.. but id like to know how do you guys deal with that
  • d

    datajoely

    07/30/2021, 6:02 PM
    We put utility functions in a folder under src
Powered by Linen
Title
d

datajoely

07/30/2021, 6:02 PM
We put utility functions in a folder under src
View count: 2