https://kedro.org/ logo
Join the conversationJoin Discord
Channels
advanced-need-help
announcements
beginners-need-help
introductions
job-posting
plugins-integrations
random
resources
welcome
Powered by Linen
beginners-need-help
  • d

    datajoely

    02/23/2022, 7:11 PM
    let's keep it on github- it's a really good topic to think through as we're going through a wider configuration design effort
  • p

    pypeaday

    02/23/2022, 7:12 PM
    flipping love that Waylon is on here as an emoji!
  • d

    datajoely

    02/23/2022, 7:13 PM
    @User is the šŸ‘‘ king kedroid šŸ™‚
  • p

    pypeaday

    02/23/2022, 7:13 PM
    we work together so I gotta get my stats up publicly I guess!
  • w

    waylonwalker

    02/23/2022, 7:31 PM
    Laughing so hard at this!
  • a

    amos

    02/23/2022, 9:45 PM
    Is there some way to ā€œrunā€ a packaged pipeline from within another python project
  • a

    amos

    02/23/2022, 9:46 PM
    I can ā€œpython -m my_pipelineā€ fine, I’m wondering if I can ā€œimport my_pipelineā€ and then ā€œmy_pipeline.runā€ or something similar
  • d

    datajoely

    02/24/2022, 9:34 AM
    So sort of - you can use the code API and do something like:
    python
    from some_package import kedro_pipeline
    
    from kedro.io import DataCatalog
    from kedro.runner import SequentialRunner
    
    catalog = DataCatalog(data_sets=datasets, feed_dict={"params:split_args": params})
    output = runner.run(pipe, catalog)
  • d

    datajoely

    02/24/2022, 9:35 AM
    but also maybe
    micropackaging
    is what you you are looking for https://kedro.readthedocs.io/en/latest/06_nodes_and_pipelines/04_micro_packaging.html
  • d

    deepyaman

    02/24/2022, 12:26 PM
    Anybody know if if it's possible to auto-sign commits from the GitHub UI? Super annoying to have to go back and rebase just because I forgot to manually add it when applying a suggestion from code review or something.
  • d

    datajoely

    02/24/2022, 12:27 PM
    Do you have permissions to click set DCO to pass?
  • a

    avan-sh

    02/24/2022, 1:01 PM
    I set up git hooks for the repo last I needed it. This doc has a ui based option https://github.com/src-d/guide/blob/master/developer-community/fix-DCO.md
  • a

    antony.milne

    02/24/2022, 1:51 PM
    @User thanks for pointing that out - I hadn't seen the Chrome extension, and I thought I had looked into every avenue for making DCO sign-off as painless as possible šŸ˜… @User if this becomes very annoying we might be able to make the "Set DCO to pass" button that @User mentioned available to members outside kedro-team (you won't be able to see it at the moment). Or, given that GitHub UI commits are GPG Verified anyway, we *might *be able to extend the configuration that makes non-explicitly signed-off commits still pass DCO (as they do already for kedro-team commits). Right now if you don't want to go through the rebase hassle just ask us to press the "DCO pass" button for you and it won't matter that you forgot to sign off previous commits though
  • a

    antony.milne

    02/24/2022, 2:05 PM
    @User in addition to what Joel said, if you do
    kedro package
    on your project (let's say its Python package name is
    xyz
    ) then you can actually do this:
    from xyz import __main__
    __main__.main()
    which will be equivalent to
    python -m xyz
    . This does seem a little ugly though. IIRC @User wrote the stuff on
    __main__
    not so long ago so might know a better way of doing this (or if there isn't one then maybe we should expose the
    main
    function so I can do
    from xyz import main
    directly)
  • w

    williamc

    02/24/2022, 4:38 PM
    Hey guys, I have this question: The end step of my pipeline is launching a Slackbot. At present I do this on a python script that loads some ML models and gets a connection to the Slack API. As such, it's a long lived process. Given this, would a pipeline node be the right place to launch the script or would you recommend to keep it as a separate step, outside my Kedro pipelines? Thanks
  • d

    datajoely

    02/24/2022, 4:41 PM
    So the answer is I’m not sure! Cool usecase either way. If you are going to do this in kedro an
    after_pipeline_run
    hook might be best
  • w

    williamc

    02/24/2022, 4:42 PM
    Gotcha, hadn't considered a hook but it sounds like a great idea. Thanks!
  • a

    amos

    02/24/2022, 5:14 PM
    What are ā€œsome packageā€ and ā€œkedro_pipelineā€ in this example?
  • d

    datajoely

    02/24/2022, 5:19 PM
    A python package in your source code or site packages which you can import a pipeline object
  • a

    amos

    02/24/2022, 5:21 PM
    So this wouldn’t work where I’ve pip-installed the packaged pipeline, just the original project?
  • d

    datajoely

    02/24/2022, 5:22 PM
    Yeah as far as I know it wouldn’t work. It’s worth trying to be sure.
  • a

    amos

    02/24/2022, 5:25 PM
    Okay thanks
  • a

    amos

    02/24/2022, 6:08 PM
    I feel like I’m going down the wrong path here. I need to create a ā€œrunā€ button, with a few sliders so a user can change the params and kick off new runs. Is there a smart way to do this?
  • d

    datajoely

    02/24/2022, 6:09 PM
    https://kedro.readthedocs.io/en/stable/09_development/03_commands_reference.html#run-the-project
  • d

    datajoely

    02/24/2022, 6:09 PM
    kedro run --params param_key1:value1,param_key2:2.0
  • d

    datajoely

    02/24/2022, 6:10 PM
    Or use the config file approach
  • a

    amos

    02/24/2022, 6:10 PM
    I’m having no problem running it from the command line, I need to drop it in a plotly dash callback
  • d

    datajoely

    02/24/2022, 6:11 PM
    Hmm it’s a very good point - I’ve wanted to make this easier for a long time
  • a

    amos

    02/24/2022, 6:11 PM
    Or something similar, some way to bolt a front end into it
  • d

    datajoely

    02/24/2022, 6:11 PM
    I might have to get back to you in the morning it’s end of day here in London
Powered by Linen
Title
d

datajoely

02/24/2022, 6:11 PM
I might have to get back to you in the morning it’s end of day here in London
View count: 1