let's keep it on github- it's a really good topic to think through as we're going through a wider configuration design effort

flipping love that Waylon is on here as an emoji!

@User is the 👑 king kedroid 🙂

we work together so I gotta get my stats up publicly I guess!

Laughing so hard at this!

Is there some way to “run” a packaged pipeline from within another python project

I can “python -m my_pipeline” fine, I’m wondering if I can “import my_pipeline” and then “my_pipeline.run” or something similar

So sort of - you can use the code API and do something like:

python
from some_package import kedro_pipeline

from kedro.io import DataCatalog
from kedro.runner import SequentialRunner

catalog = DataCatalog(data_sets=datasets, feed_dict={"params:split_args": params})
output = runner.run(pipe, catalog)

but also maybe

micropackaging

is what you you are looking for https://kedro.readthedocs.io/en/latest/06_nodes_and_pipelines/04_micro_packaging.html

Anybody know if if it's possible to auto-sign commits from the GitHub UI? Super annoying to have to go back and rebase just because I forgot to manually add it when applying a suggestion from code review or something.

Do you have permissions to click set DCO to pass?

I set up git hooks for the repo last I needed it. This doc has a ui based option https://github.com/src-d/guide/blob/master/developer-community/fix-DCO.md

@User thanks for pointing that out - I hadn't seen the Chrome extension, and I thought I had looked into every avenue for making DCO sign-off as painless as possible 😅 @User if this becomes very annoying we might be able to make the "Set DCO to pass" button that @User mentioned available to members outside kedro-team (you won't be able to see it at the moment). Or, given that GitHub UI commits are GPG Verified anyway, we *might *be able to extend the configuration that makes non-explicitly signed-off commits still pass DCO (as they do already for kedro-team commits). Right now if you don't want to go through the rebase hassle just ask us to press the "DCO pass" button for you and it won't matter that you forgot to sign off previous commits though

@User in addition to what Joel said, if you do

kedro package

on your project (let's say its Python package name is

xyz

) then you can actually do this:

from xyz import __main__
__main__.main()

which will be equivalent to

python -m xyz

. This does seem a little ugly though. IIRC @User wrote the stuff on

__main__

not so long ago so might know a better way of doing this (or if there isn't one then maybe we should expose the

main

function so I can do

from xyz import main

directly)

Hey guys, I have this question: The end step of my pipeline is launching a Slackbot. At present I do this on a python script that loads some ML models and gets a connection to the Slack API. As such, it's a long lived process. Given this, would a pipeline node be the right place to launch the script or would you recommend to keep it as a separate step, outside my Kedro pipelines? Thanks

So the answer is I’m not sure! Cool usecase either way. If you are going to do this in kedro an

after_pipeline_run

hook might be best

Gotcha, hadn't considered a hook but it sounds like a great idea. Thanks!

What are “some package” and “kedro_pipeline” in this example?

A python package in your source code or site packages which you can import a pipeline object

So this wouldn’t work where I’ve pip-installed the packaged pipeline, just the original project?

Yeah as far as I know it wouldn’t work. It’s worth trying to be sure.

Okay thanks

I feel like I’m going down the wrong path here. I need to create a “run” button, with a few sliders so a user can change the params and kick off new runs. Is there a smart way to do this?

Or use the config file approach

I’m having no problem running it from the command line, I need to drop it in a plotly dash callback

Hmm it’s a very good point - I’ve wanted to make this easier for a long time

Or something similar, some way to bolt a front end into it