Do you have permissions to click set DCO to pass?

I set up git hooks for the repo last I needed it. This doc has a ui based option https://github.com/src-d/guide/blob/master/developer-community/fix-DCO.md

@User thanks for pointing that out - I hadn't seen the Chrome extension, and I thought I had looked into every avenue for making DCO sign-off as painless as possible 😅 @User if this becomes very annoying we might be able to make the "Set DCO to pass" button that @User mentioned available to members outside kedro-team (you won't be able to see it at the moment). Or, given that GitHub UI commits are GPG Verified anyway, we *might *be able to extend the configuration that makes non-explicitly signed-off commits still pass DCO (as they do already for kedro-team commits). Right now if you don't want to go through the rebase hassle just ask us to press the "DCO pass" button for you and it won't matter that you forgot to sign off previous commits though

@User in addition to what Joel said, if you do

kedro package

on your project (let's say its Python package name is

xyz

) then you can actually do this:

from xyz import __main__
__main__.main()

which will be equivalent to

python -m xyz

. This does seem a little ugly though. IIRC @User wrote the stuff on

__main__

not so long ago so might know a better way of doing this (or if there isn't one then maybe we should expose the

main

function so I can do

from xyz import main

directly)

Hey guys, I have this question: The end step of my pipeline is launching a Slackbot. At present I do this on a python script that loads some ML models and gets a connection to the Slack API. As such, it's a long lived process. Given this, would a pipeline node be the right place to launch the script or would you recommend to keep it as a separate step, outside my Kedro pipelines? Thanks

So the answer is I’m not sure! Cool usecase either way. If you are going to do this in kedro an

after_pipeline_run

hook might be best

Gotcha, hadn't considered a hook but it sounds like a great idea. Thanks!

What are “some package” and “kedro_pipeline” in this example?

A python package in your source code or site packages which you can import a pipeline object

So this wouldn’t work where I’ve pip-installed the packaged pipeline, just the original project?

Yeah as far as I know it wouldn’t work. It’s worth trying to be sure.

Okay thanks

I feel like I’m going down the wrong path here. I need to create a “run” button, with a few sliders so a user can change the params and kick off new runs. Is there a smart way to do this?

Or use the config file approach

I’m having no problem running it from the command line, I need to drop it in a plotly dash callback

Hmm it’s a very good point - I’ve wanted to make this easier for a long time

Or something similar, some way to bolt a front end into it

I might have to get back to you in the morning it’s end of day here in London

Okay, I’ll keep playing around, was just wondering if there was a good way of doing it. I’m currently inlining all of my yamls as gigantic dicts and it doesn’t quite feel right. Thanks for your help 🙂

I agree we need to find a better way!

Can anyone link me to a project that uses partitioned datasets? Looking for some general examples, would be much appreciated.

Following the contents of this video: I see this gentleman creates a "singular" and a "plural" node so that he essentially has two separate pipelines: one for processing a specified file and one for processing the entire partitioned dataset. Is this the recommended way to design all projects that involve many files of the same type and structure?

Hi @User It’s one pattern that users have been successful with

Maybe it would best for you to tell us a little about what you’re trying to build and we can think it through together?

@User Thanks for responding. The project is nothing serious, just trying to re-work a past project using Kedro for learning purposes. In this case it's ~20 files of timeseries data of cepheid variable star brightness, where the objective is to calculate some best fit parameters such as the period of pulsation, etc. No ML, maybe 10 functions in total. In my previous implementation of the project, I just had a

main()

function that looped over all the files in the directory. Many of my Astro projects are of a similar format to this, so been wondering what the best design choices are for a Kedro implementation

I’m with you I think the DE1 pattern above makes a lot of sense. The only think that has changed since then is the introduction of modular pipelines that facilitate more powerful reuse of logic: https://kedro.readthedocs.io/en/0.17.7/06_nodes_and_pipelines/03_modular_pipelines.html

Thanks for the advice! @User PS. Is DE1 still around? Looks like he dropped off the face of the earth sometime last year

Unfortunately not he’s a bit too successful to keep doing this in his spare time! We’re actually about to start hiring a full time DevRel which should help here