copying questions here

> Data can be loaded via configuration, right? I want to use Kedro during the experiement/R&D phase. So I need to be able to mix and match raw data + feature generator + label generator + parameters. Say I have 10 raw data files. 10 feature generation functions (source files), ...etc. I want to try Raw Data 1 + Featurizer 2 + Labeler 5 + Parameter Set 6 (1/2/5/6). Then 3/4/1/7, ...etc. All while letter Mlflow record the results. > So not only raw data can be configured and loaded, also scripts (functions) used to generate the features and labels.

So this is an interesting topic

as you'll need to set up your pipelines to work in quite a dynamic fashion which we're not necessarily fans as we feel it makes things quite hard to read/maintain/debug

but it is posisble

So I think you're going to get creating with your run args

so you can inject using the technique above

> --config may be used as a starting point, but how to change the function in a pipeline node from it. I have an idea already but wanted first to check if there's a way to do it out of the box.

you'll see

we load a different Sklearn regressor object dynamically based on paramters

and then parametrise the class path

I think the same approach can be applied

You can actually follow along that sample project as it then instantiates the modelling twice

It's an advanced use of Kedro but I think it's similar to what you're trying to achieve

Unless there's something like that ready in Kedro.

I'm not sure - my first reaction is that could get a bit messy and feels like a security risk

we have plans to let the user enable 'unsafe' yaml in the future, but it's not available yet

this would allow you to reference a Python module with the special syntax at the bottom

Itertools in the standard library may be helpful https://docs.python.org/3/library/itertools.html#itertools.combinations

oh not automated or grid-search-like, this is manual experimentation based on the results.

so each pipeline run is - fixed input data - parameterised featurizer - parameterised labeler

Passed as

kedro run --config...

or something.

let me think about this more

but I think there is a chance it may not be the right tool for this approach

I'm protyping somrthing currently

regarding the dynanamic input data - would it be defined in the catalog?

Yes as the standard method. And I'm thinking of using different catalogs or find a way to replace the data with each run.

give me 10 mins to make a dummy project

Thank you.

Actually I've started to feel that Kedro is an overkill for my needs. I have only a fixed 4-step pipeline. The input data, functions, and parameters are what change so if I can put all in a repository and feed them to my pipeline via YAML configs, and use Mlflow for experiement tracking and DVC for versioning the 4 elements with each run, that would be sufficient. I'm torn between using Kedro or writing my small application.

I think I'm done with my sample project- but I'm torn if this makes sense or not

so running

kedro run --config trial_1.yml

will take data from

conf/trail_1/catalog.yml

and the nodes will take their parametrisation from the configured YAML file

But generally, what do you think about my comment above about writing a small app for that. I'm in a proof of concept phase and wrote already part of it. Either that or another project more suited for R&D you know of?

I prefer for the params and features to be Python functions in their own script files instead of conigs in YAML. I'd have more control like that and can integrate them to any other app.

so I think you may be better off with any orchestrator that doesn't take responsibility for IO

so something like Prefect, Airflow or Dagster may be more appropriate

As my functions should be loaded form the same repository where the other data resides.

It just uses importlib

So will read any module visible by the library locally or in site packages