user
04/13/2022, 11:57 AMdatajoely
04/13/2022, 12:06 PMgui42
04/14/2022, 1:20 AMdatajoely
04/14/2022, 4:16 AMpraveenbisht
04/14/2022, 2:38 PMnoklam
04/14/2022, 3:03 PMpraveenbisht
04/14/2022, 3:04 PMnoklam
04/14/2022, 3:08 PMnoklam
04/14/2022, 3:09 PMsession.run()
, you don't need to access context to execute a pipeline. In fact context.run
has been removed.praveenbisht
04/14/2022, 4:36 PMuser
04/18/2022, 6:33 AMdeepyaman
04/19/2022, 10:52 AMfaust
as a backend for this (and being more Python-native), but faust
isn't really maintained anymore (and was lacking some other necessary functionality, like joining streams).
There's an overall question of how to better support streaming workflows with Kedro, as Kedro is notoriously batch-oriented.user
04/20/2022, 5:32 PMRjify
04/20/2022, 7:01 PMOnéira
04/21/2022, 7:29 AMuser
04/21/2022, 8:50 AMdatajoely
04/21/2022, 9:11 AMRafał
04/21/2022, 10:48 AMpipe
which takes the input param params:alpha
and gives the output output
I would like to run this pipeline with different values of parameter alpha
I know I can pass extra params using CLI kedro run --pipeline pipe --params "alpha:5"
,
My problem is that I would like to create the following pipeline:
* run pipe
with alpha=x
for each x
in params:bunch_of_alphas
and name the output as output_x
* collect all the outputs from previous step and run the node which takes that collection and creates the single report
output.
I am afraid I do not know how to do that in kedro.
I am using kedro 0.18
I have already idea to create the pipeline with takes the params:bunch_of_alphas
and calls the function which does for
loop and creates fake params:alpha
in current kedro session..... and then stops since it is too complicated.
Is there any clean solution?datajoely
04/21/2022, 11:10 AMbefore_pipeline_run
hook hereRafał
04/21/2022, 11:30 AMparams:bunch_of_alphas
?
The only thing I have in my mind is to create fake kedro's catalog item with new "temporary_alpha_param"
Actually I have no how to override the pipeline params in before_pipeline_run
hook. Could you point some example code?Rafał
04/21/2022, 11:39 AMbefore_pipeline_run
hook, right ?datajoely
04/21/2022, 11:47 AMdatajoely
04/21/2022, 11:49 AMdatajoely
04/21/2022, 11:50 AMavan-sh
04/21/2022, 2:47 PMdatajoely
04/21/2022, 2:50 PMnoklam
04/21/2022, 3:14 PMnoklam
04/21/2022, 3:15 PMavan-sh
04/21/2022, 3:18 PMdatajoely
04/21/2022, 3:42 PMafter_command_run
merged on the CLI side