datajoely
10/11/2021, 2:44 PMdatajoely
10/11/2021, 2:58 PMwaylonwalker
10/14/2021, 3:28 PMdatajoely
10/14/2021, 3:34 PMdatajoely
10/14/2021, 3:35 PMwaylonwalker
10/14/2021, 4:20 PMdatajoely
10/14/2021, 4:29 PMdatajoely
10/14/2021, 4:29 PMwaylonwalker
10/14/2021, 6:46 PMdatajoely
10/15/2021, 8:35 AMkedro pipeline create <pipeline_name>
in a new project to see the folder structure in action.
The clever part is articulated in this example:
python
final_pipeline = Pipeline(
[
pipeline(cook_pipeline, outputs={"grilled_meat": "new_name"}),
pipeline(lunch_pipeline, inputs={"food": "new_name"}),
node(...),
...,
]
)
Both cook_pipeline
and lunch_pipeline
are existing modular pipeline, but by using the pipeline
method (not class) you are able to create an instance of them where you can swap catalog inputs/outputs for themuser
10/15/2021, 6:57 PMuser
10/15/2021, 6:59 PMguitar
, piano
, vocal
etc. and connect them all together to make the final master pipeline (the entire song)user
10/15/2021, 7:00 PMuser
10/15/2021, 7:03 PMuser
10/16/2021, 1:56 AMwaylonwalker
10/16/2021, 2:25 PMpython
from .pipelines import lunch_pipeline
from other_project import cook_pipeline # simply just import from another project
final_pipeline = Pipeline(
[
pipeline(cook_pipeline, outputs={"grilled_meat": "new_name"}),
pipeline(lunch_pipeline, inputs={"food": "new_name"}),
node(...),
...,
]
)
waylonwalker
10/16/2021, 2:29 PMpython
final_pipeline = Pipeline(
[
**pipeline(cook_pipeline, outputs={"grilled_meat": "new_name"}).nodes,
**pipeline(lunch_pipeline, inputs={"food": "new_name"}).nodes,
node(...),
...,
]
)
This is closer to how I have been doing it. This achieves a similar affect of reusing pipelines, but looses the history of where nodes came from. And typically I am not passing in the inputs/outputs here.SandyShocksâ˘
10/17/2021, 2:25 PMdatajoely
10/18/2021, 8:30 AMcatalog.save()
do you get the same error? Additionally it looks like your having a DeltaTable specific merge issue to do with types. You can still use Delta via the python API, but our full support for delta is WIP
https://github.com/quantumblacklabs/kedro/pull/964SandyShocksâ˘
10/18/2021, 11:31 PMdatajoely
10/19/2021, 7:31 AMEdmund M
10/21/2021, 8:27 PM%load_ext rpy2.ipython
2021-10-21 14:39:23,435 - rpy2.rinterface_lib.callbacks - WARNING - R[write to console]: Error in .Primitive("as.environment")("package:utils") :
no item called "package:utils" on the search list
Edmund M
10/21/2021, 8:28 PMdatajoely
10/22/2021, 9:02 AMEdmund M
10/22/2021, 2:17 PMdatajoely
10/22/2021, 2:18 PMEdmund M
10/22/2021, 2:20 PMkedro install
is but I'm assuming conda install
is still fair gamedatajoely
10/22/2021, 2:28 PMpip-tools compile
, requirements.in
+ requirements.txt
so kedro install
In the next major version things will be simplified into the following:
1. kedro build-reqs
is still going to prepare src/requirements.txt
so that the dependencies are fully resolved.
2. We then recommend running pip install -r src/requirements.txt
to install your compiled dependencies.
I'm pretty sure conda install
will work here too, but we do know that conda sometimes has issues with the kedro optional dependencies like pip install "kedro[pandas]"
.simon_myway
11/03/2021, 1:13 PMdatajoely
11/03/2021, 1:15 PM