Quick Q, we can create our custom starters, which ...
# advanced-need-help
b
Quick Q, we can create our custom starters, which is very useful - we use it to set company-wide standards. Is there a way to create custom pipeline layouts? So when we do
kedro pipeline create
it will use a custom cookiecutter layout created by us? Technically it seems possible as I see in the
kedro
repo there is a dedicated cookiecutter for pipelines
n
Very interesting question!
AFAIK you can have custom template for starter, but not for pipeline. The pipeline structure are for modular pipeline and they have specific structure.
How would you want to customize your current pipeline?
b
I'd say the concept is very similar to the sub-folders of the
data
directory. Very much the same idea of semantically ordering data, we also have a semantic ordering of nodes, so I want to reflect that in our organization's starters
n
So you want kedro pipeline create not just populate an empty pipeline file, but with pipeline and nodes - then the user should update and fill in the definition of these nodes?
b
Specifically instead of having all nodes in
nodes.py
I'd have them split, either
nodes/xxx.py
or just having multiple
xxx.py
files
n
I will check later - I don't remember if this is possible or not.
d
You can use cookiecutte directly for this if you want to get going quickly
That's all we're doing behind the scenes
b
1. How would I integrate the cookiecutter with the
kedro new pipeline
command? 2. Will the pipeline discovery and other features related to pipelines will just work?
d
1. It wouldn't be via kedro, you'd use it natively
2. I'll let Nok describe what's needed there
n
2 depends on the pipeline.py if i remember - the docs have more information about what structure is expected. It check the existence of pipeline.py and the create_xxxxxxxxx method.
2 Views