Hi @Bastian @wulfcrona this is a really interesting topic of discussion and in truth, something where best practice is still being defined. I would say that this is somewhere that feels similar to API versioning where a level of governances is needed, breaking changes need to be communicated and transparent etc.
I have some initial thoughts:
1. The smaller the modular pipeline the easier it is to manage changes. Several of our internal use-cases have very simple pipelines e.g. Splitting for cross validation is reused across several pipelines and projects.
2. This also brings in questions related to Governance, mono-repo vs multi-repo where things are centralised versus decentralised.
3. We currently allow users to pull as source code, rather than importable and installable libraries as we want be people to be able to remix and modify pipelines to fit their needs.
Ultimately, we're looking for you, the community, to help us understand how we can best suit your needs. I'm delighted you're using some of the newest functionality we've brought to Kedro and I'm super keen to set you up for success.
What I would love is if you could raise a GitHub discussion so we can allow our community to comment on (1) What is working for them today? (2) What isn't (3) What they would ❤️ to be added in the future?