Hello guys I am working on reformatting a ML project to Kedr

Question

Hello guys I am working on reformatting a ML project to Kedro Basically in the project I have three pipeline data engineering data science and prediction Along with having main nodes for these pipeline I also have lot of helper utility functions which need to be reformatted into kedro somewhere I am unsure how I should structure these helper functions Whether I should put them down as sub pipelines or use them as is in the form of helper scripts I would like to know what s the Kedro standard in this use case TIA

datajoely · Answer

So this is more of an art than a science

datajoely · Answer

My view is that you should have very little business logic in your nodes

datajoely · Answer

And simply call other packages within them

datajoely · Answer

Happy to help you think through in more detail

Rjify · Answer

So basically I should keep the helper scripts as is and only put the relevant logic in the nodes Does it makes sense to convert helper functions to nodes as well and have a sort of helper pipeline

datajoely · Answer

So it does depend on the complexity and contents of your helper scripts

datajoely · Answer

If they re pure python functions which don t do any IO then they re ready

datajoely · Answer

Especially if they re already tested

Rjify · Answer

Yeah they are mostly pure python functions

Rjify · Answer

and they are already tested

datajoely · Answer

So then I d focus on readability and maintainability

datajoely · Answer

Kedro nodes should be simple and in general just string together logic defined in other places

datajoely · Answer

So it sounds like you re in a good place

datajoely · Answer

Other bits of advice Feel free to `kedro pipeline create` many single purpose pipelines they can be combined easily and namespaced for both your mental model and visualisation

Rjify · Answer

It makes sense Thanks for your suggestion < datajoely>

datajoely · Answer

Good luck

datajoely · Answer

Shout if you need any sense check

datajoely · Answer

< kedroid 855153796736614411>

Rjify · Answer

Something like this < Mirko>