Hello I m looking to migrate an existing dask data science p

Question

Hello I m looking to migrate an existing dask data science project into Kedro to help with structuring the code and to help with transparency for non technical folks are there any known best practices for this use case

datajoely · Answer

This is a big topic I will come back to it tomorrow and potentially ask some the team like < antony milne> to weigh in as well

datajoely · Answer

We also have this open github discussion that touches on some relevant points

datajoely · Answer

https github com quantumblacklabs kedro discussions 859

datajoely · Answer

I think my advice is the same as any software engineering migration

datajoely · Answer

Start small prove what works and make sure the team is aligned with the approach rules constraints

datajoely · Answer

Optimise for readability rather than elegance

datajoely · Answer

Write tests Especially to check regressions

brewski · Answer

I guess one of my main questions would be along the lines of if kedro is built in such a way that it would tolerate a session transient object like a dask dataframe in a pipeline or perhaps a sqlalchemy session object both are in use in my project

datajoely · Answer

Ah we don t provide a Session object in the same singleton way In our world a Session is equivalent to a run

brewski · Answer

I m not fluent enough to follow could you clarify

brewski · Answer

the way I understand session is a single python object that references a connection made to some entity on the network

brewski · Answer

e g the engine when you spin up sqlalchemy to run queries against a db

datajoely · Answer

yes so Kedro s session doesn t work really work that way

datajoely · Answer

it s really just an object that contains the runtime reference to the catalog and pipelines in scope

brewski · Answer

I m not saying that kedro s session necessarily has to follow such a model just that other types of sessions can be juggled in one higher level kedro session

datajoely · Answer

Yeah it s not designed to be the same ways as say a Spark Dask sessions are They are used for maintaining a connection to the remote cluster environment our session is more analogous to the lifecycle of single Kedro run