Hi everyone, I'm looking to run some custom pieces of code as part of my kedro pipeline, which won't run from the same python process. At the same time, I want to avoid race conditions with saving/loading data and similar issues.
I'm thinking of running a piece of C code and some Jupyter notebooks (that need to run using a kernel).
My data is stored locally and will be versioned if I ever run pipelines in parallel.
Any suggestions? No pieces of code available on the web.