Hello guys I would like to know if it is possible to extract

Question

Hello guys I would like to know if it is possible to extract the file path of a partitioned dateset catalog used as an input in a node I tried to look in the internet for alternatives but I couldn t find a solution for it

datajoely · Answer

So this has come up before

datajoely · Answer

This thread has come up before https discord com channels 778216384475693066 846330075535769601 956485672930275388

datajoely · Answer

Maybe useful

datajoely · Answer

The other option is to use hooks that have access to the catalog at runtime

mulajumento · Answer

Hello < datajoely> First of all thanks for your suggestion Unfortunately I didn t manage to get it working However I came with two possible solutions for my problem

mulajumento · Answer

First through the catalog s attributes

mulajumento · Answer

Second one creating a new catalog and overriding it for its path

datajoely · Answer

so I m not sure either is Kedrific

datajoely · Answer

the best avenue we have for this sort of thing is to define a `before node run` hook

datajoely · Answer

https kedro readthedocs io en latest kedro framework hooks specs NodeSpecs html kedro framework hooks specs NodeSpecs before node run

datajoely · Answer

you have access to the catalog inputs and more

mulajumento · Answer

Just to contextualise my last node writes in a SQL data set some information calculated in the pipeline and also needs to write the path where is stored a STL Also generated by the pipeline The path in my database is important because the mesh is not stored directly in a SQL table Since this mesh hast to be loaded in the front end we have to have its path as a reference In my point of view hook is rather a way to observe the pipeline s flow and in my understanding in your suggestion the hook is playing the role of a node that transforms data In that way wouldn t it be more kedrific to use a node instead of a hook Or did i understand it wrong

datajoely · Answer

We like to ensure that the nodes have no knowledge of IO

datajoely · Answer

This is a good example of a bit of a grey area

datajoely · Answer

In truth we should think of hooks as the main way to extend kedro

datajoely · Answer

And if you ever find yourself creating s context it s a good smell that you ve gone too far

mulajumento · Answer

hmm now i understand why the solutions aren t kedrific I will give it some thought and find a way to solve my problem kedrificly

mulajumento · Answer

Thank you very much < datajoely> for the answers and for being so fast

datajoely · Answer

Pleasure