Hello guys! I would like to know if it is possible...
# beginners-need-help
m
Hello guys! I would like to know if it is possible to "extract" the file path of a partitioned dateset catalog used as an input in a node. I tried to look in the internet for alternatives but I couldn't find a solution for it.
d
So this has come up before
Maybe useful?
The other option is to use hooks that have access to the catalog at runtime
m
Hello @User ! First of all, thanks for your suggestion. Unfortunately I didn't manage to get it working. However I came with two possible solutions for my problem:
First, through the catalog's attributes.
Second one, creating a new catalog and overriding it for its path.
d
so I'm not sure either is 'Kedrific'
the best avenue we have for this sort of thing is to define a
before_node_run
hook
you have access to the catalog, inputs and more
m
Just to contextualise: my last node writes in a SQL data set some information calculated in the pipeline and also needs to write the path where is stored a STL (Also generated by the pipeline). The path in my database is important because the mesh is not stored directly in a SQL table . Since this mesh hast to be loaded in the front-end, we have to have its path as a reference. In my point of view, hook is rather a way to observe the pipeline's flow and, in my understanding, in your suggestion the hook is playing the role of a node that transforms data. In that way , wouldn't it be more "kedrific" to use a node instead of a hook? Or did i understand it wrong?
d
We like to ensure that the nodes have no knowledge of IO
This is a good example of a bit of a grey area
In truth we should think of hooks as the main way to extend kedro
And if you ever find yourself creating s context it's a good smell that you've gone too far
m
hmm now i understand why the solutions aren't kedrific. I will give it some thought and find a way to solve my problem "kedrificly".
Thank you very much, @User , for the answers and for being so fast 😄
d
Pleasure