The node is not aware about the type it just treats it as a

Question

The node is not aware about the type it just treats it as a variable You can also pass in named argument You can also use a dictionary of string literal as node input output For example https github com quantumblacklabs kedro starters blob main pandas iris cookiecutter repo name src cookiecutter python package pipelines data engineering pipeline py

Bruno · Answer

I ve tried with a dict ```node func=dataframe melting inputs=dict df= mapped df id vars= altitude var name= disease ```

Bruno · Answer

but it returned the following error TypeError unhashable type list

noklam · Answer

Altitude should be just a string instead of list of string

noklam · Answer

If you want the return variable as a node you need to include this logic inside your function instead of the node

Bruno · Answer

but if id vars be like ``` id vars= altitude x altitude y ```

noklam · Answer

If it is just a single variable does altitude work

Bruno · Answer

no I tried to remove from the list like this ``` node func=dataframe melting inputs=dict df= mapped df id vars= altitude var name= disease ``` and than my pipeline stopped working ``` Failed to find the pipeline named data engieering pipeline It needs to be generated and returned by the register pipelines function ```

noklam · Answer

This seems to be a separate issue Can you do `kedro registry list` and see what s the output It is suggesting you don t have the pipeline defined

Bruno · Answer

``` default data engineering gs sample s3 sample ```

Bruno · Answer

The pipeline runs using id vars= altitude but the output is an error when I remove the it stops finding the pipeline

noklam · Answer

What s your pipeline name Does it shows up here

noklam · Answer

Data engieering pipeline looks like a typo

noklam · Answer

I think it s just your invalid node throw Error earlier than the pipeline

noklam · Answer

After you fixed the node the pipeline not found error shows up instead

Bruno · Answer

I ve fixed that thank you but now the error is that it was not found in the catalog ```f Pipeline input s unsatisfied not found in the DataCatalog ValueError Pipeline input s altitude disease not found in the DataCatalog``` all inputs must be inside the catalog even a list

noklam · Answer

Great Now we are one step closer

noklam · Answer

where is the altitude disease coming from It has to be either defined in the DataCatalog or it is output from other nodes

Bruno · Answer

neither it is just a string that I have hardcoded because I already know that in this df is all about altitude

Bruno · Answer

altitude is the name of a column inside the df

Bruno · Answer

and I already know that the var name or the column name for the variable after the melt must be disease

noklam · Answer

I see so what you want to do is pass in these 2 string as a function parameter

Bruno · Answer

but sometimes in other melting maybe I would give the var name other name so I created a function willing to decide how I want my melty

Bruno · Answer

melt

Bruno · Answer

yes

noklam · Answer

in nodes input output it s all about Datasets and parameters be it an entry in DataCatalog or just a variable we called it MemoryDataSet

noklam · Answer

if you need to pass in function parameters `parameters yml` is what you are looking for

Bruno · Answer

oh I understand I tought it was just for hyper parameters and features in ML

Bruno · Answer

I will read about it and creat a parameters yml

Bruno · Answer

thank you very much nok

noklam · Answer

Kedro pipeline can be any generic python function it s not limited to ML only You may refer to the documentation here too https kedro readthedocs io en stable 06 nodes and pipelines 01 nodes html Usage of parameters yml https kedro readthedocs io en stable 04 kedro project setup 02 configuration html highlight=parameters use parameters

noklam · Answer

You are very welcome