778216384475693066 #beginners-need-help

Hello guys!! Is there a way to change dynamically the parameter.yml file based on some preprocessing/feature engineering nodes? I mean, imagine I want to remove certain columns based on the null values they have but these columns will change over time so I don't want to go to the parameter.yml file and rewrite by hand the name of the columns to delete.

antony.milne

03/28/2022, 8:57 AM

Hi @User, there's not a good way to modify a parameters file directly (it's meant to used strictly for input). The right approach here would be to make a dataset to store those column names in. If you set

versioned: true

then you can keep track of the file over time as well. There are types like

yaml.YAMLDataSet

available for this sort of thing: https://kedro.readthedocs.io/en/stable/kedro.extras.datasets.html

jcasanuevam

03/28/2022, 9:04 AM

Thanks! I'll give it a try

vivecalindahl

03/28/2022, 1:18 PM

Hi! I previously asked about parameterizing the data catalog using an environment variable for the location of a data file: https://discord.com/channels/778216384475693066/846330075535769601/953594739008077825 . I solved that by adding an env var to the globals_dict in hooks.py and then accessing it in data_catalog.yml. Similarly, now I'd now like to access metadata about the file in a node that processes the data. So something like:

Copy code

[...]
node(
name="process",
func="process_fcn",
inputs=dict(
  df="data_at_filepath",
  df_info="${DATA_INFO}"
  ),
[...]
)

where

DATA_INFO

would be an environment variable. However, AFAICT I can't inject an environment variable like this, the globals dict is not available (?). The two solutions I see are 1) just using

os.getenv

inside of the function

process_fcn

or 2) instead make the data info a parameter, refer to it as

param:data_info

and pass it in via

kedro run --params data_info:<something>

. Or is there a better way? This looks pretty similar to what I'm asking about: https://github.com/kedro-org/kedro/issues/1076

Dhaval

03/28/2022, 7:12 PM

How can I load_context of a Kedro Project on kedro 0.17.7? I am planning to use kedro with streamlit

datajoely

03/28/2022, 7:13 PM

You shouldn't need to - the correct way is via hooks

datajoely

03/28/2022, 7:13 PM

What are you trying to achieve?

Dhaval

03/28/2022, 7:13 PM

https://www.youtube.com/watch?v=fYkVtzXUEBE▾

I am trying to create a dashboard by using this tutorial

datajoely

03/28/2022, 7:13 PM

Actually in this case you are correct

datajoely

03/28/2022, 7:14 PM

This video is very old though

Dhaval

03/28/2022, 7:14 PM

I know there's no load_context now, it has been replaced by KedroContext

Dhaval

03/28/2022, 7:15 PM

But I don't know how to set the Kedro context so that I can use it on my streamlit application

Dhaval

03/28/2022, 7:15 PM

I always get this error

Copy code

2022-03-29 00:25:09.405 Traceback (most recent call last):
  File "/home/thakkar/anaconda3/envs/basic_vis/lib/python3.8/site-packages/streamlit/scriptrunner/script_runner.py", line 443, in _run_script
    exec(code, module.__dict__)
  File "/home/thakkar/Work/ramp-zendesk/app.py", line 17, in <module>
    data = context.catalog.list()
  File "/home/thakkar/anaconda3/envs/basic_vis/lib/python3.8/site-packages/kedro/framework/context/context.py", line 320, in catalog
    return self._get_catalog()
  File "/home/thakkar/anaconda3/envs/basic_vis/lib/python3.8/site-packages/kedro/framework/context/context.py", line 356, in _get_catalog
    conf_catalog = self.config_loader.get("catalog*", "catalog*/**", "**/catalog*")
  File "/home/thakkar/anaconda3/envs/basic_vis/lib/python3.8/site-packages/kedro/framework/context/context.py", line 449, in config_loader
    return self._get_config_loader()
  File "/home/thakkar/anaconda3/envs/basic_vis/lib/python3.8/site-packages/kedro/framework/context/context.py", line 432, in _get_config_loader
    raise KedroContextError(
kedro.framework.context.context.KedroContextError: Expected an instance of `ConfigLoader`, got `NoneType` instead.

datajoely

03/28/2022, 7:16 PM

You can steal the way kedro-viz does it: https://github.com/kedro-org/kedro-viz/blob/main/package/kedro_viz/integrations/kedro/data_loader.py

Dhaval

03/28/2022, 7:21 PM

Done! Thanks @User

datajoely

03/28/2022, 7:22 PM

If you wanted to share the modern way of doing this back to the community it would be much appreciated! Perhaps a gist or something?

Dhaval

03/28/2022, 7:43 PM

@User I have just used the code available on Kedro Lifecycle management from the docs It can be found here : https://kedro.readthedocs.io/en/stable/04_kedro_project_setup/03_session.html?highlight=load_context#create-a-session

Copy code

from kedro.framework.session import KedroSession
from kedro.framework.startup import bootstrap_project
from pathlib import Path

metadata = bootstrap_project(Path.cwd())
with KedroSession.create(metadata.package_name) as session:
    context = session.load_context()

Dhaval

03/28/2022, 7:44 PM

Post this I am able to access all elements. So now I can work with the context

Burn1n9m4n

03/28/2022, 10:33 PM

Is there a way to output an excel file from a Kedro pipeline that uses the autofilter function from xlsxwriter? I want to be able to provide a complete data set that comes prefiltered when it is opened by the user

Burn1n9m4n

03/28/2022, 10:33 PM

It needn’t be done entirely within the catalog either.

avan-sh

03/29/2022, 3:38 AM

xlsxwriter-autofilter

Walber Moreira

03/29/2022, 6:13 PM

Ia there an optimal way to visualize namespaced pipelines in kedro-viz? If I have like two pipelines “train” and “predict”, I can’t visualize them together. This also holds for checking the node execution order/input output through pipelines[‘pipename’].describe() on kedro ipython

Bruno

03/29/2022, 7:57 PM

Hello, anyone knows how to pass a list in the nodes input? here's an example:

Copy code

node(
            func=dataframe_melting,
            inputs=["mapped_df", ["altitude"], "disease"],
            outputs="melted_fcl_altitude_df",
            name="fcl_altitude_dataframe_melting_node"
        ),

noklam

03/29/2022, 7:59 PM

What's the function signature?

Bruno

03/29/2022, 7:59 PM

Copy code

def dataframe_melting(df, id_vars, var_name) -> pd.DataFrame:

Bruno

03/29/2022, 8:00 PM

df is a DataFrame, id_vars is a list and var_name a str