datajoely
03/24/2022, 1:55 PMdatajoely
03/24/2022, 1:56 PMnoestl
03/24/2022, 2:53 PMdatajoely
03/24/2022, 3:02 PMjcasanuevam
03/28/2022, 8:28 AMantony.milne
03/28/2022, 8:57 AMversioned: true
then you can keep track of the file over time as well. There are types like yaml.YAMLDataSet
available for this sort of thing: https://kedro.readthedocs.io/en/stable/kedro.extras.datasets.htmljcasanuevam
03/28/2022, 9:04 AMvivecalindahl
03/28/2022, 1:18 PM[...]
node(
name="process",
func="process_fcn",
inputs=dict(
df="data_at_filepath",
df_info="${DATA_INFO}"
),
[...]
)
where DATA_INFO
would be an environment variable. However, AFAICT I can't inject an environment variable like this, the globals dict is not available (?). The two solutions I see are
1) just using os.getenv
inside of the function process_fcn
or
2) instead make the data info a parameter, refer to it as param:data_info
and pass it in via kedro run --params data_info:<something>
.
Or is there a better way?
This looks pretty similar to what I'm asking about: https://github.com/kedro-org/kedro/issues/1076Dhaval
03/28/2022, 7:12 PMdatajoely
03/28/2022, 7:13 PMdatajoely
03/28/2022, 7:13 PMdatajoely
03/28/2022, 7:13 PMdatajoely
03/28/2022, 7:14 PMDhaval
03/28/2022, 7:14 PMDhaval
03/28/2022, 7:15 PMDhaval
03/28/2022, 7:15 PM2022-03-29 00:25:09.405 Traceback (most recent call last):
File "/home/thakkar/anaconda3/envs/basic_vis/lib/python3.8/site-packages/streamlit/scriptrunner/script_runner.py", line 443, in _run_script
exec(code, module.__dict__)
File "/home/thakkar/Work/ramp-zendesk/app.py", line 17, in <module>
data = context.catalog.list()
File "/home/thakkar/anaconda3/envs/basic_vis/lib/python3.8/site-packages/kedro/framework/context/context.py", line 320, in catalog
return self._get_catalog()
File "/home/thakkar/anaconda3/envs/basic_vis/lib/python3.8/site-packages/kedro/framework/context/context.py", line 356, in _get_catalog
conf_catalog = self.config_loader.get("catalog*", "catalog*/**", "**/catalog*")
File "/home/thakkar/anaconda3/envs/basic_vis/lib/python3.8/site-packages/kedro/framework/context/context.py", line 449, in config_loader
return self._get_config_loader()
File "/home/thakkar/anaconda3/envs/basic_vis/lib/python3.8/site-packages/kedro/framework/context/context.py", line 432, in _get_config_loader
raise KedroContextError(
kedro.framework.context.context.KedroContextError: Expected an instance of `ConfigLoader`, got `NoneType` instead.
datajoely
03/28/2022, 7:16 PMDhaval
03/28/2022, 7:21 PMdatajoely
03/28/2022, 7:22 PMDhaval
03/28/2022, 7:43 PMfrom kedro.framework.session import KedroSession
from kedro.framework.startup import bootstrap_project
from pathlib import Path
metadata = bootstrap_project(Path.cwd())
with KedroSession.create(metadata.package_name) as session:
context = session.load_context()
Dhaval
03/28/2022, 7:44 PMBurn1n9m4n
03/28/2022, 10:33 PMBurn1n9m4n
03/28/2022, 10:33 PMavan-sh
03/29/2022, 3:38 AMWalber Moreira
03/29/2022, 6:13 PMBruno
03/29/2022, 7:57 PMnode(
func=dataframe_melting,
inputs=["mapped_df", ["altitude"], "disease"],
outputs="melted_fcl_altitude_df",
name="fcl_altitude_dataframe_melting_node"
),
noklam
03/29/2022, 7:59 PMBruno
03/29/2022, 7:59 PMdef dataframe_melting(df, id_vars, var_name) -> pd.DataFrame:
Bruno
03/29/2022, 8:00 PM