beginners-need-help
- d
datajoely
03/28/2022, 7:22 PMIf you wanted to share the modern way of doing this back to the community it would be much appreciated! Perhaps a gist or something? - d
Dhaval
03/28/2022, 7:43 PM@User I have just used the code available on Kedro Lifecycle management from the docs It can be found here : https://kedro.readthedocs.io/en/stable/04_kedro_project_setup/03_session.html?highlight=load_context#create-a-sessionfrom kedro.framework.session import KedroSession from kedro.framework.startup import bootstrap_project from pathlib import Path metadata = bootstrap_project(Path.cwd()) with KedroSession.create(metadata.package_name) as session: context = session.load_context() - d
Dhaval
03/28/2022, 7:44 PMPost this I am able to access all elements. So now I can work with the context - b
Burn1n9m4n
03/28/2022, 10:33 PMIs there a way to output an excel file from a Kedro pipeline that uses the autofilter function from xlsxwriter? I want to be able to provide a complete data set that comes prefiltered when it is opened by the userad- 3
- 2
- b
Burn1n9m4n
03/28/2022, 10:33 PMIt needn’t be done entirely within the catalog either. - a
avan-sh
03/29/2022, 3:38 AMxlsxwriter-autofilter - w
Walber Moreira
03/29/2022, 6:13 PMIa there an optimal way to visualize namespaced pipelines in kedro-viz? If I have like two pipelines “train” and “predict”, I can’t visualize them together. This also holds for checking the node execution order/input output through pipelines[‘pipename’].describe() on kedro ipythond- 2
- 1
- b
Bruno
03/29/2022, 7:57 PMHello, anyone knows how to pass a list in the nodes input? here's an example:node( func=dataframe_melting, inputs=["mapped_df", ["altitude"], "disease"], outputs="melted_fcl_altitude_df", name="fcl_altitude_dataframe_melting_node" ), - n
noklam
03/29/2022, 7:59 PMWhat's the function signature? - b
Bruno
03/29/2022, 7:59 PMdef dataframe_melting(df, id_vars, var_name) -> pd.DataFrame: - b
Bruno
03/29/2022, 8:00 PMdf is a DataFrame, id_vars is a list and var_name a str - n
noklam
03/29/2022, 8:04 PMThen you can use it like a normal variable, the string literal is just alias for the variable. ["mapped_df", "altitude", "disease"] - b
Bruno
03/29/2022, 8:04 PMand if id_vars be a dictionary? - n
noklam
03/29/2022, 8:15 PMThe node is not aware about the type, it just treats it as a variable. You can also pass in named argument. You can also use a dictionary of string literal as node input/output For example https://github.com/quantumblacklabs/kedro-starters/blob/main/pandas-iris/{{ cookiecutter.repo_name }}/src/{{ cookiecutter.python_package }}/pipelines/data_engineering/pipeline.pyb- 2
- 31
- d
datajoely
03/29/2022, 9:01 PMvisualising namespaces - p
pypeaday
03/30/2022, 2:09 PM@User is there a way to set any kind of lifecycle on versioned datasets? I'm not seeing anything in docs about that... Or is Kedro's position that we should use an underlying filesystem's capabilities here (like ZFS snapshots or S3 lifecycle policies)? @User fyi - d
datajoely
03/30/2022, 2:30 PMwhat do you mean lifecycle? Some sort of expiry? - p
pypeaday
03/30/2022, 3:58 PMya ya exactly - like if I want to keep the last 5 versions, and then versions at the beginning of the month for the past 12 months, and then annual versions for the last X years or whatever...da- 3
- 32
- m
Matheus Serpa
03/30/2022, 5:19 PMHello there! Is there a way to get the input name inside a node function? For example,node(func=melt_data, inputs="fcl_elevation") ... def melt_data(df): # how to get "fcl_elevation" inside func?da- 3
- 7
- w
WolVez
04/01/2022, 9:16 PMis there a way to flag a dataset to not run asynchronously and to wait until other nodes are complete, if asnyc is enabled? - d
datajoely
04/01/2022, 9:24 PMThe best way to do that is to break the pipeline into pieces and execute from the CLI likekedro run --pipeline a && kedro run --pipeline b --async && kedro run --pipeline c - d
datajoely
04/01/2022, 9:25 PM&& wil wait for the previous statement to complete, single & will do both simultaneously - m
mulajumento
04/05/2022, 1:15 AMHello guys! I would like to know if it is possible to "extract" the file path of a partitioned dateset catalog used as an input in a node. I tried to look in the internet for alternatives but I couldn't find a solution for it.d- 2
- 19
- m
munchmuch
04/05/2022, 4:12 AMHi all I'm trying to get the TemplatedConfigLoader to work. I'm getting this error
. It appears the TemplatedConfigLoader is inheriting from AbstractConfigLoader in 0.18.0 any idea how to fix this? Tried to change to inheriting from ConfigLoader itself, passes the assert but doesn't use my globals.yml. Thank youIt must be a subclass of kedro.config.config.ConfigLoaderd- 2
- 2
- d
datajoely
04/05/2022, 4:32 AMusing individual partitions - d
datajoely
04/05/2022, 4:36 AMConfigLoader issue with 0.18.x - g
gui42
04/05/2022, 6:31 PMHey guys! Quick question. How could I handle in the catalog a directory that can have an unknown number of files, but conveniently named? - d
datajoely
04/05/2022, 6:31 PMPartitionedDataset! - g
gui42
04/05/2022, 6:32 PMThis seems nice. My use case is more ML driven. Think of it as train test sets but generated by another team/application. - d
datajoely
04/05/2022, 6:33 PMSo I think it should work - but we do have an assumption that things are reproducible so be careful!
Title
d
datajoely
04/05/2022, 6:33 PMSo I think it should work - but we do have an assumption that things are reproducible so be careful!
View count: 1