778216384475693066 #beginners-need-help

Hi team, wondering if anyone has run into the following error before - or what it might mean. I'm running a node that tries to save a csv dataset of two columns: slug, and domain as a YML dataset. I'm getting an error:

j c h a r l e s

01/03/2022, 12:06 AM

kedro.io.core.DataSetError: Failed while saving data to data set YAMLDataSet(filepath=~/data-flow/data/02_intermediate/meta.yml, protocol=file, save_args={'default_flow_style': False}).
'functools.partial' object has no attribute '__name__'

j c h a r l e s

01/03/2022, 12:06 AM

Somehow if I save the same dataset as

pandas.CSVDataSet

there is no error

j c h a r l e s

01/03/2022, 12:09 AM

Full traceback is here

j c h a r l e s

01/03/2022, 12:19 AM

Also, if I run the following:

import yaml
yaml.dump(node_function(INPUT).to_dict(), open("charles.yml", "w"))

No error happens

j c h a r l e s

01/03/2022, 12:55 AM

Feature request idea: let people use google sheets as a Dataset. I just added this for our pipeline and I'm pretty sure it will be very helpful for us. This would be generally useful for any people that have pipelines with data that interfaces with cross-functional partners at startups (for example the performance marketing team, or teams that work primarily through google sheets - finance)

brewski

01/03/2022, 10:06 AM

Hello again- looking to use ssl credentials to connect to a sql db. I reason that ssl credentials fall under 'config' but are files themselves- is there a way I could have kedro remember the path at which they are stored as a config variable that wouldn't be host-computer or referring-python-file dependent?

brewski

01/03/2022, 10:45 AM

Also, in the context of reading from sql to populate a dataset using [[https://kedro.readthedocs.io/en/latest/kedro.extras.datasets.pandas.SQLQueryDataSet.html#kedro.extras.datasets.pandas.SQLQueryDataSet]] is there a correct way of specifying (hopefully relative) paths to the config?

Ozol

01/03/2022, 12:32 PM

If I wish to have global constants that can be used in various nodes across multiple pipelines, where would be the most "Kedro" place to put them? I know of globals.yml and $constant_name but that seems reserved for other .yml files. Should I just create a constants.py file in the src folder of my project?

datajoely

01/03/2022, 4:10 PM

Hi @User you need to look for

TemplatedConfigLoader

https://kedro.readthedocs.io/en/stable/kedro.config.TemplatedConfigLoader.html

datajoely

01/03/2022, 4:11 PM

SSL via SQL DataSet

datajoely

01/03/2022, 4:14 PM

Partial application

datajoely

01/03/2022, 4:16 PM

YAML dump

datajoely

01/03/2022, 4:17 PM

GoogleSheets dataset

ÂNCD〆Kirito

01/04/2022, 3:44 PM

Hi, does anyone have experience in integrating Kedro pipelines through Flask RestAPI framework? I have a use case where I need to take input from user in UI and pass them to Kedro pipeline and provide the model output to UI through Flask API. TIA.

waylonwalker

01/04/2022, 3:54 PM

Quick question. I am getting a bunch of warnings on some of my projects that getting pipelines from context will not be supported in 0.18.0. Is there a way to get pipelines from a kedro session that is compatable with both 0.17 and 0.18

datajoely

01/04/2022, 4:02 PM

can you try this?

datajoely

01/04/2022, 4:03 PM

So we don't support this out of the box @User recently published this plugin https://github.com/Galileo-Galilei/kedro-serving which may do what you need

datajoely

01/04/2022, 4:03 PM

You can also look into how

kedro-viz

works as it does exactly this

waylonwalker

01/04/2022, 5:24 PM

importing pipelines works, I also dug in deeper and figured out why this did not work previously (version mismatch kedro == 0.17.6 in one and kedro == 0.17.2 in the other)

waylonwalker

01/04/2022, 5:24 PM

As always, thanks so much for the quick response @User

metalmind

01/05/2022, 4:44 PM

Hello everyone.

metalmind

01/05/2022, 4:48 PM

What is the best way to make feature and label generation configurable as one of the pipeline/node inputs same as raw data and make the associated scripts not part of the project source so they are stored in the same level as the data?

datajoely

01/05/2022, 5:17 PM

Hi @User this is an interesting topic - we have had some early conversations with these folks re what an integration could look like https://labelstud.io/

datajoely

01/05/2022, 5:18 PM

I'm not sure what you mean by

pipeline/node inputs same as raw data and make the associated scripts not part of the project source so they are stored in the same level as the data?

datajoely

01/05/2022, 5:19 PM

by I think the

--params

syntax or

--config {some.yml}

command may be your friend here https://kedro.readthedocs.io/en/latest/04_kedro_project_setup/02_configuration.html#configure-kedro-run-arguments

hoodie

01/05/2022, 5:21 PM

Hello! I have multiple time series with exogenous params. I have one dataset for each time serie, and would like to create one model for each dataset, following the same pipeline. Is it possible to easily do it with kedro? 🤔 Thanks for your response :)

Title

hoodie

01/05/2022, 5:21 PM

datajoely

01/05/2022, 5:22 PM

Yes! we are currently updating our docs to better explain this feature

but you can follow the docs here https://kedro.readthedocs.io/en/stable/06_nodes_and_pipelines/03_modular_pipelines.html

and follow my example project here https://github.com/datajoely/modular-spaceflights

You can reuse the same pipeline multiple times with different inputs

https://github.com/datajoely/modular-spaceflights/blob/main/src/modular_spaceflights/pipelines/modelling/pipeline.py

here is an example how we use the same pipeline twice, using diff modelling techniques

Wit

04/24/2022, 7:46 PM

Can modular pipelines be data driven? Taking the time series example - I want to build models for each time series but time series ids are in db and change in time. For example in a retail chain some products are removed or added.

datajoely

04/25/2022, 9:17 AM

so yes, you're able to do so - but it does harm reproducibility

As a team we don't mind dynamic pipelines, but believe in order to make them maintainable the inputs can be dynamic - but the structure should be static

The other way you can do this is loading/saving some checkpoints to another dataset to retain some idea of 'state'

happy to help you think through the problem

Wit

04/28/2022, 7:47 AM

Thanks for reply. I know it harms reproducibility. I personally prefer have one pipeline and run it many times with different inputs.

View count: 2