778216384475693066 #beginners-need-help

Channels

advanced-need-help

job-posting

welcome

j c h a r l e s

01/03/2022, 12:06 AM

Copy code

kedro.io.core.DataSetError: Failed while saving data to data set YAMLDataSet(filepath=~/data-flow/data/02_intermediate/meta.yml, protocol=file, save_args={'default_flow_style': False}).
'functools.partial' object has no attribute '__name__'

j c h a r l e s

01/03/2022, 12:06 AM

Somehow if I save the same dataset as

pandas.CSVDataSet

there is no error

j c h a r l e s

01/03/2022, 12:09 AM

Full traceback is here

j c h a r l e s

01/03/2022, 12:19 AM

Also, if I run the following:

Copy code

import yaml
yaml.dump(node_function(INPUT).to_dict(), open("charles.yml", "w"))

No error happens

j c h a r l e s

01/03/2022, 12:55 AM

Feature request idea: let people use google sheets as a Dataset. I just added this for our pipeline and I'm pretty sure it will be very helpful for us. This would be generally useful for any people that have pipelines with data that interfaces with cross-functional partners at startups (for example the performance marketing team, or teams that work primarily through google sheets - finance)

brewski

01/03/2022, 10:06 AM

Hello again- looking to use ssl credentials to connect to a sql db. I reason that ssl credentials fall under 'config' but are files themselves- is there a way I could have kedro remember the path at which they are stored as a config variable that wouldn't be host-computer or referring-python-file dependent?

brewski

01/03/2022, 10:45 AM

Also, in the context of reading from sql to populate a dataset using [[https://kedro.readthedocs.io/en/latest/kedro.extras.datasets.pandas.SQLQueryDataSet.html#kedro.extras.datasets.pandas.SQLQueryDataSet]] is there a correct way of specifying (hopefully relative) paths to the config?

Ozol

01/03/2022, 12:32 PM

If I wish to have global constants that can be used in various nodes across multiple pipelines, where would be the most "Kedro" place to put them? I know of globals.yml and $constant_name but that seems reserved for other .yml files. Should I just create a constants.py file in the src folder of my project?

datajoely

01/03/2022, 4:10 PM

Hi @User you need to look for

TemplatedConfigLoader

https://kedro.readthedocs.io/en/stable/kedro.config.TemplatedConfigLoader.html

datajoely

01/03/2022, 4:11 PM

SSL via SQL DataSet

datajoely

01/03/2022, 4:14 PM

Partial application

datajoely

01/03/2022, 4:16 PM

YAML dump

datajoely

01/03/2022, 4:17 PM

GoogleSheets dataset

ÂNCD〆Kirito

01/04/2022, 3:44 PM

Hi, does anyone have experience in integrating Kedro pipelines through Flask RestAPI framework? I have a use case where I need to take input from user in UI and pass them to Kedro pipeline and provide the model output to UI through Flask API. TIA.

waylonwalker

01/04/2022, 3:54 PM

Quick question. I am getting a bunch of warnings on some of my projects that getting pipelines from context will not be supported in 0.18.0. Is there a way to get pipelines from a kedro session that is compatable with both 0.17 and 0.18

datajoely

01/04/2022, 4:02 PM

can you try this?

datajoely

01/04/2022, 4:03 PM

So we don't support this out of the box @User recently published this plugin https://github.com/Galileo-Galilei/kedro-serving which may do what you need

datajoely

01/04/2022, 4:03 PM

You can also look into how

kedro-viz

works as it does exactly this

waylonwalker

01/04/2022, 5:24 PM

importing pipelines works, I also dug in deeper and figured out why this did not work previously (version mismatch kedro == 0.17.6 in one and kedro == 0.17.2 in the other)

waylonwalker

01/04/2022, 5:24 PM

As always, thanks so much for the quick response @User

metalmind

01/05/2022, 4:44 PM

Hello everyone.

metalmind

01/05/2022, 4:48 PM

What is the best way to make feature and label generation configurable as one of the pipeline/node inputs same as raw data and make the associated scripts not part of the project source so they are stored in the same level as the data?

datajoely

01/05/2022, 5:17 PM

Hi @User this is an interesting topic - we have had some early conversations with these folks re what an integration could look like https://labelstud.io/

datajoely

01/05/2022, 5:18 PM

I'm not sure what you mean by

pipeline/node inputs same as raw data and make the associated scripts not part of the project source so they are stored in the same level as the data?

datajoely

01/05/2022, 5:19 PM

by I think the

--params

syntax or

--config {some.yml}

command may be your friend here https://kedro.readthedocs.io/en/latest/04_kedro_project_setup/02_configuration.html#configure-kedro-run-arguments

hoodie

01/05/2022, 5:21 PM

Hello! I have multiple time series with exogenous params. I have one dataset for each time serie, and would like to create one model for each dataset, following the same pipeline. Is it possible to easily do it with kedro? 🤔 Thanks for your response :)

metalmind

01/05/2022, 5:21 PM

As far as I know, this is a tool for manual labeling. What I'm looking for is a script with a function used labeling and another for feature generation.

metalmind

01/05/2022, 5:25 PM

Data can be loaded via configuration, right? I want to use Kedro during the experiement/R&D phase. So I need to be able to mix and match raw data + feature generator + label generator + parameters. Say I have 10 raw data files. 10 feature generation functions (source files), ...etc. I want to try Raw Data 1 + Featurizer 2 + Labeler 5 + Parameter Set 6 (1/2/5/6). Then 3/4/1/7, ...etc. All while letter Mlflow record the results.

metalmind

01/05/2022, 5:25 PM

So not only raw data can be configured and loaded, also scripts (functions) used to generate the features and labels.

metalmind

01/05/2022, 5:27 PM

--config may be used as a starting point, but how to change the function in a pipeline node from it. I have an idea already but wanted first to check if there's a way to do it out of the box.