https://kedro.org/ logo
Join the conversationJoin Discord
Channels
advanced-need-help
announcements
beginners-need-help
introductions
job-posting
plugins-integrations
random
resources
welcome
Powered by Linen
beginners-need-help
  • d

    datajoely

    12/31/2021, 1:27 PM
    @User you should use #908346260224872480 for discussions re plugins not maintained by the Kedro team. I will also point you to the new experiment tracking features built into Kedro https://kedro.readthedocs.io/en/stable/08_logging/02_experiment_tracking.html https://github.com/quantumblacklabs/kedro-viz#experiment-tracking-usage
  • g

    Galileo-Galilei

    12/31/2021, 1:36 PM
    @datajoely is it possible to move the message to the right channel or can I answer here? What do you prefer?
  • d

    datajoely

    12/31/2021, 4:54 PM
    I don’t think I can move it - but I’d maybe reply there?
  • j

    j c h a r l e s

    01/03/2022, 12:05 AM
    Hi team, wondering if anyone has run into the following error before - or what it might mean. I'm running a node that tries to save a csv dataset of two columns: slug, and domain as a YML dataset. I'm getting an error:
    d
    • 2
    • 7
  • j

    j c h a r l e s

    01/03/2022, 12:06 AM
    kedro.io.core.DataSetError: Failed while saving data to data set YAMLDataSet(filepath=~/data-flow/data/02_intermediate/meta.yml, protocol=file, save_args={'default_flow_style': False}).
    'functools.partial' object has no attribute '__name__'
  • j

    j c h a r l e s

    01/03/2022, 12:06 AM
    Somehow if I save the same dataset as
    pandas.CSVDataSet
    there is no error
  • j

    j c h a r l e s

    01/03/2022, 12:09 AM
    Full traceback is here
  • j

    j c h a r l e s

    01/03/2022, 12:19 AM
    Also, if I run the following:
    import yaml
    yaml.dump(node_function(INPUT).to_dict(), open("charles.yml", "w"))
    No error happens
    d
    • 2
    • 12
  • j

    j c h a r l e s

    01/03/2022, 12:55 AM
    Feature request idea: let people use google sheets as a Dataset. I just added this for our pipeline and I'm pretty sure it will be very helpful for us. This would be generally useful for any people that have pipelines with data that interfaces with cross-functional partners at startups (for example the performance marketing team, or teams that work primarily through google sheets - finance)
    d
    • 2
    • 5
  • b

    brewski

    01/03/2022, 10:06 AM
    Hello again- looking to use ssl credentials to connect to a sql db. I reason that ssl credentials fall under 'config' but are files themselves- is there a way I could have kedro remember the path at which they are stored as a config variable that wouldn't be host-computer or referring-python-file dependent?
  • b

    brewski

    01/03/2022, 10:45 AM
    Also, in the context of reading from sql to populate a dataset using [[https://kedro.readthedocs.io/en/latest/kedro.extras.datasets.pandas.SQLQueryDataSet.html#kedro.extras.datasets.pandas.SQLQueryDataSet]] is there a correct way of specifying (hopefully relative) paths to the config?
    d
    a
    • 3
    • 76
  • o

    Ozol

    01/03/2022, 12:32 PM
    If I wish to have global constants that can be used in various nodes across multiple pipelines, where would be the most "Kedro" place to put them? I know of globals.yml and $constant_name but that seems reserved for other .yml files. Should I just create a constants.py file in the src folder of my project?
  • d

    datajoely

    01/03/2022, 4:10 PM
    Hi @User you need to look for
    TemplatedConfigLoader
    https://kedro.readthedocs.io/en/stable/kedro.config.TemplatedConfigLoader.html
  • d

    datajoely

    01/03/2022, 4:11 PM
    SSL via SQL DataSet
  • d

    datajoely

    01/03/2022, 4:14 PM
    Partial application
  • d

    datajoely

    01/03/2022, 4:16 PM
    YAML dump
  • d

    datajoely

    01/03/2022, 4:17 PM
    GoogleSheets dataset
  • â

    ÂNCD〆Kirito

    01/04/2022, 3:44 PM
    Hi, does anyone have experience in integrating Kedro pipelines through Flask RestAPI framework? I have a use case where I need to take input from user in UI and pass them to Kedro pipeline and provide the model output to UI through Flask API. TIA.
  • w

    waylonwalker

    01/04/2022, 3:54 PM
    Quick question. I am getting a bunch of warnings on some of my projects that getting pipelines from context will not be supported in 0.18.0. Is there a way to get pipelines from a kedro session that is compatable with both 0.17 and 0.18
  • d

    datajoely

    01/04/2022, 4:02 PM
    can you try this?
  • d

    datajoely

    01/04/2022, 4:03 PM
    So we don't support this out of the box @User recently published this plugin https://github.com/Galileo-Galilei/kedro-serving which may do what you need
  • d

    datajoely

    01/04/2022, 4:03 PM
    You can also look into how
    kedro-viz
    works as it does exactly this
  • w

    waylonwalker

    01/04/2022, 5:24 PM
    importing pipelines works, I also dug in deeper and figured out why this did not work previously (version mismatch kedro == 0.17.6 in one and kedro == 0.17.2 in the other)
  • w

    waylonwalker

    01/04/2022, 5:24 PM
    As always, thanks so much for the quick response @User
  • m

    metalmind

    01/05/2022, 4:44 PM
    Hello everyone.
  • m

    metalmind

    01/05/2022, 4:48 PM
    What is the best way to make feature and label generation configurable as one of the pipeline/node inputs same as raw data and make the associated scripts not part of the project source so they are stored in the same level as the data?
  • d

    datajoely

    01/05/2022, 5:17 PM
    Hi @User this is an interesting topic - we have had some early conversations with these folks re what an integration could look like https://labelstud.io/
  • d

    datajoely

    01/05/2022, 5:18 PM
    I'm not sure what you mean by
    pipeline/node inputs same as raw data and make the associated scripts not part of the project source so they are stored in the same level as the data?
  • d

    datajoely

    01/05/2022, 5:19 PM
    by I think the
    --params
    syntax or
    --config {some.yml}
    command may be your friend here https://kedro.readthedocs.io/en/latest/04_kedro_project_setup/02_configuration.html#configure-kedro-run-arguments
  • h

    hoodie

    01/05/2022, 5:21 PM
    Hello! I have multiple time series with exogenous params. I have one dataset for each time serie, and would like to create one model for each dataset, following the same pipeline. Is it possible to easily do it with kedro? 🤔 Thanks for your response :)
    d
    w
    • 3
    • 12
Powered by Linen
Title
h

hoodie

01/05/2022, 5:21 PM
Hello! I have multiple time series with exogenous params. I have one dataset for each time serie, and would like to create one model for each dataset, following the same pipeline. Is it possible to easily do it with kedro? 🤔 Thanks for your response :)
d

datajoely

01/05/2022, 5:22 PM
Yes! we are currently updating our docs to better explain this feature
but you can follow the docs here https://kedro.readthedocs.io/en/stable/06_nodes_and_pipelines/03_modular_pipelines.html
and follow my example project here https://github.com/datajoely/modular-spaceflights
You can reuse the same pipeline multiple times with different inputs
https://github.com/datajoely/modular-spaceflights/blob/main/src/modular_spaceflights/pipelines/modelling/pipeline.py
here is an example how we use the same pipeline twice, using diff modelling techniques
w

Wit

04/24/2022, 7:46 PM
Can modular pipelines be data driven? Taking the time series example - I want to build models for each time series but time series ids are in db and change in time. For example in a retail chain some products are removed or added.
d

datajoely

04/25/2022, 9:17 AM
so yes, you're able to do so - but it does harm reproducibility
As a team we don't mind dynamic pipelines, but believe in order to make them maintainable the inputs can be dynamic - but the structure should be static
The other way you can do this is loading/saving some checkpoints to another dataset to retain some idea of 'state'
happy to help you think through the problem
w

Wit

04/28/2022, 7:47 AM
Thanks for reply. I know it harms reproducibility. I personally prefer have one pipeline and run it many times with different inputs.
View count: 2