Hi Dhaval, don't think I might be right but just throwing my thoughts on why this could be happening. Is it possible that the kedro context initiated first time is being reused on the second run?

I'm guessing it's the same thing too. The flow that is being used has a fixed timestamp at all times, which doesn't make sense. There's something that goes in the backend and I don't know how to change the timestamp for the runs 😅

I'll try to replicate the issue, would following this guide[https://kedro.readthedocs.io/en/stable/10_deployment/05_prefect.html] replicate your setup?

Yes, this is exactly what I've used. Versioning datasets are causing issues while executing quick runs repetitively for a flow

prefect-versioned-datasets

Hello everyone! I wanted to know why kedro owns base and local env ? and what each one dedicated for ? why i can not create my own default env ?

Hi @User this should explain everything... https://kedro.readthedocs.io/en/latest/04_kedro_project_setup/02_configuration.html#additional-configuration-environments

not sure how I added that sticker

¯\_(ツ)_/¯

but the logic is that

base

is committed to git,

local

isn't and overrides any duplicate keys

you can then add additional `env`s in the docs above

Hi! is there a way to show the list of all the versions of a dataset in the catalog?

Hi! has anyone ever tried to run a pipeline in jupyter returning a plot? everything works till the and the figure is rendered correctly, but it raises the error

AttributeError: 'Affine2D' object has no attribute '_boxout'

what kind of plot is this? Matplotlib?

yes

I'm running it in this way :

SequentialRunner().run(pipeline=my_pipeline.from_nodes("node1").to_nodes("node10"),catalog=test)

the plot node is a memory dataset and is the last one. it is returned as a dict

{"node10": <Figure size 720x720 with 1 Axes>}

If it gets to that point I'm not sure it's a Kedro issue

it's an issue with Matplotlib render step

so i would hunt around their docs

or use Plotly because it's way easier

ok thanks !

Hi! Is there any way to save the catalog to config. I sow the from_config method but no to_config one. Thanks !

I don't think this exists today - it's something we've thought about before

I think it would be nice to be able to save the state of the catalog being used, especially if something is modified by the hooks.

Yeah not many people have asked for it so we haven't built it before, it feels quite doable

You could mimic something like this with

datasets = {dataset: dataset._describe() for dataset in catalog.datasets}
yaml.dump(datasets)

This won't give exactly the reverse of

from_config

but will go some of the way. Possibly

str(dataset)

might be useful also.

thanks for your suggestion! I' will definitely try it today.

Happy 2sday everyone! I have a SQL YAML question. One of the items in my

parameters.yml

is a long complicated SQL string:

sql_param: >
  SELECT
  ...

The weird thing is that running it in through the node produces errors. If I copy and paste the SQL string to a DB tool, it runs fine. Is yaml or Kedro treating some characters in the SQL script differently? Is there a better way to make a node run SQL? Can I use pandas.SQLQueryDataSet even though the query is not supposed to return anything?

Hi folks. I have some 50GB of data in a data warehouse (redshift). I'd like to have the initial processing (e.g. raw->primary) be done IN the warehouse to avoid heavy I/O out of the warehouse to do simple SQL queries in python. How would one handle such a scenario best? I could see two approaches: 1. run the SQL query in a kedro node with a dummy input & output to put it in the right place in the DAG 2. run the SQL query outside of kedro, e.g. in an orchestrator like airflow and do

SQL -> kedro_pipeline()