Which version of Kedro are you working with?

There were fixes to this dataset in 0.17.4 and 0.17.1

Hi everyone! Is there any way to generate the catalog config from a DataCatalog object. Something like DataCatalog.to_config() ? Has anyone already done something like this ? Thanks in advance for your help!

Versioned SparkDataSet problem

I'm working with Kedro 0.17.4

Perfect. Thanks! Obviously a very context-dependent question, but I just wanted to get a general sense for a good starting point.

monorepo?

Hello everyone. I was thinking about pipeline reuse in kedro and want to know the best way to do that. In my first pipeline, I segment my dataset into subsets dynamically. After that, I need to run a second pipeline for each set. Is it possible to run the same pipeline for each input dynamically? Does anyone already handle dynamic inputs?

So the discussion on dynamic pipelines comes up a lot. Our default position is that readability is the most important thing. You can use Jinja to define dynamic catalogs and regular Python to generate dynamic pipelines. That being said - we've observed that things become very difficult to maintain things when the pipeline logic only really exists at runtime rather than rest. With this in mind, if you do dynamic pipelines I'd encourage you to dynamically generate pipelines that are immutable in structure, but dynamic in terms of inputs/outputs. If you start building conditional logic into your pipeline generation it becomes very difficult for other team members or 'future you' to read in 6 months time

The other thing to keep in mind is custom configuration environments https://kedro.readthedocs.io/en/latest/04_kedro_project_setup/02_configuration.html#additional-configuration-environments This is where you have mirrors of the same datasets for different environments e.g. staging vs prod

@User it looks like a set of recent releases from dynaconf are causing breaking effects (shown below). It seems stable on dynaconf==3.1.5.

Yes we’ve just got reports of this elsewhere

I’m looking into it

Thanks @User - we'll put a proper fix out shortly, but for now please pinning to 3.1.5 looks like it fixes things!

Had same issue with dynaconf 3.1.7 today. Rollbacked to 3.1.4 it was fine. Will test with 3.1.5.

0.17.5 was imminent annyway and we'll put a fix for this in too

@User do you have an estimated release date for 0.17.5? Also, are there expected to be any breaking changes?

Next week is my hope - we do our absolute to not do any breaking changes in 'minor releases' e.g. 0.17.4->0.17.5

the next breaking changes will happen in 0.18.0 which I'm hoping for Q4 2021

and that will put us on the path to have a really stable API ready for 1.0

@User It should be out by tomorrow! We also have a new release for Kedro-Viz coming out soon.

And we have a new Kedro release out! Check the #778996598880862208 channel!

I might be completely stupid, but is it possible to split the catalog.yml file into different yaml files that kedro can integrate with?

Yes

To it being possible not you being stupid!

Essentially we match a pattern

So being the scenes we do this

conf_loader.get('catalog*', 'catalog*/**')

Which means we will scan the conf folder for any yaml files either prefixed with the word catalog or within a sub folder called catalog

Does the new catalog lazy loading somehow hit the describe functionality of a dataset now, prior to the dataset being initialized?

Hi. Am I correct in suspecting that the

AbstractDataSet

examples in https://kedro.readthedocs.io/en/stable/07_extend_kedro/03_custom_datasets.html are actually using code from the

AbstractVersionedDataSet

?