It’s inspired by the pandas generic dataset actually, but the idea is to be able to pass arbitrary load / save callables that follow the required signature.

What other targets would you imagine here? The rationale for the pandas one was more about us having a best effort way of not having to create a new implementation whenever they release something new like XML.

The generic-ish datasets I've been planning on pitching/developing have been around Snowpark and Ibis

In summary I'm intrigued by your pitch, but would love to learn more about what pain points it would solve

Perhaps raising a feature request on GitHub is the best way to pitch this, kick the tyres and to give you an idea of what the steering committee would accept as a contribution

One that comes to mind is supporting xarray NetCDF files or arviz InferenceData files, which have fairly standard load / save APIs.

Super interesting and something we would likely want to support

I wonder if there is slightly less generic middleground - what about a common superclass that we can just do trivial implementations of? I think there is an argument that having explicit names is good for newbies and we should do generics if they have a common parent i.e. pandas or are actually generic natively e.g. spark

A less generic middle ground could be to have generic dataset for each library then just specify the reader / writer methods I guess. The other route is to make implementation of dataset as simple as providing load / save functions with the right signature. Will try to create a feature request in the next few days.

There is some interest in Xarray NetCDF before, I am not particularly familiar with it, but as I understand this is a common data format in the scientific community. https://github.com/kedro-org/kedro/issues/1346

Hello, I have a rather nebulous issue with Kedro which I'm curious if you've ever encountered. I'm trying to modify the runner so that everything runs on Batch by following the documentation on this page: https://kedro.readthedocs.io/en/latest/deployment/aws_batch.html Everything seems quite straightforward, but one of the steps is to add a custom cli in the same location as the settings.py file using the following template: https://kedro.readthedocs.io/en/latest/development/commands_reference.html#customise-or-override-project-specific-kedro-commands Again, not very complicated according to the doc and I found that the file is resolved by the main on the other hand when I execute the kedro run command the two files are ignored so my question is, did you do a custom cli and encountered this problem?

hello, i have some questions about datasets that are dependant on parameters... For example, if I have a sqlquerydataset where the sql query has a where statement, is there a way to change the value of the where depending on a parameter in the conf/base/parameters.yml ?

You can do this with globals.yml

great, i just looked at the documentation, thank you

1. sorry for my mistake this got confused by my team with the session.py that appears on the side. (none of my team put his hands on kedro so some questions are wrong assumptions perhaps) 2. yes I use kerdo-airflow plugin 3. I am struggling on this part, I have tried following most of what I find online but nothing comes closer to making kedro DAG runs (generic directional tips that don't translate into an operational tutorial perhaps) I start thinking we shouldn't run kedro pipelines in airflow 4. Yes this is related to kedro-airflow auto-generated dag. Here is the summary of what I am trying to do: 1- I want to use kedro to make the data science team make production-ready code from the get go 2- As an MLOps I want to automate the process so that the kedro pipeline can be made into a DAG without so much friction and can be orchestrated 3- This kedro DAG should be able to run stand-alone (conf files & data should be read from a bucket -or from DWH or redis- and not from local storage) -cannot be pushing data and files cluttering the repo so leveraging bucket storage- What's your take on my approach?

Kedro template configuration does not load globals.yml configuration into catalog.yml FOR Jupyter Lab https://stackoverflow.com/questions/74052421/kedro-template-configuration-does-not-load-globals-yml-configuration-into-catalo

Hi all, I've been exploring the kedro framework over the past week or two, and been building out a proof of concept pipeline for a data project in the company I work for. What I'd like to do next, and where I'm looking for some guidance/pointers, is to see if it's possible to build a pipeline where the structure of the DAG is defined in data. I'm hoping that it's possible to build a set of templates, and then enable a less technical user to join them together via some sort of data input (JSON/YAML/...) from the data catalog? I've done this sort of thing in Airflow before, but really like the concept of connected pipelines and the use of a data catalog in Kedro. If feels like the register_pipelines might be the place to do this, but it doesn't seem to take any args? So, any tips?

TypeError: __init__() got an unexpected keyword argument 'config_loader' https://stackoverflow.com/questions/74057556/typeerror-init-got-an-unexpected-keyword-argument-config-loader

Hi fellows, I created a custom dataset to handle templated (Jinja) SQL queries. This kind of dataset would be useful for several projects. How would you deal with sharing the dataset across different kedro projects? The documentation for custom dataset talks about extending kedro extras but I don't think that this little implementation is general enough.

kedro PartitionedDataSet lazy writting to spare memory? https://stackoverflow.com/questions/74066621/kedro-partitioneddataset-lazy-writting-to-spare-memory

I think what u need is distribute this shared datasets as a python package. You can use a kedro project or just a plain python package since it won't have any pipeline in it.

Thanks @noklam , that's what I had in mind. 👍

Do you mean constructing the pipeline dynamically on the fly? You can achieve this via the hooks but it isn't something we encourage in general. Part of the reason is that we think static pipeline is easier to understand and reproduce, when you hide too many switch in the pipeline, it's difficult to reason about it.

Hi, I bumped into an unexpected behavior and would like your feedback if I did something wrong. I built a custom dataset which inherits from SQLQueryDataSet. The latter has a parameter filepath. In the former, I, first, added a parameter templated_filepath which I then pass into SQLQueryDataSet as filepath=templated_filepath. Here is the catch. If I use the custom dataset in a notebook with something like data/01_raw/blabla as templated_filepath, the catalog breaks because the full path includes notebooks/, i.e. it's using the directory of the notebook to build the complete path. But, if instead of calling the parameter templated_filepath, I use the genuine naming filepath, then everything is fine. Is there some magic around the name filepath when the catalog is parsed to create the datasets?

About paths in the catalog

@noklam yes that’s exactly what I meant. I get that it adds a significant layer of complexity, however I’m not sure that I see a better way of enabling users to construct a pipeline from a pre-defined set of building blocks without writing code. Thanks for the pointer towards the hooks, I’ll give that a look.

Include Quarto rendering in kedro pipeline and pass it inputs/outputs https://stackoverflow.com/questions/74099238/include-quarto-rendering-in-kedro-pipeline-and-pass-it-inputs-outputs

Hi, is there a way to have the configurations in settings.py configurable based on the kedro environment. For example, I want the hooks for kedro-viz and kedro-neptune disabled in production environment. Thanks

Interesting usecase - best solution off the top of my head is to use the KEDRO_ENV environment variable and just include an if statement in settings.py

I was also thinking of a using an ENV var, glad to hear that there is one already defined for the KEDRO_ENV! Thanks