For example https://kedro.readthedocs.io/en/stable/kedro.extras.datasets.pandas.SQLQueryDataSet.html

hi there, i have a question about pipeline design. i am working on a nlp project where i built several text processing pipelines for english text from different data sources, for example i have: - [env: preprocess] fetch data from source 1 - [env: preprocess] fetch data from source 2 - [env: base] preprocess - [env: base] NER - [env: base] text summarization - ... now i would like to scale to more languages and more data sources, my initial thought is that i may need to duplicate my base env per each language i support and manually update all the catalog/params by myself (although they are just conventions like prefixing by "en" or "ja" or "fr" etc). is there more "Kedro" way to do accomplish my goal?

This is a very good question - we tend to prefer explicit versus implicit. But if you have a stable pipeline, you could probably get hooks to work to ensure the catalog is tailored to the language in question

thanks @datajoely for the reply. i do see the benefits of explicit over implicit when there are a manageable number of distinct pipelines. but in my case, making one pipeline for each different language/data source implies lots of nearly duplicated codes in the repository, which i hope to avoid. i will look into your suggestion on getting hooks to work on mutating the state of the Kedro catalog.

So but alcmake sure you're using the modern modular pipeline technique of overriding inputs/outputs

essentially you can use namespaces to do a lot of the heavy lifting

this seems to be a feature introduced in 0.18.0, i am still at 0.17.5 let me check the documentation and how much work is needed for migrating to 0.18.x

No it should be there: https://kedro.readthedocs.io/en/0.17.7/06_nodes_and_pipelines/03_modular_pipelines.html

The 0.17.5 docs are uglier, but the functionality should be the same

Just encountered this error. Any fix?

Never mind found the solution

Rate limiting Kedro API requests https://stackoverflow.com/questions/72691571/rate-limiting-kedro-api-requests

What was the solution? Would you mind adding it to that StackOverflow post

I replied

I upgraded the protobuf package pip install --upgrade "protobuf<=3.20.1"

Amazing thanks

Access configuration in the pipelines definition (not only nodes) https://stackoverflow.com/questions/72700321/access-configuration-in-the-pipelines-definition-not-only-nodes

Hi all. I migrate an old Kedro project to the new version. In the old version I updated dynamically my conf in the run.py file when I loaded the configloader. I know it’s not a good practice but for the moment I need to do it. Can you tell me what is the « less worse » way to do this in kedro >0.18 ? Hooks ? Thank you very much for your help.

Hello! This depends on exactly what the custom code was doing. Generally speaking there are two routes here: * hooks, especially the new

after_context_created

one * write a custom

ConfigLoader

and reference it in

CONFIG_LOADER_CLASS

in settings.py This is quite a common question so if you can say a bit more about what your custom code does we can probably point you to examples of others who have done very similar!

Thanks you for your reply ! My code retrieve a date and create a yyyymm variable from this date and insert this new variable in the globals.

The yyyymm is used in the catalog for my outputs

ok, the simplest way to do this would be to go into settings.py and uncommnent and modify the bit of code about

CONFIG_LOADER_CLASS

. It should look something like this:

# Class that manages how configuration is loaded.
from kedro.config import TemplatedConfigLoader
CONFIG_LOADER_CLASS = TemplatedConfigLoader
# Keyword arguments to pass to the `CONFIG_LOADER_CLASS` constructor.
CONFIG_LOADER_ARGS = {
     "globals_dict": {"date": get_date_in_yyyymm_format()},
}

and then you can use

date

as a variable in your catalog.yml file.

Kedro: load existing data catalog programmatically https://stackoverflow.com/questions/72752043/kedro-load-existing-data-catalog-programmatically

Thanks !

Hi everyone, Is anyone running kedro + airflow in a "production" setting (e.g. batch prediction, automated retraining etc) and would be interested in a knowledge exchange maybe 30-60 min on their experience and lessons learned? 🙂

Could you point me towards the solution? I have the same read() not implemented for BaseSessionStore as well as save() not implemented for BaseSessionStore.

That is not an error but just INFO logging.

Does anybody have practical experience deploying Kedro pipelines as Argo Workflows? I have a couple thoughts/questions around the approach currently recommended in https://kedro.readthedocs.io/en/stable/deployment/argo.html: + Based on your experience, are nodes the correct level for containerization? Should it be one modular pipeline per step instead? The whole pipeline in one step? + Did you consider passing data between workflow steps (see https://github.com/argoproj/argo-workflows/blob/master/examples/artifact-passing.yaml )? Would it be an issue if all intermediate data passing happened like this? + Did the suggested approach (or whatever approach you took) not satisfy certain needs? Could some aspects have been easier? Please also feel free to include any other information w.r.t. your experience deploying to Argo Workflows. Thanks!!

Hi, In my new project, we are forced to use Kubeflow. Is it possible to separate pipelines into containers without creating new repositories for each pipeline?