https://kedro.org/ logo
Title
a

antony.milne

06/28/2021, 10:09 PM
Agreed - this is interesting and I'd like to hear more too 🙂 What you propose sounds very sensible. You could maybe also do it the other way round using TemplatedConfigLoader (https://kedro.readthedocs.io/en/stable/kedro.config.TemplatedConfigLoader.html), where you define the dataset name to filepath mapping in environment variables and then inject those into the kedro catalog. It depends where you want to define your original source of truth (kedro data catalog vs. environment variables) Also something that is maybe not so well known... kedro supports a variety of different file formats for config files including the catalog (yaml, json, ini, pickle, properties, xml). Although yaml is overwhelmingly popular, maybe one of the other formats is more familiar to your team/easier to parse/more natively supported in those other languages