https://kedro.org/ logo
#advanced-need-help
Title
# advanced-need-help
a

anhoang

06/24/2021, 6:17 PM
I sometimes use Kedro nodes and pipelines in my projects, but I use Kedro's Data Catalog for all of my projects! It's so useful. Trying to convince others in my team to also use the same data catalog is challenging, since the data catalog only works in Python. My team uses Bash, awk, and R also (bioinformatics). I'm thinking to just enable a mapping between dataset name and file path. To do this I would use a shell script that parses the yaml file and save this mapping into environmental variables. This way they can access the dataset-file path mapping in any language. This script would be executed in the team's
.bashrc
for team-wide dataset and be placed in the Kedro project root directory for project-specific-datasets. Any suggestions of other/easier ways to utilize the same data catalog across different languages?
5 Views