s.hedayati
06/30/2022, 11:20 AMdatajoely
06/30/2022, 11:20 AMTemplatedConfigLoader
, the instructions to do this are here:
https://kedro.readthedocs.io/en/stable/kedro_project_setup/configuration.html#template-configuration
This gives you a globals.yaml
that they lets you template your YAML using ${variable}
syntaxTemplatedConfigLoader
s.hedayati
06/30/2022, 11:27 AMdatajoely
06/30/2022, 11:28 AMbefore_pipeline_run
or after_context_created
hooks have much of this availables.hedayati
06/30/2022, 11:43 AMnoklam
06/30/2022, 12:33 PMenv
information?s.hedayati
06/30/2022, 12:38 PMdatajoely
06/30/2022, 1:09 PMbefore_pipeline_run
hook has access to both the catalog
object and the env
https://kedro.readthedocs.io/en/latest/kedro.framework.hooks.specs.PipelineSpecs.html#kedro.framework.hooks.specs.PipelineSpecs.before_pipeline_runs.hedayati
06/30/2022, 1:13 PMdatajoely
06/30/2022, 1:56 PMnoklam
06/30/2022, 2:17 PMenv
in the path
2. If your datacatalog is completely different, you can also just have 2 different catalog.yml
in base/catalog.yml
and prod/catalog.yml
s.hedayati
06/30/2022, 2:45 PMdatajoely
06/30/2022, 3:19 PMenv
argumentconf
|_base
|_local
|_prod <-- add this folder
|_staging <-- add this folder
catalog.yml
in each within the prod
and staging
folder and put different instances of your database in each file
if you use the same dataset name in eachsh
kedro run # will take base
kedro run --env prod # will take prod
kedro run --env staging # will take staging
s.hedayati
07/01/2022, 7:15 AMdatajoely
07/01/2022, 9:29 AMcatalog.yml
on runtime because this makes reproducibility difficults.hedayati
07/01/2022, 12:40 PMdatajoely
07/02/2022, 2:41 PMs.hedayati
07/06/2022, 8:21 AMdatajoely
07/06/2022, 8:21 AM