s.hedayati
06/30/2022, 11:20 AMdatajoely
06/30/2022, 11:20 AMdatajoely
06/30/2022, 11:21 AMdatajoely
06/30/2022, 11:22 AMTemplatedConfigLoader
, the instructions to do this are here:
https://kedro.readthedocs.io/en/stable/kedro_project_setup/configuration.html#template-configuration
This gives you a globals.yaml
that they lets you template your YAML using ${variable}
syntaxdatajoely
06/30/2022, 11:23 AMTemplatedConfigLoader
datajoely
06/30/2022, 11:25 AMs.hedayati
06/30/2022, 11:27 AMdatajoely
06/30/2022, 11:28 AMdatajoely
06/30/2022, 11:28 AMdatajoely
06/30/2022, 11:29 AMdatajoely
06/30/2022, 11:29 AMdatajoely
06/30/2022, 11:29 AMdatajoely
06/30/2022, 11:31 AMbefore_pipeline_run
or after_context_created
hooks have much of this availables.hedayati
06/30/2022, 11:43 AMnoklam
06/30/2022, 12:33 PMenv
information?s.hedayati
06/30/2022, 12:38 PMdatajoely
06/30/2022, 1:09 PMdatajoely
06/30/2022, 1:10 PMdatajoely
06/30/2022, 1:11 PMbefore_pipeline_run
hook has access to both the catalog
object and the env
https://kedro.readthedocs.io/en/latest/kedro.framework.hooks.specs.PipelineSpecs.html#kedro.framework.hooks.specs.PipelineSpecs.before_pipeline_runs.hedayati
06/30/2022, 1:13 PMs.hedayati
06/30/2022, 1:43 PMs.hedayati
06/30/2022, 1:48 PMdatajoely
06/30/2022, 1:56 PMdatajoely
06/30/2022, 1:56 PMnoklam
06/30/2022, 2:17 PMenv
in the path
2. If your datacatalog is completely different, you can also just have 2 different catalog.yml
in base/catalog.yml
and prod/catalog.yml
s.hedayati
06/30/2022, 2:45 PMs.hedayati
06/30/2022, 2:48 PMdatajoely
06/30/2022, 3:19 PMdatajoely
06/30/2022, 3:19 PMdatajoely
06/30/2022, 3:19 PMenv
argumentdatajoely
06/30/2022, 3:20 PMconf
|_base
|_local
|_prod <-- add this folder
|_staging <-- add this folder
datajoely
06/30/2022, 3:21 PMcatalog.yml
in each within the prod
and staging
folder and put different instances of your database in each file
if you use the same dataset name in eachdatajoely
06/30/2022, 3:22 PMsh
kedro run # will take base
kedro run --env prod # will take prod
kedro run --env staging # will take staging
datajoely
06/30/2022, 3:22 PMdatajoely
06/30/2022, 3:22 PMs.hedayati
07/01/2022, 7:15 AMs.hedayati
07/01/2022, 8:11 AMdatajoely
07/01/2022, 9:29 AMcatalog.yml
on runtime because this makes reproducibility difficultdatajoely
07/01/2022, 9:29 AMs.hedayati
07/01/2022, 12:40 PMdatajoely
07/02/2022, 2:41 PMs.hedayati
07/06/2022, 8:21 AMdatajoely
07/06/2022, 8:21 AM