ende
11/16/2021, 8:22 PMdatajoely
11/16/2021, 8:24 PMconf
├── base
│ └── catalog.yml
├── staging
│ └── globals.yml
└── prod
└── globals.yml
Template your catalog.yml
like this:
yaml
companies:
type: pandas.CSVDataSet
filepath: ${base_location}/.../x.csv
layer: raw
Then in two two globals.yml
declare base_location
argeting two different bucketskedro run --env staging
or kedro run --prod
to get two different locations targetedTemplatedConfigLoader
registration:
python
def register_config_loader(self, conf_paths: Iterable[str]) -> ConfigLoader:
return...
globals_pattern="*globals.yml",
globals_dict={
k: v for k, v in os.environ
if k.startswith("S3_key")
},
)
That way you can can set S3_key_staging
and S3_key_prod
in your environment variables and they will be available at runtimeende
11/17/2021, 12:57 AMdatajoely
11/17/2021, 9:44 AMende
11/17/2021, 8:12 PMdatajoely
11/17/2021, 8:12 PMende
11/17/2021, 9:56 PM