Option 1: Pure Kedro

Set up your config like this:

conf
├── base
│   └── catalog.yml
├── staging
│   └── globals.yml
└── prod
    └── globals.yml

Template your

catalog.yml

like this:

yaml
companies:
  type: pandas.CSVDataSet
  filepath: ${base_location}/.../x.csv
  layer: raw

Then in two two

globals.yml

declare

base_location

argeting two different buckets

Then all you have to do is run

kedro run --env staging

or

kedro run --prod

to get two different locations targeted

Option 2: Add Env variables to

TemplatedConfigLoader

registration:

python
def register_config_loader(self, conf_paths: Iterable[str]) -> ConfigLoader:
    return...        
        globals_pattern="*globals.yml",
        globals_dict={
            k: v for k, v in os.environ 
            if k.startswith("S3_key")
        },
    )

That way you can can set

S3_key_staging

and

S3_key_prod

in your environment variables and they will be available at runtime

In terms of option 1 - the state is readable in code by someone else, option 2 may be lost in version control since it happens outside of Kedro. Both are valid

make sense?

Option 2 seems like it would work well.

I love the idea of capturing all state in code, but the issue with that is if you are exposing execution of a ML pipeline to some user input (let's say a user uploading a file via some UI), then that's obviously not possible.