Thank you for the reply. I want to use the envs, yes, but I want to have pipelines that will sample my prod data to the test data with pipelines (or nodes). So I have to have a pipeline that goes from one env to the other. The way I thought to achieve this is by having a setting that always points to the test data (test) and one that can either point to the test data or to the prod data (base). In that case, the environment would make the base point to the test, so I can run stuff on a smaller dataset, and the base would point to the prod on the cloud, to use the huge datasets.

Hi @User I'm not sure if I'm following entirely, so let's start a thread to work through this. I think the answer we're going to end up will end up with more duplication than we'd like.

Wow, thanks!

As I understand it we just need to have mirror catalogs in the folder structure. You can also do this trick to inject environment variables into your TemplatedConfigLoader scope:

python
def register_config_loader(self, conf_paths: Iterable[str]) -> ConfigLoader:
    return...        globals_pattern="*globals.yml",
        globals_dict={
            k: v for k, v in os.environ 
            if k.startswith("XXXXX")
        },
    )

Basically, I want the ability to do (1)

kedro run --pipeline create-test-data

which is going to subset my prod data to my test data (which is sized in a way I can use in local) While still having the ability to run (2)

kedro run --pipeline my-usual-pipeline --env test

What complicates this is that the (1) needs to run from the prod data to the test data

So I think that's possible, but you need duplicate catalog entires

I planned on having duplicate catalog entries (thought they would be needed). In which case I wanted to make the test always available, you can always use it. The other one would be a "base", which can point to the test or to prod, depending on which --env you pass

Yeah I think it's probably the best way of doing it

I'll take a look at it (tomorrow, can't get my hands on it rn)

Yeah it's chunky, but any feedback can help us steer the future of what I feel is our biggest area of overhead