Title
s
s.hedayati
06/30/2022, 11:20 AMI am a little new to kedro. Could you please post the code that can do that?
d
datajoely
06/30/2022, 11:20 AMAbsolutely!
so this is very possible, but there are two steps to do
1) You need to change your config loader to use a version of
TemplatedConfigLoaderglobals.yaml${variable}2) Now to include the environment name within the 'global' scope you need to actually define a very simple custom
TemplatedConfigLoaderThis was actually asked previously and you can steal the implementation here:
https://discord.com/channels/778216384475693066/778998585454755870/980886228700385330
s
s.hedayati
06/30/2022, 11:27 AMthanks alot. But isn't there a simpler way to get the given parameters in cli within the code? for instance something like:
'''
from kedro.framework.context import KedroContext
print( KedroContext.env)
'''
d
datajoely
06/30/2022, 11:28 AMyes much simpler than that!
If you ever find yourself creating a context or session you have likely gone too far!
The instructions are here
you can create your own CLI.py
and then override the run command as you want
also the
before_pipeline_runafter_context_createds
s.hedayati
06/30/2022, 11:43 AMSorry for misunderstanding. Actually, I don't want to change anything about cli.py
basically I want give this command in my cli:
kedro run --env=prod
then get this environment name ('prod') within my script by calling something like :
print(context.env)
I do not want to change anything else. Just extract the info given to cli
n
noklam
06/30/2022, 12:33 PM@s.hedayati Can you elaborate a bit what you are trying to do with the
envs
s.hedayati
06/30/2022, 12:38 PMcould you please provide the code snippet of hooks, "after_context_created" which returns the given env name in cli ? I could not find any example for that
d
datajoely
06/30/2022, 1:09 PMHi sorry I was at lunch
So I think we can help you do a much better way of doing this
The
before_pipeline_runcatalogenvs
s.hedayati
06/30/2022, 1:13 PMI have defined a customized data_catalogs.py which takes the env name as input, then based on that creates that data catalog. So, if I pass "Kedro run --env=test or --env=prod" to cli, then I can get the env value within this customized catalog.py file before it gets loaded
Shouldn't be 'after_context_created' ? since this customized data_catalog is hooked with after_catalog_created? I want to use env value inside this data catalog before loading
but this is not working. Could you please tell me how should I fix this?
d
datajoely
06/30/2022, 1:56 PMWhat do you want to use the env variable to do
To change the file paths?
n
noklam
06/30/2022, 2:17 PMI think Joel is asking this because if you just need different path. You don't need to create a custom data catalog, I am very curious what does that file does.
1. Use jinja syntax and pass
envcatalog.ymlbase/catalog.ymlprod/catalog.ymls
s.hedayati
06/30/2022, 2:45 PMwell, based on the given env, it will access different databases in different environments. So, before this catalog is created( which by the way uses after_catalog_created as hooks) the env should be passed to this script, then the scripts chooses the connection url ( whether env is test/prod) then it will be passed to the hook to register that data catalog
what I want is really simple, inject env to my catalog script then use that script to register data catalog with anothe hooks. I only need that as entry point to my script before it is called by the other hooks
d
datajoely
06/30/2022, 3:19 PMOkay
so kedro has this built in
it's why we provide the
envso under
conf
|_base
|_local
|_prod <-- add this folder
|_staging <-- add this folderthen if you put a
catalog.ymlprodstagingthen all you have to do is
sh
kedro run # will take base
kedro run --env prod # will take prod
kedro run --env staging # will take stagingthis is already built in
Kedro is built against these principles https://12factor.net/config
s
s.hedayati
07/01/2022, 7:15 AMThank you very much for taking your time to help me. I will try to rearrange the catalog.yml files. in separate folders.
Unfortunately, the catalog.yml in my case need env name in order to be created. How can I pass this to my script before the catalog is created ?
d
datajoely
07/01/2022, 9:29 AMSo we, by design, don't recommend you generate the
catalog.ymlwhat does you catalog end state look like, we can maybe help you get there a more 'official' way?
s
s.hedayati
07/01/2022, 12:40 PMwell, the script, based on the given environment supposed to read all tables from database ( which is alot!) then set them az read and write data catalogs. The Problem is that, I need to give env name to read these tables first, then create a data catalog then register those. Now I can do all this, but each time before I run, I need to specify manually which env, it needs to read the tables from, which is not very efficient. I was hopping with some hooks, after_context_created pass this env variable to my data_catalog script and then use those in another hooks ( after_catalog_created) to add them to my data catalogs.
d
datajoely
07/02/2022, 2:41 PMHey sorry - only just got a chance to reply
So this is definitely achievable, but I'm not sure I would build this into the Kedro workflow just because it feels like it will be a bit fragile and hard to maintain.
Could you elaborate on what you mean by:
" read all tables from database ( which is alot!) then set them az read and write data catalogs."
Is the name / schema of the tables not known beforehand?
s
s.hedayati
07/06/2022, 8:21 AMHi, sorry for super late response. Actually, I was able to solve this with "after_context_created". Since this is the first hook, which is implemented, I could get the env name and pass it to my data_catalog script. Thanks for your support, really appreciated
d
datajoely
07/06/2022, 8:21 AMAmazing!