Title
w
waylonwalker
07/27/2022, 9:13 PMHey fellow Kedroids, has anyone had success using an external secrets manager with kedro. I think this is the last thing I need to update to get my team from 0.17.7 to 0.18. Currently I am getting errors in my tests telling me that credential keys are missing, and warnings showing me that I do not have any credential files. We use aws secrets manager with a custom hook that loads the credentials and sets them during register_catalog.
following how the ipython integration loads kedro, I get to context.catalog when I get this error.
d
datajoely
07/27/2022, 9:20 PMI think you may need an empty credentials.yml in conf
w
waylonwalker
07/27/2022, 9:31 PMThat will get rid of the WARNING's, but the ERROR telling me that a KEY is missing wont go away will it?
d
datajoely
07/27/2022, 9:31 PMCan you post the error?
w
waylonwalker
07/27/2022, 9:32 PMKeyError: "Unable to find credentials 'DATABASE_NAME': check your data catalog and credentials configuration. See https://kedro.readthedocs.io/en/stable/kedro.io.DataCatalog.html for an
example."
d
datajoely
07/27/2022, 9:33 PMOkay so your aws secrets aren't being passed or you need to mock them in your tests
w
waylonwalker
07/27/2022, 9:33 PMok, let me check
I added a breakpoint, but the breakpoint was not hit
d
datajoely
07/27/2022, 9:36 PMAh this isn't how the catalog works in 0.18.x
You need to read the upgrade guide but this now lives in settings.py
w
waylonwalker
07/27/2022, 9:37 PMah ok, I should read more 😉
I did add my ProjectHooks to settings.py
d
datajoely
07/27/2022, 9:38 PMSo I'm on my phone
But we don't do it like that
w
waylonwalker
07/27/2022, 9:38 PMwill look at the upgrade guide though
I will check your template
d
datajoely
07/27/2022, 9:39 PMIn settings.py we now have DATA_CATALOG_CLASS and DATA_CATALOG_ARGS
You can do your trick in the latter
w
waylonwalker
07/27/2022, 10:18 PMI got somewhere by defining my own
_get_config_credentialsCONTEXT_CLASSThis is what ended up with inside of
settings.pyIs there any appetite to make a credentials hook? I am not sure that I did this the right way, or same way as @datajoely would have done it, but I had to override a private method to get it done.
d
datajoely
07/28/2022, 3:56 PMI don't think anything you've done is wrong, there is potential in defining your context as well. Credentials are going to be revamped as a bigger configuration overhaul too
w
waylonwalker
07/28/2022, 5:19 PMwith the overhaul, can I give kedro a function that returns a dictionary of credentials easily?
I'm surprised this does not come up more. I'm sure others are using some sort of password vault/store/manager.
d
datajoely
07/28/2022, 5:26 PMI think people do it with the config loader not context
a
antony.milne
08/02/2022, 2:43 PM@waylonwalker very relevant, please do add your comments here! 🙏 https://github.com/kedro-org/kedro/issues/1646
t
trfir
08/10/2022, 6:50 AMHi Antony, I have seen your proposal about the credentials being loaded as env variables on github. I think it is a very good idea. Is this proposal going forward?
d
datajoely
08/10/2022, 6:51 AMIt's part of a wider configuration overhaul, so it's still being worked on. You can tweak things today with a custom config loader too.
t
trfir
08/10/2022, 6:57 AMdo you mean using TemplatedConfigLoader or something else? btw @datajoely great tutorial on youtube, very clear I really enjoyed it
d
datajoely
08/10/2022, 9:02 AMit's going to get a lot easier in the future
t
trfir
08/10/2022, 9:29 AMThanks, yeah I will do it like that for now.
Based on Antony's proposal, I was able to hack together a hook, which takes credenitals.yml and uses omegaconf to resolve the environmental variables and fills them into the catalog (see the attached image)
Then you just define your dataset as:
sqlite_table:
type: pandas.SQLQueryDataSet
credentials: sqlite_db_credentials
sql: select * from products
and credentials as:
sqlite_db_credentials:
con: "sqlite:///${oc.env:MY_DB_NAME}"
with export MY_DB_NAME=test.db
The hook will then replace MY_DB_NAME with test.db