https://kedro.org/ logo
#beginners-need-help
Title
# beginners-need-help
w

waylonwalker

07/27/2022, 9:13 PM
Hey fellow Kedroids, has anyone had success using an external secrets manager with kedro. I think this is the last thing I need to update to get my team from 0.17.7 to 0.18. Currently I am getting errors in my tests telling me that credential keys are missing, and warnings showing me that I do not have any credential files. We use aws secrets manager with a custom hook that loads the credentials and sets them during register_catalog.
following how the ipython integration loads kedro, I get to context.catalog when I get this error.
d

datajoely

07/27/2022, 9:20 PM
I think you may need an empty credentials.yml in conf
w

waylonwalker

07/27/2022, 9:31 PM
That will get rid of the WARNING's, but the ERROR telling me that a KEY is missing wont go away will it?
d

datajoely

07/27/2022, 9:31 PM
Can you post the error?
w

waylonwalker

07/27/2022, 9:32 PM
KeyError: "Unable to find credentials 'DATABASE_NAME': check your data catalog and credentials configuration. See https://kedro.readthedocs.io/en/stable/kedro.io.DataCatalog.html for an example."
d

datajoely

07/27/2022, 9:33 PM
Okay so your aws secrets aren't being passed or you need to mock them in your tests
w

waylonwalker

07/27/2022, 9:33 PM
ok, let me check
I added a breakpoint, but the breakpoint was not hit
d

datajoely

07/27/2022, 9:36 PM
Ah this isn't how the catalog works in 0.18.x
You need to read the upgrade guide but this now lives in settings.py
w

waylonwalker

07/27/2022, 9:37 PM
ah ok, I should read more 😉
I did add my ProjectHooks to settings.py
d

datajoely

07/27/2022, 9:38 PM
So I'm on my phone
But we don't do it like that
w

waylonwalker

07/27/2022, 9:38 PM
will look at the upgrade guide though
I will check your template
d

datajoely

07/27/2022, 9:39 PM
In settings.py we now have DATA_CATALOG_CLASS and DATA_CATALOG_ARGS
You can do your trick in the latter
w

waylonwalker

07/27/2022, 10:18 PM
I got somewhere by defining my own
_get_config_credentials
in a custom
CONTEXT_CLASS
. Currently I am able to just keep all credentials to get it to a working state. Previously it only got the credentials it needed by seeing them in the catalog. The way I am trying to do it I don't think I can do that.
This is what ended up with inside of
settings.py
.
Is there any appetite to make a credentials hook? I am not sure that I did this the right way, or same way as @datajoely would have done it, but I had to override a private method to get it done.
d

datajoely

07/28/2022, 3:56 PM
I don't think anything you've done is wrong, there is potential in defining your context as well. Credentials are going to be revamped as a bigger configuration overhaul too
w

waylonwalker

07/28/2022, 5:19 PM
with the overhaul, can I give kedro a function that returns a dictionary of credentials easily?
I'm surprised this does not come up more. I'm sure others are using some sort of password vault/store/manager.
d

datajoely

07/28/2022, 5:26 PM
I think people do it with the config loader not context
a

antony.milne

08/02/2022, 2:43 PM
@waylonwalker very relevant, please do add your comments here! 🙏 https://github.com/kedro-org/kedro/issues/1646
t

trfir

08/10/2022, 6:50 AM
Hi Antony, I have seen your proposal about the credentials being loaded as env variables on github. I think it is a very good idea. Is this proposal going forward?
d

datajoely

08/10/2022, 6:51 AM
It's part of a wider configuration overhaul, so it's still being worked on. You can tweak things today with a custom config loader too.
t

trfir

08/10/2022, 6:57 AM
do you mean using TemplatedConfigLoader or something else? btw @datajoely great tutorial on youtube, very clear I really enjoyed it
it's going to get a lot easier in the future
t

trfir

08/10/2022, 9:29 AM
Thanks, yeah I will do it like that for now.
Based on Antony's proposal, I was able to hack together a hook, which takes credenitals.yml and uses omegaconf to resolve the environmental variables and fills them into the catalog (see the attached image) Then you just define your dataset as: sqlite_table: type: pandas.SQLQueryDataSet credentials: sqlite_db_credentials sql: select * from products and credentials as: sqlite_db_credentials: con: "sqlite:///${oc.env:MY_DB_NAME}" with export MY_DB_NAME=test.db The hook will then replace MY_DB_NAME with test.db
34 Views