06/15/2022, 5:17 PM
question: catalog.add() seems not change local config.file Description: 1. jupyter notebook import the existing catalog.yaml to load all data (works) 2. use catalog.add and save the a new dataframe (works, data in catalog.list() and DF can be found on disk.) 3. check the local config.yaml file, cannot find the item


06/16/2022, 5:37 AM
Continuing earlier convo here. One of the common ways I've seen users interact is to add them to conf files and access them in jupyter notebooks using
. As catalog objects in notebooks are the final form after resolving a lot of user specific definitions such as paths, credentials, jinja replacements of variables, so in my opinion it's not best to write/sync to global yaml directly from catalog. That said, I see how being able to write config to a temp file would be useful. If I may ask, how are you adding objects to catalog. I feel like adding each object to catalog manually might be tedious.
Something like this might for individual datasets and specific ones but I'm not confident it'd work for every dataset & will still need cleanup


06/16/2022, 4:15 PM
wow! Totally. this explanation is quite useful.