question catalog add seems not change local config file Desc

Question

question catalog add seems not change local config file Description 1 jupyter notebook import the existing catalog yaml to load all data works 2 use catalog add and save the a new dataframe works data in catalog list and DF can be found on disk 3 check the local config yaml file cannot find the item

avan-sh · Answer

Continuing earlier convo here One of the common ways I ve seen users interact is to add them to conf files and access them in jupyter notebooks using `catalog` As catalog objects in notebooks are the final form after resolving a lot of user specific definitions such as paths credentials jinja replacements of variables so in my opinion it s not best to write sync to global yaml directly from catalog That said I see how being able to write config to a temp file would be useful If I may ask how are you adding objects to catalog I feel like adding each object to catalog manually might be tedious

avan-sh · Answer

Something like this might for individual datasets and specific ones but I m not confident it d work for every dataset will still need cleanup

JA_next · Answer

wow Totally this explanation is quite useful