AnnaRie
05/18/2022, 6:59 PMantony.milne
05/18/2022, 8:32 PMafter_catalog_created
hook a bit like this one: https://kedro.readthedocs.io/en/stable/extend_kedro/hooks.html#hook-implementation
load_versions
will be a dictionary of the form {dataset_name: load_version}
, which you can then log by using self._logger.info(load_versions)
MetricsDataSet
. Each time you do a kedro run, the run command you use is saved as part of the information that can be showed in kedro-viz. So you can ensure reproducibility that wayAnnaRie
05/19/2022, 7:11 AMafter catalog created
but I don‘t understand, how to access the load version…
I do have a catalog.yml and some information from my in catalog.yml specified information is added to a logfile.
I also have a hooks.py, where I added after_catalog_created
(some parameters, one of it is load_versions: Dict[str, str]
and I return DataCalatlog.from_config(..., load_versions, ...)
.
In catalog.yml I predefine a model
(with type (PickleDataSet), datapath, versioned=True, layer).
I start the pipeline with kedro run --pipeline pr --load-version="model:2022-05-15T05.24.31.017Z
.
How can I accedd now this load version (2022.05.15...)?antony.milne
05/20/2022, 12:30 PM# hooks.py
class DataCatalogHooks:
@property
def _logger(self):
return logging.getLogger(self.__class__.__name__)
@hook_impl
def after_catalog_created(self, catalog: DataCatalog, load_versions: Dict[str, str]) -> None:
self._logger.info(load_versions)
And then make sure DataCatalogHooks()
is in HOOKS
in settings.py 🙂AnnaRie
05/23/2022, 8:20 AM