austin-hilberg
01/26/2022, 10:38 PMdatajoely
01/26/2022, 10:39 PMdatajoely
01/26/2022, 10:40 PMaustin-hilberg
01/26/2022, 10:42 PMdatajoely
01/26/2022, 10:42 PMaustin-hilberg
01/26/2022, 10:44 PMlimdauto
01/26/2022, 10:47 PMafter_node_run
hook: https://kedro.readthedocs.io/en/stable/kedro.framework.hooks.specs.NodeSpecs.html#kedro.framework.hooks.specs.NodeSpecs.after_node_run -- use MemoryDataSet as data flow in Kedro but save to disk in after_node_run hook as backup. I think you can also do this async (eg in another process) so you don't block the pipeline execution.austin-hilberg
01/26/2022, 10:50 PMRRoger
01/27/2022, 5:28 AMdatajoely
01/27/2022, 7:52 AMDDank
01/28/2022, 12:21 PMdatajoely
01/28/2022, 12:51 PMPartitionedDataSet
and IncrementalDataSet
to help manage the sets of images.
In truth there looks like a bit of overlap between the parts Kedro tries to own in terms of (a) Config management (b) IO - so you can either choose to Kedr-ify these with our abstractions or have bits of your pipeline that use the native detectron2
stuff to do some processing steps and then switch to Kedro style pipelining downstream.DDank
01/28/2022, 10:30 PMRRoger
01/29/2022, 6:22 AMRRoger
01/29/2022, 10:54 AMbefore_pipeline_run
and after_pipeline_run
.JayG
02/01/2022, 3:34 PMdatajoely
02/01/2022, 3:56 PMDaehyun Kim
02/01/2022, 4:29 PMconf/base/logging.yml
?Daehyun Kim
02/01/2022, 4:30 PMdatajoely
02/01/2022, 4:30 PMlogging.dictConfig
anywaydatajoely
02/01/2022, 4:30 PMdatajoely
02/01/2022, 4:31 PMDaehyun Kim
02/01/2022, 4:31 PMdatajoely
02/01/2022, 4:33 PMDaehyun Kim
02/01/2022, 4:33 PMlogging.yml
and add my custom handler into it and pass it to logging.dictConfig in somewhere in any hooks?datajoely
02/01/2022, 4:34 PMkedro-telemetry
plugin it doesn't bother with logging.yml
it creates a logger the normal __name__
way
https://github.com/kedro-org/kedro-plugins/blob/main/kedro-telemetry/kedro_telemetry/plugin.pydatajoely
02/01/2022, 4:35 PMlogging.yml
by mapping one of the keys to the handler nameDaehyun Kim
02/01/2022, 4:36 PMDaehyun Kim
02/01/2022, 4:36 PMDaehyun Kim
02/01/2022, 4:36 PM