I want to save a node output to disk, but I also want to use it in a subsequent node. Should I configure the node with multiple outputs (one memory and one to disk), or should I use a memory output in the node configuration and explicitly save to the catalog within the node logic? Is there a best practice for this?

Given it will still wait for the save operation to complete before progressing I would just persist with a catalog entry and then use that output as the input to the next node

I can suggest some fancy alternatives but they feel like very over engineered

Does that mean that the next node will also load from disk?

Yes

Ok, thanks. Might it be worthwhile for "sufficiently" large datasets to save to disk but pass between nodes in memory?

You can emulate this with

after_node_run

hook: https://kedro.readthedocs.io/en/stable/kedro.framework.hooks.specs.NodeSpecs.html#kedro.framework.hooks.specs.NodeSpecs.after_node_run -- use MemoryDataSet as data flow in Kedro but save to disk in after_node_run hook as backup. I think you can also do this async (eg in another process) so you don't block the pipeline execution.

Great, thank you both!

Is it possible to get a report of the runtime by pipeline? I guess I could go through the logs and identify the first and last nodes of a pipeline, but tedious when there are a lot of nodes.

Yes! Via lifecycle hooks - this memory consumption example is similar in principle https://kedro.readthedocs.io/en/latest/07_extend_kedro/02_hooks.html#add-memory-consumption-tracking

Hey all, pretty new to kedro. I have a question on what category of ML it falls under, can you use it to structure object detection projects such as detectron2?

Hi @User - in short you absolutely can use Kedro to structure your pipelines. It's the first I've come across this particular library and it's very doable. You could probably use our

PartitionedDataSet

and

IncrementalDataSet

to help manage the sets of images. In truth there looks like a bit of overlap between the parts Kedro tries to own in terms of (a) Config management (b) IO - so you can either choose to Kedr-ify these with our abstractions or have bits of your pipeline that use the native

detectron2

stuff to do some processing steps and then switch to Kedro style pipelining downstream.

Great thanks I will look into it

Cool thanks! I'll give this ago. Haven't really used hooks at all. This will be a good opportunity.

Very cool. Used the

before_pipeline_run

and

after_pipeline_run

.

Thanks @User . Right now this is working well for us. But we might soon have to fine tune and improve the solution. We'll get in touch with you for suggestions when we start working on adding features

Nice! We'd also really appreciate any PRs straight into the docs since it's you our users who are closer to all these deployment targets 🙂

Hi Team, Is there a way to add a logging handler programmatically instead of modifying

conf/base/logging.yml

?

If possible, I'd like to saving logs in different path(e.x. logs/XXX/info.log) and so that my kedro plugin upload it to s3 for kedro run.

I think if you clear that file you can configure things the normal way. All we do is pass it to

logging.dictConfig

anyway

so if you wanted to you can put anything in that YAML file that maps to this schema https://docs.python.org/3/library/logging.config.html#configuration-dictionary-schema

i'm assuming my kedro plugin will be executed with default logging.yml

Ah for plugins I don't think it works that way

is it available to load

logging.yml

and add my custom handler into it and pass it to logging.dictConfig in somewhere in any hooks?

if you look at the

kedro-telemetry

plugin it doesn't bother with

logging.yml

it creates a logger the normal

__name__

way https://github.com/kedro-org/kedro-plugins/blob/main/kedro-telemetry/kedro_telemetry/plugin.py

now you can manage that in

logging.yml

by mapping one of the keys to the handler name

I wanted to make extra logging for my plugin instead of replacing logging

oh

i will take a look at it