Happy to discuss your use case here or via DM to work out if Kedro can help?

Hi @datajoely ok! Will DM you. A question in meantime: this is the right place to discuss about Kedro, right? The Discourse forum is not being used anymore, correct? I also had a look at the GitHub Discussions, but it looks like this server is more active

Yes last Thursday we announced this + Discussions is replacing discourse

You can rewatch our showcase event here:

We talk about the community bit right at the end 36 mins in

Welcome to the community @User , what is it that you feel makes your data not aligned with kedro?

Hi @waylonwalker @datajoely ok, let's start the discussion here, then we can move to DMs if needs be

So, basically I'm working on an instance Segmentation task, using PyTorch frameworks suitable for the task

I need to be able to read a dataset in COCO format

See here https://cocodataset.org/#format-data

Or here https://mmdetection.readthedocs.io/en/latest/2_new_data_model.html#coco-annotation-format

But there are no such data types

The

ImageDataset

supports images but not annotations

So I guess you have two options - you can use the JSON dataset to get it into your nodes https://kedro.readthedocs.io/en/stable/kedro.extras.datasets.pandas.JSONDataSet.html Or you could define a COCODataSet which inherits from the JSONDataSet and does some processing to make it easier to work with

Yes, I indeed started using

JSONDataset

, but I'm not sure where to start with defining a

COCODataset

. Extending the

JSONDataset

class to deal with the COCO syntax doesn't seem immediate. There's a Python module

pycocotools

which can be used to parse COCO annotation files, but having a COCODataset type in Kedro would be useful to, for example, examine the dataset using

kedro ipython

and then `config.load()‘

Makes sense so the steps for developing your own dataset are defined here: https://kedro.readthedocs.io/en/stable/07_extend_kedro/03_custom_datasets.html To start from scratch you need to inherit from

AbstractDataSet

and implement the

__init__()

,

load()

and

save()

methods. Looking at this example from pycocotools I think it is super easy for you to just need to import

from pycocotools.coco import COCO

and do

COCO(your_path)

within the

load()

method and you're good to go. https://github.com/cocodataset/cocoapi/blob/master/PythonAPI/pycocoDemo.ipynb The only thing worth noting is that this will only work for local files - if you want the dataset to accept either local or cloud files (s3 etc) you may want to extend / steal the basic implementation from any of the existing Kedro datasets

Thanks! Let's make it work for local files first - I'll worry about cloud storage later. I'll keep you posted. Thanks again!

Hello world, just sign up for Kedro Discord channel. Currently working as a data scientist and we have using kedro for a few projects and integrate with some of our internal library for ML experiments.

Welcome! Shout if you need anything.

Welcome to the discord @User !

Thanks! Nice to meet you here @User

@User would it be possible to write a

after_node_run

hook that checks all

CachedDataSets

to see if all dependents have been satisfied, and runs

release()

only after all dependents of that dataset have been ran automatically?

I hit enter mid message @User

ah gotcha - let me think about that

I'm not sure it can be the info passed to the node hook doesn't provide any context on which other nodes have been run

The hook provides things like node, tag , namespace etc. which could be used - but it is quite manual

This is probably be best achieved in the CachedDataSet implemenetation

does the CachedDataset automatically take care of this?

If you know the node that just completed you could check pipeline.grouped_nodes, it would not be the most efficient, but you could

release()

after you have moved completely passed all groups using a certain dataset.