This workflow is something we want to improve in general - so please shout if you run into walls

sure thing

You can check out kedro-static-viz it uses the react component in a gatsby app.

Thanks!

@User @User Let's move the discussion here

My feeling is that Kedro should have an internal DAGs already, thus it is possible to automatically release dataset that is no longer needed. I actually thought this is the default behavior but it seems not true for the moment.

@User which runner are you using? The

SequentialRunner

appears to call

.release()

whenever a dataset is not longer needed

How to make a very minimal Kedro application? Basically I only need the json which kedro produces after processing the pipelines. Since I've very much decoupled my visualization frontend by using viz as a component in another react project, I only require the json now

This repo is the best mini example we have - https://github.com/studioswong/kedro-viz-glitch/blob/layer-demo/src/App.js

on the

main

branch we have some predefined datasets which import as JSON objects, and on the

layer-demo

branch we have directly imported branches

I've then built the application and deployed in on Glitch as it will be embeded in a forthcoming Medium article

yeah, but what I need is that json only. And here in this project, they are basically importing that json pre-made

Oh do you need to generate the JSON from the kedro side?

yes, right

is it possible without the cli? through code? like can you point me to the function which can give me the json directly.

Through CLI

sequential, how is this working differently for cache or non cache dataset? As it seems that the dataset would be loaded repeatly when I am not using cachedataset, but from this snippet it seems that it will not be released until there are no reference. So why it will be loaded multiple times?

I want to make sure I understand how's the dataset life-cycle works, so I wrote some notes for myself. I think the dataset is not released due to it always in

pipeline.inputs()

or

pipeline.outputs()

. This may be a bug or at least not a desired behavior. Open for more disucssion here. https://github.com/quantumblacklabs/kedro/issues/819

Sounds great - the team will respond next week

Can I somehow manually generate that json myself, which Kedro generates after running the pipelines?

Yes there is no requirement to use Kedro core to do that - in fact as we learnt from our other conversations, you can render visualisations which break the acyclic rule of Kedro core if you really want to.

yes, that is my purpose. Since I only need the visualizations, perhaps I don't need a kedro core backend project at all

I saw a sample json and tried to mimic the structure of it. But my kedro interface was empty.

No agreed - just a warning you might run into issues in areas we've never tested, but in theory it should work

If you keep refreshing the random viz you can see it generate flowcharts that break those rules https://quantumblacklabs.github.io/kedro-viz/?data=random

I ignored the use of hex IDs for different source and targets. Would that be an issue?

also I would some info on 'modular pipelines'