https://kedro.org/ logo
Join the conversationJoin Discord
Channels
advanced-need-help
announcements
beginners-need-help
introductions
job-posting
plugins-integrations
random
resources
welcome
Powered by Linen
advanced-need-help
  • d

    datajoely

    07/01/2021, 2:04 PM
    This workflow is something we want to improve in general - so please shout if you run into walls
  • s

    sigma

    07/01/2021, 2:04 PM
    sure thing
  • w

    waylonwalker

    07/01/2021, 2:45 PM
    You can check out kedro-static-viz it uses the react component in a gatsby app.
  • s

    sigma

    07/01/2021, 3:19 PM
    Thanks!
  • n

    noklam

    07/02/2021, 1:49 AM
    @User @User Let's move the discussion here
  • n

    noklam

    07/02/2021, 1:51 AM
    My feeling is that Kedro should have an internal DAGs already, thus it is possible to automatically release dataset that is no longer needed. I actually thought this is the default behavior but it seems not true for the moment.
  • d

    datajoely

    07/02/2021, 8:41 AM
    @User which runner are you using? The
    SequentialRunner
    appears to call
    .release()
    whenever a dataset is not longer needed
  • d

    datajoely

    07/02/2021, 8:41 AM
    https://kedro.readthedocs.io/en/stable/_modules/kedro/runner/sequential_runner.html#SequentialRunner
  • s

    sigma

    07/02/2021, 1:24 PM
    How to make a very minimal Kedro application? Basically I only need the json which kedro produces after processing the pipelines. Since I've very much decoupled my visualization frontend by using viz as a component in another react project, I only require the json now
  • d

    datajoely

    07/02/2021, 1:25 PM
    This repo is the best mini example we have - https://github.com/studioswong/kedro-viz-glitch/blob/layer-demo/src/App.js
  • d

    datajoely

    07/02/2021, 1:26 PM
    on the
    main
    branch we have some predefined datasets which import as JSON objects, and on the
    layer-demo
    branch we have directly imported branches
  • d

    datajoely

    07/02/2021, 1:26 PM
    I've then built the application and deployed in on Glitch as it will be embeded in a forthcoming Medium article
  • s

    sigma

    07/02/2021, 1:26 PM
    yeah, but what I need is that json only. And here in this project, they are basically importing that json pre-made
  • d

    datajoely

    07/02/2021, 1:27 PM
    Oh do you need to generate the JSON from the kedro side?
  • s

    sigma

    07/02/2021, 1:27 PM
    yes, right
  • d

    datajoely

    07/02/2021, 1:27 PM
    kedro-viz --save-file "somefile.json"
  • s

    sigma

    07/02/2021, 1:28 PM
    is it possible without the cli? through code? like can you point me to the function which can give me the json directly.
  • d

    datajoely

    07/02/2021, 1:28 PM
    Through CLI
  • d

    datajoely

    07/02/2021, 1:28 PM
    message has been deleted
  • d

    datajoely

    07/02/2021, 1:28 PM
    https://kedro.readthedocs.io/en/stable/03_tutorial/06_visualise_pipeline.html#share-a-pipeline
  • n

    noklam

    07/02/2021, 2:07 PM
    sequential, how is this working differently for cache or non cache dataset? As it seems that the dataset would be loaded repeatly when I am not using cachedataset, but from this snippet it seems that it will not be released until there are no reference. So why it will be loaded multiple times?
  • n

    noklam

    07/03/2021, 9:58 AM
    I want to make sure I understand how's the dataset life-cycle works, so I wrote some notes for myself. I think the dataset is not released due to it always in
    pipeline.inputs()
    or
    pipeline.outputs()
    . This may be a bug or at least not a desired behavior. Open for more disucssion here. https://github.com/quantumblacklabs/kedro/issues/819
  • d

    datajoely

    07/03/2021, 10:57 AM
    Sounds great - the team will respond next week
  • s

    sigma

    07/05/2021, 12:30 PM
    Can I somehow manually generate that json myself, which Kedro generates after running the pipelines?
  • d

    datajoely

    07/05/2021, 12:31 PM
    Yes there is no requirement to use Kedro core to do that - in fact as we learnt from our other conversations, you can render visualisations which break the acyclic rule of Kedro core if you really want to.
  • s

    sigma

    07/05/2021, 12:32 PM
    yes, that is my purpose. Since I only need the visualizations, perhaps I don't need a kedro core backend project at all
  • s

    sigma

    07/05/2021, 12:32 PM
    I saw a sample json and tried to mimic the structure of it. But my kedro interface was empty.
  • d

    datajoely

    07/05/2021, 12:33 PM
    No agreed - just a warning you might run into issues in areas we've never tested, but in theory it should work
  • d

    datajoely

    07/05/2021, 12:33 PM
    If you keep refreshing the random viz you can see it generate flowcharts that break those rules https://quantumblacklabs.github.io/kedro-viz/?data=random
  • s

    sigma

    07/05/2021, 12:34 PM
    I ignored the use of hex IDs for different source and targets. Would that be an issue?
Powered by Linen
Title
s

sigma

07/05/2021, 12:34 PM
I ignored the use of hex IDs for different source and targets. Would that be an issue?
View count: 1