https://kedro.org/ logo
Join the conversationJoin Discord
Channels
advanced-need-help
announcements
beginners-need-help
introductions
job-posting
plugins-integrations
random
resources
welcome
Powered by Linen
advanced-need-help
  • d

    datajoely

    07/14/2021, 1:38 PM
    Lol sorry about that
  • u

    user

    07/14/2021, 1:38 PM
    No worries. Completly different name.
  • m

    Mad Hatter

    07/14/2021, 1:39 PM
    ok thanks for the details
  • m

    Mad Hatter

    07/14/2021, 1:40 PM
    let me see what i can do
  • d

    datajoely

    07/14/2021, 1:41 PM
    Since a few people are active - would anyone be interested in participating in some User Research? https://discord.com/channels/778216384475693066/864784648575254558/864818777899270145
  • m

    Mad Hatter

    07/15/2021, 12:38 PM
    @User thanks for help i rearranged the layers in my nodes and was able to solve the cyclic error
  • d

    datajoely

    07/15/2021, 12:38 PM
    Yes the layers constraint is a bit fiddly
  • m

    Mad Hatter

    07/15/2021, 12:38 PM
    i just want to ask how to render 20K nodes i want to generate a database lineage
  • m

    Mad Hatter

    07/15/2021, 12:38 PM
    of an enterprise data
  • d

    datajoely

    07/15/2021, 12:39 PM
    You're pushing the limits of what you can render there
  • d

    datajoely

    07/15/2021, 12:39 PM
    There is a URL flag you may want to disable
    sizewarning=false
  • d

    datajoely

    07/15/2021, 12:40 PM
    Also making sure certain nodes are tagged would allow you to render certain subgrapgs
  • m

    Mad Hatter

    07/15/2021, 12:40 PM
    every node is tagged in my case
  • m

    Mad Hatter

    07/15/2021, 12:40 PM
    ok so what is the best number of nodes in a pipeline i would create different pipelines
  • d

    datajoely

    07/15/2021, 12:42 PM
    Today - I don't think that affects performance (it will in an upcoming release) but I think rule of thunk 1K nodes on screen at any point triggers the size warning
  • m

    Mad Hatter

    07/15/2021, 12:42 PM
    it is getting hanged in firefox
  • m

    Mad Hatter

    07/15/2021, 12:42 PM
    so that's why i was asking
  • d

    datajoely

    07/15/2021, 12:43 PM
    Yes unfortunately you are pushing Viz outside of it's comfort zone
  • m

    Mad Hatter

    07/15/2021, 12:44 PM
    i have built a data lineage for 165 nodes and its working great but the database is too large with many more objects so if prototype is approved i would have to know these things
  • m

    Mad Hatter

    07/15/2021, 12:44 PM
    current look with 165 nodes
  • d

    datajoely

    07/15/2021, 12:44 PM
    maybe the best thing I can recommend is to break it up into several registered pipelines. But it's not built for that many nodes on one screen at a time
  • d

    datajoely

    07/15/2021, 12:45 PM
    I think you'll struggle to find many tools capable of rendering that many items on screen live
  • d

    datajoely

    07/15/2021, 12:46 PM
    It becomes a O(n3) problem at a certain point
  • m

    Mad Hatter

    07/15/2021, 12:47 PM
    and one more thing i haven't created any nodes or code through python i have created a json file similar to demo.mock.json so will it be better if i create nodes and everything in kedro instead of generating a kedro viz demo json file?
  • d

    datajoely

    07/15/2021, 12:48 PM
    Viz doesn't care - you just lose some of the metadata
  • m

    Mad Hatter

    07/15/2021, 12:49 PM
    ok and is there any way to let kedro viz accept more value combinations in data part of url localhost:4141/?data=demo currently only these are supported Your input should be one of the following values: 'animals', 'demo', 'json', 'selectors', or 'random'
  • d

    datajoely

    07/15/2021, 12:50 PM
    Yes - but by creating your own react project
  • d

    datajoely

    07/15/2021, 12:50 PM
    and just importing the component
  • d

    datajoely

    07/15/2021, 12:51 PM
    This is an example of something I did for a Medium article https://github.com/studioswong/kedro-viz-glitch/blob/layer-demo/src/App.js
  • m

    Mad Hatter

    07/15/2021, 12:52 PM
    got it you guys should think about integrating a data lineage portion many organisations are buying left and right data lineage tools and don't even know about this type of open source tool available
Powered by Linen
Title
m

Mad Hatter

07/15/2021, 12:52 PM
got it you guys should think about integrating a data lineage portion many organisations are buying left and right data lineage tools and don't even know about this type of open source tool available
View count: 1