#advanced-need-help
- u
user
07/14/2021, 1:38 PMNo worries. Completly different name. - m
Mad Hatter
07/14/2021, 1:39 PMok thanks for the details - m
Mad Hatter
07/14/2021, 1:40 PMlet me see what i can do - d
datajoely
07/14/2021, 1:41 PMSince a few people are active - would anyone be interested in participating in some User Research? https://discord.com/channels/778216384475693066/864784648575254558/864818777899270145 - m
Mad Hatter
07/15/2021, 12:38 PM@User thanks for help i rearranged the layers in my nodes and was able to solve the cyclic error - d
datajoely
07/15/2021, 12:38 PMYes the layers constraint is a bit fiddly - m
Mad Hatter
07/15/2021, 12:38 PMi just want to ask how to render 20K nodes i want to generate a database lineage - m
Mad Hatter
07/15/2021, 12:38 PMof an enterprise data - d
datajoely
07/15/2021, 12:39 PMYou're pushing the limits of what you can render there - d
datajoely
07/15/2021, 12:39 PMThere is a URL flag you may want to disablesizewarning=false - d
datajoely
07/15/2021, 12:40 PMAlso making sure certain nodes are tagged would allow you to render certain subgrapgs - m
Mad Hatter
07/15/2021, 12:40 PMevery node is tagged in my case - m
Mad Hatter
07/15/2021, 12:40 PMok so what is the best number of nodes in a pipeline i would create different pipelines - d
datajoely
07/15/2021, 12:42 PMToday - I don't think that affects performance (it will in an upcoming release) but I think rule of thunk 1K nodes on screen at any point triggers the size warning - m
Mad Hatter
07/15/2021, 12:42 PMit is getting hanged in firefox - m
Mad Hatter
07/15/2021, 12:42 PMso that's why i was asking - d
datajoely
07/15/2021, 12:43 PMYes unfortunately you are pushing Viz outside of it's comfort zone - m
Mad Hatter
07/15/2021, 12:44 PMi have built a data lineage for 165 nodes and its working great but the database is too large with many more objects so if prototype is approved i would have to know these things - m
Mad Hatter
07/15/2021, 12:44 PMcurrent look with 165 nodes - d
datajoely
07/15/2021, 12:44 PMmaybe the best thing I can recommend is to break it up into several registered pipelines. But it's not built for that many nodes on one screen at a time - d
datajoely
07/15/2021, 12:45 PMI think you'll struggle to find many tools capable of rendering that many items on screen live - d
datajoely
07/15/2021, 12:46 PMIt becomes a O(n3) problem at a certain point - m
Mad Hatter
07/15/2021, 12:47 PMand one more thing i haven't created any nodes or code through python i have created a json file similar to demo.mock.json so will it be better if i create nodes and everything in kedro instead of generating a kedro viz demo json file? - d
datajoely
07/15/2021, 12:48 PMViz doesn't care - you just lose some of the metadata - m
Mad Hatter
07/15/2021, 12:49 PMok and is there any way to let kedro viz accept more value combinations in data part of url localhost:4141/?data=demo currently only these are supported Your input should be one of the following values: 'animals', 'demo', 'json', 'selectors', or 'random' - d
datajoely
07/15/2021, 12:50 PMYes - but by creating your own react project - d
datajoely
07/15/2021, 12:50 PMand just importing the component - d
datajoely
07/15/2021, 12:51 PMThis is an example of something I did for a Medium article https://github.com/studioswong/kedro-viz-glitch/blob/layer-demo/src/App.js - m
Mad Hatter
07/15/2021, 12:52 PMgot it you guys should think about integrating a data lineage portion many organisations are buying left and right data lineage tools and don't even know about this type of open source tool available - d
datajoely
07/15/2021, 12:52 PMWe're aware of these people https://tokern.io/data-lineage/ and have something internally the does it for Spark