https://kedro.org/ logo
Join the conversationJoin Discord
Channels
advanced-need-help
announcements
beginners-need-help
introductions
job-posting
plugins-integrations
random
resources
welcome
Powered by Linen
advanced-need-help
  • d

    datajoely

    06/24/2021, 8:21 AM
    Yes unfortunately this isn’t the right tool for building something cyclic- it’s an interesting use case so sorry we can’t help know but it’s good to know people want to do this sort of thing
  • d

    datajoely

    06/24/2021, 8:21 AM
    In terms of drawing force directed graphs there are lots of JavaScript libraries out there like vis.js (IIRC)
  • s

    sigma

    06/24/2021, 8:22 AM
    Actually mine is an undirected graph. so isn't that technically acyclic only since I'm going source to targets in one direction only. Or you mean edge wise cyclic too?
  • s

    sigma

    06/24/2021, 8:22 AM
    Ah, thanks. I'll check this out πŸ‘
  • d

    datajoely

    06/24/2021, 8:22 AM
    Yeah it’s beyond direction, it can’t loop back
  • s

    sigma

    06/24/2021, 8:22 AM
    Right, got it. Thanks for the help! πŸ˜…
  • s

    sigma

    06/24/2021, 8:26 AM
    There's an image in the README on Kedro's github repo. There two nodes point to the same output. Can you explain how to achieve that?
  • d

    datajoely

    06/24/2021, 8:27 AM
    Let me look into the specific error. Are you okay with DMing me the JSON?
  • s

    sigma

    06/24/2021, 8:30 AM
    Pardon but I cannot provide that csv. corporate obligation πŸ˜“ . But I can provide you with a dummy CSV that fulfills the same purpose. Let me DM you!
  • d

    datajoely

    06/24/2021, 8:30 AM
    That will do!
  • a

    anhoang

    06/24/2021, 6:17 PM
    I sometimes use Kedro nodes and pipelines in my projects, but I use Kedro's Data Catalog for all of my projects! It's so useful. Trying to convince others in my team to also use the same data catalog is challenging, since the data catalog only works in Python. My team uses Bash, awk, and R also (bioinformatics). I'm thinking to just enable a mapping between dataset name and file path. To do this I would use a shell script that parses the yaml file and save this mapping into environmental variables. This way they can access the dataset-file path mapping in any language. This script would be executed in the team's
    .bashrc
    for team-wide dataset and be placed in the Kedro project root directory for project-specific-datasets. Any suggestions of other/easier ways to utilize the same data catalog across different languages?
  • a

    Arnaldo

    06/24/2021, 8:52 PM
    nice application, @User I wondering how you're using Kedro's Data Catalog in other applications. Could you provide a simple example, please?
  • d

    datajoely

    06/25/2021, 6:46 PM
    This is really interesting - @anhoang would you mind creating a GitHub discussion thread as I’d like to better understand what you need and how we can help improve the catalog/config journey?
  • a

    antony.milne

    06/28/2021, 10:09 PM
    Agreed - this is interesting and I'd like to hear more too πŸ™‚ What you propose sounds very sensible. You could maybe also do it the other way round using TemplatedConfigLoader (https://kedro.readthedocs.io/en/stable/kedro.config.TemplatedConfigLoader.html), where you define the dataset name to filepath mapping in environment variables and then inject those into the kedro catalog. It depends where you want to define your original source of truth (kedro data catalog vs. environment variables) Also something that is maybe not so well known... kedro supports a variety of different file formats for config files including the catalog (yaml, json, ini, pickle, properties, xml). Although yaml is overwhelmingly popular, maybe one of the other formats is more familiar to your team/easier to parse/more natively supported in those other languages
  • s

    sigma

    06/29/2021, 10:15 AM
    Hello! I had a query regarding kedro. Can we somehow generate pipelines on demand while the server is running?
  • s

    sigma

    06/29/2021, 10:19 AM
    Like everytime I make some changes in my pipelines code, I have to restart the server everytime. Can this be avoided somehow?
  • d

    datajoely

    06/29/2021, 10:29 AM
    Hey @User - Kedro should dynamically work with pipeline changes. How are you running your pipelines - via the CLI?
  • s

    sigma

    06/29/2021, 10:31 AM
    yes. Since I'm visualizing stuff I do ->
    kedro viz
  • d

    datajoely

    06/29/2021, 10:31 AM
    Ah you're talking about restarting the Viz server - currently yes, we do have a hot-reload prototype - but it's not out yet
  • d

    datajoely

    06/29/2021, 10:32 AM
    So the answer is... we're working on it πŸ™‚
  • s

    sigma

    06/29/2021, 10:32 AM
    oh okay πŸ˜… πŸ‘
  • d

    datajoely

    06/29/2021, 10:33 AM
    In the latest update we rewrote the Viz backend to use FastAPI and that opens up a lot possibilities for the tool to become much more interactive
  • s

    sigma

    06/29/2021, 10:34 AM
    Yes, I'm exploring that only right now.
  • s

    sigma

    07/01/2021, 2:01 PM
    Hi @User . Can you explain a bit how can I use Kedro viz as a react component in another project?
  • d

    datajoely

    07/01/2021, 2:02 PM
    Hi @User this isn't my forte - but I can point you in the direction of a project that does this
  • s

    sigma

    07/01/2021, 2:02 PM
    Yes, that would be helpful. Thanks πŸ™‚
  • d

    datajoely

    07/01/2021, 2:02 PM
    https://github.com/studioswong/kedro-viz-glitch/blob/layer-demo/src/App.js
  • d

    datajoely

    07/01/2021, 2:03 PM
    This is a hacky thing I'm working on for another purpose and isn't official
  • d

    datajoely

    07/01/2021, 2:03 PM
    but you can see in
    App.js
    on this branch how we import a JSON file and also disable the sidebar on load
  • s

    sigma

    07/01/2021, 2:04 PM
    This is what I was looking for! As always, thanks for the help πŸ™‚
Powered by Linen
Title
s

sigma

07/01/2021, 2:04 PM
This is what I was looking for! As always, thanks for the help πŸ™‚
View count: 2