https://kedro.org/ logo
Join the conversationJoin Discord
Channels
advanced-need-help
announcements
beginners-need-help
introductions
job-posting
plugins-integrations
random
resources
welcome
Powered by Linen
beginners-need-help
  • d

    datajoely

    07/21/2021, 12:59 PM
    This looks nice https://github.com/douglaslessmann/pokemon_ml_project
  • w

    waylonwalker

    07/21/2021, 1:00 PM
    we are actually going through this list live now
  • w

    waylonwalker

    07/21/2021, 1:00 PM
    we need one where nodes are added over the course of the git history
  • w

    waylonwalker

    07/21/2021, 1:01 PM
    not in a single commit as many we have seen.
  • d

    datajoely

    07/21/2021, 1:01 PM
    Ah shame - yes I'm afraid I don't have anything publicly sharable on my side
  • w

    waylonwalker

    07/21/2021, 1:02 PM
    most I have found, the history is init kedro, then updated readme a bunch of times.
  • w

    waylonwalker

    07/21/2021, 1:03 PM
    It's a hard because kedro's target audience is not generally public data.
  • w

    waylonwalker

    07/21/2021, 1:03 PM
    we will get there though 😄
  • w

    waylonwalker

    07/21/2021, 1:07 PM
    After looking for a little bit, we decided to make one.
  • w

    waylonwalker

    07/21/2021, 2:10 PM
    we made our own repo with a git history with fake dates, maybe next we can show off how to compare history over time. https://github.com/WaylonWalker/kedro-history
  • w

    waylonwalker

    07/21/2021, 2:22 PM
    adding nodes
  • w

    waylonwalker

    07/21/2021, 2:22 PM
    removing nodes
  • d

    datajoely

    07/21/2021, 2:22 PM
    ^ This is so cool
  • w

    waylonwalker

    07/21/2021, 2:22 PM
    we made enough history to test with a real repo.
  • w

    waylonwalker

    07/21/2021, 2:23 PM
    I did find one bug in that
    kedro diff <hash>
    with only a single hash passed in was showing backwards. otherwise its looking pretty good.
  • d

    datajoely

    07/21/2021, 2:24 PM
    Does it do anything regarding the pipeline registry?
  • w

    waylonwalker

    07/21/2021, 2:26 PM
    It is currently using context to get the pipelines dict
  • w

    waylonwalker

    07/21/2021, 2:27 PM
    so kedro-diff does not care if its hooks.py or pipeline_registry, it just cares that kedro is picking it up and putting it into the pipelines dict.
  • d

    datajoely

    07/21/2021, 2:27 PM
    Nice!
  • d

    datajoely

    07/21/2021, 2:28 PM
    Only thing to keep in mind is that the Context is going away at some point - so let me know if you need some help thinking how to migrate
  • w

    waylonwalker

    07/21/2021, 2:28 PM
    I might need help, I've see the warning. Is there anything release that I can use in place of context now?
  • w

    waylonwalker

    07/21/2021, 2:29 PM
    If you install kedro-diff you can see examples by running
    python -m kedro_diff
  • d

    datajoely

    07/21/2021, 2:29 PM
    thats very cool
  • w

    waylonwalker

    07/21/2021, 2:30 PM
    it even covers modified nodes
  • w

    waylonwalker

    07/21/2021, 2:31 PM
    inspired by refactoring PR's that change almost every line of code, and are not intended to change the pipeline.
  • w

    waylonwalker

    07/21/2021, 2:31 PM
    its really hard to see that the pipeline is unchanged in a git diff
  • d

    datajoely

    07/21/2021, 2:32 PM
    it's very smart - I wonder if this could be part of a CI step to give some observability
  • w

    waylonwalker

    07/21/2021, 2:32 PM
    or in normal circumstanses to see a short list of what has changed in the pipeline in the git diff.
  • w

    waylonwalker

    07/21/2021, 2:32 PM
    I really want this automatically put into our PR's, not quite sure how to do it though
  • w

    waylonwalker

    07/21/2021, 2:33 PM
    I bet if you had "owners" of different pipelines, then you could figure out who needs to approve easier too.
Powered by Linen
Title
w

waylonwalker

07/21/2021, 2:33 PM
I bet if you had "owners" of different pipelines, then you could figure out who needs to approve easier too.
View count: 1