beginners-need-help
- d
datajoely
07/21/2021, 12:59 PMThis looks nice https://github.com/douglaslessmann/pokemon_ml_project - w
waylonwalker
07/21/2021, 1:00 PMwe are actually going through this list live now - w
waylonwalker
07/21/2021, 1:00 PMwe need one where nodes are added over the course of the git history - w
waylonwalker
07/21/2021, 1:01 PMnot in a single commit as many we have seen. - d
datajoely
07/21/2021, 1:01 PMAh shame - yes I'm afraid I don't have anything publicly sharable on my side - w
waylonwalker
07/21/2021, 1:02 PMmost I have found, the history is init kedro, then updated readme a bunch of times. - w
waylonwalker
07/21/2021, 1:03 PMIt's a hard because kedro's target audience is not generally public data. - w
waylonwalker
07/21/2021, 1:03 PMwe will get there though 😄 - w
waylonwalker
07/21/2021, 1:07 PMAfter looking for a little bit, we decided to make one. - w
waylonwalker
07/21/2021, 2:10 PMwe made our own repo with a git history with fake dates, maybe next we can show off how to compare history over time. https://github.com/WaylonWalker/kedro-history - w
waylonwalker
07/21/2021, 2:22 PMadding nodes - w
waylonwalker
07/21/2021, 2:22 PMremoving nodes - d
datajoely
07/21/2021, 2:22 PM^ This is so cool - w
waylonwalker
07/21/2021, 2:22 PMwe made enough history to test with a real repo. - w
waylonwalker
07/21/2021, 2:23 PMI did find one bug in that
with only a single hash passed in was showing backwards. otherwise its looking pretty good.kedro diff <hash> - d
datajoely
07/21/2021, 2:24 PMDoes it do anything regarding the pipeline registry? - w
waylonwalker
07/21/2021, 2:26 PMIt is currently using context to get the pipelines dict - w
waylonwalker
07/21/2021, 2:27 PMso kedro-diff does not care if its hooks.py or pipeline_registry, it just cares that kedro is picking it up and putting it into the pipelines dict. - d
datajoely
07/21/2021, 2:27 PMNice! - d
datajoely
07/21/2021, 2:28 PMOnly thing to keep in mind is that the Context is going away at some point - so let me know if you need some help thinking how to migrate - w
waylonwalker
07/21/2021, 2:28 PMI might need help, I've see the warning. Is there anything release that I can use in place of context now? - w
waylonwalker
07/21/2021, 2:29 PMIf you install kedro-diff you can see examples by runningpython -m kedro_diff - d
datajoely
07/21/2021, 2:29 PMthats very cool - w
waylonwalker
07/21/2021, 2:30 PMit even covers modified nodes - w
waylonwalker
07/21/2021, 2:31 PMinspired by refactoring PR's that change almost every line of code, and are not intended to change the pipeline. - w
waylonwalker
07/21/2021, 2:31 PMits really hard to see that the pipeline is unchanged in a git diff - d
datajoely
07/21/2021, 2:32 PMit's very smart - I wonder if this could be part of a CI step to give some observability - w
waylonwalker
07/21/2021, 2:32 PMor in normal circumstanses to see a short list of what has changed in the pipeline in the git diff. - w
waylonwalker
07/21/2021, 2:32 PMI really want this automatically put into our PR's, not quite sure how to do it though - w
waylonwalker
07/21/2021, 2:33 PMI bet if you had "owners" of different pipelines, then you could figure out who needs to approve easier too.
Title
w
waylonwalker
07/21/2021, 2:33 PMI bet if you had "owners" of different pipelines, then you could figure out who needs to approve easier too.
View count: 1