https://kedro.org/ logo
Join the conversationJoin Discord
Channels
advanced-need-help
announcements
beginners-need-help
introductions
job-posting
plugins-integrations
random
resources
welcome
Powered by Linen
resources
  • w

    waylonwalker

    07/23/2021, 12:11 PM
    Going live to clean up kedro diff outputs, getting really close to a working state! https://twitch.com/waylonwalker
  • w

    waylonwalker

    07/23/2021, 2:19 PM
    https://pypi.org/project/kedro-diff/ released the initial version today
  • u

    user

    08/04/2021, 1:31 PM
    kedro catalog create
    I use kedro catalog create to boost my productivity by automatically
    
    generating yaml catalog entries for me. It will cr
    https://waylonwalker.com/kedro-catalog-create-cli
  • w

    waylonwalker

    08/04/2021, 2:20 PM
    Thanks @User for setting up the rss feed!
  • u

    user

    08/16/2021, 1:17 PM
    What is Kedro
    Kedro is an unopinionated Data Engineering framework that comes with a somewhat opinionated template. It gives the user
    https://waylonwalker.com/what-is-kedro-1
  • u

    user

    08/16/2021, 1:32 PM
    How I Kedro
    https://www.youtube.com/watch?v=bw5_FWDVRpU I recently switched over to using Ubuntu, it works well pretty much out of t
    https://waylonwalker.com/how-i-kedro
  • w

    waylonwalker

    08/16/2021, 2:00 PM

    https://youtu.be/bw5_FWDVRpU▾

    This is the start to the kedro series on YouTube
  • u

    user

    08/17/2021, 1:35 PM
    Kedro New
    https://youtu.be/uqiv5LAiJe0 Kedro new is simply a wrapper around the cookiecutter templating library. The
    
    kedro team m
    https://waylonwalker.com/kedro-new
  • w

    waylonwalker

    08/17/2021, 2:29 PM
    Create a New Data Pipeline From The Command Line

    https://www.youtube.com/watch?v=uqiv5LAiJe0▾

  • a

    Arnaldo

    08/17/2021, 4:37 PM
    private video, @User
  • w

    waylonwalker

    08/17/2021, 7:47 PM
    Darn, still learning youtube from the creator side, apparently scheduled posts are not unlisted, they are private until the schedule triggers. It will be live tomorrow. I will keep in sync with the youtube schedule from now on out. Expect many more to come.
  • w

    waylonwalker

    08/18/2021, 1:22 PM
    @User its live now.
  • a

    Arnaldo

    08/18/2021, 1:22 PM
    Added to my watch list
  • a

    anhoang

    08/18/2021, 7:18 PM
    Can we pls compile all these gems (and the ones from other users) to the documentation/Github thread?
  • d

    datajoely

    08/18/2021, 7:29 PM
    Should I start a github discussion?
  • d

    datajoely

    08/18/2021, 7:44 PM
    Alright I will do - promise you'll all contribute 🙂 ?
  • d

    datajoely

    08/19/2021, 9:22 AM
    Discussion is now live https://github.com/quantumblacklabs/kedro/discussions/859
  • u

    user

    08/19/2021, 7:08 PM
    kedro-environment
    Avoid serious version conflict issues, and use a virtual environment anytime
    
    you are running python, here are three ways
    https://waylonwalker.com/kedro-environment
  • w

    waylonwalker

    08/19/2021, 8:14 PM

    https://youtu.be/ZSxc5VVCBhM▾

  • u

    user

    08/20/2021, 2:48 PM
    Kedro Git Init
    Immediately after kedro new, before you start running kedro install or your first line of code the first
    
    thing you shoul
    https://waylonwalker.com/kedro-git-init
  • w

    waylonwalker

    08/20/2021, 4:19 PM
    Link to video as well

    https://youtu.be/IGba3ytf_6U▾

  • u

    user

    08/21/2021, 2:58 PM
    Kedro Install
    Kedro comes with an install command to install and manage all of your
    
    projects dependencies. https://youtu.be/IWimEs-hHQ
    https://waylonwalker.com/kedro-install
  • u

    user

    09/12/2021, 9:15 PM
    Writing your first kedro Nodes
    https://youtu.be/-gEwU-MrPuA Before we jump in with anything crazy, let's make some nodes with some vanilla
    
    data structu
    https://waylonwalker.com/kedro-your-first-nodes
  • j

    jakubczakon

    09/16/2021, 3:49 PM
    Hi, Jakub from neptune.ai here. First of all, I want to say that I have massive respect for this project and all the people who contribute. It is essential in day-to-day for so many teams. Great job, and thank you! A few months ago, some of our users asked if we could connect the experiment and metadata management capabilities of neptune.ai with all the great stuff that Kedro does. The main things people wanted to improve were: - Browsing, sorting, filtering pipeline executions on various metadata - Comparing pipeline executions and nodes inside a pipeline on metrics, outputs, visualizations - Display additional pipeline outputs like prediction images, ROC curves, or learning curves for deep learning models And since neptune.ai is really good at those things, we figured it could be a nice integration. It took some time, but we wanted to make it easy to use yet powerful. Here is a video walkthrough: https://www.loom.com/share/0a690ce0aa39405da955e853d2fd73d6 More info and links are in the thread (couldn't fit it all in here). I would absolutely love to hear what you think about it, how we could make it work for your workflow, where we need to improve it. If you have any questions, I am here!
  • j

    jakubczakon

    09/16/2021, 3:54 PM
    Anyhow, with a few additions to your regular Kedro project, you can do all of the things mentioned above in a friendly UI. Here are some UI examples you can play with: - Custom dashboard basic: https://app.neptune.ai/o/common/org/kedro-integration/e/KED-632/dashboard/Basic-pipeline-metadata-42874940-da74-4cdc-94a4-315a7cdfbfa8 - Compare Kedro pipeline runs: https://app.neptune.ai/o/common/org/kedro-integration/experiments?split=tbl&dash=parallel-coordinates-plot&viewId=6e96f108-97e0-4a66-aef5-eb1ed6952eff - Compare Kedro nodes: https://app.neptune.ai/o/common/org/kedro-integration/experiments?split=tbl&dash=parallel-coordinates-plot&viewId=40a782af-0b7f-4c38-aa61-a24c3bf34f45 - Group Kedro pipeline runs by dataset version: https://app.neptune.ai/o/common/org/kedro-integration/experiments?split=tbl&dash=parallel-coordinates-plot&viewId=14be06ac-8664-4a8b-a632-85b7b61a0981
  • j

    jakubczakon

    09/16/2021, 3:54 PM
    To log stuff, you need to: - Install kedro-neptune plugin - Initialize the plugin and setup credentials - Add a neptune_run handler as an input to a node - Log whatever metadata you want to it This would be a change to the node:
    def report_accuracy(predictions: np.ndarray, test_y: pd.DataFrame,
                                     neptune_run: neptune.run.Handler) -> None:
        target = np.argmax(test_y.to_numpy(), axis=1)
        accuracy = np.sum(predictions == target) / target.shape[0]
        neptune_run['nodes/report/accuracy'] = accuracy * 100
    If you are interested in trying it out, here are some step-by-step examples: - Quickstart: https://docs.neptune.ai/integrations-and-supported-tools/automation-pipelines/kedro#quickstart - Compare Kedro pipelines: https://docs.neptune.ai/integrations-and-supported-tools/automation-pipelines/kedro/compare-kedro-pipelines - Compare results between Kedro nodes: https://docs.neptune.ai/integrations-and-supported-tools/automation-pipelines/kedro/compare-results-between-kedro-nodes - Display Kedro node metadata and outputs: https://docs.neptune.ai/integrations-and-supported-tools/automation-pipelines/kedro/display-kedro-node-metadata-and-outputs
  • j

    jakubczakon

    09/16/2021, 3:55 PM
    To log stuff, you need to: - Install kedro-neptune plugin - Initialize the plugin and setup credentials - Add a neptune_run handler as an input to a node - Log whatever metadata you want to it This would be a change to the node:
    def report_accuracy(predictions: np.ndarray, test_y: pd.DataFrame,
                                     neptune_run: neptune.run.Handler) -> None:
        target = np.argmax(test_y.to_numpy(), axis=1)
        accuracy = np.sum(predictions == target) / target.shape[0]
        neptune_run['nodes/report/accuracy'] = accuracy * 100
    If you are interested in trying it out, here are some step-by-step examples: - Quickstart: https://docs.neptune.ai/integrations-and-supported-tools/automation-pipelines/kedro#quickstart - Compare Kedro pipelines: https://docs.neptune.ai/integrations-and-supported-tools/automation-pipelines/kedro/compare-kedro-pipelines - Compare results between Kedro nodes: https://docs.neptune.ai/integrations-and-supported-tools/automation-pipelines/kedro/compare-results-between-kedro-nodes - Display Kedro node metadata and outputs: https://docs.neptune.ai/integrations-and-supported-tools/automation-pipelines/kedro/display-kedro-node-metadata-and-outputs Again, thank you so much, and I really hope this will be useful to some of you here!
  • d

    datajoely

    09/16/2021, 4:00 PM
    🔥
  • a

    Arnaldo

    09/16/2021, 5:10 PM
    Looking forward to see this integration, @User
  • j

    jakubczakon

    09/16/2021, 5:34 PM
    Awesome @User! Pls share your experience, good or bad -> I'd really appreciate your feedback.
    a
    d
    • 3
    • 9
Powered by Linen
Title
j

jakubczakon

09/16/2021, 5:34 PM
Awesome @User! Pls share your experience, good or bad -> I'd really appreciate your feedback.
a

Arnaldo

09/21/2021, 4:41 PM
hi, @User I saw the video you shared and took a look at the docs
you did a really god job
the integration is very easy to setup (thanks to kedro plugins as well)
and seems very easy-to-use
I shared your video with a friend of mine that uses Neptune
if he shares a feedback with me, I'll let you know
d

datajoely

09/22/2021, 3:20 PM
Also feel free to use this new emoji
j

jakubczakon

09/23/2021, 5:23 AM
yay thanks @User!
and wow, thanks for the emoji, @User , this is so awesome!
View count: 1