Going live to clean up kedro diff outputs, getting really close to a working state! https://twitch.com/waylonwalker

https://pypi.org/project/kedro-diff/ released the initial version today

kedro catalog create

I use kedro catalog create to boost my productivity by automatically

generating yaml catalog entries for me. It will cr

https://waylonwalker.com/kedro-catalog-create-cli

Thanks @User for setting up the rss feed!

What is Kedro

Kedro is an unopinionated Data Engineering framework that comes with a somewhat opinionated template. It gives the user

https://waylonwalker.com/what-is-kedro-1

How I Kedro

https://www.youtube.com/watch?v=bw5_FWDVRpU I recently switched over to using Ubuntu, it works well pretty much out of t

https://waylonwalker.com/how-i-kedro

This is the start to the kedro series on YouTube

Kedro New

https://youtu.be/uqiv5LAiJe0 Kedro new is simply a wrapper around the cookiecutter templating library. The

kedro team m

https://waylonwalker.com/kedro-new

Create a New Data Pipeline From The Command Line

private video, @User

Darn, still learning youtube from the creator side, apparently scheduled posts are not unlisted, they are private until the schedule triggers. It will be live tomorrow. I will keep in sync with the youtube schedule from now on out. Expect many more to come.

@User its live now.

Added to my watch list

Can we pls compile all these gems (and the ones from other users) to the documentation/Github thread?

Should I start a github discussion?

Alright I will do - promise you'll all contribute 🙂 ?

Discussion is now live https://github.com/quantumblacklabs/kedro/discussions/859

kedro-environment

Avoid serious version conflict issues, and use a virtual environment anytime

you are running python, here are three ways

https://waylonwalker.com/kedro-environment

Kedro Git Init

Immediately after kedro new, before you start running kedro install or your first line of code the first

thing you shoul

https://waylonwalker.com/kedro-git-init

Link to video as well

Kedro Install

Kedro comes with an install command to install and manage all of your

projects dependencies. https://youtu.be/IWimEs-hHQ

https://waylonwalker.com/kedro-install

Writing your first kedro Nodes

https://youtu.be/-gEwU-MrPuA Before we jump in with anything crazy, let's make some nodes with some vanilla

data structu

https://waylonwalker.com/kedro-your-first-nodes

Hi, Jakub from neptune.ai here. First of all, I want to say that I have massive respect for this project and all the people who contribute. It is essential in day-to-day for so many teams. Great job, and thank you! A few months ago, some of our users asked if we could connect the experiment and metadata management capabilities of neptune.ai with all the great stuff that Kedro does. The main things people wanted to improve were: - Browsing, sorting, filtering pipeline executions on various metadata - Comparing pipeline executions and nodes inside a pipeline on metrics, outputs, visualizations - Display additional pipeline outputs like prediction images, ROC curves, or learning curves for deep learning models And since neptune.ai is really good at those things, we figured it could be a nice integration. It took some time, but we wanted to make it easy to use yet powerful. Here is a video walkthrough: https://www.loom.com/share/0a690ce0aa39405da955e853d2fd73d6 More info and links are in the thread (couldn't fit it all in here). I would absolutely love to hear what you think about it, how we could make it work for your workflow, where we need to improve it. If you have any questions, I am here!

Anyhow, with a few additions to your regular Kedro project, you can do all of the things mentioned above in a friendly UI. Here are some UI examples you can play with: - Custom dashboard basic: https://app.neptune.ai/o/common/org/kedro-integration/e/KED-632/dashboard/Basic-pipeline-metadata-42874940-da74-4cdc-94a4-315a7cdfbfa8 - Compare Kedro pipeline runs: https://app.neptune.ai/o/common/org/kedro-integration/experiments?split=tbl&dash=parallel-coordinates-plot&viewId=6e96f108-97e0-4a66-aef5-eb1ed6952eff - Compare Kedro nodes: https://app.neptune.ai/o/common/org/kedro-integration/experiments?split=tbl&dash=parallel-coordinates-plot&viewId=40a782af-0b7f-4c38-aa61-a24c3bf34f45 - Group Kedro pipeline runs by dataset version: https://app.neptune.ai/o/common/org/kedro-integration/experiments?split=tbl&dash=parallel-coordinates-plot&viewId=14be06ac-8664-4a8b-a632-85b7b61a0981

To log stuff, you need to: - Install kedro-neptune plugin - Initialize the plugin and setup credentials - Add a neptune_run handler as an input to a node - Log whatever metadata you want to it This would be a change to the node:

def report_accuracy(predictions: np.ndarray, test_y: pd.DataFrame,
                                 neptune_run: neptune.run.Handler) -> None:
    target = np.argmax(test_y.to_numpy(), axis=1)
    accuracy = np.sum(predictions == target) / target.shape[0]
    neptune_run['nodes/report/accuracy'] = accuracy * 100

If you are interested in trying it out, here are some step-by-step examples: - Quickstart: https://docs.neptune.ai/integrations-and-supported-tools/automation-pipelines/kedro#quickstart - Compare Kedro pipelines: https://docs.neptune.ai/integrations-and-supported-tools/automation-pipelines/kedro/compare-kedro-pipelines - Compare results between Kedro nodes: https://docs.neptune.ai/integrations-and-supported-tools/automation-pipelines/kedro/compare-results-between-kedro-nodes - Display Kedro node metadata and outputs: https://docs.neptune.ai/integrations-and-supported-tools/automation-pipelines/kedro/display-kedro-node-metadata-and-outputs

To log stuff, you need to: - Install kedro-neptune plugin - Initialize the plugin and setup credentials - Add a neptune_run handler as an input to a node - Log whatever metadata you want to it This would be a change to the node:

def report_accuracy(predictions: np.ndarray, test_y: pd.DataFrame,
                                 neptune_run: neptune.run.Handler) -> None:
    target = np.argmax(test_y.to_numpy(), axis=1)
    accuracy = np.sum(predictions == target) / target.shape[0]
    neptune_run['nodes/report/accuracy'] = accuracy * 100

If you are interested in trying it out, here are some step-by-step examples: - Quickstart: https://docs.neptune.ai/integrations-and-supported-tools/automation-pipelines/kedro#quickstart - Compare Kedro pipelines: https://docs.neptune.ai/integrations-and-supported-tools/automation-pipelines/kedro/compare-kedro-pipelines - Compare results between Kedro nodes: https://docs.neptune.ai/integrations-and-supported-tools/automation-pipelines/kedro/compare-results-between-kedro-nodes - Display Kedro node metadata and outputs: https://docs.neptune.ai/integrations-and-supported-tools/automation-pipelines/kedro/display-kedro-node-metadata-and-outputs Again, thank you so much, and I really hope this will be useful to some of you here!

🔥

Looking forward to see this integration, @User

Awesome @User! Pls share your experience, good or bad -> I'd really appreciate your feedback.