Yes I need to specify 2 params at runtime to make the pipeline successful

But I can't see where I could do that with the

Prefect

flow

I mean dynamically

oh or maybe I need to specify it inside the

KedroTask

yea prefect parameters are bounded to tasks

but to be honest last time I look at prefect & its ui was > 1 year ago

so can't recall how parameters flow from there to the tasks.

Ahah maybe the UI isn't as advanced as the cli/python sdk 🙂

if you still want to go the React route, the fix is one line of CSS. the

kedro-pipeline

class needs this style applied to it:

height: 100vh;

Thx for the tip. I succeed to deploy it via docker with kedro-viz python package 🙂 I keep this in mind if I need to insert it in another react app. Thanks !

If anyone want to take a look at the pipeline 🙂 https://makeusrich-viz.chainyo.tech/

Is there a way to prevent a node/pipeline to run? (maybe using hooks)

It really depends what you're trying to do

do you want to skip a node? or do you want to end execution?

skip a node

- So philosophically we not super big fans of conditional logic in Kedros execution because it creates a lot of complexity combinatorially speaking and is hard to debug - To do this sort of thing - one way is to have a 'status' object that you pass between nodes that triggers execution within it

@User Really digging your project. I'm more or less working on the same stuff. Can you tell me why didn't you choose MLFlow for model lifecycle management? It would be much more easier to track and manage models, right?

Hello all. Newbie to kedro here. I'm currently testing out using kedro for my organisation which uses Qubole for most of our ml work. In the Qubole environment we have access to a plain jupyter notebook where we are able to read files from s3 and run our models using the clusters available. I've attempted to load my kedro project as a folder in s3 and run the project from my notebook but cant seem to get the code to work. I believe it has to do with referencing the path in which the kedro project is location in s3. Attached is the code used to run in jupyter. Do you have any recommendations on how i can get this working ? Here's the error im getting * Could not find the project configuration file 'pyproject.toml' in //s3:/*

Kedro in Qubole

Hi everyone - *TLDR: I am looking for Best Practices to use Kedro to leverage Azure Machine Learning features and using MLOps techniques. * I will try to be clear but please let me know if I am not. I have shared what we are trying to build. Some context We are developing a model to be used as a prediction service by an existing application on AKS. We want to implement as much as possible "best in class" methods for MLOps using Azure Machine Learning features (Experiment tracking, model registry, maybe Azure ML Pipelines, etc.). Our code will reside in Azure Devops so we are also thinking about how to use Azure pipelines in the mix. Problematic Our DS is a very big fan of the Kedro framework. I am looking for the best way to have him adapt the framework to use the features of Azure ML to train, track, register, monitor datasets and models and deploy them to our different workload environments. I have so many questions I am not even sure where to start. - I would like to know if there are best practices to launch kedro pipeline runs after a new code push automatically in the Azure Machine Learning Service. Either for running a specific kedro pipeline or for building a whole Azure ML Pipeline exactly like the Kedro one? - What parts of the infrastructure should be leveraging the Azure SDK, and what parts should be just Azure CLI commands in the CICD pipelines? If you have any resource or example of someone successfully implementing this kind of thing it would be super helpful!

How to save keras model in kedro https://stackoverflow.com/questions/71154467/how-to-save-keras-model-in-kedro

Hi everyone, is there a kedro-sanctioned way to share code between pipelines in the same project? I have this bunch of classes I need in more than one sub-pipeline, and I don't want to duplicate code. Would a symlink help here. Thanks in advance 👍

You code is inside a

src/your_project

folder so it acts like a Python package if you add your functions to

__init__.py

file

The other important point is if you run

kedro pipeline create <pipeline_name>

it will create a pipeline that is a python package and has all of it's

__init__.py

files ready. https://kedro.readthedocs.io/en/latest/06_nodes_and_pipelines/03_modular_pipelines.html The cool part is that you can also re-use the same pipeline multiple times by overriding inputs, outputs and parameters, instructions all on that page

Thank you both @User and @User

Questions about experiment tracking plans -- CC @User @User . I read @User s post about experiment tracking plans. We are building kedro-dvc which integrates in with DVC experiment tracking (see https://github.com/FactFiber/kedro-dvc/discussions/6 for kedro-dvc discussion with links to DVC). It would seem that DVC and Kedro plans are largely orthogonal -- and could be used profitably together. For instance, DVC supports tracking data and parameter dependencies, and only partially rerunning pipelines. It supports forking experiments at checkpoints in the middle of pipelines, comparing metrics between experiments and forks. Also supports publishing experiments to git branches or pushing them "as experiments" to other repo users. (Underneath it uses the git-stash mechanism together with internal files to cache metrics.) [To this list we plan to track code dependencies as well as data dependencies in Kedro-DVC, allowing partial reruns to depend on code changes, even if not noted explicitly.] Kedro, on the other hand, seems to focus on cross experiment visualization, adding to kedro-viz. (DVC provides this via DVC-studio but that is on the other side of the freemium barrier.) To integrate, it would seem the key piece is the "session store". I wonder: a) Could the session store be a plugin with a defined API, rather than a piece of kedro-viz? (Then I could switch out the default.) b) Or will the session store have a defined API? c) How does your planned session mechanism deal with different versions of data?

Session store

How important is 3.6 support (on Windows?) at this time? I was looking into the build issues, but don't want to sink too much time if releasing 0.18 soon and don't care about this at all. I raised a draft PR that installs Rust toolchain for Windows builds, and it resolves the current issues. However, I'm still getting issues installing packages using pip in the CI, looks like for two reasons: 1. charmap can't deal with the explosion emoji coming from somewhere (click?) 2. Scarier thing is, getting stuff like

Ignoring ipython: markers 'python_version > "3.6"' don't match your environment
Ignoring pip-tools: markers 'python_version > "3.6"' don't match your environment
Ignoring pip-tools: markers 'python_version > "3.6"' don't match your environment

Guess the latter may explain the Jupyter timeouts? CI: https://app.circleci.com/pipelines/github/kedro-org/kedro/6002/workflows/a4c2f4b9-9745-4319-b8de-a5a898f40aca/jobs/92733

Everyone hates Windows 3.6 e2e tests

Anyone use Kedro and Airflow, successfully?