https://kedro.org/ logo
Join the conversationJoin Discord
Channels
advanced-need-help
announcements
beginners-need-help
introductions
job-posting
plugins-integrations
random
resources
welcome
Powered by Linen
advanced-need-help
  • l

    limdauto

    02/09/2022, 9:34 PM
    Not sure what you are after.
    --extra-params
    are only meaningful at runtime, not at session creation time.
  • c

    ChainYo

    02/09/2022, 9:35 PM
    Yes I need to specify 2 params at runtime to make the pipeline successful
  • c

    ChainYo

    02/09/2022, 9:35 PM
    But I can't see where I could do that with the
    Prefect
    flow
  • c

    ChainYo

    02/09/2022, 9:35 PM
    I mean dynamically
  • c

    ChainYo

    02/09/2022, 9:38 PM
    oh or maybe I need to specify it inside the
    KedroTask
  • l

    limdauto

    02/09/2022, 9:39 PM
    yea prefect parameters are bounded to tasks
  • l

    limdauto

    02/09/2022, 9:39 PM
    but to be honest last time I look at prefect & its ui was > 1 year ago
  • l

    limdauto

    02/09/2022, 9:40 PM
    so can't recall how parameters flow from there to the tasks.
  • c

    ChainYo

    02/09/2022, 9:41 PM
    Ahah maybe the UI isn't as advanced as the cli/python sdk ๐Ÿ™‚
  • t

    tynan

    02/10/2022, 10:28 AM
    if you still want to go the React route, the fix is one line of CSS. the
    kedro-pipeline
    class needs this style applied to it:
    height: 100vh;
  • c

    ChainYo

    02/10/2022, 10:29 AM
    Thx for the tip. I succeed to deploy it via docker with kedro-viz python package ๐Ÿ™‚ I keep this in mind if I need to insert it in another react app. Thanks !
  • c

    ChainYo

    02/10/2022, 11:53 AM
    If anyone want to take a look at the pipeline ๐Ÿ™‚ https://makeusrich-viz.chainyo.tech/
  • a

    Arnaldo

    02/11/2022, 1:44 PM
    Is there a way to prevent a node/pipeline to run? (maybe using hooks)
  • d

    datajoely

    02/11/2022, 1:50 PM
    It really depends what you're trying to do
  • d

    datajoely

    02/11/2022, 1:50 PM
    do you want to skip a node? or do you want to end execution?
  • a

    Arnaldo

    02/11/2022, 1:54 PM
    skip a node
  • d

    datajoely

    02/11/2022, 1:59 PM
    - So philosophically we not super big fans of conditional logic in Kedros execution because it creates a lot of complexity combinatorially speaking and is hard to debug - To do this sort of thing - one way is to have a 'status' object that you pass between nodes that triggers execution within it
  • d

    Dhaval

    02/14/2022, 4:38 PM
    @User Really digging your project. I'm more or less working on the same stuff. Can you tell me why didn't you choose MLFlow for model lifecycle management? It would be much more easier to track and manage models, right?
    c
    y
    • 3
    • 17
  • a

    aanan

    02/16/2022, 9:44 AM
    Hello all. Newbie to kedro here. I'm currently testing out using kedro for my organisation which uses Qubole for most of our ml work. In the Qubole environment we have access to a plain jupyter notebook where we are able to read files from s3 and run our models using the clusters available. I've attempted to load my kedro project as a folder in s3 and run the project from my notebook but cant seem to get the code to work. I believe it has to do with referencing the path in which the kedro project is location in s3. Attached is the code used to run in jupyter. Do you have any recommendations on how i can get this working ? Here's the error im getting * Could not find the project configuration file 'pyproject.toml' in //s3:/*
    l
    a
    • 3
    • 3
  • l

    Lorena

    02/16/2022, 10:22 AM
    Kedro in Qubole
  • t

    Tsakagur

    02/16/2022, 5:25 PM
    Hi everyone - *TLDR: I am looking for Best Practices to use Kedro to leverage Azure Machine Learning features and using MLOps techniques. * I will try to be clear but please let me know if I am not. I have shared what we are trying to build. Some context We are developing a model to be used as a prediction service by an existing application on AKS. We want to implement as much as possible "best in class" methods for MLOps using Azure Machine Learning features (Experiment tracking, model registry, maybe Azure ML Pipelines, etc.). Our code will reside in Azure Devops so we are also thinking about how to use Azure pipelines in the mix. Problematic Our DS is a very big fan of the Kedro framework. I am looking for the best way to have him adapt the framework to use the features of Azure ML to train, track, register, monitor datasets and models and deploy them to our different workload environments. I have so many questions I am not even sure where to start. - I would like to know if there are best practices to launch kedro pipeline runs after a new code push automatically in the Azure Machine Learning Service. Either for running a specific kedro pipeline or for building a whole Azure ML Pipeline exactly like the Kedro one? - What parts of the infrastructure should be leveraging the Azure SDK, and what parts should be just Azure CLI commands in the CICD pipelines? If you have any resource or example of someone successfully implementing this kind of thing it would be super helpful!
    d
    • 2
    • 9
  • u

    user

    02/17/2022, 8:23 AM
    How to save keras model in kedro https://stackoverflow.com/questions/71154467/how-to-save-keras-model-in-kedro
  • w

    williamc

    02/17/2022, 9:33 PM
    Hi everyone, is there a kedro-sanctioned way to share code between pipelines in the same project? I have this bunch of classes I need in more than one sub-pipeline, and I don't want to duplicate code. Would a symlink help here. Thanks in advance ๐Ÿ‘
  • c

    ChainYo

    02/18/2022, 6:12 AM
    You code is inside a
    src/your_project
    folder so it acts like a Python package if you add your functions to
    __init__.py
    file
  • d

    datajoely

    02/18/2022, 9:49 AM
    The other important point is if you run
    kedro pipeline create <pipeline_name>
    it will create a pipeline that is a python package and has all of it's
    __init__.py
    files ready. https://kedro.readthedocs.io/en/latest/06_nodes_and_pipelines/03_modular_pipelines.html The cool part is that you can also re-use the same pipeline multiple times by overriding inputs, outputs and parameters, instructions all on that page
  • w

    williamc

    02/18/2022, 5:15 PM
    Thank you both @User and @User
  • s

    shaunc

    02/21/2022, 4:03 PM
    Questions about experiment tracking plans -- CC @User @User . I read @User s post about experiment tracking plans. We are building kedro-dvc which integrates in with DVC experiment tracking (see https://github.com/FactFiber/kedro-dvc/discussions/6 for kedro-dvc discussion with links to DVC). It would seem that DVC and Kedro plans are largely orthogonal -- and could be used profitably together. For instance, DVC supports tracking data and parameter dependencies, and only partially rerunning pipelines. It supports forking experiments at checkpoints in the middle of pipelines, comparing metrics between experiments and forks. Also supports publishing experiments to git branches or pushing them "as experiments" to other repo users. (Underneath it uses the git-stash mechanism together with internal files to cache metrics.) [To this list we plan to track code dependencies as well as data dependencies in Kedro-DVC, allowing partial reruns to depend on code changes, even if not noted explicitly.] Kedro, on the other hand, seems to focus on cross experiment visualization, adding to kedro-viz. (DVC provides this via DVC-studio but that is on the other side of the freemium barrier.) To integrate, it would seem the key piece is the "session store". I wonder: a) Could the session store be a plugin with a defined API, rather than a piece of kedro-viz? (Then I could switch out the default.) b) Or will the session store have a defined API? c) How does your planned session mechanism deal with different versions of data?
    a
    d
    +2
    • 5
    • 25
  • a

    antony.milne

    02/21/2022, 4:45 PM
    Session store
  • d

    deepyaman

    02/25/2022, 12:27 PM
    How important is 3.6 support (on Windows?) at this time? I was looking into the build issues, but don't want to sink too much time if releasing 0.18 soon and don't care about this at all. I raised a draft PR that installs Rust toolchain for Windows builds, and it resolves the current issues. However, I'm still getting issues installing packages using pip in the CI, looks like for two reasons: 1. charmap can't deal with the explosion emoji coming from somewhere (click?) 2. Scarier thing is, getting stuff like
    Ignoring ipython: markers 'python_version > "3.6"' don't match your environment
    Ignoring pip-tools: markers 'python_version > "3.6"' don't match your environment
    Ignoring pip-tools: markers 'python_version > "3.6"' don't match your environment
    Guess the latter may explain the Jupyter timeouts? CI: https://app.circleci.com/pipelines/github/kedro-org/kedro/6002/workflows/a4c2f4b9-9745-4319-b8de-a5a898f40aca/jobs/92733
    a
    d
    • 3
    • 7
  • a

    antony.milne

    02/25/2022, 2:01 PM
    Everyone hates Windows 3.6 e2e tests
Powered by Linen
Title
a

antony.milne

02/25/2022, 2:01 PM
Everyone hates Windows 3.6 e2e tests
View count: 1