https://kedro.org/ logo
Join the conversationJoin Discord
Channels
advanced-need-help
announcements
beginners-need-help
introductions
job-posting
plugins-integrations
random
resources
welcome
Powered by Linen
beginners-need-help
  • s

    silvadenis

    09/15/2022, 1:06 PM
    Any suggestions for handling dynamic pipelines? I have a scenario that I would like to run a model training and model evaluation based on labels on a dataset. So each label would trigger one pipeline.
  • d

    datajoely

    09/15/2022, 1:41 PM
    So dynamic pipelines are a bit of a minefield because they become complex very quickly!
  • d

    datajoely

    09/15/2022, 1:41 PM
    In this situation I'd go for two approaches: - have your labeling process write to a dataset kedro can read - have the output of your labeling process passed in to kedro run --params
  • s

    silvadenis

    09/15/2022, 1:44 PM
    So for example we have the labels, A and B, so the labeling pipeline would output a single dataframe with values for A and B, then the training pipeline is coded for handling one label, then I can pass kedro run --param label:A, did I get it right?
  • d

    datajoely

    09/15/2022, 2:33 PM
    Something like that
  • n

    nikolarahman

    09/15/2022, 2:41 PM
    Hi Kedro community!! I just started using kedro and I got stuck. I'm pretty sure that lots of people had the same issue, but I've tried searching everywhere and I can't find the solution. Could anyone point me to the right answer? So, my pipeline is pretty standard, I assume: * load images from a directory * preprocess * CNN inference * postprocess * save output I've created a partitioned image dataset like in the pokemon example here https://kedro.readthedocs.io/en/latest/extend_kedro/custom_datasets.html. I'd like to apply the inference pipeline sequentially to each image (or each minibatch), but I can't figure out how to pass image-by-image to the pipeline. Does anyone know how to do this?
  • s

    sri

    09/16/2022, 7:49 PM
    in catalog.yml we are using runtime arguments like bucket_name below: raw_boat_data: filepath: "s3a://${bucket_name}/${key_prefix}/${folders.raw}/boats.csv" file_format: parquet bucket_name is also defined in globals.yml and is overridden by runtime parameter through commandline I see that sometimes the bucket_name is not correctly picked from the command line, it is still picking the value from globals.yml. what am i missing here?
  • d

    datajoely

    09/16/2022, 7:50 PM
    The command line part is custom, default kedro doesn't do that iirc
  • n

    noklam

    09/16/2022, 7:53 PM
    See this if you are on 0.18.x https://github.com/kedro-org/kedro/discussions/1782
  • r

    rohan_ahire

    09/17/2022, 4:35 PM
    Understanding data catalog datasets
    d
    • 2
    • 4
  • r

    rohan_ahire

    09/19/2022, 8:06 PM
    Using kedro in databricks workflow job
    • 1
    • 3
  • v

    Vici

    09/20/2022, 7:44 AM
    Hey, I wonder whether the Pipeline README markdowns are meant to end up in the automatic sphinx docs. It doesn't seem so? I searched the html files for keywords from my Pipelines' markdown files and didn't have any matches. Or is there an easy way to switch behavior? Thank you!
  • d

    datajoely

    09/20/2022, 8:47 AM
    So we just provide a very thin layer on top of sphinx here and aren't opinionated on how you construct your docs. The readme files here can easily be added to the toc tree in index.rst but aren't automatically added
  • v

    Vici

    09/20/2022, 1:33 PM
    I see, thank you for helping out a novice 😊
  • d

    datajoely

    09/20/2022, 1:47 PM
    We all were once!
  • g

    Goss

    09/22/2022, 3:53 PM
    Is there any way to have a pipeline run a bash script as a node?
  • d

    datajoely

    09/22/2022, 3:54 PM
    Via the subprocess module but it's messy
  • g

    Goss

    09/22/2022, 3:58 PM
    Anyone ever try the
    sh
    module for this?
  • r

    rafael.gildin

    09/23/2022, 2:51 PM
    Hi guys, Is there any way to continue the pipeline even if a node failed?
  • n

    noklam

    09/23/2022, 3:00 PM
    Use
    on_node_error
    hook potentially, but be cautious about the error-handling.
  • d

    datajoely

    09/23/2022, 3:06 PM
    Or a custom runner
  • r

    rafael.gildin

    09/23/2022, 3:09 PM
    thanks. Looking for them !
  • r

    rafael.gildin

    09/23/2022, 3:10 PM
    Other issue, is there a way to reduce the error message from cli?
  • n

    noklam

    09/23/2022, 3:27 PM
    You can set the level of the log to display via the
    logging.yml
    , but in general I don't think you want to hide error messages
  • r

    rafael.gildin

    09/23/2022, 3:32 PM
    thanks
  • r

    rafael.gildin

    09/23/2022, 4:22 PM
    how?
  • n

    noklam

    09/24/2022, 8:40 PM
    You simply update the LEVEL of logging that you desire
  • n

    noklam

    09/24/2022, 8:41 PM
    There should be a
    logging.yml
    , it's standard Python Logging module if you need more docs about it.
  • r

    rafael.gildin

    09/24/2022, 9:46 PM
    Even if I change it , the huge error message doesn’t disappear.
  • r

    rafael.gildin

    09/24/2022, 9:46 PM
    Thanks you anyway
Powered by Linen
Title
r

rafael.gildin

09/24/2022, 9:46 PM
Thanks you anyway
View count: 1