https://kedro.org/ logo
Join the conversationJoin Discord
Channels
advanced-need-help
announcements
beginners-need-help
introductions
job-posting
plugins-integrations
random
resources
welcome
Powered by Linen
beginners-need-help
  • r

    RafaƂ

    04/26/2022, 10:18 AM
    I tried to use
    from-inputs
    and
    from-nodes
    None of them work. However
    to-nodes
    works 😉
    run:
        pipeline: vtb_pupil2iris
        env: vision_tests
        from-inputs:
            - vision_test_bunch
  • n

    noklam

    04/26/2022, 10:23 AM
    Ok, let me quickly have a try.
  • d

    datajoely

    04/26/2022, 10:41 AM
    Do underscores work?
  • r

    RafaƂ

    04/26/2022, 10:51 AM
    YES
  • n

    noklam

    04/26/2022, 10:51 AM
    Seems so, I remember it is always like this (at least in 0.16.x), but not sure is this by design or not. I remember I have seen some discussion about making it consistent with the cli.
  • n

    noklam

    04/26/2022, 10:52 AM
    @RafaƂ to-nodes doesn't work either, you also have to do the underscore.
  • r

    RafaƂ

    04/26/2022, 10:52 AM
    Thank you. Thats why I asked for example. My observation is like that -
    to-nodes
    can be the list in YAML -
    from_nodes
    (unse underscore) - should be the string with comma separated node names Strange
  • r

    RafaƂ

    04/26/2022, 10:53 AM
    I am afraid that
    to-nodes
    worked in my case. Will check it.
  • n

    noklam

    04/26/2022, 10:53 AM
    to-nodes
    isn't pick up by the cli at all, so maybe that's why it is not complaining the list.
  • r

    RafaƂ

    04/26/2022, 10:54 AM
    What? Why not picked up? Here is some kedro --help
    Options:
      --from-inputs TEXT        A list of dataset names which should be used as a
                                starting point.
      --to-outputs TEXT         A list of dataset names which should be used as an
                                end point.
      --from-nodes TEXT         A list of node names which should be used as a
                                starting point.
      --to-nodes TEXT           A list of node names which should be used as an
                                end point.
  • n

    noklam

    04/26/2022, 10:55 AM
    @RafaƂ I mean in the
    yaml
    file, if you do
    kedro run --to-nodes
    it works .
  • n

    noklam

    04/26/2022, 10:57 AM
    As you notice in the yaml file you have to use underscore instead of the
    -
  • r

    RafaƂ

    04/26/2022, 10:58 AM
    Thanks. But anyway would be great to put such info in official kedro docs. And I still do not know how to pass the list. I am afraid that one can not use the syntax
    from_nodes:
        - node1
        - node2
  • n

    noklam

    04/26/2022, 11:00 AM
    @RafaƂ Can you try this?
    yaml
    from_nodes: node1,node2
  • r

    RafaƂ

    04/26/2022, 11:00 AM
    Providing the list-type
    from_inputs:
            - vision_test_bunch
    yields
    Pipeline does not contain data_sets named ["['vision_test_bunch']"]
  • r

    RafaƂ

    04/26/2022, 11:01 AM
    Yes, as I mentioned before. One should use single string with commas. I think it would be better to allow giving the list-type.
  • r

    RafaƂ

    04/26/2022, 11:01 AM
    For example,
    params
    section can be YAML dict and not
    string
  • r

    RafaƂ

    04/26/2022, 11:03 AM
    Any help with
    lv
    structure ? 😉
  • n

    noklam

    04/26/2022, 11:03 AM
    kedro run --config=config.yml
    does not work as expected
  • b

    Burn1n9m4n

    04/26/2022, 6:27 PM
    Hello everyone, here's the situation that I'm trying to tackle. I have a pipeline node that uses a function that raises an exception if the incoming DataFrame is empty. However, this halts the entire pipeline and isn't a very graceful exit. Is there a way that I can tell kedro to exit the node without outputting anything and continue by loading the data that is already stored in S3 via the catalog?
  • b

    Burn1n9m4n

    04/26/2022, 6:29 PM
    I tried passing a None out of a function, but kedro won't let me save a None type.
  • n

    noklam

    04/26/2022, 6:32 PM
    So the logic that u want to achieve is If data is available, run node and pass the variable to next node If is empty, skip the logic and load it from storage instead?
  • b

    Burn1n9m4n

    04/26/2022, 6:34 PM
    Yes...the reasoning behind the exception is that a bot is uploading data to a folder that we are reading from. If the bot has not uploaded anything, I have it raise an exception (this has to do with some downstream processing that will throw an error if it isn't handled). However, this causes an abrupt exit which is what I'm trying to avoid.
  • d

    datajoely

    04/26/2022, 7:03 PM
    You could save a empty dataframe with the correct schema?
  • b

    Burn1n9m4n

    04/26/2022, 7:04 PM
    Ah...that would overwrite the existing data that I have in S3 though.
  • b

    Burn1n9m4n

    04/26/2022, 7:05 PM
    But maybe if I reconfigure some things that would be possible.
  • d

    datajoely

    04/26/2022, 7:05 PM
    You could do an Incremental dataset
  • d

    datajoely

    04/26/2022, 7:05 PM
    Which wouldn't actually add anything
  • b

    Burn1n9m4n

    04/26/2022, 7:12 PM
    If it switches to an
    IncrementalDataSet
    it'll pass in nothing into the function right?
  • b

    Burn1n9m4n

    04/26/2022, 7:12 PM
    Assuming there isn't anything new in the directory
Powered by Linen
Title
b

Burn1n9m4n

04/26/2022, 7:12 PM
Assuming there isn't anything new in the directory
View count: 1