I tried to use

from-inputs

and

from-nodes

None of them work. However

to-nodes

works 😉

run:
    pipeline: vtb_pupil2iris
    env: vision_tests
    from-inputs:
        - vision_test_bunch

Ok, let me quickly have a try.

Do underscores work?

YES

Seems so, I remember it is always like this (at least in 0.16.x), but not sure is this by design or not. I remember I have seen some discussion about making it consistent with the cli.

@Rafał to-nodes doesn't work either, you also have to do the underscore.

Thank you. Thats why I asked for example. My observation is like that -

to-nodes

can be the list in YAML -

from_nodes

(unse underscore) - should be the string with comma separated node names Strange

I am afraid that

to-nodes

worked in my case. Will check it.

to-nodes

isn't pick up by the cli at all, so maybe that's why it is not complaining the list.

What? Why not picked up? Here is some kedro --help

Options:
  --from-inputs TEXT        A list of dataset names which should be used as a
                            starting point.
  --to-outputs TEXT         A list of dataset names which should be used as an
                            end point.
  --from-nodes TEXT         A list of node names which should be used as a
                            starting point.
  --to-nodes TEXT           A list of node names which should be used as an
                            end point.

@Rafał I mean in the

yaml

file, if you do

kedro run --to-nodes

it works .

As you notice in the yaml file you have to use underscore instead of the

-

Thanks. But anyway would be great to put such info in official kedro docs. And I still do not know how to pass the list. I am afraid that one can not use the syntax

from_nodes:
    - node1
    - node2

@Rafał Can you try this?

yaml
from_nodes: node1,node2

Providing the list-type

from_inputs:
        - vision_test_bunch

yields

Pipeline does not contain data_sets named ["['vision_test_bunch']"]

Yes, as I mentioned before. One should use single string with commas. I think it would be better to allow giving the list-type.

For example,

params

section can be YAML dict and not

string

Any help with

lv

structure ? 😉

kedro run --config=config.yml

does not work as expected

Hello everyone, here's the situation that I'm trying to tackle. I have a pipeline node that uses a function that raises an exception if the incoming DataFrame is empty. However, this halts the entire pipeline and isn't a very graceful exit. Is there a way that I can tell kedro to exit the node without outputting anything and continue by loading the data that is already stored in S3 via the catalog?

I tried passing a None out of a function, but kedro won't let me save a None type.

So the logic that u want to achieve is If data is available, run node and pass the variable to next node If is empty, skip the logic and load it from storage instead?

Yes...the reasoning behind the exception is that a bot is uploading data to a folder that we are reading from. If the bot has not uploaded anything, I have it raise an exception (this has to do with some downstream processing that will throw an error if it isn't handled). However, this causes an abrupt exit which is what I'm trying to avoid.

You could save a empty dataframe with the correct schema?

Ah...that would overwrite the existing data that I have in S3 though.

But maybe if I reconfigure some things that would be possible.

You could do an Incremental dataset

Which wouldn't actually add anything

If it switches to an

IncrementalDataSet

it'll pass in nothing into the function right?