@Rafał Can you try this?

yaml
from_nodes: node1,node2

Providing the list-type

from_inputs:
        - vision_test_bunch

yields

Pipeline does not contain data_sets named ["['vision_test_bunch']"]

Yes, as I mentioned before. One should use single string with commas. I think it would be better to allow giving the list-type.

For example,

params

section can be YAML dict and not

string

Any help with

lv

structure ? 😉

kedro run --config=config.yml

does not work as expected

Hello everyone, here's the situation that I'm trying to tackle. I have a pipeline node that uses a function that raises an exception if the incoming DataFrame is empty. However, this halts the entire pipeline and isn't a very graceful exit. Is there a way that I can tell kedro to exit the node without outputting anything and continue by loading the data that is already stored in S3 via the catalog?

I tried passing a None out of a function, but kedro won't let me save a None type.

So the logic that u want to achieve is If data is available, run node and pass the variable to next node If is empty, skip the logic and load it from storage instead?

Yes...the reasoning behind the exception is that a bot is uploading data to a folder that we are reading from. If the bot has not uploaded anything, I have it raise an exception (this has to do with some downstream processing that will throw an error if it isn't handled). However, this causes an abrupt exit which is what I'm trying to avoid.

You could save a empty dataframe with the correct schema?

Ah...that would overwrite the existing data that I have in S3 though.

But maybe if I reconfigure some things that would be possible.

You could do an Incremental dataset

Which wouldn't actually add anything

If it switches to an

IncrementalDataSet

it'll pass in nothing into the function right?

Assuming there isn't anything new in the directory

Interesting... Haven't thought about that as I feel this may related to dynamic pipeline but incremental dataset could be a nice solution.

I'm confused...how would that be a solution by itself?

If you write an empty dataframe with the the right schema but no rows in the case the bot returns nothing to an IncrementalDataSet it will run gracefully

So...it would go something like: (1) Setup

IncrementalDataSet

(2) Run logic within function. If dataset is empty, it will just return an empty

DataFrame

. (3) That

DataFrame

gets saved as a parquet within S3. (4) That parquet gets loaded within the next node. Its (4) that I'm not sure about. That would load a an empty parquet which would require some subsequent handling, I suspect.

Unless I'm missing something.

So you get all partitions in 4 so you can just concatenate them

Ah...let me back up. So the dataset going into the function is currently a

PartitionedDataSet

, which we setup this way because it would allow us to load the information that we need from the directory. There are checks that are performed to ensure that the file is the right one (checking filename, etc). At the end, after all the processing, the function takes each

DataFrame

and adds it to a list which is then concatenated into a single

DataFrame

.

The next node in the sequence then loads that single

DataFrame

So the incremental dataset is basically the same behind the scenes it just retains a last written checkpoint

We're now just saying an empty partition with the right schema can be concatenated fine

Hmmm....so if the

DataFrame

is empty, could I setup logic to use the previous checkpoint instead?

If you're already using partitioned dataset then just writing a zero row dataframe should work

I wasn't aware of that when you asked originally