No the point of hooks is that you have a lot of flexibility, the example is very much demonstrating functionality more than specific good practices

In my opinion

I see. Thanks for replying and helping me out.

Hello 🙂 I'm using Kedro to train a model and predict values with this model afterwards. The model is saved versioned after training and for prediction I usually take the lastes version. But if I want to use a specific version, I have to define this in the terminal after kedro run (I read on the Kedro documentation). Is there an option to get the defined version? I want to write a log for my prediction to keep things reproducible. Thanks, Anna

Logging load version

Hello. I have a question regarding to the execution order of nodes. I understand the node execution order is decided by Kedro, not exactly like the layout in pipeline.py. I would like to understand the underlying mechanism how the order is determined? Is there randomness involved? If there is randomness, it might make the testing and QA work difficult.

Under the hood it is a topological sort. There is no guarantee about the order - if there are more than 1 possible solutions. Node are suppose to be pure python function without side effect, so ordering should not affect the result.

That said, there are cases like random sequence generation are affected by the order, there is an issue to change this to a more deterministic way.

I @wwliu , I would also clarify @noklam 's point slightly that

There is no guarantee about the order

but only per dependency level, i.e. If dataset

D

requires

A

,

B

and

C

.

D

will always be the last executed, but the order in which

A

,

B

and

C

in not fixed per run.

Hi, so, I have a pipeline like this:

def _parse_inctf() -> Pipeline:
    return Pipeline(
        [
            node(
                func=nodes.insert_columns_inctf,
                inputs='external-inct-fracionada',
                outputs="inctf-preprocess-01-insert-columns",
                name="read-and-insert-columns-inctf",
            ),
            node(
                func=nodes.parse_inct_dates,
                inputs="inctf-preprocess-01-insert-columns",
                outputs="inctf-preprocess-02-parse-dates"
            ),
            node(
                func=nodes.get_pct_change,
                inputs="inctf-preprocess-02-insert-columns",
                outputs="inctf-preprocessed"
            ),
        ]
    )

From, those datasets, only the

external-inct-fracionada

and

inctl-preprocessed

are actually declared in the

catalog.yml

. I want to pass the others as MemoryDatasets, they are intermediaries to my pipeline, but when I run, I get this error:

ValueError: Pipeline input(s) {'inctf-preprocess-02-insert-columns'} not found in the DataCatalog

Is there a way of doing this without declaring each intermediary dataset in my catalog? Just so you know, this is the entrance of

external-inct-fracionada

in my catalog:

external-inct-fracionada:
  type: project.io.encrypted_excel.EncryptedExcelDataSet
  filepath: "${DATA_DIR}/External/INCT/INCTF_0222.xls"

And

EncryptedExcelDataSet

and it's implementation is seen in the attached file

The naming of intermediare data can be arbitary, you just need to have a consistent name. If it is a memory data, it must be output of some other nodes. Change 02 to 01 and it should run

Thanks!

Is there any control flow that’s supported in a Kedro Dag?

Could you explain more what you mean? Do you mean conditional nodes?

If you do, we don't be design since we want to enforce reproducibility. The way to achieve this is to have different registered pipelines/environments/instances of modular pipelines

I’m referring to the ability to run nodes conditionally depending on outputs of other nodes or configs. In many DAG solutions, you can have branching flows depending on conditions but I don’t see that supported in Kedro. I was wondering if I may be missing something.

Yeah this is a deliberate choice not to support that because we believe it harms reproducibility and makes things harder to debug

What's the pattern for numerous files as raw data? I want to download about 2000 files of the same type with different dates, e.g. "senate_2006-03-30.xml". 1. Do I create a catalog entry for each file? 2. Does the download-node

output

to a list of length 2000, i.e.

["senate_2006-03-30", "senate_2006-03-31", ...]

, i.e. a 2000-line

pipeline.py

? Or is there some sort of clever templating?

PartitionedDataset?

Yes this solved the problem thank you.

Hello people, how should I work with chunking a huge dataset when applying a function? I found an issue but it was not clear how should I deak with the context manager inside the function. Thanks!!!!

Are you using pandas? Can you post a snippet?

It's from my work, but you cant just consider a really huge pandas dataset that does not fit memory in a function where you just, let's say, add a collumn.

The only lazy evaluation I know is though Spark, which I dont have access at the time.

Sure - but is this a general pandas question or a Kedro one? This tutorial shows simple chunking + full dask https://pythonspeed.com/articles/faster-pandas-dask/

It's a Kedro question, I know how to chunk outside the node context

My question is: I know how the _load (let's say I return the chunk iterator object from pandas) can return the iterator, but how the node itself will know how to apply all the steps inside the function for EACH chunk (not just for one) without a outside loop (let's say a wrapper in the node).

I think what you get in a node will be a generator instead of a dataframe if you are using the chunk iterator. That looping would be logic that you write inside the node.

Yeah, maybe returning a "map" will do trick?