Did you do pip install kedro-viz?

Well, this was a quick fix thanks ❤️ Should

kedro-viz

be added to

requirements.txt

? Or in the documentation : https://kedro.readthedocs.io/en/0.18.1/tutorial/create_pipelines.html ?

I think the rationale behind not having it in the default requirements is due to the fact that it's not an a mandatory requirement to get a Kedro project up and running. If you are following the spaceflights tutorial it's added to the requirements during the first steps: https://kedro.readthedocs.io/en/0.18.1/tutorial/tutorial_template.html#install-dependencies

Yes this is exactly right, Viz is technically a plug-in, just a first party one

Hello, I am trying to implement Mlflow using Hooks in Kedro. I got this code snippet:

class ModelTrackingHooks:
    @hook_impl
    def after_node_run(self, node: Node, outputs: Dict[str, Any], inputs: Dict[str, Any]) -> None:
        if node._func_name == "train_model":
            model = outputs["example_model"]
            mlflow.sklearn.log_model(model, "model")
            mlflow.log_params(inputs["parameters"])

My question is, I only need to log metrics in this specific

train_model

node, while based on my understanding, this function will run every time a node finishes, and there could be a lot of nodes in the whole pipeline. Is there way I could specify which node this hook is hooked to?

Are you aware of the kedro-mlflow plugin created by our community? If the logic is only specific to a node, it may better be just included in the node's logic?

Thanks a lot for your reply. I am aware of the Kedro-mlflow plug-in, I haven’t explored much on that side, but I will. Regarding to Hooks, I am curious about the design philosophy behind hooks, or what would you suggest to be included in the Hooks instead of in the nodes?

So hooks leverage the same library that the pytest folks designed to support their plugin ecosystem, so a lot of the thinking is the same. In terms of the split between what's node logic and what's hook logic... I think you should remember that nodes are about dataflow/transformation and hooks are about tapping into different parts of the run lifecycle

Thanks, @datajoely. Could you elaborate more on different parts of the run lifecycle that could possibly use hooks? I am looking at this article https://medium.com/quantumblack/introducing-kedro-hooks-fd5bc4c03ff5, and it uses Mlflow as an example, and there are two

if

statements in the scripts.

if node._func_name == "split_data":
            mlflow.log_params(
                {"split_data_ratio": inputs["params:example_test_data_ratio"]}
            )

        elif node._func_name == "train_model":
            model = outputs["example_model"]
            mlflow.sklearn.log_model(model, "model")
            mlflow.log_params(inputs["parameters"])

These are node specific functions, Do you think these are better put in the node logic itself instead of hooks or it is a proper user case for hooks? Or as @noklam suggested using kedro-mlflow?

So the plugin is well supported and popular, that example is overly explicit, you would likely make the logging dynamic in practice and not really node specific

Would be quite fragile if you wanted to change the node name down the line

I agree that it would be fragile if we make explicit statement like this. Could you provide an example of how to logging dynamically if it is not too much to ask. Thanks.

I guess I mean you could just log every node._func_name and run the log_model method if the object is of the right type

But in the example, the object required to be logged is node specific, in

split_data

node I need to log

train_split_ratio

, and in

train_model

node, I need to log

model

object. Does this mean this senario is kind of against the hooks usage guide?

No the point of hooks is that you have a lot of flexibility, the example is very much demonstrating functionality more than specific good practices

In my opinion

I see. Thanks for replying and helping me out.

Hello 🙂 I'm using Kedro to train a model and predict values with this model afterwards. The model is saved versioned after training and for prediction I usually take the lastes version. But if I want to use a specific version, I have to define this in the terminal after kedro run (I read on the Kedro documentation). Is there an option to get the defined version? I want to write a log for my prediction to keep things reproducible. Thanks, Anna

Logging load version

Hello. I have a question regarding to the execution order of nodes. I understand the node execution order is decided by Kedro, not exactly like the layout in pipeline.py. I would like to understand the underlying mechanism how the order is determined? Is there randomness involved? If there is randomness, it might make the testing and QA work difficult.

Under the hood it is a topological sort. There is no guarantee about the order - if there are more than 1 possible solutions. Node are suppose to be pure python function without side effect, so ordering should not affect the result.

That said, there are cases like random sequence generation are affected by the order, there is an issue to change this to a more deterministic way.

I @wwliu , I would also clarify @noklam 's point slightly that

There is no guarantee about the order

but only per dependency level, i.e. If dataset

D

requires

A

,

B

and

C

.

D

will always be the last executed, but the order in which

A

,

B

and

C

in not fixed per run.

Hi, so, I have a pipeline like this:

def _parse_inctf() -> Pipeline:
    return Pipeline(
        [
            node(
                func=nodes.insert_columns_inctf,
                inputs='external-inct-fracionada',
                outputs="inctf-preprocess-01-insert-columns",
                name="read-and-insert-columns-inctf",
            ),
            node(
                func=nodes.parse_inct_dates,
                inputs="inctf-preprocess-01-insert-columns",
                outputs="inctf-preprocess-02-parse-dates"
            ),
            node(
                func=nodes.get_pct_change,
                inputs="inctf-preprocess-02-insert-columns",
                outputs="inctf-preprocessed"
            ),
        ]
    )

From, those datasets, only the

external-inct-fracionada

and

inctl-preprocessed

are actually declared in the

catalog.yml

. I want to pass the others as MemoryDatasets, they are intermediaries to my pipeline, but when I run, I get this error:

ValueError: Pipeline input(s) {'inctf-preprocess-02-insert-columns'} not found in the DataCatalog

Is there a way of doing this without declaring each intermediary dataset in my catalog? Just so you know, this is the entrance of

external-inct-fracionada

in my catalog:

external-inct-fracionada:
  type: project.io.encrypted_excel.EncryptedExcelDataSet
  filepath: "${DATA_DIR}/External/INCT/INCTF_0222.xls"

And

EncryptedExcelDataSet

and it's implementation is seen in the attached file

The naming of intermediare data can be arbitary, you just need to have a consistent name. If it is a memory data, it must be output of some other nodes. Change 02 to 01 and it should run

Thanks!

Is there any control flow that’s supported in a Kedro Dag?

Could you explain more what you mean? Do you mean conditional nodes?

If you do, we don't be design since we want to enforce reproducibility. The way to achieve this is to have different registered pipelines/environments/instances of modular pipelines

I’m referring to the ability to run nodes conditionally depending on outputs of other nodes or configs. In many DAG solutions, you can have branching flows depending on conditions but I don’t see that supported in Kedro. I was wondering if I may be missing something.