It's in stable? I think

Got it. Will read and implement. I was only able to access this link and did not know where to go from there - https://kedro.readthedocs.io/en/stable/logging/experiment_tracking.html

I agree the docs doesn't make it very clear.

Let us know if you have any problem with experiment tracking

I will raise this to the team, I think we can organise the docs slightly better here.

That would really, help as I spent too much time on it and was actually going to not continue with kedro. Luckily I found this discord channel and everything started working so fast.

Experiment tracking is working for me now 🙂

Thanks for sticking with us! Experiment tracking is a relatively new feature, as a result the doc have been update frequently and we may forget to link things properly sometimes.

Don't hesitate to ask here if you have been stucking for a while.

Is there a way to run kedro pipelines through python commands? Like calling a function from my main method. As of right now, it needs to be executed as bash commands like "kedro run" or "python -m kedro run" and also another thing is that we have to execute these commands from the project directory. Is there a way I can run from some other directory and read the input and output from s3 or adls.

It is all possible

Kedro cli is the most common entrypoint

If you need to use remote storage, you simply change the path in catalog.yml and you can still run your kedro project from your root directory

That's solution 1, and I recommend it

Soluton 2: You can run kedro pipeline via Python API, which is KedroSession, which is more common when people are using platform like Databricks, which is a notebook and you can't call cli easily

As of today, you have to run kedro from its root directory, partly due to the structure that kedro assume, i.e. Where to read the configuration

There are proposal to make these command available everywhere within a kedro project. Personally I think that's a good idea, but we haven't started this piece of work

Is this the right documentation to get started with KedroSession? https://kedro.readthedocs.io/en/stable/kedro_project_setup/session.html?highlight=session

Yes i think so

Will try this

So Kedro is only for data pipeline authoring. For workflow orchestration, we still have to rely on airflow right? I was using the airflow plugin for converting kedro pipeline to an airflow dag and it worked. However, I think I ran into the same problem there where it could not find the params. I forgot what the error was, will run it later and see.

For workflow krchestration, things like airflow and prefect are doing great work already. We focus to author the data pipeline code.

If we choose a different workflow orchestration tool other than airflow, like for example Azure Data Factory (ADF), then we will have to break each Kedro task into a ADF node right? So it defeats the purpose of using Kedro.

As you can think of a kedro pipeline, is just a python package afterall, so it doesn't make too much difference from running anything else, ofcoz the DAGs has a similar structure, which make sense to break your pipeline into sub task

One common pattern that we see more often is breaking a modular pipeline into an equivalent airflow/prefect task

That's up to your decision

The main goal of Kedro is speeding up the development of these pipeline, and making sure they are in good quality. And by writing kedro pipeline you will follow a more functional approach, spliting out io into catalog, your node only has computation logic etc, which I think it's a benefit regardless what the deployment target is.

There is nothing stopping you to deploy your entire kedro pipeline as one task

But you have to make the decision, why do you want a orchestrator at the first place? Probably for scheduling, retrying etc, and there could be many things more, maybe it's some upstream task that trigger a kedro pipeline

Which you can think your kedro pipeline is a small DAG within a larger DAG