You do need to do some work, because the Kedro DAG doesn't have a 1 to 1 mapping to your orchestrator DAG. We have plugin for airflow to help you get started, but I believe there are a lot of rooms for customization which highly depends on the use case.

In the most naive case, you can even just have a bash operator which do

kedro run

The problem with kedro pipeline being one single task in a larger dag is because it could be a big complex pipeline taking a long time to execute and we would not want to restart the entire kedro pipeline if only certain tasks fail. Also we lose the visualization feature as well when we deploy. But I like your idea about smaller kedro pipelines being part of a larger dag. Will have to think about how to deploy kedro pipelines in production. We need to figure out a way to deploy while making sure we reuse the modularity that kedro provides and the faster development time it provides.

👍I think a modular pipeline is a good starting point, but it make sense to group a few maybe

Can kedro run specific nodes in a pipeline? Like in my orchestration tool, I am thinking of calling specific kedro pipeline nodes as a part of each task in the dag.

Yes! There are all sorts of parameters you can pass to

kedro run

command

Cool! Is is also possible through kedro session?

It is but when running from an orchestrator I think the CLI is cleaner

Just look at the signature of session.run

Yup found it. So to run individual nodes, all input/output datasets must be written to disk right? In memory datasets cannot work in this case.

Exactly

What I find easiest is the use named registered pipelines to describe the blocks

And the you can just do kedro run --pipeline

You mean create small mini pipelines containing few tasks which maybe utilizing in memory datasets and chain together those mini pipelines in a dag. Like in the space flights tutorial we had data processing pipeline and data science pipeline

Exactly!

Hello, I recently used the kedro-kubeflow plugin to generate yaml files for my kedro pipelines, what is the next step to run the kubeflow.yml in a cloud service like AI Platform or Vertex AI, do you have any guide online guys?

Hello, I need to access 'session.load_context().mlflow' attribute within pipeline for current session. How can I do it? 'get_current_session()' is deprecated.

Check out this https://medium.com/google-cloud/migrate-kedro-pipeline-on-vertex-ai-fa3f2c6f7aad?s=09

Sessions are now ephemeral and linked to a run, but you can access a session mid run using hooks

I suggest using the actual plugin for Vertex AI - https://github.com/getindata/kedro-vertexai

Hello guys, I have this problem output when I run the commando kedro kubeflow compile, any idea?

So since this is matianed by the GetInData team I'd suggest posting on #908346260224872480 or raising an issue on their repo

Let us know what the solution is so we can put it on the documentation backlog

I was running my kedro pipeline from the cloud. So I replaced all the config paths for storage and logs with the azure bucket path. However, when I run the pipeline, it searches for the paths within the project directory itself. For example, it is looking for /project/dir/. Is there some other configs I am missing?

Hi @rohan_ahire , Can you share the example of the file path you've put in the catalog? In general for cloud storage, you might need to add the file system prefix as well (eg: s3:, abfs://, gs://). If it is automatically looking in the project, I suspect you're likely missing any prefix.

Here is my logging.yml

version: 1

disable_existing_loggers: False

formatters:
  simple:
    format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s"

handlers:
  console:
    class: logging.StreamHandler
    level: INFO
    formatter: simple
    stream: ext://sys.stdout

  info_file_handler:
    class: logging.handlers.RotatingFileHandler
    level: INFO
    formatter: simple
    filename: dbfs:/mnt/files/rohan/kedro_data_science/logs/info.log
    maxBytes: 10485760 # 10MB
    backupCount: 20
    encoding: utf8
    delay: True

  error_file_handler:
    class: logging.handlers.RotatingFileHandler
    level: ERROR
    formatter: simple
    filename: dbfs:/mnt/files/rohan/kedro_data_science/logs/errors.log
    maxBytes: 10485760 # 10MB
    backupCount: 20
    encoding: utf8
    delay: True

  rich:
    class: rich.logging.RichHandler

loggers:
  kedro:
    level: INFO

  kedro_logs_science_demo:
    level: INFO

root:
  handlers: [rich, info_file_handler, error_file_handler]

Hi guys, i have a really big question: If I want to integrate an event as a DataSet, (I recive a POST request from a webhook that came with some info that I want to store, process through the pipe and then send to a API DataSet) it's ok to use an incrementalDataSet?

here is the error FileNotFoundError: [Errno 2] No such file or directory: '/Workspace/Repos/Staging/kedro_data_science_demo/dbfs:/mnt/files/rohan/kedro_data_science/logs/info.log'

Do you get a similar error on data reads if you leave logs as local path but dataset paths as dbfs://. May be logging doesn't support cloud storage (not sure, just a shot in the dark)