Just look at the signature of session.run

Yup found it. So to run individual nodes, all input/output datasets must be written to disk right? In memory datasets cannot work in this case.

Exactly

What I find easiest is the use named registered pipelines to describe the blocks

And the you can just do kedro run --pipeline

You mean create small mini pipelines containing few tasks which maybe utilizing in memory datasets and chain together those mini pipelines in a dag. Like in the space flights tutorial we had data processing pipeline and data science pipeline

Exactly!

Hello, I recently used the kedro-kubeflow plugin to generate yaml files for my kedro pipelines, what is the next step to run the kubeflow.yml in a cloud service like AI Platform or Vertex AI, do you have any guide online guys?

Hello, I need to access 'session.load_context().mlflow' attribute within pipeline for current session. How can I do it? 'get_current_session()' is deprecated.

Check out this https://medium.com/google-cloud/migrate-kedro-pipeline-on-vertex-ai-fa3f2c6f7aad?s=09

Sessions are now ephemeral and linked to a run, but you can access a session mid run using hooks

I suggest using the actual plugin for Vertex AI - https://github.com/getindata/kedro-vertexai

Hello guys, I have this problem output when I run the commando kedro kubeflow compile, any idea?

So since this is matianed by the GetInData team I'd suggest posting on #908346260224872480 or raising an issue on their repo

Let us know what the solution is so we can put it on the documentation backlog

I was running my kedro pipeline from the cloud. So I replaced all the config paths for storage and logs with the azure bucket path. However, when I run the pipeline, it searches for the paths within the project directory itself. For example, it is looking for /project/dir/. Is there some other configs I am missing?

Hi @rohan_ahire , Can you share the example of the file path you've put in the catalog? In general for cloud storage, you might need to add the file system prefix as well (eg: s3:, abfs://, gs://). If it is automatically looking in the project, I suspect you're likely missing any prefix.

Here is my logging.yml

version: 1

disable_existing_loggers: False

formatters:
  simple:
    format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s"

handlers:
  console:
    class: logging.StreamHandler
    level: INFO
    formatter: simple
    stream: ext://sys.stdout

  info_file_handler:
    class: logging.handlers.RotatingFileHandler
    level: INFO
    formatter: simple
    filename: dbfs:/mnt/files/rohan/kedro_data_science/logs/info.log
    maxBytes: 10485760 # 10MB
    backupCount: 20
    encoding: utf8
    delay: True

  error_file_handler:
    class: logging.handlers.RotatingFileHandler
    level: ERROR
    formatter: simple
    filename: dbfs:/mnt/files/rohan/kedro_data_science/logs/errors.log
    maxBytes: 10485760 # 10MB
    backupCount: 20
    encoding: utf8
    delay: True

  rich:
    class: rich.logging.RichHandler

loggers:
  kedro:
    level: INFO

  kedro_logs_science_demo:
    level: INFO

root:
  handlers: [rich, info_file_handler, error_file_handler]

Hi guys, i have a really big question: If I want to integrate an event as a DataSet, (I recive a POST request from a webhook that came with some info that I want to store, process through the pipe and then send to a API DataSet) it's ok to use an incrementalDataSet?

here is the error FileNotFoundError: [Errno 2] No such file or directory: '/Workspace/Repos/Staging/kedro_data_science_demo/dbfs:/mnt/files/rohan/kedro_data_science/logs/info.log'

Do you get a similar error on data reads if you leave logs as local path but dataset paths as dbfs://. May be logging doesn't support cloud storage (not sure, just a shot in the dark)

Is there a way to disable logging so that it skips this step?

If you're using the latest version, then removing the

conf/base/logging.yml

entirely. Docs on kedro logging might be useful. https://kedro.readthedocs.io/en/stable/logging/logging.html

Logging is working now I think. I was giving the wrong cloud path.

Hi All. If I am using managed mlflow within databricks, do I still need kedro-mlflow plugin?

Hi All. If I am using managed mlflow within databricks, do I still need kedro-mlflow plugin?

When making a create experiment api call, where does it get the credentials from ?

endpoint = '/api/2.0/mlflow/experiments/create'                    │ │
│ │     host_creds = <mlflow.utils.rest_utils.MlflowHostCreds object at      │ │
│ │                  0x7f2203b6bbe0>                                         │ │
│ │      json_body = {'name': '/mnt/files/rohan/kedro_data_science/'}        │ │
│ │         method = 'POST'                                                  │ │
│ │       response = <Response [404]>                                        │ │
│ │ response_proto = <class 'rich.pretty.Node'>.__repr__ returned empty      │ │
│ │                  string

So kedro-mlflow plugin is maintained by @Galileo-Galilei he sometimes hangs on #908346260224872480 but I'd bet this answe is on the plugin docs

Hi All If I am using managed mlflow