Is there a way to disable logging so that it skips this step?

If you're using the latest version, then removing the

conf/base/logging.yml

entirely. Docs on kedro logging might be useful. https://kedro.readthedocs.io/en/stable/logging/logging.html

Logging is working now I think. I was giving the wrong cloud path.

Hi All. If I am using managed mlflow within databricks, do I still need kedro-mlflow plugin?

Hi All. If I am using managed mlflow within databricks, do I still need kedro-mlflow plugin?

When making a create experiment api call, where does it get the credentials from ?

endpoint = '/api/2.0/mlflow/experiments/create'                    │ │
│ │     host_creds = <mlflow.utils.rest_utils.MlflowHostCreds object at      │ │
│ │                  0x7f2203b6bbe0>                                         │ │
│ │      json_body = {'name': '/mnt/files/rohan/kedro_data_science/'}        │ │
│ │         method = 'POST'                                                  │ │
│ │       response = <Response [404]>                                        │ │
│ │ response_proto = <class 'rich.pretty.Node'>.__repr__ returned empty      │ │
│ │                  string

So kedro-mlflow plugin is maintained by @Galileo-Galilei he sometimes hangs on #908346260224872480 but I'd bet this answe is on the plugin docs

Hi All If I am using managed mlflow

Is there any way to create multiple pipelines based on multiple parameters in a yaml file? And then pass this on to airflow like in the same format as creating Dags programatically

@Eliãn There is

kedro-airflow

which helps you to create a Airflow Dag. However, you don't usually want to have a 1-1 mapping between Kedro pipeline and orchestrator DAGs, since they are usually larger node conceptually.

I am not very sure what do you mean by multiple pipelines base on multiple parameters, this is something Kedro support already I suppose? So what you need to do is just running different subset of Kedro pipeline within an Airflow DAG.

For example if you have pipeline "a","b","c", it could be 3 nodes in your Airflow DAG.

in each node you simply do

kedro run --pipeline a"

, or optionally the Python API (which the

kedro-airflow

helps you to do that)

Yes, I think I didn't express it very well, in fact I thought of a situation where we had the same nodes for different models. Where each model only differs because of some parameters. In airflow, dags are created in an iteration of this yaml file itself. Even though the tasks are the same, the ids are different causing each DAG to be created separately.

Also, thanks for the help

python
configs = {
    'products': {
        'schedule_interval':'@weekly'
    },
    'customers': {
        'schedule_interval':'@daily'
    },
}

def generate_dag(dag_id, start_date, schedule_interval, details):
    with DAG(dag_id, start_date=start_date, schedule_interval=schedule_interval) as dag:
        @task
        ...
    
for name, detail in configs.items():
    dag_id = f'dag_{name}'
    globals()[dag_id] = generate_dag(dag_id, ...)

@noklam something like this

I am not sure if I get it correctly, this look like just iterating some kind of configuration and then calling the same pipeline and override certain parameters? If that's the case

kedro run --params=<config>

or just use the Python API with

KedroSessions.create(extra_params=<params>)

then do a

session.run(pipeline=<some_pipeline>)

Oh, I thought this params on run was just to create an overlapping dictionary, I didn't know I could pick keys from params.yml

Thanks

Hi All. Please help with a couple of questions I have: 1. When I create a kedro session and run a kedro pipeline, my databricks job shows success even if the pipeline fails. Kedro is able to report all errors and halt the pipeline where it fails, however the databricks job is not able to catch the exception and shows the job as failure. Is there some exception handling that is required from my end and report it to databricks so that it shows a job failure?

from kedro.framework.session import KedroSession
from kedro.framework.startup import bootstrap_project
from pathlib import Path

metadata = bootstrap_project(Path.cwd())
with KedroSession.create(metadata.package_name) as session:
    session.run()

2. Does kedro have pipeline templates? Like for example, a pipeline template for regression use case or classification use case? Or do we just use the

kedro pipeline create data_processing

to create a sample template and add processing code in it?

hi all, i have a question on custom kedro dataset. Supposing that i created a partitioned dataset which is partitioned by date column, how do i take argument to read from a specific date partition? i want to be able to refer this dataset when defining the pipeline and pass it an argument with the the date value for the partition

Combine it with a

before_pipeline_run

hook!

what should i be doing in that hook? is there any example code to refer

There aren't any off the shelf examples, but people have had success accepting parameters and then updating the catalog object dynamically.

My favourite approach is to register an empty hook and use a notebook/breakpoint to work through it

Does kedro have its own feature store?

No but we treat them like any custom dataset

Like defining the dataset in data catalog right? And if someone wants to reuse the same dataset, then it is not really discoverable right? They will have to search in the catalog.yml manually?

Well you would create a custom SagemakerFeatureDataset (or equivalent) and the searchability would be in that tools UI

Does anyone have a workflow for dynamic parameters in production? I have a user trying to dynamically change a parameter to be 1 year ago in production. Is this the job of the orchestration tool above kedro? Does kedro have a better mechanism for dynamic parameters?