is there a way to give datasets tags? I've got a bunch and I'm drowning in the complexity trying to remember what each one is for

guess not: https://github.com/kedro-org/kedro/issues/1076

Comment on that issue if you want that feature!

Is there any way to load the context from an active kedro session in 0.18.X? In the version 0.17.X there was the method: get_current_session inside kedro.framework.session.session. In our code we are using the kedro context to run a pipeline with different parameters in a loop for.

So we don't typically like this pattern since it makes reproducibility difficult. We removed the current session finctionality since now 1 session = 1 run. So you can create a new one and it will be fine

Perfect, thank you for your answer, also does this imply that the catalog from the session that is using can't be accessed at runtime without creating a new session?

Yes, you can use lifecycle hooks to mutate an existing session during a run

I made some syntactic sugar that makes the following possible in jupyter: It reloads the session with the new overrides and starts the pipeline. I can send you the code if you want.

%pipe tab_adult.ingest
for e1 in (0.3, 0.5, 0.9, 1.5):
    pipe("tab_adult.privbayes.synth", {"alg.e1": e1})

It also converts "alg.e1": e1 to {alg: {e1: e1}} for you

This is super interesting! Would you mind doing a show and tell on GitHub discussions?

Sure, how do I do that?

A big problem I have with kedro is that there are no fixed value inputs for nodes. Consider the following node. It needs to know the name of the results that it's logging. This is fixed for the node that runs the function. For now, I have to use closure functions (such as

mlflow_log_model_closure

) to pin the name for the version of the function that will run. What's the better solution for this?

def mlflow_log_model_results(name: str, res: pd.DataFrame):
    if not mlflow.active_run():
        return

    ...


def mlflow_log_model_closure(name: str):
    def closure(res: pd.DataFrame):
        return mlflow_log_model_results(name, res)

    closure.__name__ = f"log_{name}_model_results"
    return closure

Yes, I think lifecycle hooks have everything you need for this

Was just wondering if there was any best practice for trying to manage the complexity of a lot of datasets? I remember in my databases class we went over ORM modeling but it was always in the context of creating entities s.t. their interactions would be represented sensibly in tables that mirror that object structure. In this context though, I feel like we're all being given data tables without a pre-made ORM

Hi

Hi I am new to Kedro. I am trying to improve the machine learning workflow using Kedro. For our use case we are using Kedro with YOLO algorithm for object detection. I implemented a pipeline using Kedro in which the dataset (images and labels ) is saved to an intermediate data directory within the kedro project and hardcoding the path in the training node. This implementaion results in a non linear style of data flow (because there is no actual connection between the data node and the training node). I was wondering if there is a better way to handle this, The problem is that YOLO and most object detection algorithms needs only the path to the data instead of the actual data being loaded. Any help is appreciated, TIA.

So the typical way to do this is to define a custom YOLO dataset which can be declared in the catalog and also removes the need for you to do the hard coding. We can help you define this, but if you're interested in contributing it back to kedro I'm sure the community would appreciate it!

Okay, How should I proceed If I am to do this? YOLO uses images(JPG/PNG) and labels(.txt) files as dataset.

Hey @romhttps://kedro.readthedocs.io/en/stable/extend_kedro/custom_datasets.html we have a tutorial here.

You could also have a go at subclassing the existing image dataset to get this working

I'd also make sure you're aware of how the PartitionedDataSet wrapper works as that's usually important for this type of work

But, For YOLO we need to handle both image files and text files(labels). I think if we are to create a custom Kedro dataset for YOLO, this dataset should handle both these files. Which might involve combining the existing image dataset and existing text dataset and using it with PartitionedDataSet. What do you think?

I don't think there is anything wrong with expecting two file paths and essentially wrapping a version of ImageDataSet, TextDataSet and PartitionedDataSet all together!

Anyone have experience creating vscode plugins? I have kedro-lsp updated to work with kedro 0.17.x and 0.18.x in a beta release. Currently all that is out is go to defintion on datasets, but I have dataset completion close to working. I use neovim and have it working there, but I have spent a day trying to get it running in vscode with no luck. Pycharm, sublime, spyder, emacs, any ide with lsp support would also be welcome.

Hi everyone! I've been using Kedro for a couple of months but I've recently come across an issue that I'm hoping someone here can give me some advice on. I'm trying to use a runtime parameter (specified via

kedro run --params ...

) with templated configuration of my data catalog so I can specify a project/site name to use as a prefix (subdirectory) on my file paths in the catalog. I'm able to achieve this functionality if I specify the parameter in my globals config, but it appears that runtime parameters provided through the CLI are not available to the TemplateConfigLoader. My goal is to enable execution of the pipeline on different raw datasets while preserving the previous dataset's data directory and without requiring the user to edit the global config file. Is this possible or is there another way I should go about this? Any advice would be greatly appreciated 😀

If you are OK with the parameters being environment variables, you can add

os.environ

in globals_dict to parametrize the paths like https://github.com/kedro-org/kedro/issues/403.

Hi folks, in a previous post here, there was a question about passing parameters to SQL queries. While randomly reading about transcoding in dataset yesterday, it gave me an idea on how to use it for the problem at stake. The idea is to use a Jinja template for the SQL query — nothing new here — and use transcoding to impose the sequential run of the nodes, namely filling the template with the parameters and only then run the query by using a common file. The catalog would look like

aggregates@query_template:
  type: text.TextDataSet
  filepath: data/01_raw/aggregates_query.sql

aggregates@query_string:
  type: text.TextDataSet
  filepath: data/02_intermediate/filled_aggregates_query.sql

aggregates@query:
  type: pandas.SQLQueryDataSet
  filepath: data/02_intermediate/filled_aggregates_query.sql
  credentials: aggregates_uri

and the pipeline

def create_pipeline(**kwargs) -> Pipeline:
    return Pipeline(
        [
            node(
                parse_parameters,
                inputs=[
                    "params:start_date",
                    "params:end_date",
                    "params:metric",
                ],
                outputs="query_parameters",
            ),
            node(
                fill_template,
                inputs=["aggregates@query_template", "query_parameters"],
                outputs="aggregates@query_string",
            ),
            node(
                perform_query,
                inputs=["aggregates@query"],
                outputs="results",
            ),
        ]
    )

Transcoding ensures that the second node runs before the third node. I like using transcoding in this situation because it makes the link between nodes more transparent than using an extra output/input. Please let me know what you think about it.

We have configuration environments for this too

So this is a neat solution - it will work. SQL execution rather than storage isn't going to be a perfect fit for kedro unless there is a data frame like api

the config loader takes in the extra parameters via the

runtime_params

property. You can extend the template config loader and just add your path via the

self._config_mapping

in

__init__()

On 0.18.x this is a little different there are a few examples out there on GitHub