I think that's going to be a form of microbatching

Which is totally fine, you only need real streaming if your data is so high volume or throuput that it needs something like KSQL

Hello, I want to do k fold and I want to save each iteration in files. Is it possible to specify in catalog.yml that saving should be done in multiple files? I want to achieve something like for iteration 1 to save train1, test1, validate1 files... For iteration 2 same files but with a 2 at the end... And so on...

So partitioned / incremental dataset sort of give you this. I think to control the file path you may way to consider a custom dataset or some funky templated config

Hello, I am new to kedro Getting following error: Object 'SparkDataSet' cannot be loaded from 'kedro.extras.datasets.spark' Failed to instantiate dataset '---' of type '---' We are using an abstract dataset Kedro 17.3 Any help would be appreciated

Can you share the full error stack trace and the

catalog.yml

(the related entry)?

Can't share much because of organization's rules

It's s bit difficult to diagnose since you're using a custom wrapper dataset. Can we see the implementation for that?

Hello, really enjoying the use of Kedro to collaborate across teams in an organized manner

Part of our development stack is a Jupyter environment that I use to host simple Plotly Dash web apps

Along with hosting the web apps, the Jupyter environment gives me access to a spark cluster and airflow capabilities. Its essentially the calculator of our stack

Any advice on best practice for setting up an initial Kredo project? Should I try and host all the folders on the Jupyter environment or develop my code locally and think about the Jupyter environment as a separate node?

If the Jupyter env is the only place that you can connect to spark etc, then you need to have your project on Jupyter. Otherwise you can always connect to Spark cluster in a local project.

Thanks @noklam, I can connect to spark locally using separate permissions. The airflow and Plotly functionalities are linked to the Jupyter env so guessing I'll need to have the project on Jupyter. I found this in the docs: https://kedro.readthedocs.io/en/stable/tools_integration/ipython.html. Will try working with this

Hey, you got anything?

from this stack it looks like you might not have installed all the kedro dependencies for spark https://kedro.readthedocs.io/en/stable/kedro_project_setup/dependencies.html?highlight=spark#install-dependencies-at-a-group-level

We are using SparkDataSet, I don't think it requires any dependency to be installed

node(
                func=split_data,
                inputs=["model_input_table", "params:model_options"],
                outputs=["X_train", "X_test", "y_train", "y_test"],
                name="split_data_node",
            )

I have a question here: here 'model_input_table' is a data frame from data catalog, while how can I know this is not a string?

Hello does anyone have an example of a Plotly Dash web app integrated with Kedro?

It will always be a dataset. The input/output of nodes are always DataSet, if it is not defined in

catalog.yml

, then it will be just a variable ->

MemoryDataSet

The string is the name of the DataSet

Thanks!

While I am confused again. here "uk_input1": "uk_input1". the first one is a string and the second one is the name of the dataset, right?

Correct, in this case the value is a dataset, the key is just a string literal.

What do you mean by integrate with Kedro? What kind of use case are u looking for?

pipeline_steps.append(node(merging_features_to_main_data, inputs = dict(main_df = 'main_df', feature_1_df = 'feature_df1', feature_2_df = 'feature_df2'),
                                                                           outputs='main_feature_df'))

This is one pipeline, merging 2 feature_df into main_df. but what if I dont know how many feature_dfs in advance, how can I do that?

You can use kwargs

I've got a question regarding the deployment/portability of a pipeline created with Kedro from one machine to another. Idon't really know if this is the correct channel for it but here it goes: Following the corresponding section in the docs and the documentation specific to the Kedro-Docker plugin I've been able to create a Docker image of my Kedro project that I can run with the

kedro docker run

command. This command mounts as volumes the required data/conf/logs folders and then runs kedro inside of it. All good! But now let's say I would like to migrate my finalised project from my development machine to the machine in my lab where we would run the pipeline directly, what are the steps needed to use the dockerized pipeline there? The documentation suggest pushing the built docker image to the registry and then pull it on the "production" env but that doesn't bring with it neither the catalog, the folder structure of the data folder nor the Kedro CLI itself.

kedro docker run

HI Kedro-experts, I have a question for which I can't yet find an answer. In the definition of the saved models (TensorFlowModelDataset) in the catalog, I use the option version=True. The filepath is something along

06_models/trained_models

. What I then see is the following file structure:

06_models/trained_models/2022-05-13T11:12:13/trained_models/

which is versioned, but not super handy. What I would expect is more something in this direction:

06_models/trained_models/2022-05-13T11:12:13/

Do you have an idea how I can get to the second file structure, or is it a "feature" :p Thanks!

Hello, this issue must be simple to fix but at this stage I quit don't understand why kedro can't find the

viz

command (I did

pip install src/requirements.txt

) Kedro version :

0.18.1

Do you have any idea?