i see. thank you. probably I need change our system that is related to ,

This isnt speaking for the project, but I wouldn't be against changing this to pure JSON

Hello! @User Could you help me with this one? I'm trying to use environment variables in mlflow.yml. I need to set

MLFLOW_TRACKING_INSECURE_TLS

to

true

. How am I supposed to declare it? It works only if I export in the terminal and executes

kedro run

Environment variables in mlflow

@User is there a way to specify params in config file? https://kedro.readthedocs.io/en/latest/04_kedro_project_setup/02_configuration.html#configure-kedro-run-arguments

Yes! --config some.yaml

thanks! having comma value param in yaml file seems to work

Looking for how others would do this. If you have a number of catalog entries that all need merged into one table on the same key, how do you do it. I've always made a node for each dataset separately, but today I read a pr that passed them all in as inputs and the function signature was

def joiner(left, *rights)

, I can't tell if this is genious or too much magic. Would love some thoughts on how others would review this PR.

I encourage this pattern on the modular spaceflights demo

I think it's a good idea

You're still being explicit where it matters

Hi I am very interested in kedro. However I am not interested in the full workflow as described in Spaceflight turtorial. At my work we have an in house built python DAG framework based on dask and I would like to replace it with kedro. What I would like to do is create pipelines dynamically, and use mostly memory datasets in the catalogue. I fear for my use case, I may not be able to use kedro and kedro-viz how I would like. Does the pipelining part of kedro stand alone without the need setting up the kedro template, setting up the data folders and packaging of project?

Using Kedro in your workflow

Let's say I have a partitioned dataset that's quite large, 0.5TB or so. Can I expect

kedro run

to work as normal, or is there anything special I need to do or configure before executing to make sure I don't run out of memory?

I think you can assume it works - but I would also suggest if you're hitting that sort of data size you may want to explore Spark or Dask as an execution engine

we have docs on both if you're interested

Thanks, I'll check that out!

The latest dask docs are on the

latest

docs branch

I don't know anything about Spark or Dask yet, so the docs may cover the following question: Let's say I have 1000 files, and there is some anomaly in file 926 that causes an error. So I fix the file itself or my nodes, and run again. How can I start from 926 and not waste time computing results I've already acquired? I've never really worked with a dataset large enough to warrant asking this question before, haha.

How to manage large partitioned datasets

Is

kedro new

supposed to create a

cli.py

file in a new project by default?

so not anymore - it's been removed

but if you place one in your project directory it will be picked up

That makes sense. I was looking into customizing my cli project commands and the

cli.py

file is mentioned in the relevant docs. Is there a sample I can use to guide myself?

Thank you!

Hi everybody, I have an issue upgrading kedro to 0.17.7. I am on windows 10, python 3.8.10 installed via pip and previously i used kedro 0.17.4 which is working fine. I upgraded kedro via

pip install kedro -U

and created a new kedro project

kedro new

and created a dedicated environment using

venv

. After activating the new environment and running

kedro build-reqs

the process fails because the restrictions for jupyter_client overlap and it cannot find a version that satisfies the requirements: > Could not find a version that matches jupyter_client<7.0,=4.1,>=5.1,>=5.3.4,>=6.1.1,>=7.0.0 (from -r C:\Users\Jan\Documents\myprojects\analytics\src\requirements.in (line 8)) Any idea how to resolve this?

funny enough,

pip install -r src/requirements.txt

seems to work fine. Only

kedro build-reqs

keeps failing...

Hi @User the

jupyter_client

library changed something two days ago which is causing this . We've merged a fix here https://github.com/kedro-org/kedro/pull/1322