Can items in

global.yml

refer to other items in the same file for templating?

date: 2022-02-02
somefile: filename_${date}.txt

I'm not actually sure - no one has ever asked that before.

after looking at the code - I don't thinks so, the files which match

*globals.yml

first are consumed and then the templating is applied to the other files

Using environment vars is a common way to get round this sort of thing. Set the var outside of Kedro and then tweak your

register_config_loader

method in

hooks.py

python
globals_pattern="*globals.yml",
        globals_dict={
            k: v for k, v in os.environ 
            if k.startswith("date")
        },
    )

Hi team, I'm trying to pass Parameters at runtime. https://kedro.readthedocs.io/en/latest/04_kedro_project_setup/02_configuration.html#specify-parameters-at-runtime One of my params have comma in value for example, kedro run --params "param1=hi, param2=a,b" is there a way that I can have comma in the value of the param such as escape sequence or so?

I'm not actually sure - can you try putting a \ before it? Otherwise there is a chance this isn't possible

kedro run --params "param1=hi, param2=a\\,b"

is not working either

Yeah can you use a different delimiter?

I think it's fair to say commas are reserved

i see. thank you. probably I need change our system that is related to ,

This isnt speaking for the project, but I wouldn't be against changing this to pure JSON

Hello! @User Could you help me with this one? I'm trying to use environment variables in mlflow.yml. I need to set

MLFLOW_TRACKING_INSECURE_TLS

to

true

. How am I supposed to declare it? It works only if I export in the terminal and executes

kedro run

Environment variables in mlflow

@User is there a way to specify params in config file? https://kedro.readthedocs.io/en/latest/04_kedro_project_setup/02_configuration.html#configure-kedro-run-arguments

Yes! --config some.yaml

thanks! having comma value param in yaml file seems to work

Looking for how others would do this. If you have a number of catalog entries that all need merged into one table on the same key, how do you do it. I've always made a node for each dataset separately, but today I read a pr that passed them all in as inputs and the function signature was

def joiner(left, *rights)

, I can't tell if this is genious or too much magic. Would love some thoughts on how others would review this PR.

I encourage this pattern on the modular spaceflights demo

I think it's a good idea

You're still being explicit where it matters

Hi I am very interested in kedro. However I am not interested in the full workflow as described in Spaceflight turtorial. At my work we have an in house built python DAG framework based on dask and I would like to replace it with kedro. What I would like to do is create pipelines dynamically, and use mostly memory datasets in the catalogue. I fear for my use case, I may not be able to use kedro and kedro-viz how I would like. Does the pipelining part of kedro stand alone without the need setting up the kedro template, setting up the data folders and packaging of project?

Using Kedro in your workflow

Let's say I have a partitioned dataset that's quite large, 0.5TB or so. Can I expect

kedro run

to work as normal, or is there anything special I need to do or configure before executing to make sure I don't run out of memory?

I think you can assume it works - but I would also suggest if you're hitting that sort of data size you may want to explore Spark or Dask as an execution engine

we have docs on both if you're interested

Thanks, I'll check that out!

The latest dask docs are on the

latest

docs branch