Cool, thanks a lot for the help!

Testing Kedro

Just curious how are other folks scheduling kedro runs? Cron jobs?

Kedro in production

If a node returns a dict with multiple keys, and the dataset expects a dict - how do I avoid the unpacking of the node to the dataset?

Is this available for

pandas.SQLQueryDataSet

? i.e. like using the SSH Tunnel feature (e.g. in DBeaver)

So Kedro uses SQLAlchemy under the hood - I think you'll need to extend / implement some sort of custom dataset which implements the sort of SSH stuff demonstrated here https://gist.github.com/danallison/7217d76d944ea4d8dabd0ba3041ebefc The other alternative is to run Kedro on the remote host like so https://stackoverflow.com/a/31508516/2010808

Currently we don't support SSH SQL connections out of the box, but we do for file based formats via fsspec

I define a list "ids" in globals.yml. I then want to make a Jinja2 loop with this list in catalog.yml how do I do that? I've tried:

{% for id in $ids %}

and` {% for id in ${ids} %}` but none of this works...

Hi @User we currently don't expose jinja variables via

globals.yml

this is currently on the backlog. I think the easiest way to make these variables available is to customise how you register

TemplatedConfigLoader

in

hooks.py

or perhaps subclass and extend

TemplatedConfigLoader

so we currently do this behind this scenes using a library called

anyconfig

the flag

ac_config=True

enables jinja

anyconfig.load

actually takes an extra parameter called

paths

where you can declare

jinja2

templates

so in short - not super easy today, but it's possible if you define your own config loader

Great and thanks for the fast reply again. Perhaps this should be added to the docs? (That you can only print stuff from the globals.yml with Jinja2, not apply logic to it etc.)

We're currently designing a whole configuration overhaul so it's part of wider piece of work, we do have a note on the jinja stuff atm that say it's a bit of a hack (IIRC)

within the team there is a strong, healthy debate on the topic on whether config should be strictly declarative or not

What is exit code

1073741819 (0xC0000005)

? It happens when using

ThreadRunner

. Using

SequentialRunner

is fine. Is it because too many processes are trying to access the same dataset simultaneously?

Are you using

ThreadRunner

with Spark or Dask? For python pipelines you should be using

ParallelRunner

I'm using

ThreadRunner

with Pandas. I tried using

ParallelRunner

a while ago and got an error about lambda functions.

Can you try with

ParallelRunner

here? We can help you work through the errors, due to the way Python works you don't get true concurrency with threads and we have to use multi-processing.

Some things can't be split into different processes - so it isn't a 100% guaranteed to work

In some cases it's good to break your pipeline into different parts and run them indepdently

using the

&&

operator in your CLI command

Using

ParallelRunner

leads to

TypeError: cannot pickle 'module' object

.

Any idea what type of object it is when it fails?

as type 'module' is quite confusing

oh it looks like it fails at the point the catalog is being created - are you using any custom datasets?

(it is getting quite late here in London, so I'll probably pick up your response in the morning)

ParallelRunner error

I see that the anyconfig.load also has the parameter

ac_context=globals

which would expose the globals to the jinja2, but it also seams that the globals.yml and catalog.yml is loaded together in arbitrary order so that the globals are not known at the time of the jinja2 rendering. I suspect (as mentioned) that

TemplatedConfigLoader

needs a major overhaul to change this