Hello folks, quick question on PartionedDatasets. I see that at load time I can access partitions at will by using the partition_id (basically the rest of the path where it is saved). Is there a way I can achieve the same incremental effect at writing time? Meaning at write time write partitions incrementally in such a way that I can relieve memory when the node is running? I think this can only be done by multiple nodes but I'm not sure.

Hello folks quick question on

Hi folks, I am trying to make a tiny PR on the documentation for a missing import in some code, literally

from pluggy import PluginManager

. If I follow the guidelines (which I partly did), I need to run

make build-docs

to check the result. When checking the content of the script, it turns out that it installs then entire test environment, including

pyspark

(300 MB) and

tensorflow

(500MB). This seems to me a bit extreme. Did I do something wrong?

You didn't do anything wrong. Sphinx needs to do that to traverse the imports to build the API section. We're actually in the process of speersting kedro datasets into its own package so this will be cleaner in the future

Hey guys, do you guys know a way or have any suggestion in order to set a DateTime param that can be used by many nodes? Like, somehow make a param inside the .yml pick the actual date when each node makes a call for it?

You can provide your own globals dict as a config loader arg in settings.py

Uhhh, does any doc mention any usage such as this one?

There are tutorials for how to configure settings.py in the docs

oh nice, thanks for that Joel

Im not sure there is this particular example but you can provide a dict of extra variables to the TemplatedConfigLoader class

uhhh, I'll be looking into that...

Yeah the idea is that when a node makes a call, he uses a datetime.now()

Well in this case you would get one date time for all nides

You could possibly achieve what you're trying to do with a before node run hook

There are docs for that too

uhh, maybe it could be creating a specific node that does that datetime.now() and uses it on every pipeline that would need it, what do you think?

oh nice one

Hi, fellows, I am still trying to make my one-line PR on the documentation — 😅 — but failing at building the docs. In both CircleCI and locally, I stumbled upon the following error

/home/circleci/project/docs/build/kedro.rst.rst:24:autosummary: stub file not found 'kedro.kedro.config'. Check your autosummary_generate setting.

Did anyone get the same problem?

I think this isn't just you, I saw it earlier last week. Raise the PR and someone from the team will look at it Monday. Thanks for the contribution!

Thanks @datajoely , I will let it rest a bit then!

is there a way to give datasets tags? I've got a bunch and I'm drowning in the complexity trying to remember what each one is for

guess not: https://github.com/kedro-org/kedro/issues/1076

Comment on that issue if you want that feature!

Is there any way to load the context from an active kedro session in 0.18.X? In the version 0.17.X there was the method: get_current_session inside kedro.framework.session.session. In our code we are using the kedro context to run a pipeline with different parameters in a loop for.

So we don't typically like this pattern since it makes reproducibility difficult. We removed the current session finctionality since now 1 session = 1 run. So you can create a new one and it will be fine

Perfect, thank you for your answer, also does this imply that the catalog from the session that is using can't be accessed at runtime without creating a new session?

Yes, you can use lifecycle hooks to mutate an existing session during a run

I made some syntactic sugar that makes the following possible in jupyter: It reloads the session with the new overrides and starts the pipeline. I can send you the code if you want.

%pipe tab_adult.ingest
for e1 in (0.3, 0.5, 0.9, 1.5):
    pipe("tab_adult.privbayes.synth", {"alg.e1": e1})

It also converts "alg.e1": e1 to {alg: {e1: e1}} for you

This is super interesting! Would you mind doing a show and tell on GitHub discussions?