this one too https://discord.com/channels/778216384475693066/778998585454755870/864551888137486336

We're in the final month of supporting our Discord server. We're all moving to Slack on the 30th of November. Check our previous announcement for the rationale for why we're doing this and remember to sign up for Kedro swag. About 400 people are in the new Slack workspace ♥️. Links #1 Join Slack: https://join.slack.com/t/kedro-org/shared_invite/zt-1eicp0iw6-nkBvDlfAYb1AUJV7DgBIvw #2 Get swag: https://www.surveys.online/jfe/form/SV_8jfTn7SQDcUiN5c

@datajoely Is there a way to tell the DAG build to wait on reading from a table until mid-way through a pipeline (rather than at the beginning). We are trying to minimize ram usage and so part of the pipeline is running in SQL utilizing a temp table that we write then read back in once joined with a ton of other stuff.

Hi all, I am using the IncrementalDataSet when converting pdf's to png's. I use this since I have lots of them and only want to convert the newly added pdf's. However, I am running into some problems with this.

CHECKPOINT

is using the latest partition_id from the previous run. Lets say that the CHECKPOINT value is Key456. When I add Key123 it will come in front of the CHECKPOINT since S3 is sorting it alphabetically. So my question is, why is the CHECKPOINT just the latest partition_id and not all the partition_ids? So that when there are new partition_ids it is compared with that list? And is there maybe a fix for this? Or just for me? Would like to hear from you! Regards, Nick

Hi there! Has anyone ever dealt with an S3 directory with an empty name while using PartitionedDataSets before? Something like what's shown below.

s3://edhenry//data_folder

I receive a

No partitions found

error when attempting to

catalog.load()

. I've tried a few things with manipulating the path string, etc. but nothing seems to work. Any ideas?

Hi all, can somebody point me how to get current running env ?

Hooks!

Are there files in that directory?

You'd have to come up with a custom runner I think

do you have maybe some example?

The sorting is down on timestamps, S3 sort shouldn't apply

There is a whole page called 'hooks' in the docs! But note you don't have access to the env name within the node, by design these are functionally pure.

If you really want to access it in both, you can send the env using the 'KEDRO_ENV' variable and access it that way

Yeah that's what I thought. Logical ofcourse. It is annoying that S3 doesn't sort it incremental. The sorting is done as it is read from S3. But I found a way around it. Thanks for the reply!

can you point me little further about KEDRO_ENV (where to put and how to access). Use case fpr me: i have multiple env which i changing form cli (--env) and i extend existing yaml dataset but i need current env example of dataset

Oh you shouldn't be doing this! Kedro does it for you! You simply have the same file in two folders that's match the env name within conf/

It will detect the right version then

The dataset shouldn't be doing this

You can also pass in yaml config using

kedro run --config

this is all on the docs

i will explain a little more just show you what i want to achieve: jenkins is running testing with diff env and also diff params (kedro run --env= --params "") dataset is actually yaml template and needs to be filled with deff params for diff env

So you can do that very easily by updating the parameters on a file level

The CLI is supposed to be ultimate override so there is no concept of env when it comes to CLI params

i have parameters set on file level (default) and form CLI i am overriding but i want to do in side custom dataset and prepare to avoid doing that on pipline or node

is maybe posible form hook to override(inject) some on current run setting and get information which env i am using(after_context_created)?

maybe i found solution

and information is available settings.CONTEXT_CLASS.env

@datajoely is it ok approach

It feels very out of bounds unfortunately - the env separation has been designed carefully against the 12factor app and I fear this is quite a fragile solution to a problem I think is solved natively https://12factor.net/

I'm still not sure why the dataset needs to be dynamic? Can't you simply declare in separate env folders?

i can use one dataset as template on multiple env and reduced creation multiple dataset files