Quite confused about code management in a modular pipelines Kedro project. On one side, to support micro-packaging I should put all my pipeline code within that pipeline's folder. On the other side, if I have nodes/functions/classes that I want to re-use across pipelines, I'd want to place their code in the

<my_project>/src/<my_project>

level - but that will prevent me from using micro-packaging (as the docs states). So what is the recommended way to work with modular pipelines that share nodes/functions/classes?

Hello I have one question regarding using kedro to train a tf/keras CNN model: on the keras documentation( https://keras.io/api/data_loading/image/#imagedatasetfromdirectory-function ), it is written to use tf.keras.utils.image_dataset_from_directory to load the images. As I am not using 'inferred' labels (I did not manage to sort image by label in my pipeline in sub folder with a partitioned dataset) I am trying to use os.walk to generate the proper label list. However I am failing with os.walk to get the proper list of files. Would someone have an example how to perform this please?

How to generate kedro pipelines automatically (like DataEngineerOne does)? https://stackoverflow.com/questions/74178198/how-to-generate-kedro-pipelines-automatically-like-dataengineerone-does

Just curious, is there any good example/best practices on how to test the kedro pipeline code with mock data, specifically integration tests ?

@mrjpz99 this is a previous thread on the topic https://discord.com/channels/778216384475693066/931533715291648041

this one too https://discord.com/channels/778216384475693066/778998585454755870/864551888137486336

We're in the final month of supporting our Discord server. We're all moving to Slack on the 30th of November. Check our previous announcement for the rationale for why we're doing this and remember to sign up for Kedro swag. About 400 people are in the new Slack workspace ♥️. Links #1 Join Slack: https://join.slack.com/t/kedro-org/shared_invite/zt-1eicp0iw6-nkBvDlfAYb1AUJV7DgBIvw #2 Get swag: https://www.surveys.online/jfe/form/SV_8jfTn7SQDcUiN5c

@datajoely Is there a way to tell the DAG build to wait on reading from a table until mid-way through a pipeline (rather than at the beginning). We are trying to minimize ram usage and so part of the pipeline is running in SQL utilizing a temp table that we write then read back in once joined with a ton of other stuff.

Hi all, I am using the IncrementalDataSet when converting pdf's to png's. I use this since I have lots of them and only want to convert the newly added pdf's. However, I am running into some problems with this.

CHECKPOINT

is using the latest partition_id from the previous run. Lets say that the CHECKPOINT value is Key456. When I add Key123 it will come in front of the CHECKPOINT since S3 is sorting it alphabetically. So my question is, why is the CHECKPOINT just the latest partition_id and not all the partition_ids? So that when there are new partition_ids it is compared with that list? And is there maybe a fix for this? Or just for me? Would like to hear from you! Regards, Nick

Hi there! Has anyone ever dealt with an S3 directory with an empty name while using PartitionedDataSets before? Something like what's shown below.

s3://edhenry//data_folder

I receive a

No partitions found

error when attempting to

catalog.load()

. I've tried a few things with manipulating the path string, etc. but nothing seems to work. Any ideas?

Hi all, can somebody point me how to get current running env ?

Hooks!

Are there files in that directory?

You'd have to come up with a custom runner I think

do you have maybe some example?

The sorting is down on timestamps, S3 sort shouldn't apply

There is a whole page called 'hooks' in the docs! But note you don't have access to the env name within the node, by design these are functionally pure.

If you really want to access it in both, you can send the env using the 'KEDRO_ENV' variable and access it that way

Yeah that's what I thought. Logical ofcourse. It is annoying that S3 doesn't sort it incremental. The sorting is done as it is read from S3. But I found a way around it. Thanks for the reply!

can you point me little further about KEDRO_ENV (where to put and how to access). Use case fpr me: i have multiple env which i changing form cli (--env) and i extend existing yaml dataset but i need current env example of dataset

Oh you shouldn't be doing this! Kedro does it for you! You simply have the same file in two folders that's match the env name within conf/

It will detect the right version then

The dataset shouldn't be doing this

You can also pass in yaml config using

kedro run --config

this is all on the docs

i will explain a little more just show you what i want to achieve: jenkins is running testing with diff env and also diff params (kedro run --env= --params "") dataset is actually yaml template and needs to be filled with deff params for diff env

So you can do that very easily by updating the parameters on a file level

The CLI is supposed to be ultimate override so there is no concept of env when it comes to CLI params

i have parameters set on file level (default) and form CLI i am overriding but i want to do in side custom dataset and prepare to avoid doing that on pipline or node

is maybe posible form hook to override(inject) some on current run setting and get information which env i am using(after_context_created)?