I hadn't made that clear

Is it possible to set global key value pairs on kedro inside a yaml file which is available to all the pipelines in a project? For example, I have a date filter that I am using and that value is repetitively accessed in multiple dataframes. Is this possible?

I'm not sure what you're looking for. Do you want the

parameter

object?

i want the start date and end date values to be available to all the pipelines

That's just possible by passing a unique key in the parameters.yml file. RIght?

so you have to add your

param:start_date

and

param:end_date

as inputs to your nodes, but yes

Sorry, I didn't think through. Thanks for the response @datajoely

Is there any support for SAP HANA for connecting to the datasets?

Sure is - install this SQL alchemy dialect and use any of the Pandas SQL datasets https://github.com/SAP/sqlalchemy-hana

You're a lifesaver

Hey guys, I just want to ask it here first, just to see if this is a known problem, otherwise I will gather more evidence before asking properly, but I am doing a project that requires processing images, and I am using pillow for it. The problem is, when I try to use s3 for my stuff, kedro is not passing the file extension to pillow which is causing it to fail. This only happens when I use s3, if I use a regular path in my computer it works. Changing the s3fs version changed the result, but also failed ultimately. Is this a known issue?

How are you using Pillow? Are you using the

kedro.extras.datasets.pillow.ImageDataSet

?

wrapped in

PartitionedDataSet

?

Yes

Exactly, I forgot to mention the partitioned, but yes

could you paste a snippet of the catalog entry?

I can't because it is from my company, I will reproduce it in my personal pc and pass you. I was setting the extension

My gut feeling is that this should work - so anonymised YAML would help

(reproducing, giving an example with enough information is what I meant by asking properly).

I will give you the anonymised later Today then, I have to go for now.

You don't need to reproduce the error, just the YAML structure would be helpful

(my work pc is shutdown, which is why I can't give it to you now)

👍

@datajoely Some versions:

s3fs==2022.1.0 
boto3==1.20.24 
aiobotocore==2.1.2

And the catalog:

yaml
my.sample.image.int: 
    type: PartitionedDataSet 
    dataset: kedro.extras.datasets.pillow.ImageDataSet 
    path: ${my.bucket}/${folder.int}/sample-images/ 
    filename_suffix: ".jpg"

Path something like: s3://mybucket/02_intermediate/sample-images

and you get no error?

I actually think you're not configuring that correctly

I get an error when trying to save, saying that the extension is unknown (the extension is being sent empty to pillow, for some reason)

don't you need two types?