https://kedro.org/ logo
Title
z

Zemeio

04/28/2022, 1:06 PM
Hmm the problem seems to be happening on pillow side, so maybe pillow version?
@datajoely Created a topic here What is happening is in the save from PIL. It does this check, but the file we pass is an s3 file-like object
python
        filename = ""
        open_fp = False
        if isinstance(fp, Path):
            filename = str(fp)
            open_fp = True
        elif isPath(fp):
            filename = fp
            open_fp = True
        elif fp == sys.stdout:
            try:
                fp = sys.stdout.buffer
            except AttributeError:
                pass
        if not filename and hasattr(fp, "name") and isPath(fp.name):
            # only set the name for metadata purposes
            filename = fp.name
This is from
PIL.Image
, and this does not support the s3 file-like that kedro is passing.
d

datajoely

04/28/2022, 1:15 PM
No one else has complained about this to my
z

Zemeio

04/28/2022, 1:15 PM
It might be something from newer versions of PIL, or something like that, but I was able to reproduce the error quite easily.
d

datajoely

04/28/2022, 1:15 PM
so in 0.18.0 we have the requirement
Pillow~=9.0
in 0.17.7 we had
"pillow.ImageDataSet": [
        "Pillow~=9.0; python_version > '3.6'",
        "Pillow~=8.0; python_version == '3.6'",
    ]
z

Zemeio

04/28/2022, 1:16 PM
Pillow==9.1.0, maybe this is the problem?
d

datajoely

04/28/2022, 1:18 PM
try downgrading to a
0.8.x
see if it works?
z

Zemeio

04/28/2022, 1:20 PM
Any restriction to s3fs?
Same error, btw.
d

datajoely

04/28/2022, 1:21 PM
there shouldn't be
this should be delegated to fsspec
I'm still struck by this not being reported, we know lots of users do this
what's your version of fsspec and s3fs?
z

Zemeio

04/28/2022, 1:24 PM
I put the code here: https://github.com/Zemeio/kedrotest that I am using, just hid my personal s3
fsspec==2022.1.0 s3fs==2022.1.0
d

datajoely

04/28/2022, 1:34 PM
So in 0.17.7 we did:
S3FS = "s3fs>=0.3.0, <0.5"
and
fsspec>=2021.4, <=2022.1
z

Zemeio

04/28/2022, 1:35 PM
Pillow is weird, the problem is happening here, maybe I should be posting this there https://github.com/python-pillow/Pillow/blob/2613429e24f94039e7517d0aff1da61d9879a922/src/PIL/Image.py#L2215
It says it accepts file, but it does not
d

datajoely

04/28/2022, 1:35 PM
maybe, just see if it works with the older versions of the libraries above
after that I'm out of ideas :S
z

Zemeio

04/28/2022, 1:37 PM
Same error. I am out of ideas as well haha
I will stop for today, but I won't give up on this. Maybe I will try to raise this on pillow, or try to reproduce the error using just s3fs and pillow to raise to them
That being said, if another kind soul happens to know how to solve this, please let me know = )
Thanks for the huge help datajoely!
@datajoely hey man, I found an easy solution! It is a little counter-intuitive, but if we pass the format as the argument as well it works (since PIL ignores that part of the code). This is the commit that fixed it: https://github.com/Zemeio/kedrotest/commit/e2551f28a7bc2cfeb1859805202e1991d971bc7e