So I'm not sure how to tell Kedro what the S3 endpoint is when not using credentials.yml as this would usually be passed in as endpoint_url along with a key/secret. In my case, the key and secret are being set as AWS\_ environment variables. How can I effectively tell s3fs what the endpoint is in this situation?

Are u using sqlitestore?

Yes that's it.

src/settings.py

has:

SESSION_STORE_ARGS = {"path": str(Path(__file__).parents[2] / "data")}

but there is a

data

under

home/kedro

Is this a path on s3 as well?

No but I'm not specifying it with an

s3://bucketname/

prefix either so it should write on the pod volume

If the folder doesn't exist it may not create the folder automatically, that was a problem a while ago, I can't remember if it's still an issue or not.

I thought there would be a data dir in $HOME on the image by virtue of the Dockerfile, but it's not there. So that is likely the problem...

I think the data/ is mean to be mount to some volume

/home/kedro

is set up in the Dockerfile created by kedro-kubeflow. The space tutorial seems like it is trying to create the path to the db file as

/home/kedro/data

. This will end up on a default volume in the pod since no volume is specified.

I see, the kedro-kubeflow plugin is maintained by our community. Maybe @em-pe @marrrcin knows better.