i see

so we need to use catalog,?

You should get something like that by just adding

versioned: True

to you catog entry

We do that already

If you really wanted to control things you could do some hook stuff to mutate the catalog live bit I wouldn't recommend it

ok thanks

@lbonini https://kedro.readthedocs.io/en/latest/06_nodes_and_pipelines/04_micro_packaging.html

You answered just when I founded 😅

Thanks

@User is there a catalog data types that handles png file or Matplotlib objects?

thank you

it's working great!

Guys, I'm in love with hooks 😄

Seriously, it is so fun to write them hahaha

Hello! I've been trying to find examples of credentials for azure blobs with no success. Anyone have an easy snippet to share? Thanks!

Just occur to me to look on the fsspec implementation, maybe I'l have some luck over there

Hello @User I'm sure we do - what dataset are you trying to use? and which Azure protocol?

I'm aiming for something able to read a parquet dataset on my azure data blob. No preferences yet, all I have is some clue that the

abfs://

protocol with the

pandas.ParquetDataset

might be what I'm looking for.

I believe I just found it. It's on example 15 in the data_catalog docs

Yes something like that! This is the advice I gave last time this came up: https://stackoverflow.com/a/69941391/2010808

I actually found this post and recognised your name on the accepted reply. It does not mention the credentials keys explicitly, if I'm not mistaken. Anyway, It all came to my lack of attention when reading the docs (as it does more frequently than I'd hope for). Thanks for your time, again!

yeah the credentials (whilst we use S3 as an example) is documented here https://kedro.readthedocs.io/en/stable/05_data/01_data_catalog.html#feeding-in-credentials and as you said we just provide this to fsspec so at that point it's best two play around with their API directly

shout if you have any questions!

Hi all, I feel this is a simple question about managing environments but can't manage to find a satisfying answer to it. The closest I've found is [this answer](https://github.com/kedro-org/kedro/discussions/909). In brief, if I want to start using Kedro for all my future projects, should I create only a single conda env

kedro

(with only Kedro installed) that would take care of all future environment needs in the future no matter the project OR should I still create a separate virtual env for each project with only Kedro installed? (I understand the requirements.txt vs requirements.in, but still can't wrap my head around the meta virtuel env 🥵 ) Thanks!

Hello! Would someone knows if it is possible to create PartionedDataset from an Azure location? A friend of mine is trying to do that and she struggles a bit. She's not sure if it is because it is not possible or because her Azure configuration is wrong...

Hello - good question! You're also not alone the python venv (and wider packaging) world is confusing and opaque (obligatory XKCD https://xkcd.com/1987/) I can try and speak to what I do: (1) I create a custom virtual environment for each kedro project (usually with the same name as my project) (2) I use conda for a couple of reasons, but mostly because I basically have this command saved

conda create -n "my-environment" python=3.8 -y

(3) The

build-reqs

and

install

is explained here and is more about making sure you can 'fail fast' and ensure all of your project dependencies play nicely together https://kedro.readthedocs.io/en/stable/04_kedro_project_setup/01_dependencies.html#project-specific-dependencies

hello, I tried to follow the guide to use the wrapper to connect two pipeline but I've got "Failed to map datasets and/or parameters". Is there an hidden requirement?

Are you just running the spaceflights tutorial or is this your own project?