Good to know, this suggests we can do a better job helping users in your position, I'll make a note. also as of today, we've moved out of quantumblacklabs into kedro-org as part of our move to the Linux foundation!

Hey folks! Great tool you guys have over here!

I just have a quick question. I'm working on a helper class to test is the dataframes that are inputs and outputs of a node/pipeline follow a given shape. Now, I'm only concerned for now about inputs/outputs that are dataframes. Is thera a way of getting the type of an IO without loading it? Either from the catalog or from the node?

I can see that if it comes from a parameter, kedro will append "params__" in front of it. Is this the most reliable way?

I think kedro hooks are the right call here https://kedro.readthedocs.io/en/latest/07_extend_kedro/02_hooks.html

You can also use pandera for this without hooks if you’re only working with pandas stuff

Pandera looks really promissing. I didn't want to use great expectations since it seems a bit overkill for my use case.

Yeah users have had good success with it

Only thing I’ll add is that if you use the decorator approach it doesn’t play nicely with running kedro in parallel mode

AttributeError: Object ParquetDataSet cannot be loaded from kedro.extras.datasets.pandas https://stackoverflow.com/questions/70719080/attributeerror-object-parquetdataset-cannot-be-loaded-from-kedro-extras-dataset

How to save kedro dataset in azure and still have it in memory https://stackoverflow.com/questions/70757448/how-to-save-kedro-dataset-in-azure-and-still-have-it-in-memory

I had a quick question about what kedro does under the hood to manage data. I love the abstraction of accessing data, but I'm curious about storage space usage. Let's say I add an s3 bucket containing imagenet to my project data catalog. When I run a pipeline that uses that imagenet dataset, does it cache the s3 bucket data locally? Or does it dynamically query s3 to pull bits and pieces as needed? Or a mixture of both?

We delegate it to the dataset so a pandas dataset will copy locally, a spark or dask dataset will work on that cluster

In general our datasets are thin layers on top of the original reader/writer

Kedro pipeline on partitioned data https://stackoverflow.com/questions/70815517/kedro-pipeline-on-partitioned-data

Hey I was wondering if there is a fix for the

graphql

error between kedro and wandb libs ? It seems that it's still complicated if I read this : https://github.com/wandb/client/issues/2813#issuecomment-1019300464 The only way is to have 2 envs ? One for the project with wandb and another one with only kedroviz ? EDIT: It seems to be complicated even with 2 envs, I uninstall

wandb

if I want to use

kedro-viz

for the moment 🙂

It’s high priority for the new week will come back to you with an update

Wow, this is indeed complicated and incredibly annoying 😬 A classic example of Python dependency hell, made even worse by (a) the fact that W&B are vendoring packages and (b) `pkg_resources`'s overzealous checking of dependency versions, which means it's surprisingly not possible to run any entry point (like

kedro viz

) unless you meet the versions specified in the requirements 😮 This took a long time to figure out, but I think might work. Note the order of commands is important: 1.

pip install --upgrade git+git://github.com/wandb/client.git@task/graphql#egg=wandb

2.

pip install kedro-viz

Check which versions you've got with

pip freeze | grep -e graphql -e gql -e kedro-viz

. This should give:

gql==3.0.0
graphql-core==3.1.7
kedro-viz==4.2.0
strawberry-graphql==0.87.3

Tagging @User - if it works, let's comment on that wandb issue for visibility

Also just for the record I thought of a way to get around (b):

python -c "from kedro_viz.launchers.cli import viz; viz()"

So if the above doesn't work then there may still be hope...

Thanks @User 🙂

I will try after work, thanks a lot !

@User if it doesn't work, you can downgrade to

kedro-viz==3.16.0

that's the last version before we require graphql

Even though this is due to wandb using a local version of old graphql which can't do much about, maybe there is something to do around allowing user to keep using Kedro even if they can't install kedro-viz

We can potentially move dependencies to optional requirements on a per feature basis. At the bare minimum, it only has the flowchart with the minimum amount of dependency. Could be a good fallback for weird edge cases like this one (or the windows issue on github)

This trick doesn't work even if I got the same packages versions 🙂 I still got an error

bash
kedro.framework.cli.utils.KedroCliError: cannot import name 'IntrospectionQuery' from 'graphql' (/home/chainyo/miniconda3/envs/make-me-rich/lib/python3.8/site-packages/graphql/__init__.py)

I can send the full error trace if needed

This is working GREAT ! I can run pipelines with

wandb

logger and watch the full pipeline in the same env !

It will run like that, I will add this hint to the issue linked above

Amazing 💪 !