Hi, I am getting a "DLL Load failed while importing win32api" error when attempting to utilize a kedro jupyter notebook kernal. I currently have winpy32 301 installed from the kedro install process, is there an version that you all recommend using that may be more stable?

@User can you run jupyter outside of kedro? i.e.

jupyter notebook

or

jupyter lab

@User I modified our teams template to have a good __main__file such that running

%run -m project_name

gets you all the same things you get from running

kedro ipython

might be worth looking into that.

Hi @User we're just looking into fixing issue and am unable to to reproduce. What version of Kedro were you running?

Hello guys! I got a question related to the kedro-airflow plugin but I'm not sure if this is the place to ask for...my company has Airflow installed in an internal server using Docker. Kedro docs says that we should install our kedro packaged project inside the airflow environment, but that means we should change the Airflow Docker Image every time we want to install a new package, module, etc. Is there a way to manage that scenario without modifying the Airflow DockerImage/Compose file??

Airflow question

Is it possible to configure the default dataset used by kedro? Is there a way to make

kedro catalog create

generate something other than

MemoryDataSets

?

Found it, its hard coded. https://github.com/quantumblacklabs/kedro/blob/master/kedro/framework/cli/catalog.py#L196

What would you like to generate instead?

parquet

I am making an issue, now if you want to join in. or maybe I can do a discord call.

twitch.tv/waylonwalker

I'm on a call already! so only able to respond here

so I think we're unlikely to change the default CLI command for this, but this is a good example of where you could override it for your own purposes

I don't want to change the cli command.

ah please create a GH issue 🙂

https://github.com/quantumblacklabs/kedro/blob/master/kedro/framework/cli/catalog.py#L196 I want to not hard code line 196, and pull defaults from settings.py. It would be a trivial change to switch the type over to something configurable, but being able to configure filepath per dataset is what is not trivial. I'll brainstorm a solution in the readme and you can tell me how far off base I am.

I would personally think doing something like

kedro catalog generate --type pandas.ParquetDataSet

would be a neat way of doing this

however, the core team are quite reluctant to make changes to the CLI overall

https://github.com/quantumblacklabs/kedro/issues/849 hashed out a quick idea. Let me know what you think.

No change to the cli at all

Thanks!

I think this would really improve the onramp of new users onto a kedro team, as the catalog could be mostly automated.

So with my PM hat on this feels like a symptom of our YAML config overhead. This is a key research/development focus for the second half of the year so stay tuned

hey guys

where is a best place to put reusable personal content in kedro structure?

i am leaving in pipelines level or extras package.. but id like to know how do you guys deal with that

We put utility functions in a folder under src

nice.. thank you 🙂

Honestly some side projects I've done with no kedro template whatsoever, starting with a blank file, I hand rolled an auto catalog in ~10 lines (probably only 3 lines of execution). I feels so fast to just ignore the catalog. Honestly I do not care where the files go, I just care that they are of the right type, and rarely (outside of the raw layer) am I fussing with any settings on them. Id rather just say dump all dataframes into parquet, in the data folder, and name the file just like the dataset, anything else make it a pickle. I can't remember ever hand loading a file, so I really don't care about the structure, I just want there to be enough there that the option is still there if I neede it.