I've had a go at making this a plug-in it's very early days https://github.com/datajoely/kedro-rich

does Kedro support memoization

You can use the CachedDataSet for this in many cases

A kedro run is ephemeral, a past run can only read persisted data

Have I correctly understood Kedro to be aimed at being a fairly generic computational graph, akin to Dask in that respect? It's from that perspective that I am asking; you could conceivably skip entire sub-graphs if ie an @idempotent decorator was used.

I think I'd frame kedro very differently - it's a framework for building robust modular code, the execution part can be pure python or delegated to something optimised like dask or spark

What do you think about node(..., final_output='other_node', memoize=True)

Within the same session I'm pretty sure that's what the cached dataset does. In general the nodes should have no knowledge of IO

Thanks for the clarification

Perhaps we could efficiently return hashes to nodes that would allow them to divert computation

kedroid: "I have automatically compiled 13 shortcuts"

Have you seen PsyNeuLink's new optimization path?

It's called Distill; it lowers your code to LLVM IR. http://www.cs.yale.edu/homes/abhishek/jvesely-cgo22.pdf

I gave two #AI / #ML / #DataScience talks in a span of two weeks in March: - AI/ML/DL/RL comparisons: https://bit.ly/ai-ml-dl-rl-comparisons - Studying the limitations of statistical measurements: https://bit.ly/studying-stats-limits Enjoy and share the knowledge

Waiting for the 0.18.0 release announcement 👀

Just received an email from Prefect: Prefect 2.0 is here: https://orion-docs.prefect.io/

Yeah cool stuff about having a ray runner right

Nice

Let me check

@hereWe're making the changes official. We have moved to Slack (#1) and you'll find 500 members in that channel supporting each other! Check #778996598880862208 for the rationale and remember to sign up for swag (#2). We're also running a survey to gauge interest in a "Software Engineering for Data Scientists and Data Engineers" course that we want to write and run next year so please fill out this survey if this is something you want (#3). We'll see you on Slack ❤️ Links #1 Join Slack: https://slack.kedro.org/ #2 Get swag when we ship our new branding: https://www.surveys.online/jfe/form/SV_8jfTn7SQDcUiN5c #3 Software Engineering for Data Scientists and Data Engineers survey: https://www.surveys.online/jfe/form/SV_37WxRUy5xkC0xkW