A colleague and I looked into this a while back, and created a POC based on spark-streaming. You can check it out on https://github.com/deepyaman/kedro-streaming/blob/develop/conf/base/catalog.yml#L11. No guarantees it works with latest Kedro. 🙂 We also explored using

faust

as a backend for this (and being more Python-native), but

faust

isn't really maintained anymore (and was lacking some other necessary functionality, like joining streams). There's an overall question of how to better support streaming workflows with Kedro, as Kedro is notoriously batch-oriented.