Title
#advanced-need-help
datajoely

datajoely

03/07/2022, 1:57 PM
I don't think this is a common enough issue for us to change the
SparkDataSet
implementation itself. We try to mirror the underlying API as much as possible. What I would recommend is the two simple ways to add this to yourself. I think the easiest thing you can do is subclass the
SparkDataSet
and then override the
save()
method - you can copy the implementation from us and simply add those two lines from the screenshot below to the operation. You can see how to create a custom dataset here https://kedro.readthedocs.io/en/stable/07_extend_kedro/03_custom_datasets.html There is also a route to doing this with a hook (https://kedro.readthedocs.io/en/latest/07_extend_kedro/02_hooks.html) but I think the dataset is easier