Understanding data catalog datasets

Question

rohan_ahire · Answer

SparkHiveDataSet Does this read from hive and load data in spark dataframe or does it write a spark dataframe to hive table Or does it do both I am getting confused with the data catalog definitions for not just hive but any type I don t understand what it is reading from or what is it writing to

datajoely · Answer

It s a very thin wrapper over the native spark functionality https github com quantumblacklabs kedro blob main kedro extras datasets spark spark hive dataset py L153

rohan_ahire · Answer

thanks this is helpful

datajoely · Answer

Pleasure