Understanding data catalog datasets
# beginners-need-helpr
rohan_ahire
09/17/2022, 4:35 PMUnderstanding data catalog datasets
rohan_ahire
09/17/2022, 4:35 PMSparkHiveDataSet - Does this read from hive and load data in spark dataframe or does it write a spark dataframe to hive table? Or does it do both? I am getting confused with the data catalog definitions for not just hive but any type. I don't understand what it is reading from or what is it writing to.
d
datajoely
09/19/2022, 7:09 PMIt's a very thin wrapper over the native spark functionality
https://github.com/quantumblacklabs/kedro/blob/main/kedro/extras/datasets/spark/spark_hive_dataset.py#L153
r
rohan_ahire
09/19/2022, 8:00 PMthanks, this is helpful
d
datajoely
09/19/2022, 8:00 PMPleasure