antony.milne
02/02/2022, 8:00 AMpandas.CSVDataSet
over sftp was just added to our docs a couple of days ago - see Example 16 here: https://kedro.readthedocs.io/en/latest/05_data/01_data_catalog.html#using-the-data-catalog-with-the-yaml-api
It should be straightforward to combine this with PartitionedDataSet
by following the example here https://kedro.readthedocs.io/en/latest/kedro.io.PartitionedDataSet.html#kedro.io.PartitionedDataSet. You'd end up with something like this:
my_dataset:
credentials: cluster_credentials
path: sftp:///path/to/remote_cluster/
filename_suffix: .csv
type: PartitionedDataSet
dataset:
type: pandas.CSVDataSet