@User you're in luck (partially, at least) since an example of

pandas.CSVDataSet

over sftp was just added to our docs a couple of days ago - see Example 16 here: https://kedro.readthedocs.io/en/latest/05_data/01_data_catalog.html#using-the-data-catalog-with-the-yaml-api It should be straightforward to combine this with

PartitionedDataSet

by following the example here https://kedro.readthedocs.io/en/latest/kedro.io.PartitionedDataSet.html#kedro.io.PartitionedDataSet. You'd end up with something like this:

my_dataset:
  credentials: cluster_credentials
  path: sftp:///path/to/remote_cluster/
  filename_suffix: .csv
  type: PartitionedDataSet
  dataset:
    type: pandas.CSVDataSet