Title
#beginners-need-help
antony.milne

antony.milne

02/02/2022, 8:00 AM
@User you're in luck (partially, at least) since an example of
pandas.CSVDataSet
over sftp was just added to our docs a couple of days ago - see Example 16 here: https://kedro.readthedocs.io/en/latest/05_data/01_data_catalog.html#using-the-data-catalog-with-the-yaml-api It should be straightforward to combine this with
PartitionedDataSet
by following the example here https://kedro.readthedocs.io/en/latest/kedro.io.PartitionedDataSet.html#kedro.io.PartitionedDataSet. You'd end up with something like this:
my_dataset:
  credentials: cluster_credentials
  path: sftp:///path/to/remote_cluster/
  filename_suffix: .csv
  type: PartitionedDataSet
  dataset:
    type: pandas.CSVDataSet