ende
09/27/2021, 11:06 PMAbstractDataSet
examples in https://kedro.readthedocs.io/en/stable/07_extend_kedro/03_custom_datasets.html are actually using code from the AbstractVersionedDataSet
?from pathlib import PurePosixPath
from typing import Any, Dict
from kedro.io.core import (
AbstractVersionedDataSet,
get_filepath_str,
get_protocol_and_path,
)
import fsspec
import numpy as np
from PIL import Image
class ImageDataSet(AbstractVersionedDataSet):
"""``ImageDataSet`` loads / save image data from a given filepath as `numpy` array using Pillow.
Example:
::
>>> ImageDataSet(filepath='/img/file/path.png')
"""
def __init__(self, filepath: str):
"""Creates a new instance of ImageDataSet to load / save image data for given filepath.
Args:
filepath: The location of the image file to load / save data.
"""
# parse the path and protocol (e.g. file, http, s3, etc.)
protocol, path = get_protocol_and_path(filepath)
self._protocol = protocol
self._filepath = PurePosixPath(path)
self._fs = fsspec.filesystem(self._protocol)
def _load(self) -> np.ndarray:
"""Loads data from the image file.
Returns:
Data from the image file as a numpy array
"""
# using get_filepath_str ensures that the protocol and path are appended correctly for different filesystems
load_path = get_filepath_str(self._get_load_path(), self._protocol)
...
This is supposed to just be a complete example of AbstractDataSet_get_load_path
is not inherited from the AbstractDataSet
base class.datajoely
09/28/2021, 8:38 AMAbstractDataSet
ende
09/28/2021, 12:18 PMdatajoely
09/28/2021, 12:34 PMAbstractDataSet
?AbstractVersionedDataSet
on that page?ende
09/28/2021, 6:34 PMAbstractDataSet
(but presumably the first example should be)AbstractVersionedDataSet
example case and copy+modified that for the AbstractDataSet section.datajoely
09/29/2021, 8:06 AMende
09/29/2021, 3:58 PMdatajoely
09/29/2021, 3:58 PMende
09/29/2021, 5:35 PM