Burn1n9m4n
04/18/2022, 9:38 PMnoklam
04/18/2022, 9:38 PMnoklam
04/18/2022, 9:38 PMBurn1n9m4n
04/18/2022, 9:39 PMnoklam
04/18/2022, 9:39 PMnoklam
04/18/2022, 9:40 PMBurn1n9m4n
04/18/2022, 9:41 PMBurn1n9m4n
04/18/2022, 9:41 PMnoklam
04/18/2022, 9:42 PMBurn1n9m4n
04/18/2022, 9:44 PMBurn1n9m4n
04/18/2022, 9:47 PMs3bucketname/project_folder/data/layer/data.parquet/2022-04-18T21.43.00.910Z/data.parquet
for ParquetDataSet(filepath=s3bucketname/project_folder/data/layer/data.parquet, load_args={}, protocol=s3, save_args={}, version=Version(load=None, save='2022-04-18T21.43.00.910Z')) must not exist if versioning is enabled.Burn1n9m4n
04/18/2022, 9:47 PMBurn1n9m4n
04/18/2022, 9:49 PMnoklam
04/18/2022, 9:50 PMs3bucketname/project_folder/data/layer/data.parquet
?Burn1n9m4n
04/18/2022, 9:51 PMs3bucketname/project_folder/data/layer/data.parquet/
noklam
04/18/2022, 9:51 PMfilename/timestamp/filename
Burn1n9m4n
04/18/2022, 9:52 PMBurn1n9m4n
04/18/2022, 9:53 PMs3bucketname/project_folder/data/layer/data.parquet/timestamp/data.parquet
vivekumar
04/19/2022, 6:48 AMvivekumar
04/19/2022, 6:50 AMnoklam
04/19/2022, 7:36 AMkedro.py
Python is confused that this is the kedro module instead of the installed one.noklam
04/19/2022, 7:39 AMgui42
04/19/2022, 3:08 PMavan-sh
04/19/2022, 3:58 PMsession.run
with to_outputs
arg. But this will only return a dictionary of your datasets, not as memory datasets retrievable in catalog.
Reference to session.run
function specs: https://kedro.readthedocs.io/en/stable/kedro.framework.session.session.KedroSession.html#kedro.framework.session.session.KedroSession.runnoklam
04/19/2022, 4:36 PMgui42
04/19/2022, 6:48 PMsession.run
only returns an empty dict, and from what I understand, only datasets with some catalog issues are returned.
from the session.run docstring:
Returns:
Any node outputs that cannot be processed by the ``DataCatalog``.
These are returned in a dictionary, where the keys are defined
by the node outputs.
gui42
04/19/2022, 6:52 PMavan-sh
04/19/2022, 6:58 PMto_outputs
arg for it to return them in the dictionary.
Also what noklam might be looking to know the reason you're trying to do this.gui42
04/19/2022, 7:00 PMsession.run(to_outputs=['my_dataset'])
And the return value is an empty dict. The pipeline runs smoothly, and everything is defined in the catalog.gui42
04/19/2022, 7:01 PM