antony.milne
12/20/2021, 10:29 AMfs_args
instead of `save_args`:
fs_args:
open_args_save:
compression: gzip
antony.milne
12/20/2021, 10:29 AMdatajoely
12/20/2021, 10:46 AMyaml
example_iris_data_gzip:
type: pandas.CSVDataSet
filepath: data/01_raw/iris.csv.gzip
fs_args:
open_args_save:
compression: gzip
open_args_load:
compression: gzip
datajoely
12/20/2021, 10:46 AMfanzipei
12/20/2021, 11:46 AMRRoger
12/21/2021, 4:42 AMClass `pandas.SQLQueryDataSet` not found or one of its dependencieshas not been installed.
In requirements.in
I have kedro[pandas,pickle,yaml,json]==0.17.5
which should install all pandas DataSets right?
I do kedro build-reqs
then pip install -r src\requirements.txt
.datajoely
12/21/2021, 10:54 AMDaehyun Kim
12/21/2021, 10:09 PMDhaval
12/22/2021, 10:03 AMError: Failed to map datasets and/or parameters: train
I don't know what to do on this frontdatajoely
12/22/2021, 11:28 AMRRoger
12/23/2021, 3:42 AMdatajoely
12/23/2021, 12:01 PMDaehyun Kim
12/23/2021, 8:34 PMastro dev start
https://kedro.readthedocs.io/en/stable/10_deployment/11_airflow_astronomer.html#step-4-launch-the-local-airflow-cluster-with-astronomer
(kedro) kepricon@kepricon-G732LXS:~/git/kedro_test/kedro-airflow-iris$ astro dev logs
Error checking feature flag no context set, have you authenticated to a cluster
Error checking feature flag no context set, have you authenticated to a cluster
scheduler_1 | Waiting for host: 0.0.0.0 5432
webserver_1 | Waiting for host: 0.0.0.0 5432
and here's the logDaehyun Kim
12/23/2021, 8:35 PMRRoger
12/23/2021, 8:55 PMdatajoely
12/23/2021, 9:23 PMDaehyun Kim
12/28/2021, 11:44 PMpickle.PickleDataSet
type using plain pickle
module?
for example, I have model_metrics.pickle
that is PickleDataSet and how can I load it via pickle.load()
?datajoely
12/29/2021, 11:20 AMj c h a r l e s
12/30/2021, 12:00 AMpip install -U git+https://github.com/quantumblacklabs/kedro.git@b10bb69775e519598f4344ed1d2be5cc05a22533
j c h a r l e s
12/30/2021, 12:04 AMgit+https://github.com/quantumblacklabs/kedro.git@35e78cc5a5d7b64a034ce6561fc90ec579375569
j c h a r l e s
12/30/2021, 1:43 AMERROR: git+https://github.com/quantumblacklabs/kedro-viz.git@8da5a164637bfbd9c2f526b4f7a68f7a8a1114f2 does not appear to be a Python project: neither 'setup.py' nor 'pyproject.toml' found.
j c h a r l e s
12/30/2021, 4:02 AMj c h a r l e s
12/30/2021, 9:56 AMconf/base/parameters/<pipeline-name>.yml
. I use a custom hook that iterates through this yml file and uses catalog.add_all(new_entries, replace=True)
to generate all the downstream datasets. Then in each pipeline, rather than mapping inputs and outputs, I again iterate through this same set of entities in conf/base/parameters/<pipeline-name>.yml
and use functools.partial in order to create the node func
, freezing the input values to be values found in this parameters file.j c h a r l e s
12/30/2021, 10:07 AMlist object has no attribute filter
, a potentially more useful error might be something that throws ValueError, expected a pipeline object and received list
. Another error I found was when I tried to pass literal values to a node function, the error was something like cannot split
, and could be improved to be ValueError: inputs are not allowed to contain literal values (like integers). Please use functools.partial to create a node function with the desired literal argument specified
. These errors are very hard to debug because they are thrown deep in the kedro library code. Could save a lot of hassle by having better validation errors.datajoely
12/30/2021, 10:23 AMdatajoely
12/31/2021, 1:27 PMGalileo-Galilei
12/31/2021, 1:36 PMdatajoely
12/31/2021, 4:54 PMj c h a r l e s
01/03/2022, 12:05 AM