datajoely
03/09/2022, 3:26 PMdatajoely
03/09/2022, 3:26 PMnoestl
03/09/2022, 10:13 PMdatajoely
03/09/2022, 10:23 PMkedro build-docs
that's one of the command available in a project directory:datajoely
03/09/2022, 10:25 PMmake build-docs
will generate what goes onto http://kedro.readthedocs.io/beats-like-a-helix
03/10/2022, 11:50 AMkedro build-reqs
is still failing for me after changing to jupyter_client>=5.1, <8.0
. Is there anything else I'm supposed to do?datajoely
03/10/2022, 11:52 AMbeats-like-a-helix
03/10/2022, 12:03 PMNo requirements.in found. Copying contents from requirements.txt...
/Users/jordan/mambaforge/envs/test-env/bin/python3.8 -m piptools compile -q /Users/jordan/Documents/University/4/test_project/src/requirements.in
Could not find a version that matches jupyter_client<7.0,<8.0,>=4.1,>=5.1,>=5.3.4,>=6.1.1,>=7.0.0 (from -r /Users/jordan/Documents/University/4/test_project/src/requirements.in (line 8))
Tried: 4.0.0, 4.0.0, 4.0.0, 4.1.0, 4.1.0, 4.1.1, 4.1.1, 4.1.1, 4.2.0, 4.2.0, 4.2.0, 4.2.1, 4.2.1, 4.2.1, 4.2.2, 4.2.2, 4.2.2, 4.3.0, 4.3.0, 4.3.0, 4.4.0, 4.4.0, 5.0.0, 5.0.0, 5.0.1, 5.0.1, 5.1.0, 5.1.0, 5.2.0, 5.2.0, 5.2.1, 5.2.1, 5.2.2, 5.2.2, 5.2.3, 5.2.3, 5.2.4, 5.2.4, 5.3.0, 5.3.0, 5.3.1, 5.3.1, 5.3.2, 5.3.2, 5.3.3, 5.3.3, 5.3.4, 5.3.4, 5.3.5, 5.3.5, 6.0.0, 6.0.0, 6.1.0, 6.1.0, 6.1.1, 6.1.1, 6.1.2, 6.1.2, 6.1.3, 6.1.3, 6.1.5, 6.1.5, 6.1.6, 6.1.6, 6.1.7, 6.1.7, 6.1.8, 6.1.8, 6.1.9, 6.1.9, 6.1.10, 6.1.10, 6.1.11, 6.1.11, 6.1.12, 6.1.12, 6.1.13, 6.1.13, 6.2.0, 6.2.0, 7.0.0, 7.0.0, 7.0.1, 7.0.1, 7.0.2, 7.0.2, 7.0.3, 7.0.3, 7.0.4, 7.0.4, 7.0.5, 7.0.5, 7.0.6, 7.0.6, 7.1.0, 7.1.0, 7.1.1, 7.1.1, 7.1.2, 7.1.2
Skipped pre-versions: 7.0.0a0, 7.0.0a0, 7.0.0a1, 7.0.0a1, 7.0.0rc0, 7.0.0rc0, 7.0.0rc1, 7.0.0rc1
beats-like-a-helix
03/10/2022, 12:04 PMThere are incompatible versions in the resolved dependencies:
jupyter_client<8.0,>=5.1 (from -r /Users/jordan/Documents/University/4/test_project/src/requirements.in (line 8))
jupyter-client>=4.1 (from qtconsole==5.2.2->jupyter==1.0.0->-r /Users/jordan/Documents/University/4/test_project/src/requirements.in (line 7))
jupyter-client>=7.0.0 (from jupyter-console==6.4.3->jupyter==1.0.0->-r /Users/jordan/Documents/University/4/test_project/src/requirements.in (line 7))
jupyter-client>=6.1.1 (from jupyter-server==1.13.5->jupyterlab==3.3.1->-r /Users/jordan/Documents/University/4/test_project/src/requirements.in (line 9))
jupyter-client>=5.3.4 (from notebook==6.4.8->jupyter==1.0.0->-r /Users/jordan/Documents/University/4/test_project/src/requirements.in (line 7))
jupyter-client<8.0 (from ipykernel==6.9.1->jupyter==1.0.0->-r /Users/jordan/Documents/University/4/test_project/src/requirements.in (line 7))
jupyter-client<7.0,>=5.1 (from kedro==0.17.7->-r /Users/jordan/Documents/University/4/test_project/src/requirements.in (line 10))
deepyaman
03/10/2022, 1:08 PMjupyter-console<6.4.3 # 6.4.3 requires jupyter_client>=7.0
in your src/requirements.txt
(bumping the jupyter_client
range yourself isn't feasible since the Kedro dependency will still restrict it until a new release).datajoely
03/10/2022, 1:10 PMbeats-like-a-helix
03/10/2022, 5:24 PMmscrts
03/11/2022, 7:36 AMdatajoely
03/11/2022, 7:39 AMWolVez
03/15/2022, 2:33 PMdatajoely
03/15/2022, 2:34 PMMemoryDataSet
out of the box - what else would you like?WolVez
03/15/2022, 2:35 PMdatajoely
03/15/2022, 2:59 PMMemoryDataSet
and (2) you could tweak to get it to return the last dataset too
https://kedro.readthedocs.io/en/0.17.2/06_nodes_and_pipelines/04_run_a_pipeline.html#custom-runnersWolVez
03/15/2022, 3:01 PMdatajoely
03/15/2022, 3:59 PMpdb
or ipdb
debuggers to achieve the same thingWolVez
03/15/2022, 4:02 PMlbonini
03/15/2022, 8:44 PMyaml
example_csv: # Input
type: pandas.ParquetDataSet
filepath: "s3://my_bucket/yyyy-mm-dd/*.parquet
I need to save it inside a today()
folder...
I was wondering about doing this with templateconfigloader, using environment variables but I need to know if it is simpler than I expect...datajoely
03/15/2022, 8:46 PMPartitionedDataSet
to do this with pandas. Spark will do this automatically.lbonini
03/15/2022, 8:51 PMdatajoely
03/15/2022, 8:54 PMDIVINE
03/15/2022, 8:54 PMdatajoely
03/15/2022, 8:57 PMdatajoely
03/15/2022, 8:58 PMantony.milne
03/16/2022, 9:32 AMvivecalindahl
03/16/2022, 10:05 AMFILEPATH=/path/to/file/
and in my data catalog I'd want something like
example_iris_data:
type: pandas.CSVDataSet
filepath: "${params:filepath}"
Then I could run kedro as kedro run --params filepath=$FILEPATH
.
I'm aware of the TemplatedConfigLoader
https://kedro.readthedocs.io/en/latest/04_kedro_project_setup/02_configuration.html#template-configuration . But is there a way of skipping the extra loop through the config file globals.yaml
? I haven't been able to figure out how to do what I outlined above, i.e. using the parameters dict directly.
Basically it would be nice to directly provide the env variable, without first creating a globals.yaml
containing a single variable.