778216384475693066 #beginners-need-help

Channels

advanced-need-help

job-posting

welcome

datajoely

03/09/2022, 3:26 PM

If you bump the version it should then compile

datajoely

03/09/2022, 3:26 PM

shout if you have any problems!

noestl

03/09/2022, 10:13 PM

Hello, I am trying to use on Mac, the command "make build-docs" to generate some documentations. It worked with "kedro build-docs" but not with make. Following error is attached. Do someone know what's happening ?

datajoely

03/09/2022, 10:23 PM

Hello @User so I think we're talking about two different things: If you are generating a Kedro project and you want to generate a documentation for your Data project you can run

kedro build-docs

that's one of the command available in a project directory:

datajoely

03/09/2022, 10:25 PM

We don't actually ship a make file in Kedro projects because it's not great on Windows. Kedro itself has a makefile for developers contributing to the open source project and that

make build-docs

will generate what goes onto http://kedro.readthedocs.io/

beats-like-a-helix

03/10/2022, 11:50 AM

kedro build-reqs

is still failing for me after changing to

jupyter_client>=5.1, <8.0

. Is there anything else I'm supposed to do?

datajoely

03/10/2022, 11:52 AM

can you post your error?

beats-like-a-helix

03/10/2022, 12:03 PM

Copy code

No requirements.in found. Copying contents from requirements.txt...
/Users/jordan/mambaforge/envs/test-env/bin/python3.8 -m piptools compile -q /Users/jordan/Documents/University/4/test_project/src/requirements.in
Could not find a version that matches jupyter_client<7.0,<8.0,>=4.1,>=5.1,>=5.3.4,>=6.1.1,>=7.0.0 (from -r /Users/jordan/Documents/University/4/test_project/src/requirements.in (line 8))
Tried: 4.0.0, 4.0.0, 4.0.0, 4.1.0, 4.1.0, 4.1.1, 4.1.1, 4.1.1, 4.2.0, 4.2.0, 4.2.0, 4.2.1, 4.2.1, 4.2.1, 4.2.2, 4.2.2, 4.2.2, 4.3.0, 4.3.0, 4.3.0, 4.4.0, 4.4.0, 5.0.0, 5.0.0, 5.0.1, 5.0.1, 5.1.0, 5.1.0, 5.2.0, 5.2.0, 5.2.1, 5.2.1, 5.2.2, 5.2.2, 5.2.3, 5.2.3, 5.2.4, 5.2.4, 5.3.0, 5.3.0, 5.3.1, 5.3.1, 5.3.2, 5.3.2, 5.3.3, 5.3.3, 5.3.4, 5.3.4, 5.3.5, 5.3.5, 6.0.0, 6.0.0, 6.1.0, 6.1.0, 6.1.1, 6.1.1, 6.1.2, 6.1.2, 6.1.3, 6.1.3, 6.1.5, 6.1.5, 6.1.6, 6.1.6, 6.1.7, 6.1.7, 6.1.8, 6.1.8, 6.1.9, 6.1.9, 6.1.10, 6.1.10, 6.1.11, 6.1.11, 6.1.12, 6.1.12, 6.1.13, 6.1.13, 6.2.0, 6.2.0, 7.0.0, 7.0.0, 7.0.1, 7.0.1, 7.0.2, 7.0.2, 7.0.3, 7.0.3, 7.0.4, 7.0.4, 7.0.5, 7.0.5, 7.0.6, 7.0.6, 7.1.0, 7.1.0, 7.1.1, 7.1.1, 7.1.2, 7.1.2
Skipped pre-versions: 7.0.0a0, 7.0.0a0, 7.0.0a1, 7.0.0a1, 7.0.0rc0, 7.0.0rc0, 7.0.0rc1, 7.0.0rc1

beats-like-a-helix

03/10/2022, 12:04 PM

Copy code

There are incompatible versions in the resolved dependencies:
  jupyter_client<8.0,>=5.1 (from -r /Users/jordan/Documents/University/4/test_project/src/requirements.in (line 8))
  jupyter-client>=4.1 (from qtconsole==5.2.2->jupyter==1.0.0->-r /Users/jordan/Documents/University/4/test_project/src/requirements.in (line 7))
  jupyter-client>=7.0.0 (from jupyter-console==6.4.3->jupyter==1.0.0->-r /Users/jordan/Documents/University/4/test_project/src/requirements.in (line 7))
  jupyter-client>=6.1.1 (from jupyter-server==1.13.5->jupyterlab==3.3.1->-r /Users/jordan/Documents/University/4/test_project/src/requirements.in (line 9))
  jupyter-client>=5.3.4 (from notebook==6.4.8->jupyter==1.0.0->-r /Users/jordan/Documents/University/4/test_project/src/requirements.in (line 7))
  jupyter-client<8.0 (from ipykernel==6.9.1->jupyter==1.0.0->-r /Users/jordan/Documents/University/4/test_project/src/requirements.in (line 7))
  jupyter-client<7.0,>=5.1 (from kedro==0.17.7->-r /Users/jordan/Documents/University/4/test_project/src/requirements.in (line 10))

deepyaman

03/10/2022, 1:08 PM

Include

jupyter-console<6.4.3  # 6.4.3 requires jupyter_client>=7.0

in your

src/requirements.txt

(bumping the

jupyter_client

range yourself isn't feasible since the Kedro dependency will still restrict it until a new release).

datajoely

03/10/2022, 1:10 PM

Thanks @User

beats-like-a-helix

03/10/2022, 5:24 PM

Much obliged @User 👌

mscrts

03/11/2022, 7:36 AM

Hello ser may I ask is there any best practices to integrate scrapy in kedro pipeline to automate the data collection job? Thanks!🙏🏻

datajoely

03/11/2022, 7:39 AM

So since that's sort of an IO step I would recommend you define a custom dataset that only provides and implementation for load not a save operation https://kedro.readthedocs.io/en/stable/07_extend_kedro/03_custom_datasets.html You may want to take inspiration from the APIDataSet which look a bit like this https://kedro.readthedocs.io/en/stable/_modules/kedro/extras/datasets/api/api_dataset.html#APIDataSet

WolVez

03/15/2022, 2:33 PM

What is the default dataset if a node output is not specified in the config? It can clearly still be referenced and assigned even in a jupyter notebook. Is it just a MemoryDataset?

datajoely

03/15/2022, 2:34 PM

You can technically change this, but it is a

MemoryDataSet

out of the box - what else would you like?

WolVez

03/15/2022, 2:35 PM

No, that is perfect. The functionality for debugging with the memorydataset in a jupyter notebook is fantastic. However, when we specify an output the jupyter notebook doesn't return the memory dataset output, so we are looking at adding just a general debug config for quickly being able to add the needed debugging output to the various needed nodes.

datajoely

03/15/2022, 2:59 PM

I'm not sure how/why this dropped out the latest docs but there is a way you can create a custom runner that (1) converts everything to a

MemoryDataSet

and (2) you could tweak to get it to return the last dataset too https://kedro.readthedocs.io/en/0.17.2/06_nodes_and_pipelines/04_run_a_pipeline.html#custom-runners

WolVez

03/15/2022, 3:01 PM

tweak to get the last dataset would be huge, that way when we specify a pipeline or node run we always get the one we are focusing on without bogging down the system by holding unneeded information

datajoely

03/15/2022, 3:59 PM

I can coach you through getting that working, it might be a useful thing to include in our docs when you're done. The other thing to maybe consider is to use

pdb

ipdb

debuggers to achieve the same thing

WolVez

03/15/2022, 4:02 PM

Would love that.

lbonini

03/15/2022, 8:44 PM

Hello guys! Is there a easy way to export a file from catalog to a dynamic url like:

Copy code

yaml
example_csv: # Input
  type: pandas.ParquetDataSet
  filepath: "s3://my_bucket/yyyy-mm-dd/*.parquet

I need to save it inside a

today()

folder... I was wondering about doing this with templateconfigloader, using environment variables but I need to know if it is simpler than I expect...

datajoely

03/15/2022, 8:46 PM

Hello you may be able to use

PartitionedDataSet

to do this with pandas. Spark will do this automatically.

lbonini

03/15/2022, 8:51 PM

Is there another way to do this?

datajoely

03/15/2022, 8:54 PM

I think you just wrap your dataset one level higher

DIVINE

03/15/2022, 8:54 PM

Hello! What's the best practice to update micro-packages between projects? For example, let's say I develop a pipeline P-1 in a project P-A then micro-package it into a project P-B then I find out that I need to update the pipeline (for example for a bugfix). Should I update the micro-package manually then publish it? Should I do it in project P-A or project P-B?

datajoely

03/15/2022, 8:57 PM

Hello! This is an advanced question but I have a couple of thoughts here. (1) you could designate one of these the "lead" project where consuming projects just take things as is (2) you can set up CI rules to let you know when your versions get stale when compared to the artefactory / repo where the 'latest' lived

datajoely

03/15/2022, 8:58 PM

Either way it's somewhat of a team workflow question rather than one I think we have a specific view on

antony.milne

03/16/2022, 9:32 AM

FYI @User we are currently thinking about improvements to our interactive Jupyter notebook workflow, how users should debug pipelines, etc. Sounds very relevant to what you're trying to do so please do leave suggestions, comments, feature requests etc. here 🙂 https://github.com/kedro-org/kedro/issues/1075

vivecalindahl

03/16/2022, 10:05 AM

Hi all! I'd like to parameterize my data catalog using an environment variable. For instance, say I have an environment variable

FILEPATH=/path/to/file/

and in my data catalog I'd want something like

Copy code

example_iris_data:
 type: pandas.CSVDataSet
 filepath: "${params:filepath}"

Then I could run kedro as

kedro run --params filepath=$FILEPATH

. I'm aware of the

TemplatedConfigLoader

https://kedro.readthedocs.io/en/latest/04_kedro_project_setup/02_configuration.html#template-configuration . But is there a way of skipping the extra loop through the config file

globals.yaml

? I haven't been able to figure out how to do what I outlined above, i.e. using the parameters dict directly. Basically it would be nice to directly provide the env variable, without first creating a

globals.yaml

containing a single variable.