Ah nice, namespacing issue will get easier in the future

There are ways you could do this but I'd recommend you don't because it will break reproducibility

As don't typically recommend this because the combinatorial complexity introduced makes things v hard to manage/maintain/debug

I see, what would you suggest to do to make some similar effects or near as that purpose?

It comes down to how kedro is called

If you're doing conditional logic it's important to think about the side effects

So, basically, I do run kedro via execution of a .yml file that contains the kedro run command....

That .yml is called on a scheduled basis to execute the whole of the pipelines

Following @Thiago Poletto question, How can I trigger one or more pipelines based on the current date?

So this is the time to explore an orchestrator like Airflow to do this type of execution logic

The trick is to separate the scheduling from the business logic codified in kedro

could that be done through hooks?

It can, everything is possible!

But it does go against the reproducibility principle we try to enforce

I see, although I'm a still a little stuck to find a solution for that purpose

the idea is to run a different pipeline depending on a date period, so normally pipeline 1 will run, and when that period comes, pipeline 2 runs...

What is the suggested way of working for python code that is used in multiple nodes in different pipelines? Do I place that code in /src, or should I put it into a certain node.py, and import it from there? I understand that in principle we want modularity and create nodes which can run independently. However, I still encounter situations where I can re-use certain pieces of code in multiple places.

We are moving to Slack, please join us with https://join.slack.com/t/kedro-org/shared_invite/zt-1eicp0iw6-nkBvDlfAYb1AUJV7DgBIvw For the short answer. You are free to organise your code as a typical python library, not everything belongs to a pipeline and node. For example, you can have a

package.common

or

package.utils

module. Reusing a function in multiple nodes is not uncommon.

Thanks, and thanks!

Has anybody got any experience with debugging unittests using Kedro and VSCode (using pytest)? I followed the guide "Set up Visual Studio Code" from the docs and used the suggested launch.json file. But when using VSCode's "debug test" option, the debugger doesn't stop at the breakpoints :(. Any way to fix this?

The output view of the terminal is a good place to start here. Typically it shows something is broken in the start up process before the tests run.

Thanks for the tip, indeed the debug console throws me some interesting output:

PYDEV DEBUGGER WARNING:
sys.settrace() should not be used when the debugger is being used.
This may cause the debugger to stop working correctly.
If this is needed, please check: 
http://pydev.blogspot.com/2007/06/why-cant-pydev-debugger-work-with.html
to see how to restore the debug tracing back correctly.
Call Location:
  File "c:\Users\my_project_directory\env\lib\site-packages\coverage\collector.py", line 292, in _installation_trace
    sys.settrace(None)

I'm in the works of googling this... A bit confused, though, why this problem shows up.

Ah sometimes coverage causes issues - try --no-cov in their pytest config

Thanks for helping solve my issue, even though it didn't directly involve Kedro :). It's also addressed by VS Code themselves: https://code.visualstudio.com/docs/python/testing#_pytest-configuration-settings. They suggest putting the --no-cov into vscode's launch.json -- which didn't solve my issue, because the setting was superseded by the pytest settings in Kedro's default pyproject.toml file. But now that I've added it to pyproject.toml,

[tool.pytest.ini_options] \n addopts=...

, it works like a charm 🥰

Been there before!

Hopefully you saw our message about discord being decommissioned in favour of more enterprise friendly slack org. Please join us there!

Hello everyone! I'm relatively new to Kedro. I'm using it together with Dask for some data processing, and I have some issues with regards to data locality. I have a pipeline that has three nodes where the datasets are loaded like follows:

dask.ParquetDataSet from s3 -> MemoryDataSet -> dask.ParquetDataSet to s3

I run this pipeline from my local workstation for testing purposes. My Dask Cluster is then deployed on AWS EC2 (Scheduler+Workers) and they communicate privately. I noticed that on the last node, the

MemoryDataSet -> dask.ParquetDataSet to s3

causes the data to be transferred to my local machine where the Kedro pipeline is being run, and then transferred back to s3. Needless to say this introduces costs and lag and is not what I intended. Can I tell the workers to write this data directly to s3? If not, what is the intended way to do this? I read through the documentation, and there is some very good information on getting the Pipeline to run as either step functions (https://kedro.readthedocs.io/en/stable/deployment/aws_step_functions.html) or on AWS Batch (https://kedro.readthedocs.io/en/stable/deployment/aws_batch.html), but this is not quite the deployment flow I had in mind. Is it intended for the pipeline to be run on the same infrastructure where the workers are deployed?

Oops, just read this : ) I'll try again there. Thanks for the heads-up.

Hello everyone! I'm new here. I'm trying the starter Iris project, reading the raw .csv files from S3, but I'm having issues with versions (kedro, boto3, s3fs) using Virtualenv. Python==3.8.5 kedro==0.18.3 Sometimes I get the message "*Install s3fs to access S3*" when that library has already been installed, and other times "N*o node was executed. Repeat the above command to try a new execution*". Do you know if there are any specific dependencies between libraries?

You need to install the specific dataset e.g. kedro[pandas.CSVDataSet]