A module that I'm working with that uses

multiprocessing

gives me this complaint while being used within Kedro:

from pycbc.waveform import get_fd_waveform
  File "/Users/jordan/mambaforge/envs/gravitational_waves/lib/python3.8/site-packages/pycbc/__init__.py", line 150, in <module>
    multiprocessing.set_start_method('fork')
  File "/Users/jordan/mambaforge/envs/gravitational_waves/lib/python3.8/multiprocessing/context.py", line 243, in set_start_method
    raise RuntimeError('context has already been set')
RuntimeError: context has already been set

Does anyone have an idea how I can fix this? Cheers.

I'm going to ask if anyone else can help here

That would be great!

I looked into pycbc code here. I feel like the issue is down to a pycbc wanting to create a new multiprocessing context during its init

multiprocessing.set_start_method('fork')

. Due to this, any possible of multiprocessing context that might be set before importing pycbc is likely to create similar error. Possibly pycbc kill/remove any context before trying to create a new one as it is always possible that other imports that precede could set a context already. (Not sure if this removing context is correct or even correct) In this case kedro is not using any multiprocessing functions, but has an import of multiprocessing which is somehow creating a context. A quick fix to unblock your development is to importing pycbc in your bin/kedro before any kedro import. Another option that works is to comment out the ShelveStore code in your

kedro.framework.session.store

module in your local installation. Sadly, both are temporary fixes, I don't have anything permanent

Absoute legend, thanks for this! Commenting out ShelveStore has done the trick for me. The only catch is I can't pass the

-a

flag to

kedro viz

or the same error will occur, but this is a small price to pay.

@avan-sh very much is is a legend 💪

You too @datajoely ! I have no idea how you have time for anything else in life with how active you are fixing people's problems in here haha

This is possibly a design problem with Pycbc, I'd say expecting that no multiprocessing context exists is the main problem here. if it's not kedro then possibly some other package might create it.

I'm tempted to make an issue documenting the remediation i.e. the shelve store stuff in a generic sense

@datajoely @beats-like-a-helix created an issue to document the problem and fix https://github.com/kedro-org/kedro/issues/1442