A module that I'm working with that uses `multipro...
# advanced-need-help
A module that I'm working with that uses
gives me this complaint while being used within Kedro:
Copy code
from pycbc.waveform import get_fd_waveform
  File "/Users/jordan/mambaforge/envs/gravitational_waves/lib/python3.8/site-packages/pycbc/__init__.py", line 150, in <module>
  File "/Users/jordan/mambaforge/envs/gravitational_waves/lib/python3.8/multiprocessing/context.py", line 243, in set_start_method
    raise RuntimeError('context has already been set')
RuntimeError: context has already been set
Does anyone have an idea how I can fix this? Cheers.
I'm going to ask if anyone else can help here
That would be great!
I looked into pycbc code here. I feel like the issue is down to a pycbc wanting to create a new multiprocessing context during its init
. Due to this, any possible of multiprocessing context that might be set before importing pycbc is likely to create similar error. Possibly pycbc kill/remove any context before trying to create a new one as it is always possible that other imports that precede could set a context already. (Not sure if this removing context is correct or even correct) In this case kedro is not using any multiprocessing functions, but has an import of multiprocessing which is somehow creating a context. A quick fix to unblock your development is to importing pycbc in your bin/kedro before any kedro import. Another option that works is to comment out the ShelveStore code in your
module in your local installation. Sadly, both are temporary fixes, I don't have anything permanent
Absoute legend, thanks for this! Commenting out ShelveStore has done the trick for me. The only catch is I can't pass the
flag to
kedro viz
or the same error will occur, but this is a small price to pay.
@avan-sh very much is is a legend 💪
You too @datajoely ! I have no idea how you have time for anything else in life with how active you are fixing people's problems in here haha
@avan-sh I appreciate this is extremely low priority, but I was going to open a Github issue about this just to have it documented somewhere. Can this even reasonably be classified as a Kedro issue, or is it more of a design problem with PyCBC?
This is possibly a design problem with Pycbc, I'd say expecting that no multiprocessing context exists is the main problem here. if it's not kedro then possibly some other package might create it.
I'm tempted to make an issue documenting the remediation i.e. the shelve store stuff in a generic sense
@datajoely @beats-like-a-helix created an issue to document the problem and fix https://github.com/kedro-org/kedro/issues/1442
Please modify the Title if necessary