Hi Guys ,I am a AWS Data Engineer with just a week...
# advanced-need-help
Hi Guys ,I am a AWS Data Engineer with just a week worth experience Kedro. I have been assigned a task to deploy one of our Kedro pipelines which has 67 nodes and 4 pipelines into AWS Step Functions. I was able to successfully deploy my Kedro Pipeline as AWS Step Functions using the instructions provided here : https://kedro.readthedocs.io/en/latest/10_deployment/10_aws_step_functions.html . But when I try to run the Step Function State machine , the execution fails parallel processing error as AWS Lambda does not support parallel processing. { "resourceType": "lambda", "resource": "invoke", "error": "OSError", "cause": { "errorMessage": "[Errno 38] Function not implemented", "errorType": "OSError", "requestId": "6f1de43c-8aff-4294-a888-1a905c8fb7eb", "stackTrace": [ .............. " File \"/home/app/kedro/framework/session/store.py\", line 76, in ShelveStore\n _lock = Lock()\n", " File \"/usr/local/lib/python3.8/multiprocessing/context.py\", line 68, in Lock\n return Lock(ctx=self.get_context())\n", " File \"/usr/local/lib/python3.8/multiprocessing/synchronize.py\", line 162, in __init__\n SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)\n", " File \"/usr/local/lib/python3.8/multiprocessing/synchronize.py\", line 57, in __init__\n sl = self._semlock = _multiprocessing.SemLock(\n" ] } } Can someone help resolve this as I believe this is because of the def _convert_kedro_pipeline_to_step_functions_state_machine(self) in the deploy.py file provided in the documentation. Any help would be much appreciated. Thanks