Hi Guys ,I am a AWS Data Engineer with just a week worth experience Kedro. I have been assigned a task to deploy one of our Kedro pipelines which has 67 nodes and 4 pipelines into AWS Step Functions. I was able to successfully deploy my Kedro Pipeline as AWS Step Functions using the instructions provided here :
https://kedro.readthedocs.io/en/latest/10_deployment/10_aws_step_functions.html . But when I try to run the Step Function State machine , the execution fails parallel processing error as AWS Lambda does not support parallel processing. {
"resourceType": "lambda",
"resource": "invoke",
"error": "OSError",
"cause": {
"errorMessage": "[Errno 38] Function not implemented",
"errorType": "OSError",
"requestId": "6f1de43c-8aff-4294-a888-1a905c8fb7eb",
"stackTrace": [ ..............
" File \"/home/app/kedro/framework/session/store.py\", line 76, in ShelveStore\n _lock = Lock()\n",
" File \"/usr/local/lib/python3.8/multiprocessing/context.py\", line 68, in Lock\n return Lock(ctx=self.get_context())\n",
" File \"/usr/local/lib/python3.8/multiprocessing/synchronize.py\", line 162, in __init__\n SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)\n",
" File \"/usr/local/lib/python3.8/multiprocessing/synchronize.py\", line 57, in __init__\n sl = self._semlock = _multiprocessing.SemLock(\n"
]
}
} Can someone help resolve this as I believe this is because of the
def _convert_kedro_pipeline_to_step_functions_state_machine(self) in the deploy.py file provided in the documentation. Any help would be much appreciated. Thanks