noklam
07/08/2022, 4:34 PMdeepyaman
07/08/2022, 6:24 PMpackages_to_install
based on the requirements for the modular pipelines, or you can just package the modular pipeline itself and build based on that. Most of my understanding is of the Kubeflow v2 SDK in Python (which is still in beta), but you can achieve similar results in earlier versions I believe.deepyaman
07/08/2022, 6:26 PMuser
07/10/2022, 4:26 PMvenncit
07/10/2022, 8:10 PM{{standard_task_runner.py:52}} INFO - Started process 227 to run task
and seems like a setting/dependency is not properly met for some reason.noklam
07/10/2022, 8:12 PMnoklam
07/10/2022, 8:12 PMvenncit
07/10/2022, 8:42 PMdef execute(self, context):
it won't even print itvenncit
07/10/2022, 8:42 PMvenncit
07/10/2022, 8:45 PM*** Reading remote log from Cloudwatch log_group: airflow-Airflow-FCC-Task log_stream: ads-f9329-forecast-customer-care/feature-engineering-prophet-node/2022-07-10T20_40_29.715933+00_00/1.log.
[2022-07-10 20:40:30,595] {{taskinstance.py:877}} INFO - Dependencies all met for <TaskInstance: ads-f9329-forecast-customer-care.feature-engineering-prophet-node 2022-07-10T20:40:29.715933+00:00 [queued]>
--------------------------------------------------------------------------------
[2022-07-10 20:40:30,622] {{taskinstance.py:1089}} INFO - Executing <Task(KedroOperator): feature-engineering-prophet-node> on 2022-07-10T20:40:29.715933+00:00
[2022-07-10 20:40:30,625] {{standard_task_runner.py:52}} INFO - Started process 314 to run task
venncit
07/11/2022, 9:15 PMMatthias Roels
07/12/2022, 6:18 PMdatajoely
07/12/2022, 6:19 PMenv
attribute is a common use case hereMatthias Roels
07/12/2022, 6:37 PMdatajoely
07/12/2022, 6:39 PMMatthias Roels
07/12/2022, 7:02 PMenv
attribute available in register_pipelines?datajoely
07/12/2022, 7:26 PMMatthias Roels
07/12/2022, 7:40 PMdatajoely
07/13/2022, 9:20 AMMatthias Roels
07/13/2022, 9:30 AMconfig_loader
object (or the env
var) available in register_pipelines
and I have no idea how (as the register_pipelines
does not accept any argument)datajoely
07/13/2022, 10:04 AMMatthias Roels
07/13/2022, 10:04 AMself.config_loader
) and register_pipeline hook (that leverages self.config_loader
)Matthias Roels
07/13/2022, 10:05 AMdatajoely
07/13/2022, 10:05 AMMatthias Roels
07/13/2022, 10:05 AMdatajoely
07/13/2022, 10:05 AMMatthias Roels
07/13/2022, 10:06 AMMatthias Roels
07/13/2022, 10:06 AMdatajoely
07/13/2022, 10:07 AMMatthias Roels
07/13/2022, 10:08 AMenv
set as an actual environment variable (KEDRO_ENV
) and use that in my custom register_pipelines
function to create a new instance of config_loader
and fetch my pipelines from there. But it is not an ideal work around...