Hi, I'm coming back to you regarding the execution of my "real" pipeline that didn't work with Airflow. I think I figured out where the problem comes from. It seems that hooks implementations are not taken into account when the pipeline is launched in Airflow. Are you aware of that? The problem is that I use hooks not only for tracking experiments using MLflow and Azure integration but also to initialize and update the parameters.yml file. I'm doing forecasting and needed the parameters.yml to be updated easily based on a specific parameter also defined in hooks.py. The nodes are not running at all in Airflow mainly because the parameters.yml file is not initialized but it is also problematic that the MLFlowTrackingClass() I created in the hooks.py file does not execute. Here's an example for the UpdateParametersFile class defined in the hooks.py file.

class UpdateParametersFile:
    
    @hook_impl
    def before_pipeline_run(self,  run_params: Dict[str, Any]) -> None:
        conf_paths = ['conf/base', 'conf/local']
        conf_loader = ConfigLoader(conf_paths)
        config_params = conf_loader.get("parameters*", "parameters*/**")
        with open('conf/base/parameters.yml', 'w') as file:
            pass
        config_params["param1"] = dict(key1=10, key2=24)
        # Initialize key-values pairs
        config_params["param3"] = dict(key1='sum')
        with open('conf/base/parameters.yml', 'w') as file:
            yaml.dump(config_params, file)

I also made the config_params available to the nodes that require "parameters" by adding the following to the ProjectHooks class:

class ProjectHooks:
    @hook_impl
    def before_node_run(self, node: Node, inputs):
        conf_paths = ['conf/base', 'conf/local']
        conf_loader = ConfigLoader(conf_paths)
        config_params = conf_loader.get("parameters*", "parameters*/**")
        if node.name == "node_name":
            return {"parameters": config_params}

Thank you in advance