datajoely
07/13/2022, 10:07 AMMatthias Roels
07/13/2022, 10:08 AMenv
set as an actual environment variable (KEDRO_ENV
) and use that in my custom register_pipelines
function to create a new instance of config_loader
and fetch my pipelines from there. But it is not an ideal work around...datajoely
07/13/2022, 10:09 AMdatajoely
07/13/2022, 10:19 AMdatajoely
07/13/2022, 10:19 AMMatthias Roels
07/13/2022, 10:53 AMantony.milne
07/13/2022, 10:55 AMafter_context_created
that makes available context.config_loader
. You can pass this as a global variable to your register_pipelines
function and do config_loader.get("pipeline")
(or whatever) there.
This still feels very hacky to me and I would like to have a better way to do it, but unfortunately since per-environment pipeline.yml files isn't really the "kedro way" it is awkward to support. IMO we should make it much easier for people to use a plugin that enables this functionalityantony.milne
07/13/2022, 11:01 AMregister_pipelines
(which is how the config loader is instantiated internally):
import settings
config_loader_class = settings.CONFIG_LOADER_CLASS
config_loader_class(
conf_source=str(self._project_path / settings.CONF_SOURCE),
env=env,
runtime_params=extra_params,
**settings.CONFIG_LOADER_ARGS,
)
To get this working fully you still need to get env
and extra_params
into register_pipelines
. I think the cleanest way to do that is again through the after_context_created
hook, but in theory you could just directly extract it from the CLI arguments if you always run kedro that way.Matthias Roels
07/13/2022, 12:06 PMMatthias Roels
07/13/2022, 7:09 PMantony.milne
07/13/2022, 7:43 PMbefore_command_run
for this. Better to access it using click
as something like click.get_current_context(silent=True).params["env"]
.antony.milne
07/13/2022, 7:45 PMsession.run
directly rather than doing kedro run
from the CLI)antony.milne
07/13/2022, 7:47 PMclick
stuff directly in register_pipelines
Matthias Roels
07/14/2022, 7:39 AMxxavier
07/14/2022, 8:34 AMarum
07/14/2022, 9:19 AMdatajoely
07/14/2022, 9:32 AMdatajoely
07/14/2022, 9:34 AMarum
07/14/2022, 10:02 AMarum
07/14/2022, 10:04 AMarum
07/14/2022, 10:05 AMarum
07/14/2022, 10:06 AMarum
07/14/2022, 10:06 AMdatajoely
07/14/2022, 10:07 AMpython -m build
, custom-py-tools
isn't provided by Kedro, nor is the Nexus artifactory manager.arum
07/14/2022, 10:25 AMarum
07/14/2022, 10:26 AMarum
07/14/2022, 10:26 AM[2022-07-14 13:21:06,000] {logging_mixin.py:115} INFO - Model version 20220713-163948
[2022-07-14 13:21:06,000] {taskinstance.py:1909} ERROR - Task failed with exception
Traceback (most recent call last):
File "/Users/sl/airflow/dags/epc_fi.py", line 36, in execute
with KedroSession.create(self.package_name,
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/kedro/framework/session/session.py", line 172, in create
session._setup_logging()
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/kedro/framework/session/session.py", line 188, in _setup_logging
conf_logging = self._get_logging_config()
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/kedro/framework/session/session.py", line 176, in _get_logging_config
conf_logging = self._get_config_loader().get(
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/kedro/config/templated_config.py", line 161, in get
config_raw = _get_config_from_patterns(
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/kedro/config/common.py", line 69, in _get_config_from_patterns
raise ValueError(
ValueError: Given configuration path either does not exist or is not a valid directory: /conf/base
[2022-07-14 13:21:06,004] {taskinstance.py:1415} INFO - Marking task as UP_FOR_RETRY. dag_id=epc-fi, task_id=preprocess, execution_date=20220714T102039, start_date=20220714T102055, end_date=20220714T102106
[2022-07-14 13:21:06,007] {standard_task_runner.py:92} ERROR - Failed to execute job 121 for task preprocess (Given configuration path either does not exist or is not a valid directory: /conf/base; 4330)
[2022-07-14 13:21:06,051] {local_task_job.py:156} INFO - Task exited with return code 1
[2022-07-14 13:21:06,060] {local_task_job.py:273} INFO - 0 downstream tasks scheduled from follow-on schedule check
arum
07/14/2022, 10:27 AMdatajoely
07/14/2022, 10:29 AMarum
07/14/2022, 12:54 PMarum
07/14/2022, 12:54 PM