datajoely
01/19/2022, 10:10 PMdatajoely
01/19/2022, 10:12 PMRroger
01/19/2022, 10:38 PMmartinlarsalbert
01/20/2022, 9:34 AMac_context=globals
which would expose the globals to the jinja2, but it also seams that the globals.yml and catalog.yml is loaded together in arbitrary order so that the globals are not known at the time of the jinja2 rendering. I suspect (as mentioned) that TemplatedConfigLoader
needs a major overhaul to change thismartinlarsalbert
01/20/2022, 9:47 AManyconfig
as a backend.
Args:
config_file: Path to a config file to process.
Returns:
Parsed configuration.
"""
# for performance reasons
import anyconfig # pylint: disable=import-outside-toplevel
if "globals" in str(config_file):
globals = {}
else:
globals_pattern = "*globals.yml"
globals = self.get(globals_pattern) if globals_pattern else {}
return {
k: v
for k, v in anyconfig.load(
config_file, ac_template=True, ac_context=globals
).items()
if not k.startswith("_")
}
```datajoely
01/20/2022, 10:35 AMmartinlarsalbert
01/20/2022, 12:17 PMif "globals" in str(config_file):
This is however not folowing the globals_pattern so it is a bit of a hack, but it works for my project at leastdatajoely
01/20/2022, 12:17 PMdatajoely
01/20/2022, 12:18 PMdatajoely
01/20/2022, 12:18 PMuser
01/23/2022, 8:43 AMjaweiss2305
01/23/2022, 11:06 AMdatajoely
01/23/2022, 12:57 PMspark.SparkHiveDataSet
user
01/23/2022, 3:16 PMlucas.scholl
01/23/2022, 9:26 PMdatajoely
01/23/2022, 9:44 PMChainYo
01/25/2022, 3:10 PMNone
as outputs ?datajoely
01/25/2022, 3:12 PMkedro run --pipeline preprocessing && kedro run --model_train
will run sequentially, or orchestrate these explicitly in a higher level tool like Airflow/Prefect etcChainYo
01/25/2022, 3:13 PMexplicit dependencies
are done with kedro directly in pipeline_registry
?ChainYo
01/25/2022, 3:16 PMThe order in which you add the pipelines together is not significant and data_science_pipeline + data_processing_pipeline will result in the same pipeline, since Kedro automatically detects the correct execution order for all the nodes in the resulting pipeline.
ChainYo
01/25/2022, 3:16 PMdatajoely
01/25/2022, 3:18 PMdatajoely
01/25/2022, 3:18 PMdatajoely
01/25/2022, 3:18 PMdatajoely
01/25/2022, 3:18 PMChainYo
01/25/2022, 3:20 PMparameters
, Maybe they could output something useful.ChainYo
01/25/2022, 3:21 PMtraining_done=True
) in the second pipeline, that could work ?datajoely
01/25/2022, 3:26 PMChainYo
01/25/2022, 3:49 PMChainYo
01/25/2022, 3:49 PMChainYo
01/25/2022, 3:49 PM