martinlarsalbert
01/20/2022, 9:47 AManyconfig
as a backend.
Args:
config_file: Path to a config file to process.
Returns:
Parsed configuration.
"""
# for performance reasons
import anyconfig # pylint: disable=import-outside-toplevel
if "globals" in str(config_file):
globals = {}
else:
globals_pattern = "*globals.yml"
globals = self.get(globals_pattern) if globals_pattern else {}
return {
k: v
for k, v in anyconfig.load(
config_file, ac_template=True, ac_context=globals
).items()
if not k.startswith("_")
}
```datajoely
01/20/2022, 10:35 AMmartinlarsalbert
01/20/2022, 12:17 PMif "globals" in str(config_file):
This is however not folowing the globals_pattern so it is a bit of a hack, but it works for my project at leastdatajoely
01/20/2022, 12:17 PMdatajoely
01/20/2022, 12:18 PMdatajoely
01/20/2022, 12:18 PMuser
01/23/2022, 8:43 AMjaweiss2305
01/23/2022, 11:06 AMdatajoely
01/23/2022, 12:57 PMspark.SparkHiveDataSet
user
01/23/2022, 3:16 PMlucas.scholl
01/23/2022, 9:26 PMdatajoely
01/23/2022, 9:44 PMChainYo
01/25/2022, 3:10 PMNone
as outputs ?datajoely
01/25/2022, 3:12 PMkedro run --pipeline preprocessing && kedro run --model_train
will run sequentially, or orchestrate these explicitly in a higher level tool like Airflow/Prefect etcChainYo
01/25/2022, 3:13 PMexplicit dependencies
are done with kedro directly in pipeline_registry
?ChainYo
01/25/2022, 3:16 PMThe order in which you add the pipelines together is not significant and data_science_pipeline + data_processing_pipeline will result in the same pipeline, since Kedro automatically detects the correct execution order for all the nodes in the resulting pipeline.
ChainYo
01/25/2022, 3:16 PMdatajoely
01/25/2022, 3:18 PMdatajoely
01/25/2022, 3:18 PMdatajoely
01/25/2022, 3:18 PMdatajoely
01/25/2022, 3:18 PMChainYo
01/25/2022, 3:20 PMparameters
, Maybe they could output something useful.ChainYo
01/25/2022, 3:21 PMtraining_done=True
) in the second pipeline, that could work ?datajoely
01/25/2022, 3:26 PMChainYo
01/25/2022, 3:49 PMJayG
01/26/2022, 12:50 PMdatajoely
01/26/2022, 12:58 PMdatajoely
01/26/2022, 1:09 PMkedro run --pipeline={name}
limdauto
01/26/2022, 1:18 PM