desrame
03/05/2022, 6:32 PMdatajoely
03/05/2022, 6:32 PMdesrame
03/05/2022, 6:32 PMdatajoely
03/05/2022, 6:33 PMdesrame
03/05/2022, 6:33 PMdatajoely
03/05/2022, 6:33 PMdesrame
03/05/2022, 6:33 PMdesrame
03/05/2022, 6:33 PMdesrame
03/05/2022, 6:33 PMdesrame
03/05/2022, 6:34 PMdesrame
03/05/2022, 6:34 PMdatajoely
03/05/2022, 6:34 PMpipeline_registry.py
you can just remove the keys from the dictionarydatajoely
03/05/2022, 6:34 PMdatajoely
03/05/2022, 6:34 PMdesrame
03/05/2022, 6:35 PMdesrame
03/05/2022, 6:35 PMdesrame
03/06/2022, 11:42 PMdatajoely
03/06/2022, 11:48 PMPickleDataSet
is probably the best shout - potentially with the joblib
backenddatajoely
03/06/2022, 11:49 PMMemoryDataSet
desrame
03/07/2022, 12:01 AMRRoger
03/07/2022, 4:57 AMglobal.yml
refer to other items in the same file for templating?
date: 2022-02-02
somefile: filename_${date}.txt
datajoely
03/07/2022, 9:27 AMdatajoely
03/07/2022, 9:28 AM*globals.yml
first are consumed and then the templating is applied to the other filesdatajoely
03/07/2022, 9:32 AMregister_config_loader
method in hooks.py
python
globals_pattern="*globals.yml",
globals_dict={
k: v for k, v in os.environ
if k.startswith("date")
},
)
Daehyun Kim
03/07/2022, 7:16 PMdatajoely
03/07/2022, 7:18 PMDaehyun Kim
03/07/2022, 7:19 PMDaehyun Kim
03/07/2022, 7:19 PMdatajoely
03/07/2022, 7:19 PMdatajoely
03/07/2022, 7:19 PM