RRoger
03/07/2022, 4:57 AMglobal.yml
refer to other items in the same file for templating?
date: 2022-02-02
somefile: filename_${date}.txt
datajoely
03/07/2022, 9:27 AMdatajoely
03/07/2022, 9:28 AM*globals.yml
first are consumed and then the templating is applied to the other filesdatajoely
03/07/2022, 9:32 AMregister_config_loader
method in hooks.py
python
globals_pattern="*globals.yml",
globals_dict={
k: v for k, v in os.environ
if k.startswith("date")
},
)
Daehyun Kim
03/07/2022, 7:16 PMdatajoely
03/07/2022, 7:18 PMDaehyun Kim
03/07/2022, 7:19 PMDaehyun Kim
03/07/2022, 7:19 PMdatajoely
03/07/2022, 7:19 PMdatajoely
03/07/2022, 7:19 PMDaehyun Kim
03/07/2022, 7:20 PMdatajoely
03/07/2022, 7:21 PMlbonini
03/07/2022, 7:58 PMMLFLOW_TRACKING_INSECURE_TLS
to true
. How am I supposed to declare it? It works only if I export in the terminal and executes kedro run
Galileo-Galilei
03/07/2022, 8:46 PMDaehyun Kim
03/07/2022, 8:52 PMdatajoely
03/07/2022, 8:53 PMdatajoely
03/07/2022, 8:54 PMDaehyun Kim
03/07/2022, 9:13 PMwaylonwalker
03/07/2022, 10:24 PMdef joiner(left, *rights)
, I can't tell if this is genious or too much magic. Would love some thoughts on how others would review this PR.datajoely
03/07/2022, 10:30 PMdatajoely
03/07/2022, 10:30 PMdatajoely
03/07/2022, 10:31 PMMelen
03/08/2022, 2:35 AMYetunde
03/08/2022, 9:10 AMbeats-like-a-helix
03/08/2022, 11:57 AMkedro run
to work as normal, or is there anything special I need to do or configure before executing to make sure I don't run out of memory?datajoely
03/08/2022, 12:17 PMdatajoely
03/08/2022, 12:17 PMbeats-like-a-helix
03/08/2022, 12:19 PMdatajoely
03/08/2022, 12:21 PMlatest
docs branchbeats-like-a-helix
03/08/2022, 12:44 PMbeats-like-a-helix
03/08/2022, 12:44 PMdatajoely
03/08/2022, 1:40 PMbeats-like-a-helix
03/08/2022, 2:14 PMdatajoely
03/08/2022, 2:37 PMon_pipeline_error
hook is really useful here
(2) Try and set up a data profiling pipeline that allows you to do low-cost analysis of the data that can maybe give you an idea of what may cause problems downstream
(3) for logic errors, you may want to use exception handlers to gracefully swallow and log errors rather than killing the processbeats-like-a-helix
03/08/2022, 2:58 PM