07/08/2021, 3:48 PM
Great article @User oel. I need to take some time to think about it for sure. I have been using some of the layers a bit differently, I would be curious to hear your thoughts on it. The largest difference I see is between intermediate and primary. At the intermediate layer I only really do automated (off-the-shelf) functions, plus anything that is needed to just get it to parquet. Sometime datetimes dont want to store properly. I generally think of this intermediate layer as applying assumptions that my project has adopted, such as all strings are pre-stripped, all column names are lowercase and free of special characters. My primary layer looks a bit more like your intermediate layer. It most often starts as an identity function but gives us a place to do any manual cohersion needed.