What's the pattern for numerous files as raw data?...
# beginners-need-help
r
What's the pattern for numerous files as raw data? I want to download about 2000 files of the same type with different dates, e.g. "senate_2006-03-30.xml". 1. Do I create a catalog entry for each file? 2. Does the download-node
output
to a list of length 2000, i.e.
["senate_2006-03-30", "senate_2006-03-31", ...]
, i.e. a 2000-line
pipeline.py
? Or is there some sort of clever templating?