Are duplicates defined by an ID column? I think we may want to do some sort of UPSERT operation here.