Due to some weirdness and presumably thumbnail rendering, if a folder is set to "Optimize for Pictures", it takes 10+ times as long as it should to load. This was happening for my Downloads folder. It seems to only apply when it's accessed through "This PC".
Anyway, to fix it, in the properties of the folder in question, under customize, change
Optimize this folder for: to General Items and it will work much better.