Creative Speculation

In an idealized symmetrical collapse, the energy of the collapse would be spread out evenly throughout the lower structure, maximizing the ability of the lower structure to resist collapse.

In an asymmetrical collapse, as happens in the real world, the same kinetic energy will focus its destructive powers on a much smaller area of the lower structure, greatly reducing the ability of the lower structure to resist collapse.

The potential energy of different sections of the top structure aren't going to magically wait their turns. Once the supporting columns for a floor are greatly compromised, and the collapse begins, the potential energy of the top structure very quickly gets turned into kinetic energy.