Your list above indicates all Xorg drivers (which are the most basic of drivers), the additional few Kb they consume is a moot point IMO
I find the way to reduce boot times with xf86 drivers is intelligently selecting only those which are required for all systems, like xf86-input-xxx .
The other nv, nouveau, etc etc drivers (not likely to support modern hardware anyhow) are all unnecessary when loading up proprietary Nvidia drivers.
When you bring in real proprietary Nvidia and ATI drivers into the mix, this changes perspective somewhat. But I would argue that a few Mb of memory unnecessarily used against several Gb of available memory and the few extra milliseconds it takes to load or mount is almost negligible in today's systems, the upside of this is functionality across many systems. If you are still thinking lst files based on configuration, by now we're down to maybe only two lst options, which is much more manageable Xorg + Nvidia or Xorg + ATI lst files.
Striving for a balance of minimal drivers to support the most hardware configurations is always challenging, but can be done