I finally had the courage to try LTO, with GCC 5.2. -flto=6 caused a swapstorm, -flto=3 allowed things to build. After waiting a while and having spent gigs of RAM, the fifth binary is 14.5% smaller, and seems a bit faster too.
No crashes or anything surprising yet. If it survives a week, time for a new release.