Out of curiosity, what is so important/critically desired for XFCE?
I mean I like very much XFCE (in Alpine Linux), but my understanding is that XFCE is, like GNOME or KDE, just a desktop-environment, aka a bunch of applications (file manager, control panel, etc) + windows-manager + task-bar.
1.So, if small SIZE is the main goal, then Tiny Core has already the FLTK +WBAR + tc-panel, you just need a small file manager (gkt1 /2), some sound/memory applets (like flit, etc) for icons of ram/cpu/sound/network etc.
2. But if the goal is super-flue, like convenience, or "consistency" of colours (blah), or icons (who cares to remember their goal, if more than 10-20 icons) then ... good luck. BTW, XFCE was, once upon time, a relative small desktop paradigm alternative; not anymore.
3. if you want more "security" then CPU + kernel + libc/musl should be the latest (and bloated) -- because vulnerabilities were discovered and corrected (keep dreaming). The price must be paid somewhere, as time/ money /speed /size/ resources.
PS: in my conception, humanity must pay for at lest 3 things: laziness, or luxury or stupidity.