Mixers
Mixer architectures

LatMLP and PatchTSMixer are variants of this one. Mixers involve doing some sort of permutation, then projection, then combine operation to exchange information to mimic a lightweight attention.
PatchTSMixer is from IBM developed of timeseries data. In short, it permutes thrice for each dimension.
LatMLP is an architecture that I was trying. It has patches similar to CaloDiT, hence two dimension only, patches X features.
EDM
Cell Energy:
For 500 GeV, LatMLPs are a bit better, though slightly. For 50 GeV, LatMLP is much better. Surprisingly, the more complex PatchTSMixer with gated attention is worse than non-gated attention one.
I can try to get the low energy distribution right (the left side) and the 50 GeV, but that'll result in making the model more complex, hence slow.