Skip to main content

Mixers

Mixer architectures on Dataset-2 granularity on Par04 SiW 1M samples
Last updated: February 26, 2025

Mixer architectures

MLPMixer architecture

LatMLP and PatchTSMixer are variants of this one. Mixers involve doing some sort of permutation, then projection, then combine operation to exchange information to mimic a lightweight attention.

PatchTSMixer is from IBM developed of timeseries data. In short, it permutes thrice for each dimension.

LatMLP is an architecture that I was trying. It has patches similar to CaloDiT, hence two dimension only, patches X features.



EDM



Cell Energy:

For 500 GeV, LatMLPs are a bit better, though slightly. For 50 GeV, LatMLP is much better. Surprisingly, the more complex PatchTSMixer with gated attention is worse than non-gated attention one.

I can try to get the low energy distribution right (the left side) and the 50 GeV, but that'll result in making the model more complex, hence slow.

4 selected runs


Comparing only LatMLP (non-wide) and PatchTSMixer (without gated attention)


2 selected runs


Distillation


2 selected runs