Parallel vs Recurrent models

Meetings

Slides

parallel_vs_recurrent_0919.pdf
Discuss the goal & plan of the reading group.
Go over the basics of Transformers, RNNs, and state-space models (SSMs).
- Ref to SSMs (just think of them as linear RNNs): S4 / Mamba; I found this video helpful.
  - Key points: convolutional form if time homogenous; associative scan if not.
References mentioned in the meeting
- Neural Turing Machine (Graves 14): adding memory to RNNs; using softmax.

SSM survey: survey1 / survey2
Vision: survey on visual Mamba
Questions to be answered from an empirical survey
(uncleaned) Discussion notes
Tokenization (thanks @Yuchen Li for suggesting this!!): how the models handle inputs that are not naturally tokenized (e.g. images) or tokenized in a non-semantic way (e.g. character or byte-level)?
- Motivation: one limitation of Transformer (as mentioned in Albert's talk) is that it may not do well when the tokens are not at the proper semantic level. However, attention layers are able to compose (e.g. using the induction head construction in @Daniel’s talk). So the difference may lie in training, difficulties induced by sequence length, and effect on learned representations due to tokenization choices.
- Potential readings: MambaByte (and maybe some Transformer counterparts), ViT / Vision Mamba, @Vatsal Sharan’s paper on Fourier embeddings
Hybrid architectures:
- Motivation: what do we mean by parallel vs recurrent models?
- How should we think of linear attention variants (e.g. SumFormer, among many others)? Do they have the same memory bottleneck as RNNs? This is related to a discussion @Spencer Frei mentioned in the hardware reading group.
- What do we know about RNN with attention (e.g. @Kaifeng Lyu's paper) ?
- MambaOut shows that the S6 layer is not needed for some image tasks, whereas local conv is key. Are there domains where local convolution outperform both ViTs and SSMs?