Context
Most sentiment models pick one architecture and commit. Convolution is good at local phrase patterns, recurrence is good at order and long context, attention is good at deciding what matters. I wanted to know what happens when you stop choosing and wire all three into a single network.
Problem
The task is binary sentiment on IMDB movie reviews. The real question underneath it is whether a hybrid earns its extra complexity, or whether it just adds parameters and training pain for a result a plain BiLSTM would have given you anyway.
Approach
The network reads a review through one shared embedding layer, then splits the work:
- a multi-scale CNN with kernels of size 3 and 5, so it captures both trigram and five-gram patterns,
- a bidirectional LSTM for context running in both directions,
- four-head self-attention to weight the sentiment-heavy tokens,
- dual pooling that takes both the average and the max, then merges them.
Residual connections and layer normalization hold the training steady, and spatial dropout on the embeddings keeps it from leaning on any single word too hard.
Outcome
The hybrid design holds together and trains stably, which was not a given with this many moving parts stacked on each other. The interesting part was watching the attention heads and the convolution branch disagree on edge cases, which is exactly the kind of redundancy that makes a combined model more robust than its pieces.
Next Iteration
- Swap the embedding layer for a pretrained encoder and measure the lift.
- Add an ablation table that turns each branch off in turn, to show what each one is really worth.