NLP Case Study

IMDb Sentiment Analysis: RNN vs Pretrained Transformers

XLNet won with 0.9578 TestAccuracy, but the more useful design lesson was that context gain flattens around 1,024 tokens.

Long, noisy IMDb reviews with HTML markup, contrast structures, and late sentiment turns. Compared BiLSTM against DistilGPT-2 and XLNet with max-length and classifier-head ablations.

NLPSentiment AnalysisTransformersRNNTruncation StudyKaggle
GitHub Report PDF

Best Result

0.9578

XLNet TestAccuracy at max_len=1,024

BiLSTM (max_len=500)89.05%
DistilGPT-2 (base head, 500)90.48%
DistilGPT-2 (MLP head, 1024)92.99%
XLNet (max_len=1024)95.78%

The Problem I Was Testing

IMDb sentiment isn’t straightforward. Reviews are long, structurally complex, and full of real-world noise. A model that scores well on short-clean-text benchmarks can still fail here.

01

Long-Context Problem

IMDb reviews average 234 words, max 2,470. Most models plateau after ~512 tokens, but late-structure reviews flip sentiment in the final sentences. Truncating loses the signal.

02

Noisy Natural Text

58.67% of reviews contain HTML markup. 36.49% have repeated punctuation. 22.56% use all-caps emphasis. This isn't clean academic text — models need to parse real-world noise.

03

Diminishing Returns After BiLSTM

A simple BiLSTM already reaches 89.05%. Each transformer model costs more compute for incrementally smaller gains. The gap between DistilGPT-2 and XLNet is the design question.

A Few Reviews That Explain the Task

negativeContrast + Complaint
I have no idea how this movie got such a high rating. ... The camera work is incompetent, the editing non-existent, and the screenplay is about as compelling as a shopping list written by a third grader.

Late-structure reversal: starts with disbelief, builds evidence over 200+ words, ends with a sharp dismissal. Sentiment isn't obvious until the final clause.

positiveCraft + Recommendation
I rented this movie at the recommendation of a friend and was a bit skeptical about how it would play out, but wasn't disappointed in the least. ... An amazing story of triumph over adversity, and a triumph for the craft of acting.

Positive signal is clear throughout, but the review is structurally similar to a negative one in length and detail. Distinguishing requires understanding the valence of each clause.

noisyMessy Real-World Text
<br /><br />I thought this movie was okay. &lt;spoiler&gt;The characters are somewhat shallow and underdeveloped.&lt;/spoiler&gt; ... I would rate this movie 5/10.

Contains HTML tags, spoiler markup, URL fragments, and a mid-review score that contradicts the opening. Models need to integrate signals across noisy formatting.

Approach

BiLSTM from scratch

Two ablations: max_len=500 and max_len=1024. Trained from random initialization on IMDb texts.

Baseline RNN. Tests whether sequence length alone improves a simple architecture.

DistilGPT-2 fine-tune

Two ablations: built-in classifier head vs custom MLP head. Both tested at max_len=500 and 1024.

Tests whether adding a learned classifier head on top of a pretrained transformer helps or just adds parameters.

XLNet fine-tune

Three max_len values: 500, 1024, and 1200. Tests whether the permutation-based architecture benefits from longer context.

XLNet's autoregressive-permutation approach reads context bidirectionally. Tests whether this matters for long reviews.

Dataset

Train / Test

25,000 each

Balanced

50% pos / 50% neg

Avg Length

234 words

Max Lengths Tested

500 / 1024 / 1200

Results

BiLSTM (max_len=500)89.05%
BiLSTM (max_len=1024)89.05%
DistilGPT-2 (base head, 500)90.48%
DistilGPT-2 (MLP head, 1024)92.99%
XLNet (max_len=1024)95.78%
XLNet (max_len=1200)95.78%
Modelmax_len=500max_len=1024max_len=1200
BiLSTM (base)89.05%89.05%—
DistilGPT-2 (base head)90.48%91.81%—
DistilGPT-2 (MLP head)89.41%92.99%—
XLNet (base head)93.34%95.78%95.78%
XLNet (MLP head)92.87%94.12%94.12%
XLNet 0.9578 @ max_len=1024

Takeaways

  • 1Transformers significantly outperform BiLSTM on long, noisy text — the gap is ~6 points for the best model.
  • 2XLNet achieves the best accuracy (0.9578) at max_len=1024, but accuracy does not improve at max_len=1200 — the model plateaus.
  • 3The MLP classifier head gives a small lift for DistilGPT-2 (+2.5 points), suggesting that a learned aggregation helps on noisy input.
  • 4Context beyond ~1,024 tokens yields no additional gain. More tokens add compute cost without better predictions — the useful signal is in the first ~1K tokens.