Or why Wikipedia is the perfect Data Quality testing ground (not just for linguists)
Every developer who has ever scraped Wikipedia for an NLP task thinks: "It's human-written text. That means the dataset is balanced, natural, and ready to feed into my model."
Spoiler: it's not.
We analyzed 6 language editions of Wikipedia (Russian, Tatar, Bashkir, Mari, Mordvin, Komi) using the crudest but most effective tool in a data engineer's arsenal — frequency analysis of top tokens.
The results will change how you think about building training sets and text corpora for any IT system.
The litmus test: healthy dataset vs. train wreck
In any natural, human-written corpus, the most frequent tokens are function words — prepositions, conjunctions, pronouns, auxiliary verbs. This is a universal law across languages.
Example from the Russian National Corpus:
| # | Word | POS |
|---|
| 1 | и (and) | conj |
| 2 | в (in) | prep |
| 3 | не (not) | part |
If your top-20 contains "river", "basin", "water", "orchid" — your data pipeline is broken, even if your scraping code is flawless.
Russian Wikipedia — the gold standard (mostly)
1.06M articles. Top tokens:
| # | Word | Frequency |
|---|
| 1 | в (in) | 10,859,129 |
| 2 | и (and) | 5,761,105 |
| 3 | на (on) | 3,214,393 |
✅ Function words dominate. The dataset is usable for language modeling.
But Tatar and Bashkir Wikipedia? Data Quality Nightmare
~50K articles each. Top-5 for Tatar:
| # | Word | Translation | Frequency |
|---|
| 1 | елга | river | 132,567 |
| 2 | бассейны | basin | 75,706 |
| 3 | су | water | 54,689 |
| 4 | буенча | by (postposition) | 48,838 |
| 5 | Русия | Russia | 48,722 |
"River" appears 3x more often than the conjunction "and". Bashkir is identical: йылга (river), бассейны, һыу (water), Рәсәй (Russia).
Why? Auto-generated content strikes again
Most articles in these editions weren't written by humans. They were auto-generated from a template using Russia's State Water Registry.
Thousands of river entries — each repeating "river", "basin", "water", "km", "mouth". This is a textbook case of systematic bias that kills any ML pipeline.
For a developer, this means:
- Your word embeddings will be obsessed with hydrology.
- Any text classifier will overfit on geographic patterns.
- You'll spend months debugging why your model keeps generating "river" in every context.
Mordvin Wikipedias: The Orchid Syndrome
Erzya and Moksha editions (~1,100–1,500 articles each). Top tokens include "category", "see", "day", "year" — but also:
| Erzya | Moksha |
|---|
| истэжо (also) | ди (and) |
| категория | и (and) |
| ван (see) | тыналста (from family) |
| чи (day) | касыкссь (plant) |
| ды (and) | панчф (flower) |
| ие (year) | орхидея (orchid) |
The explanation: hundreds of articles about orchids (family Orchidaceae) were bot-generated. Each repeats "orchid", "flower", "species", "genus".
They call it the Orchid Phenomenon — a dataset hijacked by bots writing about a single narrow topic.
Quick sanity check: average article length
| Edition | Avg length (words) |
|---|
| Russian | 184.5 |
| Tatar | 58.5 |
| Bashkir | 56.8 |
| Komi-Zyrian | 29.6 |
| Erzya | 34.5 |
| Moksha | 34.9 |
Short articles = high risk of template-driven, repetitive content. This is a red flag for any Data Engineer: you need additional filtering.
What this means for you as a developer building IT systems
1. Never trust a dataset blindly — especially for low-resource languages or small editions. Dataset size ≠ dataset quality.
2. Frequency analysis is your cheap, powerful friend. Generate a top-N word list in 5 minutes with any language (Python, collections.Counter, a few lines of code). Make it a mandatory step in your ETL pipeline.
3. Auto-generated content is a systemic risk. It's everywhere: not just rivers and orchids, but also geocoordinates, biological taxa, movie stubs, sports tournaments. If you don't clean it, your model will think the world consists of rivers and flowers.
4. Data Drift starts here. If your training distribution doesn't match production traffic, your system makes decisions based on a distorted reality. That's not a bug in your code — it's a bug in your data.
5. Version and profile your data like you version your code. Implement Data Quality checks (Great Expectations, Pytest for dataframes) in your CI/CD pipeline. If the top-10 tokens suddenly change — halt the pipeline.
The bottom line
Wikipedia is free, but not always natural.
For major languages (English, Russian, German) the corpus is surprisingly balanced. For many minority languages, the Wikipedia you scrape might be a botanical reference, not a linguistic corpus.
For an engineer, this means: always profile your data. A frequency list takes 5 minutes to generate and can save you months of debugging weird model predictions and overfitting on noise patterns.
Further reading: Orekhov, Reshetnikov (2014). "Towards an evaluation of Wikipedia as a linguistic source: a comparative study." In Modern Russian Language on the Internet, Moscow.
Liked this? Share with your fellow engineers. They'll never look at a Wikipedia dump as "clean data" again.