Or why language vitality online has nothing to do with speaker counts — and everything to do with data engineering
Russia speaks dozens of minority languages — Tatar, Bashkir, Udmurt, and even tiny ones like South Yukaghir (fewer than 50 speakers). Some have no Wikipedia, no standard corpus, and almost no digital footprint.
But they do exist online — scattered across forums, blogs, and VK communities.
We wanted to measure their online presence. As NLP engineers, we quickly realized: no global corpus, no labelled dataset, no pre-trained embeddings, and no simple way to ask a search engine "give me all pages in Chechen".
So we built our own pipeline. Here's what every developer building NLP systems for low-resource languages needs to know.
The Core Problem: Low-Resource NLP Without a Dictionary
In mainstream NLP, you start with a corpus. Wikipedia. Common Crawl. Something.
For 96 minority languages, the corpus doesn't exist. You have to find the data first — before you can even think about training embeddings, fine-tuning BERT, or building a translation system.
This is the ultimate cold-start problem.
Every stage of a standard NLP pipeline breaks:
| NLP Stage | The Problem |
|---|
| Data Acquisition | No central repository. Data is scattered across VK, forums, blogs. |
| Language Identification | No pre-trained classifier for 90% of these languages. |
| Tokenization | No standard orthography. People type without diacritics. |
| Embeddings | No pre-trained word vectors. You must build from scratch. |
| Evaluation | No gold-standard test sets. You can't measure what you can't label. |
Step 1: Data Acquisition — The Search API as Your Crawler
You can't download the whole internet (unless you're Google). But search engines already crawled it.
We used Yandex.XML — a limited API with 10,000 requests per day. Each request returns one page of results, and each next page costs another request.
For an engineer, this means: you have to be extremely efficient with your queries.
The challenge: What do you search for?
We needed word markers — words that are:
- Frequent in the target language
- Unique — not appearing in Russian or other languages
- Free of special characters — because real users don't type them
That last point is critical for any NLP system dealing with real-world user-generated content.
Most minority languages use Cyrillic extensions: ҙ, ә, ö, ҷ. There's no standard keyboard for them. People write "менан" instead of "менән". If we query with the "correct" spelling, we lose 90% of real texts.
We call this "household spelling" — like medieval birch-bark manuscripts, but with modern laziness.
For your NLP pipeline: If you're building systems for languages without standard keyboards, you need fuzzy matching at the query stage, not just at the evaluation stage. This means:
- Levenshtein-based query expansion
- Phonetic normalization
- Handling diacritic stripping before search
Step 2: Noise Filtering — The Precision Problem
Even good markers return junk. We had to filter:
- Wikipedia duplicates — we can download those in one click, so exclude them
- Linguistic papers — Russian text containing one line of Rutul as an example
- Music sites — song titles but no lyrics
- Typos — e.g., Russian
"пазух" (bosom) mistyped as "пазох", which is a valid Khakas word meaning "again"
The solution: Multi-marker verification
If one marker appears on a domain but others from the same language don't — it's probably a false positive.
For your ML pipeline: This is ensemble validation — using multiple weak signals to confirm a strong one. The trade-off is extra API calls, which are expensive at 10k/day.
Alternative approach: Train a binary classifier (Russian vs. everything else) on the fly. We did this later — more on that below.
Step 3: Domain Classification — Building a Tiered Index
We asked Yandex to categorize each domain into three tiers:
| Tier | Description | Example |
|---|
| 1. Language-first | Most pages in the minority language | Small blogs, community sites |
| 2. Mixed | Several pages, but not dominant | Regional news portals |
| 3. Giant platforms | Minority content is buried | YouTube, stihi.ru |
Additionally, we specifically queried VK.com communities because post-2012, most standalone language sites died — and everything moved to social networks.
For your data engineering pipeline: You need a stratified crawling strategy. Don't treat all domains equally. Allocate more budget to Tier 1, and use depth-limited sampling for Tier 3.
Step 4: Language Detection Without a Training Corpus
We downloaded pages and needed to separate Russian from, say, Tuvan.
Classic n-gram language detection requires a pre-labelled corpus — which we didn't have for most languages.
The solution: We did have Russian — plenty of it. So we built a binary classifier: Russian vs. everything else.
If a paragraph isn't Russian, we assume it's our target language.
For your ML system:
- This is a one-class classification problem with a twist.
- It's not perfect — but it's good enough to build the first ever web corpora for these languages.
- This is a classic weak supervision approach: use what you have (Russian) to learn what you don't (everything else).
What We Learned: Online vs. Offline — Zero Correlation
We compared online metrics (Wikipedia articles, VK communities, text volume) with offline metrics (speaker count, regional economy).
| Metric Pair | Correlation |
|---|
| Among online metrics | >0.7 (Spearman) |
| Online vs. offline | ≈ 0 |
This is the key insight for anyone building IT systems for low-resource languages:
How many people speak a language does not predict how alive it is on the internet.
What matters is a small, active, passionate community.
Example: Bashkir has fewer speakers than Chechen, but far more online communities. Why?
- Chechnya's internet infrastructure recovered later (1990s-2000s conflicts delayed digital adoption).
- Online activity is a function of infrastructure, not just population.
For your product strategy: If you're building a language model for a low-resource language, prioritize communities, not census data. Find the VK groups, Discord servers, and Telegram channels. That's where the data lives.
A Few Geeky Findings
| Finding | Implication |
|---|
| Post-2012 shift | Almost all minority language activity moved to VK.com. Standalone forums died. |
| Typical user | 19-31 years old, lives in the region's capital (Ufa for Bashkir, Izhevsk for Udmurt). |
| Typical post length | ~5 words per post in minority language. Most conversation is in Russian; people sprinkle native words as identity markers. |
| Interconnectivity | Minority language sites link to each other across language families. A rare case of "peoples united" online. |
For your NLP models: If you're building embeddings from this data, expect sparse, short texts. Standard word2vec on 5-word posts won't work well. Consider:
- Character-level embeddings (more robust to spelling variation)
- Cross-lingual transfer (use Russian or English as a bridge)
- Zero-shot approaches (label propagation from related languages)
The Data Engineering Pipeline — Summary
┌─────────────────┐
│ Yandex.XML API │ ← 10k requests/day budget
│ (search engine)│
└────────┬────────┘
▼
┌─────────────────┐
│ Word Markers │ ← Frequent + Unique + No diacritics
│ (handcrafted) │ ← "Household spelling" handling
└────────┬────────┘
▼
┌─────────────────┐
│ Domain Filter │ ← Remove Wikipedia, linguistic papers
│ (multi-marker) │ ← Ensemble validation
└────────┬────────┘
▼
┌─────────────────┐
│ Tiered Index │ ← Language-first, Mixed, Giant platforms
│ (stratified) │ ← VK.com special treatment
└────────┬────────┘
▼
┌─────────────────┐
│ Binary Class. │ ← Russian vs. everything else
│ (weak super.) │ ← No pre-labelled corpus needed
└────────┬────────┘
▼
┌─────────────────┐
│ Web Corpora │ ← First ever for these languages
│ (downloadable) │ ← Available at web-corpora.net
└─────────────────┘
The Data
Downloadable corpora and domain lists:
web-corpora.net/wsgi3/minorlangs/download
And yes, there's a VK community with cute memes in minority languages — featuring cats.
The Bottom Line for Engineers
Web scraping isn't just about avoiding robots.txt. It's about understanding human behaviour — typos, keyboard limitations, social network migration, and why a language with 50 speakers can thrive online if those 50 are determined enough.
Key takeaways for building NLP systems for low-resource languages:
- Start with search APIs, not crawlers. The internet is already indexed. Use it.
- Handle real-world spelling. People don't use diacritics. Your queries shouldn't either.
- Use weak supervision. You don't need a labelled corpus to start — use what you have.
- Prioritize communities, not speakers. Online vitality ≠ offline speaker count.
- Expect short texts. ~5-word posts are the norm. Plan your embeddings accordingly.
Further reading:
- Our downloadable corpora: web-corpora.net
- Yandex.XML API documentation
- VK.com API for social media data extraction
Liked this? Share with your NLP engineering team. They'll never look at "low-resource" the same way again.
