Two hundred and fifty thousand articles, and the thing editors complained about was the search box. Not the accuracy of it. The wait. Type a name into the newsroom's admin search, go and do something else, come back.
That is the easy complaint, because it arrives. Someone tells you. What never arrives is the other one, and it took me far too long to notice it was missing: nobody has ever told me the search came back empty for a story we definitely published. Nobody ever will. A search that returns nothing does not look like a bug. It looks like proof you never covered it.
So we fixed the loud one, and left the quiet one running for years.
It is the same seam as an earlier piece on Arabic slugs: Arabic quietly breaking an assumption WordPress never knew it was making. Last time it was the URL. This time it is the search box.
What WordPress actually does with an Arabic query
Type a term into WordPress search and WP_Query builds this, near enough:
WHERE post_title LIKE '%term%'
OR post_excerpt LIKE '%term%'
OR post_content LIKE '%term%'
That is the whole mechanism. Not a search engine, not an index, a substring test run three times per row. It knows nothing about words, and nothing at all about Arabic.
We measured what that means on a throwaway WordPress 7.0.2 and MySQL 8.0.35: one post per spelling, then the obvious searches.
| you search | you get back |
|---|
| محمد | only the plain post |
| مُحَمَّد (with tashkeel) | only the tashkeel post |
| محمـــد (with tatweel) | only the tatweel post |
| أحمد (hamza alef) | only the hamza post |
| احمد (bare alef) | only the bare alef post |
| مصطفى (alef maqsura) | only the maqsura post |
| مصطفي (yaa) | only the yaa post |
| مكتبه (haa for taa marbuta) | nothing at all |
Read that right-hand column again. Every search returns only its own exact spelling, and the last returns nothing at all, even though every term on the left is the same word to a human being. A reader who types a name without diacritics never sees the article that has them. Someone who writes أحمد never finds احمد. Someone who ends a word with ه instead of ة gets an empty page.
The one result that looks like a success is the exception that proves the point. Searching كتاب does return الكتاب. That is not the search understanding the definite article, it is the letters ك ت ا ب happening to sit inside الكتاب as a literal substring. Search وكتاب, the same word with a prefixed و, and you get nothing, because the bare word is no longer embedded in it.
The admin post list behaves identically to the front end, which is the part that matters, because that is where your editors live.
The same word, written three ways, and a reader means one thing by all of them. WordPress compares raw bytes with LIKE, so the three stay three different strings and none finds the others. Fold them with the same normaliser and they collapse to a single token that matches every time.
The database already knows. WordPress never asks it.
This is the part that genuinely surprised me.
WordPress does not inherit your server's collation. It forces its own onto its tables, utf8mb4_unicode_520_ci, whatever MySQL's default happens to be. And that collation is not naive about Arabic. Ask MySQL directly, with an equality test:
SELECT 'مُحَمَّد' = 'محمد' COLLATE utf8mb4_unicode_520_ci; -- 1
One. True. The collation folds the diacritics away and says these are the same string. On MySQL 8's utf8mb4_0900_ai_ci it folds tatweel as well.
Now ask it the way WordPress asks it:
SELECT 'مُحَمَّد' LIKE '%محمد%' COLLATE utf8mb4_unicode_520_ci; -- 0
Zero. Same data, same collation, same server, opposite answer. Wildcard pattern matching does not apply the collation's character folding, so every piece of Arabic knowledge sitting in your database is thrown away the moment you wrap the term in %. We tested four collations. Under =, the good ones fold. Under LIKE '%…%', all four return zero.
The capability was already in the building. The search architecture walks straight past it.
The identical pair, the diacritic-marked spelling and the plain one, under WordPress's own collation. Compared with =, the collation folds the tashkeel away and returns 1: they are equal. Compared with LIKE and a wildcard, that folding is skipped and it returns 0. Same data, same collation, opposite answer, and WordPress search asks with LIKE.
Even reaching it would not be enough
It is tempting to stop here and go looking for a way to make WordPress compare with =. It would not save you, and the measurements show why.
A collation folds accents. Tashkeel are accents, combining marks that sit on top of a letter, which is exactly why they fold.
Hamza is not an accent. أ and ا are different letters. So are ى and ي. So are ة and ه. None of them fold, in any collation we tested, under any comparison. And these are not edge cases in Arabic, they are the ordinary daily variation: the same name spelled أحمد by one editor and احمد by another, the same word ending ى in one headline and ي in the next.
So the database will hand you diacritics for free if you ask it correctly, and it will never hand you the rest. Anything that actually works has to normalise the text itself, deliberately, on both sides of the comparison.
What it costs, and why the broken searches are the expensive ones
The other half of this is the half your editors do complain about.
A LIKE with a leading wildcard cannot use an index. Not "is slow to use one", cannot use one, ever. EXPLAIN on the query WordPress generates says it plainly:
type: ALL key: NULL rows: 50013
Full scan, no key, every row. We seeded 50,000 realistic Arabic posts and timed it with a fresh PHP process per run, because repeating it inside one process makes WordPress's in-request cache report a fraction of a millisecond, which is a lie:
- a term matching one post: about 215 ms
- a term matching nothing: about 225 ms
The search that finds nothing is the slowest, and that is not a rounding artefact. LIKE '%term%' gives up on a row the instant it finds the substring, so a row that matches is cheap: often the title matches and the long body is never read at all. A row that does not match has to be read to the last byte of every field before MySQL can be certain the word is absent. Proving absence is the expensive case. And in Arabic, the searches that come back empty are precisely the variant spellings in the table above.
You might expect LIMIT 10 to rescue this. It cannot, and MySQL says so in plain words. WordPress needs a total for pagination, so it asks for SQL_CALC_FOUND_ROWS, and the query plan reports the consequence itself:
Limit: 10 row(s) (no early end due to SQL_CALC_FOUND_ROWS)
Ten results on the page, every matching row enumerated to produce them.
The scan cost tracks table size, so a 250,000-article archive lands somewhere around a second per search, per concurrent searcher, with no index able to help. (That figure is an extrapolation from the 50k measurement, not a measurement of a 250k table.)
Now put the two halves together. The searches that silently return nothing are the same searches that scan your entire archive in order to return nothing. In Arabic, those are every variant spelling your readers and editors actually type. The complaint you get and the complaint you never get are the same query.
What we built, and what it fixed
We did the obvious right thing, and it worked.
We built a table outside WordPress, on AWS, one row per article: id, title, body, author, categories. Every time a post was published, updated or deleted, that row was kept in step. Search stopped asking the WordPress database to scan a quarter of a million rows and started asking a store built for the job. Public search had already been handed off to a Google Custom Search Engine, so between the two, search stopped being the thing anyone raised in a meeting.
(That table later grew into something much larger, a recommendation engine that learns what a reader is interested in and chooses their next article. That is a story for another day.)
By every measure we were watching, we had fixed search.
What it did not fix
We never normalised the Arabic.
The new table was faster. It was still storing and matching the same raw strings, with the same orthographic variation, in the same literal way. A reader typing محمد still missed مُحَمَّد. أحمد still missed احمد. مكتبه still found nothing. We had rebuilt the entire search path, moved it to another cloud, and put a proper store underneath it, and every one of those searches failed exactly as it had before.
That is the lesson, and it took us years to see it: the speed problem was an infrastructure problem, and the matching problem never was. Scale, indexes and a store designed for querying will fix a scan. None of them will turn two spellings of the same word into the same token. Normalisation is a text problem, and if you do not solve it, it follows you to AWS.
And because nobody ever complains about a search that returns nothing, nothing ever forced the issue. The loud problem got an entire re-architecture. The quiet one just carried on.
The layer we were missing
The fix is small, and it is the same fix wherever your index lives.
Normalise both sides. Keep a normalised copy of the searchable text, index that, and put every incoming query through the identical normaliser before matching. Both halves matter: normalise only the stored text and a query carrying diacritics still misses, normalise only the query and it misses the stored variants.
What the normaliser has to do:
- Strip the diacritics. Tashkeel are combining marks (U+064B to U+0652 and friends). Remove them entirely.
- Strip tatweel (U+0640). It is decoration and carries no meaning.
- Unify the alef forms. أ إ آ ٱ all become ا.
- Unify ى to ي, and decide about ة to ه. These are conventions, not truths. Pick one, apply it to both sides, write it down.
- Strip the definite article الـ, so الكتاب and كتاب fold to one token. Guard it, so it only comes off when enough of the word is left to still be a word, otherwise ال on the front of a short word gets eaten too. And stop there. It is tempting to strip the single-letter prefixes و ف ب ك ل as well, but those letters begin ordinary words, so بيت would lose its ب and فكرة would become كرة, a genuinely different word. So وكتاب will not fold to كتاب, and that conservative choice is the correct one.
- Leave Latin alone. Newsroom copy is full of WordPress, AWS, S3 and half-transliterated terms. A normaliser that mangles those breaks more than it fixes.
Do that and the table from earlier collapses into a single row: every spelling of محمد, however it was typed, becomes one token that matches all the others. And because the normalised copy is matched through an index instead of a leading-wildcard scan, the query that used to take a second stops scanning the table at all.
And none of this needs a cloud. The story above happened to run through a table we put on AWS, but that was our answer to scale, not to matching. On an ordinary single-server WordPress, the same normalised copy in the same database, with the query folded the same way, fixes both the misses and the scan with nothing external at all. That is the layer we were missing, and it would have been missing just as badly on one server as on twenty.
One change, both complaints.
The plugin
The whole fix is one file. It goes in wp-content/mu-plugins/, and in its default mode it needs nothing else: no cloud, no service, no configuration.
composer require mantekio/wp-arabic-search
On save it folds everything searchable about a post, title, excerpt and body, into one normalised column in a shadow table with a real FULLTEXT index, and rewrites WordPress search to match against that column through the identical normaliser. The public site and the wp-admin post list both run through WP_Query, so both are fixed at once. Build the index over what you already have with one command:
wp arabic-search reindex
Media gets the same treatment, which turned out to matter more than expected. An attachment keeps its text in four places: the title, the caption, the alt text and the filename. All four go into the same column, so a picture captioned in Arabic is findable by that caption, and صورة-بيروت.jpg is findable by its own name. WordPress does not search alt text at all, so on that one the plugin is not restoring parity, it is adding something core never had. This changes what the Media Library can find, not what appears anywhere else: the plugin decides which rows match the text, and WordPress still decides which post types and statuses are allowed in the results.
It ships with two modes, because the normaliser is the reusable part and the index is not. The default, index, is the self-contained one just described. The other, normaliser, maintains nothing and rewrites nothing: it exposes the normalising function and fires an action carrying the normalised text on every save, so a pipeline that indexes somewhere else (the AWS table we built, OpenSearch, anything) can fold its stored text and its queries with the same code. That is the guarantee that actually matters, and it is the one we did not have for years: both sides normalised by identical code.
define( 'WPAS_MODE', 'normaliser' );
One thing it deliberately refuses to do. InnoDB does not index words shorter than three characters, which in Arabic covers من, في, ما and عن: real words, typed constantly. An index that has never seen a word cannot find it, so requiring one would return an empty page for a search plain WordPress answers perfectly well. That is exactly the failure this article is about, and it would have been ours. So the plugin drops those tokens rather than requiring them, and where a search contains nothing else it hands the query back to core's LIKE untouched. Search a two-letter word and you get precisely what WordPress would have given you; search two words where only one clears the threshold and the long one still matches. The rule it follows is that where it cannot help, it stands aside rather than answer worse than the search it replaced. There is nothing to configure for this, and no reason to go changing innodb_ft_min_token_size on your server: it needs a MySQL restart and a rebuild of every full-text index on the box, and it buys you nothing here.
Which raises the question any search tool eventually gets asked, usually by an editor who is certain the article exists: why did this not match? Answering it used to mean reading the source, so there is now a command that just tells you.
$ wp arabic-search explain "من المكتبة"
normalised: من مكتبه
+--------+------+--------------------------------------------------+
| token | used | reason |
+--------+------+--------------------------------------------------+
| من | no | shorter than innodb_ft_min_token_size (3) |
| مكتبه | yes | indexable |
+--------+------+--------------------------------------------------+
matches with: MATCH(search_text) AGAINST ('+مكتبه' IN BOOLEAN MODE)
The folded form, every token with whether it was used and why not, and the exact query that runs. It is the smallest feature in the plugin and the one I would miss most, because "search is broken" and "search is working exactly as designed on a word the index cannot hold" look identical from the outside, and this is the difference between them.
We eventually ran it against the real thing: a quarter of a million Arabic articles, plus half a million media items, on a copy of a live newsroom archive. One search settles the whole argument.
| searching مُحَمَّد across 250,000 articles | results | time |
|---|
| WordPress | 1 | 4.8 s |
| with the plugin | 28,238 | 0.55 s |
One. Out of twenty-eight thousand articles that contain that name, plain WordPress found a single one, because only one of them happened to spell it with exactly those diacritics. The other 28,237 were in the database the whole time, and no one searching for that name would ever have seen them.
The speed is the part you would have put in the ticket. The other column is the part nobody was ever going to report.
The rest of the numbers come from a controlled 50,000-post rig, where the match count can be dialled precisely, before and after:
| matches, out of 50,000 | default WordPress | with the plugin |
|---|
| 1 | about 215 ms | 5 ms |
| 500 (1%) | about 212 ms | 10 ms |
| 5,000 (10%) | about 230 ms | 24 ms |
| 25,000 (50%) | about 250 ms | 115 ms |
| 50,000 (100%) | about 310 ms | about 210 ms |
EXPLAIN tells the same story as the clock. Before, every search was the full scan:
type: ALL key: NULL rows: 50013
After, the index drives it:
type: fulltext key: search_ft rows: 1
We had that bottom row wrong for a while, and the way it was wrong is worth repeating, because it is a trap set for anyone who benchmarks a text search. Our first measurement said the index lost on a term matching everything. It did not. The word we had seeded happened to sit near the start of every document, and since LIKE abandons a row the moment it finds a match, we were timing the scan at its luckiest: first few bytes, done, next row. Move the same word to the end of the same documents, and the result inverts.
| a term matching all 50,000 rows | default WordPress | with the plugin |
|---|
| the word sits at the start of every document | 140 ms | 208 ms |
| the word sits at the end of every document | 300 ms | 205 ms |
Read the plugin column first: it barely moves, because an index does not care where in the text the word happens to sit. Now read the other one, which doubles. We had benchmarked the scan's single luckiest case and called it a fair fight.
So the honest shape is a crossover, not a caveat. Against a term the scan actually has to read, the index wins across the entire range in the table above. Against a term positioned for the scan's best possible case, the index still wins until roughly 70% of the archive matches. A query that matches seventy per cent of a newsroom archive is not a search, it is a SELECT * with extra steps.
And the case that used to hurt most is now the cheapest. The old worst case was the search that came back empty, because proving absence meant reading every byte of every row. An index that finds nothing has nothing to read.
One honest word on where it is in its life. It is new, and it ships as 0.9.3. That it is already on its fourth release says something worth saying plainly: six rounds of adversarial testing found a bug every single time, and not one of them was found by reading the code.
The worst was in the first release. Search a two-letter Arabic word, من or في, and it returned nothing at all. The check meant to drop words too short for the index counted bytes, and an Arabic letter is two bytes, so every one of those words sailed through and became a required term the index had never held. A plugin whose entire argument is that Arabic search silently returns nothing had shipped a way to make Arabic search silently return nothing. Others followed: a health check that could not detect the condition it existed to detect, and a reindex command that announced success having written not one row.
They are all fixed, and every fix is in the changelog, but the pattern is the point. Each one was invisible from the code and obvious the moment something actually ran it. If this piece has a moral it is that one, and it applies to your search too: the failure that returns nothing looks exactly like the absence of anything to find.
What it has not yet had is months in a live newsroom, so it is out in the open under the GPL, on GitHub and Packagist, being hardened in daylight. The repository is github.com/mantekio/wp-arabic-search. Same loop as the pieces before it: the article explains the thinking, the repository is the plugin, the package is the one-line install, and each points at the other two.
Known limits
Being clear about what a tool does not do is part of the tool.
- Single-letter prefixes are deliberately not stripped. Folding the definite article is safe behind a length guard; folding و ف ب ك ل is not, because they also start ordinary words, so it would turn بيت into يت and فكرة into كرة. The cost of that safety is that وكتاب does not fold to كتاب. Real prefix handling needs morphology, not a character map.
- Folding is not stemming. It will not connect كتاب to its broken plural كتب, because Arabic plurals rewrite the word rather than append to it. That needs a root dictionary, and that is a different project.
- The results are only as complete as the index. Nothing walks your archive when you install it, so run
wp arabic-search reindex once. After that each save keeps its own row current, and wp arabic-search status exits non-zero the moment the index falls behind, goes stale, or carries rows for posts that no longer exist. Drift is something you get told about rather than something a reader discovers for you.
- It costs disk, inside the database. A second normalised copy of your searchable text is still a copy. On that quarter-million-article archive the table came to about 1.1 GB, roughly 3 KB per article, though the
FULLTEXT index on top of it was only 17 MB. Media is nearly free: attachments were two thirds of the rows and a sixteenth of the bytes.
- Words the index cannot hold are correct, not fast. A search that falls back gets core's behaviour entirely, including core's speed, because it is core's query. On that same archive a two-letter word took about three seconds, exactly as it always had, while an indexed search of the same corpus took well under one. Nothing is worse than before. Nothing is better either.
- Normalisation is lossy on purpose. Folding ة into ه, or ى into ي, will occasionally merge two words that genuinely differ. In an archive search that trade is nearly always worth it, but it is a trade: decide it deliberately and document it, rather than discovering it later.
- Past a certain size, use a real search engine. OpenSearch ships an
arabic analyser that does normalisation, stemming and stopwords properly, and gives you relevance ranking a LIKE will never have. If the archive is large and search matters, that is the destination. You still need the normaliser, because the same text problem travels with you. Ours did.
A quarter of a million articles, and the only thing anyone ever told us about search was that it was slow. So we made it fast. We built the table, moved the queries to AWS, kept it in sync on every post, the complaints stopped, and for a long time we took that to mean we had finished.
The searches that returned nothing never produced a single complaint, because a search that returns nothing looks exactly like a subject you never covered. The archive was full of articles nobody could find, and the silence around them read as success.
That is the part worth keeping, whatever your stack: complaints tell you about the loud failures and nothing at all about the quiet ones. Someone has to go and look.
That is how we run WordPress on AWS for newsrooms: fix what people report, then go hunting for what they cannot. If your archive is in Arabic and you have never tested what your search does with a hamza, let's talk.
