Arabic-aware search algorithm

This document describes the search behavior independently of C#, Entity Framework Core, and PostgreSQL. It can be used to reproduce the same behavior in another language or database. It reflects the implementation as of July 30, 2026. The algorithm is a pragmatic text search rather than a linguistic analyzer. It does not stem words, infer roots, transliterate Arabic, or correct arbitrary spelling mistakes. It expands a query into a bounded set of forms that account for:

an accidentally selected Arabic or QWERTY keyboard layout;
common Arabic and Persian forms of kaf and yeh;
alif maqsura;
terminal heh and taa marbuta;
case-insensitive substring matching;
weighted relevance for a primary field, normally a person’s full name.

Terminology

Term	Meaning
Query	The complete text entered by the user.
Term	One non-empty part of the query after splitting it on spaces.
Variation	One alternative representation of a term or phrase.
Searchable field	A text field in which a match may be found.
Primary field	The field given extra ranking weight, such as `fullName`.
Contains match	A variation occurs anywhere in a field.
Prefix match	A field begins with a variation.

Unless stated otherwise, variation sets remove duplicate strings. The current implementation compares variation-set entries case-insensitively at the outer level and exactly at the inner Arabic/Persian expansion levels. A port may use one case-insensitive ordered set throughout if it produces the same observable matches.

High-level process

For ranked search:

Trim the query and split it into terms.
Generate variations for every term.
Generate variations for the complete trimmed phrase.
Keep a record if at least one term variation occurs in at least one searchable field.
Compute an additive relevance score.
Sort records by descending score.
Apply an application-defined tie-breaker, such as descending record ID.

The primary field is also included in the set of searchable fields, even if the caller did not include it explicitly.

rankedSearch(records, query, fields, primaryField):
    terms = splitOnSpaces(query)

    if terms is empty:
        return records ordered by a constant score

    allFields = distinct(fields + primaryField)
    termVariations = [variations(term) for term in terms]
    phraseVariations = variations(trim(query))

    candidates = records where
        any term variation is contained in any allFields value

    for each candidate:
        score = relevanceScore(
            candidate,
            termVariations,
            phraseVariations,
            allFields,
            primaryField
        )

    return candidates ordered by score descending

Query tokenization

Split the query on the ordinary space character (U+0020). Remove empty entries and trim each result. For example:

"  احمد   علي " -> ["احمد", "علي"]

The canonical normalization operation described later can collapse other Unicode whitespace, but ranked search does not currently call it before tokenization. A compatible port should preserve that distinction.

Variation generation

Generate variations for a string in three stages.

1. Initial forms

Start with a case-insensitive set containing:

the original input;
the input interpreted as QWERTY keystrokes and converted to Arabic;
the input interpreted as Arabic-layout keystrokes and converted to QWERTY.

Discard blank results.

initial = distinctIgnoreCase([
    input,
    qwertyToArabic(input),
    arabicToQwerty(input)
])

This is keyboard-layout repair, not transliteration. For example, it is meant to recover text entered while the wrong keyboard layout was active.

2. Arabic and Persian character forms

For each initial form, produce whole-string styles for kaf:

kafForms = distinct([
    value,
    replaceAll(value, "ک", "ك"),
    replaceAll(value, "ك", "ک")
])

For every kaf form, produce whole-string styles for yeh:

yehForms = distinct([
    value,
    replaceAll(value, ["ی", "ى"], "ي"),
    replaceAll(value, ["ي", "ى"], "ی"),
    replaceAll(value, ["ي", "ی"], "ى")
])

These are whole-string styles rather than every possible per-character combination. For example, a string containing several yeh characters gets an all-Arabic-yeh form and an all-Persian-yeh form, not every mixture of the two. This limits query growth.

3. Terminal heh and taa marbuta

Split each form into words. For every word ending in ة or ه, produce both endings while preserving the stem:

terminalForms("فاطمة") = ["فاطمة", "فاطمه"]
terminalForms("فاطمه") = ["فاطمة", "فاطمه"]
terminalForms("احمد")  = ["احمد"]

For a phrase, compute the Cartesian product of the alternatives for all its words, then join each combination with a single space.

endingVariations(value):
    words = splitOnSpaces(value)
    combinations = [[]]

    for word in words:
        if word ends with "ة" or "ه":
            stem = word without its final character
            wordForms = [stem + "ة", stem + "ه"]
        else:
            wordForms = [word]

        combinations = cartesianAppend(combinations, wordForms)

    return distinct(joinWithSingleSpace(x) for x in combinations)

The final variation set is the union of these expansions for all initial forms:

variations(input):
    if input is blank:
        return []

    result = empty case-insensitive set

    for initialForm in initialForms(input):
        for kafForm in kafForms(initialForm):
            for yehForm in yehForms(kafForm):
                for endingForm in endingVariations(yehForm):
                    add endingForm to result

    return result

Keyboard-layout mapping

The mapping represents the Arabic keyboard layout used by the implementation. Conversion leaves characters that do not appear in the table unchanged.

Arabic	QWERTY	Arabic	QWERTY	Arabic	QWERTY
`لا`	`b`	`ض`	`q`	`ص`	`w`
`ث`	`e`	`ق`	`r`	`ف`	`t`
`غ`	`y`	`ع`	`u`	`ه`	`i`
`خ`	`o`	`ح`	`p`	`ج`	`[`
`د`	`]`	`ش`	`a`	`س`	`s`
`ي`	`d`	`ب`	`f`	`ل`	`g`
`ا`	`h`	`ت`	`j`	`ن`	`k`
`م`	`l`	`ك`	`;`	`ط`	`'`
`ئ`	`z`	`ء`	`x`	`ؤ`	`c`
`ر`	`v`	`ى`	`n`	`ة`	`m`
`و`	`,`	`ز`	`.`	`ظ`	`/`
`ذ`	`

When converting Arabic to QWERTY:

recognize لا before processing individual characters and emit b;
treat Persian ک like Arabic ك;
treat Persian ی like Arabic ي.

When converting QWERTY to Arabic:

look up Latin letters in lowercase, so Caps Lock does not prevent repair;
map b to the two-character sequence لا;
preserve unmapped characters.

Matching

For a term, a record matches when any variation is contained in any searchable field:

termMatchesAnywhere(record, termForms):
    return any(
        containsIgnoreCase(fieldValue, form)
        for fieldValue in searchableFieldValues(record)
        for form in termForms
    )

The ranked search filter uses ANY semantics between terms:

recordMatches(record):
    return any(
        termMatchesAnywhere(record, forms)
        for forms in termVariations
    )

Consequently, a query containing two terms retains records matching either term. Matching more terms affects ranking. The current PostgreSQL implementation uses ILIKE:

field ILIKE '%variation%'  -- contains
field ILIKE 'variation%'   -- prefix

Other databases should use an equivalent case-insensitive operation. Exact case-folding behavior may depend on the database, collation, and locale.

Compatibility warning: The current implementation does not escape SQL pattern characters in the query. % and _ therefore act as wildcards under SQL LIKE/ILIKE semantics. A new implementation should either preserve this for strict compatibility or explicitly define and document escaping.

Null field values behave as non-matches.

Ranked relevance score

The score is additive. Every true condition contributes its points, including conditions that overlap. Let:

T0 be the first term;
Ti be a later term at index i;
P be the primary field;
F be all searchable fields, including P.

Add the following values:

Condition	Points
The full phrase is a prefix of `P`	50,000
The full phrase occurs anywhere in `P`	30,000
Every term occurs somewhere in `P`	20,000
`T0` is a prefix of `P`	10,000
`T0` occurs anywhere in `P`	5,000
`T0` occurs anywhere in `F`	2,000
Every term occurs somewhere in `F`	1,000
Each later `Ti` occurs in `P`	100 per term
Each later `Ti` occurs in `F`	20 per term

Each occurrence test means that at least one variation satisfies the condition. It does not count the number of matching variations or repeated occurrences in a field.

relevanceScore(record, termForms, phraseForms, fields, primary):
    score = 0

    primaryValue = value(record, primary)
    allValues = values(record, fields)

    primaryTermMatches = [
        any(containsIgnoreCase(primaryValue, form) for form in forms)
        for forms in termForms
    ]

    anywhereTermMatches = [
        any(
            containsIgnoreCase(fieldValue, form)
            for fieldValue in allValues
            for form in forms
        )
        for forms in termForms
    ]

    firstForms = termForms[0]

    if any(startsWithIgnoreCase(primaryValue, x) for x in phraseForms):
        score += 50_000

    if any(containsIgnoreCase(primaryValue, x) for x in phraseForms):
        score += 30_000

    if all(primaryTermMatches):
        score += 20_000

    if any(startsWithIgnoreCase(primaryValue, x) for x in firstForms):
        score += 10_000

    if primaryTermMatches[0]:
        score += 5_000

    if anywhereTermMatches[0]:
        score += 2_000

    if all(anywhereTermMatches):
        score += 1_000

    for i from 1 to length(termForms) - 1:
        if primaryTermMatches[i]:
            score += 100

        if anywhereTermMatches[i]:
            score += 20

    return score

The first term is deliberately more influential than later terms. A primary field beginning with the first term should generally rank above a record that matches only later refinements elsewhere. Because scoring is additive, a full-phrase prefix normally also receives the full-phrase contains points and several term-level points. This stacking is intentional.

Unranked filtering variant

The implementation also defines an unranked filter. It generates the same per-term variations and supports two ways to combine terms:

strict mode: every term must match at least one searchable field;
loose mode: at least one term must match at least one searchable field.

An additional caller-supplied predicate is always combined with the search predicate using logical AND.

unrankedFilter(records, query, fields, loose, additionalPredicate):
    termConditions = [
        record -> termMatchesAnywhere(record, variations(term))
        for term in splitOnSpaces(query)
    ]

    if loose:
        searchCondition = OR(termConditions)
    else:
        searchCondition = AND(termConditions)

    finalCondition = searchCondition AND additionalPredicate
    return records where finalCondition

If no usable search condition exists, apply only the additional predicate. If neither exists, return the input unchanged.

Canonical text normalization

Canonical normalization is a separate utility. The query-variation algorithm does not currently invoke it automatically. Apply these steps in order:

Return an empty or whitespace-only input trimmed.
Apply Unicode compatibility normalization, NFKC.
Normalize Arabic characters to their Persian forms:
- ك to ک;
- ي to ی;
- ى to ی.
Remove Unicode non-spacing marks and tatweel (ـ).
Apply the following mappings:

Input	Output
`أ`, `إ`, `ٱ`, `آ`	`ا`
`ؤ`	`و`
`ئ`	`ی`
`ة`, `ۀ`	`ه`
ZWNJ (`U+200C`), ZWJ (`U+200D`)	space
Persian digits `۰`–`۹`	ASCII digits `0`–`9`
Arabic-Indic digits `٠`–`٩`	ASCII digits `0`–`9`

Replace every Unicode whitespace run with one ordinary space.
Trim leading and trailing spaces.

normalizeSearchText(input):
    if input is blank:
        return trim(input or "")

    value = unicodeNormalizeNFKC(input)
    value = replace(value, {
        "ك": "ک",
        "ي": "ی",
        "ى": "ی"
    })

    output = ""
    previousWasSpace = false

    for character in value:
        if unicodeCategory(character) == NonSpacingMark:
            continue

        if character == "ـ":
            continue

        character = mapCanonicalCharacterAndDigits(character)
        isSpace = isUnicodeWhitespace(character)

        if isSpace and previousWasSpace:
            continue

        output += " " if isSpace else character
        previousWasSpace = isSpace

    return trim(output)

There are also two narrower conversions:

normalizePersianCharacters:
    ك -> ک
    ي -> ی
    ى -> ی

normalizeArabicCharacters:
    ک -> ك
    ی -> ي
    ى -> ي

Complexity and limits

For each initial form, character expansion produces at most three kaf styles and four yeh styles. Terminal ه/ة alternatives can double for every affected word. Before duplicate removal, the upper bound is approximately:

3 initial forms × 3 kaf forms × 4 yeh forms × 2^E

where E is the number of words ending in ه or ة. For ordinary single-term searches the set stays small, and duplicate removal usually reduces it substantially. Long phrases with many affected endings can grow exponentially. Implementations should set reasonable query-length and term-count limits, or cap generated variations. Database predicate count grows approximately with:

searchable field count × term count × variations per term

Implementations should inspect generated queries and add appropriate database indexes. Leading-wildcard substring searches such as %value% usually cannot use a normal B-tree index efficiently.

Compatibility checklist

A port should test at least:

empty and whitespace-only queries;
Arabic and Persian kaf: ك and ک;
Arabic yeh, Persian yeh, and alif maqsura: ي, ی, and ى;
terminal ه and ة;
Arabic text typed with a QWERTY layout selected, and the reverse;
Caps Lock during QWERTY-to-Arabic repair;
the لا two-character mapping;
null searchable fields;
one-term and multi-term candidate filtering;
overlapping additive ranking conditions;
records tied on relevance score;
% and _ behavior if the backend uses SQL patterns;
canonical normalization of diacritics, tatweel, joiners, whitespace, and Arabic-script digits.

Worked examples

These examples combine the individual rules into complete inputs and outputs. Variation lists labeled “selected” are illustrative subsets because keyboard-layout expansion can add forms that are not relevant to the point being demonstrated.

Canonical normalization

Input:    "  أَحْمَـد  ١٢۳ "
NFKC:     "  أَحْمَـد  ١٢۳ "
Marks:    "  أحمـد  ١٢۳ "
Tatweel:  "  أحمد  ١٢۳ "
Mapping:  "  احمد  123 "
Spaces:   "احمد 123"

The result is:

احمد 123

Wrong keyboard layout

If the user intended to enter احمد but had the QWERTY layout selected, the physical keystrokes appear as hpl].

Input:            hpl]
QWERTY to Arabic: احمد
Selected search variations:
  - hpl]
  - احمد

The repair works in the other direction as well:

Input:            اثممخ
Arabic to QWERTY: hello
Selected search variations:
  - اثممخ
  - hello

This does not translate either word. It reinterprets the same physical keys under the other keyboard layout.

Arabic and Persian forms

For the input علي, the selected variations include:

علي  # Arabic yeh
علی  # Persian yeh
على  # alif maqsura
ugd  # the corresponding physical QWERTY keys

For the input فاطمة, terminal-ending expansion includes:

فاطمة
فاطمه

A stored value using any of these Arabic/Persian forms can therefore satisfy the same contains condition.

Multiword ranking

Assume the query is:

احمد علي

Search fullName and address, with fullName as the primary field:

ID	`fullName`	`address`
A	`احمد علي`	`النجف`
B	`علي احمد`	`بغداد`
C	`علي حسن`	`قرب احمد`

All three records pass the candidate filter:

A contains both terms in the primary field.
B contains both terms in the primary field, but in the reverse order.
C contains علي in the primary field and احمد in another searchable field.

Their scores are:

Condition	A	B	C
Full phrase prefixes primary field	50,000	0	0
Full phrase occurs in primary field	30,000	0	0
Every term occurs in primary field	20,000	20,000	0
First term prefixes primary field	10,000	0	0
First term occurs in primary field	5,000	5,000	0
First term occurs in any field	2,000	2,000	2,000
Every term occurs in any field	1,000	1,000	1,000
Later term occurs in primary field	100	100	100
Later term occurs in any field	20	20	20
Total	118,120	28,120	3,120

The resulting order is:

A, B, C

This example also shows that conditions stack. Record A receives both the full-phrase prefix and full-phrase contains points, while a match in the primary field also counts as a match in any searchable field.

Strict and loose filtering

Using the same records and the query احمد علي:

Strict mode (all terms must match):
  A, B, C

Loose mode (at least one term must match):
  A, B, C

If record C instead had address = "كربلاء", it would match only علي:

Strict mode:
  A, B

Loose mode:
  A, B, C

Overview

People

Reference

Arabic-aware search algorithm

Arabic-aware search algorithm

Terminology

High-level process

Query tokenization

Variation generation

1. Initial forms

2. Arabic and Persian character forms

3. Terminal heh and taa marbuta

Keyboard-layout mapping

Matching

Ranked relevance score

Unranked filtering variant

Canonical text normalization

Complexity and limits

Compatibility checklist

Worked examples

Canonical normalization

Wrong keyboard layout

Arabic and Persian forms

Multiword ranking

Strict and loose filtering

​Arabic-aware search algorithm

​Terminology

​High-level process

​Query tokenization

​Variation generation

​1. Initial forms

​2. Arabic and Persian character forms

​3. Terminal heh and taa marbuta

​Keyboard-layout mapping

​Matching

​Ranked relevance score

​Unranked filtering variant

​Canonical text normalization

​Complexity and limits

​Compatibility checklist

​Worked examples

​Canonical normalization

​Wrong keyboard layout

​Arabic and Persian forms

​Multiword ranking

​Strict and loose filtering

Arabic-aware search algorithm

Terminology

High-level process

Query tokenization

Variation generation

1. Initial forms

2. Arabic and Persian character forms

3. Terminal heh and taa marbuta

Keyboard-layout mapping

Matching

Ranked relevance score

Unranked filtering variant

Canonical text normalization

Complexity and limits

Compatibility checklist

Worked examples

Canonical normalization

Wrong keyboard layout

Arabic and Persian forms

Multiword ranking

Strict and loose filtering