> ## Documentation Index > Fetch the complete documentation index at: https://api-docs.taeh.org/llms.txt > Use this file to discover all available pages before exploring further. # Arabic-aware search algorithm > A language-neutral specification of Taeh's Arabic/Persian query expansion, matching, and ranking behavior. # Arabic-aware search algorithm This document describes the search behavior independently of C#, Entity Framework Core, and PostgreSQL. It can be used to reproduce the same behavior in another language or database. It reflects the implementation as of July 30, 2026. The algorithm is a pragmatic text search rather than a linguistic analyzer. It does not stem words, infer roots, transliterate Arabic, or correct arbitrary spelling mistakes. It expands a query into a bounded set of forms that account for: * an accidentally selected Arabic or QWERTY keyboard layout; * common Arabic and Persian forms of kaf and yeh; * alif maqsura; * terminal heh and taa marbuta; * case-insensitive substring matching; * weighted relevance for a primary field, normally a person's full name. ## Terminology | Term | Meaning | | ---------------- | ------------------------------------------------------------- | | Query | The complete text entered by the user. | | Term | One non-empty part of the query after splitting it on spaces. | | Variation | One alternative representation of a term or phrase. | | Searchable field | A text field in which a match may be found. | | Primary field | The field given extra ranking weight, such as `fullName`. | | Contains match | A variation occurs anywhere in a field. | | Prefix match | A field begins with a variation. | Unless stated otherwise, variation sets remove duplicate strings. The current implementation compares variation-set entries case-insensitively at the outer level and exactly at the inner Arabic/Persian expansion levels. A port may use one case-insensitive ordered set throughout if it produces the same observable matches. ## High-level process For ranked search: 1. Trim the query and split it into terms. 2. Generate variations for every term. 3. Generate variations for the complete trimmed phrase. 4. Keep a record if at least one term variation occurs in at least one searchable field. 5. Compute an additive relevance score. 6. Sort records by descending score. 7. Apply an application-defined tie-breaker, such as descending record ID. The primary field is also included in the set of searchable fields, even if the caller did not include it explicitly. ```text theme={null} rankedSearch(records, query, fields, primaryField): terms = splitOnSpaces(query) if terms is empty: return records ordered by a constant score allFields = distinct(fields + primaryField) termVariations = [variations(term) for term in terms] phraseVariations = variations(trim(query)) candidates = records where any term variation is contained in any allFields value for each candidate: score = relevanceScore( candidate, termVariations, phraseVariations, allFields, primaryField ) return candidates ordered by score descending ``` ## Query tokenization Split the query on the ordinary space character (`U+0020`). Remove empty entries and trim each result. For example: ```text theme={null} " احمد علي " -> ["احمد", "علي"] ``` The canonical normalization operation described later can collapse other Unicode whitespace, but ranked search does not currently call it before tokenization. A compatible port should preserve that distinction. ## Variation generation Generate variations for a string in three stages. ### 1. Initial forms Start with a case-insensitive set containing: 1. the original input; 2. the input interpreted as QWERTY keystrokes and converted to Arabic; 3. the input interpreted as Arabic-layout keystrokes and converted to QWERTY. Discard blank results. ```text theme={null} initial = distinctIgnoreCase([ input, qwertyToArabic(input), arabicToQwerty(input) ]) ``` This is keyboard-layout repair, not transliteration. For example, it is meant to recover text entered while the wrong keyboard layout was active. ### 2. Arabic and Persian character forms For each initial form, produce whole-string styles for kaf: ```text theme={null} kafForms = distinct([ value, replaceAll(value, "ک", "ك"), replaceAll(value, "ك", "ک") ]) ``` For every kaf form, produce whole-string styles for yeh: ```text theme={null} yehForms = distinct([ value, replaceAll(value, ["ی", "ى"], "ي"), replaceAll(value, ["ي", "ى"], "ی"), replaceAll(value, ["ي", "ی"], "ى") ]) ``` These are whole-string styles rather than every possible per-character combination. For example, a string containing several yeh characters gets an all-Arabic-yeh form and an all-Persian-yeh form, not every mixture of the two. This limits query growth. ### 3. Terminal heh and taa marbuta Split each form into words. For every word ending in `ة` or `ه`, produce both endings while preserving the stem: ```text theme={null} terminalForms("فاطمة") = ["فاطمة", "فاطمه"] terminalForms("فاطمه") = ["فاطمة", "فاطمه"] terminalForms("احمد") = ["احمد"] ``` For a phrase, compute the Cartesian product of the alternatives for all its words, then join each combination with a single space. ```text theme={null} endingVariations(value): words = splitOnSpaces(value) combinations = [[]] for word in words: if word ends with "ة" or "ه": stem = word without its final character wordForms = [stem + "ة", stem + "ه"] else: wordForms = [word] combinations = cartesianAppend(combinations, wordForms) return distinct(joinWithSingleSpace(x) for x in combinations) ``` The final variation set is the union of these expansions for all initial forms: ```text theme={null} variations(input): if input is blank: return [] result = empty case-insensitive set for initialForm in initialForms(input): for kafForm in kafForms(initialForm): for yehForm in yehForms(kafForm): for endingForm in endingVariations(yehForm): add endingForm to result return result ``` ## Keyboard-layout mapping The mapping represents the Arabic keyboard layout used by the implementation. Conversion leaves characters that do not appear in the table unchanged. | Arabic | QWERTY | Arabic | QWERTY | Arabic | QWERTY | | ------ | ------- | ------ | ------ | ------ | ------ | | `لا` | `b` | `ض` | `q` | `ص` | `w` | | `ث` | `e` | `ق` | `r` | `ف` | `t` | | `غ` | `y` | `ع` | `u` | `ه` | `i` | | `خ` | `o` | `ح` | `p` | `ج` | `[` | | `د` | `]` | `ش` | `a` | `س` | `s` | | `ي` | `d` | `ب` | `f` | `ل` | `g` | | `ا` | `h` | `ت` | `j` | `ن` | `k` | | `م` | `l` | `ك` | `;` | `ط` | `'` | | `ئ` | `z` | `ء` | `x` | `ؤ` | `c` | | `ر` | `v` | `ى` | `n` | `ة` | `m` | | `و` | `,` | `ز` | `.` | `ظ` | `/` | | `ذ` | `` ` `` | | | | | When converting Arabic to QWERTY: * recognize `لا` before processing individual characters and emit `b`; * treat Persian `ک` like Arabic `ك`; * treat Persian `ی` like Arabic `ي`. When converting QWERTY to Arabic: * look up Latin letters in lowercase, so Caps Lock does not prevent repair; * map `b` to the two-character sequence `لا`; * preserve unmapped characters. ## Matching For a term, a record matches when any variation is contained in any searchable field: ```text theme={null} termMatchesAnywhere(record, termForms): return any( containsIgnoreCase(fieldValue, form) for fieldValue in searchableFieldValues(record) for form in termForms ) ``` The ranked search filter uses `ANY` semantics between terms: ```text theme={null} recordMatches(record): return any( termMatchesAnywhere(record, forms) for forms in termVariations ) ``` Consequently, a query containing two terms retains records matching either term. Matching more terms affects ranking. The current PostgreSQL implementation uses `ILIKE`: ```sql theme={null} field ILIKE '%variation%' -- contains field ILIKE 'variation%' -- prefix ``` Other databases should use an equivalent case-insensitive operation. Exact case-folding behavior may depend on the database, collation, and locale. > **Compatibility warning:** The current implementation does not escape SQL > pattern characters in the query. `%` and `_` therefore act as wildcards > under SQL `LIKE`/`ILIKE` semantics. A new implementation should either > preserve this for strict compatibility or explicitly define and document > escaping. Null field values behave as non-matches. ## Ranked relevance score The score is additive. Every true condition contributes its points, including conditions that overlap. Let: * `T0` be the first term; * `Ti` be a later term at index `i`; * `P` be the primary field; * `F` be all searchable fields, including `P`. Add the following values: | Condition | Points | | -------------------------------------- | -----------: | | The full phrase is a prefix of `P` | 50,000 | | The full phrase occurs anywhere in `P` | 30,000 | | Every term occurs somewhere in `P` | 20,000 | | `T0` is a prefix of `P` | 10,000 | | `T0` occurs anywhere in `P` | 5,000 | | `T0` occurs anywhere in `F` | 2,000 | | Every term occurs somewhere in `F` | 1,000 | | Each later `Ti` occurs in `P` | 100 per term | | Each later `Ti` occurs in `F` | 20 per term | Each occurrence test means that at least one variation satisfies the condition. It does not count the number of matching variations or repeated occurrences in a field. ```text theme={null} relevanceScore(record, termForms, phraseForms, fields, primary): score = 0 primaryValue = value(record, primary) allValues = values(record, fields) primaryTermMatches = [ any(containsIgnoreCase(primaryValue, form) for form in forms) for forms in termForms ] anywhereTermMatches = [ any( containsIgnoreCase(fieldValue, form) for fieldValue in allValues for form in forms ) for forms in termForms ] firstForms = termForms[0] if any(startsWithIgnoreCase(primaryValue, x) for x in phraseForms): score += 50_000 if any(containsIgnoreCase(primaryValue, x) for x in phraseForms): score += 30_000 if all(primaryTermMatches): score += 20_000 if any(startsWithIgnoreCase(primaryValue, x) for x in firstForms): score += 10_000 if primaryTermMatches[0]: score += 5_000 if anywhereTermMatches[0]: score += 2_000 if all(anywhereTermMatches): score += 1_000 for i from 1 to length(termForms) - 1: if primaryTermMatches[i]: score += 100 if anywhereTermMatches[i]: score += 20 return score ``` The first term is deliberately more influential than later terms. A primary field beginning with the first term should generally rank above a record that matches only later refinements elsewhere. Because scoring is additive, a full-phrase prefix normally also receives the full-phrase contains points and several term-level points. This stacking is intentional. ## Unranked filtering variant The implementation also defines an unranked filter. It generates the same per-term variations and supports two ways to combine terms: * strict mode: every term must match at least one searchable field; * loose mode: at least one term must match at least one searchable field. An additional caller-supplied predicate is always combined with the search predicate using logical `AND`. ```text theme={null} unrankedFilter(records, query, fields, loose, additionalPredicate): termConditions = [ record -> termMatchesAnywhere(record, variations(term)) for term in splitOnSpaces(query) ] if loose: searchCondition = OR(termConditions) else: searchCondition = AND(termConditions) finalCondition = searchCondition AND additionalPredicate return records where finalCondition ``` If no usable search condition exists, apply only the additional predicate. If neither exists, return the input unchanged. ## Canonical text normalization Canonical normalization is a separate utility. The query-variation algorithm does not currently invoke it automatically. Apply these steps in order: 1. Return an empty or whitespace-only input trimmed. 2. Apply Unicode compatibility normalization, NFKC. 3. Normalize Arabic characters to their Persian forms: * `ك` to `ک`; * `ي` to `ی`; * `ى` to `ی`. 4. Remove Unicode non-spacing marks and tatweel (`ـ`). 5. Apply the following mappings: | Input | Output | | ------------------------------- | -------------------- | | `أ`, `إ`, `ٱ`, `آ` | `ا` | | `ؤ` | `و` | | `ئ` | `ی` | | `ة`, `ۀ` | `ه` | | ZWNJ (`U+200C`), ZWJ (`U+200D`) | space | | Persian digits `۰`–`۹` | ASCII digits `0`–`9` | | Arabic-Indic digits `٠`–`٩` | ASCII digits `0`–`9` | 6. Replace every Unicode whitespace run with one ordinary space. 7. Trim leading and trailing spaces. ```text theme={null} normalizeSearchText(input): if input is blank: return trim(input or "") value = unicodeNormalizeNFKC(input) value = replace(value, { "ك": "ک", "ي": "ی", "ى": "ی" }) output = "" previousWasSpace = false for character in value: if unicodeCategory(character) == NonSpacingMark: continue if character == "ـ": continue character = mapCanonicalCharacterAndDigits(character) isSpace = isUnicodeWhitespace(character) if isSpace and previousWasSpace: continue output += " " if isSpace else character previousWasSpace = isSpace return trim(output) ``` There are also two narrower conversions: ```text theme={null} normalizePersianCharacters: ك -> ک ي -> ی ى -> ی normalizeArabicCharacters: ک -> ك ی -> ي ى -> ي ``` ## Complexity and limits For each initial form, character expansion produces at most three kaf styles and four yeh styles. Terminal `ه`/`ة` alternatives can double for every affected word. Before duplicate removal, the upper bound is approximately: ```text theme={null} 3 initial forms × 3 kaf forms × 4 yeh forms × 2^E ``` where `E` is the number of words ending in `ه` or `ة`. For ordinary single-term searches the set stays small, and duplicate removal usually reduces it substantially. Long phrases with many affected endings can grow exponentially. Implementations should set reasonable query-length and term-count limits, or cap generated variations. Database predicate count grows approximately with: ```text theme={null} searchable field count × term count × variations per term ``` Implementations should inspect generated queries and add appropriate database indexes. Leading-wildcard substring searches such as `%value%` usually cannot use a normal B-tree index efficiently. ## Compatibility checklist A port should test at least: * empty and whitespace-only queries; * Arabic and Persian kaf: `ك` and `ک`; * Arabic yeh, Persian yeh, and alif maqsura: `ي`, `ی`, and `ى`; * terminal `ه` and `ة`; * Arabic text typed with a QWERTY layout selected, and the reverse; * Caps Lock during QWERTY-to-Arabic repair; * the `لا` two-character mapping; * null searchable fields; * one-term and multi-term candidate filtering; * overlapping additive ranking conditions; * records tied on relevance score; * `%` and `_` behavior if the backend uses SQL patterns; * canonical normalization of diacritics, tatweel, joiners, whitespace, and Arabic-script digits. ## Worked examples These examples combine the individual rules into complete inputs and outputs. Variation lists labeled "selected" are illustrative subsets because keyboard-layout expansion can add forms that are not relevant to the point being demonstrated. ### Canonical normalization ```text theme={null} Input: " أَحْمَـد ١٢۳ " NFKC: " أَحْمَـد ١٢۳ " Marks: " أحمـد ١٢۳ " Tatweel: " أحمد ١٢۳ " Mapping: " احمد 123 " Spaces: "احمد 123" ``` The result is: ```text theme={null} احمد 123 ``` ### Wrong keyboard layout If the user intended to enter `احمد` but had the QWERTY layout selected, the physical keystrokes appear as `hpl]`. ```text theme={null} Input: hpl] QWERTY to Arabic: احمد Selected search variations: - hpl] - احمد ``` The repair works in the other direction as well: ```text theme={null} Input: اثممخ Arabic to QWERTY: hello Selected search variations: - اثممخ - hello ``` This does not translate either word. It reinterprets the same physical keys under the other keyboard layout. ### Arabic and Persian forms For the input `علي`, the selected variations include: ```text theme={null} علي # Arabic yeh علی # Persian yeh على # alif maqsura ugd # the corresponding physical QWERTY keys ``` For the input `فاطمة`, terminal-ending expansion includes: ```text theme={null} فاطمة فاطمه ``` A stored value using any of these Arabic/Persian forms can therefore satisfy the same contains condition. ### Multiword ranking Assume the query is: ```text theme={null} احمد علي ``` Search `fullName` and `address`, with `fullName` as the primary field: | ID | `fullName` | `address` | | -- | ---------- | ---------- | | A | `احمد علي` | `النجف` | | B | `علي احمد` | `بغداد` | | C | `علي حسن` | `قرب احمد` | All three records pass the candidate filter: * A contains both terms in the primary field. * B contains both terms in the primary field, but in the reverse order. * C contains `علي` in the primary field and `احمد` in another searchable field. Their scores are: | Condition | A | B | C | | ----------------------------------- | ----------: | ---------: | --------: | | Full phrase prefixes primary field | 50,000 | 0 | 0 | | Full phrase occurs in primary field | 30,000 | 0 | 0 | | Every term occurs in primary field | 20,000 | 20,000 | 0 | | First term prefixes primary field | 10,000 | 0 | 0 | | First term occurs in primary field | 5,000 | 5,000 | 0 | | First term occurs in any field | 2,000 | 2,000 | 2,000 | | Every term occurs in any field | 1,000 | 1,000 | 1,000 | | Later term occurs in primary field | 100 | 100 | 100 | | Later term occurs in any field | 20 | 20 | 20 | | **Total** | **118,120** | **28,120** | **3,120** | The resulting order is: ```text theme={null} A, B, C ``` This example also shows that conditions stack. Record A receives both the full-phrase prefix and full-phrase contains points, while a match in the primary field also counts as a match in any searchable field. ### Strict and loose filtering Using the same records and the query `احمد علي`: ```text theme={null} Strict mode (all terms must match): A, B, C Loose mode (at least one term must match): A, B, C ``` If record C instead had `address = "كربلاء"`, it would match only `علي`: ```text theme={null} Strict mode: A, B Loose mode: A, B, C ```