Script Transliterators

The first task in building a script transform is to determine which system of transliteration to use as a model. There are dozens of different systems for each language and script.

The International Organization for Standardization (ISO ) uses a strict definition of transliteration, which requires it to be reversible. Although the goal for ICU script transforms is to be reversible, they do not have to adhere to this definition. In general, most transliteration systems in use are not reversible. This tutorial will describe the process for building a reversible transform since it illustrates more of the issues involved in the rules. (For guidelines in building transforms, see "Guidelines for Designing Script Transliterations" (§) in the General Transforms chapter. For external sources for script transforms, see Script Transliterator Sources (§) in that same chapter)

See Properties and ICU Rule Syntax for information regarding syntax characters.

In this example, we start with a set of rules for Greek since they provide a real example based on mathematics. We will use the rules that do not involve the pronunciation of Modern Greek; instead, we will use rules that correspond to the way that Greek words were incorporated into the English language. For example, we will transliterate "Βιολογία-Φυσιολογία" as "Biología-Physiología", not as "Violohía-Fisiolohía". To illustrate some of the trickier cases, we will also transliterate the Greek accents that are no longer in use in modern Greek.

Some of the characters may not be visible on the screen unless you have a Unicode font with all the Greek letters. If you have a licensed copy of Microsoft® Office, you can use the "Arial Unicode MS" font, or you can download the CODE2000 font for free. For more information, see Display Problems? on the Unicode web site.

We will also verify that every Latin letter maps to a Greek letter. This insures that when we reverse the transliteration that the process can handle all the Latin letters.

This direction is not reversible. The following table illustrates this situation:

Basics

In non-complex cases, we have a one-to-one relationship between letters in both Greek and Latin. These rules map between a source string and a target string. The following shows this relationship:
π <> p;

This rule states that when you transliterate from Greek to Latin, convert π to p and when you transliterate from Latin to Greek, convert p to π. The syntax is
string1 <> string2 ;

We will start by adding a whole batch of simple mappings. These mappings will not work yet, but we will start with them. For now, we will not use the uppercase versions of characters.

We will also add rules for completeness. These provide fallback mappings for Latin characters that do not normally result from transliterating Greek characters.

Context and Range

We have completed the simple one-to-one mappings and the rules for completeness. The next step is to look at the characters in context. In Greek, for example, the transform converts a "γ" to an "n" if it is before any of the following characters: γ, κ, ξ, or χ. Otherwise the transform converts it to a "g". The following list a all of the possibilities:

γγ > ng;
γκ > nk;
γξ > nx;
γχ > nch;
γ > g;

All the rules are evaluated in the order they are listed. The transform will first try to match the first four rules. If all of these rules fail, it will use the last one.

However, this method quickly becomes tiresome when you consider all the possible uppercase and lowercase combinations. An alternative is to use two additional features: context and range.

Context

First, we will consider the impact of context on a transform. We already have rules for converting γ, κ, ξ, and χ. We must consider how to convert the γ character when it is followed by ³, κ, ξ, and χ. Otherwise we must permit those characters to be converted using their specific rules. This is done with the following:

γ } γ > n;
γ } κ > n;
γ } ξ > n;
γ } χ > n;
γ > g;

A left curly brace marks the start of a context rule. The context rule will be followed when the transform matches the rules against the source text, but itself will not be converted. For example, if we had the sequence γγ, the transform converts the first γ into an "n" using the first rule, then the second γ is unaffected by that rule. The "γ" matches a "k" rule and is converts it into a "k". The result is "nk".

Range

Using context, we have the same number of rules. But, by using range, we can collapse the first four rules into one. The following shows how we can use range:

{γ}[γκξχ] > n;
γ > g;

Any list of characters within square braces will match any one of the characters. We can then add the uppercase variants for completeness, to get:

γ } [ΓΚΞΧγκξχ] > n;
γ > g; 

Remember that we can use spaces for clarity. We can also write this rule as the following:

γ } [ Γ Κ Ξ Χ   γ κ ξ χ ] > n ;
γ > g ;

If a range of characters happens to have adjacent code numbers, we can just use a hyphen to abbreviate it. For example, instead of writing [a b c d e f g m n o], we can simplify the range by writing [a-g m-o].

Styled Text

Another reason to use context is that transforms will convert styled text. When transforms convert styled text, they copy the style source text to the target text. However, the transforms are limited in that they can only convert whole replacements since it is impossible to know how any boundaries within the source text will correspond to the target text. Thus the following shows the effects of the two types of rules on some sample text:

For example, suppose that we were to convert "γγ" to "ng". By using context, if there is a different style on the first gamma than on the second (such as font, size, color, etc), then that style difference is preserved in the resulting two characters. That is, the "n" will have the style of the first gamma, while the "g" will have the style of the second gamma.

Contexts preserve the styles at a much finer granularity.

Case

When converting from Greek to Latin, we can just convert "θ" to and from "th". But what happens with the uppercase theta (Θ)? Sometimes we need to convert it to uppercase "TH", and sometimes to uppercase "T" and lowercase "h". We can choose between these based on the letters before and afterwards. If there is a lowercase letter after an uppercase letter, we can choose "Th", otherwise we will use "TH".

We could manually list all the lowercase letters, but we also can use ranges. Ranges not only list characters explicitly, but they also give you access to all the characters that have a given Unicode property. Although the abbreviations are a bit arcane, we can specify common sets of characters such as all the uppercase letters. The following example shows how case and range can be used together:

Θ } [:LowercaseLetter:] <> Th;
Θ <> TH;

The example allows words like Θεολογικές‚ to map to Theologikés and not THeologikés

You either can specify properties with the POSIX-style syntax, such as [:LowercaseLetter:], or with the Perl-style syntax, such as \p{LowercaseLetter}.

Properties and Values

A Greek sigma is written as "ς" if it is at the end of a word (but not completely separate) and as "σ" otherwise. When we convert characters from Greek to Latin, this is not a problem. However, it is a problem when we convert the character back to Greek from Latin. We need to convert an s depending on the context. While we could list all the possible letters in a range, we can also use a character property. Although the range [:Letter:] stands for all letters, we really want all the characters that aren't letters. To accomplish this, we can use a negated range: [:^Letter:]. The following shows a negated range:

σ < [:^Letter:] { s } [:^Letter:] ;
ς < s } [:^Letter:] ;
σ < s ;

These rules state that if an "s" is surrounded by non-letters, convert it to "σ". Otherwise, if the "s" is followed by a non-letter, convert it to "ς". If all else fails, convert it to "σ"

Negated ranges [^...] will match at the beginning and the end of a string. This makes the rules much easier to write.

To make the rules clearer, you can use variables. Instead of the example above, we can write the following:

$nonletter = [:^Letter:] ;

σ < $nonletter { s } $nonletter ;
ς < s } $nonletter ;
σ < s ;

There are many more properties available that can be used in combination. For following table lists some examples:

For more on properties, see the UnicodeSet and Properties chapters.

Repetition

Elements in a rule can also repeat. For example, in the following rules, the transform converts an iota-subscript into a capital I if the preceding base letter is an uppercase character. Otherwise, the transform converts the iota-subscript into a lowercase character.

[:Uppercase Letter:] { ͅ } > I;
ͅ > i;

However, this is not sufficient, since the base letter may be optionally followed by non-spacing marks. To capture that, we can use the * syntax, which means repeat zero or more times. The following shows this syntax:

[:Uppercase Letter:] [:Nonspacing Mark:] * { ͅ } > I ;
ͅ > i ;

The following operators can be used for repetition:

We can also use these operators as sequences with parentheses for grouping. For example, "a ( b c ) * d" will match against "ad" or "abcd" or "abcbcd".

Currently, any repetition will cause the sequence to match as many times as allowed even if that causes the rest of the rule to fail. For example, suppose we have the following (contrived) rules:

Æther

The start and end of a string are treated specially. Essentially, characters off the end of the string are handled as if they were the noncharacter \uFFFF, which is called "æther". (The code point \uFFFF will never occur in any valid Unicode text). In particular, a negative Unicode set will generally also match against the start/end of a string. For example, the following rule will execute on the first a in a string, as well as an a that is actually preceded by a non-letter.

This is because \uFFFF is an element of [:^L:], which includes all codepoints that do not represent letters. To refer explicitly to æther, you can use a $ at the end of a range, such as in the following rules:

In these rules, an a before or after a number -- or at the start or end of a string -- will be matched. (You could also use \uFFFF explicitly, but the $ is recommended).

Thus to disallow a match against æther in a negation, you need to add the $ to the list of negated items. For example, the first rule and results from above would change to the following (notice that the first a is not replaced):

Characters that are outside the context limits -- contextStart to contextEnd -- are also treated as æther.

The property [:any:] can be used to match all code points, including æther. Thus the following are equivalent:

However, since the transform is always greedy with no backup, this property is not very useful in practice. What is more often required is dealing with the end of lines. If you want to match the start or end of a line, then you can define a variable that includes all the line separator characters, and then use it in the context of your rules. For example:

There is also a special character, the period (.), that is equivalent to the negation of the $break variable we defined above. It can be used to match any characters excluding those for linebreaks or æther. However, it cannot be used within a range: you can't have [[.] - \u000A], for example. If you want to have different behavior you can define your own variables and use them instead of the period.

There are a few other special escapes, that can be used in ranges. These are listed in the table below. However, instead of the latter two it is safest to use the above $break definition since it works for line endings across different platforms.

Accents

We could handle each accented character by itself with rules such as the following:

ά > á;
έ > é;
...

This procedure is very complicated when we consider all the possible combinations of accents and the fact that the text might not be normalized. In ICU 1.8, we can add other transforms as rules either before or after all the other rules. We then can modify the rules to the following:

:: NFD (NFC) ;
α <> a;
...
ω <> ō;
:: NFC (NFD);

These modified rules first separate accents from their base characters and then put them in a canonical order. We can then deal with the individual components, as desired. We can use NFC (NFC) at the end to put the entire result into standard canonical form. The inverse uses the transform rules in reverse order, so the (NFD) goes at the bottom and (NFC) at the top.

A global filter can also be used with the transform rules. The following example shows a filter used in the rules:

:: [[:Greek:][:Inherited:]];
:: NFD (NFC) ;
α <> a;
...
ω <> ō;
:: NFC (NFD);
:: ([[:Latin:][:Inherited:]]);

The global filter will cause any other characters to be unaffected. In particular, the NFD then only applies to Greek characters and accents, leaving all other characters

Disambiguation

If the transliteration is to be completely reversible, what would happen if we happened to have the Greek combination νγ? Because ν converts to n, both νγ and γγ convert to "ng" and we have an ambiguity. Normally, this sequence does not occur in the Greek language. However, for consistency -- and especially to aid in mechanical testing– we must consider this situation. (There are other cases in this and other languages where both sequences occur.)

To resolve this ambiguity, use the mechanism recommended by the Japanese and Korean transliteration standards by inserting an apostrophe or hyphen to disambiguate the results. We can add a rule like the following that inserts an apostrophe after an "n" if we need to reverse the transliteration process:
ν } [ΓΚΞΧγκξχ] > n\';

In ICU, there are several of these mechanisms for the Greek rules. The ICU rules undergo some fairly rigorous mechanical testing to ensure reversibility. Adding these disambiguation rules ensure that the rules can pass these tests and handle all possible sequences of characters correctly.

There are some character forms that never occur in normal context. By convention, we use tilde (~) for such cases to allow for reverse transliteration. Thus, if you had the text "Θεολογικές (ς)", it would transliterate to "Theologikés (~s)". Using the tilde allows the reverse transliteration to detect the character and convert correctly back to the original: "Θεολογικές (ς)". Similarly, if we had the phrase "Θεολογικέσ", it would transliterate to "Theologiké~s". These are called anomalous characters.

Revisiting

Rules allow for characters to be revisited after they are replaced. For example, the following converts "C" back "S" in front of "E", "I" or "Y". The vertical bar means that the character will be revisited, so that the "S" or "K" in a Greek transform will be applied to the result and will eventually produce a sigma (Σ, σ, or ς) or kappa (Κ or κ).

$softener = [eiyEIY] ;

| S < C } $softener ;
| K < C ;
| s < c } $softener ;
| k < c ;

The ability to revisit is particularly useful in reducing the number of rules required for a given language. For example, in Japanese there are a large number of cases that follow the same pattern: "kyo" maps to a large hiragana for "ki" (き) followed by a small hiragana for "yo" (ょ). This can be done with a small number of rules with the following pattern:

First, the ASCII punctuation mark, tilde "~", represents characters that never normally occur in isolation. This is a general convention for anomalous characters within the ICU rules in any event.

'~yu' > ゅ;
'~ye' > ぇ;
'~yo' > ょ;

Second, any syllables that use this pattern are broken into the first hiragana and are followed by letters that will form the small hiragana.

by > び|'~y';
ch > ち|'~y';
dj > ぢ|'~y';
gy > ぎ|'~y';
j > じ|'~y';
ky > き|'~y';
my > み|'~y';
ny > に|'~y';
py > ぴ|'~y';
ry > り|'~y';
sh > し|'~y';

Using these rules, "kyo" is first converted into "き~yo". Since the "~yo" is then revisited, this produces the desired final result, "きょ". Thus, a small number of rules (3 + 11 = 14) provide for a large number of cases. If all of the combinations of rules were used instead, it would require 3 x 11 = 33 rules.

You can set the new revisit point (called the cursor) anywhere in the replacement text. You can even set the revisit point before or after the target text. The at-sign, as in the following example, is used as a filler to indicate the position, for those cases:

[aeiou] { x > | @ ks ;
ak > ack ;

The first rule will convert "x", when preceded by a vowel, into "ks". The transform will then backup to the position before the vowel and continue. In the next pass, the "ak" will match and be invoked. Thus, if the source text is "ax", the result will be "ack".

Although you can move the cursor forward or backward, it is limited in two ways: (a) to the text that is matched, (b) within the original substring that is to be converted. For example, if we have the rule "a b* {x} > |@@@@@y" and it matches in the text "mabbx", the result will be "m|abby" (| represents the cursor position). Even though there are five @ signs, the cursor will only backup to the first character that is matched.

Copying

We can copy part of the matched string to the target text. Use parenthesis to group the text to copy and use "$n" (where n is a number from 1 to 99) to indicate which group. For example, in Korean, any vowel that does not have a consonant before it gets the null consonant (?) inserted before it. The following example shows this rule:

([aeiouwy]) > ?| $1 ;

To revisit the vowel again, insert the null consonant, insert the vowel, and then backup before the vowel to reconsider it. Similarly, we have a following rule that inserts a null vowel (?), if no real vowel is found after a consonant:

([b-dg-hj-km-npr-t]) > | $1 eu;

In this case, since we are going to reconsider the text again, we put in the Latin equivalent of the Korean null vowel, which is "eu".

Order Matters

Two rules overlap when there is a string that both rules could match at the start. For example, the first part of the following rule does not overlap, but the last two parts do overlap:

β > b;
γ } [ Γ Κ Ξ Χ   γ κ ξ χ ] > n ;
γ > g ;

When rules do not overlap, they will produce the same result no matter what order they are in. It does not matter whether we have either of the following:

β > b;
γ > g ;

or

γ > g ;
β > b;

When rules do overlap, order is important. In fact, a rule could be rendered completely useless. Suppose we have:

β } [aeiou] > b;
β } [^aeiou] > v;
β > p;

In this case, the last rule is masked as none of the text that will match the rule will already be matched by previous rules. If a rule is masked, then a warning will be issued when you attempt to build a transform with the rules.

Combinations

In Greek, a rough breathing mark on one of the first two vowels in a word represents an "H". This mark is invalid anywhere else in the language. In the normalize (NFD) form, the rough-breathing mark will be first accent after the vowel (with perhaps other accents following). So, we will start with the following variables and rule. The rule transforms a rough breathing mark into an "H", and moves it to before the vowels.

$gvowel = [ΑΕΗΙΟΥΩαεηιουω];

($gvowel + ) ̔ > H | $1;

A word like ὍΤΑΝ" is transformed into "HOTAN". This transformation does not work with a lowercase word like "ὅταν". To handle lowercase words, we insert another rule that moves the "H" over lowercase vowels and changes it to lowercase. The following shows this rule:

$gvowel = [ΑΕΗΙΟΥΩαεηιουω];
$lcgvowel = [αεηιουω];

($lcgvowel +) ̔ > h | $1; # fix lowercase
($gvowel + ) ̔ > H | $1;

This rule provides the correct results as the lowercase word "ὅταν" is transformed into "hotan".

There are also titlecase words such as "Ὅταν". For this situation, we need to lowercase the uppercase letters as the transform passes over them. We need to do that in two circumstances: (a) the breathing mark is on a capital letter followed by a lowercase, or (b) the breathing mark is on a lowercase vowel. The following shows how to write a rule for this situation:

$gvowel = [ΑΕΗΙΟΥΩαεηιουω];
$lcgvowel = [αεηιουω];

{Ο    ̔ } [:Nonspacing Mark:]* [:Ll:] > H | ο; # fix Titlecase
{Ο ( $lcgvowel * )    ̔ } > H | ο $1; # fix Titlecase

( $lcgvowel + )    ̔ > h | $1 ; # fix lowercase
($gvowel + )    ̔ > H | $1 ;

This rule gives the correct results for lowercase as "Ὅταν" is transformed into "Hotan". We must copy the above insertion and modify it for each of the vowels since each has a different lowercase.

We must also write a rule to handle a single letter word like "ὃ". In that case, we would need to look beyond the word, either forward or backward, to know whether to transform it to "HO" or to transform it to "Ho". Unlike the case of a capital theta (Θ), there are cases in the Greek language where single-vowel words have rough breathing marks. In this case, we would use several rules to match either before or after the word and ignore certain characters like punctuation and space (watch out for combining marks).

Pitfalls

1. Case When executing script conversions, if the source script has uppercase and lowercase characters, and the target is lowercase, then lowercase everything before your first rule. For example:
   
   :: [:Latin:] lower (); # lowercase target before applying forward rules
   
   This will allow the rules to work even when they are given a mixture of upper and lower case character. This procedure is done in the following ICU transforms:

1. Latin-Hangul

2. Latin-Greek

3. Latin-Cyrillic

4. Latin-Devanagari

5. Latin-Gujarati

6. etc

1. Punctuation. When executing script conversions, remember that scripts have different punctuation conventions. For example, in the Greek language, the ";" means a question mark. Generally, these punctuation marks also should be converted when transliterating scripts.

2. Normalization Always design transform rules so that they work no matter whether the source is normalized or not. (This is also true for the target, in the case of backwards rules.) Generally, the best way to do this is to have :: NFD (NFC); as the first line of the rules, and :: NFC (NFD); as the last line. To supply filters, as described above, break each of these lines into two separate lines. Then, apply the filter to either the normal or inverse direction. Each of the accents then can be manipulated as separate items that are always in a canonical order. If we are not using any accent manipulation, we could use :: NFC (NFC) ; at the top of the rules instead.

3. Ignorable Characters   Letters may have following accents such as the following example:
   
   [:lowercase letter:] } z > s ; # convert z after letters into s
   
   Normally, we want to ignore any accents that are on the z in performing the rule. To do that, restate the rule as:
   
   [:lowercase letter:] [:mark:]* } z > s ; # convert z after letters into s
   
   Even if we are not using NFD, this is still a good idea since some languages use separate accents that cannot be combined.
   Moreover, some languages may have embedded format codes, such as a Left-Right Mark, or a Non-Joiner. Because of that, it is even safer to use the following:
   
   

TODO: this code should be part of the preceding list item #4.
$ignore = [ [:mark:] [:format:] ] * ; # define at the top of your file
...
[:letter:] $ignore } z > s ; # convert z after letters into sh

Remember that the rules themselves must be in the same normalization format. Otherwise, nothing will match. To do this, run NFD on the rules themselves. In some cases, we must rearrange the order of the rules because of masking. For example, consider the following rules: