Character set detection is the process of determining the character set, or encoding, of character data in an unknown format. This is, at best, an imprecise operation using statistics and heuristics. Because of this, detection works best if you supply at least a few hundred bytes of character data that's mostly in a single language. In some cases, the language can be determined along with the encoding.
Several different techniques are used for character set detection. For multi-byte encodings, the sequence of bytes is checked for legal patterns. The detected characters are also check against a list of frequently used characters in that encoding. For single byte encodings, the data is checked against a list of the most commonly occurring three letter groups for each language that can be written using that encoding. The detection process can be configured to optionally ignore html or xml style markup, which can interfere with the detection process by changing the statistics.
The input data can either be a Java input stream, or an array of bytes. The output of the detection process is a list of possible character sets, with the most likely one first. For simplicity, you can also ask for a Java Reader that will read the data in the detected encoding.
There is another character set detection C++ library, the Compact Encoding Detector, that may have a lower error rate, particularly when working with short samples of text.
The CharsetMatch class holds the result of comparing the input data to a particular encoding. You can use an instance of this class to get the name of the character set, the language, and how good the match is. You can also use this class to decode the input data.
To find out how good the match is, you use the getConfidence() method to get a confidence value. This is an integer from 0 to 100. The higher the value, the more confidence there is in the match For example:
In C, you can use ucsdet_getConfidence(const UCharsetMatch *ucsm, UErrorCode *status) method to get a confidence value
To get the name of the character set, which can be used as an encoding name in Java, you use the getName() method:
To get the name of the character set in C :
To get the three letter ISO code for the detected language, you use the getLanguage() method. If the language could not be determined, getLanguage() will return null. Note that language detection does not work with all charsets, and includes only a very small set of possible languages. It should not used if robust, reliable language detection is required.
The ucsdet_getLanguage(const UCharsetMatch *ucsm, UErrorCode *status) method can be used in C to get the language code. If the language could not be determined, the method will return an empty string.
If you want to get a Java String containing the converted data you can use the getString() method:
If you want to limit the number of characters in the string, pass the maximum number of characters you want to the getString() method:
To get a java.io.Reader to read the converted data, use the getReader() method:
The CharsetDetector class does the actual detection. It matches the input data against all character sets, and computes a list of CharsetMatch objects to hold the results. The input data can be supplied as an array of bytes, or as a java.io.InputStream.
To use a CharsetDetector object, first you construct it, and then you set the input data, using the setText() method. Because setting the input data is separate from the construction, it is easy to reuse a CharsetDetector object:
If you want to know which character set matches your input data with the highest confidence, you can use the detect() method, which will return a CharsetMatch object for the match with the highest confidence:
If you want to know which character set matches your input data in C, you can use the ucsdet_detect( UCharsetDetector *csd , UErrorCode *status) method.
If you want to know all of the character sets that could match your input data with a non-zero confidence, you can use the detectAll() method, which will return an array of CharsetMatch objects sorted by confidence, from highest to lowest.:
The CharsetDetector class also implements a crude input filter that can strip out html and xml style tags. If you want to enable the input filter, which is disabled when you construct a CharsetDetector, you use the enableInputFilter() method, which takes a boolean. Pass in true if you want to enable the input filter, and false if you want to disable it:
To enable an input filter in C, you can use ucsdet_enableInputFilter( UCharsetDetector* csd, UBool filter) function.
If you have more detailed knowledge about the structure of the input data, it is better to filter the data yourself before you pass it to CharsetDetector. For example, you might know that the data is from an html page that contains CSS styles, which will not be stripped by the input filter.
You can use the inputFilterEnabled() method to see if the input filter is enabled:
The CharsetDetector class also has two convenience methods that let you detect and convert the input data in one step: the getReader() and getString() methods:
The following code is equivalent to using the convenience methods:
The following table shows all the encodings that can be detected. You can get this list (without the languages) by calling the getAllDetectableCharsets() method: