Apache Tika - Content Detection

Apache Tika - Content Detection
The Detector Interface
The org.apache.tika.detect.Detector interface is the basis for most of the content type detection in Apache Tika. 
In general, only two keys on the Metadata object are used by Detectors. These areMetadata.RESOURCE_NAME_KEY which should hold the name of the file (where known), and Metadata.CONTENT_TYPE which should hold the advertised content type of the file (eg from a webserver or a content repository).

Mime Magic Detction

By looking for special ("magic") patterns of bytes near the start of the file, it is often possible to detect the type of the file. For some file types, this is a simple process. For others, typically container based formats, the magic detection may not be enough. (More detail on detecting container formats below)

Tika is able to make use of a a mime magic info file, in the Freedesktop MIME-infoformat to peform mime magic detection. (Note that Tika supports a few more match types than Freedesktop does)

This is provided within Tika by org.apache.tika.detect.MagicDetector. It is most commonly access via org.apache.tika.mime.MimeTypes, normally sourced from thetika-mimetypes.xml and custom-mimetypes.xml files. For more information on defining your own custom mimetypes, see the new parser guide.

Resource Name Based Detection

Where the name of the file is known, it is sometimes possible to guess the file type from the name or extension. Within the tika-mimetypes.xml file is a list of patterns which are used to identify the type from the filename.
However, because files may be renamed, this method of detection is quick but not always as accurate.
This is provided within Tika by org.apache.tika.detect.NameDetector.

Known Content Type "Detection
The default Mime Types Detector

By default, the mime type detection in Tika is provided byorg.apache.tika.mime.MimeTypes. This detector makes use of tika-mimetypes.xml to power magic based and filename based detection.
Firstly, magic based detection is used on the start of the file. If the file is an XML file, then the start of the XML is processed to look for root elements. Next, if available, the filename (from Metadata.RESOURCE_NAME_KEY) is then used to improve the detail of the detection, such as when magic detects a text file, and the filename hints it's really a CSV. Finally, if available, the supplied content type (fromMetadata.CONTENT_TYPE) is used to further refine the type.

Container Aware Detection

Tika provides a wrapping detector in the form of org.apache.tika.detect.DefaultDetector. This uses the service loader to discover all available detectors, including any available container aware ones, and tries them in turn. For container aware detection, include theTika Parsers jar and its dependencies in your project, then use DefaultDetector along with a TikaInputStream.

Because these container detectors needs to read the whole file to open and inspect the container, they must be used with a org.apache.tika.io.TikaInputStream. If called with a regular InputStream, then all work will be done by the default Mime Magic detection only.

The default Tika Detector

Just as with Parsers, Tika provides a special detectororg.apache.tika.detect.DefaultDetector which auto-detects (based on service files) the available detectors at runtime, and tries these in turn to identify the file type.

If only Tika Core is available, the Default Detector will work only with Mime Magic and Resource Name detection. However, if Tika Parsers (and its dependencies!) are available, additional detectors which known about containers (such as zip and ole2) will be used as appropriate, provided that detection is being performed with aorg.apache.tika.io.TikaInputStream. Custom detectors can also be used as desired, they simply need to be listed in a service file much as is done for custom parsers.

Ways of triggering Detection

The simplest way to detect is through the Tika Facade class, which provides methods to detect based on File, InputStream, InputStream and Filename, Filename or a few others. It works best with a File or TikaInputStream.

Alternately, detection can be performed on a specific Detector, or usingDefaultDetector to have all available Detectors used. A typical pattern would be something like:

TikaConfig tika = new TikaConfig();

for (File f : myListOfFiles) {
   Metadata metadata = new Metadata();
   metadata.set(Metadata.RESOURCE_NAME_KEY, f.toString());
   String mimetype = tika.getDetector().detect(
        TikaInputStream.get(f), metadata);
   System.out.println("File " + f + " is " + mimetype);
}
for (InputStream is : myListOfStreams) {
   String mimetype = tika.getDetector().detect(
        TikaInputStream.get(is), new Metadata());
   System.out.println("Stream " + is + " is " + mimetype);
}

The language detection is provided by org.apache.tika.language.LanguageIdentifier
Read full article from Apache Tika - Content Detection