Apache Tika - Get Tika parsing up and running in 5 minutes

Apache Tika - Get Tika parsing up and running in 5 minutes
Add your MIME-Type

Tika loads the core, standard MIME-Types from the file "org/apache/tika/mime/tika-mimetypes.xml", which comes from tika-core/src/main/resources/org/apache/tika/mime/tika-mimetypes.xml . If your new MIME-Type is a standard one which is missing from Tika, submit a patch for this file!

If your MIME-Type needs adding, create a new file "org/apache/tika/mime/custom-mimetypes.xml" in your codebase. You should add to it something like this:

 <?xml version="1.0" encoding="UTF-8"?>
 <mime-info>
   <mime-type type="application/hello">
          <glob pattern="*.hi"/>
   </mime-type>
 </mime-info>

public class HelloParser extends AbstractParser {

        private static final Set<MediaType> SUPPORTED_TYPES = Collections.singleton(MediaType.application("hello"));
        public static final String HELLO_MIME_TYPE = "application/hello";
        
        public Set<MediaType> getSupportedTypes(ParseContext context) {
                return SUPPORTED_TYPES;
        }

        public void parse(
                        InputStream stream, ContentHandler handler,
                        Metadata metadata, ParseContext context)
                        throws IOException, SAXException, TikaException {

                metadata.set(Metadata.CONTENT_TYPE, HELLO_MIME_TYPE);
                metadata.set("Hello", "World");

                XHTMLContentHandler xhtml = new XHTMLContentHandler(handler, metadata);
                xhtml.startDocument();
                xhtml.endDocument();
        }
}

If your MIME-Types aren't standard ones, ensure you listed them in a "custom-mimetypes.xml" file so that Tika knows about them (see above).
Is in the "parse" method where you will do all your work. This is, extract the information of the resource and then set the metadata.

List the new parser

Finally, you should explicitly tell the AutoDetectParser to include your new parser. This step is only needed if you want to use the AutoDetectParser functionality. 

List your new parser in: tika-parsers/src/main/resources/META-INF/services/org.apache.tika.parser.Parser
Read full article from Apache Tika - Get Tika parsing up and running in 5 minutes

Apache Tika - Content Detection

Apache Tika - Content Detection
The Detector Interface
The org.apache.tika.detect.Detector interface is the basis for most of the content type detection in Apache Tika. 
In general, only two keys on the Metadata object are used by Detectors. These areMetadata.RESOURCE_NAME_KEY which should hold the name of the file (where known), and Metadata.CONTENT_TYPE which should hold the advertised content type of the file (eg from a webserver or a content repository).

Mime Magic Detction

By looking for special ("magic") patterns of bytes near the start of the file, it is often possible to detect the type of the file. For some file types, this is a simple process. For others, typically container based formats, the magic detection may not be enough. (More detail on detecting container formats below)

Tika is able to make use of a a mime magic info file, in the Freedesktop MIME-infoformat to peform mime magic detection. (Note that Tika supports a few more match types than Freedesktop does)

This is provided within Tika by org.apache.tika.detect.MagicDetector. It is most commonly access via org.apache.tika.mime.MimeTypes, normally sourced from thetika-mimetypes.xml and custom-mimetypes.xml files. For more information on defining your own custom mimetypes, see the new parser guide.

Resource Name Based Detection

Where the name of the file is known, it is sometimes possible to guess the file type from the name or extension. Within the tika-mimetypes.xml file is a list of patterns which are used to identify the type from the filename.
However, because files may be renamed, this method of detection is quick but not always as accurate.
This is provided within Tika by org.apache.tika.detect.NameDetector.

Known Content Type "Detection
The default Mime Types Detector

By default, the mime type detection in Tika is provided byorg.apache.tika.mime.MimeTypes. This detector makes use of tika-mimetypes.xml to power magic based and filename based detection.
Firstly, magic based detection is used on the start of the file. If the file is an XML file, then the start of the XML is processed to look for root elements. Next, if available, the filename (from Metadata.RESOURCE_NAME_KEY) is then used to improve the detail of the detection, such as when magic detects a text file, and the filename hints it's really a CSV. Finally, if available, the supplied content type (fromMetadata.CONTENT_TYPE) is used to further refine the type.

Container Aware Detection

Tika provides a wrapping detector in the form of org.apache.tika.detect.DefaultDetector. This uses the service loader to discover all available detectors, including any available container aware ones, and tries them in turn. For container aware detection, include theTika Parsers jar and its dependencies in your project, then use DefaultDetector along with a TikaInputStream.

Because these container detectors needs to read the whole file to open and inspect the container, they must be used with a org.apache.tika.io.TikaInputStream. If called with a regular InputStream, then all work will be done by the default Mime Magic detection only.

The default Tika Detector

Just as with Parsers, Tika provides a special detectororg.apache.tika.detect.DefaultDetector which auto-detects (based on service files) the available detectors at runtime, and tries these in turn to identify the file type.

If only Tika Core is available, the Default Detector will work only with Mime Magic and Resource Name detection. However, if Tika Parsers (and its dependencies!) are available, additional detectors which known about containers (such as zip and ole2) will be used as appropriate, provided that detection is being performed with aorg.apache.tika.io.TikaInputStream. Custom detectors can also be used as desired, they simply need to be listed in a service file much as is done for custom parsers.

Ways of triggering Detection

The simplest way to detect is through the Tika Facade class, which provides methods to detect based on File, InputStream, InputStream and Filename, Filename or a few others. It works best with a File or TikaInputStream.

Alternately, detection can be performed on a specific Detector, or usingDefaultDetector to have all available Detectors used. A typical pattern would be something like:

TikaConfig tika = new TikaConfig();

for (File f : myListOfFiles) {
   Metadata metadata = new Metadata();
   metadata.set(Metadata.RESOURCE_NAME_KEY, f.toString());
   String mimetype = tika.getDetector().detect(
        TikaInputStream.get(f), metadata);
   System.out.println("File " + f + " is " + mimetype);
}
for (InputStream is : myListOfStreams) {
   String mimetype = tika.getDetector().detect(
        TikaInputStream.get(is), new Metadata());
   System.out.println("Stream " + is + " is " + mimetype);
}

The language detection is provided by org.apache.tika.language.LanguageIdentifier
Read full article from Apache Tika - Content Detection

Tika JAX-RS Server

http://wiki.apache.org/tika/TikaJAXRS

tika's JSR 311 network server, tika-server. The server package uses the Apache CXF framework that provides an implementation of JAX-RS for Java. The Tika server component builds to a standalone package in Tika, tika-server.

mvn install
cd ./tika-server/target/
java -jar tika-server-x.x.jar

java -jar tika-server-x.x.jar --host=intranet.local --port=12345

All services that take files use HTTP "PUT" requests. Original file must be sent in request body without any additional encoding (do not use multipart/form-data or other containers).

Information services (eg defined mimetypes, defined parsers etc) work with HTML "GET" requests.

You may optionally specify content type in "Content-Type" header. If you do not specify mime type, Tika will use its detectors to guess it.

You may specify additional identifier in URL after resource name, like "/tika/my-file-i-sent-to-tika-resource" for "/tika" resource. Tikaserver uses this name only for logging, so you may put there file name, UUID or any other identifier (do not forget to url-encode any special characters).

$ curl -X PUT -d @zipcode.csv http://localhost:9998/meta --header "Content-Type: text/csv"
$ curl -T price.xls http://localhost:9998/meta

Returns:

"Content-Encoding","ISO-8859-2"
"Content-Type","text/plain"

HTTP PUTs a document to the /tika service and you get back the extracted text. HTTP GET prints a greeting stating the server is up.

curl -X GET http://localhost:9998/tika

$ curl -X PUT -d @GeoSPARQL.pdf http://localhost:9998/tika --header "Content-type: application/pdf"
$ curl -T price.xls http://localhost:9998/tika --header "Accept: text/html"
$ curl -T price.xls http://localhost:9998/tika --header "Accept: text/plain"

HTTP PUTs a document and uses the Default Detector from Tika to identify its MIME/media type. The caveat here is that providing a hint for the filename can increase the quality of detection.

curl -X PUT -d @TODO.rtf http://localhost:9998/detect/stream

PUT a CSV file without filename hint and get back text/plain

$ curl -X PUT --upload-file foo.csv http://localhost:9998/detect/stream

PUT a CSV file with filename hint and get back text/csv

$ curl -X PUT -H "Content-Disposition: attachment; filename=foo.csv" --upload-file foo.csv http://localhost:9998/detect/stream

PUT zip file and get back met file zip

$ curl -X PUT -d @foo.zip http://localhost:9998/unpacker --header "Content-type: application/zip"

PUT doc file and get back met file tar

$ curl -T Doc1_ole.doc -H "Accept: application/x-tar" http://localhost:9998/unpacker > /var/tmp/x.tar

"All" resource

Get text, metadata and attachments in one request.

$ curl -T Doc1_ole.doc http://localhost:9998/all > /var/tmp/x.zip

/mime-types

/detectors

Extracting A Document From A URL

It is possible to use a remote file with TikaJAXRS by downloading it via its URL first then piping it to the appropriate service:

$ curl -s "http://url/to/my.file" | curl -X PUT -T - http://localhost:9998/meta
$ curl -s "http://url/to/my.file" | curl -X PUT -T - http://localhost:9998/tika

The caveat with above is that it fetches the entire file, so large files such as video can take some time to download. Therefore, you may wish to use curl to get preliminary information (content type, name and size) about the file before you proceed:

$ curl -I http://url/to/my.file

If the file should be parsed (E.g. you only want to get information about mp3s, mp4s and PDFs), send it on to TikaJAXRS.

Please read full article from http://wiki.apache.org/tika/TikaJAXRS

RecursiveMetadata - Tika Wiki

RecursiveMetadata - Tika Wiki

If you parse an archive (zip, tar, etc.) the parsed document contains other documents, and any of those documents could also be archives containing other documents, and so on. The example on this page shows you how to do the following:

• Set up the parse context so nested documents will be parsed.
• Wrap the AutoDetectParser so you can get the text and metadata for each nested document.

This example writes the metadata and body text for each nested document to standard output.

public static void main(String[] args) throws Exception {
       Parser parser = new RecursiveMetadataParser(new AutoDetectParser());
       ParseContext context = new ParseContext();
       context.set(Parser.class, parser);

       ContentHandler handler = new DefaultHandler();
       Metadata metadata = new Metadata();

       InputStream stream = TikaInputStream.get(new File(args[0]));
       try {
           parser.parse(stream, handler, metadata, context);
       } finally {
           stream.close();
       }
   }

   private static class RecursiveMetadataParser extends ParserDecorator {

       public RecursiveMetadataParser(Parser parser) {
           super(parser);
       }

       @Override
       public void parse(
               InputStream stream, ContentHandler ignore,
               Metadata metadata, ParseContext context)
               throws IOException, SAXException, TikaException {
           ContentHandler content = new BodyContentHandler();
           super.parse(stream, content, metadata, context);

           System.out.println("----");
           System.out.println(metadata);
           System.out.println("----");
           System.out.println(content.toString());
       }
   }

Setting up Recursive Parsing

  public static void main(String[] args) throws Exception {
       Parser parser = new RecursiveMetadataParser(new AutoDetectParser());
       ParseContext context = new ParseContext();
       context.set(Parser.class, parser);

The example starts by setting up recursive parsing. If you are parsing text files, word documents, etc. then you'll never notice if recursive parsing is enable or not. If you are parsing containers like zip files and tar.gz files, the only way to get the text for the files contained by the containers is to enable recursive parsing.

The way to enable recursive parsing is to create a ParseContext and add a parser to it as shown on the line context.set(Parser.class, parser). This is the parser that will be used to parse any nested documents.

Parsing a File

       ContentHandler handler = new DefaultHandler();
       Metadata metadata = new Metadata();

       InputStream stream = TikaInputStream.get(new File(args[0]));
       try {
           parser.parse(stream, handler, metadata, context);
       } finally {
           stream.close();
       }

The rest of the main function parses a file. The parser used to parse the root document is the same parser that was added to the ParseContext as the parser to use for nested documents.

Looking at the Tika API (http://tika.apache.org/0.7/api/), I don't see a DefaultHandler class or a TikaInputStream. In the place of DefaultHandler you could use BodyContentHandler, and in the place of TikaInputStream you could use FileInputStream.

RecursiveMetadataParser parse

       public void parse(
               InputStream stream, ContentHandler ignore,
               Metadata metadata, ParseContext context)
               throws IOException, SAXException, TikaException {
           ContentHandler content = new BodyContentHandler();
           super.parse(stream, content, metadata, context);
           System.out.println("----");
           System.out.println(metadata);
           System.out.println("----");
           System.out.println(content.toString());
       }
\
   }

The parse method is where you get access to the metadata and the body text. When the parser set in ParseContext is used to parse a nested document, a new Metadata object is created and passed to the parse method. Since the example put a RecursiveMetadataParser in the ParseContext,RecursiveMetadataParser's parse method is called. Before calling super.parse, the metadata object is empty. After super.parse returns, the metadata object contains all of the metadata the decorated parser found and System.out.println(metadata) prints all of the metadata to standard output.

By creating a new BodyContentHandler and passing that to super.parse, the text for each document is captured without mixing it with text from other documents.

The great thing about AutoDetectParser is that it can parse and extract text from almost anything. In particular, it can parse zip, tar, tar.bz2, and other archives that contain documents. If you have a zip file with 100 text files in it, using Jukka's example code you can get the text and metadata for each file nested inside of the zip file. What you might not expect is that you also get metadata and body text for the zip file itself.

If you aren't interested in seeing text and metadata for the zip file itself, you'll want to take a look at metadata.get(Metadata.CONTENT_TYPE)) for each file Tika parses so you can skip the archives themselves. For a zip file, the content type is "application/zip".

Read full article from RecursiveMetadata - Tika Wiki

Tika Official Docs

Tika Official Docs 

void parse(
    InputStream stream, ContentHandler handler, Metadata metadata,
    ParseContext context) throws IOException, SAXException, TikaException;

The parse method takes the document to be parsed and related metadata as input and outputs the results as XHTML SAX events and extra metadata. The parse context argument is used to specify context information (like the current local) that is not related to any individual document.

Input metadata

A client application should be able to include metadata like the file name or declared content type with the document to be parsed. The parser implementation can use this information to better guide the parsing process.

The parsed content of the document stream is returned to the client application as a sequence of XHTML SAX events. XHTML is used to express structured content of the document and SAX events enable streamed processing. 

Dealing with the raw SAX events can be a bit complex, so Apache Tika comes with a number of utility classes that can be used to process and convert the event stream to other representations.

For example, the BodyContentHandler class can be used to extract just the body part of the XHTML output and feed it either as SAX events to another content handler or as characters to an output stream, a writer, or simply a string.

Another useful class is ParsingReader that uses a background thread to parse the document and returns the extracted text content as a character stream:

InputStream stream = ...; // the document to be parsed
Reader reader = new ParsingReader(parser, stream, ...);
try {
    ...;                  // read the document text using the reader
} finally {
    reader.close();       // the document stream is closed automatically
}

Parse context

The final argument to the parse method is used to inject context-specific information to the parsing process. This is useful for example when dealing with locale-specific date and number formats in Microsoft Excel spreadsheets. Another important use of the parse context is passing in the delegate parser instance to be used by two-phase parsers like the PackageParser subclasses. Some parser classes allow customization of the parsing process through strategy objects in the parse context.

The goal of Tika is to reuse existing parser libraries like PDFBox or Apache POIas much as possible, and so most of the parser classes in Tika are adapters to such external libraries.

Tika also contains some general purpose parser implementations that are not targeted at any specific document formats. The most notable of these is the AutoDetectParser class that encapsulates all Tika functionality into a single parser that can handle any types of documents. This parser will automatically determine the type of the incoming document based on various heuristics and will then parse the document accordingly.

http://tika.apache.org/1.6/formats.html

Please read full article from Tika Official Docs 

Content mining with Apache Tika

Content mining with Apache Tika

To extract metadata or content by running Tika from the command line, use the prepackaged jar file. For example, this command outputs the contents of the file test.doc to standard output in text format:

java -jar tika-app-1.4.jar --text test.doc

If you just want the file's metadata, again in text format, try:

java -jar tika-app-1.4.jar --metadata test.doc

Short forms of these commands are also available; run java -jar tika-app-1.4.jar --help to get the full list of available options. You can output the content information in HTML (replace --text with --html) or XHTML (replace --text with --xml) if you prefer. You can output the metadata as JSON (replace --metadata with --json) or XMP (replace --metadata with --xmp).

You can also hook Tika into a standard Unix pipeline, as with any other Unix-style command. For example, you can use cURL to fetch a file, parse its content into HTML using Tika, and then send that HTML output to a file:

curl http://example.com/test.doc | java -jar tika-app-1.4.jar --html > test.html

In addition to working with metadata and content, Tika can also detect the file type and even the language that a file is written in. This can be useful if metadata is lacking:

$ java -jar tika-app-1.4.jar --detect test.doc 
application/rtf
$ java -jar tika-app-1.4.jar --language test_french.doc 

You could also use the filetype detection output to hook a file into another pipeline or another part of a Java app. Tika can even handle metadata from files that contain EXIF information.

Please read full article from Content mining with Apache Tika

Metadata extraction with Apache Tika

Metadata extraction with Apache Tika

Tika defines a standard API and makes use of existing libraries like POI and PDFBox for it's content extraction. While writing this post the current release of Tika is version 0.6 and the following file formats are already supported:

• HyperText Markup Language
• XML and derived formats
• Microsoft Office document formats
• OpenDocument Format
• Portable Document Format
• Electronic Publication Format
• Rich Text Format
• Compression and packaging formats
• Text formats
• Audio formats
• Image formats
• Video formats
• Java class files and archives
• The mbox format

I want to see what kind of EXIF information can be retrieved from an image by using Tika.
The most important part of the above code example is using the JpegParser to parse the .JPG file and the creation of the Metadata object with the appropriate information.

Of course in the above test case I only test for the current Camera Model, but the Metadata object holds much more information then just that. Viewing all the fields found in the metadata of the image can be achieved quite easily by using for instance the following method.

private void listAvailableMetaDataFields(final Metadata metadata) {
    for(int i = 0; i <metadata.names().length; i++) {
        String name = metadata.names()[i];
        System.out.println(name + " : " + metadata.get(name));
    }
}

Read full article from Metadata extraction with Apache Tika