1<!-- 2 Licensed to the Apache Software Foundation (ASF) under one or more 3 contributor license agreements. See the NOTICE file distributed with 4 this work for additional information regarding copyright ownership. 5 The ASF licenses this file to You under the Apache License, Version 2.0 6 (the "License"); you may not use this file except in compliance with 7 the License. You may obtain a copy of the License at 8 9 http://www.apache.org/licenses/LICENSE-2.0 10 11 Unless required by applicable law or agreed to in writing, software 12 distributed under the License is distributed on an "AS IS" BASIS, 13 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 See the License for the specific language governing permissions and 15 limitations under the License. 16--> 17<html> 18<head> 19 <META http-equiv="Content-Type" content="text/html; charset=UTF-8"> 20 <title> 21 Apache Lucene OpenNLP integration module 22 </title> 23</head> 24<body> 25<p> 26 This module exposes functionality from 27 <a href="http://opennlp.apache.org">Apache OpenNLP</a> to Apache Lucene. 28 The Apache OpenNLP library is a machine learning based toolkit for the processing of natural language text. 29<p> 30 For an introduction to Lucene's analysis API, see the {@link org.apache.lucene.analysis} package documentation. 31<p> 32 The OpenNLP Tokenizer behavior is similar to the WhiteSpaceTokenizer but is smart about 33 inter-word punctuation. The term stream looks very much like the way you parse words and 34 punctuation while reading. The major difference between this tokenizer and most other 35 tokenizers shipped with Lucene is that punctuation is tokenized. This is required for 36 the following taggers to operate properly. 37<p> 38 The OpenNLP taggers annotate terms using the <code>TypeAttribute</code>. 39<ul> 40 <li><code>OpenNLPTokenizer</code> segments text into sentences or words. This Tokenizer 41 uses the OpenNLP Sentence Detector and/or Tokenizer classes. When used together, the 42 Tokenizer receives sentences and can do a better job.</li> 43 <li><code>OpenNLPFilter</code> tags words using one or more technologies: Part-of-Speech, 44 Chunking, and Named Entity Recognition. These tags are assigned as token types. Note that 45 only of these operations will tag 46 </li> 47</ul> 48<p> 49 Since the <code>TypeAttribute</code> is not stored in the index, it is recommended that one 50 of these filters is used following <code>OpenNLPFilter</code> to enable search against the 51 assigned tags: 52<ul> 53 <li><code>TypeAsPayloadFilter</code> copies the <code>TypeAttribute</code> value to the 54 <code>PayloadAttribute</code></li> 55 <li><code>TypeAsSynonymFilter</code> creates a cloned token at the same position as each 56 tagged token, and copies the {{TypeAttribute}} value to the {{CharTermAttribute}}, optionally 57 with a customized prefix (so that tags effectively occupy a different namespace from token 58 text).</li> 59</ul> 60</body> 61</html> 62