You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opennlp.apache.org by rz...@apache.org on 2026/09/17 17:57:34 UTC
(opennlp-addons) branch main updated: OPENNLP-1964: Remove the Japanese add-on (#198)
This is an automated email from the ASF dual-hosted git repository.
rzo1 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/opennlp-addons.git
The following commit(s) were added to refs/heads/main by this push:
new 0d9d2cc OPENNLP-1964: Remove the Japanese add-on (#198)
0d9d2cc is described below
commit 0d9d2cc91c83a5dfe3ba3aee8af177ed28ffaf80
Author: Richard Zowalla <13...@users.noreply.github.com>
AuthorDate: Thu Sep 17 19:57:28 2026 +0200
OPENNLP-1964: Remove the Japanese add-on (#198)
---
README.md | 5 +-
japanese/build.xml | 79 ---------------
japanese/pom.xml | 90 -----------------
.../AuxiliaryInfoNameContextGenerator.java | 81 ---------------
.../AuxiliaryInfoTokenNameFinderFactory.java | 57 -----------
.../opennlp/tools/namefind/AuxiliaryInfoUtil.java | 63 ------------
...AuxiliaryInfoAwareDelegateFeatureGenerator.java | 65 ------------
...ryInfoAwareDelegateFeatureGeneratorFactory.java | 40 --------
.../lang/jpn/BigramNameFeatureGenerator.java | 41 --------
.../jpn/BigramNameFeatureGeneratorFactory.java | 35 -------
...wnClusterTokenClassFeatureGeneratorFactory.java | 60 -----------
.../lang/jpn/BrownTokenClassFeatureGenerator.java | 49 ---------
.../featuregen/lang/jpn/FeatureGeneratorUtil.java | 90 -----------------
.../lang/jpn/TokenClassFeatureGenerator.java | 53 ----------
.../jpn/TokenClassFeatureGeneratorFactory.java | 38 -------
.../lang/jpn/TokenPatternFeatureGenerator.java | 90 -----------------
.../jpn/TokenPatternFeatureGeneratorFactory.java | 38 -------
.../lang/jpn/TrigramNameFeatureGenerator.java | 47 ---------
.../jpn/TrigramNameFeatureGeneratorFactory.java | 35 -------
.../tools/namefind/ner-auxinfo-features.xml | 59 -----------
.../tools/namefind/AuxiliaryInfoUtilTest.java | 75 --------------
...liaryInfoAwareDelegateFeatureGeneratorTest.java | 66 ------------
.../lang/jpn/BigramNameFeatureGeneratorTest.java | 96 ------------------
.../lang/jpn/FeatureGeneratorUtilTest.java | 68 -------------
.../lang/jpn/TokenClassFeatureGeneratorTest.java | 66 ------------
.../lang/jpn/TokenPatternFeatureGeneratorTest.java | 77 --------------
.../lang/jpn/TrigramNameFeatureGeneratorTest.java | 111 ---------------------
opennlp-addons-docs/src/docbkx/building.xml | 2 +-
opennlp-addons-docs/src/docbkx/modules.xml | 4 -
opennlp-addons-docs/src/docbkx/using-addons.xml | 4 +-
pom.xml | 1 -
31 files changed, 5 insertions(+), 1680 deletions(-)
diff --git a/README.md b/README.md
index 9d80125..8192e02 100644
--- a/README.md
+++ b/README.md
@@ -39,7 +39,6 @@ Each add-on is published as its own Maven artifact, so an application can take a
Currently, the repository has the following modules:
* `geoentitylinker` : Links place names found in text to entries of a gazetteer index.
-* `japanese` : Feature generators and a name finder factory for Japanese text.
* `liblinear` : LIBLINEAR based machine learning implementation.
* `modelbuilder` : Utilities for semi-supervised generation of name finder models.
* `opennlp-addons-docs` : The DocBook sources of the add-ons manual.
@@ -53,7 +52,7 @@ Depend on the module you need, not on the parent `opennlp-addons` POM:
```
<dependency>
<groupId>org.apache.opennlp.addons</groupId>
- <artifactId>japanese</artifactId>
+ <artifactId>${addon.artifactId}</artifactId>
<version>${opennlp-addons.version}</version>
</dependency>
```
@@ -61,7 +60,7 @@ Depend on the module you need, not on the parent `opennlp-addons` POM:
#### Gradle
```
-compile group: "org.apache.opennlp.addons", name: "japanese", version: "${opennlp-addons.version}"
+compile group: "org.apache.opennlp.addons", name: "${addon.artifactId}", version: "${opennlp-addons.version}"
```
> [!NOTE]
diff --git a/japanese/build.xml b/japanese/build.xml
deleted file mode 100644
index e02fecc..0000000
--- a/japanese/build.xml
+++ /dev/null
@@ -1,79 +0,0 @@
-<!--
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
--->
-<project name="japanese"
- default="jar" basedir=".">
-
- <property name="src.dir" value="src"/>
- <property name="cls.dir" value="classes"/>
- <property name="lib.dir" value="lib"/>
- <property name="test.result.dir" value="test-result"/>
- <property name="product.jar" value="japanese-3.0.0-SNAPSHOT.jar"/>
-
- <target name="compile" description="compile source and test code">
- <mkdir dir="${cls.dir}"/>
- <javac srcdir="${src.dir}/main/java" destdir="${cls.dir}" debug="on" includes="**/*.java">
- <classpath path="${cls.dir}"/>
- <classpath>
- <fileset dir="${lib.dir}" includes="**/*.jar"/>
- </classpath>
- </javac>
- <javac srcdir="${src.dir}/test/java" destdir="${cls.dir}" debug="on" includes="**/*Test.java">
- <classpath path="${cls.dir}"/>
- <classpath>
- <fileset dir="${lib.dir}" includes="**/*.jar"/>
- </classpath>
- </javac>
- </target>
-
- <target name="test" depends="compile" description="run all tests">
- <mkdir dir="${test.result.dir}"/>
- <junit printsummary="on"
- haltonfailure="no"
- errorProperty="tests.failed"
- failureProperty="tests.failed"
- dir="${src.dir}/test/resources">
- <classpath path="${cls.dir}"/>
- <classpath>
- <fileset dir="${lib.dir}" includes="**/*.jar"/>
- </classpath>
- <jvmarg value="-ea"/>
- <formatter type="plain"/>
- <batchtest fork="yes" todir="${test.result.dir}" unless="testcase">
- <fileset dir="${src.dir}/test/java" includes="**/*Test.java"/>
- </batchtest>
- <batchtest fork="yes" todir="${test.result.dir}" if="testcase">
- <fileset dir="${src.dir}/test/java" includes="**/${testcase}.java"/>
- </batchtest>
- </junit>
- <fail if="tests.failed">***** Tests failed! *****</fail>
- </target>
-
- <target name="jar" depends="compile" description="create a jar file">
- <jar destfile="${product.jar}" basedir="${cls.dir}"/>
- </target>
-
- <target name="clean" description="clean all">
- <delete dir="${cls.dir}"/>
- <delete dir="${test.result.dir}"/>
- <delete>
- <fileset dir="." includes="*.jar"/>
- </delete>
- </target>
-
-</project>
diff --git a/japanese/pom.xml b/japanese/pom.xml
deleted file mode 100644
index 4d7cb5f..0000000
--- a/japanese/pom.xml
+++ /dev/null
@@ -1,90 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-
-<!--
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
--->
-
-<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
- xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
- <modelVersion>4.0.0</modelVersion>
- <parent>
- <groupId>org.apache.opennlp.addons</groupId>
- <artifactId>opennlp-addons</artifactId>
- <version>3.0.0-SNAPSHOT</version>
- </parent>
-
- <artifactId>japanese</artifactId>
- <name>Apache OpenNLP Japanese Addon</name>
- <description>Name-finder features for Japanese text.</description>
- <packaging>jar</packaging>
-
- <dependencies>
- <dependency>
- <groupId>org.apache.opennlp</groupId>
- <artifactId>opennlp-api</artifactId>
- </dependency>
- <dependency>
- <groupId>org.apache.opennlp</groupId>
- <artifactId>opennlp-runtime</artifactId>
- </dependency>
-
- <dependency>
- <groupId>org.junit.jupiter</groupId>
- <artifactId>junit-jupiter-api</artifactId>
- </dependency>
-
- <dependency>
- <groupId>org.junit.jupiter</groupId>
- <artifactId>junit-jupiter-engine</artifactId>
- </dependency>
-
- <dependency>
- <groupId>org.junit.jupiter</groupId>
- <artifactId>junit-jupiter-params</artifactId>
- </dependency>
-
- <dependency>
- <groupId>org.apache.logging.log4j</groupId>
- <artifactId>log4j-api</artifactId>
- </dependency>
-
- <dependency>
- <groupId>org.apache.logging.log4j</groupId>
- <artifactId>log4j-core</artifactId>
- </dependency>
-
- <dependency>
- <groupId>org.apache.logging.log4j</groupId>
- <artifactId>log4j-slf4j2-impl</artifactId>
- </dependency>
- </dependencies>
-
- <build>
- <plugins>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-compiler-plugin</artifactId>
- <configuration>
- <source>${maven.compiler.source}</source>
- <target>${maven.compiler.target}</target>
- <compilerArgument>-Xlint</compilerArgument>
- </configuration>
- </plugin>
- </plugins>
- </build>
-</project>
diff --git a/japanese/src/main/java/opennlp/tools/namefind/AuxiliaryInfoNameContextGenerator.java b/japanese/src/main/java/opennlp/tools/namefind/AuxiliaryInfoNameContextGenerator.java
deleted file mode 100644
index 333379f..0000000
--- a/japanese/src/main/java/opennlp/tools/namefind/AuxiliaryInfoNameContextGenerator.java
+++ /dev/null
@@ -1,81 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package opennlp.tools.namefind;
-
-import java.util.ArrayList;
-import java.util.List;
-
-import opennlp.tools.util.featuregen.AdaptiveFeatureGenerator;
-import opennlp.tools.util.featuregen.FeatureGeneratorUtil;
-
-/**
- * If a token contains auxiliary information, e.g. POS tag, this class can be used
- * to extract word part in {@link #getContext(int, String[], String[], Object[])} method.
- *
- * <strong>EXPERIMENTAL</strong>.
- * This class has been added as part of a work in progress and might change without notice.
- */
-public class AuxiliaryInfoNameContextGenerator extends DefaultNameContextGenerator {
-
- public AuxiliaryInfoNameContextGenerator(AdaptiveFeatureGenerator... featureGenerators) {
- super(featureGenerators);
- }
-
- /**
- * Return the context for finding names at the specified index.
- * @param index The index of the token in the specified toks array for which the
- * context should be constructed.
- * @param tokens The tokens of the sentence. The <code>toString</code> methods
- * of these objects should return the token text.
- * @param preds The previous decisions made in the tagging of this sequence.
- * Only indices less than {@code i} will be examined.
- * @param additionalContext Addition features which may be based on a context outside the sentence.
- *
- * @return the context for finding names at the specified index.
- */
- @Override
- public String[] getContext(int index, String[] tokens, String[] preds, Object[] additionalContext) {
- List<String> features = new ArrayList<>();
-
- for (AdaptiveFeatureGenerator featureGenerator : featureGenerators) {
- featureGenerator.createFeatures(features, tokens, index, preds);
- }
-
- //previous outcome features
- String po = NameFinderME.OTHER;
- String ppo = NameFinderME.OTHER;
-
- // TODO: These should be moved out here in its own feature generator!
- if (preds != null) {
- if (index > 1) {
- ppo = preds[index - 2];
- }
-
- if (index > 0) {
- po = preds[index - 1];
- }
- features.add("po=" + po);
- String word = AuxiliaryInfoUtil.getWordPart(tokens[index]);
- features.add("pow=" + po + "," + word);
- features.add("powf=" + po + "," + FeatureGeneratorUtil.tokenFeature(word));
- features.add("ppo=" + ppo);
- }
-
- return features.toArray(new String[0]);
- }
-}
diff --git a/japanese/src/main/java/opennlp/tools/namefind/AuxiliaryInfoTokenNameFinderFactory.java b/japanese/src/main/java/opennlp/tools/namefind/AuxiliaryInfoTokenNameFinderFactory.java
deleted file mode 100644
index bcd6439..0000000
--- a/japanese/src/main/java/opennlp/tools/namefind/AuxiliaryInfoTokenNameFinderFactory.java
+++ /dev/null
@@ -1,57 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package opennlp.tools.namefind;
-
-import opennlp.tools.util.featuregen.AdaptiveFeatureGenerator;
-import opennlp.tools.util.featuregen.AggregatedFeatureGenerator;
-import opennlp.tools.util.featuregen.BigramNameFeatureGenerator;
-import opennlp.tools.util.featuregen.CachedFeatureGenerator;
-import opennlp.tools.util.featuregen.OutcomePriorFeatureGenerator;
-import opennlp.tools.util.featuregen.PreviousMapFeatureGenerator;
-import opennlp.tools.util.featuregen.SentenceFeatureGenerator;
-import opennlp.tools.util.featuregen.TokenClassFeatureGenerator;
-import opennlp.tools.util.featuregen.TokenFeatureGenerator;
-import opennlp.tools.util.featuregen.WindowFeatureGenerator;
-
-/**
- * If a token contains auxiliary information, e.g. POS tag, in the training data,
- * you can use this class via -factory command line option.
- *
- * <strong>EXPERIMENTAL</strong>.
- * This class has been added as part of a work in progress and might change without notice.
- */
-public class AuxiliaryInfoTokenNameFinderFactory extends TokenNameFinderFactory {
-
- @Override
- public NameContextGenerator createContextGenerator() {
-
- AdaptiveFeatureGenerator featureGenerator = createFeatureGenerators();
-
- if (featureGenerator == null) {
- featureGenerator = new CachedFeatureGenerator(new AggregatedFeatureGenerator(
- new WindowFeatureGenerator(new TokenFeatureGenerator(), 2, 2),
- new WindowFeatureGenerator(new TokenClassFeatureGenerator(true), 2, 2),
- new OutcomePriorFeatureGenerator(),
- new PreviousMapFeatureGenerator(),
- new BigramNameFeatureGenerator(),
- new SentenceFeatureGenerator(true, false)));
- }
-
- return new AuxiliaryInfoNameContextGenerator(featureGenerator);
- }
-}
diff --git a/japanese/src/main/java/opennlp/tools/namefind/AuxiliaryInfoUtil.java b/japanese/src/main/java/opennlp/tools/namefind/AuxiliaryInfoUtil.java
deleted file mode 100644
index a9eb7f8..0000000
--- a/japanese/src/main/java/opennlp/tools/namefind/AuxiliaryInfoUtil.java
+++ /dev/null
@@ -1,63 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package opennlp.tools.namefind;
-
-/**
- * If a token contains auxiliary information, e.g. POS tag, this class can be used
- * to extract word part or auxiliary information part.<br>
- *
- * ex) token := word '/' POStag
- *
- * <strong>EXPERIMENTAL</strong>.
- * This class has been added as part of a work in progress and might change without notice.
- */
-public class AuxiliaryInfoUtil {
-
- public static int getSeparatorIndex(String wordAndAux) {
- int idx = wordAndAux.lastIndexOf('/');
- if (idx < 0)
- throw new RuntimeException(String.format("token %s doesn't have Auxiliary info", wordAndAux));
- return idx;
- }
-
- public static String getWordPart(String wordAndAux) {
- int idx = getSeparatorIndex(wordAndAux);
- return idx == 0 ? " " : wordAndAux.substring(0, idx);
- }
-
- public static String[] getWordParts(String[] wordAndAuxes) {
- String[] results = new String[wordAndAuxes.length];
- for (int i = 0; i < wordAndAuxes.length; i++) {
- results[i] = getWordPart(wordAndAuxes[i]);
- }
- return results;
- }
-
- public static String getAuxPart(String wordAndAux) {
- int idx = getSeparatorIndex(wordAndAux);
- return wordAndAux.substring(idx + 1);
- }
-
- public static String[] getAuxParts(String[] wordAndAuxes) {
- String[] results = new String[wordAndAuxes.length];
- for (int i = 0; i < wordAndAuxes.length; i++) {
- results[i] = getAuxPart(wordAndAuxes[i]);
- }
- return results;
- }
-}
diff --git a/japanese/src/main/java/opennlp/tools/util/featuregen/AuxiliaryInfoAwareDelegateFeatureGenerator.java b/japanese/src/main/java/opennlp/tools/util/featuregen/AuxiliaryInfoAwareDelegateFeatureGenerator.java
deleted file mode 100644
index aa9178d..0000000
--- a/japanese/src/main/java/opennlp/tools/util/featuregen/AuxiliaryInfoAwareDelegateFeatureGenerator.java
+++ /dev/null
@@ -1,65 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package opennlp.tools.util.featuregen;
-
-import java.util.Arrays;
-import java.util.List;
-
-import opennlp.tools.namefind.AuxiliaryInfoUtil;
-
-/**
- * If a token contains auxiliary information, e.g. POS tag, in the training data,
- * you can use this feature generator in order to let the feature generator choose
- * word part or auxiliary information part.<br>
- *
- * ex) token := word '/' POStag
- *
- * <strong>EXPERIMENTAL</strong>.
- * This class has been added as part of a work in progress and might change without notice.
- */
-public class AuxiliaryInfoAwareDelegateFeatureGenerator implements AdaptiveFeatureGenerator {
-
- private String[] cachedWordAndAuxes;
- private String[] cached;
-
- private final AdaptiveFeatureGenerator generator;
- private final boolean useAux;
-
- public AuxiliaryInfoAwareDelegateFeatureGenerator(AdaptiveFeatureGenerator generator, boolean useAux) {
- this.generator = generator;
- this.useAux = useAux;
- }
-
- @Override
- public void createFeatures(List<String> features, String[] wordAndAuxes, int index,
- String[] previousOutcomes) {
- if (useAux) {
- if (!Arrays.equals(cachedWordAndAuxes, wordAndAuxes)) {
- cachedWordAndAuxes = wordAndAuxes;
- cached = AuxiliaryInfoUtil.getAuxParts(wordAndAuxes);
- }
- }
- else {
- if (!Arrays.equals(cachedWordAndAuxes, wordAndAuxes)) {
- cachedWordAndAuxes = wordAndAuxes;
- cached = AuxiliaryInfoUtil.getWordParts(wordAndAuxes);
- }
- }
- generator.createFeatures(features, cached, index, previousOutcomes);
- }
-}
diff --git a/japanese/src/main/java/opennlp/tools/util/featuregen/AuxiliaryInfoAwareDelegateFeatureGeneratorFactory.java b/japanese/src/main/java/opennlp/tools/util/featuregen/AuxiliaryInfoAwareDelegateFeatureGeneratorFactory.java
deleted file mode 100644
index 28d71e4..0000000
--- a/japanese/src/main/java/opennlp/tools/util/featuregen/AuxiliaryInfoAwareDelegateFeatureGeneratorFactory.java
+++ /dev/null
@@ -1,40 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package opennlp.tools.util.featuregen;
-
-import opennlp.tools.util.InvalidFormatException;
-
-/**
- * @see opennlp.tools.util.featuregen.AuxiliaryInfoAwareDelegateFeatureGenerator
- *
- * <strong>EXPERIMENTAL</strong>.
- * This class has been added as part of a work in progress and might change without notice.
- */
-public class AuxiliaryInfoAwareDelegateFeatureGeneratorFactory
- extends GeneratorFactory.AbstractXmlFeatureGeneratorFactory {
-
- @Override
- public AdaptiveFeatureGenerator create() throws InvalidFormatException {
- AdaptiveFeatureGenerator generator = (AdaptiveFeatureGenerator)args.get("generator#0");
- if (generator == null) {
- throw new InvalidFormatException("AuxiliaryInfoAwareDelegate feature generator must contain" +
- " an aggregator element");
- }
- return new AuxiliaryInfoAwareDelegateFeatureGenerator(generator, getBool("useAux", false));
- }
-}
diff --git a/japanese/src/main/java/opennlp/tools/util/featuregen/lang/jpn/BigramNameFeatureGenerator.java b/japanese/src/main/java/opennlp/tools/util/featuregen/lang/jpn/BigramNameFeatureGenerator.java
deleted file mode 100644
index d1a6305..0000000
--- a/japanese/src/main/java/opennlp/tools/util/featuregen/lang/jpn/BigramNameFeatureGenerator.java
+++ /dev/null
@@ -1,41 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package opennlp.tools.util.featuregen.lang.jpn;
-
-import java.util.List;
-
-import opennlp.tools.util.featuregen.AdaptiveFeatureGenerator;
-
-public class BigramNameFeatureGenerator implements AdaptiveFeatureGenerator {
-
- public void createFeatures(List<String> features, String[] tokens, int index,
- String[] previousOutcomes) {
- String wc = FeatureGeneratorUtil.tokenFeature(tokens[index]);
- //bi-gram features
- if (index > 0) {
- features.add("pw,w=" + tokens[index - 1] + "," + tokens[index]);
- String pwc = FeatureGeneratorUtil.tokenFeature(tokens[index - 1]);
- features.add("pwc,wc=" + pwc + "," + wc);
- }
- if (index + 1 < tokens.length) {
- features.add("w,nw=" + tokens[index] + "," + tokens[index + 1]);
- String nwc = FeatureGeneratorUtil.tokenFeature(tokens[index + 1]);
- features.add("wc,nc=" + wc + "," + nwc);
- }
- }
-}
diff --git a/japanese/src/main/java/opennlp/tools/util/featuregen/lang/jpn/BigramNameFeatureGeneratorFactory.java b/japanese/src/main/java/opennlp/tools/util/featuregen/lang/jpn/BigramNameFeatureGeneratorFactory.java
deleted file mode 100644
index 50e2dfa..0000000
--- a/japanese/src/main/java/opennlp/tools/util/featuregen/lang/jpn/BigramNameFeatureGeneratorFactory.java
+++ /dev/null
@@ -1,35 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package opennlp.tools.util.featuregen.lang.jpn;
-
-import opennlp.tools.util.InvalidFormatException;
-import opennlp.tools.util.featuregen.AdaptiveFeatureGenerator;
-import opennlp.tools.util.featuregen.GeneratorFactory;
-
-public class BigramNameFeatureGeneratorFactory
- extends GeneratorFactory.AbstractXmlFeatureGeneratorFactory {
-
- public BigramNameFeatureGeneratorFactory() {
- super();
- }
-
- @Override
- public AdaptiveFeatureGenerator create() throws InvalidFormatException {
- return new BigramNameFeatureGenerator();
- }
-}
diff --git a/japanese/src/main/java/opennlp/tools/util/featuregen/lang/jpn/BrownClusterTokenClassFeatureGeneratorFactory.java b/japanese/src/main/java/opennlp/tools/util/featuregen/lang/jpn/BrownClusterTokenClassFeatureGeneratorFactory.java
deleted file mode 100644
index 6d005d3..0000000
--- a/japanese/src/main/java/opennlp/tools/util/featuregen/lang/jpn/BrownClusterTokenClassFeatureGeneratorFactory.java
+++ /dev/null
@@ -1,60 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package opennlp.tools.util.featuregen.lang.jpn;
-
-import java.util.HashMap;
-import java.util.Map;
-
-import opennlp.tools.util.InvalidFormatException;
-import opennlp.tools.util.featuregen.AdaptiveFeatureGenerator;
-import opennlp.tools.util.featuregen.BrownCluster;
-import opennlp.tools.util.featuregen.GeneratorFactory;
-import opennlp.tools.util.model.ArtifactSerializer;
-
-/**
- * Generates Brown clustering features for token classes.
- */
-public class BrownClusterTokenClassFeatureGeneratorFactory
- extends GeneratorFactory.AbstractXmlFeatureGeneratorFactory {
-
- public BrownClusterTokenClassFeatureGeneratorFactory() {
- super();
- }
-
- @Override
- public AdaptiveFeatureGenerator create() throws InvalidFormatException {
- // if resourceManager is null, we don't instantiate
- if (resourceManager == null)
- return null;
-
- String dictResourceKey = getStr("dict");
- Object dictResource = resourceManager.getResource(dictResourceKey);
- if (!(dictResource instanceof BrownCluster)) {
- throw new InvalidFormatException("Not a BrownLexicon resource for key: " + dictResourceKey);
- }
-
- return new BrownTokenClassFeatureGenerator((BrownCluster) dictResource);
- }
-
- @Override
- public Map<String, ArtifactSerializer<?>> getArtifactSerializerMapping() throws InvalidFormatException {
- Map<String, ArtifactSerializer<?>> mapping = new HashMap<>();
- mapping.put(getStr("dict"), new BrownCluster.BrownClusterSerializer());
- return mapping;
- }
-}
diff --git a/japanese/src/main/java/opennlp/tools/util/featuregen/lang/jpn/BrownTokenClassFeatureGenerator.java b/japanese/src/main/java/opennlp/tools/util/featuregen/lang/jpn/BrownTokenClassFeatureGenerator.java
deleted file mode 100644
index edae61f..0000000
--- a/japanese/src/main/java/opennlp/tools/util/featuregen/lang/jpn/BrownTokenClassFeatureGenerator.java
+++ /dev/null
@@ -1,49 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package opennlp.tools.util.featuregen.lang.jpn;
-
-import java.util.List;
-
-import opennlp.tools.util.featuregen.AdaptiveFeatureGenerator;
-import opennlp.tools.util.featuregen.BrownCluster;
-import opennlp.tools.util.featuregen.BrownTokenClasses;
-
-/**
- * Generates Brown cluster features for current token and token class.
- */
-public class BrownTokenClassFeatureGenerator implements AdaptiveFeatureGenerator {
-
- private final BrownCluster brownLexicon;
-
- public BrownTokenClassFeatureGenerator(BrownCluster dict) {
- this.brownLexicon = dict;
- }
-
- public void createFeatures(List<String> features, String[] tokens, int index,
- String[] previousOutcomes) {
-
- String wordShape = FeatureGeneratorUtil.tokenFeature(tokens[index]);
- List<String> wordClasses = BrownTokenClasses.getWordClasses(tokens[index], brownLexicon);
-
- for (String wordClass : wordClasses) {
- features.add("c," + "browncluster" + "=" + wordShape + "," + wordClass);
- }
- }
-
-}
-
diff --git a/japanese/src/main/java/opennlp/tools/util/featuregen/lang/jpn/FeatureGeneratorUtil.java b/japanese/src/main/java/opennlp/tools/util/featuregen/lang/jpn/FeatureGeneratorUtil.java
deleted file mode 100644
index 03b5659..0000000
--- a/japanese/src/main/java/opennlp/tools/util/featuregen/lang/jpn/FeatureGeneratorUtil.java
+++ /dev/null
@@ -1,90 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-
-package opennlp.tools.util.featuregen.lang.jpn;
-
-import java.util.Objects;
-
-/**
- * This class provide common utilities for feature generation.
- */
-public class FeatureGeneratorUtil {
-
- /**
- * Generates a class name for the specified token.
- * The classes are as follows where the first matching class is used:
- * <ul>
- * <li>hira - Japanese Hiragana</li>
- * <li>kata - Japanese Katakana</li>
- * <li>digit - all number letters</li>
- * <li>alpha - all alphabet letters</li>
- * <li>other - other </li>
- * </ul>
- * @param token A token or word.
- * @return The class name that the specified token belongs in.
- */
- public static String tokenFeature(String token) {
-
- Objects.requireNonNull(token, "token must be not null!");
-
- if (token.isEmpty()) return "other";
-
- // scan token only once
- char c = token.charAt(0);
- if (Character.isDigit(c)) {
- for (int i = 1; i < token.length(); i++) {
- c = token.charAt(i);
- if (!Character.isDigit(c)) return "other";
- }
- return "digit";
- }
- else {
- Character.UnicodeBlock ub = Character.UnicodeBlock.of(c);
- if (ub.equals(Character.UnicodeBlock.HIRAGANA)) {
- for (int i = 1; i < token.length(); i++) {
- c = token.charAt(i);
- if (c != '・' && c != 'ー' && c != '〜') {
- ub = Character.UnicodeBlock.of(c);
- if (!ub.equals(Character.UnicodeBlock.HIRAGANA)) return "other";
- }
- }
- return "hira";
- }
- else if (ub.equals(Character.UnicodeBlock.KATAKANA)) {
- for (int i = 1; i < token.length(); i++) {
- c = token.charAt(i);
- if (c != '・' && c != 'ー' && c != '〜') {
- ub = Character.UnicodeBlock.of(c);
- if (!ub.equals(Character.UnicodeBlock.KATAKANA)) return "other";
- }
- }
- return "kata";
- }
- else if (Character.isAlphabetic(c) &&
- !ub.equals(Character.UnicodeBlock.CJK_UNIFIED_IDEOGRAPHS)) {
- for (int i = 1; i < token.length(); i++) {
- c = token.charAt(i);
- if (!Character.isAlphabetic(c)) return "other";
- }
- return "alpha";
- }
- }
-
- return "other";
- }
-}
diff --git a/japanese/src/main/java/opennlp/tools/util/featuregen/lang/jpn/TokenClassFeatureGenerator.java b/japanese/src/main/java/opennlp/tools/util/featuregen/lang/jpn/TokenClassFeatureGenerator.java
deleted file mode 100644
index be4cadf..0000000
--- a/japanese/src/main/java/opennlp/tools/util/featuregen/lang/jpn/TokenClassFeatureGenerator.java
+++ /dev/null
@@ -1,53 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package opennlp.tools.util.featuregen.lang.jpn;
-
-import java.util.List;
-
-import opennlp.tools.util.StringUtil;
-import opennlp.tools.util.featuregen.AdaptiveFeatureGenerator;
-
-
-/**
- * Generates features for different for the class of the token.
- */
-public class TokenClassFeatureGenerator implements AdaptiveFeatureGenerator {
-
- private static final String TOKEN_CLASS_PREFIX = "wc";
- private static final String TOKEN_AND_CLASS_PREFIX = "w&c";
-
- private final boolean generateWordAndClassFeature;
-
- public TokenClassFeatureGenerator() {
- this(false);
- }
-
- public TokenClassFeatureGenerator(boolean generateWordAndClassFeature) {
- this.generateWordAndClassFeature = generateWordAndClassFeature;
- }
-
- public void createFeatures(List<String> features, String[] tokens, int index, String[] preds) {
- String wordClass = FeatureGeneratorUtil.tokenFeature(tokens[index]);
- features.add(TOKEN_CLASS_PREFIX + "=" + wordClass);
-
- if (generateWordAndClassFeature) {
- features.add(TOKEN_AND_CLASS_PREFIX + "=" + StringUtil.toLowerCase(tokens[index]) +
- "," + wordClass);
- }
- }
-}
diff --git a/japanese/src/main/java/opennlp/tools/util/featuregen/lang/jpn/TokenClassFeatureGeneratorFactory.java b/japanese/src/main/java/opennlp/tools/util/featuregen/lang/jpn/TokenClassFeatureGeneratorFactory.java
deleted file mode 100644
index dd7c0ed..0000000
--- a/japanese/src/main/java/opennlp/tools/util/featuregen/lang/jpn/TokenClassFeatureGeneratorFactory.java
+++ /dev/null
@@ -1,38 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package opennlp.tools.util.featuregen.lang.jpn;
-
-import opennlp.tools.util.InvalidFormatException;
-import opennlp.tools.util.featuregen.AdaptiveFeatureGenerator;
-import opennlp.tools.util.featuregen.GeneratorFactory;
-
-/**
- * @see TokenClassFeatureGenerator
- */
-public class TokenClassFeatureGeneratorFactory
- extends GeneratorFactory.AbstractXmlFeatureGeneratorFactory {
-
- public TokenClassFeatureGeneratorFactory() {
- super();
- }
-
- @Override
- public AdaptiveFeatureGenerator create() throws InvalidFormatException {
- return new TokenClassFeatureGenerator(getBool("wordAndClass", true));
- }
-}
diff --git a/japanese/src/main/java/opennlp/tools/util/featuregen/lang/jpn/TokenPatternFeatureGenerator.java b/japanese/src/main/java/opennlp/tools/util/featuregen/lang/jpn/TokenPatternFeatureGenerator.java
deleted file mode 100644
index 624a6bd..0000000
--- a/japanese/src/main/java/opennlp/tools/util/featuregen/lang/jpn/TokenPatternFeatureGenerator.java
+++ /dev/null
@@ -1,90 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-
-package opennlp.tools.util.featuregen.lang.jpn;
-
-import java.util.List;
-import java.util.regex.Pattern;
-
-import opennlp.tools.tokenize.SimpleTokenizer;
-import opennlp.tools.tokenize.Tokenizer;
-import opennlp.tools.util.StringUtil;
-import opennlp.tools.util.featuregen.AdaptiveFeatureGenerator;
-
-/**
- * Partitions tokens into sub-tokens based on character classes and generates
- * class features for each of the sub-tokens and combinations of those sub-tokens.
- */
-public class TokenPatternFeatureGenerator implements AdaptiveFeatureGenerator {
-
- private final Pattern noLetters = Pattern.compile("[^a-zA-Z]");
- private final Tokenizer tokenizer;
-
- /**
- * Initializes a new instance.
- * For tokinization the {@link SimpleTokenizer} is used.
- */
- public TokenPatternFeatureGenerator() {
- this(SimpleTokenizer.INSTANCE);
- }
-
- /**
- * Initializes a new instance.
- *
- * @param supportTokenizer
- */
- public TokenPatternFeatureGenerator(Tokenizer supportTokenizer) {
- tokenizer = supportTokenizer;
- }
-
- public void createFeatures(List<String> feats, String[] toks, int index, String[] preds) {
-
- String[] tokenized = tokenizer.tokenize(toks[index]);
-
- if (tokenized.length == 1) {
- feats.add("st=" + StringUtil.toLowerCase(toks[index]));
- return;
- }
-
- feats.add("stn=" + tokenized.length);
-
- StringBuilder pattern = new StringBuilder();
-
- for (int i = 0; i < tokenized.length; i++) {
-
- if (i < tokenized.length - 1) {
- feats.add("pt2=" + FeatureGeneratorUtil.tokenFeature(tokenized[i]) +
- FeatureGeneratorUtil.tokenFeature(tokenized[i + 1]));
- }
-
- if (i < tokenized.length - 2) {
- feats.add("pt3=" + FeatureGeneratorUtil.tokenFeature(tokenized[i]) +
- FeatureGeneratorUtil.tokenFeature(tokenized[i + 1]) +
- FeatureGeneratorUtil.tokenFeature(tokenized[i + 2]));
- }
-
- pattern.append(FeatureGeneratorUtil.tokenFeature(tokenized[i]));
-
- if (!noLetters.matcher(tokenized[i]).find()) {
- feats.add("st=" + StringUtil.toLowerCase(tokenized[i]));
- }
- }
-
- feats.add("pta=" + pattern.toString());
- }
-}
diff --git a/japanese/src/main/java/opennlp/tools/util/featuregen/lang/jpn/TokenPatternFeatureGeneratorFactory.java b/japanese/src/main/java/opennlp/tools/util/featuregen/lang/jpn/TokenPatternFeatureGeneratorFactory.java
deleted file mode 100644
index 35ccc08..0000000
--- a/japanese/src/main/java/opennlp/tools/util/featuregen/lang/jpn/TokenPatternFeatureGeneratorFactory.java
+++ /dev/null
@@ -1,38 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package opennlp.tools.util.featuregen.lang.jpn;
-
-import opennlp.tools.util.InvalidFormatException;
-import opennlp.tools.util.featuregen.AdaptiveFeatureGenerator;
-import opennlp.tools.util.featuregen.GeneratorFactory;
-
-/**
- * @see TokenPatternFeatureGenerator
- */
-public class TokenPatternFeatureGeneratorFactory
- extends GeneratorFactory.AbstractXmlFeatureGeneratorFactory {
-
- public TokenPatternFeatureGeneratorFactory() {
- super();
- }
-
- @Override
- public AdaptiveFeatureGenerator create() throws InvalidFormatException {
- return new TokenPatternFeatureGenerator();
- }
-}
diff --git a/japanese/src/main/java/opennlp/tools/util/featuregen/lang/jpn/TrigramNameFeatureGenerator.java b/japanese/src/main/java/opennlp/tools/util/featuregen/lang/jpn/TrigramNameFeatureGenerator.java
deleted file mode 100644
index 10a6af8..0000000
--- a/japanese/src/main/java/opennlp/tools/util/featuregen/lang/jpn/TrigramNameFeatureGenerator.java
+++ /dev/null
@@ -1,47 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package opennlp.tools.util.featuregen.lang.jpn;
-
-import java.util.List;
-
-import opennlp.tools.util.featuregen.AdaptiveFeatureGenerator;
-
-/**
- * Adds trigram features based on tokens and token classes.
- *
- */
-public class TrigramNameFeatureGenerator implements AdaptiveFeatureGenerator {
-
- public void createFeatures(List<String> features, String[] tokens, int index,
- String[] previousOutcomes) {
- String wc = FeatureGeneratorUtil.tokenFeature(tokens[index]);
- // trigram features
- if (index > 1) {
- features.add("ppw,pw,w=" + tokens[index - 2] + "," + tokens[index - 1] + "," + tokens[index]);
- String pwc = FeatureGeneratorUtil.tokenFeature(tokens[index - 1]);
- String ppwc = FeatureGeneratorUtil.tokenFeature(tokens[index - 2]);
- features.add("ppwc,pwc,wc=" + ppwc + "," + pwc + "," + wc);
- }
- if (index + 2 < tokens.length) {
- features.add("w,nw,nnw=" + tokens[index] + "," + tokens[index + 1] + "," + tokens[index + 2]);
- String nwc = FeatureGeneratorUtil.tokenFeature(tokens[index + 1]);
- String nnwc = FeatureGeneratorUtil.tokenFeature(tokens[index + 2]);
- features.add("wc,nwc,nnwc=" + wc + "," + nwc + "," + nnwc);
- }
- }
-}
diff --git a/japanese/src/main/java/opennlp/tools/util/featuregen/lang/jpn/TrigramNameFeatureGeneratorFactory.java b/japanese/src/main/java/opennlp/tools/util/featuregen/lang/jpn/TrigramNameFeatureGeneratorFactory.java
deleted file mode 100644
index 5b007b0..0000000
--- a/japanese/src/main/java/opennlp/tools/util/featuregen/lang/jpn/TrigramNameFeatureGeneratorFactory.java
+++ /dev/null
@@ -1,35 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package opennlp.tools.util.featuregen.lang.jpn;
-
-import opennlp.tools.util.InvalidFormatException;
-import opennlp.tools.util.featuregen.AdaptiveFeatureGenerator;
-import opennlp.tools.util.featuregen.GeneratorFactory;
-
-public class TrigramNameFeatureGeneratorFactory
- extends GeneratorFactory.AbstractXmlFeatureGeneratorFactory {
-
- public TrigramNameFeatureGeneratorFactory() {
- super();
- }
-
- @Override
- public AdaptiveFeatureGenerator create() throws InvalidFormatException {
- return new TrigramNameFeatureGenerator();
- }
-}
diff --git a/japanese/src/main/resources/opennlp/tools/namefind/ner-auxinfo-features.xml b/japanese/src/main/resources/opennlp/tools/namefind/ner-auxinfo-features.xml
deleted file mode 100644
index 1a1c272..0000000
--- a/japanese/src/main/resources/opennlp/tools/namefind/ner-auxinfo-features.xml
+++ /dev/null
@@ -1,59 +0,0 @@
-<!--
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
--->
-
-<!-- [EXPERIMENTAL] Auxiliary information name finder feature generator configuration -->
-<featureGenerators cache="true" name="nameFinder">
- <generator class="opennlp.tools.util.featuregen.AuxiliaryInfoAwareDelegateFeatureGeneratorFactory">
- <generator class="opennlp.tools.util.featuregen.WindowFeatureGeneratorFactory">
- <int name="prevLength">2</int>
- <int name="nextLength">2</int>
- <generator class="opennlp.tools.util.featuregen.lang.jpn.TokenClassFeatureGeneratorFactory"/>
- </generator>
- </generator>
- <generator class="opennlp.tools.util.featuregen.AuxiliaryInfoAwareDelegateFeatureGeneratorFactory">
- <generator class="opennlp.tools.util.featuregen.WindowFeatureGeneratorFactory">
- <int name="prevLength">2</int>
- <int name="nextLength">2</int>
- <generator class="opennlp.tools.util.featuregen.TokenFeatureGeneratorFactory"/>
- </generator>
- </generator>
- <generator class="opennlp.tools.util.featuregen.AuxiliaryInfoAwareDelegateFeatureGeneratorFactory">
- <bool name="useAux">true</bool>
- <generator class="opennlp.tools.util.featuregen.WindowFeatureGeneratorFactory">
- <int name="prevLength">2</int>
- <int name="nextLength">2</int>
- <generator class="opennlp.tools.util.featuregen.TokenFeatureGeneratorFactory"/>
- </generator>
- </generator>
- <generator class="opennlp.tools.util.featuregen.AuxiliaryInfoAwareDelegateFeatureGeneratorFactory">
- <generator class="opennlp.tools.util.featuregen.DefinitionFeatureGeneratorFactory"/>
- </generator>
- <generator class="opennlp.tools.util.featuregen.AuxiliaryInfoAwareDelegateFeatureGeneratorFactory">
- <generator class="opennlp.tools.util.featuregen.PreviousMapFeatureGeneratorFactory"/>
- </generator>
- <generator class="opennlp.tools.util.featuregen.AuxiliaryInfoAwareDelegateFeatureGeneratorFactory">
- <generator class="opennlp.tools.util.featuregen.lang.jpn.BigramNameFeatureGeneratorFactory"/>
- </generator>
- <generator class="opennlp.tools.util.featuregen.AuxiliaryInfoAwareDelegateFeatureGeneratorFactory">
- <generator class="opennlp.tools.util.featuregen.SentenceFeatureGeneratorFactory">
- <bool name="begin">true</bool>
- <bool name="end">false</bool>
- </generator>
- </generator>
-</featureGenerators>
diff --git a/japanese/src/test/java/opennlp/tools/namefind/AuxiliaryInfoUtilTest.java b/japanese/src/test/java/opennlp/tools/namefind/AuxiliaryInfoUtilTest.java
deleted file mode 100644
index db9bad2..0000000
--- a/japanese/src/test/java/opennlp/tools/namefind/AuxiliaryInfoUtilTest.java
+++ /dev/null
@@ -1,75 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package opennlp.tools.namefind;
-
-import org.junit.jupiter.api.Assertions;
-import org.junit.jupiter.api.Test;
-
-import static org.junit.jupiter.api.Assertions.assertEquals;
-
-public class AuxiliaryInfoUtilTest {
-
- @Test
- public void testGetSeparatorIndex() {
- assertEquals(0, AuxiliaryInfoUtil.getSeparatorIndex("/POStag"));
- assertEquals(1, AuxiliaryInfoUtil.getSeparatorIndex("1/POStag"));
- assertEquals(10, AuxiliaryInfoUtil.getSeparatorIndex("word/stuff/POStag"));
- }
-
- @Test
- public void testGetSeparatorIndexNoPos() {
- Assertions.assertThrows(RuntimeException.class, () ->
- AuxiliaryInfoUtil.getSeparatorIndex("NOPOStags"));
- }
-
- @Test
- public void testGetWordPart() {
- assertEquals(" ", AuxiliaryInfoUtil.getWordPart("/POStag"));
- assertEquals("1", AuxiliaryInfoUtil.getWordPart("1/POStag"));
- assertEquals("word", AuxiliaryInfoUtil.getWordPart("word/POStag"));
- assertEquals("word/stuff", AuxiliaryInfoUtil.getWordPart("word/stuff/POStag"));
- }
-
- @Test
- public void testGetWordParts() {
- String[] results = AuxiliaryInfoUtil.getWordParts(new String[]{"1/A", "234/B", "3456/C", "/D"});
- assertEquals(4, results.length);
- assertEquals("1", results[0]);
- assertEquals("234", results[1]);
- assertEquals("3456", results[2]);
- assertEquals(" ", results[3]);
- }
-
- @Test
- public void testGetAuxPart() {
- assertEquals("POStag", AuxiliaryInfoUtil.getAuxPart("/POStag"));
- assertEquals("POStag", AuxiliaryInfoUtil.getAuxPart("1/POStag"));
- assertEquals("POStag", AuxiliaryInfoUtil.getAuxPart("word/POStag"));
- assertEquals("POStag", AuxiliaryInfoUtil.getAuxPart("word/stuff/POStag"));
- }
-
- @Test
- public void testGetAuxParts() {
- String[] results = AuxiliaryInfoUtil.getAuxParts(new String[] {"1/ABC", "234/B", "3456/CD", "/DEFGH"});
- assertEquals(4, results.length);
- assertEquals("ABC", results[0]);
- assertEquals("B", results[1]);
- assertEquals("CD", results[2]);
- assertEquals("DEFGH", results[3]);
- }
-}
diff --git a/japanese/src/test/java/opennlp/tools/util/featuregen/AuxiliaryInfoAwareDelegateFeatureGeneratorTest.java b/japanese/src/test/java/opennlp/tools/util/featuregen/AuxiliaryInfoAwareDelegateFeatureGeneratorTest.java
deleted file mode 100644
index 40f34e6..0000000
--- a/japanese/src/test/java/opennlp/tools/util/featuregen/AuxiliaryInfoAwareDelegateFeatureGeneratorTest.java
+++ /dev/null
@@ -1,66 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package opennlp.tools.util.featuregen;
-
-import java.util.ArrayList;
-import java.util.List;
-
-import org.junit.jupiter.api.BeforeEach;
-import org.junit.jupiter.api.Test;
-
-import static org.junit.jupiter.api.Assertions.assertEquals;
-
-public class AuxiliaryInfoAwareDelegateFeatureGeneratorTest {
-
- private final String[] testSentence = "w1/pos1 w2/pos2 w3/pos3 w4/pos4".split("\\s+");
-
- private List<String> features;
-
- @BeforeEach
- public void setUp() {
- features = new ArrayList<>();
- }
-
- @Test
- public void testWord() {
- AdaptiveFeatureGenerator featureGenerator = new AuxiliaryInfoAwareDelegateFeatureGenerator(
- new IdentityFeatureGenerator(), false);
-
- featureGenerator.createFeatures(features, testSentence, 2, null);
- assertEquals(1, features.size());
- assertEquals("w3", features.get(0));
- }
-
- @Test
- public void testAuxInfo() {
- AdaptiveFeatureGenerator featureGenerator = new AuxiliaryInfoAwareDelegateFeatureGenerator(
- new IdentityFeatureGenerator(), true);
-
- featureGenerator.createFeatures(features, testSentence, 3, null);
- assertEquals(1, features.size());
- assertEquals("pos4", features.get(0));
- }
-
- static class IdentityFeatureGenerator implements AdaptiveFeatureGenerator {
-
- @Override
- public void createFeatures(List<String> features, String[] tokens, int index, String[] previousOutcomes) {
- features.add(tokens[index]);
- }
- }
-}
diff --git a/japanese/src/test/java/opennlp/tools/util/featuregen/lang/jpn/BigramNameFeatureGeneratorTest.java b/japanese/src/test/java/opennlp/tools/util/featuregen/lang/jpn/BigramNameFeatureGeneratorTest.java
deleted file mode 100644
index 83fdb18..0000000
--- a/japanese/src/test/java/opennlp/tools/util/featuregen/lang/jpn/BigramNameFeatureGeneratorTest.java
+++ /dev/null
@@ -1,96 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package opennlp.tools.util.featuregen.lang.jpn;
-
-import java.util.ArrayList;
-import java.util.List;
-
-import org.junit.jupiter.api.BeforeEach;
-import org.junit.jupiter.api.Test;
-
-import opennlp.tools.util.featuregen.AdaptiveFeatureGenerator;
-
-import static org.junit.jupiter.api.Assertions.assertEquals;
-
-public class BigramNameFeatureGeneratorTest {
-
- private List<String> features;
- private static final String[] TEST_SENTENCE = new String[] {"This", "is", "an", "example", "sentence"};
-
- @BeforeEach
- public void setUp() {
- features = new ArrayList<>();
- }
-
- @Test
- public void testBegin() {
-
- final int testTokenIndex = 0;
-
- AdaptiveFeatureGenerator generator = new BigramNameFeatureGenerator();
-
- generator.createFeatures(features, TEST_SENTENCE, testTokenIndex, null);
-
- assertEquals(2, features.size());
- assertEquals("w,nw=This,is", features.get(0));
- assertEquals("wc,nc=alpha,alpha", features.get(1));
- }
-
- @Test
- public void testMiddle() {
-
- final int testTokenIndex = 2;
-
- AdaptiveFeatureGenerator generator = new BigramNameFeatureGenerator();
-
- generator.createFeatures(features, TEST_SENTENCE, testTokenIndex, null);
-
- assertEquals(4, features.size());
- assertEquals("pw,w=is,an", features.get(0));
- assertEquals("pwc,wc=alpha,alpha", features.get(1));
- assertEquals("w,nw=an,example", features.get(2));
- assertEquals("wc,nc=alpha,alpha", features.get(3));
- }
-
- @Test
- public void testEnd() {
-
- final int testTokenIndex = 4;
-
- AdaptiveFeatureGenerator generator = new BigramNameFeatureGenerator();
-
- generator.createFeatures(features, TEST_SENTENCE, testTokenIndex, null);
-
- assertEquals(2, features.size());
- assertEquals("pw,w=example,sentence", features.get(0));
- assertEquals("pwc,wc=alpha,alpha", features.get(1));
- }
-
- @Test
- public void testShort() {
-
- String[] shortSentence = new String[] {"word"};
-
- final int testTokenIndex = 0;
-
- AdaptiveFeatureGenerator generator = new BigramNameFeatureGenerator();
- generator.createFeatures(features, shortSentence, testTokenIndex, null);
-
- assertEquals(0, features.size());
- }
-}
diff --git a/japanese/src/test/java/opennlp/tools/util/featuregen/lang/jpn/FeatureGeneratorUtilTest.java b/japanese/src/test/java/opennlp/tools/util/featuregen/lang/jpn/FeatureGeneratorUtilTest.java
deleted file mode 100644
index 1b52161..0000000
--- a/japanese/src/test/java/opennlp/tools/util/featuregen/lang/jpn/FeatureGeneratorUtilTest.java
+++ /dev/null
@@ -1,68 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package opennlp.tools.util.featuregen.lang.jpn;
-
-import org.junit.jupiter.api.Test;
-
-import static org.junit.jupiter.api.Assertions.assertEquals;
-
-public class FeatureGeneratorUtilTest {
-
- @Test
- public void test() {
- // digits
- assertEquals("digit", FeatureGeneratorUtil.tokenFeature("12"));
- assertEquals("digit", FeatureGeneratorUtil.tokenFeature("1234"));
- assertEquals("other", FeatureGeneratorUtil.tokenFeature("abcd234"));
- assertEquals("other", FeatureGeneratorUtil.tokenFeature("1234-56"));
- assertEquals("other", FeatureGeneratorUtil.tokenFeature("4/6/2017"));
- assertEquals("other", FeatureGeneratorUtil.tokenFeature("1,234,567"));
- assertEquals("other", FeatureGeneratorUtil.tokenFeature("12.34567"));
- assertEquals("other", FeatureGeneratorUtil.tokenFeature("123(456)7890"));
-
- // letters
- assertEquals("alpha", FeatureGeneratorUtil.tokenFeature("opennlp"));
- assertEquals("alpha", FeatureGeneratorUtil.tokenFeature("O"));
- assertEquals("alpha", FeatureGeneratorUtil.tokenFeature("OPENNLP"));
- assertEquals("other", FeatureGeneratorUtil.tokenFeature("A."));
- assertEquals("alpha", FeatureGeneratorUtil.tokenFeature("Mike"));
- assertEquals("alpha", FeatureGeneratorUtil.tokenFeature("somethingStupid"));
-
- // symbols
- assertEquals("other", FeatureGeneratorUtil.tokenFeature(","));
- assertEquals("other", FeatureGeneratorUtil.tokenFeature("."));
- assertEquals("other", FeatureGeneratorUtil.tokenFeature("?"));
- assertEquals("other", FeatureGeneratorUtil.tokenFeature("!"));
- assertEquals("other", FeatureGeneratorUtil.tokenFeature("#"));
- assertEquals("other", FeatureGeneratorUtil.tokenFeature("%"));
- assertEquals("other", FeatureGeneratorUtil.tokenFeature("&"));
- }
-
- @Test
- public void testJapanese() {
- // Hiragana
- assertEquals("hira", FeatureGeneratorUtil.tokenFeature("そういえば"));
- assertEquals("hira", FeatureGeneratorUtil.tokenFeature("おーぷん・そ〜す・そふとうぇあ"));
- assertEquals("other", FeatureGeneratorUtil.tokenFeature("あぱっち・そふとうぇあ財団"));
-
- // Katakana
- assertEquals("kata", FeatureGeneratorUtil.tokenFeature("ジャパン"));
- assertEquals("kata", FeatureGeneratorUtil.tokenFeature("オープン・ソ〜ス・ソフトウェア"));
- assertEquals("other", FeatureGeneratorUtil.tokenFeature("アパッチ・ソフトウェア財団"));
- }
-}
diff --git a/japanese/src/test/java/opennlp/tools/util/featuregen/lang/jpn/TokenClassFeatureGeneratorTest.java b/japanese/src/test/java/opennlp/tools/util/featuregen/lang/jpn/TokenClassFeatureGeneratorTest.java
deleted file mode 100644
index b7096eb..0000000
--- a/japanese/src/test/java/opennlp/tools/util/featuregen/lang/jpn/TokenClassFeatureGeneratorTest.java
+++ /dev/null
@@ -1,66 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package opennlp.tools.util.featuregen.lang.jpn;
-
-import java.util.ArrayList;
-import java.util.List;
-
-import org.junit.jupiter.api.BeforeEach;
-import org.junit.jupiter.api.Test;
-
-import opennlp.tools.util.featuregen.AdaptiveFeatureGenerator;
-
-import static org.junit.jupiter.api.Assertions.assertEquals;
-
-public class TokenClassFeatureGeneratorTest {
-
- private List<String> features;
- private static final String[] TEST_SENTENCE = new String[] {"This", "is", "an", "Example", "sentence"};
-
- @BeforeEach
- public void setUp() {
- features = new ArrayList<>();
- }
-
- @Test
- public void testGenWAC() {
-
- final int testTokenIndex = 3;
-
- AdaptiveFeatureGenerator generator = new TokenClassFeatureGenerator(true);
-
- generator.createFeatures(features, TEST_SENTENCE, testTokenIndex, null);
-
- assertEquals(2, features.size());
- assertEquals("wc=alpha", features.get(0));
- assertEquals("w&c=example,alpha", features.get(1));
- }
-
- @Test
- public void testNoWAC() {
-
- final int testTokenIndex = 3;
-
- AdaptiveFeatureGenerator generator = new TokenClassFeatureGenerator(false);
-
- generator.createFeatures(features, TEST_SENTENCE, testTokenIndex, null);
-
- assertEquals(1, features.size());
- assertEquals("wc=alpha", features.get(0));
- }
-}
diff --git a/japanese/src/test/java/opennlp/tools/util/featuregen/lang/jpn/TokenPatternFeatureGeneratorTest.java b/japanese/src/test/java/opennlp/tools/util/featuregen/lang/jpn/TokenPatternFeatureGeneratorTest.java
deleted file mode 100644
index de2fd26..0000000
--- a/japanese/src/test/java/opennlp/tools/util/featuregen/lang/jpn/TokenPatternFeatureGeneratorTest.java
+++ /dev/null
@@ -1,77 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package opennlp.tools.util.featuregen.lang.jpn;
-
-import java.util.ArrayList;
-import java.util.List;
-
-import org.junit.jupiter.api.BeforeEach;
-import org.junit.jupiter.api.Test;
-
-import opennlp.tools.util.featuregen.AdaptiveFeatureGenerator;
-
-import static org.junit.jupiter.api.Assertions.assertEquals;
-
-public class TokenPatternFeatureGeneratorTest {
-
- private List<String> features;
-
- @BeforeEach
- public void setUp() {
- features = new ArrayList<>();
- }
-
- @Test
- public void testSingleToken() {
-
- String[] testSentence = new String[] {"This", "is", "an", "example", "sentence"};
- final int testTokenIndex = 3;
-
- AdaptiveFeatureGenerator generator = new TokenPatternFeatureGenerator();
-
- generator.createFeatures(features, testSentence, testTokenIndex, null);
- assertEquals(1, features.size());
- assertEquals("st=example", features.get(0));
- }
-
- @Test
- public void testSentence() {
-
- String[] testSentence = new String[] {"This is an example sentence"};
- final int testTokenIndex = 0;
-
- AdaptiveFeatureGenerator generator = new TokenPatternFeatureGenerator();
-
- generator.createFeatures(features, testSentence, testTokenIndex, null);
- assertEquals(14, features.size());
- assertEquals("stn=5", features.get(0));
- assertEquals("pt2=alphaalpha", features.get(1));
- assertEquals("pt3=alphaalphaalpha", features.get(2));
- assertEquals("st=this", features.get(3));
- assertEquals("pt2=alphaalpha", features.get(4));
- assertEquals("pt3=alphaalphaalpha", features.get(5));
- assertEquals("st=is", features.get(6));
- assertEquals("pt2=alphaalpha", features.get(7));
- assertEquals("pt3=alphaalphaalpha", features.get(8));
- assertEquals("st=an", features.get(9));
- assertEquals("pt2=alphaalpha", features.get(10));
- assertEquals("st=example", features.get(11));
- assertEquals("st=sentence", features.get(12));
- assertEquals("pta=alphaalphaalphaalphaalpha", features.get(13));
- }
-}
diff --git a/japanese/src/test/java/opennlp/tools/util/featuregen/lang/jpn/TrigramNameFeatureGeneratorTest.java b/japanese/src/test/java/opennlp/tools/util/featuregen/lang/jpn/TrigramNameFeatureGeneratorTest.java
deleted file mode 100644
index ba397db..0000000
--- a/japanese/src/test/java/opennlp/tools/util/featuregen/lang/jpn/TrigramNameFeatureGeneratorTest.java
+++ /dev/null
@@ -1,111 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package opennlp.tools.util.featuregen.lang.jpn;
-
-import java.util.ArrayList;
-import java.util.List;
-
-import org.junit.jupiter.api.BeforeEach;
-import org.junit.jupiter.api.Test;
-
-import opennlp.tools.util.featuregen.AdaptiveFeatureGenerator;
-
-import static org.junit.jupiter.api.Assertions.assertEquals;
-
-public class TrigramNameFeatureGeneratorTest {
-
- private List<String> features;
- private static final String[] TEST_SENTENCE = new String[] {"This", "is", "an", "example", "sentence"};
-
- @BeforeEach
- public void setUp() {
- features = new ArrayList<>();
- }
-
- @Test
- public void testBegin() {
-
- final int testTokenIndex = 0;
-
- AdaptiveFeatureGenerator generator = new TrigramNameFeatureGenerator();
-
- generator.createFeatures(features, TEST_SENTENCE, testTokenIndex, null);
-
- assertEquals(2, features.size());
- assertEquals("w,nw,nnw=This,is,an", features.get(0));
- assertEquals("wc,nwc,nnwc=alpha,alpha,alpha", features.get(1));
- }
-
- @Test
- public void testNextOfBegin() {
-
- final int testTokenIndex = 1;
-
- AdaptiveFeatureGenerator generator = new TrigramNameFeatureGenerator();
-
- generator.createFeatures(features, TEST_SENTENCE, testTokenIndex, null);
-
- assertEquals(2, features.size());
- assertEquals("w,nw,nnw=is,an,example", features.get(0));
- assertEquals("wc,nwc,nnwc=alpha,alpha,alpha", features.get(1));
- }
-
- @Test
- public void testMiddle() {
-
- final int testTokenIndex = 2;
-
- AdaptiveFeatureGenerator generator = new TrigramNameFeatureGenerator();
-
- generator.createFeatures(features, TEST_SENTENCE, testTokenIndex, null);
-
- assertEquals(4, features.size());
- assertEquals("ppw,pw,w=This,is,an", features.get(0));
- assertEquals("ppwc,pwc,wc=alpha,alpha,alpha", features.get(1));
- assertEquals("w,nw,nnw=an,example,sentence", features.get(2));
- assertEquals("wc,nwc,nnwc=alpha,alpha,alpha", features.get(3));
- }
-
- @Test
- public void testEnd() {
-
- final int testTokenIndex = 4;
-
- AdaptiveFeatureGenerator generator = new TrigramNameFeatureGenerator();
-
- generator.createFeatures(features, TEST_SENTENCE, testTokenIndex, null);
-
- assertEquals(2, features.size());
- assertEquals("ppw,pw,w=an,example,sentence", features.get(0));
- assertEquals("ppwc,pwc,wc=alpha,alpha,alpha", features.get(1));
- }
-
- @Test
- public void testShort() {
-
- String[] shortSentence = new String[] {"I", "know", "it"};
-
- final int testTokenIndex = 1;
-
- AdaptiveFeatureGenerator generator = new TrigramNameFeatureGenerator();
-
- generator.createFeatures(features, shortSentence, testTokenIndex, null);
-
- assertEquals(0, features.size());
- }
-}
diff --git a/opennlp-addons-docs/src/docbkx/building.xml b/opennlp-addons-docs/src/docbkx/building.xml
index 4d1f023..65d6a37 100644
--- a/opennlp-addons-docs/src/docbkx/building.xml
+++ b/opennlp-addons-docs/src/docbkx/building.xml
@@ -37,7 +37,7 @@
classes, unit and integration tests, RAT license-header checks, and dependency-license reporting.
Add <code>-Pjacoco</code> to generate coverage reports.</para>
<para>Build one module and its reactor dependencies with <code>-pl</code> and <code>-am</code>:</para>
- <screen><![CDATA[mvn -pl japanese -am verify -Dopennlp.forkCount=1]]></screen>
+ <screen><![CDATA[mvn -pl <module> -am verify -Dopennlp.forkCount=1]]></screen>
</section>
<section xml:id="building.manual">
<title>Generate the manual</title>
diff --git a/opennlp-addons-docs/src/docbkx/modules.xml b/opennlp-addons-docs/src/docbkx/modules.xml
index bd33f2b..eb83f5a 100644
--- a/opennlp-addons-docs/src/docbkx/modules.xml
+++ b/opennlp-addons-docs/src/docbkx/modules.xml
@@ -31,10 +31,6 @@
<entry><code>geoentitylinker</code></entry>
<entry>Detects and links geographic entities</entry>
</row>
- <row>
- <entry><code>japanese</code></entry>
- <entry>Adds features for Japanese text processing</entry>
- </row>
<row>
<entry><code>liblinear</code></entry>
<entry>Supplies LIBLINEAR-backed model support</entry>
diff --git a/opennlp-addons-docs/src/docbkx/using-addons.xml b/opennlp-addons-docs/src/docbkx/using-addons.xml
index a9f71fc..97d1173 100644
--- a/opennlp-addons-docs/src/docbkx/using-addons.xml
+++ b/opennlp-addons-docs/src/docbkx/using-addons.xml
@@ -28,7 +28,7 @@
<code>opennlp-addons</code> artifact is a build POM and is not an application dependency.</para>
<programlisting language="xml"><![CDATA[<dependency>
<groupId>org.apache.opennlp.addons</groupId>
- <artifactId>japanese</artifactId>
+ <artifactId>${addon.artifactId}</artifactId>
<version>${opennlp-addons.version}</version>
</dependency>]]></programlisting>
<para>Use the same add-ons version for each add-on in one application.</para>
@@ -39,7 +39,7 @@
against the OpenNLP 3.x release selected by the root <code>opennlp.version</code> property. A module
that requires an API still under review can override that property on its development branch.</para>
<para>Before a 3.x release is available from Maven Central, install the selected add-on from a local checkout:</para>
- <screen><![CDATA[mvn -pl japanese -am install]]></screen>
+ <screen><![CDATA[mvn -pl <module> -am install]]></screen>
<para>Then use <code><?eval ${project.version}?></code> as <code>opennlp-addons.version</code> in the
consuming build.</para>
</section>
diff --git a/pom.xml b/pom.xml
index bd26cfb..6388b36 100644
--- a/pom.xml
+++ b/pom.xml
@@ -109,7 +109,6 @@
<modules>
<module>opennlp-addons-docs</module>
<module>geoentitylinker</module>
- <module>japanese</module>
<module>liblinear</module>
<module>modelbuilder</module>
<module>jwnl</module>