You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hivemall.apache.org by helenahm <gi...@git.apache.org> on 2017/07/02 04:21:55 UTC
[GitHub] incubator-hivemall pull request #93: Maximum Entropy Model
GitHub user helenahm opened a pull request:
https://github.com/apache/incubator-hivemall/pull/93
Maximum Entropy Model
## What changes were proposed in this pull request?
A Distributed Max Entropy Model
## What type of PR is it?
Feature
## What is the Jira issue?
?
## How was this patch tested?
There are two tests at the moment, hivemall.smile.classification.MaxEntUDTFTest.java
and hivemall.smile.tools.TreePredictUDFTest.java
plus I have tested the code on EMR:
add jar hivemall-core-0.4.2-rc.2-maxent-with-dependencies.jar;
add jar opennlp-maxent-3.0.0.jar;
source define-all.hive;
create temporary function train_maxent_classifier as 'hivemall.smile.classification.MaxEntUDTF';
create temporary function predict_maxent_classifier as 'hivemall.smile.tools.MaxEntPredictUDF';
drop table tmodel_maxent;
CREATE TABLE tmodel_maxent
STORED AS SEQUENCEFILE
AS
select
train_maxent_classifier(features, klass, "-attrs
Q,Q,Q,Q,Q,Q,Q,Q,Q,Q,Q,Q,Q,Q,Q,Q,Q,Q,Q,Q,Q,Q,Q,Q,Q,Q,Q,Q,Q,Q,Q,Q,Q,Q,Q,Q,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,Q,Q,Q,Q,Q,Q,Q,Q")
from
t_test_maxent;
create table tmodel_combined as
select model, attributes, features, klass from t_test_maxent join tmodel_maxent;
create table tmodel_predicted as
select
predict_maxent_classifier(model, attributes, features) result, klass from tmodel_combined;
Source table:
drop table t_test_maxent;
create table t_test_maxent as select
array( x1,x2,x3,x4,x5,x6,x7,x8,x9,x10,x11,x12,x13,x14,x15,x16,x17,x18,x19,x20,x21,x22,x23,x24,x25,x26,x27,x28,x29,x30,x31,x32,x33,x34,x35,x36,
cast(tWord(x37) as double),
cast(tWord(x38) as double),
cast(tWord(x39) as double),
cast(tWord(x40) as double),
cast(tWord(x41) as double),
cast(tWord(x42) as double),
cast(tWord(x43) as double),
cast(tWord(x44) as double),
cast(contentWord(x45) as double),
cast(contentWord(x46) as double),
cast(contentWord(x47) as double),
cast(contentWord(x48) as double),
cast(contentWord(x49) as double),
cast(contentWord(x50) as double),
cast(contentWord(x51) as double),
cast(contentWord(x52) as double),
cast(contentWord(x53) as double),
cast(presentationWord(x54) as double),
cast(presentationWord(x55) as double),
cast(presentationWord(x56) as double),
cast(presentationWord(x57) as double),
cast(presentationWord(x58) as double),
cast(presentationWord(x59) as double),
cast(presentationWord(x60) as double),
cast(presentationWord(x61) as double),
cast(presentationWord(x62) as double),
x63,x64,x65,x66,x67,x68,x69,x70) features
, klass from pdfs_and_tiffs_instances_combined_instances where regexp_replace(tp, 'T', '') == '76_698_855_347';
## How to use this feature?
Maximum Entropy Classifier is, from my point of view, the most useful classification technique for many NLP tasks and many other tasks that are not related to NLP. It is used for part of speech tagging, NER, and some other tasks.
I have been searching for a distributed version of it and found one article only that talks about it. "Efficient Large Scale Distributed Training of Conditional Maximum Entropy Models" by Mehryar Mohri [quite well-known] and his colleagues at Google. (Please, let me know how I can send you the article if you will not get it by googling). Thus, I think it is time to implement that. I plan to use Mixture Weight Method they describe.
By now a final udaf is still to be implemented (the one that collects all the models and averages the weights), that I plan to commit next week.
See if you like the idea and will accept the code. It is based on Apache maxent, that is open source and is written in a simple way.
Regards,
Elena.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/helenahm/incubator-hivemall master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-hivemall/pull/93.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #93
----
commit 45a656aa7278066ce3fc36fcd81fb1eca11f1079
Author: helenahm <he...@users.noreply.github.com>
Date: 2017-06-02T05:10:13Z
Update LDAUDTFTest.java
commit fef9c1ce719d3924a28cc90d71d40728dc5c7563
Author: helenahm <he...@users.noreply.github.com>
Date: 2017-06-02T05:22:54Z
Merge pull request #1 from helenahm/helenahm-patch-1
Update LDAUDTFTest.java
commit e92b13aa3cb4fc193ea3da3fadd8a8fe8a6a073b
Author: AKHMATOVA, Elena <el...@suncorp.com.au>
Date: 2017-07-02T03:41:14Z
maxent
commit d4031550f80007045353f1e24e58c99244ab3db3
Author: AKHMATOVA, Elena <el...@suncorp.com.au>
Date: 2017-07-02T03:49:16Z
maxent cont.
commit f921d91fe8a1958cfd198236219c129355ef2fea
Author: AKHMATOVA, Elena <el...@suncorp.com.au>
Date: 2017-07-02T03:54:38Z
maxent cont.
commit 2a712edfd9bbe765bb2781f84b519e283fe6bd56
Author: helenahm <he...@users.noreply.github.com>
Date: 2017-07-02T03:59:59Z
Update LDAUDTFTest.java
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #93: [WIP][HIVEMALL-126] Maximum Entropy Model
Posted by helenahm <gi...@git.apache.org>.
Github user helenahm commented on the issue:
https://github.com/apache/incubator-hivemall/pull/93
I will do more tests too, as I actually need the model for a project. So I plan to test it under "load" too. I will write about the results.
It may have similar issues that Random Forest has. You are right. In a nutshell the implementation and memory concerns are similar.
The implementation is as scalable as the implementation of Random Forest: one or more models per mapper and then a UDAF that combines all the learned models into one final model.
I still use the Random Forest even though on EMR r4 machines _numTrees greater than 1 does not work for me for my dataset. MaxEnt though will give me a better model, I think, I will not have to think whether there is overfitting because of the tree structure, etc.
Iterative Scaling can be re-written from scratch too without using any third-party software. This is an option too.
I am sure that NLP community will more likely accept the implementation and will use it in exactly the way those guys have written it. We very much value Adwait Ratnaparkhi's work. Many published articles use exactly that Max Ent implementation. That means that people will be able to use HiveMall and compare their newer results with results of their previous work.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #93: [WIP][HIVEMALL-126] Maximum Entropy Model usin...
Posted by helenahm <gi...@git.apache.org>.
Github user helenahm commented on the issue:
https://github.com/apache/incubator-hivemall/pull/93
Yes. I will do all that in the next few days.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #93: [WIP][HIVEMALL-126] Maximum Entropy Model usin...
Posted by myui <gi...@git.apache.org>.
Github user myui commented on the issue:
https://github.com/apache/incubator-hivemall/pull/93
Not sure. Please avoid about it for coveralls.
It seems we need to fix Travis config due to this issue.
https://github.com/travis-ci/travis-ci/issues/7964
Verbose outputs from `MaxEntUDTFTest` and `MaxEntPredictUDFTest` should be avoided.
https://travis-ci.org/apache/incubator-hivemall/jobs/256887101
Also, it seems CI is killed due to some reason while the latest commit in the master is build passing.
https://docs.travis-ci.com/user/common-build-problems/#My-build-script-is-killed-without-any-error
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #93: Maximum Entropy Model
Posted by helenahm <gi...@git.apache.org>.
Github user helenahm commented on the issue:
https://github.com/apache/incubator-hivemall/pull/93
The official page for OpenNLP MaxEnt:
http://maxent.sourceforge.net/about.html
Models choice is 100% NLP influenced.
The opennlp.maxent package was originally built by Jason Baldridge, Tom Morton, and Gann Bierner. We owe a big thanks to Adwait Ratnaparkhi for his work on maximum entropy models for natural language processing applications. His introduction to maxent for NLP and dissertation are what really made opennlp.maxent and our Grok maxent components (POS tagger, end of sentence detector, tokenizer, name finder) possible!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #93: [WIP][HIVEMALL-126] Maximum Entropy Model
Posted by myui <gi...@git.apache.org>.
Github user myui commented on the issue:
https://github.com/apache/incubator-hivemall/pull/93
• Shall I commit all the changes? Or would it better to pinpoint my own files only?
Just push your files by `git add <select only your files>` .
• Could you rename the title of this PR to `[WIP][HVIEMALL-126] Maximum Entropy Model using OpenNLP` to clarify this PR more?
• As commented in the review, why you are using multiple threads for training and OOB tests? Single thread execution is enough for MaxEntropy classifier and your OOB test scheme is incorrect.
• I have a little concern about memory usage of OpenNLP MaxEnt implementation. Not like other online classifier implementation, it holds entire training data in memory. Hivemall's RandomForest implementation holds entire dataset in memory but tried to consume less memory using CSRMatrix.
Max Entropy classifier is also known as Multinominal Logistic Regression. It might be better to evaluate [Smile's one](https://github.com/haifengl/smile/blob/master/core/src/main/java/smile/classification/LogisticRegression.java#L264) as well. I need to evaluate memory consumptions of your implementation and OpenNLP maxent because each task of Hadoop workers can use limited relatively small memory space.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall pull request #93: Maximum Entropy Model
Posted by myui <gi...@git.apache.org>.
Github user myui commented on a diff in the pull request:
https://github.com/apache/incubator-hivemall/pull/93#discussion_r125178438
--- Diff: core/src/main/java/hivemall/smile/tools/MaxEntPredictUDF.java ---
@@ -0,0 +1,174 @@
+package hivemall.smile.tools;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.LinkedList;
+import java.util.List;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import org.apache.hadoop.hive.ql.exec.UDFArgumentException;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.udf.generic.GenericUDF;
+import org.apache.hadoop.hive.ql.udf.generic.GenericUDF.DeferredObject;
+import org.apache.hadoop.hive.serde2.io.DoubleWritable;
+import org.apache.hadoop.hive.serde2.objectinspector.ListObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory;
+import org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;
+import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils;
+import org.apache.hadoop.hive.serde2.objectinspector.primitive.StringObjectInspector;
+import org.apache.hadoop.io.IntWritable;
+import org.apache.hadoop.io.Text;
+
+import hivemall.math.matrix.builders.CSRMatrixBuilder;
+import hivemall.math.matrix.builders.MatrixBuilder;
+import hivemall.math.vector.DenseVector;
+import hivemall.math.vector.SparseVector;
+import hivemall.math.vector.Vector;
+import hivemall.smile.classification.DecisionTree;
+import hivemall.smile.classification.PredictionHandler;
+import hivemall.smile.data.Attribute;
+import hivemall.smile.data.Attribute.AttributeType;
+import hivemall.smile.regression.RegressionTree;
+import hivemall.smile.utils.SmileExtUtils;
+import hivemall.utils.codec.Base91;
+import hivemall.utils.hadoop.HiveUtils;
+import hivemall.utils.hadoop.WritableUtils;
+import hivemall.utils.lang.Preconditions;
+import opennlp.model.AbstractModel;
+import opennlp.model.GenericModelReader;
+import opennlp.model.MaxentModel;
+import opennlp.model.RealValueFileEventStream;
+
+public class MaxEntPredictUDF extends GenericUDF {
+
+ private StringObjectInspector modelOI;
+ private StringObjectInspector attributesOI;
+ private ListObjectInspector featureListOI;
+ private PrimitiveObjectInspector featureElemOI;
+
+ @Nullable
+ private Vector featuresProbe;
+ private Attribute[] attributes;
+
+ @Nullable
+ private transient MaxentModel evaluator;
+
+ @Override
+ public ObjectInspector initialize(ObjectInspector[] argOIs) throws UDFArgumentException {
+ if (argOIs.length != 3) {
+ throw new UDFArgumentException("_FUNC_ takes 3");
+ }
+
+ this.modelOI = HiveUtils.asStringOI(argOIs[0]);
+ this.attributesOI = HiveUtils.asStringOI(argOIs[1]);
+
+ ListObjectInspector listOI = HiveUtils.asListOI(argOIs[2]);
+ this.featureListOI = listOI;
+ ObjectInspector elemOI = listOI.getListElementObjectInspector();
+ if (HiveUtils.isNumberOI(elemOI)) {
+ this.featureElemOI = HiveUtils.asDoubleCompatibleOI(elemOI);
+ } else {
+ throw new UDFArgumentException(
+ "_FUNC_ takes array<double> for the second argument: "
+ + listOI.getTypeName());
+ }
+
+ List<String> fieldNames = new ArrayList<String>(2);
+ List<ObjectInspector> fieldOIs = new ArrayList<ObjectInspector>(2);
+ fieldNames.add("value");
+ fieldOIs.add(PrimitiveObjectInspectorFactory.writableStringObjectInspector);
+ fieldNames.add("posteriori");
+ fieldOIs.add(ObjectInspectorFactory.getStandardListObjectInspector(PrimitiveObjectInspectorFactory.writableDoubleObjectInspector));
+ return ObjectInspectorFactory.getStandardStructObjectInspector(fieldNames, fieldOIs);
+ }
+
+ @Override
+ public Object evaluate(@Nonnull DeferredObject[] arguments) throws HiveException {
+ Object arg0 = arguments[0].get();
+ if (arg0 == null) {
+ throw new HiveException("Model was null");
+ }
+
+ Text model = modelOI.getPrimitiveWritableObject(arg0);
+ try {
+ evaluator = new SepDelimitedTextGISModelReader(model).constructModel();
+ } catch (IOException e) {
+ throw new HiveException(e.getMessage());
+ }
+
+ Object arg1 = arguments[1].get();
+ if (arg1 == null) {
+ throw new HiveException("arguments were null");
+ }
+ attributes = SmileExtUtils.resolveAttributes(attributesOI.getPrimitiveWritableObject(arg1).toString());
+
+ Object arg2 = arguments[2].get();
+ if (arg2 == null) {
+ throw new HiveException("array<double> features was null");
+ }
+ this.featuresProbe = parseFeatures(arg2, featuresProbe);
+
+ double[] obs = this.featuresProbe.toArray();
+
+ String[] names = new String[obs.length];
+ float[] values = new float[obs.length];
+ for (int i = 0; i < obs.length; i++){
+ if (attributes[i].type == AttributeType.NOMINAL){
+ names[i] = i + "_" + String.valueOf(obs[i]).toString();
+ values[i] = Double.valueOf(1.0).floatValue();
+ }else{
+ names[i] = String.valueOf(i);
+ values[i] = Double.valueOf(obs[i]).floatValue();
+ }
+ }
+
+ double[] ocs = evaluator.eval(names,values);
+ String klass = evaluator.getBestOutcome(ocs);
+
+ List<DoubleWritable> ocss = new ArrayList<DoubleWritable>();
+ for (int i = 0; i < ocs.length; i++){
+ ocss.add(new DoubleWritable(ocs[i]));
+ }
+
+ return new Object[]{new Text(klass), ocss};
+ }
+
+ @Nonnull
+ private Vector parseFeatures(@Nonnull final Object argObj, @Nonnull Vector probe) {
+ final int length = featureListOI.getListLength(argObj);
+ if (probe == null) {
+ probe = new DenseVector(length);
+ } else if (length != probe.size()) {
+ probe = new DenseVector(length);
+ }
+
+ for (int i = 0; i < length; i++) {
+ Object o = featureListOI.getListElement(argObj, i);
+ if (o == null) {
+ continue;
+ }
+ double v = PrimitiveObjectInspectorUtils.getDouble(o, featureElemOI);
+ probe.set(i, v);
+ }
+ return probe;
+ }
+
+ @Override
+ public void close() throws IOException {
+ this.modelOI = null;
+ this.featureElemOI = null;
+ this.featureListOI = null;
+ this.evaluator = null;
+ }
+
+ @Override
+ public String getDisplayString(String[] children) {
+ return "tree_predict(" + Arrays.toString(children) + ")";
--- End diff --
`tree_predict` is not correct. Better to be renamed to`matext_predict` or something.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall pull request #93: [WIP][HIVEMALL-126] Maximum Entropy Mod...
Posted by helenahm <gi...@git.apache.org>.
Github user helenahm commented on a diff in the pull request:
https://github.com/apache/incubator-hivemall/pull/93#discussion_r125551895
--- Diff: core/src/main/java/hivemall/smile/classification/MaxEntUDTF.java ---
@@ -0,0 +1,440 @@
+package hivemall.smile.classification;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.BitSet;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import javax.annotation.Nonnegative;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import javax.annotation.concurrent.GuardedBy;
+
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.Options;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.hive.ql.exec.MapredContext;
+import org.apache.hadoop.hive.ql.exec.MapredContextAccessor;
+import org.apache.hadoop.hive.ql.exec.UDFArgumentException;
+import org.apache.hadoop.hive.ql.exec.UDFArgumentTypeException;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.serde2.io.DoubleWritable;
+import org.apache.hadoop.hive.serde2.objectinspector.ListObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory;
+import org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;
+import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils;
+import org.apache.hadoop.io.IntWritable;
+import org.apache.hadoop.io.Text;
+import org.apache.hadoop.mapred.Reporter;
+import org.apache.hadoop.mapred.Counters.Counter;
+
+import hivemall.UDTFWithOptions;
+import hivemall.math.matrix.Matrix;
+import hivemall.math.matrix.MatrixUtils;
+import hivemall.math.matrix.builders.CSRMatrixBuilder;
+import hivemall.math.matrix.builders.MatrixBuilder;
+import hivemall.math.matrix.builders.RowMajorDenseMatrixBuilder;
+import hivemall.math.matrix.ints.ColumnMajorIntMatrix;
+import hivemall.math.matrix.ints.DoKIntMatrix;
+import hivemall.math.matrix.ints.IntMatrix;
+import hivemall.math.random.PRNG;
+import hivemall.math.random.RandomNumberGeneratorFactory;
+import hivemall.math.vector.Vector;
+import hivemall.math.vector.VectorProcedure;
+import hivemall.smile.classification.DecisionTree.SplitRule;
+import hivemall.smile.data.Attribute;
+import hivemall.smile.tools.MatrixEventStream;
+import hivemall.smile.tools.SepDelimitedTextGISModelWriter;
+import hivemall.smile.utils.SmileExtUtils;
+import hivemall.smile.utils.SmileTaskExecutor;
+import hivemall.utils.codec.Base91;
+import hivemall.utils.collections.lists.IntArrayList;
+import hivemall.utils.hadoop.HiveUtils;
+import hivemall.utils.hadoop.WritableUtils;
+import hivemall.utils.lang.Preconditions;
+import hivemall.utils.lang.Primitives;
+import hivemall.utils.lang.RandomUtils;
+
+import opennlp.maxent.GIS;
+import opennlp.maxent.io.GISModelWriter;
+import opennlp.model.AbstractModel;
+import opennlp.model.Event;
+import opennlp.model.EventStream;
+import opennlp.model.OnePassRealValueDataIndexer;
+
+@Description(
+ name = "train_maxent_classifier",
+ value = "_FUNC_(array<double> features, int label [, const boolean classification])"
+ + " - Returns a maximum entropy model per subset of data.")
+@UDFType(deterministic = true, stateful = false)
+public class MaxEntUDTF extends UDTFWithOptions{
+ private static final Log logger = LogFactory.getLog(MaxEntUDTF.class);
+
+ private ListObjectInspector featureListOI;
+ private PrimitiveObjectInspector featureElemOI;
+ private PrimitiveObjectInspector labelOI;
+
+ private MatrixBuilder matrixBuilder;
+ private IntArrayList labels;
+
+ private boolean _real;
+ private Attribute[] _attributes;
+ private static boolean _USE_SMOOTHING;
+ private double _SMOOTHING_OBSERVATION;
+
+ private int _numTrees = 1;
+
+ @Nullable
+ private Reporter _progressReporter;
+ @Nullable
+ private Counter _treeBuildTaskCounter;
+
+ @Override
+ protected Options getOptions() {
+ Options opts = new Options();
+ opts.addOption("real", "quantative_feature_presence_indication", true,
+ "true or false [default: true]");
+ opts.addOption("smoothing", "smoothimg", true, "Shall smoothing be performed [default: false]");
+ opts.addOption("constant", "smoothing_constant", true, "real number [default: 1.0]");
+ opts.addOption("attrs", "attribute_types", true, "Comma separated attribute types "
+ + "(Q for quantitative variable and C for categorical variable. e.g., [Q,C,Q,C])");
+ return opts;
+ }
+
+ @Override
+ protected CommandLine processOptions(ObjectInspector[] argOIs) throws UDFArgumentException {
+ boolean real = true;
+ boolean USE_SMOOTHING = false;
+ double SMOOTHING_OBSERVATION = 0.1;
+
+ Attribute[] attrs = null;
+
+ CommandLine cl = null;
+ if (argOIs.length >= 3) {
+ String rawArgs = HiveUtils.getConstString(argOIs[2]);
+ cl = parseOptions(rawArgs);
+
+ real = Primitives.parseBoolean(cl.getOptionValue("quantative_feature_presence_indication"), real);
+ attrs = SmileExtUtils.resolveAttributes(cl.getOptionValue("attribute_types"));
+ USE_SMOOTHING = Primitives.parseBoolean(cl.getOptionValue("smoothing"), USE_SMOOTHING);
+ SMOOTHING_OBSERVATION = Primitives.parseDouble(cl.getOptionValue("smoothing_constant"), SMOOTHING_OBSERVATION);
+ }
+
+ this._real = real;
+ this._attributes = attrs;
+ this._USE_SMOOTHING = USE_SMOOTHING;
+ this._SMOOTHING_OBSERVATION = SMOOTHING_OBSERVATION;
+
+ return cl;
+ }
+
+ @Override
+ public StructObjectInspector initialize(ObjectInspector[] argOIs) throws UDFArgumentException {
+ if (argOIs.length < 2 || argOIs.length > 3) {
+ throw new UDFArgumentException(
+ "_FUNC_ takes 2 ~ 3 arguments: array<double> features, int label [, const string options]: "
+ + argOIs.length);
+ }
+
+ ListObjectInspector listOI = HiveUtils.asListOI(argOIs[0]);
+ ObjectInspector elemOI = listOI.getListElementObjectInspector();
+ this.featureListOI = listOI;
+ if (HiveUtils.isNumberOI(elemOI)) {
+ this.featureElemOI = HiveUtils.asDoubleCompatibleOI(elemOI);
+ this.matrixBuilder = new CSRMatrixBuilder(8192);
+ } else {
+ throw new UDFArgumentException(
+ "_FUNC_ takes double[] for the first argument: " + listOI.getTypeName());
+ }
+ this.labelOI = HiveUtils.asIntCompatibleOI(argOIs[1]);
+
+ processOptions(argOIs);
+
+ this.labels = new IntArrayList(1024);
+
+ final ArrayList<String> fieldNames = new ArrayList<String>(6);
+ final ArrayList<ObjectInspector> fieldOIs = new ArrayList<ObjectInspector>(6);
+
+ fieldNames.add("model_id");
+ fieldOIs.add(PrimitiveObjectInspectorFactory.writableStringObjectInspector);
+ fieldNames.add("model_weight");
+ fieldOIs.add(PrimitiveObjectInspectorFactory.writableDoubleObjectInspector);
+ fieldNames.add("model");
+ fieldOIs.add(PrimitiveObjectInspectorFactory.writableStringObjectInspector);
+ fieldNames.add("attributes");
+ fieldOIs.add(PrimitiveObjectInspectorFactory.writableStringObjectInspector);
+ fieldNames.add("oob_errors");
+ fieldOIs.add(PrimitiveObjectInspectorFactory.writableIntObjectInspector);
+ fieldNames.add("oob_tests");
+ fieldOIs.add(PrimitiveObjectInspectorFactory.writableIntObjectInspector);
+
+ return ObjectInspectorFactory.getStandardStructObjectInspector(fieldNames, fieldOIs);
+ }
+
+ @Override
+ public void process(Object[] args) throws HiveException {
+ if (args[0] == null) {
+ throw new HiveException("array<double> features was null");
+ }
+ parseFeatures(args[0], matrixBuilder);
+ int label = PrimitiveObjectInspectorUtils.getInt(args[1], labelOI);
+ labels.add(label);
+ }
+
+ private void parseFeatures(@Nonnull final Object argObj, @Nonnull final MatrixBuilder builder) {
+ final int length = featureListOI.getListLength(argObj);
+ for (int i = 0; i < length; i++) {
+ Object o = featureListOI.getListElement(argObj, i);
+ if (o == null) {
+ continue;
+ }
+ double v = PrimitiveObjectInspectorUtils.getDouble(o, featureElemOI);
+ builder.nextColumn(i, v);
+ }
+ builder.nextRow();
+ }
+
+ @Override
+ public void close() throws HiveException {
+ this._progressReporter = getReporter();
+ this._treeBuildTaskCounter = (_progressReporter == null) ? null
+ : _progressReporter.getCounter("hivemall.smile.MaxEntClassifier$Counter",
+ "finishedGISTask");
+ reportProgress(_progressReporter);
+
+ if (!labels.isEmpty()) {
+ Matrix x = matrixBuilder.buildMatrix();
+ this.matrixBuilder = null;
+ int[] y = labels.toArray();
+ this.labels = null;
+
+ // run training
+ train(x, y);
+ }
+
+ // clean up
+ this.featureListOI = null;
+ this.featureElemOI = null;
+ this.labelOI = null;
+ }
+
+ private void checkOptions() throws HiveException {
+ if (_USE_SMOOTHING == false && _SMOOTHING_OBSERVATION != 0.1) {
+ throw new HiveException("Instructions received to avoid smoothing, but smoothing constant is set [" + _SMOOTHING_OBSERVATION + "]");
+ }
+ }
+
+ /**
+ * @param x features
+ * @param y label
+ * @param attrs attribute types
+ * @param numTrees The number of trees
+ * @param numVars The number of variables to pick up in each node.
+ * @param seed The seed number for Random Forest
+ */
+ private void train(@Nonnull Matrix x, @Nonnull final int[] y) throws HiveException {
+ final int numExamples = x.numRows();
+ if (numExamples != y.length) {
+ throw new HiveException(String.format("The sizes of X and Y don't match: %d != %d",
+ numExamples, y.length));
+ }
+ checkOptions();
+
+ int[] labels = SmileExtUtils.classLables(y);
+ Attribute[] attributes = SmileExtUtils.attributeTypes(_attributes, x);
+
+ if (logger.isInfoEnabled()) {
+ logger.info("real: " + _real + ", smoothing: " + this._USE_SMOOTHING + ", smoothing constant: "
+ + _SMOOTHING_OBSERVATION);
+ }
+
+ IntMatrix prediction = new DoKIntMatrix(numExamples, labels.length); // placeholder for out-of-bag prediction
+ AtomicInteger remainingTasks = new AtomicInteger(_numTrees);
+ List<TrainingTask> tasks = new ArrayList<TrainingTask>();
+ for (int i = 0; i < _numTrees; i++) {
--- End diff --
I left an ability to divide training set further into subsets for memory reasons. Perhaps at times it is faster to subdivide the data that mapper accepts into a number of chunks and train a model per each training set chunk. Implementation-wise it is same to Random Forests indeed. At the moment _numTrees is set to 1. If it works with 1 all the time then perhaps it is a good setting, otherwise it may be better to allow people to set it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall pull request #93: [WIP][HIVEMALL-126] Maximum Entropy Mod...
Posted by helenahm <gi...@git.apache.org>.
Github user helenahm commented on a diff in the pull request:
https://github.com/apache/incubator-hivemall/pull/93#discussion_r130533347
--- Diff: core/pom.xml ---
@@ -103,6 +103,12 @@
<version>${guava.version}</version>
<scope>provided</scope>
</dependency>
+ <dependency>
+ <groupId>opennlp</groupId>
+ <artifactId>maxent</artifactId>
+ <version>3.0.0</version>
--- End diff --
In general I totally agree. I think it would be good to perform the move to another version of maxent in a few steps.
1. The code I have re-used is that of GISTrainer. That is more or less updating the weights in a matrix where matrix is hivemall's matrix. Everything else is just following your class structure. I have checked that the resulting models are the same and I have also confirmed that the resulting model makes sense on my own data. So the resulting weights must be correct. Can we say that training is correct and accept the current version as the correct and functioning one?
2. After that there are a few options:
we could try to re-write the code in a way that will accept the newest version of opennlp maxent and all the following versions. I guess that would require changes in opennlp maxent too, but perhaps it is better than manual alteration of GISTrainer every time you update something, and both projects will benefit from such collaboration.
if not, perhaps for Hivemall as a project, we may consider re-writing iterative scaling from scratch to make it Hivemall efficient, perhaps using the tricks OpenNLP uses to make the code more efficient, and making sure that the resulting weights are comparable, but without aiming to being able to plug a new OpenNLP jar each time new version appears.
What do you think?
Regards,
Elena.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #93: Maximum Entropy Model
Posted by helenahm <gi...@git.apache.org>.
Github user helenahm commented on the issue:
https://github.com/apache/incubator-hivemall/pull/93
Just added a MaxEntMixtureWeightUDAF to aggregate weights of models obtained on each part of data.
create temporary function aggregate_classifiers as 'hivemall.smile.tools.MaxEntMixtureWeightUDAF';
tested it on EMR only:
add jar hivemall-core-0.4.2-rc.2-maxent-with-dependencies.jar;
add jar opennlp-maxent-3.0.0.jar;
source define-all.hive;
create temporary function train_maxent_classifier as 'hivemall.smile.classification.MaxEntUDTF';
create temporary function predict_maxent_classifier as 'hivemall.smile.tools.MaxEntPredictUDF';
create temporary function aggregate_classifiers as 'hivemall.smile.tools.MaxEntMixtureWeightUDAF';
select aggregate_classifiers(model) from tmodel5;
Where tmodel5 contains 5 lines of same model.
I will do more testing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall pull request #93: [WIP][HIVEMALL-126] Maximum Entropy Mod...
Posted by myui <gi...@git.apache.org>.
Github user myui commented on a diff in the pull request:
https://github.com/apache/incubator-hivemall/pull/93#discussion_r125545255
--- Diff: core/src/main/java/hivemall/smile/classification/MaxEntUDTF.java ---
@@ -0,0 +1,440 @@
+package hivemall.smile.classification;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.BitSet;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import javax.annotation.Nonnegative;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import javax.annotation.concurrent.GuardedBy;
+
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.Options;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.hive.ql.exec.MapredContext;
+import org.apache.hadoop.hive.ql.exec.MapredContextAccessor;
+import org.apache.hadoop.hive.ql.exec.UDFArgumentException;
+import org.apache.hadoop.hive.ql.exec.UDFArgumentTypeException;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.serde2.io.DoubleWritable;
+import org.apache.hadoop.hive.serde2.objectinspector.ListObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory;
+import org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;
+import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils;
+import org.apache.hadoop.io.IntWritable;
+import org.apache.hadoop.io.Text;
+import org.apache.hadoop.mapred.Reporter;
+import org.apache.hadoop.mapred.Counters.Counter;
+
+import hivemall.UDTFWithOptions;
+import hivemall.math.matrix.Matrix;
+import hivemall.math.matrix.MatrixUtils;
+import hivemall.math.matrix.builders.CSRMatrixBuilder;
+import hivemall.math.matrix.builders.MatrixBuilder;
+import hivemall.math.matrix.builders.RowMajorDenseMatrixBuilder;
+import hivemall.math.matrix.ints.ColumnMajorIntMatrix;
+import hivemall.math.matrix.ints.DoKIntMatrix;
+import hivemall.math.matrix.ints.IntMatrix;
+import hivemall.math.random.PRNG;
+import hivemall.math.random.RandomNumberGeneratorFactory;
+import hivemall.math.vector.Vector;
+import hivemall.math.vector.VectorProcedure;
+import hivemall.smile.classification.DecisionTree.SplitRule;
+import hivemall.smile.data.Attribute;
+import hivemall.smile.tools.MatrixEventStream;
+import hivemall.smile.tools.SepDelimitedTextGISModelWriter;
+import hivemall.smile.utils.SmileExtUtils;
+import hivemall.smile.utils.SmileTaskExecutor;
+import hivemall.utils.codec.Base91;
+import hivemall.utils.collections.lists.IntArrayList;
+import hivemall.utils.hadoop.HiveUtils;
+import hivemall.utils.hadoop.WritableUtils;
+import hivemall.utils.lang.Preconditions;
+import hivemall.utils.lang.Primitives;
+import hivemall.utils.lang.RandomUtils;
+
+import opennlp.maxent.GIS;
+import opennlp.maxent.io.GISModelWriter;
+import opennlp.model.AbstractModel;
+import opennlp.model.Event;
+import opennlp.model.EventStream;
+import opennlp.model.OnePassRealValueDataIndexer;
+
+@Description(
+ name = "train_maxent_classifier",
+ value = "_FUNC_(array<double> features, int label [, const boolean classification])"
+ + " - Returns a maximum entropy model per subset of data.")
+@UDFType(deterministic = true, stateful = false)
+public class MaxEntUDTF extends UDTFWithOptions{
+ private static final Log logger = LogFactory.getLog(MaxEntUDTF.class);
+
+ private ListObjectInspector featureListOI;
+ private PrimitiveObjectInspector featureElemOI;
+ private PrimitiveObjectInspector labelOI;
+
+ private MatrixBuilder matrixBuilder;
+ private IntArrayList labels;
+
+ private boolean _real;
+ private Attribute[] _attributes;
+ private static boolean _USE_SMOOTHING;
+ private double _SMOOTHING_OBSERVATION;
+
+ private int _numTrees = 1;
+
+ @Nullable
+ private Reporter _progressReporter;
+ @Nullable
+ private Counter _treeBuildTaskCounter;
+
+ @Override
+ protected Options getOptions() {
+ Options opts = new Options();
+ opts.addOption("real", "quantative_feature_presence_indication", true,
+ "true or false [default: true]");
+ opts.addOption("smoothing", "smoothimg", true, "Shall smoothing be performed [default: false]");
+ opts.addOption("constant", "smoothing_constant", true, "real number [default: 1.0]");
+ opts.addOption("attrs", "attribute_types", true, "Comma separated attribute types "
+ + "(Q for quantitative variable and C for categorical variable. e.g., [Q,C,Q,C])");
+ return opts;
+ }
+
+ @Override
+ protected CommandLine processOptions(ObjectInspector[] argOIs) throws UDFArgumentException {
+ boolean real = true;
+ boolean USE_SMOOTHING = false;
+ double SMOOTHING_OBSERVATION = 0.1;
+
+ Attribute[] attrs = null;
+
+ CommandLine cl = null;
+ if (argOIs.length >= 3) {
+ String rawArgs = HiveUtils.getConstString(argOIs[2]);
+ cl = parseOptions(rawArgs);
+
+ real = Primitives.parseBoolean(cl.getOptionValue("quantative_feature_presence_indication"), real);
+ attrs = SmileExtUtils.resolveAttributes(cl.getOptionValue("attribute_types"));
+ USE_SMOOTHING = Primitives.parseBoolean(cl.getOptionValue("smoothing"), USE_SMOOTHING);
+ SMOOTHING_OBSERVATION = Primitives.parseDouble(cl.getOptionValue("smoothing_constant"), SMOOTHING_OBSERVATION);
+ }
+
+ this._real = real;
+ this._attributes = attrs;
+ this._USE_SMOOTHING = USE_SMOOTHING;
+ this._SMOOTHING_OBSERVATION = SMOOTHING_OBSERVATION;
+
+ return cl;
+ }
+
+ @Override
+ public StructObjectInspector initialize(ObjectInspector[] argOIs) throws UDFArgumentException {
+ if (argOIs.length < 2 || argOIs.length > 3) {
+ throw new UDFArgumentException(
+ "_FUNC_ takes 2 ~ 3 arguments: array<double> features, int label [, const string options]: "
+ + argOIs.length);
+ }
+
+ ListObjectInspector listOI = HiveUtils.asListOI(argOIs[0]);
+ ObjectInspector elemOI = listOI.getListElementObjectInspector();
+ this.featureListOI = listOI;
+ if (HiveUtils.isNumberOI(elemOI)) {
+ this.featureElemOI = HiveUtils.asDoubleCompatibleOI(elemOI);
+ this.matrixBuilder = new CSRMatrixBuilder(8192);
+ } else {
+ throw new UDFArgumentException(
+ "_FUNC_ takes double[] for the first argument: " + listOI.getTypeName());
+ }
+ this.labelOI = HiveUtils.asIntCompatibleOI(argOIs[1]);
+
+ processOptions(argOIs);
+
+ this.labels = new IntArrayList(1024);
+
+ final ArrayList<String> fieldNames = new ArrayList<String>(6);
+ final ArrayList<ObjectInspector> fieldOIs = new ArrayList<ObjectInspector>(6);
+
+ fieldNames.add("model_id");
+ fieldOIs.add(PrimitiveObjectInspectorFactory.writableStringObjectInspector);
+ fieldNames.add("model_weight");
+ fieldOIs.add(PrimitiveObjectInspectorFactory.writableDoubleObjectInspector);
+ fieldNames.add("model");
+ fieldOIs.add(PrimitiveObjectInspectorFactory.writableStringObjectInspector);
+ fieldNames.add("attributes");
+ fieldOIs.add(PrimitiveObjectInspectorFactory.writableStringObjectInspector);
+ fieldNames.add("oob_errors");
+ fieldOIs.add(PrimitiveObjectInspectorFactory.writableIntObjectInspector);
+ fieldNames.add("oob_tests");
+ fieldOIs.add(PrimitiveObjectInspectorFactory.writableIntObjectInspector);
+
+ return ObjectInspectorFactory.getStandardStructObjectInspector(fieldNames, fieldOIs);
+ }
+
+ @Override
+ public void process(Object[] args) throws HiveException {
+ if (args[0] == null) {
+ throw new HiveException("array<double> features was null");
+ }
+ parseFeatures(args[0], matrixBuilder);
+ int label = PrimitiveObjectInspectorUtils.getInt(args[1], labelOI);
+ labels.add(label);
+ }
+
+ private void parseFeatures(@Nonnull final Object argObj, @Nonnull final MatrixBuilder builder) {
+ final int length = featureListOI.getListLength(argObj);
+ for (int i = 0; i < length; i++) {
+ Object o = featureListOI.getListElement(argObj, i);
+ if (o == null) {
+ continue;
+ }
+ double v = PrimitiveObjectInspectorUtils.getDouble(o, featureElemOI);
+ builder.nextColumn(i, v);
+ }
+ builder.nextRow();
+ }
+
+ @Override
+ public void close() throws HiveException {
+ this._progressReporter = getReporter();
+ this._treeBuildTaskCounter = (_progressReporter == null) ? null
+ : _progressReporter.getCounter("hivemall.smile.MaxEntClassifier$Counter",
+ "finishedGISTask");
+ reportProgress(_progressReporter);
+
+ if (!labels.isEmpty()) {
+ Matrix x = matrixBuilder.buildMatrix();
+ this.matrixBuilder = null;
+ int[] y = labels.toArray();
+ this.labels = null;
+
+ // run training
+ train(x, y);
+ }
+
+ // clean up
+ this.featureListOI = null;
+ this.featureElemOI = null;
+ this.labelOI = null;
+ }
+
+ private void checkOptions() throws HiveException {
+ if (_USE_SMOOTHING == false && _SMOOTHING_OBSERVATION != 0.1) {
+ throw new HiveException("Instructions received to avoid smoothing, but smoothing constant is set [" + _SMOOTHING_OBSERVATION + "]");
+ }
+ }
+
+ /**
+ * @param x features
+ * @param y label
+ * @param attrs attribute types
+ * @param numTrees The number of trees
+ * @param numVars The number of variables to pick up in each node.
+ * @param seed The seed number for Random Forest
+ */
+ private void train(@Nonnull Matrix x, @Nonnull final int[] y) throws HiveException {
+ final int numExamples = x.numRows();
+ if (numExamples != y.length) {
+ throw new HiveException(String.format("The sizes of X and Y don't match: %d != %d",
+ numExamples, y.length));
+ }
+ checkOptions();
+
+ int[] labels = SmileExtUtils.classLables(y);
+ Attribute[] attributes = SmileExtUtils.attributeTypes(_attributes, x);
+
+ if (logger.isInfoEnabled()) {
+ logger.info("real: " + _real + ", smoothing: " + this._USE_SMOOTHING + ", smoothing constant: "
+ + _SMOOTHING_OBSERVATION);
+ }
+
+ IntMatrix prediction = new DoKIntMatrix(numExamples, labels.length); // placeholder for out-of-bag prediction
+ AtomicInteger remainingTasks = new AtomicInteger(_numTrees);
+ List<TrainingTask> tasks = new ArrayList<TrainingTask>();
+ for (int i = 0; i < _numTrees; i++) {
+ tasks.add(new TrainingTask(this, i, attributes, x, y, prediction, remainingTasks));
+ }
+
+ MapredContext mapredContext = MapredContextAccessor.get();
+ final SmileTaskExecutor executor = new SmileTaskExecutor(mapredContext);
+ try {
+ executor.run(tasks);
+ } catch (Exception ex) {
+ throw new HiveException(ex);
+ } finally {
+ executor.shotdown();
+ }
+ }
+
+ /**
+ * Synchronized because {@link #forward(Object)} should be called from a single thread.
+ *
+ * @param accuracy
+ */
+ synchronized void forward(final int taskId, @Nonnull final Text model,
+ @Nonnull Attribute[] attributes,
+ @Nonnegative final double accuracy, final int[] y,
+ @Nonnull final IntMatrix prediction, final boolean lastTask) throws HiveException {
+ int oobErrors = 0;
+ int oobTests = 0;
+ if (lastTask) {
+ // out-of-bag error estimate
+ for (int i = 0; i < y.length; i++) {
+ final int pred = MatrixUtils.whichMax(prediction, i);
+ if (pred != -1 && prediction.get(i, pred) > 0) {
+ oobTests++;
+ if (pred != y[i]) {
+ oobErrors++;
+ }
+ }
+ }
+ }
+
+ String attributesString = SmileExtUtils.resolveAttributes(attributes);
+
+ final Object[] forwardObjs = new Object[6];
+ String modelId = RandomUtils.getUUID();
+ forwardObjs[0] = new Text(modelId);
+ forwardObjs[1] = new DoubleWritable(accuracy);
+ forwardObjs[2] = model;
+ forwardObjs[3] = new Text(SmileExtUtils.resolveAttributes(attributes));
+ forwardObjs[4] = new IntWritable(oobErrors);
+ forwardObjs[5] = new IntWritable(oobTests);
+ forward(forwardObjs);
+
+ reportProgress(_progressReporter);
+ incrCounter(_treeBuildTaskCounter, 1);
+
+ logger.info("Forwarded " + taskId + "-th DecisionTree out of " + _numTrees);
+ }
+
+ /**
+ * Trains a regression tree.
+ */
+ private static final class TrainingTask implements Callable<Integer> {
+
+ /**
+ * Training instances.
+ */
+ @Nonnull
+ private final Matrix _x;
+ /**
+ * Training sample labels.
+ */
+ @Nonnull
+ private final int[] _y;
+
+ /**
+ * Attribute properties.
+ */
+ @Nonnull
+ private final Attribute[] _attributes;
+
+ /**
+ * The out-of-bag predictions.
+ */
+ @Nonnull
+ @GuardedBy("_udtf")
+ private final IntMatrix _prediction;
+
+ @Nonnull
+ private final MaxEntUDTF _udtf;
+ private final int _taskId;
+
+ @Nonnull
+ private final AtomicInteger _remainingTasks;
+
+ TrainingTask(@Nonnull MaxEntUDTF udtf, int taskId,
+ @Nonnull Attribute[] attributes, @Nonnull Matrix x, @Nonnull int[] y,
+ @Nonnull IntMatrix prediction, @Nonnull AtomicInteger remainingTasks) {
+ this._udtf = udtf;
+ this._taskId = taskId;
+ this._attributes = attributes;
+ this._x = x;
+ this._y = y;
+ this._prediction = prediction;
+ this._remainingTasks = remainingTasks;
+ }
+
+ @Override
+ public Integer call() throws HiveException {
+ final int N = _x.numRows();
+
+ EventStream es = new MatrixEventStream(_x, _y, _attributes);
+ AbstractModel model;
+ try {
+ model = GIS.trainModel(1000, new OnePassRealValueDataIndexer(es,0), _USE_SMOOTHING);
--- End diff --
`1000` ( `-iterations` ?) should be parameterized.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #93: [WIP][HIVEMALL-126] Maximum Entropy Model usin...
Posted by kottmann <gi...@git.apache.org>.
Github user kottmann commented on the issue:
https://github.com/apache/incubator-hivemall/pull/93
@helenahm as far as I know the training data is stored once in memory, and then for each thread a copy of the parameters is stored.
Yeah, so if you have a lot of training data then running out of memory is one symptom you run into, but that is not the actual problem of this implementation. The actual cause is that it won't scale beyond one machine.
Bottom line if you want to use GIS training with lots of data don't use this implementation, the training requires a certain amount of CPU time and it increases with the amount of training data. In case you manage to make this run with much more data the time it will take to run will be uncomfortably high.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #93: [WIP][HIVEMALL-126] Maximum Entropy Model usin...
Posted by myui <gi...@git.apache.org>.
Github user myui commented on the issue:
https://github.com/apache/incubator-hivemall/pull/93
- [ ] CI is failing due to missing License header. You can use `bin/format_header.sh` to update headers.
- [ ] Could you apply source code formatter, i.e., `mvn formatter:format`, for your commit?
- [ ] package should be `hivemall.opennlp` , not `hivemall.smile` .
- [ ] This function should be `hivemall-nlp` module (refer KuromojiUDF)
- [ ] DDLs should be described in `resources/ddl/define-additional.hive`. (grep `tokenize_ja` for a reference)
- [ ] Documentation is required in `docs/gitbook` . `npm install gitbook-cli; gitbook install; gitbook serve` on `docs/gitbook` .
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #93: [WIP][HIVEMALL-126] Maximum Entropy Model usin...
Posted by helenahm <gi...@git.apache.org>.
Github user helenahm commented on the issue:
https://github.com/apache/incubator-hivemall/pull/93
It will include some work.
Let me explain.
You were right when you have said that OpenNLP implementation is poor memory-wise. Indeed, they store data in [][] and few times. Using their code directly causes Java Heap Space, GC errors, etc. (Tested that on my 97 mil of data rows. Newer version of code has same problems.) And you were right about the wonderful CSRMatrix. And DoKMatrix too. They allow to store more data. Thus, more or less, I have changed all the [][] (related to input data) to CSRMatrix and [][] holding weights to DoKMatrix.
To explain that more, it is best to look at source code for the GISTrainer. In fact all 3 of them, old maxent, new maxent, and Hivemall's BigGISTrainer. The links are below.
Newer GISTrainer:
https://github.com/apache/opennlp/blob/master/opennlp-tools/src/main/java/opennlp/tools/ml/maxent/GISTrainer.java
Older (3.0.0) GISTrainer:
https://sourceforge.net/projects/maxent/files/ - whole achive
GISTrainer attached:
[GISTrainer.txt](https://github.com/apache/incubator-hivemall/files/1192806/GISTrainer.txt)
Hivemall GISTrainer:
https://github.com/helenahm/incubator-hivemall/blob/master/core/src/main/java/hivemall/opennlp/tools/BigGISTrainer.java
Notice how trainModel of BigGISTrainer gets MatrixForTraining (https://github.com/helenahm/incubator-hivemall/blob/master/core/src/main/java/hivemall/opennlp/tools/MatrixForTraining.java), that contains references to Matrix, and outcomes. This is CSRMatrix.
And row data is collected from the CSRMatrix in MatrixForTraining instead of the double[][].
when
ComparableEvent ev = x.createComparableEvent(ti, di.getPredicateIndex(), di.getOMap());
(they use this convenience Event thing to work with a row of data. Instead of storing a List of Events in memory the modified code also builds an event when needed.)
and results are stored in
Matrix predCount = new DoKMatrix(numPreds, numOutcomes); instead of [][] again.
GISTrainer did not change very dramatically. If 3.0.0 training is reliable enough, I would, of course, consider the existing version as 1.0, and did all the effort to adapt GISTrainer later on. It makes sense to do that, I totally agree. And perhaps it makes sense to continue after that to understanding training process in greater details and perhaps write a newer comparable trainer that will be independent from OpenNLP.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall pull request #93: [WIP][HIVEMALL-126] Maximum Entropy Mod...
Posted by myui <gi...@git.apache.org>.
Github user myui commented on a diff in the pull request:
https://github.com/apache/incubator-hivemall/pull/93#discussion_r125544508
--- Diff: core/src/main/java/hivemall/smile/classification/MaxEntUDTF.java ---
@@ -0,0 +1,440 @@
+package hivemall.smile.classification;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.BitSet;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import javax.annotation.Nonnegative;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import javax.annotation.concurrent.GuardedBy;
+
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.Options;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.hive.ql.exec.MapredContext;
+import org.apache.hadoop.hive.ql.exec.MapredContextAccessor;
+import org.apache.hadoop.hive.ql.exec.UDFArgumentException;
+import org.apache.hadoop.hive.ql.exec.UDFArgumentTypeException;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.serde2.io.DoubleWritable;
+import org.apache.hadoop.hive.serde2.objectinspector.ListObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory;
+import org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;
+import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils;
+import org.apache.hadoop.io.IntWritable;
+import org.apache.hadoop.io.Text;
+import org.apache.hadoop.mapred.Reporter;
+import org.apache.hadoop.mapred.Counters.Counter;
+
+import hivemall.UDTFWithOptions;
+import hivemall.math.matrix.Matrix;
+import hivemall.math.matrix.MatrixUtils;
+import hivemall.math.matrix.builders.CSRMatrixBuilder;
+import hivemall.math.matrix.builders.MatrixBuilder;
+import hivemall.math.matrix.builders.RowMajorDenseMatrixBuilder;
+import hivemall.math.matrix.ints.ColumnMajorIntMatrix;
+import hivemall.math.matrix.ints.DoKIntMatrix;
+import hivemall.math.matrix.ints.IntMatrix;
+import hivemall.math.random.PRNG;
+import hivemall.math.random.RandomNumberGeneratorFactory;
+import hivemall.math.vector.Vector;
+import hivemall.math.vector.VectorProcedure;
+import hivemall.smile.classification.DecisionTree.SplitRule;
+import hivemall.smile.data.Attribute;
+import hivemall.smile.tools.MatrixEventStream;
+import hivemall.smile.tools.SepDelimitedTextGISModelWriter;
+import hivemall.smile.utils.SmileExtUtils;
+import hivemall.smile.utils.SmileTaskExecutor;
+import hivemall.utils.codec.Base91;
+import hivemall.utils.collections.lists.IntArrayList;
+import hivemall.utils.hadoop.HiveUtils;
+import hivemall.utils.hadoop.WritableUtils;
+import hivemall.utils.lang.Preconditions;
+import hivemall.utils.lang.Primitives;
+import hivemall.utils.lang.RandomUtils;
+
+import opennlp.maxent.GIS;
+import opennlp.maxent.io.GISModelWriter;
+import opennlp.model.AbstractModel;
+import opennlp.model.Event;
+import opennlp.model.EventStream;
+import opennlp.model.OnePassRealValueDataIndexer;
+
+@Description(
+ name = "train_maxent_classifier",
+ value = "_FUNC_(array<double> features, int label [, const boolean classification])"
+ + " - Returns a maximum entropy model per subset of data.")
+@UDFType(deterministic = true, stateful = false)
+public class MaxEntUDTF extends UDTFWithOptions{
+ private static final Log logger = LogFactory.getLog(MaxEntUDTF.class);
+
+ private ListObjectInspector featureListOI;
+ private PrimitiveObjectInspector featureElemOI;
+ private PrimitiveObjectInspector labelOI;
+
+ private MatrixBuilder matrixBuilder;
+ private IntArrayList labels;
+
+ private boolean _real;
+ private Attribute[] _attributes;
+ private static boolean _USE_SMOOTHING;
+ private double _SMOOTHING_OBSERVATION;
+
+ private int _numTrees = 1;
+
+ @Nullable
+ private Reporter _progressReporter;
+ @Nullable
+ private Counter _treeBuildTaskCounter;
+
+ @Override
+ protected Options getOptions() {
+ Options opts = new Options();
+ opts.addOption("real", "quantative_feature_presence_indication", true,
+ "true or false [default: true]");
+ opts.addOption("smoothing", "smoothimg", true, "Shall smoothing be performed [default: false]");
+ opts.addOption("constant", "smoothing_constant", true, "real number [default: 1.0]");
+ opts.addOption("attrs", "attribute_types", true, "Comma separated attribute types "
+ + "(Q for quantitative variable and C for categorical variable. e.g., [Q,C,Q,C])");
+ return opts;
+ }
+
+ @Override
+ protected CommandLine processOptions(ObjectInspector[] argOIs) throws UDFArgumentException {
+ boolean real = true;
+ boolean USE_SMOOTHING = false;
+ double SMOOTHING_OBSERVATION = 0.1;
+
+ Attribute[] attrs = null;
+
+ CommandLine cl = null;
+ if (argOIs.length >= 3) {
+ String rawArgs = HiveUtils.getConstString(argOIs[2]);
+ cl = parseOptions(rawArgs);
+
+ real = Primitives.parseBoolean(cl.getOptionValue("quantative_feature_presence_indication"), real);
+ attrs = SmileExtUtils.resolveAttributes(cl.getOptionValue("attribute_types"));
+ USE_SMOOTHING = Primitives.parseBoolean(cl.getOptionValue("smoothing"), USE_SMOOTHING);
+ SMOOTHING_OBSERVATION = Primitives.parseDouble(cl.getOptionValue("smoothing_constant"), SMOOTHING_OBSERVATION);
+ }
+
+ this._real = real;
+ this._attributes = attrs;
+ this._USE_SMOOTHING = USE_SMOOTHING;
+ this._SMOOTHING_OBSERVATION = SMOOTHING_OBSERVATION;
+
+ return cl;
+ }
+
+ @Override
+ public StructObjectInspector initialize(ObjectInspector[] argOIs) throws UDFArgumentException {
+ if (argOIs.length < 2 || argOIs.length > 3) {
+ throw new UDFArgumentException(
+ "_FUNC_ takes 2 ~ 3 arguments: array<double> features, int label [, const string options]: "
+ + argOIs.length);
+ }
+
+ ListObjectInspector listOI = HiveUtils.asListOI(argOIs[0]);
+ ObjectInspector elemOI = listOI.getListElementObjectInspector();
+ this.featureListOI = listOI;
+ if (HiveUtils.isNumberOI(elemOI)) {
+ this.featureElemOI = HiveUtils.asDoubleCompatibleOI(elemOI);
+ this.matrixBuilder = new CSRMatrixBuilder(8192);
+ } else {
+ throw new UDFArgumentException(
+ "_FUNC_ takes double[] for the first argument: " + listOI.getTypeName());
+ }
+ this.labelOI = HiveUtils.asIntCompatibleOI(argOIs[1]);
+
+ processOptions(argOIs);
+
+ this.labels = new IntArrayList(1024);
+
+ final ArrayList<String> fieldNames = new ArrayList<String>(6);
+ final ArrayList<ObjectInspector> fieldOIs = new ArrayList<ObjectInspector>(6);
+
+ fieldNames.add("model_id");
+ fieldOIs.add(PrimitiveObjectInspectorFactory.writableStringObjectInspector);
+ fieldNames.add("model_weight");
+ fieldOIs.add(PrimitiveObjectInspectorFactory.writableDoubleObjectInspector);
+ fieldNames.add("model");
+ fieldOIs.add(PrimitiveObjectInspectorFactory.writableStringObjectInspector);
+ fieldNames.add("attributes");
+ fieldOIs.add(PrimitiveObjectInspectorFactory.writableStringObjectInspector);
+ fieldNames.add("oob_errors");
+ fieldOIs.add(PrimitiveObjectInspectorFactory.writableIntObjectInspector);
+ fieldNames.add("oob_tests");
+ fieldOIs.add(PrimitiveObjectInspectorFactory.writableIntObjectInspector);
+
+ return ObjectInspectorFactory.getStandardStructObjectInspector(fieldNames, fieldOIs);
+ }
+
+ @Override
+ public void process(Object[] args) throws HiveException {
+ if (args[0] == null) {
+ throw new HiveException("array<double> features was null");
+ }
+ parseFeatures(args[0], matrixBuilder);
+ int label = PrimitiveObjectInspectorUtils.getInt(args[1], labelOI);
+ labels.add(label);
+ }
+
+ private void parseFeatures(@Nonnull final Object argObj, @Nonnull final MatrixBuilder builder) {
+ final int length = featureListOI.getListLength(argObj);
+ for (int i = 0; i < length; i++) {
+ Object o = featureListOI.getListElement(argObj, i);
+ if (o == null) {
+ continue;
+ }
+ double v = PrimitiveObjectInspectorUtils.getDouble(o, featureElemOI);
+ builder.nextColumn(i, v);
+ }
+ builder.nextRow();
+ }
+
+ @Override
+ public void close() throws HiveException {
+ this._progressReporter = getReporter();
+ this._treeBuildTaskCounter = (_progressReporter == null) ? null
+ : _progressReporter.getCounter("hivemall.smile.MaxEntClassifier$Counter",
+ "finishedGISTask");
+ reportProgress(_progressReporter);
+
+ if (!labels.isEmpty()) {
+ Matrix x = matrixBuilder.buildMatrix();
+ this.matrixBuilder = null;
+ int[] y = labels.toArray();
+ this.labels = null;
+
+ // run training
+ train(x, y);
+ }
+
+ // clean up
+ this.featureListOI = null;
+ this.featureElemOI = null;
+ this.labelOI = null;
+ }
+
+ private void checkOptions() throws HiveException {
+ if (_USE_SMOOTHING == false && _SMOOTHING_OBSERVATION != 0.1) {
+ throw new HiveException("Instructions received to avoid smoothing, but smoothing constant is set [" + _SMOOTHING_OBSERVATION + "]");
+ }
+ }
+
+ /**
+ * @param x features
+ * @param y label
+ * @param attrs attribute types
+ * @param numTrees The number of trees
+ * @param numVars The number of variables to pick up in each node.
+ * @param seed The seed number for Random Forest
+ */
+ private void train(@Nonnull Matrix x, @Nonnull final int[] y) throws HiveException {
+ final int numExamples = x.numRows();
+ if (numExamples != y.length) {
+ throw new HiveException(String.format("The sizes of X and Y don't match: %d != %d",
+ numExamples, y.length));
+ }
+ checkOptions();
+
+ int[] labels = SmileExtUtils.classLables(y);
+ Attribute[] attributes = SmileExtUtils.attributeTypes(_attributes, x);
+
+ if (logger.isInfoEnabled()) {
+ logger.info("real: " + _real + ", smoothing: " + this._USE_SMOOTHING + ", smoothing constant: "
+ + _SMOOTHING_OBSERVATION);
+ }
+
+ IntMatrix prediction = new DoKIntMatrix(numExamples, labels.length); // placeholder for out-of-bag prediction
+ AtomicInteger remainingTasks = new AtomicInteger(_numTrees);
+ List<TrainingTask> tasks = new ArrayList<TrainingTask>();
+ for (int i = 0; i < _numTrees; i++) {
+ tasks.add(new TrainingTask(this, i, attributes, x, y, prediction, remainingTasks));
+ }
+
+ MapredContext mapredContext = MapredContextAccessor.get();
+ final SmileTaskExecutor executor = new SmileTaskExecutor(mapredContext);
+ try {
+ executor.run(tasks);
+ } catch (Exception ex) {
+ throw new HiveException(ex);
+ } finally {
+ executor.shotdown();
+ }
+ }
+
+ /**
+ * Synchronized because {@link #forward(Object)} should be called from a single thread.
+ *
+ * @param accuracy
+ */
+ synchronized void forward(final int taskId, @Nonnull final Text model,
+ @Nonnull Attribute[] attributes,
+ @Nonnegative final double accuracy, final int[] y,
+ @Nonnull final IntMatrix prediction, final boolean lastTask) throws HiveException {
+ int oobErrors = 0;
+ int oobTests = 0;
+ if (lastTask) {
+ // out-of-bag error estimate
--- End diff --
Out-Of-Bag testing is a method for RandomForest. Should not use training examples used in the input of training.
https://www.quora.com/What-is-the-out-of-bag-error-in-Random-Forests
OOB testing is not needed for MaxEnt classifier in my understanding.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall pull request #93: [WIP][HIVEMALL-126] Maximum Entropy Mod...
Posted by myui <gi...@git.apache.org>.
Github user myui commented on a diff in the pull request:
https://github.com/apache/incubator-hivemall/pull/93#discussion_r125544365
--- Diff: core/src/main/java/hivemall/smile/classification/MaxEntUDTF.java ---
@@ -0,0 +1,440 @@
+package hivemall.smile.classification;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.BitSet;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import javax.annotation.Nonnegative;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import javax.annotation.concurrent.GuardedBy;
+
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.Options;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.hive.ql.exec.MapredContext;
+import org.apache.hadoop.hive.ql.exec.MapredContextAccessor;
+import org.apache.hadoop.hive.ql.exec.UDFArgumentException;
+import org.apache.hadoop.hive.ql.exec.UDFArgumentTypeException;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.serde2.io.DoubleWritable;
+import org.apache.hadoop.hive.serde2.objectinspector.ListObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory;
+import org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;
+import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils;
+import org.apache.hadoop.io.IntWritable;
+import org.apache.hadoop.io.Text;
+import org.apache.hadoop.mapred.Reporter;
+import org.apache.hadoop.mapred.Counters.Counter;
+
+import hivemall.UDTFWithOptions;
+import hivemall.math.matrix.Matrix;
+import hivemall.math.matrix.MatrixUtils;
+import hivemall.math.matrix.builders.CSRMatrixBuilder;
+import hivemall.math.matrix.builders.MatrixBuilder;
+import hivemall.math.matrix.builders.RowMajorDenseMatrixBuilder;
+import hivemall.math.matrix.ints.ColumnMajorIntMatrix;
+import hivemall.math.matrix.ints.DoKIntMatrix;
+import hivemall.math.matrix.ints.IntMatrix;
+import hivemall.math.random.PRNG;
+import hivemall.math.random.RandomNumberGeneratorFactory;
+import hivemall.math.vector.Vector;
+import hivemall.math.vector.VectorProcedure;
+import hivemall.smile.classification.DecisionTree.SplitRule;
+import hivemall.smile.data.Attribute;
+import hivemall.smile.tools.MatrixEventStream;
+import hivemall.smile.tools.SepDelimitedTextGISModelWriter;
+import hivemall.smile.utils.SmileExtUtils;
+import hivemall.smile.utils.SmileTaskExecutor;
+import hivemall.utils.codec.Base91;
+import hivemall.utils.collections.lists.IntArrayList;
+import hivemall.utils.hadoop.HiveUtils;
+import hivemall.utils.hadoop.WritableUtils;
+import hivemall.utils.lang.Preconditions;
+import hivemall.utils.lang.Primitives;
+import hivemall.utils.lang.RandomUtils;
+
+import opennlp.maxent.GIS;
+import opennlp.maxent.io.GISModelWriter;
+import opennlp.model.AbstractModel;
+import opennlp.model.Event;
+import opennlp.model.EventStream;
+import opennlp.model.OnePassRealValueDataIndexer;
+
+@Description(
+ name = "train_maxent_classifier",
+ value = "_FUNC_(array<double> features, int label [, const boolean classification])"
+ + " - Returns a maximum entropy model per subset of data.")
+@UDFType(deterministic = true, stateful = false)
+public class MaxEntUDTF extends UDTFWithOptions{
+ private static final Log logger = LogFactory.getLog(MaxEntUDTF.class);
+
+ private ListObjectInspector featureListOI;
+ private PrimitiveObjectInspector featureElemOI;
+ private PrimitiveObjectInspector labelOI;
+
+ private MatrixBuilder matrixBuilder;
+ private IntArrayList labels;
+
+ private boolean _real;
+ private Attribute[] _attributes;
+ private static boolean _USE_SMOOTHING;
+ private double _SMOOTHING_OBSERVATION;
+
+ private int _numTrees = 1;
+
+ @Nullable
+ private Reporter _progressReporter;
+ @Nullable
+ private Counter _treeBuildTaskCounter;
+
+ @Override
+ protected Options getOptions() {
+ Options opts = new Options();
+ opts.addOption("real", "quantative_feature_presence_indication", true,
+ "true or false [default: true]");
+ opts.addOption("smoothing", "smoothimg", true, "Shall smoothing be performed [default: false]");
+ opts.addOption("constant", "smoothing_constant", true, "real number [default: 1.0]");
+ opts.addOption("attrs", "attribute_types", true, "Comma separated attribute types "
+ + "(Q for quantitative variable and C for categorical variable. e.g., [Q,C,Q,C])");
+ return opts;
+ }
+
+ @Override
+ protected CommandLine processOptions(ObjectInspector[] argOIs) throws UDFArgumentException {
+ boolean real = true;
+ boolean USE_SMOOTHING = false;
+ double SMOOTHING_OBSERVATION = 0.1;
+
+ Attribute[] attrs = null;
+
+ CommandLine cl = null;
+ if (argOIs.length >= 3) {
+ String rawArgs = HiveUtils.getConstString(argOIs[2]);
+ cl = parseOptions(rawArgs);
+
+ real = Primitives.parseBoolean(cl.getOptionValue("quantative_feature_presence_indication"), real);
+ attrs = SmileExtUtils.resolveAttributes(cl.getOptionValue("attribute_types"));
+ USE_SMOOTHING = Primitives.parseBoolean(cl.getOptionValue("smoothing"), USE_SMOOTHING);
+ SMOOTHING_OBSERVATION = Primitives.parseDouble(cl.getOptionValue("smoothing_constant"), SMOOTHING_OBSERVATION);
+ }
+
+ this._real = real;
+ this._attributes = attrs;
+ this._USE_SMOOTHING = USE_SMOOTHING;
+ this._SMOOTHING_OBSERVATION = SMOOTHING_OBSERVATION;
+
+ return cl;
+ }
+
+ @Override
+ public StructObjectInspector initialize(ObjectInspector[] argOIs) throws UDFArgumentException {
+ if (argOIs.length < 2 || argOIs.length > 3) {
+ throw new UDFArgumentException(
+ "_FUNC_ takes 2 ~ 3 arguments: array<double> features, int label [, const string options]: "
+ + argOIs.length);
+ }
+
+ ListObjectInspector listOI = HiveUtils.asListOI(argOIs[0]);
+ ObjectInspector elemOI = listOI.getListElementObjectInspector();
+ this.featureListOI = listOI;
+ if (HiveUtils.isNumberOI(elemOI)) {
+ this.featureElemOI = HiveUtils.asDoubleCompatibleOI(elemOI);
+ this.matrixBuilder = new CSRMatrixBuilder(8192);
+ } else {
+ throw new UDFArgumentException(
+ "_FUNC_ takes double[] for the first argument: " + listOI.getTypeName());
+ }
+ this.labelOI = HiveUtils.asIntCompatibleOI(argOIs[1]);
+
+ processOptions(argOIs);
+
+ this.labels = new IntArrayList(1024);
+
+ final ArrayList<String> fieldNames = new ArrayList<String>(6);
+ final ArrayList<ObjectInspector> fieldOIs = new ArrayList<ObjectInspector>(6);
+
+ fieldNames.add("model_id");
+ fieldOIs.add(PrimitiveObjectInspectorFactory.writableStringObjectInspector);
+ fieldNames.add("model_weight");
+ fieldOIs.add(PrimitiveObjectInspectorFactory.writableDoubleObjectInspector);
+ fieldNames.add("model");
+ fieldOIs.add(PrimitiveObjectInspectorFactory.writableStringObjectInspector);
+ fieldNames.add("attributes");
+ fieldOIs.add(PrimitiveObjectInspectorFactory.writableStringObjectInspector);
+ fieldNames.add("oob_errors");
+ fieldOIs.add(PrimitiveObjectInspectorFactory.writableIntObjectInspector);
+ fieldNames.add("oob_tests");
+ fieldOIs.add(PrimitiveObjectInspectorFactory.writableIntObjectInspector);
+
+ return ObjectInspectorFactory.getStandardStructObjectInspector(fieldNames, fieldOIs);
+ }
+
+ @Override
+ public void process(Object[] args) throws HiveException {
+ if (args[0] == null) {
+ throw new HiveException("array<double> features was null");
+ }
+ parseFeatures(args[0], matrixBuilder);
+ int label = PrimitiveObjectInspectorUtils.getInt(args[1], labelOI);
+ labels.add(label);
+ }
+
+ private void parseFeatures(@Nonnull final Object argObj, @Nonnull final MatrixBuilder builder) {
+ final int length = featureListOI.getListLength(argObj);
+ for (int i = 0; i < length; i++) {
+ Object o = featureListOI.getListElement(argObj, i);
+ if (o == null) {
+ continue;
+ }
+ double v = PrimitiveObjectInspectorUtils.getDouble(o, featureElemOI);
+ builder.nextColumn(i, v);
+ }
+ builder.nextRow();
+ }
+
+ @Override
+ public void close() throws HiveException {
+ this._progressReporter = getReporter();
+ this._treeBuildTaskCounter = (_progressReporter == null) ? null
+ : _progressReporter.getCounter("hivemall.smile.MaxEntClassifier$Counter",
+ "finishedGISTask");
+ reportProgress(_progressReporter);
+
+ if (!labels.isEmpty()) {
+ Matrix x = matrixBuilder.buildMatrix();
+ this.matrixBuilder = null;
+ int[] y = labels.toArray();
+ this.labels = null;
+
+ // run training
+ train(x, y);
+ }
+
+ // clean up
+ this.featureListOI = null;
+ this.featureElemOI = null;
+ this.labelOI = null;
+ }
+
+ private void checkOptions() throws HiveException {
+ if (_USE_SMOOTHING == false && _SMOOTHING_OBSERVATION != 0.1) {
+ throw new HiveException("Instructions received to avoid smoothing, but smoothing constant is set [" + _SMOOTHING_OBSERVATION + "]");
+ }
+ }
+
+ /**
+ * @param x features
+ * @param y label
+ * @param attrs attribute types
+ * @param numTrees The number of trees
+ * @param numVars The number of variables to pick up in each node.
+ * @param seed The seed number for Random Forest
+ */
+ private void train(@Nonnull Matrix x, @Nonnull final int[] y) throws HiveException {
+ final int numExamples = x.numRows();
+ if (numExamples != y.length) {
+ throw new HiveException(String.format("The sizes of X and Y don't match: %d != %d",
+ numExamples, y.length));
+ }
+ checkOptions();
+
+ int[] labels = SmileExtUtils.classLables(y);
+ Attribute[] attributes = SmileExtUtils.attributeTypes(_attributes, x);
+
+ if (logger.isInfoEnabled()) {
+ logger.info("real: " + _real + ", smoothing: " + this._USE_SMOOTHING + ", smoothing constant: "
+ + _SMOOTHING_OBSERVATION);
+ }
+
+ IntMatrix prediction = new DoKIntMatrix(numExamples, labels.length); // placeholder for out-of-bag prediction
+ AtomicInteger remainingTasks = new AtomicInteger(_numTrees);
+ List<TrainingTask> tasks = new ArrayList<TrainingTask>();
+ for (int i = 0; i < _numTrees; i++) {
--- End diff --
Why are you using `_numTrees` and multiple threads for training? It's not tree model.
It's specialized for RandomForest (Constructs DecisionTrees in parallel) training and I think a single thread execution is enough for Max Entropy classifier.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #93: [WIP][HIVEMALL-126] Maximum Entropy Model usin...
Posted by kottmann <gi...@git.apache.org>.
Github user kottmann commented on the issue:
https://github.com/apache/incubator-hivemall/pull/93
@myui that was done for the 1.6.0 release, and in maxent 3.0.3 it was modified to run in multiple threads.
You probably need to take a similar approach as we took for multi-threaded training e.g. split the amount of work done per iteration and scale it out to multiple machines, merge the parameters, and repeat for the next iteration.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall pull request #93: [WIP][HIVEMALL-126] Maximum Entropy Mod...
Posted by helenahm <gi...@git.apache.org>.
Github user helenahm commented on a diff in the pull request:
https://github.com/apache/incubator-hivemall/pull/93#discussion_r125551962
--- Diff: core/src/main/java/hivemall/smile/classification/MaxEntUDTF.java ---
@@ -0,0 +1,440 @@
+package hivemall.smile.classification;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.BitSet;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import javax.annotation.Nonnegative;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import javax.annotation.concurrent.GuardedBy;
+
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.Options;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.hive.ql.exec.MapredContext;
+import org.apache.hadoop.hive.ql.exec.MapredContextAccessor;
+import org.apache.hadoop.hive.ql.exec.UDFArgumentException;
+import org.apache.hadoop.hive.ql.exec.UDFArgumentTypeException;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.serde2.io.DoubleWritable;
+import org.apache.hadoop.hive.serde2.objectinspector.ListObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory;
+import org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;
+import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils;
+import org.apache.hadoop.io.IntWritable;
+import org.apache.hadoop.io.Text;
+import org.apache.hadoop.mapred.Reporter;
+import org.apache.hadoop.mapred.Counters.Counter;
+
+import hivemall.UDTFWithOptions;
+import hivemall.math.matrix.Matrix;
+import hivemall.math.matrix.MatrixUtils;
+import hivemall.math.matrix.builders.CSRMatrixBuilder;
+import hivemall.math.matrix.builders.MatrixBuilder;
+import hivemall.math.matrix.builders.RowMajorDenseMatrixBuilder;
+import hivemall.math.matrix.ints.ColumnMajorIntMatrix;
+import hivemall.math.matrix.ints.DoKIntMatrix;
+import hivemall.math.matrix.ints.IntMatrix;
+import hivemall.math.random.PRNG;
+import hivemall.math.random.RandomNumberGeneratorFactory;
+import hivemall.math.vector.Vector;
+import hivemall.math.vector.VectorProcedure;
+import hivemall.smile.classification.DecisionTree.SplitRule;
+import hivemall.smile.data.Attribute;
+import hivemall.smile.tools.MatrixEventStream;
+import hivemall.smile.tools.SepDelimitedTextGISModelWriter;
+import hivemall.smile.utils.SmileExtUtils;
+import hivemall.smile.utils.SmileTaskExecutor;
+import hivemall.utils.codec.Base91;
+import hivemall.utils.collections.lists.IntArrayList;
+import hivemall.utils.hadoop.HiveUtils;
+import hivemall.utils.hadoop.WritableUtils;
+import hivemall.utils.lang.Preconditions;
+import hivemall.utils.lang.Primitives;
+import hivemall.utils.lang.RandomUtils;
+
+import opennlp.maxent.GIS;
+import opennlp.maxent.io.GISModelWriter;
+import opennlp.model.AbstractModel;
+import opennlp.model.Event;
+import opennlp.model.EventStream;
+import opennlp.model.OnePassRealValueDataIndexer;
+
+@Description(
+ name = "train_maxent_classifier",
+ value = "_FUNC_(array<double> features, int label [, const boolean classification])"
+ + " - Returns a maximum entropy model per subset of data.")
+@UDFType(deterministic = true, stateful = false)
+public class MaxEntUDTF extends UDTFWithOptions{
+ private static final Log logger = LogFactory.getLog(MaxEntUDTF.class);
+
+ private ListObjectInspector featureListOI;
+ private PrimitiveObjectInspector featureElemOI;
+ private PrimitiveObjectInspector labelOI;
+
+ private MatrixBuilder matrixBuilder;
+ private IntArrayList labels;
+
+ private boolean _real;
+ private Attribute[] _attributes;
+ private static boolean _USE_SMOOTHING;
+ private double _SMOOTHING_OBSERVATION;
+
+ private int _numTrees = 1;
+
+ @Nullable
+ private Reporter _progressReporter;
+ @Nullable
+ private Counter _treeBuildTaskCounter;
+
+ @Override
+ protected Options getOptions() {
+ Options opts = new Options();
+ opts.addOption("real", "quantative_feature_presence_indication", true,
+ "true or false [default: true]");
+ opts.addOption("smoothing", "smoothimg", true, "Shall smoothing be performed [default: false]");
+ opts.addOption("constant", "smoothing_constant", true, "real number [default: 1.0]");
+ opts.addOption("attrs", "attribute_types", true, "Comma separated attribute types "
+ + "(Q for quantitative variable and C for categorical variable. e.g., [Q,C,Q,C])");
+ return opts;
+ }
+
+ @Override
+ protected CommandLine processOptions(ObjectInspector[] argOIs) throws UDFArgumentException {
+ boolean real = true;
+ boolean USE_SMOOTHING = false;
+ double SMOOTHING_OBSERVATION = 0.1;
+
+ Attribute[] attrs = null;
+
+ CommandLine cl = null;
+ if (argOIs.length >= 3) {
+ String rawArgs = HiveUtils.getConstString(argOIs[2]);
+ cl = parseOptions(rawArgs);
+
+ real = Primitives.parseBoolean(cl.getOptionValue("quantative_feature_presence_indication"), real);
+ attrs = SmileExtUtils.resolveAttributes(cl.getOptionValue("attribute_types"));
+ USE_SMOOTHING = Primitives.parseBoolean(cl.getOptionValue("smoothing"), USE_SMOOTHING);
+ SMOOTHING_OBSERVATION = Primitives.parseDouble(cl.getOptionValue("smoothing_constant"), SMOOTHING_OBSERVATION);
+ }
+
+ this._real = real;
+ this._attributes = attrs;
+ this._USE_SMOOTHING = USE_SMOOTHING;
+ this._SMOOTHING_OBSERVATION = SMOOTHING_OBSERVATION;
+
+ return cl;
+ }
+
+ @Override
+ public StructObjectInspector initialize(ObjectInspector[] argOIs) throws UDFArgumentException {
+ if (argOIs.length < 2 || argOIs.length > 3) {
+ throw new UDFArgumentException(
+ "_FUNC_ takes 2 ~ 3 arguments: array<double> features, int label [, const string options]: "
+ + argOIs.length);
+ }
+
+ ListObjectInspector listOI = HiveUtils.asListOI(argOIs[0]);
+ ObjectInspector elemOI = listOI.getListElementObjectInspector();
+ this.featureListOI = listOI;
+ if (HiveUtils.isNumberOI(elemOI)) {
+ this.featureElemOI = HiveUtils.asDoubleCompatibleOI(elemOI);
+ this.matrixBuilder = new CSRMatrixBuilder(8192);
+ } else {
+ throw new UDFArgumentException(
+ "_FUNC_ takes double[] for the first argument: " + listOI.getTypeName());
+ }
+ this.labelOI = HiveUtils.asIntCompatibleOI(argOIs[1]);
+
+ processOptions(argOIs);
+
+ this.labels = new IntArrayList(1024);
+
+ final ArrayList<String> fieldNames = new ArrayList<String>(6);
+ final ArrayList<ObjectInspector> fieldOIs = new ArrayList<ObjectInspector>(6);
+
+ fieldNames.add("model_id");
+ fieldOIs.add(PrimitiveObjectInspectorFactory.writableStringObjectInspector);
+ fieldNames.add("model_weight");
+ fieldOIs.add(PrimitiveObjectInspectorFactory.writableDoubleObjectInspector);
+ fieldNames.add("model");
+ fieldOIs.add(PrimitiveObjectInspectorFactory.writableStringObjectInspector);
+ fieldNames.add("attributes");
+ fieldOIs.add(PrimitiveObjectInspectorFactory.writableStringObjectInspector);
+ fieldNames.add("oob_errors");
+ fieldOIs.add(PrimitiveObjectInspectorFactory.writableIntObjectInspector);
+ fieldNames.add("oob_tests");
+ fieldOIs.add(PrimitiveObjectInspectorFactory.writableIntObjectInspector);
+
+ return ObjectInspectorFactory.getStandardStructObjectInspector(fieldNames, fieldOIs);
+ }
+
+ @Override
+ public void process(Object[] args) throws HiveException {
+ if (args[0] == null) {
+ throw new HiveException("array<double> features was null");
+ }
+ parseFeatures(args[0], matrixBuilder);
+ int label = PrimitiveObjectInspectorUtils.getInt(args[1], labelOI);
+ labels.add(label);
+ }
+
+ private void parseFeatures(@Nonnull final Object argObj, @Nonnull final MatrixBuilder builder) {
+ final int length = featureListOI.getListLength(argObj);
+ for (int i = 0; i < length; i++) {
+ Object o = featureListOI.getListElement(argObj, i);
+ if (o == null) {
+ continue;
+ }
+ double v = PrimitiveObjectInspectorUtils.getDouble(o, featureElemOI);
+ builder.nextColumn(i, v);
+ }
+ builder.nextRow();
+ }
+
+ @Override
+ public void close() throws HiveException {
+ this._progressReporter = getReporter();
+ this._treeBuildTaskCounter = (_progressReporter == null) ? null
+ : _progressReporter.getCounter("hivemall.smile.MaxEntClassifier$Counter",
+ "finishedGISTask");
+ reportProgress(_progressReporter);
+
+ if (!labels.isEmpty()) {
+ Matrix x = matrixBuilder.buildMatrix();
+ this.matrixBuilder = null;
+ int[] y = labels.toArray();
+ this.labels = null;
+
+ // run training
+ train(x, y);
+ }
+
+ // clean up
+ this.featureListOI = null;
+ this.featureElemOI = null;
+ this.labelOI = null;
+ }
+
+ private void checkOptions() throws HiveException {
+ if (_USE_SMOOTHING == false && _SMOOTHING_OBSERVATION != 0.1) {
+ throw new HiveException("Instructions received to avoid smoothing, but smoothing constant is set [" + _SMOOTHING_OBSERVATION + "]");
+ }
+ }
+
+ /**
+ * @param x features
+ * @param y label
+ * @param attrs attribute types
+ * @param numTrees The number of trees
+ * @param numVars The number of variables to pick up in each node.
+ * @param seed The seed number for Random Forest
+ */
+ private void train(@Nonnull Matrix x, @Nonnull final int[] y) throws HiveException {
+ final int numExamples = x.numRows();
+ if (numExamples != y.length) {
+ throw new HiveException(String.format("The sizes of X and Y don't match: %d != %d",
+ numExamples, y.length));
+ }
+ checkOptions();
+
+ int[] labels = SmileExtUtils.classLables(y);
+ Attribute[] attributes = SmileExtUtils.attributeTypes(_attributes, x);
+
+ if (logger.isInfoEnabled()) {
+ logger.info("real: " + _real + ", smoothing: " + this._USE_SMOOTHING + ", smoothing constant: "
+ + _SMOOTHING_OBSERVATION);
+ }
+
+ IntMatrix prediction = new DoKIntMatrix(numExamples, labels.length); // placeholder for out-of-bag prediction
+ AtomicInteger remainingTasks = new AtomicInteger(_numTrees);
+ List<TrainingTask> tasks = new ArrayList<TrainingTask>();
+ for (int i = 0; i < _numTrees; i++) {
+ tasks.add(new TrainingTask(this, i, attributes, x, y, prediction, remainingTasks));
+ }
+
+ MapredContext mapredContext = MapredContextAccessor.get();
+ final SmileTaskExecutor executor = new SmileTaskExecutor(mapredContext);
+ try {
+ executor.run(tasks);
+ } catch (Exception ex) {
+ throw new HiveException(ex);
+ } finally {
+ executor.shotdown();
+ }
+ }
+
+ /**
+ * Synchronized because {@link #forward(Object)} should be called from a single thread.
+ *
+ * @param accuracy
+ */
+ synchronized void forward(final int taskId, @Nonnull final Text model,
+ @Nonnull Attribute[] attributes,
+ @Nonnegative final double accuracy, final int[] y,
+ @Nonnull final IntMatrix prediction, final boolean lastTask) throws HiveException {
+ int oobErrors = 0;
+ int oobTests = 0;
+ if (lastTask) {
+ // out-of-bag error estimate
--- End diff --
Yes. It is not that important for this model. You are right.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall pull request #93: Maximum Entropy Model
Posted by myui <gi...@git.apache.org>.
Github user myui commented on a diff in the pull request:
https://github.com/apache/incubator-hivemall/pull/93#discussion_r125185211
--- Diff: core/src/main/java/hivemall/smile/tools/MatrixEventStream.java ---
@@ -0,0 +1,81 @@
+package hivemall.smile.tools;
+
+import java.util.Iterator;
+import java.util.List;
+
+import hivemall.math.matrix.Matrix;
+import hivemall.smile.data.Attribute;
+import hivemall.smile.data.Attribute.AttributeType;
+import opennlp.model.AbstractEventStream;
+import opennlp.model.Event;
+
+public class MatrixEventStream extends AbstractEventStream {
+ Matrix ds;
+ int[] y;
+ Attribute[] attrs;
+ int rowNum;
+ Event next;
+
+ public MatrixEventStream (Matrix ds, int[] y, Attribute[] attrs) {
+ this.ds = ds;
+ this.y = y;
+ this.attrs = attrs;
+ rowNum = 0;
+
+ if (rowNum < ds.numRows()){
+ next = createEvent(ds.getRow(rowNum), y[rowNum]);
+ };
+ }
+
+ /**
+ * Returns the next Event object held in this EventStream. Each call to nextEvent advances the EventStream.
+ *
+ * @return the Event object which is next in this EventStream
+ */
+ public Event next () {
+ while (next == null && (rowNum < ds.numRows())){
+ next = createEvent(ds.getRow(rowNum), y[rowNum]);
+ }
+
+ Event current = next;
+ if (rowNum < ds.numRows()) {
+ next = createEvent(ds.getRow(rowNum), y[rowNum]);
+ }
+ else {
+ next = null;
+ }
+ return current;
+ }
+
+ /**
+ * Test whether there are any Events remaining in this EventStream.
+ *
+ * @return true if this EventStream has more Events
+ */
+ public boolean hasNext () {
+ while (next == null && rowNum < ds.numRows()){
+ next = createEvent(ds.getRow(rowNum), y[rowNum]);
+ }
+ return next != null;
+ }
+
+ private Event createEvent(double[] obs, int y) {
+ rowNum++;
+ if (obs == null){
+ return null;
+ }else{
+ String[] names = new String[obs.length];
+ float[] values = new float[obs.length];
+ for (int i = 0; i < obs.length; i++){
+ if (attrs[i].type == AttributeType.NOMINAL){
+ names[i] = i + "_" + String.valueOf(obs[i]).toString();
+ values[i] = Double.valueOf(1.0).floatValue();
--- End diff --
`values[i] = 1.f;`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall pull request #93: [WIP][HIVEMALL-126] Maximum Entropy Mod...
Posted by helenahm <gi...@git.apache.org>.
Github user helenahm commented on a diff in the pull request:
https://github.com/apache/incubator-hivemall/pull/93#discussion_r125552049
--- Diff: core/src/main/java/hivemall/smile/classification/MaxEntUDTF.java ---
@@ -0,0 +1,440 @@
+package hivemall.smile.classification;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.BitSet;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import javax.annotation.Nonnegative;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import javax.annotation.concurrent.GuardedBy;
+
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.Options;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.hive.ql.exec.MapredContext;
+import org.apache.hadoop.hive.ql.exec.MapredContextAccessor;
+import org.apache.hadoop.hive.ql.exec.UDFArgumentException;
+import org.apache.hadoop.hive.ql.exec.UDFArgumentTypeException;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.serde2.io.DoubleWritable;
+import org.apache.hadoop.hive.serde2.objectinspector.ListObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory;
+import org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;
+import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils;
+import org.apache.hadoop.io.IntWritable;
+import org.apache.hadoop.io.Text;
+import org.apache.hadoop.mapred.Reporter;
+import org.apache.hadoop.mapred.Counters.Counter;
+
+import hivemall.UDTFWithOptions;
+import hivemall.math.matrix.Matrix;
+import hivemall.math.matrix.MatrixUtils;
+import hivemall.math.matrix.builders.CSRMatrixBuilder;
+import hivemall.math.matrix.builders.MatrixBuilder;
+import hivemall.math.matrix.builders.RowMajorDenseMatrixBuilder;
+import hivemall.math.matrix.ints.ColumnMajorIntMatrix;
+import hivemall.math.matrix.ints.DoKIntMatrix;
+import hivemall.math.matrix.ints.IntMatrix;
+import hivemall.math.random.PRNG;
+import hivemall.math.random.RandomNumberGeneratorFactory;
+import hivemall.math.vector.Vector;
+import hivemall.math.vector.VectorProcedure;
+import hivemall.smile.classification.DecisionTree.SplitRule;
+import hivemall.smile.data.Attribute;
+import hivemall.smile.tools.MatrixEventStream;
+import hivemall.smile.tools.SepDelimitedTextGISModelWriter;
+import hivemall.smile.utils.SmileExtUtils;
+import hivemall.smile.utils.SmileTaskExecutor;
+import hivemall.utils.codec.Base91;
+import hivemall.utils.collections.lists.IntArrayList;
+import hivemall.utils.hadoop.HiveUtils;
+import hivemall.utils.hadoop.WritableUtils;
+import hivemall.utils.lang.Preconditions;
+import hivemall.utils.lang.Primitives;
+import hivemall.utils.lang.RandomUtils;
+
+import opennlp.maxent.GIS;
+import opennlp.maxent.io.GISModelWriter;
+import opennlp.model.AbstractModel;
+import opennlp.model.Event;
+import opennlp.model.EventStream;
+import opennlp.model.OnePassRealValueDataIndexer;
+
+@Description(
+ name = "train_maxent_classifier",
+ value = "_FUNC_(array<double> features, int label [, const boolean classification])"
+ + " - Returns a maximum entropy model per subset of data.")
+@UDFType(deterministic = true, stateful = false)
+public class MaxEntUDTF extends UDTFWithOptions{
+ private static final Log logger = LogFactory.getLog(MaxEntUDTF.class);
+
+ private ListObjectInspector featureListOI;
+ private PrimitiveObjectInspector featureElemOI;
+ private PrimitiveObjectInspector labelOI;
+
+ private MatrixBuilder matrixBuilder;
+ private IntArrayList labels;
+
+ private boolean _real;
+ private Attribute[] _attributes;
+ private static boolean _USE_SMOOTHING;
+ private double _SMOOTHING_OBSERVATION;
+
+ private int _numTrees = 1;
+
+ @Nullable
+ private Reporter _progressReporter;
+ @Nullable
+ private Counter _treeBuildTaskCounter;
+
+ @Override
+ protected Options getOptions() {
+ Options opts = new Options();
+ opts.addOption("real", "quantative_feature_presence_indication", true,
+ "true or false [default: true]");
+ opts.addOption("smoothing", "smoothimg", true, "Shall smoothing be performed [default: false]");
+ opts.addOption("constant", "smoothing_constant", true, "real number [default: 1.0]");
+ opts.addOption("attrs", "attribute_types", true, "Comma separated attribute types "
+ + "(Q for quantitative variable and C for categorical variable. e.g., [Q,C,Q,C])");
+ return opts;
+ }
+
+ @Override
+ protected CommandLine processOptions(ObjectInspector[] argOIs) throws UDFArgumentException {
+ boolean real = true;
+ boolean USE_SMOOTHING = false;
+ double SMOOTHING_OBSERVATION = 0.1;
+
+ Attribute[] attrs = null;
+
+ CommandLine cl = null;
+ if (argOIs.length >= 3) {
+ String rawArgs = HiveUtils.getConstString(argOIs[2]);
+ cl = parseOptions(rawArgs);
+
+ real = Primitives.parseBoolean(cl.getOptionValue("quantative_feature_presence_indication"), real);
+ attrs = SmileExtUtils.resolveAttributes(cl.getOptionValue("attribute_types"));
+ USE_SMOOTHING = Primitives.parseBoolean(cl.getOptionValue("smoothing"), USE_SMOOTHING);
+ SMOOTHING_OBSERVATION = Primitives.parseDouble(cl.getOptionValue("smoothing_constant"), SMOOTHING_OBSERVATION);
+ }
+
+ this._real = real;
+ this._attributes = attrs;
+ this._USE_SMOOTHING = USE_SMOOTHING;
+ this._SMOOTHING_OBSERVATION = SMOOTHING_OBSERVATION;
+
+ return cl;
+ }
+
+ @Override
+ public StructObjectInspector initialize(ObjectInspector[] argOIs) throws UDFArgumentException {
+ if (argOIs.length < 2 || argOIs.length > 3) {
+ throw new UDFArgumentException(
+ "_FUNC_ takes 2 ~ 3 arguments: array<double> features, int label [, const string options]: "
+ + argOIs.length);
+ }
+
+ ListObjectInspector listOI = HiveUtils.asListOI(argOIs[0]);
+ ObjectInspector elemOI = listOI.getListElementObjectInspector();
+ this.featureListOI = listOI;
+ if (HiveUtils.isNumberOI(elemOI)) {
+ this.featureElemOI = HiveUtils.asDoubleCompatibleOI(elemOI);
+ this.matrixBuilder = new CSRMatrixBuilder(8192);
+ } else {
+ throw new UDFArgumentException(
+ "_FUNC_ takes double[] for the first argument: " + listOI.getTypeName());
+ }
+ this.labelOI = HiveUtils.asIntCompatibleOI(argOIs[1]);
+
+ processOptions(argOIs);
+
+ this.labels = new IntArrayList(1024);
+
+ final ArrayList<String> fieldNames = new ArrayList<String>(6);
+ final ArrayList<ObjectInspector> fieldOIs = new ArrayList<ObjectInspector>(6);
+
+ fieldNames.add("model_id");
+ fieldOIs.add(PrimitiveObjectInspectorFactory.writableStringObjectInspector);
+ fieldNames.add("model_weight");
+ fieldOIs.add(PrimitiveObjectInspectorFactory.writableDoubleObjectInspector);
+ fieldNames.add("model");
+ fieldOIs.add(PrimitiveObjectInspectorFactory.writableStringObjectInspector);
+ fieldNames.add("attributes");
+ fieldOIs.add(PrimitiveObjectInspectorFactory.writableStringObjectInspector);
+ fieldNames.add("oob_errors");
+ fieldOIs.add(PrimitiveObjectInspectorFactory.writableIntObjectInspector);
+ fieldNames.add("oob_tests");
+ fieldOIs.add(PrimitiveObjectInspectorFactory.writableIntObjectInspector);
+
+ return ObjectInspectorFactory.getStandardStructObjectInspector(fieldNames, fieldOIs);
+ }
+
+ @Override
+ public void process(Object[] args) throws HiveException {
+ if (args[0] == null) {
+ throw new HiveException("array<double> features was null");
+ }
+ parseFeatures(args[0], matrixBuilder);
+ int label = PrimitiveObjectInspectorUtils.getInt(args[1], labelOI);
+ labels.add(label);
+ }
+
+ private void parseFeatures(@Nonnull final Object argObj, @Nonnull final MatrixBuilder builder) {
+ final int length = featureListOI.getListLength(argObj);
+ for (int i = 0; i < length; i++) {
+ Object o = featureListOI.getListElement(argObj, i);
+ if (o == null) {
+ continue;
+ }
+ double v = PrimitiveObjectInspectorUtils.getDouble(o, featureElemOI);
+ builder.nextColumn(i, v);
+ }
+ builder.nextRow();
+ }
+
+ @Override
+ public void close() throws HiveException {
+ this._progressReporter = getReporter();
+ this._treeBuildTaskCounter = (_progressReporter == null) ? null
+ : _progressReporter.getCounter("hivemall.smile.MaxEntClassifier$Counter",
+ "finishedGISTask");
+ reportProgress(_progressReporter);
+
+ if (!labels.isEmpty()) {
+ Matrix x = matrixBuilder.buildMatrix();
+ this.matrixBuilder = null;
+ int[] y = labels.toArray();
+ this.labels = null;
+
+ // run training
+ train(x, y);
+ }
+
+ // clean up
+ this.featureListOI = null;
+ this.featureElemOI = null;
+ this.labelOI = null;
+ }
+
+ private void checkOptions() throws HiveException {
+ if (_USE_SMOOTHING == false && _SMOOTHING_OBSERVATION != 0.1) {
+ throw new HiveException("Instructions received to avoid smoothing, but smoothing constant is set [" + _SMOOTHING_OBSERVATION + "]");
+ }
+ }
+
+ /**
+ * @param x features
+ * @param y label
+ * @param attrs attribute types
+ * @param numTrees The number of trees
+ * @param numVars The number of variables to pick up in each node.
+ * @param seed The seed number for Random Forest
+ */
+ private void train(@Nonnull Matrix x, @Nonnull final int[] y) throws HiveException {
+ final int numExamples = x.numRows();
+ if (numExamples != y.length) {
+ throw new HiveException(String.format("The sizes of X and Y don't match: %d != %d",
+ numExamples, y.length));
+ }
+ checkOptions();
+
+ int[] labels = SmileExtUtils.classLables(y);
+ Attribute[] attributes = SmileExtUtils.attributeTypes(_attributes, x);
+
+ if (logger.isInfoEnabled()) {
+ logger.info("real: " + _real + ", smoothing: " + this._USE_SMOOTHING + ", smoothing constant: "
+ + _SMOOTHING_OBSERVATION);
+ }
+
+ IntMatrix prediction = new DoKIntMatrix(numExamples, labels.length); // placeholder for out-of-bag prediction
+ AtomicInteger remainingTasks = new AtomicInteger(_numTrees);
+ List<TrainingTask> tasks = new ArrayList<TrainingTask>();
+ for (int i = 0; i < _numTrees; i++) {
+ tasks.add(new TrainingTask(this, i, attributes, x, y, prediction, remainingTasks));
+ }
+
+ MapredContext mapredContext = MapredContextAccessor.get();
+ final SmileTaskExecutor executor = new SmileTaskExecutor(mapredContext);
+ try {
+ executor.run(tasks);
+ } catch (Exception ex) {
+ throw new HiveException(ex);
+ } finally {
+ executor.shotdown();
+ }
+ }
+
+ /**
+ * Synchronized because {@link #forward(Object)} should be called from a single thread.
+ *
+ * @param accuracy
+ */
+ synchronized void forward(final int taskId, @Nonnull final Text model,
+ @Nonnull Attribute[] attributes,
+ @Nonnegative final double accuracy, final int[] y,
+ @Nonnull final IntMatrix prediction, final boolean lastTask) throws HiveException {
+ int oobErrors = 0;
+ int oobTests = 0;
+ if (lastTask) {
+ // out-of-bag error estimate
+ for (int i = 0; i < y.length; i++) {
+ final int pred = MatrixUtils.whichMax(prediction, i);
+ if (pred != -1 && prediction.get(i, pred) > 0) {
+ oobTests++;
+ if (pred != y[i]) {
+ oobErrors++;
+ }
+ }
+ }
+ }
+
+ String attributesString = SmileExtUtils.resolveAttributes(attributes);
+
+ final Object[] forwardObjs = new Object[6];
+ String modelId = RandomUtils.getUUID();
+ forwardObjs[0] = new Text(modelId);
+ forwardObjs[1] = new DoubleWritable(accuracy);
+ forwardObjs[2] = model;
+ forwardObjs[3] = new Text(SmileExtUtils.resolveAttributes(attributes));
+ forwardObjs[4] = new IntWritable(oobErrors);
+ forwardObjs[5] = new IntWritable(oobTests);
+ forward(forwardObjs);
+
+ reportProgress(_progressReporter);
+ incrCounter(_treeBuildTaskCounter, 1);
+
+ logger.info("Forwarded " + taskId + "-th DecisionTree out of " + _numTrees);
+ }
+
+ /**
+ * Trains a regression tree.
+ */
+ private static final class TrainingTask implements Callable<Integer> {
+
+ /**
+ * Training instances.
+ */
+ @Nonnull
+ private final Matrix _x;
+ /**
+ * Training sample labels.
+ */
+ @Nonnull
+ private final int[] _y;
+
+ /**
+ * Attribute properties.
+ */
+ @Nonnull
+ private final Attribute[] _attributes;
+
+ /**
+ * The out-of-bag predictions.
+ */
+ @Nonnull
+ @GuardedBy("_udtf")
+ private final IntMatrix _prediction;
+
+ @Nonnull
+ private final MaxEntUDTF _udtf;
+ private final int _taskId;
+
+ @Nonnull
+ private final AtomicInteger _remainingTasks;
+
+ TrainingTask(@Nonnull MaxEntUDTF udtf, int taskId,
+ @Nonnull Attribute[] attributes, @Nonnull Matrix x, @Nonnull int[] y,
+ @Nonnull IntMatrix prediction, @Nonnull AtomicInteger remainingTasks) {
+ this._udtf = udtf;
+ this._taskId = taskId;
+ this._attributes = attributes;
+ this._x = x;
+ this._y = y;
+ this._prediction = prediction;
+ this._remainingTasks = remainingTasks;
+ }
+
+ @Override
+ public Integer call() throws HiveException {
+ final int N = _x.numRows();
+
+ EventStream es = new MatrixEventStream(_x, _y, _attributes);
+ AbstractModel model;
+ try {
+ model = GIS.trainModel(1000, new OnePassRealValueDataIndexer(es,0), _USE_SMOOTHING);
--- End diff --
Yes. Should be definitely.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #93: [WIP][HIVEMALL-126] Maximum Entropy Model usin...
Posted by helenahm <gi...@git.apache.org>.
Github user helenahm commented on the issue:
https://github.com/apache/incubator-hivemall/pull/93
I have also looked at the resulting models last week. Formally, I added only one extra test to tests: hivemall.opennlp.tools.MaxEntPredictUDFTest.java.
/**
* Compare MaxEntropy in HiveMall with that of OpenNLP 3.0.0
*/
@Test
public void testResemblenceToOpenNLP() throws Exception {
}
The code inside the test gets access to internal model representation and compares feature weights.
Using similar code I have looked at the feature weights in 3 models that were relevant for my dataset. All the models look reasonable, that is, key class features get high weights, and models predict what they should. One of the models is an aggregated model (aggregated from the 302 I have got from mappers). This one looks reasonable too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #93: Maximum Entropy Model
Posted by helenahm <gi...@git.apache.org>.
Github user helenahm commented on the issue:
https://github.com/apache/incubator-hivemall/pull/93
I have just put myself as a watcher on Hivemall-126. And run mvn formatter. The formatter made changers in heaps of files. Shall I commit all the changes? Or would it better to pinpoint my own files only?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #93: [WIP][HIVEMALL-126] Maximum Entropy Model usin...
Posted by helenahm <gi...@git.apache.org>.
Github user helenahm commented on the issue:
https://github.com/apache/incubator-hivemall/pull/93
Sure. OpenNLP one will be scalable too. The code is still not perfect now, but (!)
97304256 rows of data
that required 302 mappers on my machine, and
set hive.execution.engine=mr;
set mapreduce.task.timeout=1800000;
no memory issues, I had 302 models back and aggregated them into one after.
Still have to work on the code and check whether the resulting model makes sense.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #93: [WIP][HIVEMALL-126] Maximum Entropy Model usin...
Posted by myui <gi...@git.apache.org>.
Github user myui commented on the issue:
https://github.com/apache/incubator-hivemall/pull/93
@helenahm I think it's better to use [Apache opennlp-tools](https://mvnrepository.com/artifact/org.apache.opennlp/opennlp-tools) `v1.8.1` as @kottmann
mentioned.
`v3.0.0` is not supported anymore and may have some bugs for training other datasets.
Could you explain difficulties in applying `v1.8.1` a bit more? Has API been changed significantly between `v3.0.0` and `v1.8.1`?
[Notice file](https://github.com/apache/incubator-hivemall/blob/master/NOTICE) should be updated too if ported sources (not jar) are included.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #93: [WIP][HIVEMALL-126] Maximum Entropy Model usin...
Posted by kottmann <gi...@git.apache.org>.
Github user kottmann commented on the issue:
https://github.com/apache/incubator-hivemall/pull/93
Sure, there are ways to make this work across multiple machines, but then you can't use it like we ship it. Maybe the best solution for you would be to just take the code you need, strip it down and get rid of opennlp as a dependency?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #93: Maximum Entropy Model
Posted by myui <gi...@git.apache.org>.
Github user myui commented on the issue:
https://github.com/apache/incubator-hivemall/pull/93
@helenahm Thank you for your contribution!
- Could you resolve the conflict? You can add `public void testUTF8()` and rename existing `public void test2()` to `public void testASCII()`.
- Could you apply Hivemall code formatting? You can use `mvn formatter:format` or [this style file](https://github.com/apache/incubator-hivemall/blob/master/resources/eclipse-style.xml) for Eclipse/IntelliJ.
BTW, `hivemall.smile` package is used for the fork of [smile](https://github.com/haifengl/smile/). It's better to use another package. I'm considering to move general classes such as [hivemall.smile.data.Attribute](https://github.com/apache/incubator-hivemall/blob/master/core/src/main/java/hivemall/smile/data/Attribute.java) to [hivemall.common](https://github.com/apache/incubator-hivemall/tree/master/core/src/main/java/hivemall/common).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #93: [WIP][HIVEMALL-126] Maximum Entropy Model usin...
Posted by helenahm <gi...@git.apache.org>.
Github user helenahm commented on the issue:
https://github.com/apache/incubator-hivemall/pull/93
"Changes Unknown when pulling b4383db on helenahm:master into ** on apache:master**."
What does that mean?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall pull request #93: Maximum Entropy Model
Posted by myui <gi...@git.apache.org>.
Github user myui commented on a diff in the pull request:
https://github.com/apache/incubator-hivemall/pull/93#discussion_r125178624
--- Diff: core/src/main/java/hivemall/smile/tools/MatrixEventStream.java ---
@@ -0,0 +1,79 @@
+package hivemall.smile.tools;
+
+import java.util.Iterator;
+import java.util.List;
+
+import hivemall.math.matrix.Matrix;
+import hivemall.smile.data.Attribute;
+import hivemall.smile.data.Attribute.AttributeType;
+import opennlp.model.AbstractEventStream;
+import opennlp.model.Event;
+
+public class MatrixEventStream extends AbstractEventStream {
+ Matrix ds;
+ int[] y;
+ Attribute[] attrs;
+ int rowNum;
+ Event next;
+
+ public MatrixEventStream (Matrix ds, int[] y, Attribute[] attrs) {
+ this.ds = ds;
+ this.y = y;
+ this.attrs = attrs;
+ rowNum = 0;
+
+ if (rowNum < ds.numRows()){
+ next = createEvent(ds.getRow(rowNum), y[rowNum]);
+ };
+ }
+
+ /**
+ * Returns the next Event object held in this EventStream. Each call to nextEvent advances the EventStream.
+ *
+ * @return the Event object which is next in this EventStream
+ */
+ public Event next () {
+ while (next == null && (rowNum < ds.numRows()))
--- End diff --
`{}` is needed to [Google coding style](https://google.github.io/styleguide/javaguide.html#s4.1.2-blocks-k-r-style) that Hivemall follows.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #93: [WIP][HIVEMALL-126] Maximum Entropy Model usin...
Posted by coveralls <gi...@git.apache.org>.
Github user coveralls commented on the issue:
https://github.com/apache/incubator-hivemall/pull/93
[](https://coveralls.io/builds/12520483)
Changes Unknown when pulling **b4383dbe3880494bd24fc54c1a22edbc9c51aa7e on helenahm:master** into ** on apache:master**.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #93: Maximum Entropy Model
Posted by myui <gi...@git.apache.org>.
Github user myui commented on the issue:
https://github.com/apache/incubator-hivemall/pull/93
> •Could you create a JIRA issue for MaxEntropy classifier and rename the PR title to [WIP][HIVEMALL-xxx] ..... ?
>
> How do I do that?
I created an issue for you. https://issues.apache.org/jira/browse/HIVEMALL-126
You need to create an account at Apache JIRA ^.
> •Could you apply Hivemall code formatting? You can use mvn formatter:format or this style file for Eclipse/IntelliJ.
>
> just mvn formatter:format ? How does it work? Is there something in the pom that tells maven how to re-format files?
`mvn formatter:format` uses the following style format internally.
https://github.com/apache/incubator-hivemall/blob/master/resources/eclipse-style.xml
If you are using Eclipse or IntelliJ, just import the above style file as the project code formatter and run source code formatting on IDE.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall pull request #93: [WIP][HIVEMALL-126] Maximum Entropy Mod...
Posted by kottmann <gi...@git.apache.org>.
Github user kottmann commented on a diff in the pull request:
https://github.com/apache/incubator-hivemall/pull/93#discussion_r130337244
--- Diff: core/pom.xml ---
@@ -103,6 +103,12 @@
<version>${guava.version}</version>
<scope>provided</scope>
</dependency>
+ <dependency>
+ <groupId>opennlp</groupId>
+ <artifactId>maxent</artifactId>
+ <version>3.0.0</version>
--- End diff --
We fixed a couple of bugs over time, and added new features, usual maintenance (e.g. testing on recent java versions), so yeah, it probably makes sense to use a recent version when you build something new. Also it supports now multi-threaded training.
The version you are linking above is 7 years old.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall pull request #93: [WIP][HIVEMALL-126] Maximum Entropy Mod...
Posted by helenahm <gi...@git.apache.org>.
Github user helenahm commented on a diff in the pull request:
https://github.com/apache/incubator-hivemall/pull/93#discussion_r130332904
--- Diff: core/pom.xml ---
@@ -103,6 +103,12 @@
<version>${guava.version}</version>
<scope>provided</scope>
</dependency>
+ <dependency>
+ <groupId>opennlp</groupId>
+ <artifactId>maxent</artifactId>
+ <version>3.0.0</version>
--- End diff --
Thank you for the comment. Is there a reason for chasing the latest version? Algorithms do not age... Was an error discovered in the maxent 3.0.0 training?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #93: [WIP][HIVEMALL-126] Maximum Entropy Model usin...
Posted by helenahm <gi...@git.apache.org>.
Github user helenahm commented on the issue:
https://github.com/apache/incubator-hivemall/pull/93
"Yeah, so if you have a lot of training data then running out of memory is one symptom you run into, but that is not the actual problem of this implementation."
- it was the big problem for me to use on Hadoop and that is why i had to alter the training code
- the newer version of the code is as bad as the old one from this point of view
"The actual cause is that it won't scale beyond one machine."
- yes, that is why I really like what Hivemall project is about, and that is why i needed MaxEnt for Hive
"In case you manage to make this run with much more data the time it will take to run will be uncomfortably high."
-- that is why i have tested my new implementation on almost 100 mils of training samples and saw each of 302 mappers finish work in very reasonable time
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #93: [WIP][HIVEMALL-126] Maximum Entropy Model usin...
Posted by myui <gi...@git.apache.org>.
Github user myui commented on the issue:
https://github.com/apache/incubator-hivemall/pull/93
I'll keep my eye on this issue. Let me know when your conclusion is led.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall pull request #93: Maximum Entropy Model
Posted by myui <gi...@git.apache.org>.
Github user myui commented on a diff in the pull request:
https://github.com/apache/incubator-hivemall/pull/93#discussion_r125185131
--- Diff: core/src/main/java/hivemall/smile/tools/MaxEntPredictUDF.java ---
@@ -0,0 +1,179 @@
+package hivemall.smile.tools;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.LinkedList;
+import java.util.List;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import org.apache.hadoop.hive.ql.exec.UDFArgumentException;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.udf.generic.GenericUDF;
+import org.apache.hadoop.hive.ql.udf.generic.GenericUDF.DeferredObject;
+import org.apache.hadoop.hive.serde2.io.DoubleWritable;
+import org.apache.hadoop.hive.serde2.objectinspector.ListObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory;
+import org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;
+import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils;
+import org.apache.hadoop.hive.serde2.objectinspector.primitive.StringObjectInspector;
+import org.apache.hadoop.io.IntWritable;
+import org.apache.hadoop.io.Text;
+
+import hivemall.math.matrix.builders.CSRMatrixBuilder;
+import hivemall.math.matrix.builders.MatrixBuilder;
+import hivemall.math.vector.DenseVector;
+import hivemall.math.vector.SparseVector;
+import hivemall.math.vector.Vector;
+import hivemall.smile.classification.DecisionTree;
+import hivemall.smile.classification.PredictionHandler;
+import hivemall.smile.data.Attribute;
+import hivemall.smile.data.Attribute.AttributeType;
+import hivemall.smile.regression.RegressionTree;
+import hivemall.smile.utils.SmileExtUtils;
+import hivemall.utils.codec.Base91;
+import hivemall.utils.hadoop.HiveUtils;
+import hivemall.utils.hadoop.WritableUtils;
+import hivemall.utils.lang.Preconditions;
+import opennlp.model.AbstractModel;
+import opennlp.model.GenericModelReader;
+import opennlp.model.MaxentModel;
+import opennlp.model.RealValueFileEventStream;
+
+@Description(
+ name = "predict_maxent_classifier",
+ value = "_FUNC_(string model, string attributes, array<double> features)"
+ + " - Returns best class and probability distribution among all the classes per instance.")
+@UDFType(deterministic = true, stateful = false)
+public class MaxEntPredictUDF extends GenericUDF {
+
+ private StringObjectInspector modelOI;
+ private StringObjectInspector attributesOI;
+ private ListObjectInspector featureListOI;
+ private PrimitiveObjectInspector featureElemOI;
+
+ @Nullable
+ private Vector featuresProbe;
+ private Attribute[] attributes;
+
+ @Nullable
+ private transient MaxentModel evaluator;
+
+ @Override
+ public ObjectInspector initialize(ObjectInspector[] argOIs) throws UDFArgumentException {
+ if (argOIs.length != 3) {
+ throw new UDFArgumentException("_FUNC_ takes 3");
+ }
+
+ this.modelOI = HiveUtils.asStringOI(argOIs[0]);
+ this.attributesOI = HiveUtils.asStringOI(argOIs[1]);
+
+ ListObjectInspector listOI = HiveUtils.asListOI(argOIs[2]);
+ this.featureListOI = listOI;
+ ObjectInspector elemOI = listOI.getListElementObjectInspector();
+ if (HiveUtils.isNumberOI(elemOI)) {
+ this.featureElemOI = HiveUtils.asDoubleCompatibleOI(elemOI);
+ } else {
+ throw new UDFArgumentException(
+ "_FUNC_ takes array<double> for the second argument: "
+ + listOI.getTypeName());
+ }
+
+ List<String> fieldNames = new ArrayList<String>(2);
+ List<ObjectInspector> fieldOIs = new ArrayList<ObjectInspector>(2);
+ fieldNames.add("value");
+ fieldOIs.add(PrimitiveObjectInspectorFactory.writableStringObjectInspector);
+ fieldNames.add("posteriori");
+ fieldOIs.add(ObjectInspectorFactory.getStandardListObjectInspector(PrimitiveObjectInspectorFactory.writableDoubleObjectInspector));
+ return ObjectInspectorFactory.getStandardStructObjectInspector(fieldNames, fieldOIs);
+ }
+
+ @Override
+ public Object evaluate(@Nonnull DeferredObject[] arguments) throws HiveException {
+ Object arg0 = arguments[0].get();
+ if (arg0 == null) {
+ throw new HiveException("Model was null");
+ }
+
+ Text model = modelOI.getPrimitiveWritableObject(arg0);
+ try {
+ evaluator = new SepDelimitedTextGISModelReader(model).constructModel();
+ } catch (IOException e) {
+ throw new HiveException(e.getMessage());
+ }
+
+ Object arg1 = arguments[1].get();
+ if (arg1 == null) {
+ throw new HiveException("arguments were null");
+ }
+ attributes = SmileExtUtils.resolveAttributes(attributesOI.getPrimitiveWritableObject(arg1).toString());
+
+ Object arg2 = arguments[2].get();
+ if (arg2 == null) {
+ throw new HiveException("array<double> features was null");
+ }
+ this.featuresProbe = parseFeatures(arg2, featuresProbe);
+
+ double[] obs = this.featuresProbe.toArray();
+
+ String[] names = new String[obs.length];
+ float[] values = new float[obs.length];
+ for (int i = 0; i < obs.length; i++){
+ if (attributes[i].type == AttributeType.NOMINAL){
+ names[i] = i + "_" + String.valueOf(obs[i]).toString();
--- End diff --
`String.valueOf(obs[i])` is enough. `toString()` is redundant.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #93: [WIP][HIVEMALL-126] Maximum Entropy Model usin...
Posted by myui <gi...@git.apache.org>.
Github user myui commented on the issue:
https://github.com/apache/incubator-hivemall/pull/93
@helenahm opennlp-based code might also be useful. Instead of rewriting, having both Smile-based Multinominal Logistic Regression and OpenNLP-based MaxEnt classifier is an option.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #93: [WIP][HIVEMALL-126] Maximum Entropy Model usin...
Posted by helenahm <gi...@git.apache.org>.
Github user helenahm commented on the issue:
https://github.com/apache/incubator-hivemall/pull/93
I think you are right about CSRMatrix (_x here). I should use it without converting it back to unscaleable structures via DataIndexer.
As I literary did by:
EventStream es = new MatrixEventStream(_x, _y, _attributes);
AbstractModel model;
try {
model = GIS.trainModel(1000, new OnePassRealValueDataIndexer(es,0), _USE_SMOOTHING);
} catch (IOException e) {
throw new HiveException(e.getMessage());
}
I will re-write the code to be more scalable, by changing OpenNLP code to a code that uses Smile's and Hivemall's structures. I was mistaken about the ways OpenNLP processes the data from the EventStream.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #93: [WIP][HIVEMALL-126] Maximum Entropy Model usin...
Posted by myui <gi...@git.apache.org>.
Github user myui commented on the issue:
https://github.com/apache/incubator-hivemall/pull/93
@helenahm I agree to use Hivemall's Matrix to reduce memory consumption and create a custom BigGISTrainer for Hivemall.
My concern is that the modification can be based on the latest release of Apache OpenNLP, `v1.8.1` if there are no reason to use pre-apache release.
Anyway, I look into your PR after merging https://github.com/apache/incubator-hivemall/pull/105
Maybe in the next week. Some refactoring would be applied (such as removing debug prints and unused codes) forking your PR branch.
BTW, multi-thresholding should be avoided when running a task in a Yarn container. Better to be parallelized by Hive.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall pull request #93: Maximum Entropy Model
Posted by myui <gi...@git.apache.org>.
Github user myui commented on a diff in the pull request:
https://github.com/apache/incubator-hivemall/pull/93#discussion_r125185167
--- Diff: core/src/main/java/hivemall/smile/utils/SmileExtUtils.java ---
@@ -78,6 +78,42 @@ private SmileExtUtils() {}
return attr;
}
+/**
+ * Q for {@link NumericAttribute}, C for {@link NominalAttribute}.
+ */
+ @Nullable
+ public static String resolveAttributes(@Nullable final Attribute[] opt)
+ throws UDFArgumentException {
+ if (opt == null) {
+ return null;
+ }
+
+ final int size = opt.length;
+ final String[] opts = new String[size];
+ final NumericAttribute immutableNumAttr = new NumericAttribute();
+
+ for (int i = 0; i < size; i++) {
+ final Attribute type = opt[i];
+ if (type.type == AttributeType.NUMERIC){
+ opts[i] = "Q";
+ } else if (type.type == AttributeType.NOMINAL) {
+ opts[i] = "C";
+ } else {
+ throw new UDFArgumentException("Unexpected type: " + type);
+ }
+ }
+
+ String opts_str = "";
--- End diff --
Better to use `StringBuilder` for `opts_str`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall pull request #93: [WIP][HIVEMALL-126] Maximum Entropy Mod...
Posted by kottmann <gi...@git.apache.org>.
Github user kottmann commented on a diff in the pull request:
https://github.com/apache/incubator-hivemall/pull/93#discussion_r130319727
--- Diff: core/pom.xml ---
@@ -103,6 +103,12 @@
<version>${guava.version}</version>
<scope>provided</scope>
</dependency>
+ <dependency>
+ <groupId>opennlp</groupId>
+ <artifactId>maxent</artifactId>
+ <version>3.0.0</version>
--- End diff --
Consider using a recent version of OpenNLP instead. The maxent code was moved into opennlp-tools (latest version is 1.8.1). The version you use here is from a couple of years ago.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall pull request #93: Maximum Entropy Model
Posted by myui <gi...@git.apache.org>.
Github user myui commented on a diff in the pull request:
https://github.com/apache/incubator-hivemall/pull/93#discussion_r125178629
--- Diff: core/src/main/java/hivemall/smile/tools/MatrixEventStream.java ---
@@ -0,0 +1,79 @@
+package hivemall.smile.tools;
+
+import java.util.Iterator;
+import java.util.List;
+
+import hivemall.math.matrix.Matrix;
+import hivemall.smile.data.Attribute;
+import hivemall.smile.data.Attribute.AttributeType;
+import opennlp.model.AbstractEventStream;
+import opennlp.model.Event;
+
+public class MatrixEventStream extends AbstractEventStream {
+ Matrix ds;
+ int[] y;
+ Attribute[] attrs;
+ int rowNum;
+ Event next;
+
+ public MatrixEventStream (Matrix ds, int[] y, Attribute[] attrs) {
+ this.ds = ds;
+ this.y = y;
+ this.attrs = attrs;
+ rowNum = 0;
+
+ if (rowNum < ds.numRows()){
+ next = createEvent(ds.getRow(rowNum), y[rowNum]);
+ };
+ }
+
+ /**
+ * Returns the next Event object held in this EventStream. Each call to nextEvent advances the EventStream.
+ *
+ * @return the Event object which is next in this EventStream
+ */
+ public Event next () {
+ while (next == null && (rowNum < ds.numRows()))
+ next = createEvent(ds.getRow(rowNum), y[rowNum]);
+
+ Event current = next;
+ if (rowNum < ds.numRows()) {
+ next = createEvent(ds.getRow(rowNum), y[rowNum]);
+ }
+ else {
+ next = null;
+ }
+ return current;
+ }
+
+ /**
+ * Test whether there are any Events remaining in this EventStream.
+ *
+ * @return true if this EventStream has more Events
+ */
+ public boolean hasNext () {
+ while (next == null && rowNum < ds.numRows())
+ next = createEvent(ds.getRow(rowNum), y[rowNum]);
+ return next != null;
+ }
+
+ private Event createEvent(double[] obs, int y) {
+ rowNum++;
+ if (obs == null)
--- End diff --
`{}` same as the above.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #93: [WIP][HIVEMALL-126] Maximum Entropy Model usin...
Posted by myui <gi...@git.apache.org>.
Github user myui commented on the issue:
https://github.com/apache/incubator-hivemall/pull/93
@kottmann Do you know in which version maxent classifier is moved to opennlp-tools?
Versioning scheme of opennlp-maxent and opennlp-tools modules are very different.
https://mvnrepository.com/artifact/org.apache.opennlp/opennlp-maxent
https://mvnrepository.com/artifact/org.apache.opennlp/opennlp-tools
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #93: [WIP][HIVEMALL-126] Maximum Entropy Model usin...
Posted by kottmann <gi...@git.apache.org>.
Github user kottmann commented on the issue:
https://github.com/apache/incubator-hivemall/pull/93
@myui the maxent 3.0.1 version went through Apache IP clearance when the code base was moved from SourceForge, and should be almost identical to 3.0.0.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall pull request #93: Maximum Entropy Model
Posted by myui <gi...@git.apache.org>.
Github user myui commented on a diff in the pull request:
https://github.com/apache/incubator-hivemall/pull/93#discussion_r125185138
--- Diff: core/src/main/java/hivemall/smile/tools/MaxEntPredictUDF.java ---
@@ -0,0 +1,179 @@
+package hivemall.smile.tools;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.LinkedList;
+import java.util.List;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import org.apache.hadoop.hive.ql.exec.UDFArgumentException;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.udf.generic.GenericUDF;
+import org.apache.hadoop.hive.ql.udf.generic.GenericUDF.DeferredObject;
+import org.apache.hadoop.hive.serde2.io.DoubleWritable;
+import org.apache.hadoop.hive.serde2.objectinspector.ListObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory;
+import org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;
+import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils;
+import org.apache.hadoop.hive.serde2.objectinspector.primitive.StringObjectInspector;
+import org.apache.hadoop.io.IntWritable;
+import org.apache.hadoop.io.Text;
+
+import hivemall.math.matrix.builders.CSRMatrixBuilder;
+import hivemall.math.matrix.builders.MatrixBuilder;
+import hivemall.math.vector.DenseVector;
+import hivemall.math.vector.SparseVector;
+import hivemall.math.vector.Vector;
+import hivemall.smile.classification.DecisionTree;
+import hivemall.smile.classification.PredictionHandler;
+import hivemall.smile.data.Attribute;
+import hivemall.smile.data.Attribute.AttributeType;
+import hivemall.smile.regression.RegressionTree;
+import hivemall.smile.utils.SmileExtUtils;
+import hivemall.utils.codec.Base91;
+import hivemall.utils.hadoop.HiveUtils;
+import hivemall.utils.hadoop.WritableUtils;
+import hivemall.utils.lang.Preconditions;
+import opennlp.model.AbstractModel;
+import opennlp.model.GenericModelReader;
+import opennlp.model.MaxentModel;
+import opennlp.model.RealValueFileEventStream;
+
+@Description(
+ name = "predict_maxent_classifier",
+ value = "_FUNC_(string model, string attributes, array<double> features)"
+ + " - Returns best class and probability distribution among all the classes per instance.")
+@UDFType(deterministic = true, stateful = false)
+public class MaxEntPredictUDF extends GenericUDF {
+
+ private StringObjectInspector modelOI;
+ private StringObjectInspector attributesOI;
+ private ListObjectInspector featureListOI;
+ private PrimitiveObjectInspector featureElemOI;
+
+ @Nullable
+ private Vector featuresProbe;
+ private Attribute[] attributes;
+
+ @Nullable
+ private transient MaxentModel evaluator;
+
+ @Override
+ public ObjectInspector initialize(ObjectInspector[] argOIs) throws UDFArgumentException {
+ if (argOIs.length != 3) {
+ throw new UDFArgumentException("_FUNC_ takes 3");
+ }
+
+ this.modelOI = HiveUtils.asStringOI(argOIs[0]);
+ this.attributesOI = HiveUtils.asStringOI(argOIs[1]);
+
+ ListObjectInspector listOI = HiveUtils.asListOI(argOIs[2]);
+ this.featureListOI = listOI;
+ ObjectInspector elemOI = listOI.getListElementObjectInspector();
+ if (HiveUtils.isNumberOI(elemOI)) {
+ this.featureElemOI = HiveUtils.asDoubleCompatibleOI(elemOI);
+ } else {
+ throw new UDFArgumentException(
+ "_FUNC_ takes array<double> for the second argument: "
+ + listOI.getTypeName());
+ }
+
+ List<String> fieldNames = new ArrayList<String>(2);
+ List<ObjectInspector> fieldOIs = new ArrayList<ObjectInspector>(2);
+ fieldNames.add("value");
+ fieldOIs.add(PrimitiveObjectInspectorFactory.writableStringObjectInspector);
+ fieldNames.add("posteriori");
+ fieldOIs.add(ObjectInspectorFactory.getStandardListObjectInspector(PrimitiveObjectInspectorFactory.writableDoubleObjectInspector));
+ return ObjectInspectorFactory.getStandardStructObjectInspector(fieldNames, fieldOIs);
+ }
+
+ @Override
+ public Object evaluate(@Nonnull DeferredObject[] arguments) throws HiveException {
+ Object arg0 = arguments[0].get();
+ if (arg0 == null) {
+ throw new HiveException("Model was null");
+ }
+
+ Text model = modelOI.getPrimitiveWritableObject(arg0);
+ try {
+ evaluator = new SepDelimitedTextGISModelReader(model).constructModel();
+ } catch (IOException e) {
+ throw new HiveException(e.getMessage());
+ }
+
+ Object arg1 = arguments[1].get();
+ if (arg1 == null) {
+ throw new HiveException("arguments were null");
+ }
+ attributes = SmileExtUtils.resolveAttributes(attributesOI.getPrimitiveWritableObject(arg1).toString());
+
+ Object arg2 = arguments[2].get();
+ if (arg2 == null) {
+ throw new HiveException("array<double> features was null");
+ }
+ this.featuresProbe = parseFeatures(arg2, featuresProbe);
+
+ double[] obs = this.featuresProbe.toArray();
+
+ String[] names = new String[obs.length];
+ float[] values = new float[obs.length];
+ for (int i = 0; i < obs.length; i++){
+ if (attributes[i].type == AttributeType.NOMINAL){
+ names[i] = i + "_" + String.valueOf(obs[i]).toString();
+ values[i] = Double.valueOf(1.0).floatValue();
--- End diff --
Why not just `values[i] = 1.f`;
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #93: [WIP][HIVEMALL-126] Maximum Entropy Model usin...
Posted by helenahm <gi...@git.apache.org>.
Github user helenahm commented on the issue:
https://github.com/apache/incubator-hivemall/pull/93
@myui I share the concern that the modification can be based on the latest release of Apache OpenNLP, v1.8.1 if there are no reason to use pre-apache release. If I knew about the newer version of maxent at the very beginning I would have used it.
I will examine the newer maxent code in the next few days.
As you said, have a look at the PR when you have time. And then a decision what to do can be made.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall pull request #93: Maximum Entropy Model
Posted by myui <gi...@git.apache.org>.
Github user myui commented on a diff in the pull request:
https://github.com/apache/incubator-hivemall/pull/93#discussion_r125185193
--- Diff: core/src/main/java/hivemall/smile/tools/MatrixEventStream.java ---
@@ -0,0 +1,81 @@
+package hivemall.smile.tools;
+
+import java.util.Iterator;
+import java.util.List;
+
+import hivemall.math.matrix.Matrix;
+import hivemall.smile.data.Attribute;
+import hivemall.smile.data.Attribute.AttributeType;
+import opennlp.model.AbstractEventStream;
+import opennlp.model.Event;
+
+public class MatrixEventStream extends AbstractEventStream {
+ Matrix ds;
+ int[] y;
+ Attribute[] attrs;
+ int rowNum;
+ Event next;
+
+ public MatrixEventStream (Matrix ds, int[] y, Attribute[] attrs) {
+ this.ds = ds;
+ this.y = y;
+ this.attrs = attrs;
+ rowNum = 0;
+
+ if (rowNum < ds.numRows()){
+ next = createEvent(ds.getRow(rowNum), y[rowNum]);
+ };
--- End diff --
`};`=> `}`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #93: Maximum Entropy Model
Posted by helenahm <gi...@git.apache.org>.
Github user helenahm commented on the issue:
https://github.com/apache/incubator-hivemall/pull/93
I have completed all the small code changes you have commented on.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #93: Maximum Entropy Model
Posted by helenahm <gi...@git.apache.org>.
Github user helenahm commented on the issue:
https://github.com/apache/incubator-hivemall/pull/93
And please, disregard the LDAUDTFTest.java. We have already discussed that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall pull request #93: [WIP][HIVEMALL-126] Maximum Entropy Mod...
Posted by myui <gi...@git.apache.org>.
Github user myui commented on a diff in the pull request:
https://github.com/apache/incubator-hivemall/pull/93#discussion_r125542839
--- Diff: resources/ddl/define-all.hive ---
@@ -60,7 +60,7 @@ create temporary function train_multiclass_pa as 'hivemall.classifier.multiclass
drop temporary function if exists train_multiclass_pa1;
create temporary function train_multiclass_pa1 as 'hivemall.classifier.multiclass.MulticlassPassiveAggressiveUDTF$PA1';
-
+aggregate_classifier
--- End diff --
remove this line.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #93: [WIP][HIVEMALL-126] Maximum Entropy Model usin...
Posted by helenahm <gi...@git.apache.org>.
Github user helenahm commented on the issue:
https://github.com/apache/incubator-hivemall/pull/93
I think the code is ready to be checked and pulled now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #93: Maximum Entropy Model
Posted by helenahm <gi...@git.apache.org>.
Github user helenahm commented on the issue:
https://github.com/apache/incubator-hivemall/pull/93
•Could you create a JIRA issue for MaxEntropy classifier and rename the PR title to [WIP][HIVEMALL-xxx] ..... ?
How do I do that?
•Could you apply Hivemall code formatting? You can use mvn formatter:format or this style file for Eclipse/IntelliJ.
just mvn formatter:format ? How does it work? Is there something in the pom that tells maven how to re-format files?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall issue #93: Maximum Entropy Model
Posted by helenahm <gi...@git.apache.org>.
Github user helenahm commented on the issue:
https://github.com/apache/incubator-hivemall/pull/93
In Hivemall-126:
Max Entropy Classifier (a.k.a. Multi-nominal/Multiclass Logistic Regression) [1,2] is useful for Text classification.
Max Entropy Classifier is more often used for Part-of-Speech Tagging and Named Entity Recognition, and some other tasks where context is used as features. Those are also fundamental tasks of NLP. Even though Text Classification is a candidate too.
Mohri as his colleagues also put POS task first. As Mohri writes in the article that is a basis for the implementation I have chosen:
Our first set of experiments were carried out with “medium” scale data sets containing 1M-300Minstances.
These included: English part-of-speech tagging, generated from the Penn Treebank
[16] using the first character of each part-of-speech tag as output, sections 2-21 for training, section
23 for testing and a feature representation based on the identity, affixes, and orthography of the input
word and the words in a window of size two; Sentiment analysis, generated from a set of
online product, service, and merchant reviews with a three-label output (positive, negative, neutral),
with a bag of words feature representation; RCV1-v2 as described by [14], where documents having
multiple labels were included multiple times, once for each label; Acoustic Speech Data, a 39-
dimensional input consisting of 13 PLP coefficients, plus their first and second derivatives, and 129
outputs (43 phones × 3 acoustic states); and the Deja News Archive, a text topic classification
problem generated from a collection of Usenet discussion forums from the years 1995-2000. For all
text experiments, we used random feature mixing [9, 20] to control the size of the feature space.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall pull request #93: Maximum Entropy Model
Posted by myui <gi...@git.apache.org>.
Github user myui commented on a diff in the pull request:
https://github.com/apache/incubator-hivemall/pull/93#discussion_r125185311
--- Diff: core/src/main/java/hivemall/smile/tools/MaxEntPredictUDF.java ---
@@ -0,0 +1,179 @@
+package hivemall.smile.tools;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.LinkedList;
+import java.util.List;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import org.apache.hadoop.hive.ql.exec.UDFArgumentException;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.udf.generic.GenericUDF;
+import org.apache.hadoop.hive.ql.udf.generic.GenericUDF.DeferredObject;
+import org.apache.hadoop.hive.serde2.io.DoubleWritable;
+import org.apache.hadoop.hive.serde2.objectinspector.ListObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory;
+import org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;
+import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils;
+import org.apache.hadoop.hive.serde2.objectinspector.primitive.StringObjectInspector;
+import org.apache.hadoop.io.IntWritable;
+import org.apache.hadoop.io.Text;
+
+import hivemall.math.matrix.builders.CSRMatrixBuilder;
+import hivemall.math.matrix.builders.MatrixBuilder;
+import hivemall.math.vector.DenseVector;
+import hivemall.math.vector.SparseVector;
+import hivemall.math.vector.Vector;
+import hivemall.smile.classification.DecisionTree;
+import hivemall.smile.classification.PredictionHandler;
+import hivemall.smile.data.Attribute;
+import hivemall.smile.data.Attribute.AttributeType;
+import hivemall.smile.regression.RegressionTree;
+import hivemall.smile.utils.SmileExtUtils;
+import hivemall.utils.codec.Base91;
+import hivemall.utils.hadoop.HiveUtils;
+import hivemall.utils.hadoop.WritableUtils;
+import hivemall.utils.lang.Preconditions;
+import opennlp.model.AbstractModel;
+import opennlp.model.GenericModelReader;
+import opennlp.model.MaxentModel;
+import opennlp.model.RealValueFileEventStream;
+
+@Description(
+ name = "predict_maxent_classifier",
+ value = "_FUNC_(string model, string attributes, array<double> features)"
+ + " - Returns best class and probability distribution among all the classes per instance.")
+@UDFType(deterministic = true, stateful = false)
+public class MaxEntPredictUDF extends GenericUDF {
+
+ private StringObjectInspector modelOI;
+ private StringObjectInspector attributesOI;
+ private ListObjectInspector featureListOI;
+ private PrimitiveObjectInspector featureElemOI;
+
+ @Nullable
+ private Vector featuresProbe;
+ private Attribute[] attributes;
+
+ @Nullable
+ private transient MaxentModel evaluator;
+
+ @Override
+ public ObjectInspector initialize(ObjectInspector[] argOIs) throws UDFArgumentException {
+ if (argOIs.length != 3) {
+ throw new UDFArgumentException("_FUNC_ takes 3");
+ }
+
+ this.modelOI = HiveUtils.asStringOI(argOIs[0]);
+ this.attributesOI = HiveUtils.asStringOI(argOIs[1]);
+
+ ListObjectInspector listOI = HiveUtils.asListOI(argOIs[2]);
+ this.featureListOI = listOI;
+ ObjectInspector elemOI = listOI.getListElementObjectInspector();
+ if (HiveUtils.isNumberOI(elemOI)) {
+ this.featureElemOI = HiveUtils.asDoubleCompatibleOI(elemOI);
+ } else {
+ throw new UDFArgumentException(
+ "_FUNC_ takes array<double> for the second argument: "
+ + listOI.getTypeName());
+ }
+
+ List<String> fieldNames = new ArrayList<String>(2);
+ List<ObjectInspector> fieldOIs = new ArrayList<ObjectInspector>(2);
+ fieldNames.add("value");
+ fieldOIs.add(PrimitiveObjectInspectorFactory.writableStringObjectInspector);
+ fieldNames.add("posteriori");
+ fieldOIs.add(ObjectInspectorFactory.getStandardListObjectInspector(PrimitiveObjectInspectorFactory.writableDoubleObjectInspector));
+ return ObjectInspectorFactory.getStandardStructObjectInspector(fieldNames, fieldOIs);
+ }
+
+ @Override
+ public Object evaluate(@Nonnull DeferredObject[] arguments) throws HiveException {
+ Object arg0 = arguments[0].get();
+ if (arg0 == null) {
+ throw new HiveException("Model was null");
+ }
+
+ Text model = modelOI.getPrimitiveWritableObject(arg0);
+ try {
+ evaluator = new SepDelimitedTextGISModelReader(model).constructModel();
+ } catch (IOException e) {
+ throw new HiveException(e.getMessage());
+ }
+
+ Object arg1 = arguments[1].get();
+ if (arg1 == null) {
+ throw new HiveException("arguments were null");
+ }
+ attributes = SmileExtUtils.resolveAttributes(attributesOI.getPrimitiveWritableObject(arg1).toString());
+
+ Object arg2 = arguments[2].get();
+ if (arg2 == null) {
+ throw new HiveException("array<double> features was null");
+ }
+ this.featuresProbe = parseFeatures(arg2, featuresProbe);
+
+ double[] obs = this.featuresProbe.toArray();
+
+ String[] names = new String[obs.length];
+ float[] values = new float[obs.length];
+ for (int i = 0; i < obs.length; i++){
+ if (attributes[i].type == AttributeType.NOMINAL){
+ names[i] = i + "_" + String.valueOf(obs[i]).toString();
--- End diff --
`i + "_" + obs[i]` will be perfectly compiled into
`new StringBuilder().append(i).append("_").append(obj[i])`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall pull request #93: Maximum Entropy Model
Posted by myui <gi...@git.apache.org>.
Github user myui commented on a diff in the pull request:
https://github.com/apache/incubator-hivemall/pull/93#discussion_r125178426
--- Diff: core/src/main/java/hivemall/smile/tools/MaxEntPredictUDF.java ---
@@ -0,0 +1,174 @@
+package hivemall.smile.tools;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.LinkedList;
+import java.util.List;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import org.apache.hadoop.hive.ql.exec.UDFArgumentException;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.udf.generic.GenericUDF;
+import org.apache.hadoop.hive.ql.udf.generic.GenericUDF.DeferredObject;
+import org.apache.hadoop.hive.serde2.io.DoubleWritable;
+import org.apache.hadoop.hive.serde2.objectinspector.ListObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory;
+import org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;
+import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils;
+import org.apache.hadoop.hive.serde2.objectinspector.primitive.StringObjectInspector;
+import org.apache.hadoop.io.IntWritable;
+import org.apache.hadoop.io.Text;
+
+import hivemall.math.matrix.builders.CSRMatrixBuilder;
+import hivemall.math.matrix.builders.MatrixBuilder;
+import hivemall.math.vector.DenseVector;
+import hivemall.math.vector.SparseVector;
+import hivemall.math.vector.Vector;
+import hivemall.smile.classification.DecisionTree;
+import hivemall.smile.classification.PredictionHandler;
+import hivemall.smile.data.Attribute;
+import hivemall.smile.data.Attribute.AttributeType;
+import hivemall.smile.regression.RegressionTree;
+import hivemall.smile.utils.SmileExtUtils;
+import hivemall.utils.codec.Base91;
+import hivemall.utils.hadoop.HiveUtils;
+import hivemall.utils.hadoop.WritableUtils;
+import hivemall.utils.lang.Preconditions;
+import opennlp.model.AbstractModel;
+import opennlp.model.GenericModelReader;
+import opennlp.model.MaxentModel;
+import opennlp.model.RealValueFileEventStream;
+
+public class MaxEntPredictUDF extends GenericUDF {
--- End diff --
UDF/UDTF/UDAF classes should have UDF annotation as seen in https://github.com/apache/incubator-hivemall/blob/master/core/src/main/java/hivemall/smile/tools/TreePredictUDF.java#L56.
It is used when [`desc function extended xxx`](https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-HiveOperatorsandUser-DefinedFunctions(UDFs)).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall pull request #93: [WIP][HIVEMALL-126] Maximum Entropy Mod...
Posted by kottmann <gi...@git.apache.org>.
Github user kottmann commented on a diff in the pull request:
https://github.com/apache/incubator-hivemall/pull/93#discussion_r130568528
--- Diff: core/pom.xml ---
@@ -103,6 +103,12 @@
<version>${guava.version}</version>
<scope>provided</scope>
</dependency>
+ <dependency>
+ <groupId>opennlp</groupId>
+ <artifactId>maxent</artifactId>
+ <version>3.0.0</version>
--- End diff --
The same as before. I recommend to use a recent version.
Since you are including parts of the code directly I kindly ask you to also update the NOTICE file with Apache OpenNLP attribution.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall pull request #93: [WIP][HIVEMALL-126] Maximum Entropy Mod...
Posted by myui <gi...@git.apache.org>.
Github user myui commented on a diff in the pull request:
https://github.com/apache/incubator-hivemall/pull/93#discussion_r125543784
--- Diff: core/src/main/java/hivemall/smile/classification/MaxEntUDTF.java ---
@@ -0,0 +1,440 @@
+package hivemall.smile.classification;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.BitSet;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import javax.annotation.Nonnegative;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import javax.annotation.concurrent.GuardedBy;
+
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.Options;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.hive.ql.exec.MapredContext;
+import org.apache.hadoop.hive.ql.exec.MapredContextAccessor;
+import org.apache.hadoop.hive.ql.exec.UDFArgumentException;
+import org.apache.hadoop.hive.ql.exec.UDFArgumentTypeException;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.serde2.io.DoubleWritable;
+import org.apache.hadoop.hive.serde2.objectinspector.ListObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory;
+import org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;
+import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils;
+import org.apache.hadoop.io.IntWritable;
+import org.apache.hadoop.io.Text;
+import org.apache.hadoop.mapred.Reporter;
+import org.apache.hadoop.mapred.Counters.Counter;
+
+import hivemall.UDTFWithOptions;
+import hivemall.math.matrix.Matrix;
+import hivemall.math.matrix.MatrixUtils;
+import hivemall.math.matrix.builders.CSRMatrixBuilder;
+import hivemall.math.matrix.builders.MatrixBuilder;
+import hivemall.math.matrix.builders.RowMajorDenseMatrixBuilder;
+import hivemall.math.matrix.ints.ColumnMajorIntMatrix;
+import hivemall.math.matrix.ints.DoKIntMatrix;
+import hivemall.math.matrix.ints.IntMatrix;
+import hivemall.math.random.PRNG;
+import hivemall.math.random.RandomNumberGeneratorFactory;
+import hivemall.math.vector.Vector;
+import hivemall.math.vector.VectorProcedure;
+import hivemall.smile.classification.DecisionTree.SplitRule;
+import hivemall.smile.data.Attribute;
+import hivemall.smile.tools.MatrixEventStream;
+import hivemall.smile.tools.SepDelimitedTextGISModelWriter;
+import hivemall.smile.utils.SmileExtUtils;
+import hivemall.smile.utils.SmileTaskExecutor;
+import hivemall.utils.codec.Base91;
+import hivemall.utils.collections.lists.IntArrayList;
+import hivemall.utils.hadoop.HiveUtils;
+import hivemall.utils.hadoop.WritableUtils;
+import hivemall.utils.lang.Preconditions;
+import hivemall.utils.lang.Primitives;
+import hivemall.utils.lang.RandomUtils;
+
+import opennlp.maxent.GIS;
+import opennlp.maxent.io.GISModelWriter;
+import opennlp.model.AbstractModel;
+import opennlp.model.Event;
+import opennlp.model.EventStream;
+import opennlp.model.OnePassRealValueDataIndexer;
+
+@Description(
+ name = "train_maxent_classifier",
+ value = "_FUNC_(array<double> features, int label [, const boolean classification])"
+ + " - Returns a maximum entropy model per subset of data.")
+@UDFType(deterministic = true, stateful = false)
+public class MaxEntUDTF extends UDTFWithOptions{
+ private static final Log logger = LogFactory.getLog(MaxEntUDTF.class);
+
+ private ListObjectInspector featureListOI;
--- End diff --
Inconsistent indentation. Formatter should be applied.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---