You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hivemall.apache.org by takuti <gi...@git.apache.org> on 2017/04/13 02:33:40 UTC

[GitHub] incubator-hivemall pull request #71: [HIVEMALL-74] Implement pLSA

GitHub user takuti opened a pull request:

    https://github.com/apache/incubator-hivemall/pull/71

    [HIVEMALL-74] Implement pLSA

    ## What changes were proposed in this pull request?
    
    Implement (incremental) probabilistic latent semantic analysis (pLSA) algorithm:
    
    - Original papers:
      - [Probabilistic Latent Semantic Indexing](http://dl.acm.org/citation.cfm?id=312649)
      - [Probabilistic Latent Semantic Analysis](http://www.iro.umontreal.ca/~nie/IFT6255/Hofmann-UAI99.pdf)
    - Incremental variant which is implemented in this PR:
      - [Incremental Probabilistic Latent Semantic Analysis for Automatic Question Recommendation](https://pdfs.semanticscholar.org/b66e/c7faf2e4888503f7ad1537d284f350fb3e58.pdf)
    
    ## What type of PR is it?
    
    Feature
    
    ## What is the Jira issue?
    
    https://issues.apache.org/jira/browse/HIVEMALL-74
    
    ## How was this patch tested?
    
    - unit tests
    - manual tests on local
    
    ## How to use this feature?
    
    Basically, interfaces are similar to Online LDA implemented in #66; we have `train_plsa()` and `plsa_predict()` functions.
    
    For a `docs` table:
    
    | docid | doc  |
    |:---:|:---|
    | 1  | "Fruits and vegetables are healthy." |
    |2 | "I like apples, oranges, and avocados. I do not like the flu or colds." |
    
    the following query successfully learns topics behind the two documents:
    
    ```sql
    with word_counts as (
      select
        docid,
        feature(word, count(word)) as f
      from docs t1 lateral view explode(tokenize(doc, true)) t2 as word
      where
        not is_stopword(word)
      group by
        docid, word
    ),
    plsa_model as (
      select
        train_plsa(feature, "-topic 2 -iter 10000 -eps 0.00001 -delta 0.00001 -alpha 0.00001") as (label, word, prob)
      from (
        select docid, collect_set(f) as feature
        from word_counts
        group by docid
      ) t
    ),
    test as (
      select
        docid,
        word,
        count(word) as value
      from docs t1 LATERAL VIEW explode(tokenize(doc, true)) t2 as word
      where
        not is_stopword(word)
      group by
        docid, word
    ),
    topic as (
      select
        t.docid,
        plsa_predict(t.word, t.value, m.label, m.prob, "-topic 2 -delta 0.01 -alpha 0.00001") as probabilities
      from
        test t
        JOIN plsa_model m ON (t.word = m.word)
      group by
        t.docid
    )
    select docid, probabilities, probabilities[0].label, m.words -- topic each document should be assigned
    from topic t
    join (
      select label, collect_set(feature(word, prob)) as words
      from plsa_model
      group by label
    ) m on t.probabilities[0].label = m.label
    ;
    ```
    
    | docid | doc  |
    |:---:|:---|
    | 1  | "Fruits and vegetables are healthy." |
    |2 | "I like apples, oranges, and avocados. I do not like the flu or colds." |
    
    |docid  | probabilities  | label |  words |
    |:---:|:---|:---:|:---|
    |1   |    [{"label":0,"probability":1.0},{"label":1,"probability":1.1246405E-32}]  | 0|["fruits:0.33333066","healthy:0.33333066","vegetables:0.33333066","like:2.4555745E-6","avocados:2.0867487E-6","colds:1.665714E-6","flu:1.0358361E-6","apples:5.5809795E-7","oranges:2.2456922E-7"] |
    |2 |      [{"label":1,"probability":0.9999961},{"label":0,"probability":3.886718E-6}]  |   1     |  ["like:0.28571412","oranges:0.14285718","colds:0.14285718","avocados:0.14285718","apples:0.14285718","flu:0.14285718","healthy:1.841767E-32","vegetables:1.2376679E-32","fruits:7.812756E-34"] |


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/takuti/incubator-hivemall plsa

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-hivemall/pull/71.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #71
    
----
commit afac23fc002d9cbc7cd446a713cba9607564fe54
Author: Takuya Kitazawa <k....@gmail.com>
Date:   2017-04-12T05:25:21Z

    Implement Incremental pLSA model

commit 0af3783ec89c56e09c410f39bc1a4b359cffc2d9
Author: Takuya Kitazawa <k....@gmail.com>
Date:   2017-04-12T08:57:26Z

    Implement `train_plsa` UDTF and its test

commit d7ebee3d8d3c8e950f803fe07aefaee12ae2c806
Author: Takuya Kitazawa <k....@gmail.com>
Date:   2017-04-12T09:26:05Z

    Off debug print

commit f247542f88397f28c93a53d8a0b5140f785b131b
Author: Takuya Kitazawa <k....@gmail.com>
Date:   2017-04-13T02:12:22Z

    Implement `plsa_predict` UDAF and its test

commit 82597a2ca4ffefefb301b89aba57e24960fb0436
Author: Takuya Kitazawa <k....@gmail.com>
Date:   2017-04-13T02:21:38Z

    Add pLSA UDFs to ddl

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall pull request #71: [WIP][HIVEMALL-74] Implement pLSA

Posted by myui <gi...@git.apache.org>.
Github user myui commented on a diff in the pull request:

    https://github.com/apache/incubator-hivemall/pull/71#discussion_r112697171
  
    --- Diff: core/src/main/java/hivemall/topicmodel/IncrementalPLSAModel.java ---
    @@ -0,0 +1,300 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *   http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing,
    + * software distributed under the License is distributed on an
    + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    + * KIND, either express or implied.  See the License for the
    + * specific language governing permissions and limitations
    + * under the License.
    + */
    +package hivemall.topicmodel;
    +
    +import hivemall.model.FeatureValue;
    +import hivemall.utils.lang.ArrayUtils;
    +
    +import java.util.Arrays;
    +import java.util.ArrayList;
    +import java.util.Collections;
    +import java.util.HashMap;
    +import java.util.List;
    +import java.util.Map;
    +import java.util.Random;
    +import java.util.SortedMap;
    +import java.util.TreeMap;
    +
    +import javax.annotation.Nonnegative;
    +import javax.annotation.Nonnull;
    +
    +public final class IncrementalPLSAModel {
    +
    +    // ---------------------------------
    +    // HyperParameters
    +
    +    // number of topics
    +    private final int _K;
    +
    +    // control how much P(w|z) update is affected by the last value
    +    private final float _alpha;
    +
    +    // check convergence of P(w|z) for a document
    +    private final double _delta;
    +
    +    // ---------------------------------
    +
    +    // random number generator
    +    @Nonnull
    +    private final Random _rnd;
    +
    +    // optimized in the E step
    +    private List<Map<String, float[]>> _p_dwz; // P(z|d,w) probability of topics for each document-label pair
    +
    +    // optimized in the M step
    +    @Nonnull
    +    private List<float[]> _p_dz; // P(z|d) probability of topics for documents
    +    private Map<String, float[]> _p_zw; // P(w|z) probability of labels for each topic
    +
    +    @Nonnull
    +    private final List<Map<String, Float>> _miniBatchDocs;
    +    private int _miniBatchSize;
    +
    +    public IncrementalPLSAModel(int K, float alpha, double delta) {
    +        this._K = K;
    +        this._alpha = alpha;
    +        this._delta = delta;
    +
    +        this._rnd = new Random(1001);
    +
    +        this._p_zw = new HashMap<String, float[]>();
    +
    +        this._miniBatchDocs = new ArrayList<Map<String, Float>>();
    +    }
    +
    +    public void train(@Nonnull final String[][] miniBatch) {
    +        initMiniBatch(miniBatch, _miniBatchDocs);
    +
    +        this._miniBatchSize = _miniBatchDocs.size();
    +
    +        initParams();
    +
    +        final List<float[]> pPrev_dz = new ArrayList<float[]>();
    +
    +        for (int d = 0; d < _miniBatchSize; d++) {
    +            do {
    +                pPrev_dz.clear();
    +                pPrev_dz.addAll(_p_dz);
    +
    +                // Expectation
    +                eStep(d);
    +
    +                // Maximization
    +                mStep(d);
    +            } while (!isPdzConverged(d, pPrev_dz, _p_dz)); // until get stable value of P(z|d)
    +        }
    +    }
    +
    +    private static void initMiniBatch(@Nonnull final String[][] miniBatch,
    +            @Nonnull final List<Map<String, Float>> docs) {
    +        docs.clear();
    +
    +        final FeatureValue probe = new FeatureValue();
    +
    +        // parse document
    +        for (final String[] e : miniBatch) {
    +            if (e == null || e.length == 0) {
    +                continue;
    +            }
    +
    +            final Map<String, Float> doc = new HashMap<String, Float>();
    +
    +            // parse features
    +            for (String fv : e) {
    +                if (fv == null) {
    +                    continue;
    +                }
    +                FeatureValue.parseFeatureAsString(fv, probe);
    +                String label = probe.getFeatureAsString();
    +                float value = probe.getValueAsFloat();
    +                doc.put(label, Float.valueOf(value));
    +            }
    +
    +            docs.add(doc);
    +        }
    +    }
    +
    +    private void initParams() {
    +        final List<float[]> p_dz = new ArrayList<float[]>();
    +        final List<Map<String, float[]>> p_dwz = new ArrayList<Map<String, float[]>>();
    +
    +        for (int d = 0; d < _miniBatchSize; d++) {
    +            // init P(z|d)
    +            float[] p_dz_d = ArrayUtils.newRandomFloatArray(_K, _rnd);
    +            ArrayUtils.normalize(p_dz_d);
    +            p_dz.add(p_dz_d);
    +
    +            final Map<String, float[]> p_dwz_d = new HashMap<String, float[]>();
    +            p_dwz.add(p_dwz_d);
    +
    +            for (final String label : _miniBatchDocs.get(d).keySet()) {
    +                // init P(z|d,w)
    +                float[] p_dwz_dw = ArrayUtils.newRandomFloatArray(_K, _rnd);
    +                ArrayUtils.normalize(p_dwz_dw);
    +                p_dwz_d.put(label, p_dwz_dw);
    +
    +                // insert new labels to P(w|z)
    +                if (!_p_zw.containsKey(label)) {
    +                    float[] p_zw_w = ArrayUtils.newRandomFloatArray(_K, _rnd);
    +                    ArrayUtils.normalize(p_zw_w);
    +                    _p_zw.put(label, p_zw_w);
    +                }
    +            }
    +        }
    +
    +        this._p_dz = p_dz;
    +        this._p_dwz = p_dwz;
    +    }
    +
    +    private void eStep(@Nonnegative final int d) {
    +        final Map<String, float[]> p_dwz_d = _p_dwz.get(d);
    +        final float[] p_dz_d = _p_dz.get(d);
    +
    +        // update P(z|d,w) = P(z|d) * P(w|z)
    +        for (final String label : _miniBatchDocs.get(d).keySet()) {
    +            final float[] p_dwz_dw = p_dwz_d.get(label);
    +            final float[] p_zw_w = _p_zw.get(label);
    +            for (int z = 0; z < _K; z++) {
    +                p_dwz_dw[z] = p_dz_d[z] * p_zw_w[z];
    +            }
    +            ArrayUtils.normalize(p_dwz_dw);
    +        }
    +    }
    +
    +    private void mStep(@Nonnegative final int d) {
    +        final Map<String, Float> doc = _miniBatchDocs.get(d);
    +        final Map<String, float[]> p_dwz_d = _p_dwz.get(d);
    +
    +        // update P(z|d) = n(d,w) * P(z|d,w)
    +        final float[] p_dz_d = _p_dz.get(d);
    +        Arrays.fill(p_dz_d, 0.f); // zero-fill w/ keeping pointer to _p_dz.get(d)
    +        for (Map.Entry<String, Float> e : doc.entrySet()) {
    +            final float[] p_dwz_dw = p_dwz_d.get(e.getKey());
    +            final float n = e.getValue().floatValue();
    +            for (int z = 0; z < _K; z++) {
    +                p_dz_d[z] += n * p_dwz_dw[z];
    +            }
    +        }
    +        ArrayUtils.normalize(p_dz_d);
    +
    +        // update P(w|z) = n(d,w) * P(z|d,w) + alpha * P(w|z)
    +        for (int z = 0; z < _K; z++) {
    +            double npSumInDoc_zw_w = 0.d; // sum over the labels in the document
    +            double pSumAll_zw_w = 0.d; // sum over the all existing labels
    +
    +            for (Map.Entry<String, float[]> e : _p_zw.entrySet()) {
    +                final String label = e.getKey();
    +                final float[] p_zw_w = e.getValue();
    +
    +                if (doc.containsKey(label)) {
    +                    p_zw_w[z] *= _alpha; // alpha * P(w|z)
    +                    float np = doc.get(label) * p_dwz_d.get(label)[z]; // n(d,w) * P(z|d,w)
    --- End diff --
    
    Avoid autoboxing `doc.get(label)`.
    
    ```java
    Float label_value = doc.get(label);
    if(label_value == null) {
       ..
    } else {
      p_zw_w[z] *= _alpha; // alpha * P(w|z)
      float np = label_value.floatValue() * p_dwz_d.get(label)[z];
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall issue #71: [HIVEMALL-74] Implement pLSA

Posted by myui <gi...@git.apache.org>.
Github user myui commented on the issue:

    https://github.com/apache/incubator-hivemall/pull/71
  
    @takuti I think probability normalization is missing in `plsa_predict`.
    
    ```
    [{"label":0,"probability":1.0},{"label":1,"probability":1.1246405E-32}]
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall pull request #71: [HIVEMALL-74] Implement pLSA

Posted by myui <gi...@git.apache.org>.
Github user myui commented on a diff in the pull request:

    https://github.com/apache/incubator-hivemall/pull/71#discussion_r113690121
  
    --- Diff: core/src/main/java/hivemall/topicmodel/PLSAPredictUDAF.java ---
    @@ -0,0 +1,477 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *   http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing,
    + * software distributed under the License is distributed on an
    + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    + * KIND, either express or implied.  See the License for the
    + * specific language governing permissions and limitations
    + * under the License.
    + */
    +package hivemall.topicmodel;
    +
    +import hivemall.utils.hadoop.HiveUtils;
    +import hivemall.utils.lang.CommandLineUtils;
    +import hivemall.utils.lang.Primitives;
    +
    +import java.io.PrintWriter;
    +import java.io.StringWriter;
    +import java.util.ArrayList;
    +import java.util.Collections;
    +import java.util.HashMap;
    +import java.util.List;
    +import java.util.Map;
    +import java.util.SortedMap;
    +import java.util.TreeMap;
    +
    +import javax.annotation.Nonnull;
    +
    +import org.apache.commons.cli.CommandLine;
    +import org.apache.commons.cli.HelpFormatter;
    +import org.apache.commons.cli.Options;
    +import org.apache.commons.logging.Log;
    +import org.apache.commons.logging.LogFactory;
    +import org.apache.hadoop.hive.ql.exec.Description;
    +import org.apache.hadoop.hive.ql.exec.UDFArgumentException;
    +import org.apache.hadoop.hive.ql.exec.UDFArgumentLengthException;
    +import org.apache.hadoop.hive.ql.exec.UDFArgumentTypeException;
    +import org.apache.hadoop.hive.ql.metadata.HiveException;
    +import org.apache.hadoop.hive.ql.parse.SemanticException;
    +import org.apache.hadoop.hive.ql.udf.generic.AbstractGenericUDAFResolver;
    +import org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator;
    +import org.apache.hadoop.hive.serde2.io.DoubleWritable;
    +import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
    +import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory;
    +import org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector;
    +import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector;
    +import org.apache.hadoop.hive.serde2.objectinspector.StructField;
    +import org.apache.hadoop.hive.serde2.objectinspector.StandardListObjectInspector;
    +import org.apache.hadoop.hive.serde2.objectinspector.StandardMapObjectInspector;
    +import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;
    +import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils;
    +import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
    +import org.apache.hadoop.io.FloatWritable;
    +import org.apache.hadoop.io.IntWritable;
    +
    +@Description(name = "plsa_predict",
    +        value = "_FUNC_(string word, float value, int label, float prob[, const string options])"
    +                + " - Returns a list which consists of <int label, float prob>")
    +public final class PLSAPredictUDAF extends AbstractGenericUDAFResolver {
    +    private static final Log logger = LogFactory.getLog(PLSAPredictUDAF.class);
    +
    +    @Override
    +    public Evaluator getEvaluator(TypeInfo[] typeInfo) throws SemanticException {
    +        if (typeInfo.length != 4 && typeInfo.length != 5) {
    +            throw new UDFArgumentLengthException(
    +                "Expected argument length is 4 or 5 but given argument length was "
    +                        + typeInfo.length);
    +        }
    +
    +        if (!HiveUtils.isStringTypeInfo(typeInfo[0])) {
    +            throw new UDFArgumentTypeException(0,
    +                "String type is expected for the first argument word: " + typeInfo[0].getTypeName());
    +        }
    +        if (!HiveUtils.isNumberTypeInfo(typeInfo[1])) {
    +            throw new UDFArgumentTypeException(1,
    +                "Number type is expected for the second argument value: "
    +                        + typeInfo[1].getTypeName());
    +        }
    +        if (!HiveUtils.isIntegerTypeInfo(typeInfo[2])) {
    +            throw new UDFArgumentTypeException(2,
    +                "Integer type is expected for the third argument label: "
    +                        + typeInfo[2].getTypeName());
    +        }
    +        if (!HiveUtils.isNumberTypeInfo(typeInfo[3])) {
    +            throw new UDFArgumentTypeException(3,
    +                "Number type is expected for the forth argument prob: " + typeInfo[3].getTypeName());
    +        }
    +
    +        if (typeInfo.length == 5) {
    +            if (!HiveUtils.isStringTypeInfo(typeInfo[4])) {
    +                throw new UDFArgumentTypeException(4,
    +                    "String type is expected for the fifth argument prob: "
    +                            + typeInfo[4].getTypeName());
    +            }
    +        }
    +
    +        return new Evaluator();
    +    }
    +
    +    public static class Evaluator extends GenericUDAFEvaluator {
    +
    +        // input OI
    +        private PrimitiveObjectInspector wordOI;
    +        private PrimitiveObjectInspector valueOI;
    +        private PrimitiveObjectInspector labelOI;
    +        private PrimitiveObjectInspector probOI;
    +
    +        // Hyperparameters
    +        private int topic;
    +        private float alpha;
    +        private double delta;
    +
    +        // merge OI
    +        private StructObjectInspector internalMergeOI;
    +        private StructField wcListField;
    +        private StructField probMapField;
    +        private StructField topicOptionField;
    +        private StructField alphaOptionField;
    +        private StructField deltaOptionField;
    +        private PrimitiveObjectInspector wcListElemOI;
    +        private StandardListObjectInspector wcListOI;
    +        private StandardMapObjectInspector probMapOI;
    +        private PrimitiveObjectInspector probMapKeyOI;
    +        private StandardListObjectInspector probMapValueOI;
    +        private PrimitiveObjectInspector probMapValueElemOI;
    +
    +        public Evaluator() {}
    +
    +        protected Options getOptions() {
    +            Options opts = new Options();
    +            opts.addOption("k", "topic", true, "The number of topics [required]");
    +            opts.addOption("alpha", true, "The hyperparameter for P(w|z) update [default: 0.5]");
    +            opts.addOption("delta", true,
    +                "Check convergence in the expectation step [default: 1E-5]");
    +            return opts;
    +        }
    +
    +        @Nonnull
    +        protected final CommandLine parseOptions(String optionValue) throws UDFArgumentException {
    +            String[] args = optionValue.split("\\s+");
    +            Options opts = getOptions();
    +            opts.addOption("help", false, "Show function help");
    +            CommandLine cl = CommandLineUtils.parseOptions(args, opts);
    +
    +            if (cl.hasOption("help")) {
    +                Description funcDesc = getClass().getAnnotation(Description.class);
    +                final String cmdLineSyntax;
    +                if (funcDesc == null) {
    +                    cmdLineSyntax = getClass().getSimpleName();
    +                } else {
    +                    String funcName = funcDesc.name();
    +                    cmdLineSyntax = funcName == null ? getClass().getSimpleName()
    +                            : funcDesc.value().replace("_FUNC_", funcDesc.name());
    +                }
    +                StringWriter sw = new StringWriter();
    +                sw.write('\n');
    +                PrintWriter pw = new PrintWriter(sw);
    +                HelpFormatter formatter = new HelpFormatter();
    +                formatter.printHelp(pw, HelpFormatter.DEFAULT_WIDTH, cmdLineSyntax, null, opts,
    +                    HelpFormatter.DEFAULT_LEFT_PAD, HelpFormatter.DEFAULT_DESC_PAD, null, true);
    +                pw.flush();
    +                String helpMsg = sw.toString();
    +                throw new UDFArgumentException(helpMsg);
    +            }
    +
    +            return cl;
    +        }
    +
    +        protected CommandLine processOptions(ObjectInspector[] argOIs) throws UDFArgumentException {
    +            CommandLine cl = null;
    +
    +            if (argOIs.length != 5) {
    +                throw new UDFArgumentException("At least 1 option `-topic` MUST be specified");
    +            }
    +
    +            String rawArgs = HiveUtils.getConstString(argOIs[4]);
    +            cl = parseOptions(rawArgs);
    +
    +            this.topic = Primitives.parseInt(cl.getOptionValue("topic"), 0);
    +            if (topic < 1) {
    +                throw new UDFArgumentException(
    +                    "A positive integer MUST be set to an option `-topic`: " + topic);
    +            }
    +
    +            this.alpha = Primitives.parseFloat(cl.getOptionValue("alpha"), 0.5f);
    +            this.delta = Primitives.parseDouble(cl.getOptionValue("delta"), 1E-3d);
    +
    +            return cl;
    +        }
    +
    +        @Override
    +        public ObjectInspector init(Mode mode, ObjectInspector[] parameters) throws HiveException {
    +            assert (parameters.length == 4 || parameters.length == 5);
    +            super.init(mode, parameters);
    +
    +            // initialize input
    +            if (mode == Mode.PARTIAL1 || mode == Mode.COMPLETE) {// from original data
    +                processOptions(parameters);
    +                this.wordOI = HiveUtils.asStringOI(parameters[0]);
    +                this.valueOI = HiveUtils.asDoubleCompatibleOI(parameters[1]);
    +                this.labelOI = HiveUtils.asIntegerOI(parameters[2]);
    +                this.probOI = HiveUtils.asDoubleCompatibleOI(parameters[3]);
    +            } else {// from partial aggregation
    +                StructObjectInspector soi = (StructObjectInspector) parameters[0];
    +                this.internalMergeOI = soi;
    +                this.wcListField = soi.getStructFieldRef("wcList");
    +                this.probMapField = soi.getStructFieldRef("probMap");
    +                this.topicOptionField = soi.getStructFieldRef("topic");
    +                this.alphaOptionField = soi.getStructFieldRef("alpha");
    +                this.deltaOptionField = soi.getStructFieldRef("delta");
    +                this.wcListElemOI = PrimitiveObjectInspectorFactory.javaStringObjectInspector;
    +                this.wcListOI = ObjectInspectorFactory.getStandardListObjectInspector(wcListElemOI);
    +                this.probMapKeyOI = PrimitiveObjectInspectorFactory.javaStringObjectInspector;
    +                this.probMapValueElemOI = PrimitiveObjectInspectorFactory.javaStringObjectInspector;
    +                this.probMapValueOI = ObjectInspectorFactory.getStandardListObjectInspector(probMapValueElemOI);
    +                this.probMapOI = ObjectInspectorFactory.getStandardMapObjectInspector(probMapKeyOI,
    +                    probMapValueOI);
    +            }
    +
    +            // initialize output
    +            final ObjectInspector outputOI;
    +            if (mode == Mode.PARTIAL1 || mode == Mode.PARTIAL2) {// terminatePartial
    +                outputOI = internalMergeOI();
    +            } else {
    +                final ArrayList<String> fieldNames = new ArrayList<String>();
    +                final ArrayList<ObjectInspector> fieldOIs = new ArrayList<ObjectInspector>();
    +                fieldNames.add("label");
    +                fieldOIs.add(PrimitiveObjectInspectorFactory.writableIntObjectInspector);
    +                fieldNames.add("probability");
    +                fieldOIs.add(PrimitiveObjectInspectorFactory.writableFloatObjectInspector);
    +
    +                outputOI = ObjectInspectorFactory.getStandardListObjectInspector(ObjectInspectorFactory.getStandardStructObjectInspector(
    +                    fieldNames, fieldOIs));
    +            }
    +            return outputOI;
    +        }
    +
    +        private static StructObjectInspector internalMergeOI() {
    +            ArrayList<String> fieldNames = new ArrayList<String>();
    +            ArrayList<ObjectInspector> fieldOIs = new ArrayList<ObjectInspector>();
    +
    +            fieldNames.add("wcList");
    +            fieldOIs.add(ObjectInspectorFactory.getStandardListObjectInspector(PrimitiveObjectInspectorFactory.javaStringObjectInspector));
    +
    +            fieldNames.add("probMap");
    +            fieldOIs.add(ObjectInspectorFactory.getStandardMapObjectInspector(
    +                PrimitiveObjectInspectorFactory.javaStringObjectInspector,
    +                ObjectInspectorFactory.getStandardListObjectInspector(PrimitiveObjectInspectorFactory.javaFloatObjectInspector)));
    +
    +            fieldNames.add("topic");
    +            fieldOIs.add(PrimitiveObjectInspectorFactory.writableIntObjectInspector);
    +
    +            fieldNames.add("alpha");
    +            fieldOIs.add(PrimitiveObjectInspectorFactory.writableFloatObjectInspector);
    +
    +            fieldNames.add("delta");
    +            fieldOIs.add(PrimitiveObjectInspectorFactory.writableDoubleObjectInspector);
    +
    +            return ObjectInspectorFactory.getStandardStructObjectInspector(fieldNames, fieldOIs);
    +        }
    +
    +        @Override
    +        public AggregationBuffer getNewAggregationBuffer() throws HiveException {
    +            AggregationBuffer myAggr = new PLSAPredictAggregationBuffer();
    +            reset(myAggr);
    +            return myAggr;
    +        }
    +
    +        @Override
    +        public void reset(@SuppressWarnings("deprecation") AggregationBuffer agg)
    +                throws HiveException {
    +            PLSAPredictAggregationBuffer myAggr = (PLSAPredictAggregationBuffer) agg;
    +            myAggr.reset();
    +            myAggr.setOptions(topic, alpha, delta);
    +        }
    +
    +        @Override
    +        public void iterate(@SuppressWarnings("deprecation") AggregationBuffer agg,
    +                Object[] parameters) throws HiveException {
    +            PLSAPredictAggregationBuffer myAggr = (PLSAPredictAggregationBuffer) agg;
    +
    +            if (parameters[0] == null || parameters[1] == null || parameters[2] == null
    +                    || parameters[3] == null) {
    +                return;
    +            }
    +
    +            String word = PrimitiveObjectInspectorUtils.getString(parameters[0], wordOI);
    +            float value = HiveUtils.getFloat(parameters[1], valueOI);
    +            int label = PrimitiveObjectInspectorUtils.getInt(parameters[2], labelOI);
    +            float prob = HiveUtils.getFloat(parameters[3], probOI);
    +
    +            myAggr.iterate(word, value, label, prob);
    +        }
    +
    +        @Override
    +        public Object terminatePartial(@SuppressWarnings("deprecation") AggregationBuffer agg)
    +                throws HiveException {
    +            PLSAPredictAggregationBuffer myAggr = (PLSAPredictAggregationBuffer) agg;
    +            if (myAggr.wcList.size() == 0) {
    +                return null;
    +            }
    +
    +            Object[] partialResult = new Object[5];
    +            partialResult[0] = myAggr.wcList;
    +            partialResult[1] = myAggr.probMap;
    +            partialResult[2] = new IntWritable(myAggr.topic);
    +            partialResult[3] = new FloatWritable(myAggr.alpha);
    +            partialResult[4] = new DoubleWritable(myAggr.delta);
    +
    +            return partialResult;
    +        }
    +
    +        @Override
    +        public void merge(@SuppressWarnings("deprecation") AggregationBuffer agg, Object partial)
    +                throws HiveException {
    +            if (partial == null) {
    +                return;
    +            }
    +
    +            Object wcListObj = internalMergeOI.getStructFieldData(partial, wcListField);
    +
    +            List<?> wcListRaw = wcListOI.getList(HiveUtils.castLazyBinaryObject(wcListObj));
    +
    +            // fix list elements to Java String objects
    +            int wcListSize = wcListRaw.size();
    +            List<String> wcList = new ArrayList<String>();
    +            for (int i = 0; i < wcListSize; i++) {
    +                wcList.add(PrimitiveObjectInspectorUtils.getString(wcListRaw.get(i), wcListElemOI));
    +            }
    +
    +            Object probMapObj = internalMergeOI.getStructFieldData(partial, probMapField);
    +            Map<?, ?> probMapRaw = probMapOI.getMap(HiveUtils.castLazyBinaryObject(probMapObj));
    +
    +            Map<String, List<Float>> probMap = new HashMap<String, List<Float>>();
    +            for (Map.Entry<?, ?> e : probMapRaw.entrySet()) {
    +                // fix map keys to Java String objects
    +                String word = PrimitiveObjectInspectorUtils.getString(e.getKey(), probMapKeyOI);
    +
    +                Object probMapValueObj = e.getValue();
    +                List<?> probMapValueRaw = probMapValueOI.getList(HiveUtils.castLazyBinaryObject(probMapValueObj));
    +
    +                // fix map values to lists of Java Float objects
    +                int probMapValueSize = probMapValueRaw.size();
    +                List<Float> prob_word = new ArrayList<Float>();
    +                for (int i = 0; i < probMapValueSize; i++) {
    +                    prob_word.add(HiveUtils.getFloat(probMapValueRaw.get(i), probMapValueElemOI));
    +                }
    +
    +                probMap.put(word, prob_word);
    +            }
    +
    +            // restore options from partial result
    +            Object topicObj = internalMergeOI.getStructFieldData(partial, topicOptionField);
    +            this.topic = PrimitiveObjectInspectorFactory.writableIntObjectInspector.get(topicObj);
    +
    +            Object alphaObj = internalMergeOI.getStructFieldData(partial, alphaOptionField);
    +            this.alpha = PrimitiveObjectInspectorFactory.writableFloatObjectInspector.get(alphaObj);
    +
    +            Object deltaObj = internalMergeOI.getStructFieldData(partial, deltaOptionField);
    +            this.delta = PrimitiveObjectInspectorFactory.writableDoubleObjectInspector.get(deltaObj);
    +
    +            PLSAPredictAggregationBuffer myAggr = (PLSAPredictAggregationBuffer) agg;
    +            myAggr.setOptions(topic, alpha, delta);
    +            myAggr.merge(wcList, probMap);
    +        }
    +
    +        @Override
    +        public Object terminate(@SuppressWarnings("deprecation") AggregationBuffer agg)
    +                throws HiveException {
    +            PLSAPredictAggregationBuffer myAggr = (PLSAPredictAggregationBuffer) agg;
    +            float[] topicDistr = myAggr.get();
    +
    +            SortedMap<Float, Integer> sortedDistr = new TreeMap<Float, Integer>(
    +                Collections.reverseOrder());
    +            for (int i = 0; i < topicDistr.length; i++) {
    +                sortedDistr.put(topicDistr[i], i);
    +            }
    +
    +            List<Object[]> result = new ArrayList<Object[]>();
    +            for (Map.Entry<Float, Integer> e : sortedDistr.entrySet()) {
    +                Object[] struct = new Object[2];
    +                struct[0] = new IntWritable(e.getValue()); // label
    +                struct[1] = new FloatWritable(e.getKey()); // probability
    +                result.add(struct);
    +            }
    +            return result;
    +        }
    +
    +    }
    +
    +    public static class PLSAPredictAggregationBuffer extends
    +            GenericUDAFEvaluator.AbstractAggregationBuffer {
    +
    +        private List<String> wcList;
    +        private Map<String, List<Float>> probMap;
    +
    +        private int topic;
    +        private float alpha;
    +        private double delta;
    +
    +        PLSAPredictAggregationBuffer() {
    +            super();
    +        }
    +
    +        void setOptions(int topic, float alpha, double delta) {
    +            this.topic = topic;
    +            this.alpha = alpha;
    +            this.delta = delta;
    +        }
    +
    +        void reset() {
    +            this.wcList = new ArrayList<String>();
    +            this.probMap = new HashMap<String, List<Float>>();
    +        }
    +
    +        void iterate(String word, float value, int label, float prob) {
    +            wcList.add(word + ":" + value);
    +
    +            // for an unforeseen word, initialize its probs w/ -1s
    +            if (!probMap.containsKey(word)) {
    +                List<Float> probEmpty_word = new ArrayList<Float>(Collections.nCopies(topic, -1.f));
    --- End diff --
    
    Q. Is this necessary?
    
    ```java
    List<Float> probEmpty_word = new ArrayList<Float>(Collections.nCopies(topic, -1.f));
    ```
    
    Why not as follows?
    
    ```java
    List<Float> probEmpty_word = Collections.nCopies(topic, -1.f);
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall pull request #71: [HIVEMALL-74] Implement pLSA

Posted by myui <gi...@git.apache.org>.
Github user myui commented on a diff in the pull request:

    https://github.com/apache/incubator-hivemall/pull/71#discussion_r113690440
  
    --- Diff: core/src/main/java/hivemall/topicmodel/PLSAPredictUDAF.java ---
    @@ -0,0 +1,477 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *   http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing,
    + * software distributed under the License is distributed on an
    + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    + * KIND, either express or implied.  See the License for the
    + * specific language governing permissions and limitations
    + * under the License.
    + */
    +package hivemall.topicmodel;
    +
    +import hivemall.utils.hadoop.HiveUtils;
    +import hivemall.utils.lang.CommandLineUtils;
    +import hivemall.utils.lang.Primitives;
    +
    +import java.io.PrintWriter;
    +import java.io.StringWriter;
    +import java.util.ArrayList;
    +import java.util.Collections;
    +import java.util.HashMap;
    +import java.util.List;
    +import java.util.Map;
    +import java.util.SortedMap;
    +import java.util.TreeMap;
    +
    +import javax.annotation.Nonnull;
    +
    +import org.apache.commons.cli.CommandLine;
    +import org.apache.commons.cli.HelpFormatter;
    +import org.apache.commons.cli.Options;
    +import org.apache.commons.logging.Log;
    +import org.apache.commons.logging.LogFactory;
    +import org.apache.hadoop.hive.ql.exec.Description;
    +import org.apache.hadoop.hive.ql.exec.UDFArgumentException;
    +import org.apache.hadoop.hive.ql.exec.UDFArgumentLengthException;
    +import org.apache.hadoop.hive.ql.exec.UDFArgumentTypeException;
    +import org.apache.hadoop.hive.ql.metadata.HiveException;
    +import org.apache.hadoop.hive.ql.parse.SemanticException;
    +import org.apache.hadoop.hive.ql.udf.generic.AbstractGenericUDAFResolver;
    +import org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator;
    +import org.apache.hadoop.hive.serde2.io.DoubleWritable;
    +import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
    +import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory;
    +import org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector;
    +import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector;
    +import org.apache.hadoop.hive.serde2.objectinspector.StructField;
    +import org.apache.hadoop.hive.serde2.objectinspector.StandardListObjectInspector;
    +import org.apache.hadoop.hive.serde2.objectinspector.StandardMapObjectInspector;
    +import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;
    +import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils;
    +import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
    +import org.apache.hadoop.io.FloatWritable;
    +import org.apache.hadoop.io.IntWritable;
    +
    +@Description(name = "plsa_predict",
    +        value = "_FUNC_(string word, float value, int label, float prob[, const string options])"
    +                + " - Returns a list which consists of <int label, float prob>")
    +public final class PLSAPredictUDAF extends AbstractGenericUDAFResolver {
    +    private static final Log logger = LogFactory.getLog(PLSAPredictUDAF.class);
    +
    +    @Override
    +    public Evaluator getEvaluator(TypeInfo[] typeInfo) throws SemanticException {
    +        if (typeInfo.length != 4 && typeInfo.length != 5) {
    +            throw new UDFArgumentLengthException(
    +                "Expected argument length is 4 or 5 but given argument length was "
    +                        + typeInfo.length);
    +        }
    +
    +        if (!HiveUtils.isStringTypeInfo(typeInfo[0])) {
    +            throw new UDFArgumentTypeException(0,
    +                "String type is expected for the first argument word: " + typeInfo[0].getTypeName());
    +        }
    +        if (!HiveUtils.isNumberTypeInfo(typeInfo[1])) {
    +            throw new UDFArgumentTypeException(1,
    +                "Number type is expected for the second argument value: "
    +                        + typeInfo[1].getTypeName());
    +        }
    +        if (!HiveUtils.isIntegerTypeInfo(typeInfo[2])) {
    +            throw new UDFArgumentTypeException(2,
    +                "Integer type is expected for the third argument label: "
    +                        + typeInfo[2].getTypeName());
    +        }
    +        if (!HiveUtils.isNumberTypeInfo(typeInfo[3])) {
    +            throw new UDFArgumentTypeException(3,
    +                "Number type is expected for the forth argument prob: " + typeInfo[3].getTypeName());
    +        }
    +
    +        if (typeInfo.length == 5) {
    +            if (!HiveUtils.isStringTypeInfo(typeInfo[4])) {
    +                throw new UDFArgumentTypeException(4,
    +                    "String type is expected for the fifth argument prob: "
    +                            + typeInfo[4].getTypeName());
    +            }
    +        }
    +
    +        return new Evaluator();
    +    }
    +
    +    public static class Evaluator extends GenericUDAFEvaluator {
    +
    +        // input OI
    +        private PrimitiveObjectInspector wordOI;
    +        private PrimitiveObjectInspector valueOI;
    +        private PrimitiveObjectInspector labelOI;
    +        private PrimitiveObjectInspector probOI;
    +
    +        // Hyperparameters
    +        private int topic;
    +        private float alpha;
    +        private double delta;
    +
    +        // merge OI
    +        private StructObjectInspector internalMergeOI;
    +        private StructField wcListField;
    +        private StructField probMapField;
    +        private StructField topicOptionField;
    +        private StructField alphaOptionField;
    +        private StructField deltaOptionField;
    +        private PrimitiveObjectInspector wcListElemOI;
    +        private StandardListObjectInspector wcListOI;
    +        private StandardMapObjectInspector probMapOI;
    +        private PrimitiveObjectInspector probMapKeyOI;
    +        private StandardListObjectInspector probMapValueOI;
    +        private PrimitiveObjectInspector probMapValueElemOI;
    +
    +        public Evaluator() {}
    +
    +        protected Options getOptions() {
    +            Options opts = new Options();
    +            opts.addOption("k", "topic", true, "The number of topics [required]");
    +            opts.addOption("alpha", true, "The hyperparameter for P(w|z) update [default: 0.5]");
    +            opts.addOption("delta", true,
    +                "Check convergence in the expectation step [default: 1E-5]");
    +            return opts;
    +        }
    +
    +        @Nonnull
    +        protected final CommandLine parseOptions(String optionValue) throws UDFArgumentException {
    +            String[] args = optionValue.split("\\s+");
    +            Options opts = getOptions();
    +            opts.addOption("help", false, "Show function help");
    +            CommandLine cl = CommandLineUtils.parseOptions(args, opts);
    +
    +            if (cl.hasOption("help")) {
    +                Description funcDesc = getClass().getAnnotation(Description.class);
    +                final String cmdLineSyntax;
    +                if (funcDesc == null) {
    +                    cmdLineSyntax = getClass().getSimpleName();
    +                } else {
    +                    String funcName = funcDesc.name();
    +                    cmdLineSyntax = funcName == null ? getClass().getSimpleName()
    +                            : funcDesc.value().replace("_FUNC_", funcDesc.name());
    +                }
    +                StringWriter sw = new StringWriter();
    +                sw.write('\n');
    +                PrintWriter pw = new PrintWriter(sw);
    +                HelpFormatter formatter = new HelpFormatter();
    +                formatter.printHelp(pw, HelpFormatter.DEFAULT_WIDTH, cmdLineSyntax, null, opts,
    +                    HelpFormatter.DEFAULT_LEFT_PAD, HelpFormatter.DEFAULT_DESC_PAD, null, true);
    +                pw.flush();
    +                String helpMsg = sw.toString();
    +                throw new UDFArgumentException(helpMsg);
    +            }
    +
    +            return cl;
    +        }
    +
    +        protected CommandLine processOptions(ObjectInspector[] argOIs) throws UDFArgumentException {
    +            CommandLine cl = null;
    +
    +            if (argOIs.length != 5) {
    +                throw new UDFArgumentException("At least 1 option `-topic` MUST be specified");
    +            }
    +
    +            String rawArgs = HiveUtils.getConstString(argOIs[4]);
    +            cl = parseOptions(rawArgs);
    +
    +            this.topic = Primitives.parseInt(cl.getOptionValue("topic"), 0);
    +            if (topic < 1) {
    +                throw new UDFArgumentException(
    +                    "A positive integer MUST be set to an option `-topic`: " + topic);
    +            }
    +
    +            this.alpha = Primitives.parseFloat(cl.getOptionValue("alpha"), 0.5f);
    +            this.delta = Primitives.parseDouble(cl.getOptionValue("delta"), 1E-3d);
    +
    +            return cl;
    +        }
    +
    +        @Override
    +        public ObjectInspector init(Mode mode, ObjectInspector[] parameters) throws HiveException {
    +            assert (parameters.length == 4 || parameters.length == 5);
    +            super.init(mode, parameters);
    +
    +            // initialize input
    +            if (mode == Mode.PARTIAL1 || mode == Mode.COMPLETE) {// from original data
    +                processOptions(parameters);
    +                this.wordOI = HiveUtils.asStringOI(parameters[0]);
    +                this.valueOI = HiveUtils.asDoubleCompatibleOI(parameters[1]);
    +                this.labelOI = HiveUtils.asIntegerOI(parameters[2]);
    +                this.probOI = HiveUtils.asDoubleCompatibleOI(parameters[3]);
    +            } else {// from partial aggregation
    +                StructObjectInspector soi = (StructObjectInspector) parameters[0];
    +                this.internalMergeOI = soi;
    +                this.wcListField = soi.getStructFieldRef("wcList");
    +                this.probMapField = soi.getStructFieldRef("probMap");
    +                this.topicOptionField = soi.getStructFieldRef("topic");
    +                this.alphaOptionField = soi.getStructFieldRef("alpha");
    +                this.deltaOptionField = soi.getStructFieldRef("delta");
    +                this.wcListElemOI = PrimitiveObjectInspectorFactory.javaStringObjectInspector;
    +                this.wcListOI = ObjectInspectorFactory.getStandardListObjectInspector(wcListElemOI);
    +                this.probMapKeyOI = PrimitiveObjectInspectorFactory.javaStringObjectInspector;
    +                this.probMapValueElemOI = PrimitiveObjectInspectorFactory.javaStringObjectInspector;
    +                this.probMapValueOI = ObjectInspectorFactory.getStandardListObjectInspector(probMapValueElemOI);
    +                this.probMapOI = ObjectInspectorFactory.getStandardMapObjectInspector(probMapKeyOI,
    +                    probMapValueOI);
    +            }
    +
    +            // initialize output
    +            final ObjectInspector outputOI;
    +            if (mode == Mode.PARTIAL1 || mode == Mode.PARTIAL2) {// terminatePartial
    +                outputOI = internalMergeOI();
    +            } else {
    +                final ArrayList<String> fieldNames = new ArrayList<String>();
    +                final ArrayList<ObjectInspector> fieldOIs = new ArrayList<ObjectInspector>();
    +                fieldNames.add("label");
    +                fieldOIs.add(PrimitiveObjectInspectorFactory.writableIntObjectInspector);
    +                fieldNames.add("probability");
    +                fieldOIs.add(PrimitiveObjectInspectorFactory.writableFloatObjectInspector);
    +
    +                outputOI = ObjectInspectorFactory.getStandardListObjectInspector(ObjectInspectorFactory.getStandardStructObjectInspector(
    +                    fieldNames, fieldOIs));
    +            }
    +            return outputOI;
    +        }
    +
    +        private static StructObjectInspector internalMergeOI() {
    +            ArrayList<String> fieldNames = new ArrayList<String>();
    +            ArrayList<ObjectInspector> fieldOIs = new ArrayList<ObjectInspector>();
    +
    +            fieldNames.add("wcList");
    +            fieldOIs.add(ObjectInspectorFactory.getStandardListObjectInspector(PrimitiveObjectInspectorFactory.javaStringObjectInspector));
    +
    +            fieldNames.add("probMap");
    +            fieldOIs.add(ObjectInspectorFactory.getStandardMapObjectInspector(
    +                PrimitiveObjectInspectorFactory.javaStringObjectInspector,
    +                ObjectInspectorFactory.getStandardListObjectInspector(PrimitiveObjectInspectorFactory.javaFloatObjectInspector)));
    +
    +            fieldNames.add("topic");
    +            fieldOIs.add(PrimitiveObjectInspectorFactory.writableIntObjectInspector);
    +
    +            fieldNames.add("alpha");
    +            fieldOIs.add(PrimitiveObjectInspectorFactory.writableFloatObjectInspector);
    +
    +            fieldNames.add("delta");
    +            fieldOIs.add(PrimitiveObjectInspectorFactory.writableDoubleObjectInspector);
    +
    +            return ObjectInspectorFactory.getStandardStructObjectInspector(fieldNames, fieldOIs);
    +        }
    +
    +        @Override
    +        public AggregationBuffer getNewAggregationBuffer() throws HiveException {
    +            AggregationBuffer myAggr = new PLSAPredictAggregationBuffer();
    +            reset(myAggr);
    +            return myAggr;
    +        }
    +
    +        @Override
    +        public void reset(@SuppressWarnings("deprecation") AggregationBuffer agg)
    +                throws HiveException {
    +            PLSAPredictAggregationBuffer myAggr = (PLSAPredictAggregationBuffer) agg;
    +            myAggr.reset();
    +            myAggr.setOptions(topic, alpha, delta);
    +        }
    +
    +        @Override
    +        public void iterate(@SuppressWarnings("deprecation") AggregationBuffer agg,
    +                Object[] parameters) throws HiveException {
    +            PLSAPredictAggregationBuffer myAggr = (PLSAPredictAggregationBuffer) agg;
    +
    +            if (parameters[0] == null || parameters[1] == null || parameters[2] == null
    +                    || parameters[3] == null) {
    +                return;
    +            }
    +
    +            String word = PrimitiveObjectInspectorUtils.getString(parameters[0], wordOI);
    +            float value = HiveUtils.getFloat(parameters[1], valueOI);
    +            int label = PrimitiveObjectInspectorUtils.getInt(parameters[2], labelOI);
    +            float prob = HiveUtils.getFloat(parameters[3], probOI);
    +
    +            myAggr.iterate(word, value, label, prob);
    +        }
    +
    +        @Override
    +        public Object terminatePartial(@SuppressWarnings("deprecation") AggregationBuffer agg)
    +                throws HiveException {
    +            PLSAPredictAggregationBuffer myAggr = (PLSAPredictAggregationBuffer) agg;
    +            if (myAggr.wcList.size() == 0) {
    +                return null;
    +            }
    +
    +            Object[] partialResult = new Object[5];
    +            partialResult[0] = myAggr.wcList;
    +            partialResult[1] = myAggr.probMap;
    +            partialResult[2] = new IntWritable(myAggr.topic);
    +            partialResult[3] = new FloatWritable(myAggr.alpha);
    +            partialResult[4] = new DoubleWritable(myAggr.delta);
    +
    +            return partialResult;
    +        }
    +
    +        @Override
    +        public void merge(@SuppressWarnings("deprecation") AggregationBuffer agg, Object partial)
    +                throws HiveException {
    +            if (partial == null) {
    +                return;
    +            }
    +
    +            Object wcListObj = internalMergeOI.getStructFieldData(partial, wcListField);
    +
    +            List<?> wcListRaw = wcListOI.getList(HiveUtils.castLazyBinaryObject(wcListObj));
    +
    +            // fix list elements to Java String objects
    +            int wcListSize = wcListRaw.size();
    +            List<String> wcList = new ArrayList<String>();
    +            for (int i = 0; i < wcListSize; i++) {
    +                wcList.add(PrimitiveObjectInspectorUtils.getString(wcListRaw.get(i), wcListElemOI));
    +            }
    +
    +            Object probMapObj = internalMergeOI.getStructFieldData(partial, probMapField);
    +            Map<?, ?> probMapRaw = probMapOI.getMap(HiveUtils.castLazyBinaryObject(probMapObj));
    +
    +            Map<String, List<Float>> probMap = new HashMap<String, List<Float>>();
    +            for (Map.Entry<?, ?> e : probMapRaw.entrySet()) {
    +                // fix map keys to Java String objects
    +                String word = PrimitiveObjectInspectorUtils.getString(e.getKey(), probMapKeyOI);
    +
    +                Object probMapValueObj = e.getValue();
    +                List<?> probMapValueRaw = probMapValueOI.getList(HiveUtils.castLazyBinaryObject(probMapValueObj));
    +
    +                // fix map values to lists of Java Float objects
    +                int probMapValueSize = probMapValueRaw.size();
    +                List<Float> prob_word = new ArrayList<Float>();
    +                for (int i = 0; i < probMapValueSize; i++) {
    +                    prob_word.add(HiveUtils.getFloat(probMapValueRaw.get(i), probMapValueElemOI));
    +                }
    +
    +                probMap.put(word, prob_word);
    +            }
    +
    +            // restore options from partial result
    +            Object topicObj = internalMergeOI.getStructFieldData(partial, topicOptionField);
    +            this.topic = PrimitiveObjectInspectorFactory.writableIntObjectInspector.get(topicObj);
    +
    +            Object alphaObj = internalMergeOI.getStructFieldData(partial, alphaOptionField);
    +            this.alpha = PrimitiveObjectInspectorFactory.writableFloatObjectInspector.get(alphaObj);
    +
    +            Object deltaObj = internalMergeOI.getStructFieldData(partial, deltaOptionField);
    +            this.delta = PrimitiveObjectInspectorFactory.writableDoubleObjectInspector.get(deltaObj);
    +
    +            PLSAPredictAggregationBuffer myAggr = (PLSAPredictAggregationBuffer) agg;
    +            myAggr.setOptions(topic, alpha, delta);
    +            myAggr.merge(wcList, probMap);
    +        }
    +
    +        @Override
    +        public Object terminate(@SuppressWarnings("deprecation") AggregationBuffer agg)
    +                throws HiveException {
    +            PLSAPredictAggregationBuffer myAggr = (PLSAPredictAggregationBuffer) agg;
    +            float[] topicDistr = myAggr.get();
    +
    +            SortedMap<Float, Integer> sortedDistr = new TreeMap<Float, Integer>(
    +                Collections.reverseOrder());
    +            for (int i = 0; i < topicDistr.length; i++) {
    +                sortedDistr.put(topicDistr[i], i);
    +            }
    +
    +            List<Object[]> result = new ArrayList<Object[]>();
    +            for (Map.Entry<Float, Integer> e : sortedDistr.entrySet()) {
    +                Object[] struct = new Object[2];
    +                struct[0] = new IntWritable(e.getValue()); // label
    +                struct[1] = new FloatWritable(e.getKey()); // probability
    +                result.add(struct);
    +            }
    +            return result;
    +        }
    +
    +    }
    +
    +    public static class PLSAPredictAggregationBuffer extends
    +            GenericUDAFEvaluator.AbstractAggregationBuffer {
    +
    +        private List<String> wcList;
    +        private Map<String, List<Float>> probMap;
    +
    +        private int topic;
    +        private float alpha;
    +        private double delta;
    +
    +        PLSAPredictAggregationBuffer() {
    +            super();
    +        }
    +
    +        void setOptions(int topic, float alpha, double delta) {
    +            this.topic = topic;
    +            this.alpha = alpha;
    +            this.delta = delta;
    +        }
    +
    +        void reset() {
    +            this.wcList = new ArrayList<String>();
    +            this.probMap = new HashMap<String, List<Float>>();
    +        }
    +
    +        void iterate(String word, float value, int label, float prob) {
    +            wcList.add(word + ":" + value);
    +
    +            // for an unforeseen word, initialize its probs w/ -1s
    +            if (!probMap.containsKey(word)) {
    +                List<Float> probEmpty_word = new ArrayList<Float>(Collections.nCopies(topic, -1.f));
    --- End diff --
    
    ahh... It's immutable.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall pull request #71: [WIP][HIVEMALL-74] Implement pLSA

Posted by myui <gi...@git.apache.org>.
Github user myui commented on a diff in the pull request:

    https://github.com/apache/incubator-hivemall/pull/71#discussion_r112698445
  
    --- Diff: core/src/main/java/hivemall/topicmodel/IncrementalPLSAModel.java ---
    @@ -0,0 +1,300 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *   http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing,
    + * software distributed under the License is distributed on an
    + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    + * KIND, either express or implied.  See the License for the
    + * specific language governing permissions and limitations
    + * under the License.
    + */
    +package hivemall.topicmodel;
    +
    +import hivemall.model.FeatureValue;
    +import hivemall.utils.lang.ArrayUtils;
    +
    +import java.util.Arrays;
    +import java.util.ArrayList;
    +import java.util.Collections;
    +import java.util.HashMap;
    +import java.util.List;
    +import java.util.Map;
    +import java.util.Random;
    +import java.util.SortedMap;
    +import java.util.TreeMap;
    +
    +import javax.annotation.Nonnegative;
    +import javax.annotation.Nonnull;
    +
    +public final class IncrementalPLSAModel {
    +
    +    // ---------------------------------
    +    // HyperParameters
    +
    +    // number of topics
    +    private final int _K;
    +
    +    // control how much P(w|z) update is affected by the last value
    +    private final float _alpha;
    +
    +    // check convergence of P(w|z) for a document
    +    private final double _delta;
    +
    +    // ---------------------------------
    +
    +    // random number generator
    +    @Nonnull
    +    private final Random _rnd;
    +
    +    // optimized in the E step
    +    private List<Map<String, float[]>> _p_dwz; // P(z|d,w) probability of topics for each document-label pair
    +
    +    // optimized in the M step
    +    @Nonnull
    +    private List<float[]> _p_dz; // P(z|d) probability of topics for documents
    +    private Map<String, float[]> _p_zw; // P(w|z) probability of labels for each topic
    +
    +    @Nonnull
    +    private final List<Map<String, Float>> _miniBatchDocs;
    +    private int _miniBatchSize;
    +
    +    public IncrementalPLSAModel(int K, float alpha, double delta) {
    +        this._K = K;
    +        this._alpha = alpha;
    +        this._delta = delta;
    +
    +        this._rnd = new Random(1001);
    +
    +        this._p_zw = new HashMap<String, float[]>();
    +
    +        this._miniBatchDocs = new ArrayList<Map<String, Float>>();
    +    }
    +
    +    public void train(@Nonnull final String[][] miniBatch) {
    +        initMiniBatch(miniBatch, _miniBatchDocs);
    +
    +        this._miniBatchSize = _miniBatchDocs.size();
    +
    +        initParams();
    +
    +        final List<float[]> pPrev_dz = new ArrayList<float[]>();
    +
    +        for (int d = 0; d < _miniBatchSize; d++) {
    +            do {
    +                pPrev_dz.clear();
    +                pPrev_dz.addAll(_p_dz);
    +
    +                // Expectation
    +                eStep(d);
    +
    +                // Maximization
    +                mStep(d);
    +            } while (!isPdzConverged(d, pPrev_dz, _p_dz)); // until get stable value of P(z|d)
    +        }
    +    }
    +
    +    private static void initMiniBatch(@Nonnull final String[][] miniBatch,
    +            @Nonnull final List<Map<String, Float>> docs) {
    +        docs.clear();
    +
    +        final FeatureValue probe = new FeatureValue();
    +
    +        // parse document
    +        for (final String[] e : miniBatch) {
    +            if (e == null || e.length == 0) {
    +                continue;
    +            }
    +
    +            final Map<String, Float> doc = new HashMap<String, Float>();
    +
    +            // parse features
    +            for (String fv : e) {
    +                if (fv == null) {
    +                    continue;
    +                }
    +                FeatureValue.parseFeatureAsString(fv, probe);
    +                String label = probe.getFeatureAsString();
    +                float value = probe.getValueAsFloat();
    +                doc.put(label, Float.valueOf(value));
    +            }
    +
    +            docs.add(doc);
    +        }
    +    }
    +
    +    private void initParams() {
    +        final List<float[]> p_dz = new ArrayList<float[]>();
    +        final List<Map<String, float[]>> p_dwz = new ArrayList<Map<String, float[]>>();
    +
    +        for (int d = 0; d < _miniBatchSize; d++) {
    +            // init P(z|d)
    +            float[] p_dz_d = ArrayUtils.newRandomFloatArray(_K, _rnd);
    +            ArrayUtils.normalize(p_dz_d);
    +            p_dz.add(p_dz_d);
    +
    +            final Map<String, float[]> p_dwz_d = new HashMap<String, float[]>();
    +            p_dwz.add(p_dwz_d);
    +
    +            for (final String label : _miniBatchDocs.get(d).keySet()) {
    +                // init P(z|d,w)
    +                float[] p_dwz_dw = ArrayUtils.newRandomFloatArray(_K, _rnd);
    +                ArrayUtils.normalize(p_dwz_dw);
    --- End diff --
    
    It's better to return a value in `normalize` so that 
    
    ```java
    import static hivemall.utils.MathUtils.normalize;
    
    float[] p_dwz_dw = normalize(ArrayUtils.newRandomFloatArray(_K, _rnd));
    ```
    
    can be applied.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall issue #71: [HIVEMALL-74] Implement pLSA

Posted by myui <gi...@git.apache.org>.
Github user myui commented on the issue:

    https://github.com/apache/incubator-hivemall/pull/71
  
    @takuti LGTM for other codes. Merged! Thanks.
    
    Please send PR `[HIVEMALL-74-2]` if required.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall issue #71: [WIP][HIVEMALL-74] Implement pLSA

Posted by coveralls <gi...@git.apache.org>.
Github user coveralls commented on the issue:

    https://github.com/apache/incubator-hivemall/pull/71
  
    
    [![Coverage Status](https://coveralls.io/builds/11206371/badge)](https://coveralls.io/builds/11206371)
    
    Coverage increased (+0.7%) to 38.728% when pulling **6fe22cab2f70037efdea1cc162d4146de6463841 on takuti:plsa** into **6e24d3a958c8e1eeb724ce980bb0c2d23d99a398 on apache:master**.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall issue #71: [WIP][HIVEMALL-74] Implement pLSA

Posted by myui <gi...@git.apache.org>.
Github user myui commented on the issue:

    https://github.com/apache/incubator-hivemall/pull/71
  
    @takuti please update this PR following the way in LDA.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall issue #71: [HIVEMALL-74] Implement pLSA

Posted by myui <gi...@git.apache.org>.
Github user myui commented on the issue:

    https://github.com/apache/incubator-hivemall/pull/71
  
    @takuti Revised branch is already merged into master.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall issue #71: [WIP][HIVEMALL-74] Implement pLSA

Posted by coveralls <gi...@git.apache.org>.
Github user coveralls commented on the issue:

    https://github.com/apache/incubator-hivemall/pull/71
  
    
    [![Coverage Status](https://coveralls.io/builds/11177470/badge)](https://coveralls.io/builds/11177470)
    
    Coverage increased (+0.7%) to 38.738% when pulling **c1dda3b084e8ad577c15e9e6eb49a92856939269 on takuti:plsa** into **6e24d3a958c8e1eeb724ce980bb0c2d23d99a398 on apache:master**.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall issue #71: [HIVEMALL-74] Implement pLSA

Posted by takuti <gi...@git.apache.org>.
Github user takuti commented on the issue:

    https://github.com/apache/incubator-hivemall/pull/71
  
    Since interfaces are similar to Online LDA, this PR should be handled after #66 is merged.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall pull request #71: [HIVEMALL-74] Implement pLSA

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/incubator-hivemall/pull/71


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall issue #71: [HIVEMALL-74] Implement pLSA

Posted by myui <gi...@git.apache.org>.
Github user myui commented on the issue:

    https://github.com/apache/incubator-hivemall/pull/71
  
    revising at https://github.com/myui/incubator-hivemall/tree/HIVEMALL-74


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall pull request #71: [WIP][HIVEMALL-74] Implement pLSA

Posted by myui <gi...@git.apache.org>.
Github user myui commented on a diff in the pull request:

    https://github.com/apache/incubator-hivemall/pull/71#discussion_r112696086
  
    --- Diff: core/src/main/java/hivemall/topicmodel/IncrementalPLSAModel.java ---
    @@ -0,0 +1,300 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *   http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing,
    + * software distributed under the License is distributed on an
    + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    + * KIND, either express or implied.  See the License for the
    + * specific language governing permissions and limitations
    + * under the License.
    + */
    +package hivemall.topicmodel;
    +
    +import hivemall.model.FeatureValue;
    +import hivemall.utils.lang.ArrayUtils;
    +
    +import java.util.Arrays;
    +import java.util.ArrayList;
    +import java.util.Collections;
    +import java.util.HashMap;
    +import java.util.List;
    +import java.util.Map;
    +import java.util.Random;
    +import java.util.SortedMap;
    +import java.util.TreeMap;
    +
    +import javax.annotation.Nonnegative;
    +import javax.annotation.Nonnull;
    +
    +public final class IncrementalPLSAModel {
    +
    +    // ---------------------------------
    +    // HyperParameters
    +
    +    // number of topics
    +    private final int _K;
    +
    +    // control how much P(w|z) update is affected by the last value
    +    private final float _alpha;
    +
    +    // check convergence of P(w|z) for a document
    +    private final double _delta;
    +
    +    // ---------------------------------
    +
    +    // random number generator
    +    @Nonnull
    +    private final Random _rnd;
    +
    +    // optimized in the E step
    +    private List<Map<String, float[]>> _p_dwz; // P(z|d,w) probability of topics for each document-label pair
    +
    +    // optimized in the M step
    +    @Nonnull
    +    private List<float[]> _p_dz; // P(z|d) probability of topics for documents
    +    private Map<String, float[]> _p_zw; // P(w|z) probability of labels for each topic
    +
    +    @Nonnull
    +    private final List<Map<String, Float>> _miniBatchDocs;
    +    private int _miniBatchSize;
    +
    +    public IncrementalPLSAModel(int K, float alpha, double delta) {
    +        this._K = K;
    +        this._alpha = alpha;
    +        this._delta = delta;
    +
    +        this._rnd = new Random(1001);
    +
    +        this._p_zw = new HashMap<String, float[]>();
    +
    +        this._miniBatchDocs = new ArrayList<Map<String, Float>>();
    +    }
    +
    +    public void train(@Nonnull final String[][] miniBatch) {
    +        initMiniBatch(miniBatch, _miniBatchDocs);
    +
    +        this._miniBatchSize = _miniBatchDocs.size();
    +
    +        initParams();
    +
    +        final List<float[]> pPrev_dz = new ArrayList<float[]>();
    +
    +        for (int d = 0; d < _miniBatchSize; d++) {
    +            do {
    +                pPrev_dz.clear();
    +                pPrev_dz.addAll(_p_dz);
    +
    +                // Expectation
    +                eStep(d);
    +
    +                // Maximization
    +                mStep(d);
    +            } while (!isPdzConverged(d, pPrev_dz, _p_dz)); // until get stable value of P(z|d)
    +        }
    +    }
    +
    +    private static void initMiniBatch(@Nonnull final String[][] miniBatch,
    +            @Nonnull final List<Map<String, Float>> docs) {
    +        docs.clear();
    +
    +        final FeatureValue probe = new FeatureValue();
    +
    +        // parse document
    +        for (final String[] e : miniBatch) {
    +            if (e == null || e.length == 0) {
    +                continue;
    +            }
    +
    +            final Map<String, Float> doc = new HashMap<String, Float>();
    +
    +            // parse features
    +            for (String fv : e) {
    +                if (fv == null) {
    +                    continue;
    +                }
    +                FeatureValue.parseFeatureAsString(fv, probe);
    +                String label = probe.getFeatureAsString();
    +                float value = probe.getValueAsFloat();
    +                doc.put(label, Float.valueOf(value));
    +            }
    +
    +            docs.add(doc);
    +        }
    +    }
    +
    +    private void initParams() {
    +        final List<float[]> p_dz = new ArrayList<float[]>();
    +        final List<Map<String, float[]>> p_dwz = new ArrayList<Map<String, float[]>>();
    +
    +        for (int d = 0; d < _miniBatchSize; d++) {
    +            // init P(z|d)
    +            float[] p_dz_d = ArrayUtils.newRandomFloatArray(_K, _rnd);
    +            ArrayUtils.normalize(p_dz_d);
    --- End diff --
    
    `normalize` should be moved to `MathUtils`, not a normal array operation.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall issue #71: [HIVEMALL-74] Implement pLSA

Posted by takuti <gi...@git.apache.org>.
Github user takuti commented on the issue:

    https://github.com/apache/incubator-hivemall/pull/71
  
    @myui Thanks! Are you planning to push [your revise branch](https://github.com/myui/incubator-hivemall/tree/HIVEMALL-74) to master? Or, can I cherry-pick them and start making additional effort on my own?
    
    For normalization, limitation of floating point numbers?
    
    ```java
    float[] ary = new float[] {0.99999f, 1e-32f};
    System.out.println("Original: " + ary[0] + " " + ary[1]);
    double sum = 0.d;
    for (int i = 0; i < 2; i++) {
        sum += ary[i];
    }
    for (int i = 0; i < 2; i++) {
        ary[i] /= sum;
    }
    System.out.println("Normalized: " + ary[0] + " " + ary[1]);
    ```
    
    ```
    Original: 0.99999 1.0E-32
    Normalized: 1.0 1.00001E-32
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall issue #71: [WIP][HIVEMALL-74] Implement pLSA

Posted by takuti <gi...@git.apache.org>.
Github user takuti commented on the issue:

    https://github.com/apache/incubator-hivemall/pull/71
  
    I've realized that the main difference between following two papers is in **how to initialize P(w|z) for newly observed words**. 
    
    - [Incremental Probabilistic Latent Semantic Analysis for
    Automatic Question Recommendation](https://pdfs.semanticscholar.org/b66e/c7faf2e4888503f7ad1537d284f350fb3e58.pdf) 
    - [Using Incremental PLSI for Threshold-Resilient Online Event Analysis](https://pdfs.semanticscholar.org/a258/b33e285da2e93b59e50311d50ff46045a38b.pdf)
    
    The former (i.e., current implementation) simply initializes w/ random values. Previous P(w|z) could be incorporated by setting a hyper-parameter `alpha` if we wanted (, and `alpha=0` is also possible). 
    
    Meanwhile, the latter requires to undergo certain fold-in procedure to compute "better" P(w|z) by setting a window size. IMO, this approach is too much complex to achieve our goal (=implement pLSA UDTF which repeats EM iterations over the same set of mini-batches).
    
    Thus, I will finalize this PR with current implementation.
    
    Todo:
    
    - [ ] Double-check if the algorithm is implemented correctly
    - [ ] Documentation
      - Difference with LDA
      - Explain the effect of `alpha`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall issue #71: [HIVEMALL-74] Implement pLSA

Posted by myui <gi...@git.apache.org>.
Github user myui commented on the issue:

    https://github.com/apache/incubator-hivemall/pull/71
  
    oops... need to fix CI errors as well.
    https://travis-ci.org/apache/incubator-hivemall/jobs/226400990


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall issue #71: [HIVEMALL-74] Implement pLSA

Posted by coveralls <gi...@git.apache.org>.
Github user coveralls commented on the issue:

    https://github.com/apache/incubator-hivemall/pull/71
  
    
    [![Coverage Status](https://coveralls.io/builds/11066189/badge)](https://coveralls.io/builds/11066189)
    
    Coverage increased (+0.7%) to 37.545% when pulling **82597a2ca4ffefefb301b89aba57e24960fb0436 on takuti:plsa** into **c53b9ff9b23755c7d211390ea97bc911a80674bf on apache:master**.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall issue #71: [HIVEMALL-74] Implement pLSA

Posted by myui <gi...@git.apache.org>.
Github user myui commented on the issue:

    https://github.com/apache/incubator-hivemall/pull/71
  
    Sure \U0001f44d 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall pull request #71: [WIP][HIVEMALL-74] Implement pLSA

Posted by myui <gi...@git.apache.org>.
Github user myui commented on a diff in the pull request:

    https://github.com/apache/incubator-hivemall/pull/71#discussion_r112865163
  
    --- Diff: core/src/main/java/hivemall/utils/math/MathUtils.java ---
    @@ -401,4 +401,13 @@ public static double logsumexp(@Nonnull final double[] arr, final double max) {
             return Math.log(logsumexp) + max;
         }
     
    +    @Nonnull
    +    public static float[] normalize(@Nonnull final float[] arr) {
    --- End diff --
    
    `sum` is l1norm? Then, remove this to `l1normalize`.
    
    `l1norm` should be the sum of the **absolute** values of the columns.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall issue #71: [HIVEMALL-74] Implement pLSA

Posted by takuti <gi...@git.apache.org>.
Github user takuti commented on the issue:

    https://github.com/apache/incubator-hivemall/pull/71
  
    Ack! Checking your modifications.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall issue #71: [WIP][HIVEMALL-74] Implement pLSA

Posted by coveralls <gi...@git.apache.org>.
Github user coveralls commented on the issue:

    https://github.com/apache/incubator-hivemall/pull/71
  
    
    [![Coverage Status](https://coveralls.io/builds/11206599/badge)](https://coveralls.io/builds/11206599)
    
    Coverage increased (+0.7%) to 38.721% when pulling **c3709e9098c188f966d6f1d5609b7ad3c3649336 on takuti:plsa** into **6e24d3a958c8e1eeb724ce980bb0c2d23d99a398 on apache:master**.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall issue #71: [HIVEMALL-74] Implement pLSA

Posted by takuti <gi...@git.apache.org>.
Github user takuti commented on the issue:

    https://github.com/apache/incubator-hivemall/pull/71
  
    ^ Fixed and gonna be sent as a new pr soon


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall issue #71: [HIVEMALL-74] Implement pLSA

Posted by coveralls <gi...@git.apache.org>.
Github user coveralls commented on the issue:

    https://github.com/apache/incubator-hivemall/pull/71
  
    
    [![Coverage Status](https://coveralls.io/builds/11225273/badge)](https://coveralls.io/builds/11225273)
    
    Coverage increased (+0.7%) to 38.68% when pulling **94a8fe70088161be6b94c31476d52cd906d1398c on takuti:plsa** into **0c4798ebaeb2e4e4e388e38501396c4823336edf on apache:master**.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall issue #71: [WIP][HIVEMALL-74] Implement pLSA

Posted by coveralls <gi...@git.apache.org>.
Github user coveralls commented on the issue:

    https://github.com/apache/incubator-hivemall/pull/71
  
    
    [![Coverage Status](https://coveralls.io/builds/11179365/badge)](https://coveralls.io/builds/11179365)
    
    Coverage increased (+0.7%) to 38.733% when pulling **37dfb9885577d77f1e88e52c82a08d44f4429542 on takuti:plsa** into **6e24d3a958c8e1eeb724ce980bb0c2d23d99a398 on apache:master**.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall issue #71: [HIVEMALL-74] Implement pLSA

Posted by takuti <gi...@git.apache.org>.
Github user takuti commented on the issue:

    https://github.com/apache/incubator-hivemall/pull/71
  
    @myui almost finished. could you review current code and docs?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall issue #71: [WIP][HIVEMALL-74] Implement pLSA

Posted by takuti <gi...@git.apache.org>.
Github user takuti commented on the issue:

    https://github.com/apache/incubator-hivemall/pull/71
  
    **Observations**:
    
    Current implementation is based on: [Incremental Probabilistic Latent Semantic Analysis for
    Automatic Question Recommendation](https://pdfs.semanticscholar.org/b66e/c7faf2e4888503f7ad1537d284f350fb3e58.pdf), and this technique requires us to choose `alpha` as a hyper-parameter used in Eq. (22). However, choosing an appropriate value of `alpha` is difficult for the following reasons:
    
    - In the original formulation Eq. (22), `\sum_w P(w|z)` does not seem to be 1.
    - Depending on data and `alpha`, perplexity highly fluctuates (i.e., not monotonically decreasing), and thus checking convergence based on perplexity is difficult.
    
    One possible direction is to consider alternative incremental PLSA techniques. In particular, [Using Incremental PLSI for Threshold-Resilient Online Event Analysis](https://pdfs.semanticscholar.org/a258/b33e285da2e93b59e50311d50ff46045a38b.pdf) is one of the most cited incremental PLSA papers.
    
    I still keep this PR as "WIP" for the reasons that I mentioned above. Will try to implement the alternative technique and report results in a week...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall issue #71: [WIP][HIVEMALL-74] Implement pLSA

Posted by coveralls <gi...@git.apache.org>.
Github user coveralls commented on the issue:

    https://github.com/apache/incubator-hivemall/pull/71
  
    
    [![Coverage Status](https://coveralls.io/builds/11205962/badge)](https://coveralls.io/builds/11205962)
    
    Coverage increased (+0.7%) to 38.726% when pulling **7616ccb96eeb1603e8bcd1bf221087d108e7a98f on takuti:plsa** into **6e24d3a958c8e1eeb724ce980bb0c2d23d99a398 on apache:master**.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall issue #71: [WIP][HIVEMALL-74] Implement pLSA

Posted by coveralls <gi...@git.apache.org>.
Github user coveralls commented on the issue:

    https://github.com/apache/incubator-hivemall/pull/71
  
    
    [![Coverage Status](https://coveralls.io/builds/11206718/badge)](https://coveralls.io/builds/11206718)
    
    Coverage increased (+1.0%) to 39.058% when pulling **c3709e9098c188f966d6f1d5609b7ad3c3649336 on takuti:plsa** into **6e24d3a958c8e1eeb724ce980bb0c2d23d99a398 on apache:master**.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall issue #71: [WIP][HIVEMALL-74] Implement pLSA

Posted by coveralls <gi...@git.apache.org>.
Github user coveralls commented on the issue:

    https://github.com/apache/incubator-hivemall/pull/71
  
    
    [![Coverage Status](https://coveralls.io/builds/11206307/badge)](https://coveralls.io/builds/11206307)
    
    Coverage increased (+0.7%) to 38.723% when pulling **6fe22cab2f70037efdea1cc162d4146de6463841 on takuti:plsa** into **6e24d3a958c8e1eeb724ce980bb0c2d23d99a398 on apache:master**.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall issue #71: [WIP][HIVEMALL-74] Implement pLSA

Posted by coveralls <gi...@git.apache.org>.
Github user coveralls commented on the issue:

    https://github.com/apache/incubator-hivemall/pull/71
  
    
    [![Coverage Status](https://coveralls.io/builds/11179515/badge)](https://coveralls.io/builds/11179515)
    
    Coverage increased (+0.7%) to 38.741% when pulling **37dfb9885577d77f1e88e52c82a08d44f4429542 on takuti:plsa** into **6e24d3a958c8e1eeb724ce980bb0c2d23d99a398 on apache:master**.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall issue #71: [WIP][HIVEMALL-74] Implement pLSA

Posted by coveralls <gi...@git.apache.org>.
Github user coveralls commented on the issue:

    https://github.com/apache/incubator-hivemall/pull/71
  
    
    [![Coverage Status](https://coveralls.io/builds/11179095/badge)](https://coveralls.io/builds/11179095)
    
    Coverage increased (+1.007%) to 39.071% when pulling **68d80c6f56ca26bab6c6c1a2843bd9da3b6df366 on takuti:plsa** into **6e24d3a958c8e1eeb724ce980bb0c2d23d99a398 on apache:master**.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---