You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hivemall.apache.org by helenahm <gi...@git.apache.org> on 2017/06/02 05:37:33 UTC

[GitHub] incubator-hivemall pull request #82: Encoding related bug in LDA.

GitHub user helenahm opened a pull request:

    https://github.com/apache/incubator-hivemall/pull/82

    Encoding related bug in LDA.

    What changes were proposed in this pull request?
    
    I have found a major bug, without fixing it people will not be able to use LDA, or perhaps other algorithms too.
    
    I spotted the bug and made a small fix, I would like you to fix the rest.
    
    What type of PR is it?
    
    [Bug Fix]
    
    What is the Jira issue?
    
    ???
    
    How was this patch tested?
    
    I intended to use LDA for my data, on EMR as usually, LDA failed to process my text. So I checked your test, and when added code to add my data to your test instead of your two lines. When it run successfully, I realized that the test may be faulty, and indeed, I think close() is called in real life, but not in the test. Same errors as on EMR showed up.
    
    I found the lines in features that coursed the errors:
     na‹ve:1
     xž:1
    
    Why I do not know, but that means that I have to pre-process data prior to testing LDA further, plus I start doubting whether it will work for other languages.
    
    EMR error messages for different options for memory and number of reduces are below. Same source, same reason.
    
    Diagnostic Messages for this Task:
     Error: java.lang.RuntimeException: Hive Runtime Error while closing operators: Exception caused in the iterative training
     at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:287)
     at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:454)
     at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:393)
     at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
     at java.security.AccessController.doPrivileged(Native Method)
     at javax.security.auth.Subject.doAs(Subject.java:422)
     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
     at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
     Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Exception caused in the iterative training
     at hivemall.topicmodel.LDAUDTF.runIterativeTraining(LDAUDTF.java:511)
     at hivemall.topicmodel.LDAUDTF.close(LDAUDTF.java:309)
     at org.apache.hadoop.hive.ql.exec.UDTFOperator.closeOp(UDTFOperator.java:152)
     at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:683)
     at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
     at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
     at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:279)
     ... 7 more
     Caused by: java.lang.OutOfMemoryError: Java heap space
     at hivemall.topicmodel.LDAUDTF.runIterativeTraining(LDAUDTF.java:352)
     ... 13 more
    
    Error: java.lang.RuntimeException: Hive Runtime Error while closing operators: Exception caused in the iterative training
     at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:287)
     at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:454)
     at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:393)
     at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
     at java.security.AccessController.doPrivileged(Native Method)
     at javax.security.auth.Subject.doAs(Subject.java:422)
     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
     at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
     Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Exception caused in the iterative training
     at hivemall.topicmodel.LDAUDTF.runIterativeTraining(LDAUDTF.java:511)
     at hivemall.topicmodel.LDAUDTF.close(LDAUDTF.java:309)
     at org.apache.hadoop.hive.ql.exec.UDTFOperator.closeOp(UDTFOperator.java:152)
     at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:683)
     at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
     at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
     at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:279)
     ... 7 more
     Caused by: java.nio.BufferUnderflowException
     at java.nio.DirectByteBuffer.get(DirectByteBuffer.java:271)
     at java.nio.ByteBuffer.get(ByteBuffer.java:715)
     at hivemall.topicmodel.LDAUDTF.runIterativeTraining(LDAUDTF.java:356)
     ... 13 more
    
    Error: java.lang.RuntimeException: Hive Runtime Error while closing operators: Exception caused in the iterative training
     at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:287)
     at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:454)
     at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:393)
     at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
     at java.security.AccessController.doPrivileged(Native Method)
     at javax.security.auth.Subject.doAs(Subject.java:422)
     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
     at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
     Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Exception caused in the iterative training
     at hivemall.topicmodel.LDAUDTF.runIterativeTraining(LDAUDTF.java:511)
     at hivemall.topicmodel.LDAUDTF.close(LDAUDTF.java:309)
     at org.apache.hadoop.hive.ql.exec.UDTFOperator.closeOp(UDTFOperator.java:152)
     at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:683)
     at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
     at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
     at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:279)
     ... 7 more
     Caused by: java.lang.NegativeArraySizeException
     at hivemall.topicmodel.LDAUDTF.runIterativeTraining(LDAUDTF.java:352)
     ... 13 more
    
    How to use this feature?
    
    When fixed people will be able to run LDA.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/helenahm/incubator-hivemall master

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-hivemall/pull/82.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #82
    
----
commit 45a656aa7278066ce3fc36fcd81fb1eca11f1079
Author: helenahm <he...@users.noreply.github.com>
Date:   2017-06-02T05:10:13Z

    Update LDAUDTFTest.java

commit fef9c1ce719d3924a28cc90d71d40728dc5c7563
Author: helenahm <he...@users.noreply.github.com>
Date:   2017-06-02T05:22:54Z

    Merge pull request #1 from helenahm/helenahm-patch-1
    
    Update LDAUDTFTest.java

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall issue #82: Encoding related bug in LDA.

Posted by takuti <gi...@git.apache.org>.
Github user takuti commented on the issue:

    https://github.com/apache/incubator-hivemall/pull/82
  
    @helenahm No worries, and I have fixed both `train_lda()` and `tokenize()` in #83. 
    
    If the PR is merged:
    
    - you can use multi-byte strings e.g. `xž` and `na‹ve`
    - NULL document is allowed (and has no negative effect to the later topic modeling)
    
    Could you check if the update works fine for you? The branch is `fix-topicmodel` in [takuti/incubator-hivemall](https://github.com/takuti/incubator-hivemall/tree/fix-topicmodel); you can build the fixed version by cloning the repository and checking out to the branch.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall issue #82: Encoding related bug in LDA.

Posted by myui <gi...@git.apache.org>.
Github user myui commented on the issue:

    https://github.com/apache/incubator-hivemall/pull/82
  
    @takuti Root cause is NPE. The RuntimeException is caught and re-thrown in https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java#L1039
    
    The exception message is generated in Hive runtime. So, we cannot change it. 
    Root cause is that UDFs should not throw RuntimeException.
    
    ```
    Caused by: java.lang.NullPointerException
    at hivemall.tools.text.TokenizeUDF.evaluate(TokenizeUDF.java:42)
    ... 26 more
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall issue #82: Encoding related bug in LDA.

Posted by takuti <gi...@git.apache.org>.
Github user takuti commented on the issue:

    https://github.com/apache/incubator-hivemall/pull/82
  
    @helenahm Good catch, and thank you for reporting. You are right; this bug is definitely caused by the encoding-related issue. I'll take care of fixing the bug.
    
    Basically, the critical reason is in:
    
    ```java
    buf.putInt(wc.length());
    buf.put(wc.getBytes());
    ```
    
    These lines are totally wrong and need be fixed to:
    
    ```java
    buf.putInt(wc.getBytes().length);
    buf.put(wc.getBytes());
    ```
    
    At the same time, as @myui mentioned above, using `bytes` things should be avoided.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall issue #82: Encoding related bug in LDA.

Posted by myui <gi...@git.apache.org>.
Github user myui commented on the issue:

    https://github.com/apache/incubator-hivemall/pull/82
  
    @helenahm @takuti close this PR as resolved.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall issue #82: Encoding related bug in LDA.

Posted by takuti <gi...@git.apache.org>.
Github user takuti commented on the issue:

    https://github.com/apache/incubator-hivemall/pull/82
  
    @myui IMO, in that case, throwing well-described `HiveException` looks more reasonable because input and output type of `tokenize` UDF is dfifferent. Or, returning empty array is also acceptable.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall issue #82: Encoding related bug in LDA.

Posted by takuti <gi...@git.apache.org>.
Github user takuti commented on the issue:

    https://github.com/apache/incubator-hivemall/pull/82
  
    @helenahm Cool! Thank you for your cooperation. Could you close this PR if everything is okay? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall issue #82: Encoding related bug in LDA.

Posted by helenahm <gi...@git.apache.org>.
Github user helenahm commented on the issue:

    https://github.com/apache/incubator-hivemall/pull/82
  
    i am pretty sure that in my case the document is empty, and not null, first of all because Ln != "" saved that query, not Ln is not null; also it is a table obtained as an external table, so if anything nulls would look like "null", that is a non empty string.
    
    It is up to you, but as for the user of hivemall, LDA is at the end of a much longer pipeline, and nulls and empty strings will not be uncommon, your users may be upset to have such an error. 
    
    Also, as a user, after building a model and running predict, I would feel good if null and empty strings would get a topic, -1, or "none", or "Na". I do not know how it works now, but I think processing nulls and empty strings would make people feel better about the hivemall library.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall pull request #82: Encoding related bug in LDA.

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/incubator-hivemall/pull/82


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall issue #82: Encoding related bug in LDA.

Posted by myui <gi...@git.apache.org>.
Github user myui commented on the issue:

    https://github.com/apache/incubator-hivemall/pull/82
  
    @takuti It's another issue. `tokenize` should return null where input is null.
    https://github.com/apache/incubator-hivemall/blob/master/core/src/main/java/hivemall/tools/text/TokenizeUDF.java#L42


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall issue #82: Encoding related bug in LDA.

Posted by myui <gi...@git.apache.org>.
Github user myui commented on the issue:

    https://github.com/apache/incubator-hivemall/pull/82
  
    @helenahm thank you for the bug report.
    
    @takuti 
    
    `String.getBytes()` and `new String(byte[])` should be avoided to avoid encoding/decoding problem (UTF16 to . See the JDK implementation. 
    
    Better to replace the current code using `String.charAt(int)` and `new String(char[])`. See `hivemall.utils.IOUtils`.
    
    ```
            final int len = s.length();
            out.writeInt(len);
            for (int i = 0; i < len; i++) {
                int v = s.charAt(i);
                out.writeChar(v);
            }
    
            final char[] ch = new char[len];
            for (int i = 0; i < len; i++) {
                ch[i] = in.readChar();
            }
            return new String(ch);
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall issue #82: Encoding related bug in LDA.

Posted by helenahm <gi...@git.apache.org>.
Github user helenahm commented on the issue:

    https://github.com/apache/incubator-hivemall/pull/82
  
    Also, in my query (below), I had to add ln != null, to not submit empty text, otherwise an error (below below) is thrown
    
    create temporary function train_lda as 'hivemall.topicmodel.LDAUDTF';
    with word_counts as (
      select
        cast(rownum as int) as docid,
        feature(word, count(word)) as word_count
      from data_rownum t1 LATERAL VIEW explode(tokenize(ln, true)) t2 as word
      where
        not is_stopword(word) and ln != ""  
        group by
        rownum, word
    )
    select
      label, word, avg(lambda) as lambda
    from (
      select
        train_lda(feature, "-topics 2 -iter 20") as (label, word, lambda)
      from (
        select docid, collect_set(word_count) as feature
        from word_counts
        group by docid
      ) t1
    ) t2
    group by label, word
    order by lambda desc
    ;
    
    
    Diagnostic Messages for this Task:
    Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"rownum":"179","ln":null}
            at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:169)
            at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
            at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:455)
            at org.apache.hadoop.mapred.MapTask.run(MapTask.java:344)
            at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
            at java.security.AccessController.doPrivileged(Native Method)
            at javax.security.auth.Subject.doAs(Subject.java:422)
            at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
            at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
    Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"rownum":"179","ln":null}
            at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:499)
            at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:160)
            ... 8 more
    Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to execute method public java.util.List hivemall.tools.text.TokenizeUDF.evaluate(org.apache.hadoop.io.Text,boolean)  on object hivemall.tools.text.TokenizeUDF@6240651f of class hivemall.tools.text.TokenizeUDF with arguments {null, true:java.lang.Boolean} of size 2
            at org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:991)
            at org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.evaluate(GenericUDFBridge.java:194)
            at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:186)
            at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
            at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65)
            at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:88)
            at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
            at org.apache.hadoop.hive.ql.exec.LateralViewForwardOperator.process(LateralViewForwardOperator.java:39)
            at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
            at org.apache.hadoop.hive.ql.exec.FilterOperator.process(FilterOperator.java:126)
            at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
            at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
            at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:149)
            at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:489)
            ... 9 more
    Caused by: java.lang.reflect.InvocationTargetException
            at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:498)
            at org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:967)
            ... 22 more
    Caused by: java.lang.NullPointerException
            at hivemall.tools.text.TokenizeUDF.evaluate(TokenizeUDF.java:42)
            ... 26 more



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall issue #82: Encoding related bug in LDA.

Posted by helenahm <gi...@git.apache.org>.
Github user helenahm commented on the issue:

    https://github.com/apache/incubator-hivemall/pull/82
  
    You are right though, my bad, the error comes from null rows only. Sorry.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall issue #82: Encoding related bug in LDA.

Posted by helenahm <gi...@git.apache.org>.
Github user helenahm commented on the issue:

    https://github.com/apache/incubator-hivemall/pull/82
  
    Thank you for the library you are developing. Please, let me know when the code is ready to be pulled to build a new hivemall-core jar and use it for LDA.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall issue #82: Encoding related bug in LDA.

Posted by helenahm <gi...@git.apache.org>.
Github user helenahm commented on the issue:

    https://github.com/apache/incubator-hivemall/pull/82
  
    Yes, thank you very very much, the compiled code works without any errors! :-) 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall issue #82: Encoding related bug in LDA.

Posted by takuti <gi...@git.apache.org>.
Github user takuti commented on the issue:

    https://github.com/apache/incubator-hivemall/pull/82
  
    @myui ah, I see.
    
    @helenahm I'm addressing the `tokenize(null)` issue, plz wait :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall issue #82: Encoding related bug in LDA.

Posted by takuti <gi...@git.apache.org>.
Github user takuti commented on the issue:

    https://github.com/apache/incubator-hivemall/pull/82
  
    @helenahm If I understand correctly, it sounds natural. Make sure the difference between "**null**" and "**empty**" document. **Null** document must not exist in your document collection (or you manually need to add the `AND ln != null` clause for workaround as you already tried; i.e. carefully undergo preprocessing step).
    
    In fact, since `doc#3` is **null**, the following query throws an exception which you wrote:
    
    ```sql
    with docs as (
      select docid, doc
      from (
        select 1 as docid, "Fruits and vegetables are healthy na‹ve." as doc
        union all
        select 2 as docid, "I like apples, oranges, and avocados. I do not like the flu or colds." as doc
        union all
        select 3 as docid, null as doc
      ) t1
    ),
    word_counts as (
      select
        docid,
        feature(word, count(word)) as f
      from docs t1 LATERAL VIEW explode(tokenize(doc, true)) t2 as word
      where
        not is_stopword(word)
      group by
        docid, word
    )
    select label, word, avg(lambda) as lambda
    from (
      select
        -- train_plsa(feature, "-topics 2 -eps 0.00001 -iter 2048 -alpha 0.01") as (label, word, lambda)
        train_lda(feature, "-topics 2 -iter 20") as (label, word, lambda)
      from (
        select docid, collect_set(f) as feature
        from word_counts
        group by docid
        -- order by docid
      ) t1
    ) t2
    group by label, word
    order by lambda desc
    ;
    ```
    
    However, if the document is just **empty**, it works:
    
    ```sql
    with docs as (
      select docid, doc
      from (
        select 1 as docid, "Fruits and vegetables are healthy na‹ve." as doc
        union all
        select 2 as docid, "I like apples, oranges, and avocados. I do not like the flu or colds." as doc
        union all
        select 3 as docid, "" as doc
      ) t1
    ),
    ...
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---