You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hivemall.apache.org by "Makoto Yui (JIRA)" <ji...@apache.org> on 2019/05/28 04:29:00 UTC
[jira] [Commented] (HIVEMALL-255) Error:
java.lang.RuntimeException: Hive Runtime Error while closing operators
[ https://issues.apache.org/jira/browse/HIVEMALL-255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16849325#comment-16849325 ]
Makoto Yui commented on HIVEMALL-255:
-------------------------------------
[~jshaw6] sorry to be late.
What's hive version?
{code:java}
select version();{code}
The query worked fine with my environment (hive 2.3.0).
{code:java}
array("price"), – quantitative feature names{code}
is expected to be const string array but it seems it is not provided as const value.
> Error: java.lang.RuntimeException: Hive Runtime Error while closing operators
> -----------------------------------------------------------------------------
>
> Key: HIVEMALL-255
> URL: https://issues.apache.org/jira/browse/HIVEMALL-255
> Project: Hivemall
> Issue Type: Bug
> Affects Versions: 0.5.2
> Environment: Ubuntu 16.04 4.4.0-146-generic
> Hadoop 3.1.2
> Hivemall 0.52
> Reporter: Jacob Shaw
> Assignee: Makoto Yui
> Priority: Minor
> Labels: Hivemall, Regression, Tutorial
>
> Hello when attempting to use hive-malls regression tool kit I run into errors when attempting to build the feature representation.
>
> I've been following this guide [https://hivemall.incubator.apache.org/userguide/supervised_learning/tutorial.html] and have been attempting to reproduce it. I've used the code provided however I'm running into issues when running.
>
> My issue seems to be with this part of the guide
> {color:#569cd6}create table if not exists purchase_history{color} {color:#c586c0}as{color}
> {color:#c586c0}select{color} {color:#b5cea8}1{color} {color:#c586c0}as{color}{color:#d4d4d4} id, {color}{color:#ce9178}"Saturday"{color} {color:#c586c0}as{color}{color:#d4d4d4} day_of_week, {color}{color:#ce9178}"male"{color} {color:#c586c0}as{color}{color:#d4d4d4} gender, {color}{color:#b5cea8}600{color} {color:#c586c0}as{color}{color:#d4d4d4} price, {color}{color:#ce9178}"book"{color} {color:#c586c0}as{color}{color:#d4d4d4} category, {color}{color:#b5cea8}1{color} {color:#c586c0}as{color}{color:#d4d4d4} label{color}
> {color:#c586c0}union{color} {color:#c586c0}all{color}
> {color:#c586c0}select{color} {color:#b5cea8}2{color} {color:#c586c0}as{color}{color:#d4d4d4} id, {color}{color:#ce9178}"Friday"{color} {color:#c586c0}as{color}{color:#d4d4d4} day_of_week, {color}{color:#ce9178}"female"{color} {color:#c586c0}as{color}{color:#d4d4d4} gender, {color}{color:#b5cea8}4800{color} {color:#c586c0}as{color}{color:#d4d4d4} price, {color}{color:#ce9178}"sports"{color} {color:#c586c0}as{color}{color:#d4d4d4} category, {color}{color:#b5cea8}0{color} {color:#c586c0}as{color}{color:#d4d4d4} label{color}
> {color:#c586c0}union{color} {color:#c586c0}all{color}
> {color:#c586c0}select{color} {color:#b5cea8}3{color} {color:#c586c0}as{color}{color:#d4d4d4} id, {color}{color:#ce9178}"Friday"{color} {color:#c586c0}as{color}{color:#d4d4d4} day_of_week, {color}{color:#ce9178}"other"{color} {color:#c586c0}as{color}{color:#d4d4d4} gender, {color}{color:#b5cea8}18000{color} {color:#c586c0}as{color}{color:#d4d4d4} price, {color}{color:#ce9178}"entertainment"{color} {color:#c586c0}as{color}{color:#d4d4d4} category, {color}{color:#b5cea8}0{color} {color:#c586c0}as{color}{color:#d4d4d4} label{color}
> {color:#c586c0}union{color} {color:#c586c0}all{color}
> {color:#c586c0}select{color} {color:#b5cea8}4{color} {color:#c586c0}as{color}{color:#d4d4d4} id, {color}{color:#ce9178}"Thursday"{color} {color:#c586c0}as{color}{color:#d4d4d4} day_of_week, {color}{color:#ce9178}"male"{color} {color:#c586c0}as{color}{color:#d4d4d4} gender, {color}{color:#b5cea8}200{color} {color:#c586c0}as{color}{color:#d4d4d4} price, {color}{color:#ce9178}"food"{color} {color:#c586c0}as{color}{color:#d4d4d4} category, {color}{color:#b5cea8}0{color} {color:#c586c0}as{color}{color:#d4d4d4} label{color}
> {color:#c586c0}union{color} {color:#c586c0}all{color}
> {color:#c586c0}select{color} {color:#b5cea8}5{color} {color:#c586c0}as{color}{color:#d4d4d4} id, {color}{color:#ce9178}"Wednesday"{color} {color:#c586c0}as{color}{color:#d4d4d4} day_of_week, {color}{color:#ce9178}"female"{color} {color:#c586c0}as{color}{color:#d4d4d4} gender, {color}{color:#b5cea8}1000{color} {color:#c586c0}as{color}{color:#d4d4d4} price, {color}{color:#ce9178}"electronics"{color} {color:#c586c0}as{color}{color:#d4d4d4} category, {color}{color:#b5cea8}1{color} {color:#c586c0}as{color}{color:#d4d4d4} label{color}
> {color:#d4d4d4};{color}
> {color:#569cd6}create table if not exists training{color} {color:#c586c0}as{color}
> {color:#c586c0}select{color}
> {color:#d4d4d4} id,{color}
> {color:#d4d4d4} array_concat( {color}{color:#6a9955}-- concatenate two arrays of quantitative and categorical features into single array{color}
> {color:#d4d4d4} quantitative_features({color}
> {color:#4ec9b0}array{color}{color:#d4d4d4}({color}{color:#ce9178}"price"{color}{color:#d4d4d4}), {color}{color:#6a9955}-- quantitative feature names{color}
> {color:#d4d4d4} price {color}{color:#6a9955}-- corresponding column names{color}
> {color:#d4d4d4} ),{color}
> {color:#d4d4d4} categorical_features({color}
> {color:#4ec9b0}array{color}{color:#d4d4d4}({color}{color:#ce9178}"day of week"{color}{color:#d4d4d4}, {color}{color:#ce9178}"gender"{color}{color:#d4d4d4}, {color}{color:#ce9178}"category"{color}{color:#d4d4d4}), {color}{color:#6a9955}-- categorical feature names{color}
> {color:#d4d4d4} day_of_week, gender, category {color}{color:#6a9955}-- corresponding column names{color}
> {color:#d4d4d4} ){color}
> {color:#d4d4d4} ) {color}{color:#c586c0}as{color}{color:#d4d4d4} features,{color}
> {color:#d4d4d4} label{color}
> {color:#c586c0}from{color}
> {color:#d4d4d4} purchase_history{color}
> {color:#d4d4d4};{color}
> This is copied straight from the guide. When running I am getting this error
>
> :24,657 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id 2019-05-16 10:09:24,692 INFO [main] org.apache.hadoop.hive.ql.exec.FileSinkOperator: Using serializer : class org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe[[[B@43b0ade]:[id, features, label]:[int, array<string>, int]] and formatter : org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat@2d66530f 2019-05-16 10:09:24,692 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: mapred.healthChecker.script.timeout is deprecated. Instead, use mapreduce.tasktracker.healthchecker.script.timeout 2019-05-16 10:09:24,706 INFO [main] org.apache.hadoop.hive.ql.exec.Utilities: PLAN PATH = hdfs://localhost:9000/tmp/hive/jshaw6/970622c3-bfd6-407c-93f9-953184696ebf/hive_2019-05-16_10-09-11_357_6286825630727418123-1/-mr-10006/3f3f0199-3af0-40dd-abb4-6bad4df12ba7/map.xml 2019-05-16 10:09:24,745 ERROR [main] org.apache.hadoop.hive.ql.exec.mr.ExecMapper: Hit error while closing operators - failing tree 2019-05-16 10:09:24,746 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.RuntimeException: Hive Runtime Error while closing operators at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:211) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61) at org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Error evaluating id at org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:149) at org.apache.hadoop.hive.ql.exec.Operator.vectorForward(Operator.java:966) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:939) at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125) at org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.closeOp(VectorMapOperator.java:990) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:733) at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:193) ... 9 more Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.exec.UDFArgumentException: argument must be a constant value: array<string> at org.apache.hadoop.hive.ql.exec.vector.udf.VectorUDFAdaptor.evaluate(VectorUDFAdaptor.java:106) at org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:271) at org.apache.hadoop.hive.ql.exec.vector.udf.VectorUDFAdaptor.evaluate(VectorUDFAdaptor.java:111) at org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:146) ... 15 more Caused by: org.apache.hadoop.hive.ql.exec.UDFArgumentException: argument must be a constant value: array<string> at hivemall.utils.hadoop.HiveUtils.getConstStringArray(HiveUtils.java:502) at hivemall.ftvec.trans.QuantitativeFeaturesUDF.initialize(QuantitativeFeaturesUDF.java:80) at org.apache.hadoop.hive.ql.exec.vector.udf.VectorUDFAdaptor.init(VectorUDFAdaptor.java:89) at org.apache.hadoop.hive.ql.exec.vector.udf.VectorUDFAdaptor.evaluate(VectorUDFAdaptor.java:104) ... 18 more
>
> However when I run the same query alone, without creating a table, I get the right results.
>
> 1 ["price:600.0","day of week#Saturday","gender#male","category#book"] 1
> 2 ["price:4800.0","day of week#Friday","gender#female","category#sports"]0
> 3 ["price:18000.0","day of week#Friday","gender#other","category#entertainment"] 0
> 4 ["price:200.0","day of week#Thursday","gender#male","category#food"] 0
> 5 ["price:1000.0","day of week#Wednesday","gender#female","category#electronics"] 1
>
> Any idea why I am not able to save this information in a table?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)