You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2020/12/09 15:11:08 UTC
[GitHub] [iceberg] islandgit opened a new issue #1898: Hive read iceberg table ERROR
islandgit opened a new issue #1898:
URL: https://github.com/apache/iceberg/issues/1898
I added iceberg-hive- Runtime-0.10.0. jar in the Lib patch of hive. after starting Hive ‘set iceberg.mr.catalog=hive’
I can running
select * from table limit 10,
but Can't running
select * from table where id = 100
The error message is:
20/12/09 22:50:28 ERROR exec.TaskRunner: Error in executeTask
java.lang.NoSuchMethodError: org.apache.hadoop.hive.ql.io.sarg.ConvertAstToSearchArg.create(Lorg/apache/hadoop/conf/Configuration;Lorg/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc;)Lorg/apache/hadoop/hive/ql/io/sarg/SearchArgument;
at org.apache.iceberg.mr.hive.HiveIcebergInputFormat.getSplits(HiveIcebergInputFormat.java:55)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:349)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:468)
at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:542)
at org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:338)
at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:329)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:200)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1570)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1567)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1567)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:576)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:571)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:571)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:562)
at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:444)
at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:151)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2200)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1843)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1563)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1339)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1328)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:187)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:409)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:836)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:772)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:699)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:313)
at org.apache.hadoop.util.RunJar.main(RunJar.java:227)
FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. org.apache.hadoop.hive.ql.io.sarg.ConvertAstToSearchArg.create(Lorg/apache/hadoop/conf/Configuration;Lorg/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc;)Lorg/apache/hadoop/hive/ql/io/sarg/SearchArgument;
20/12/09 22:50:28 ERROR ql.Driver: FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. org.apache.hadoop.hive.ql.io.sarg.ConvertAstToSearchArg.create(Lorg/apache/hadoop/conf/Configuration;Lorg/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc;)Lorg/apache/hadoop/hive/ql/io/sarg/SearchArgument;
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary closed issue #1898: Hive read iceberg table ERROR
Posted by GitBox <gi...@apache.org>.
pvary closed issue #1898:
URL: https://github.com/apache/iceberg/issues/1898
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on issue #1898: Hive read iceberg table ERROR
Posted by GitBox <gi...@apache.org>.
pvary commented on issue #1898:
URL: https://github.com/apache/iceberg/issues/1898#issuecomment-755958727
@islandgit: I just realized I can not 😢. Could you please close this?
Thanks,
Peter
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on issue #1898: Hive read iceberg table ERROR
Posted by GitBox <gi...@apache.org>.
pvary commented on issue #1898:
URL: https://github.com/apache/iceberg/issues/1898#issuecomment-755266152
@islandgit: May I close this issue?
Thanks,
Peter
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on issue #1898: Hive read iceberg table ERROR
Posted by GitBox <gi...@apache.org>.
pvary commented on issue #1898:
URL: https://github.com/apache/iceberg/issues/1898#issuecomment-741941924
@islandgit: The method was introduced by [HIVE-15269](https://issues.apache.org/jira/browse/HIVE-15269). This is only in Hive 2.2. Iceberg supports Hive 2.3.7(?), and Hive 3.1.2. The CDH version you have mentioned is using Hive 2.1.1, and it does not contain HIVE-15269. 😢
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] islandgit commented on issue #1898: Hive read iceberg table ERROR
Posted by GitBox <gi...@apache.org>.
islandgit commented on issue #1898:
URL: https://github.com/apache/iceberg/issues/1898#issuecomment-741886984
> @islandgit：我看不到任何命名的列`id`。您也可以发送确切的查询吗？
>
> 谢谢彼得
@pvary
sorry ， select * from table where id = 100 is just an example.
The actual SQL
**_select * from sinoiov_original_data where vehicle = 'AP3068';_**
**This is the query information for Beeline，The error message above is the 'hive' query**
INFO : Compiling command(queryId=hive_20201210002100_62b991d1-d5a6-40e1-835d-10f157291e0e): select * from sinoiov_original_data where vehicle = 'AP3068'
INFO : Semantic Analysis Completed
INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:sinoiov_original_data.gpstime, type:bigint, comment:null), FieldSchema(name:sinoiov_original_data.systime, type:bigint, comment:null), FieldSchema(name:sinoiov_original_data.lon, type:int, comment:null), FieldSchema(name:sinoiov_original_data.lat, type:int, comment:null), FieldSchema(name:sinoiov_original_data.gpsspeed, type:int, comment:null), FieldSchema(name:sinoiov_original_data.direction, type:int, comment:null), FieldSchema(name:sinoiov_original_data.altitude, type:int, comment:null), FieldSchema(name:sinoiov_original_data.mileage, type:int, comment:null), FieldSchema(name:sinoiov_original_data.state, type:string, comment:null), FieldSchema(name:sinoiov_original_data.alarm, type:string, comment:null), FieldSchema(name:sinoiov_original_data.color, type:int, comment:null), FieldSchema(name:sinoiov_original_data.vehicle, type:string, comment:null), FieldSchema(name:sinoiov_original_data.p_time, type:timestamp,
comment:null)], properties:null)
INFO : Completed compiling command(queryId=hive_20201210002100_62b991d1-d5a6-40e1-835d-10f157291e0e); Time taken: 0.211 seconds
INFO : Executing command(queryId=hive_20201210002100_62b991d1-d5a6-40e1-835d-10f157291e0e): select * from sinoiov_original_data where vehicle = 'AP3068'
WARN :
INFO : Query ID = hive_20201210002100_62b991d1-d5a6-40e1-835d-10f157291e0e
INFO : Total jobs = 1
INFO : Launching Job 1 out of 1
INFO : Starting task [Stage-1:MAPRED] in serial mode
INFO : Number of reduce tasks is set to 0 since there's no reduce operator
INFO : Cleaning up the staging area /user/hive/.staging/job_1607332098484_0070
ERROR : FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. org.apache.hadoop.hive.ql.io.sarg.ConvertAstToSearchArg.create(Lorg/apache/hadoop/conf/Configuration;Lorg/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc;)Lorg/apache/hadoop/hive/ql/io/sarg/SearchArgument;
INFO : Completed executing command(queryId=hive_20201210002100_62b991d1-d5a6-40e1-835d-10f157291e0e); Time taken: 1.329 seconds
Error: Error while processing statement: FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. org.apache.hadoop.hive.ql.io.sarg.ConvertAstToSearchArg.create(Lorg/apache/hadoop/conf/Configuration;Lorg/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc;)Lorg/apache/hadoop/hive/ql/io/sarg/SearchArgument; (state=08S01,code=-101)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] islandgit commented on issue #1898: Hive read iceberg table ERROR
Posted by GitBox <gi...@apache.org>.
islandgit commented on issue #1898:
URL: https://github.com/apache/iceberg/issues/1898#issuecomment-741870471
> @islandgit: Could you please tell us which version of Hive were you using?
> Also could you please provide the output of the `SHOW CREATE TABLE` output for the table mentioned above?
> Thanks,
> Peter
Thanks for your answer
my hive version is **_2.1.1-cdh6.3.2_**
SHOW CREATE TABLE is:
+----------------------------------------------------+
| createtab_stmt |
+----------------------------------------------------+
| CREATE EXTERNAL TABLE `sinoiov_original_data`( |
| `gtime` bigint COMMENT '', |
| `stime` bigint COMMENT '', |
| `ln` int COMMENT '', |
| `lt` int COMMENT '', |
| `speed` int COMMENT '', |
| `direction` int COMMENT '', |
| `altitude` int COMMENT '', |
| `mileage` int COMMENT '', |
| `state` string COMMENT '', |
| `alarm` string COMMENT '', |
| `color` int COMMENT '', |
| `vehicle` string COMMENT '', |
| `p_time` timestamp COMMENT '') |
| ROW FORMAT SERDE |
| 'org.apache.iceberg.mr.hive.HiveIcebergSerDe' |
| STORED BY |
| 'org.apache.iceberg.mr.hive.HiveIcebergStorageHandler' |
| |
| LOCATION |
| 'hdfs://lbc/user/hive/warehouse/iceberg_gps_db.db/sinoiov_original_data' |
| TBLPROPERTIES ( |
| 'metadata_location'='hdfs://lbc/user/hive/warehouse/iceberg_gps_db.db/sinoiov_original_data/metadata/00001-c12049be-d808-40f8-b712-d0fe53647997.metadata.json', |
| 'previous_metadata_location'='hdfs://lbc/user/hive/warehouse/iceberg_gps_db.db/sinoiov_original_data/metadata/00000-b4c0733a-b0c3-4ae2-8546-05a761b6880c.metadata.json', |
| 'table_type'='ICEBERG', |
| 'transient_lastDdlTime'='1607431371') |
+----------------------------------------------------+
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] islandgit commented on issue #1898: Hive read iceberg table ERROR
Posted by GitBox <gi...@apache.org>.
islandgit commented on issue #1898:
URL: https://github.com/apache/iceberg/issues/1898#issuecomment-755376427
> @islandgit: May I close this issue?
>
> Thanks,
> Peter
@pvary
Thank you very much
you can close it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] islandgit edited a comment on issue #1898: Hive read iceberg table ERROR
Posted by GitBox <gi...@apache.org>.
islandgit edited a comment on issue #1898:
URL: https://github.com/apache/iceberg/issues/1898#issuecomment-741870471
> @islandgit: Could you please tell us which version of Hive were you using?
> Also could you please provide the output of the `SHOW CREATE TABLE` output for the table mentioned above?
> Thanks,
> Peter
@pvary
Thanks for your answer
my hive version is **_2.1.1-cdh6.3.2_**
SHOW CREATE TABLE is:
+----------------------------------------------------+
| createtab_stmt |
+----------------------------------------------------+
| CREATE EXTERNAL TABLE `sinoiov_original_data`( |
| `gtime` bigint COMMENT '', |
| `stime` bigint COMMENT '', |
| `ln` int COMMENT '', |
| `lt` int COMMENT '', |
| `speed` int COMMENT '', |
| `direction` int COMMENT '', |
| `altitude` int COMMENT '', |
| `mileage` int COMMENT '', |
| `state` string COMMENT '', |
| `alarm` string COMMENT '', |
| `color` int COMMENT '', |
| `vehicle` string COMMENT '', |
| `p_time` timestamp COMMENT '') |
| ROW FORMAT SERDE |
| 'org.apache.iceberg.mr.hive.HiveIcebergSerDe' |
| STORED BY |
| 'org.apache.iceberg.mr.hive.HiveIcebergStorageHandler' |
| |
| LOCATION |
| 'hdfs://lbc/user/hive/warehouse/iceberg_gps_db.db/sinoiov_original_data' |
| TBLPROPERTIES ( |
| 'metadata_location'='hdfs://lbc/user/hive/warehouse/iceberg_gps_db.db/sinoiov_original_data/metadata/00001-c12049be-d808-40f8-b712-d0fe53647997.metadata.json', |
| 'previous_metadata_location'='hdfs://lbc/user/hive/warehouse/iceberg_gps_db.db/sinoiov_original_data/metadata/00000-b4c0733a-b0c3-4ae2-8546-05a761b6880c.metadata.json', |
| 'table_type'='ICEBERG', |
| 'transient_lastDdlTime'='1607431371') |
+----------------------------------------------------+
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on issue #1898: Hive read iceberg table ERROR
Posted by GitBox <gi...@apache.org>.
pvary commented on issue #1898:
URL: https://github.com/apache/iceberg/issues/1898#issuecomment-741860625
@islandgit: Could you please tell us which version of Hive were you using?
Also could you please provide the output of the `SHOW CREATE TABLE` output for the table mentioned above?
Thanks,
Peter
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on issue #1898: Hive read iceberg table ERROR
Posted by GitBox <gi...@apache.org>.
pvary commented on issue #1898:
URL: https://github.com/apache/iceberg/issues/1898#issuecomment-741879164
@islandgit: I can not see any colum named `id`. Could you please send the exact query as well?
Thanks, Peter
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org