You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/10/14 02:40:05 UTC
[GitHub] [hudi] Neo966 opened a new issue #3795: [SUPPORT] hive query hudi error
Neo966 opened a new issue #3795:
URL: https://github.com/apache/hudi/issues/3795
hive version:2.1.1
flink:1.12.2 scala:2.11
1、hive.input.format=org.apache.hadoop.hive.ql.io.CombineHiveInputFormat;
select count(*) from xxx; //103828120, It's not right. actual number is 18874368.
2、hive.input.format=org.apache.hudi.hadoop.hive.HoodieCombineHiveInputFormat;
select count(*) from xxx; //Error: Error while processing statement: FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. org/apache/hadoop/hive/common/StringInternUtils (state=08S01,code=-101)
3、hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat;
select count(*) from xxx; //18874368
select * from xxx limit 18874360, 10; //it's works, display the last 8 records normally.
select count(*) from xxx where name = 'lisi'; //2097152
select * from xxx where name = 'lisi' limit 2097150, 10; //the result error, no record return, should return last 2 record.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3795: [SUPPORT] hive query hudi error
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3795:
URL: https://github.com/apache/hudi/issues/3795#issuecomment-1002311142
@harsh1231 to look into this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3795: [SUPPORT] hive query hudi error
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3795:
URL: https://github.com/apache/hudi/issues/3795#issuecomment-1039624510
@codope : any updates on this ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #3795: [SUPPORT] hive query hudi error
Posted by GitBox <gi...@apache.org>.
danny0405 commented on issue #3795:
URL: https://github.com/apache/hudi/issues/3795#issuecomment-971101657
Hi @codope Can we solve this in release 0.10.0 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Neo966 commented on issue #3795: [SUPPORT] hive query hudi error
Posted by GitBox <gi...@apache.org>.
Neo966 commented on issue #3795:
URL: https://github.com/apache/hudi/issues/3795#issuecomment-945489538
@codope thank you for your reply.
the table type is cow and the query type is snapshot queries.
the full stacktrace of hiveserver log as follow:
Query ID = hive_20211018161259_48b4d920-d763-4ab4-8692-b6ad63c9b034
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. org/apache/hadoop/hive/common/StringInternUtils
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #3795: [SUPPORT] hive query hudi error
Posted by GitBox <gi...@apache.org>.
codope commented on issue #3795:
URL: https://github.com/apache/hudi/issues/3795#issuecomment-945577657
@Neo966 The stacktrace does not give sufficient info. You will have to check the full log for the mapreduce job corresponding to the queryID.
Meanwhile, please give docker demo a try in the environment where you're getting the error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3795: [SUPPORT] hive query hudi error
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3795:
URL: https://github.com/apache/hudi/issues/3795#issuecomment-1039624510
@codope : any updates on this ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #3795: [SUPPORT] hive query hudi error
Posted by GitBox <gi...@apache.org>.
codope commented on issue #3795:
URL: https://github.com/apache/hudi/issues/3795#issuecomment-944926936
@Neo966 What's the [table type and query types](https://hudi.apache.org/docs/overview#table-types--queries) for which you're getting error? Also, can you share the full stacktrace?
By the way, I checked the different input formats in the [docker setup](https://hudi.apache.org/docs/docker_demo/) and it works fine. Maybe, you could also give it a try. Then, we can work with your table to see if it is isolated issue or some edge condition in `HoodieCombineHiveInputFormat `.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hellochueng commented on issue #3795: [SUPPORT] hive query hudi error
Posted by GitBox <gi...@apache.org>.
hellochueng commented on issue #3795:
URL: https://github.com/apache/hudi/issues/3795#issuecomment-971137251
CREATE EXTERNAL TABLE flink_hudi_hive_test1(
id INT, test1 STRING, test2 STRING, test3 STRING
) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
STORED AS INPUTFORMAT 'org.apache.hudi.hadoop.realtime.HoodieParquetRealtimeInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
LOCATION 'hdfs://idc-bigdata-185-56.jdy.kd.internal/hudi/flink_hudi_cdc4';
set hive.execution.engine=mr;
set hive.input.format=org.apache.hudi.hadoop.HoodieParquetInputFormat;
When running
[select * from flink_hudi_hive_test1 where id > 30000]
[select count(*) from flink_hudi_hive_test1]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org