You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "mengqi (Jira)" <ji...@apache.org> on 2022/01/21 08:32:00 UTC
[jira] [Resolved] (HBASE-26695) Throw exception if region has same start key when writePartitions use HFileOutputFormat2
[ https://issues.apache.org/jira/browse/HBASE-26695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
mengqi resolved HBASE-26695.
----------------------------
Resolution: Duplicate
> Throw exception if region has same start key when writePartitions use HFileOutputFormat2
> ----------------------------------------------------------------------------------------
>
> Key: HBASE-26695
> URL: https://issues.apache.org/jira/browse/HBASE-26695
> Project: HBase
> Issue Type: Improvement
> Reporter: mengqi
> Priority: Minor
>
> Some exception info is not friendly in MapReduce task log such as:
> {code:java}
> 2022-01-21 06:23:39,156 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.IllegalArgumentException: Can't read partitions file
> at org.apache.hadoop.mapreduce.lib.partition.TotalOrderPartitioner.setConf(TotalOrderPartitioner.java:116)
> at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
> at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
> at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:719)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:797)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:344)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:169)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1865)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
> Caused by: java.io.IOException: Wrong number of partitions in keyset
> at org.apache.hadoop.mapreduce.lib.partition.TotalOrderPartitioner.setConf(TotalOrderPartitioner.java:90)
> ... 10 more
> {code}
> when I write hfile use HFileOutputFormat2 in a MapReduce task.
> I can not understand what happend to my job. And got some wrong suggestions such as: trucate table or add cancelDeleteOnExit code in my code. Those are useless and stupid.
>
> *The real reason is my table has same start key ,and I didn't find this problem in hbck web ui overlaps list immediately.*
> {*}So{*}， I thanks it's a good idea to check same start key in writePartitions(), we can print each same start key and throw a same start key exception if my table regions has same start key.
> It is more friendly exception info than 'Can't read partitions file'.
> Here is my code [https://github.com/Mengqi777/hbase/commit/d8dc093695d02cfb16ba303389de9f7cef778ee1|https://github.com/Mengqi777/hbase/commit/d8dc093695d02cfb16ba303389de9f7cef778ee1],]
> can I create a pull request ?
>
>
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)