You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Julian Jaffe <jj...@marinsoftware.com> on 2016/11/21 22:48:55 UTC
WrongRowIOException
When importing data into a fresh HBase instance, after some time the import
throws the following exception:
Error: org.apache.hadoop.hbase.client.WrongRowIOException: The row in
\x00\x00\x0767341283611_10153807927108612\x00\x80\x00\x00\x00\x84)L\xA7/IN:nme/1461847340445/Put/vlen=42/seqid=0
doesn't match the original one
\x00\x00\x0767341283611_10153805927108612\x00\x80\x00\x00\x00\x84)L\xA7
(The non-matching row differs on different runs).
If the import is allowed to run to completion, the row count of the data
imported is less than the row count of the source data.
Googling for this error only turns up the source code that generates the
error, so it doesn't seem to be a common problem.
Can anyone provide any guidance?
Julian Jaffe
Re: WrongRowIOException
Posted by Julian Jaffe <jj...@marinsoftware.com>.
Both clusters were are running the same version of hbase, and hadoop, with
matching compile dates and checksums. Also, `hbase hbck` showed no
inconsistencies in the source hbase instance.
Stack trace:
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of
TaskAttempt attempt_1470780584817_0448_m_000346_3 is : 0.17224467
2016-11-20 06:06:01,200 FATAL [IPC Server handler 28 on 39202]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task:
attempt_1470780584817_0448_m_000346_3 - exited :
org.apache.hadoop.hbase.client.WrongRowIOException: The row in
\x00\x00\x0710356613704525352\x00\x80\x00\x00\x00\x85B\xBCU/IN:crdAt/1462937192582/Put/vlen=12/seqid=0
doesn't match the original one
\x00\x00\x0710156613704525352\x00\x80\x00\x00\x00\x85B\xBCU
at org.apache.hadoop.hbase.client.Put.add(Put.java:321)
at org.apache.hadoop.hbase.mapreduce.Import$Importer.addPutToKv(Import.java:215)
at org.apache.hadoop.hbase.mapreduce.Import$Importer.processKV(Import.java:195)
at org.apache.hadoop.hbase.mapreduce.Import$Importer.writeResult(Import.java:158)
at org.apache.hadoop.hbase.mapreduce.Import$Importer.map(Import.java:143)
at org.apache.hadoop.hbase.mapreduce.Import$Importer.map(Import.java:126)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
2016-11-20 06:06:01,200 INFO [IPC Server handler 28 on 39202]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report
from attempt_1470780584817_0448_m_000346_3: Error:
org.apache.hadoop.hbase.client.WrongRowIOException: The row in
\x00\x00\x0710356613704525352\x00\x80\x00\x00\x00\x85B\xBCU/IN:crdAt/1462937192582/Put/vlen=12/seqid=0
doesn't match the original one
\x00\x00\x0710156613704525352\x00\x80\x00\x00\x00\x85B\xBCU
at org.apache.hadoop.hbase.client.Put.add(Put.java:321)
at org.apache.hadoop.hbase.mapreduce.Import$Importer.addPutToKv(Import.java:215)
at org.apache.hadoop.hbase.mapreduce.Import$Importer.processKV(Import.java:195)
at org.apache.hadoop.hbase.mapreduce.Import$Importer.writeResult(Import.java:158)
at org.apache.hadoop.hbase.mapreduce.Import$Importer.map(Import.java:143)
at org.apache.hadoop.hbase.mapreduce.Import$Importer.map(Import.java:126)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
2016-11-20 06:06:01,200 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
Diagnostics report from attempt_1470780584817_0448_m_000346_3: Error:
org.apache.hadoop.hbase.client.WrongRowIOException: The row in
\x00\x00\x0710356613704525352\x00\x80\x00\x00\x00\x85B\xBCU/IN:crdAt/1462937192582/Put/vlen=12/seqid=0
doesn't match the original one
\x00\x00\x0710156613704525352\x00\x80\x00\x00\x00\x85B\xBCU
at org.apache.hadoop.hbase.client.Put.add(Put.java:321)
at org.apache.hadoop.hbase.mapreduce.Import$Importer.addPutToKv(Import.java:215)
at org.apache.hadoop.hbase.mapreduce.Import$Importer.processKV(Import.java:195)
at org.apache.hadoop.hbase.mapreduce.Import$Importer.writeResult(Import.java:158)
at org.apache.hadoop.hbase.mapreduce.Import$Importer.map(Import.java:143)
at org.apache.hadoop.hbase.mapreduce.Import$Importer.map(Import.java:126)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
On Mon, Nov 21, 2016 at 4:58 PM, Ted Yu <yu...@gmail.com> wrote:
> Can you give the whole stack trace for WrongRowIOException ?
>
> Was the cluster running Export using the same version of hbase (
> 1.0.0-cdh5.5.2) ?
>
> Thanks
>
> On Mon, Nov 21, 2016 at 4:35 PM, Julian Jaffe <jj...@marinsoftware.com>
> wrote:
>
> > Hbase Version: 1.0.0-cdh5.5.2
> >
> > We're importing the data using `hbase
> > org.apache.hadoop.hbase.mapreduce.Import 'table.name' /path/to/backup`
> > (The data was exported from an HBase instance on another cluster using
> > `hbase org.apache.hadoop.hbase.mapreduce.Export` and then distcp'd
> between
> > the clusters).
> >
> > On Mon, Nov 21, 2016 at 4:29 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > I did a quick search - there was no relevant JIRA or discussion thread
> at
> > > first glance.
> > >
> > > Which hbase release are you using ?
> > >
> > > How do you import the data ?
> > >
> > > More details would be helpful.
> > >
> > > Thanks
> > >
> > > On Mon, Nov 21, 2016 at 2:48 PM, Julian Jaffe <
> jjaffe@marinsoftware.com>
> > > wrote:
> > >
> > > > When importing data into a fresh HBase instance, after some time the
> > > import
> > > > throws the following exception:
> > > >
> > > > Error: org.apache.hadoop.hbase.client.WrongRowIOException: The row
> in
> > > > \x00\x00\x0767341283611_10153807927108612\x00\x80\x00\
> > > > x00\x00\x84)L\xA7/IN:nme/1461847340445/Put/vlen=42/seqid=0
> > > > doesn't match the original one
> > > > \x00\x00\x0767341283611_10153805927108612\x00\x80\x00\
> > x00\x00\x84)L\xA7
> > > >
> > > > (The non-matching row differs on different runs).
> > > >
> > > > If the import is allowed to run to completion, the row count of the
> > data
> > > > imported is less than the row count of the source data.
> > > >
> > > > Googling for this error only turns up the source code that generates
> > the
> > > > error, so it doesn't seem to be a common problem.
> > > >
> > > > Can anyone provide any guidance?
> > > >
> > > > Julian Jaffe
> > > >
> > >
> >
>
Re: WrongRowIOException
Posted by Ted Yu <yu...@gmail.com>.
Can you give the whole stack trace for WrongRowIOException ?
Was the cluster running Export using the same version of hbase (
1.0.0-cdh5.5.2) ?
Thanks
On Mon, Nov 21, 2016 at 4:35 PM, Julian Jaffe <jj...@marinsoftware.com>
wrote:
> Hbase Version: 1.0.0-cdh5.5.2
>
> We're importing the data using `hbase
> org.apache.hadoop.hbase.mapreduce.Import 'table.name' /path/to/backup`
> (The data was exported from an HBase instance on another cluster using
> `hbase org.apache.hadoop.hbase.mapreduce.Export` and then distcp'd between
> the clusters).
>
> On Mon, Nov 21, 2016 at 4:29 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > I did a quick search - there was no relevant JIRA or discussion thread at
> > first glance.
> >
> > Which hbase release are you using ?
> >
> > How do you import the data ?
> >
> > More details would be helpful.
> >
> > Thanks
> >
> > On Mon, Nov 21, 2016 at 2:48 PM, Julian Jaffe <jj...@marinsoftware.com>
> > wrote:
> >
> > > When importing data into a fresh HBase instance, after some time the
> > import
> > > throws the following exception:
> > >
> > > Error: org.apache.hadoop.hbase.client.WrongRowIOException: The row in
> > > \x00\x00\x0767341283611_10153807927108612\x00\x80\x00\
> > > x00\x00\x84)L\xA7/IN:nme/1461847340445/Put/vlen=42/seqid=0
> > > doesn't match the original one
> > > \x00\x00\x0767341283611_10153805927108612\x00\x80\x00\
> x00\x00\x84)L\xA7
> > >
> > > (The non-matching row differs on different runs).
> > >
> > > If the import is allowed to run to completion, the row count of the
> data
> > > imported is less than the row count of the source data.
> > >
> > > Googling for this error only turns up the source code that generates
> the
> > > error, so it doesn't seem to be a common problem.
> > >
> > > Can anyone provide any guidance?
> > >
> > > Julian Jaffe
> > >
> >
>
Re: WrongRowIOException
Posted by Julian Jaffe <jj...@marinsoftware.com>.
Hbase Version: 1.0.0-cdh5.5.2
We're importing the data using `hbase
org.apache.hadoop.hbase.mapreduce.Import 'table.name' /path/to/backup`
(The data was exported from an HBase instance on another cluster using
`hbase org.apache.hadoop.hbase.mapreduce.Export` and then distcp'd between
the clusters).
On Mon, Nov 21, 2016 at 4:29 PM, Ted Yu <yu...@gmail.com> wrote:
> I did a quick search - there was no relevant JIRA or discussion thread at
> first glance.
>
> Which hbase release are you using ?
>
> How do you import the data ?
>
> More details would be helpful.
>
> Thanks
>
> On Mon, Nov 21, 2016 at 2:48 PM, Julian Jaffe <jj...@marinsoftware.com>
> wrote:
>
> > When importing data into a fresh HBase instance, after some time the
> import
> > throws the following exception:
> >
> > Error: org.apache.hadoop.hbase.client.WrongRowIOException: The row in
> > \x00\x00\x0767341283611_10153807927108612\x00\x80\x00\
> > x00\x00\x84)L\xA7/IN:nme/1461847340445/Put/vlen=42/seqid=0
> > doesn't match the original one
> > \x00\x00\x0767341283611_10153805927108612\x00\x80\x00\x00\x00\x84)L\xA7
> >
> > (The non-matching row differs on different runs).
> >
> > If the import is allowed to run to completion, the row count of the data
> > imported is less than the row count of the source data.
> >
> > Googling for this error only turns up the source code that generates the
> > error, so it doesn't seem to be a common problem.
> >
> > Can anyone provide any guidance?
> >
> > Julian Jaffe
> >
>
Re: WrongRowIOException
Posted by Ted Yu <yu...@gmail.com>.
I did a quick search - there was no relevant JIRA or discussion thread at
first glance.
Which hbase release are you using ?
How do you import the data ?
More details would be helpful.
Thanks
On Mon, Nov 21, 2016 at 2:48 PM, Julian Jaffe <jj...@marinsoftware.com>
wrote:
> When importing data into a fresh HBase instance, after some time the import
> throws the following exception:
>
> Error: org.apache.hadoop.hbase.client.WrongRowIOException: The row in
> \x00\x00\x0767341283611_10153807927108612\x00\x80\x00\
> x00\x00\x84)L\xA7/IN:nme/1461847340445/Put/vlen=42/seqid=0
> doesn't match the original one
> \x00\x00\x0767341283611_10153805927108612\x00\x80\x00\x00\x00\x84)L\xA7
>
> (The non-matching row differs on different runs).
>
> If the import is allowed to run to completion, the row count of the data
> imported is less than the row count of the source data.
>
> Googling for this error only turns up the source code that generates the
> error, so it doesn't seem to be a common problem.
>
> Can anyone provide any guidance?
>
> Julian Jaffe
>