You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by J Mohamed Zahoor <jm...@gmail.com> on 2012/10/22 17:47:04 UTC
Scan exception when running MR
Hi
I am facing a scanner exception like this when i run a mr job.
Both the input and output are hbase tables (different tables)…
This comes sporadically on some mapper and all other mappers runs fine..
Even the failed mapper gets passed in the next attempt.
Any clue on what might be wrong?
java.lang.NullPointerException
at org.apache.hadoop.hbase.client.Scan.<init>(Scan.java:147)
at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.createRecordReader(TableInputFormatBase.java:123)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.<init>(MapTask.java:489)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:731)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
./zahoor
Re: Scan exception when running MR
Posted by J Mohamed Zahoor <jm...@gmail.com>.
No Worries.
It was a stray hbase jar that caused the problem.
./zahoor
On 22-Oct-2012, at 10:43 PM, Bryan Beaudreault <bb...@hubspot.com> wrote:
> Oh, sorry.
>
> This sounds like a version mismatch. Do you have the same version
> installed on your servers and being pulled in your hadoop job?
>
> On Mon, Oct 22, 2012 at 12:32 PM, J Mohamed Zahoor <jm...@gmail.com> wrote:
>
>> Cool… But my map reduce doesn't even start…
>> It fails while creating a record reader...
>> The record reader fails in
>>
>> TableMapReduceUtil.convertStringToScan(conf.get(SCAN));
>>
>> and throws a
>>
>> java.io.IOException: version not supported
>> at org.apache.hadoop.hbase.client.Scan.readFields(Scan.java:558)
>> at
>> org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.convertStringToScan(TableMapReduceUtil.java:255)
>> at
>> org.apache.hadoop.hbase.mapreduce.TableInputFormat.setConf(TableInputFormat.java:105)
>> at
>> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
>> at
>> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
>> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:723)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
>> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:415)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
>> at org.apache.hadoop.mapred.Child.main(Child.java:249)
>>
>>
>> Basically the deserializing of Scan object from the conf is failing for
>> some reason.
>>
>> ./zahoor
>>
>>
>>
>>
>>
>> On 22-Oct-2012, at 9:52 PM, Bryan Beaudreault <bb...@hubspot.com>
>> wrote:
>>
>>> I'm not on 0.94.1, but I've found a lot of situations that can cause
>>> scanner timeouts and other scanner exceptions from M/R. The primary ones
>>> probably still apply in later versions:
>>>
>>>
>>> - Caching or batching set too high. If caching is set to, e.g. 1000,
>>> and hbase.rpc.timeout is set to 30 seconds, it means you need to be
>> able to
>>> be able to process 1000 records in your mapper in under 30 seconds
>> (minus
>>> the overhead of actually returning that many records). Otherwise the
>>> mapper's next call to next() will throw a timeout.
>>> - Similar to the above, this can happen if just the logic in your
>> mapper
>>> is too heavy and taking too long. Just keep in mind that the
>>> hbase.rpc.timeout can be triggered by the time between calls to next()
>>> - hbase.rpc.timeout > hbase.regionserver.lease.period. If this is the
>>> case, the RS will timeout first and kill the scan. Then your mapper
>> will
>>> call next() and since the scan no longer exists will throw a scan/lease
>>> exception.
>>> - The filters on the mapper scan are causing too many rows to be
>>> skipped, such that not enough rows can be collected to return within
>> the
>>> timeout.
>>>
>>> Hope this helps and/or is still accurate for your version.
>>>
>>> On Mon, Oct 22, 2012 at 11:51 AM, J Mohamed Zahoor <jm...@gmail.com>
>> wrote:
>>>
>>>> I am using 0.94.1
>>>>
>>>> ./zahoor
>>>>
>>>>
>>>> On 22-Oct-2012, at 9:17 PM, J Mohamed Zahoor <jm...@gmail.com> wrote:
>>>>
>>>>> Hi
>>>>>
>>>>> I am facing a scanner exception like this when i run a mr job.
>>>>> Both the input and output are hbase tables (different tables)…
>>>>> This comes sporadically on some mapper and all other mappers runs
>> fine..
>>>>> Even the failed mapper gets passed in the next attempt.
>>>>> Any clue on what might be wrong?
>>>>>
>>>>> java.lang.NullPointerException
>>>>> at org.apache.hadoop.hbase.client.Scan.<init>(Scan.java:147)
>>>>> at
>>>>
>> org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.createRecordReader(TableInputFormatBase.java:123)
>>>>> at
>>>>
>> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.<init>(MapTask.java:489)
>>>>> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:731)
>>>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
>>>>> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>> at javax.security.auth.Subject.doAs(Subject.java:415)
>>>>> at
>>>>
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
>>>>> at org.apache.hadoop.mapred.Child.main(Child.java:249)
>>>>>
>>>>>
>>>>> ./zahoor
>>>>
>>>>
>>
>>
Re: Scan exception when running MR
Posted by Bryan Beaudreault <bb...@hubspot.com>.
Oh, sorry.
This sounds like a version mismatch. Do you have the same version
installed on your servers and being pulled in your hadoop job?
On Mon, Oct 22, 2012 at 12:32 PM, J Mohamed Zahoor <jm...@gmail.com> wrote:
> Cool… But my map reduce doesn't even start…
> It fails while creating a record reader...
> The record reader fails in
>
> TableMapReduceUtil.convertStringToScan(conf.get(SCAN));
>
> and throws a
>
> java.io.IOException: version not supported
> at org.apache.hadoop.hbase.client.Scan.readFields(Scan.java:558)
> at
> org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.convertStringToScan(TableMapReduceUtil.java:255)
> at
> org.apache.hadoop.hbase.mapreduce.TableInputFormat.setConf(TableInputFormat.java:105)
> at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:723)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
>
>
> Basically the deserializing of Scan object from the conf is failing for
> some reason.
>
> ./zahoor
>
>
>
>
>
> On 22-Oct-2012, at 9:52 PM, Bryan Beaudreault <bb...@hubspot.com>
> wrote:
>
> > I'm not on 0.94.1, but I've found a lot of situations that can cause
> > scanner timeouts and other scanner exceptions from M/R. The primary ones
> > probably still apply in later versions:
> >
> >
> > - Caching or batching set too high. If caching is set to, e.g. 1000,
> > and hbase.rpc.timeout is set to 30 seconds, it means you need to be
> able to
> > be able to process 1000 records in your mapper in under 30 seconds
> (minus
> > the overhead of actually returning that many records). Otherwise the
> > mapper's next call to next() will throw a timeout.
> > - Similar to the above, this can happen if just the logic in your
> mapper
> > is too heavy and taking too long. Just keep in mind that the
> > hbase.rpc.timeout can be triggered by the time between calls to next()
> > - hbase.rpc.timeout > hbase.regionserver.lease.period. If this is the
> > case, the RS will timeout first and kill the scan. Then your mapper
> will
> > call next() and since the scan no longer exists will throw a scan/lease
> > exception.
> > - The filters on the mapper scan are causing too many rows to be
> > skipped, such that not enough rows can be collected to return within
> the
> > timeout.
> >
> > Hope this helps and/or is still accurate for your version.
> >
> > On Mon, Oct 22, 2012 at 11:51 AM, J Mohamed Zahoor <jm...@gmail.com>
> wrote:
> >
> >> I am using 0.94.1
> >>
> >> ./zahoor
> >>
> >>
> >> On 22-Oct-2012, at 9:17 PM, J Mohamed Zahoor <jm...@gmail.com> wrote:
> >>
> >>> Hi
> >>>
> >>> I am facing a scanner exception like this when i run a mr job.
> >>> Both the input and output are hbase tables (different tables)…
> >>> This comes sporadically on some mapper and all other mappers runs
> fine..
> >>> Even the failed mapper gets passed in the next attempt.
> >>> Any clue on what might be wrong?
> >>>
> >>> java.lang.NullPointerException
> >>> at org.apache.hadoop.hbase.client.Scan.<init>(Scan.java:147)
> >>> at
> >>
> org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.createRecordReader(TableInputFormatBase.java:123)
> >>> at
> >>
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.<init>(MapTask.java:489)
> >>> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:731)
> >>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
> >>> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> >>> at java.security.AccessController.doPrivileged(Native Method)
> >>> at javax.security.auth.Subject.doAs(Subject.java:415)
> >>> at
> >>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
> >>> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> >>>
> >>>
> >>> ./zahoor
> >>
> >>
>
>
Re: Scan exception when running MR
Posted by J Mohamed Zahoor <jm...@gmail.com>.
Cool… But my map reduce doesn't even start…
It fails while creating a record reader...
The record reader fails in
TableMapReduceUtil.convertStringToScan(conf.get(SCAN));
and throws a
java.io.IOException: version not supported
at org.apache.hadoop.hbase.client.Scan.readFields(Scan.java:558)
at org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.convertStringToScan(TableMapReduceUtil.java:255)
at org.apache.hadoop.hbase.mapreduce.TableInputFormat.setConf(TableInputFormat.java:105)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:723)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Basically the deserializing of Scan object from the conf is failing for some reason.
./zahoor
On 22-Oct-2012, at 9:52 PM, Bryan Beaudreault <bb...@hubspot.com> wrote:
> I'm not on 0.94.1, but I've found a lot of situations that can cause
> scanner timeouts and other scanner exceptions from M/R. The primary ones
> probably still apply in later versions:
>
>
> - Caching or batching set too high. If caching is set to, e.g. 1000,
> and hbase.rpc.timeout is set to 30 seconds, it means you need to be able to
> be able to process 1000 records in your mapper in under 30 seconds (minus
> the overhead of actually returning that many records). Otherwise the
> mapper's next call to next() will throw a timeout.
> - Similar to the above, this can happen if just the logic in your mapper
> is too heavy and taking too long. Just keep in mind that the
> hbase.rpc.timeout can be triggered by the time between calls to next()
> - hbase.rpc.timeout > hbase.regionserver.lease.period. If this is the
> case, the RS will timeout first and kill the scan. Then your mapper will
> call next() and since the scan no longer exists will throw a scan/lease
> exception.
> - The filters on the mapper scan are causing too many rows to be
> skipped, such that not enough rows can be collected to return within the
> timeout.
>
> Hope this helps and/or is still accurate for your version.
>
> On Mon, Oct 22, 2012 at 11:51 AM, J Mohamed Zahoor <jm...@gmail.com> wrote:
>
>> I am using 0.94.1
>>
>> ./zahoor
>>
>>
>> On 22-Oct-2012, at 9:17 PM, J Mohamed Zahoor <jm...@gmail.com> wrote:
>>
>>> Hi
>>>
>>> I am facing a scanner exception like this when i run a mr job.
>>> Both the input and output are hbase tables (different tables)…
>>> This comes sporadically on some mapper and all other mappers runs fine..
>>> Even the failed mapper gets passed in the next attempt.
>>> Any clue on what might be wrong?
>>>
>>> java.lang.NullPointerException
>>> at org.apache.hadoop.hbase.client.Scan.<init>(Scan.java:147)
>>> at
>> org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.createRecordReader(TableInputFormatBase.java:123)
>>> at
>> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.<init>(MapTask.java:489)
>>> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:731)
>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
>>> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at javax.security.auth.Subject.doAs(Subject.java:415)
>>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
>>> at org.apache.hadoop.mapred.Child.main(Child.java:249)
>>>
>>>
>>> ./zahoor
>>
>>
Re: Scan exception when running MR
Posted by Bryan Beaudreault <bb...@hubspot.com>.
I'm not on 0.94.1, but I've found a lot of situations that can cause
scanner timeouts and other scanner exceptions from M/R. The primary ones
probably still apply in later versions:
- Caching or batching set too high. If caching is set to, e.g. 1000,
and hbase.rpc.timeout is set to 30 seconds, it means you need to be able to
be able to process 1000 records in your mapper in under 30 seconds (minus
the overhead of actually returning that many records). Otherwise the
mapper's next call to next() will throw a timeout.
- Similar to the above, this can happen if just the logic in your mapper
is too heavy and taking too long. Just keep in mind that the
hbase.rpc.timeout can be triggered by the time between calls to next()
- hbase.rpc.timeout > hbase.regionserver.lease.period. If this is the
case, the RS will timeout first and kill the scan. Then your mapper will
call next() and since the scan no longer exists will throw a scan/lease
exception.
- The filters on the mapper scan are causing too many rows to be
skipped, such that not enough rows can be collected to return within the
timeout.
Hope this helps and/or is still accurate for your version.
On Mon, Oct 22, 2012 at 11:51 AM, J Mohamed Zahoor <jm...@gmail.com> wrote:
> I am using 0.94.1
>
> ./zahoor
>
>
> On 22-Oct-2012, at 9:17 PM, J Mohamed Zahoor <jm...@gmail.com> wrote:
>
> > Hi
> >
> > I am facing a scanner exception like this when i run a mr job.
> > Both the input and output are hbase tables (different tables)…
> > This comes sporadically on some mapper and all other mappers runs fine..
> > Even the failed mapper gets passed in the next attempt.
> > Any clue on what might be wrong?
> >
> > java.lang.NullPointerException
> > at org.apache.hadoop.hbase.client.Scan.<init>(Scan.java:147)
> > at
> org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.createRecordReader(TableInputFormatBase.java:123)
> > at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.<init>(MapTask.java:489)
> > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:731)
> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
> > at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:415)
> > at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
> > at org.apache.hadoop.mapred.Child.main(Child.java:249)
> >
> >
> > ./zahoor
>
>
Re: Scan exception when running MR
Posted by J Mohamed Zahoor <jm...@gmail.com>.
I am using 0.94.1
./zahoor
On 22-Oct-2012, at 9:17 PM, J Mohamed Zahoor <jm...@gmail.com> wrote:
> Hi
>
> I am facing a scanner exception like this when i run a mr job.
> Both the input and output are hbase tables (different tables)…
> This comes sporadically on some mapper and all other mappers runs fine..
> Even the failed mapper gets passed in the next attempt.
> Any clue on what might be wrong?
>
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.client.Scan.<init>(Scan.java:147)
> at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.createRecordReader(TableInputFormatBase.java:123)
> at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.<init>(MapTask.java:489)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:731)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
>
>
> ./zahoor