Progress percentage for gora-hbase record readers

[Alparslan Avcı <al...@gmail.com>]

Hi all,

While upgrading Nutch 2.x to gora-0.4, we have noticed that gora-hbase
always returns 0 (zero) for progress percentage when GoraRecordReader is
used in Mappers. And when I looked into the HBaseScannerResult, which
handles and calculates the progress of the result of a query based on an
HBase scanner, I've seen that there is a TODO in getProgress() method.
Moreoever, it has not been updated from the beginning of the project.

Is there any work-on-progress or discussion about this issue?

Thanks.

-- 
Alparslan Avcı

Re: Progress percentage for gora-hbase record readers

[Renato Marroquín Mogrovejo <re...@gmail.com>]

Hi Alparslan,

I think this is because we are not able to know before hand how many tuples
are going to be read or fetched. We could do this but then we would have to
do an extra operation to figure this out.
I think this is the same behaviour inside the MapReduce examples from HBase
itself [1]. Maybe if we knew the maximum number of records that a request
is going to get, then we would be able to tell how many we have already
seen, and then pass this result as the request progress.


Renato M.

[1]
https://github.com/apache/hbase/blob/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java#L297


2014-05-03 10:02 GMT+02:00 Alparslan Avcı <al...@gmail.com>:

> Hi all,
>
> While upgrading Nutch 2.x to gora-0.4, we have noticed that gora-hbase
> always returns 0 (zero) for progress percentage when GoraRecordReader is
> used in Mappers. And when I looked into the HBaseScannerResult, which
> handles and calculates the progress of the result of a query based on an
> HBase scanner, I've seen that there is a TODO in getProgress() method.
> Moreoever, it has not been updated from the beginning of the project.
>
> Is there any work-on-progress or discussion about this issue?
>
> Thanks.
>
> --
> Alparslan Avcı
>