Bitmaps field entries

[Nishadi Kirielle <nd...@cse.mrt.ac.lk>]

Hi all,

I am a final year undergraduate from University of Moratuwa, Sri Lanka.
For the final year project, we are using the bitmap indexes created by Hive
and I need a little clarification on the bitmaps field entries in the
bitmap indexes.

After creating the bitmap indexes, the '_bitmaps' field of all the records
are similar and indicate an array as follows;

[1,2,4,8589934592,1,0]
>

Will it be possible to know what is represented by each entry in the array,
how is the Java Ewah created compressed bitmap array get converted to this
and why do all of them are similar in a single index table.


Thank you and regards
-- 
Nishadi Kirielle

Undergraduate
University of Moratuwa - Sri Lanka

Mobile : +94 70 204 5934
Blog : nishadikirielle.wordpress.com

Re: Bitmaps field entries

[Nishadi Kirielle <nd...@cse.mrt.ac.lk>]

Thanks a lot for the guide.

Regards
Nishadi

On 20 December 2016 at 12:10, Gopal Vijayaraghavan <go...@apache.org>
wrote:

>
> > After creating the bitmap indexes, the '_bitmaps' field of all the
> records
> > are similar and indicate an array as follows;
>
> This is a direct deserialized object of EWAHBitMap (writeInt(),
> writeLong() as a sequence of arrays)
>
> https://github.com/lemire/javaewah/blob/master/src/main/
> java/com/googlecode/javaewah/EWAHCompressedBitmap.java#L608
>
> I really wish it was a bit more robust to read/use, but the bitmaps are
> more of an internal structure
>
> org.apache.hadoop.hive.ql.index.bitmap.BitmapObjectOutput;
>
> with no real docs.
>
> Cheers,
> Gopal
>
>
>


-- 
Nishadi Kirielle

Undergraduate
University of Moratuwa - Sri Lanka

Mobile : +94 70 204 5934
Blog : nishadikirielle.wordpress.com

Re: Bitmaps field entries

[Gopal Vijayaraghavan <go...@apache.org>]

> After creating the bitmap indexes, the '_bitmaps' field of all the records
> are similar and indicate an array as follows;

This is a direct deserialized object of EWAHBitMap (writeInt(), writeLong() as a sequence of arrays)

https://github.com/lemire/javaewah/blob/master/src/main/java/com/googlecode/javaewah/EWAHCompressedBitmap.java#L608

I really wish it was a bit more robust to read/use, but the bitmaps are more of an internal structure 

org.apache.hadoop.hive.ql.index.bitmap.BitmapObjectOutput;

with no real docs.

Cheers,
Gopal