protobuf 2.4.1 and ObjectInspector

["kulkarni.swarnim@gmail.com" <ku...@gmail.com>]

I am trying to use the ReflectionStructObjectInspector to extract fields
from a protobuf generated from 2.4.1 compiler. I am seeing that reflections
fails to extract fields out of the generated protobuf class. Specifically,
this code snippet:

public static Field[] getDeclaredNonStaticFields(Class<?> c) {

    Field[] f = c.getDeclaredFields();    // This returns back the correct
number of fields

    ArrayList<Field> af = new ArrayList<Field>();

    for (int i = 0; i < f.length; ++i) {

      *//* *The logic here falls flat as it is looking only for the
non-static fields and all generated fields *

*     // seem to be static*

      if (!Modifier.isStatic(f[i].getModifiers())) {

        af.add(f[i]);

      }

    }

    Field[] r = new Field[af.size()];

    for (int i = 0; i < af.size(); ++i) {

      r[i] = af.get(i);

    }

    return r;

  }

This causes the whole ObjectInspector to fail. Has anyone else seen this
issue too?

Re: protobuf 2.4.1 and ObjectInspector

["kulkarni.swarnim@gmail.com" <ku...@gmail.com>]

Thanks Ed for your reply on this.

I specifically have been working on enhancing the existing hbase hive
integration to handle advanced proto/thrift structures and also logged [1]
in the process. I wasn't able to use the existing
ProtocolBuffersObjectInspector for this purpose as they extend the
ReflectionStructObjectInspector which I have seen to fail in some cases. I
decided to provided the struct schema explicitly in the CREATE EXTERNAL
TABLE command itself and then use that to lazily deserialize the structure.

This approach worked well for most cases until I hit more complicated cases
like circular references and nested repeated messages in protobufs. I
suspect that reflections "might" have worked better for such complicated
cases.

You mentioned writing a serde that takes the protobuf objects directly and
deserialize them. Did you mean to use reflections or follow a method
similar to one I described above? Any suggestions?

Thanks,

On Tue, May 22, 2012 at 9:00 PM, Edward Capriolo <ed...@gmail.com>wrote:

> I am trying to decipher my way though the code as well. Apparently the
> work is half done to return ProfotBuf as a native type. As there is
> support for ObjectInstector.THRIFT and ObjectInspector.ProtoBuffer. I
> currently want to write a Serde that works like the thrift serde where
> protobuf objects can be given directly to hive. Come hang out in the
> IRC room and maybe we can chat more about this.
>
> On Tue, May 22, 2012 at 6:09 PM, kulkarni.swarnim@gmail.com
> <ku...@gmail.com> wrote:
> > I am trying to use the ReflectionStructObjectInspector to extract fields
> > from a protobuf generated from 2.4.1 compiler. I am seeing that
> reflections
> > fails to extract fields out of the generated protobuf class.
> Specifically,
> > this code snippet:
> >
> > public static Field[] getDeclaredNonStaticFields(Class<?> c) {
> >
> >     Field[] f = c.getDeclaredFields();    // This returns back the
> correct
> > number of fields
> >
> >     ArrayList<Field> af = new ArrayList<Field>();
> >
> >     for (int i = 0; i < f.length; ++i) {
> >
> >       // The logic here falls flat as it is looking only for the
> non-static
> > fields and all generated fields
> >
> >      // seem to be static
> >
> >       if (!Modifier.isStatic(f[i].getModifiers())) {
> >
> >         af.add(f[i]);
> >
> >       }
> >
> >     }
> >
> >     Field[] r = new Field[af.size()];
> >
> >     for (int i = 0; i < af.size(); ++i) {
> >
> >       r[i] = af.get(i);
> >
> >     }
> >
> >     return r;
> >
> >   }
> >
> >
> > This causes the whole ObjectInspector to fail. Has anyone else seen this
> > issue too?
> >
>



-- 
Swarnim

Re: protobuf 2.4.1 and ObjectInspector

[Edward Capriolo <ed...@gmail.com>]

I am trying to decipher my way though the code as well. Apparently the
work is half done to return ProfotBuf as a native type. As there is
support for ObjectInstector.THRIFT and ObjectInspector.ProtoBuffer. I
currently want to write a Serde that works like the thrift serde where
protobuf objects can be given directly to hive. Come hang out in the
IRC room and maybe we can chat more about this.

On Tue, May 22, 2012 at 6:09 PM, kulkarni.swarnim@gmail.com
<ku...@gmail.com> wrote:
> I am trying to use the ReflectionStructObjectInspector to extract fields
> from a protobuf generated from 2.4.1 compiler. I am seeing that reflections
> fails to extract fields out of the generated protobuf class. Specifically,
> this code snippet:
>
> public static Field[] getDeclaredNonStaticFields(Class<?> c) {
>
>     Field[] f = c.getDeclaredFields();    // This returns back the correct
> number of fields
>
>     ArrayList<Field> af = new ArrayList<Field>();
>
>     for (int i = 0; i < f.length; ++i) {
>
>       // The logic here falls flat as it is looking only for the non-static
> fields and all generated fields
>
>      // seem to be static
>
>       if (!Modifier.isStatic(f[i].getModifiers())) {
>
>         af.add(f[i]);
>
>       }
>
>     }
>
>     Field[] r = new Field[af.size()];
>
>     for (int i = 0; i < af.size(); ++i) {
>
>       r[i] = af.get(i);
>
>     }
>
>     return r;
>
>   }
>
>
> This causes the whole ObjectInspector to fail. Has anyone else seen this
> issue too?
>