Re: problems with protoc versions / anyone tried under hadoop 2.0?

[Jacques Nadeau <ja...@apache.org>]

Yeah...

It's painful that 2.5 and 2.4.1 don't coeexist better together.  It
seems like they should simply separate the classes that are bw/fw
compatible and those that aren't.  C'est la vie.

My current thinking is that we generate an alternate 2.5 jar that
modifies all the package names and then use a mvn filter on the
generated protoc code as part of the compile part of the build.  I
much prefer doing it compile time rather than trying to fight through
classloader issues.  The last piece is modifying the protoc builds to
be allowed to be pointed at an alternative path for the protoc
generator so people can have both on their system (if people want to
build a combination of Drill and Hadoop or HBase).

But I haven't gotten to it yet.  We should get a jira in for it and
see if a maven maestro can help us out.

J

On Wed, Jul 31, 2013 at 12:17 AM, Lisen Mu <im...@gmail.com> wrote:
> Hi drillers,
>
> Has anyone tried scan under hadoop 2.0?
>
> We've built a HBase scanner, with hadoop 2.0.x.
>
> The problem is, drill depends on protoc 2.5.0, while hadoop 2.0.x currently
> depends on 2.4.x.
>
> Is there any suggestion to keep both drill and hadoop 2.0 working together?
>
> Keeping a branch of drill with protoc downgraded to 2.4 is just painful. It
> would be great if the drill master could downgrade to 2.4, however, the
> most promising solution is likely to be some class loading techniques to
> keep both happy. Additional requirement is to keep 'mvn test' happy, too.
> Any suggestion on this? What's the easiest way?
>
> Thanks,
>
> Lisen

Re: problems with protoc versions / anyone tried under hadoop 2.0?

[Lisen Mu <im...@gmail.com>]

I think I might have put together a way to make things work as you said,
with maven-shade-plugin. Will verify tomorrow.

The idea is to rename classes in protobuf-java and references to pb-java in
'common' into renamed package '250.com.google.protobuf.*' and to package
together into an alternative jar: common-1.0-SNAPSHOT-rebuffed.jar

The compilation of java-exec would depend on normal common-1.0-SNAPSHOT.jar
and normal protobuf-java without a problem. java-exec also packages an
alternative jar: java-exec-1.0-SNAPSHOT-rebuffed.jar, with all references
to pb-java renamed, but without renamed pb-java itself.

In this way nothing changed under normal workflow; If one wants to run with
hadoop 2.0, he/she can have dependency on rebuffed jars with
<classifier>rebuffed</classifier> .

the patch is at: https://issues.apache.org/jira/browse/DRILL-161

Is there any problem? Or any better way to do this?






On Fri, Aug 2, 2013 at 3:00 AM, Jacques Nadeau <ja...@apache.org> wrote:

> Yeah...
>
> It's painful that 2.5 and 2.4.1 don't coeexist better together.  It
> seems like they should simply separate the classes that are bw/fw
> compatible and those that aren't.  C'est la vie.
>
> My current thinking is that we generate an alternate 2.5 jar that
> modifies all the package names and then use a mvn filter on the
> generated protoc code as part of the compile part of the build.  I
> much prefer doing it compile time rather than trying to fight through
> classloader issues.  The last piece is modifying the protoc builds to
> be allowed to be pointed at an alternative path for the protoc
> generator so people can have both on their system (if people want to
> build a combination of Drill and Hadoop or HBase).
>
> But I haven't gotten to it yet.  We should get a jira in for it and
> see if a maven maestro can help us out.
>
> J
>
> On Wed, Jul 31, 2013 at 12:17 AM, Lisen Mu <im...@gmail.com> wrote:
> > Hi drillers,
> >
> > Has anyone tried scan under hadoop 2.0?
> >
> > We've built a HBase scanner, with hadoop 2.0.x.
> >
> > The problem is, drill depends on protoc 2.5.0, while hadoop 2.0.x
> currently
> > depends on 2.4.x.
> >
> > Is there any suggestion to keep both drill and hadoop 2.0 working
> together?
> >
> > Keeping a branch of drill with protoc downgraded to 2.4 is just painful.
> It
> > would be great if the drill master could downgrade to 2.4, however, the
> > most promising solution is likely to be some class loading techniques to
> > keep both happy. Additional requirement is to keep 'mvn test' happy, too.
> > Any suggestion on this? What's the easiest way?
> >
> > Thanks,
> >
> > Lisen
>