Building Crunch with JDK 1.7

[Christian Tzolov <tz...@apache.org>]

Hey,

Although Java7 is not officially supported in Hadoop I've tested it with
Crunch.

My environment is: Maven 3.0.3, Java 1.7.0_17, Mac OSX - 10.8.3 (x86_64),
 with java.source=1.6 and java.target=1.6 POM settings.

Building Crunch with both profiles (hadoop-1 and hadoop-2) shows the
following two issues:

1.  o.a.c.l.j.MapsideJoinIT.testMapsideJoin() fails because the
System.setProperty("java.io.tmpdir", "/tmp") does not work in this
environment.
The workaround is to set -Djava.io.tmpdir=/tmp on the command line.

2.  o.a.c.i.h.WordCountHBaseIT fails due to some internal HBase issue. Same
problem affects both hadoop-1/hbase-0.90.4 and
hadoop-2/hbase-0.94.1 profiles. Not sure if there is a workaround for this
problem. It might be related to HBASE-5261.


Has someone experimented with Crunch and Java7? any insides?

Cheers,
Chris

Re: Building Crunch with JDK 1.7

[Josh Wills <jo...@gmail.com>]

On Tue, Mar 26, 2013 at 1:48 PM, Matthias Friedrich <ma...@mafr.de> wrote:

> On Tuesday, 2013-03-26, Josh Wills wrote:
> > Very cool that this is all that's broken. :)
> >
> > Re: the mapside join IT, I'm wondering if that should be set to a temp
> > directory using the TemporaryPaths stuff-- @Matthias, isn't that the
> right
> > thing to do?
>
> Eventually, yes, but I remember there was some issue with Mac OS and
> the distributed cache that required this particular hack. Maybe we
> need to add some special handling to avoid using Hadoop's distributed
> cache altogether when running in local mode.
>

Ah, you're right.


>
> > Re: HBase, my HBase friends are saying that there were test issues and
> that
> > they think things work in trunk. Not sure if anyone wants to volunteer to
> > brave trying that out. ;-)
>
> Yeah. Whoever takes this on has to either know HBase very well or must
> be completely insane :)
>

The funny thing is that those two statements are completely equivalent. ;-)


>
> Regards,
>   Matthias
>

Re: Building Crunch with JDK 1.7

[Matthias Friedrich <ma...@mafr.de>]

On Tuesday, 2013-03-26, Josh Wills wrote:
> Very cool that this is all that's broken. :)
> 
> Re: the mapside join IT, I'm wondering if that should be set to a temp
> directory using the TemporaryPaths stuff-- @Matthias, isn't that the right
> thing to do?

Eventually, yes, but I remember there was some issue with Mac OS and
the distributed cache that required this particular hack. Maybe we
need to add some special handling to avoid using Hadoop's distributed
cache altogether when running in local mode.

> Re: HBase, my HBase friends are saying that there were test issues and that
> they think things work in trunk. Not sure if anyone wants to volunteer to
> brave trying that out. ;-)

Yeah. Whoever takes this on has to either know HBase very well or must
be completely insane :)

Regards,
  Matthias

Re: Building Crunch with JDK 1.7

[Josh Wills <jo...@gmail.com>]

Very cool that this is all that's broken. :)

Re: the mapside join IT, I'm wondering if that should be set to a temp
directory using the TemporaryPaths stuff-- @Matthias, isn't that the right
thing to do?

Re: HBase, my HBase friends are saying that there were test issues and that
they think things work in trunk. Not sure if anyone wants to volunteer to
brave trying that out. ;-)

J


On Tue, Mar 26, 2013 at 9:19 AM, Christian Tzolov <tz...@apache.org> wrote:

> Hey,
>
> Although Java7 is not officially supported in Hadoop I've tested it with
> Crunch.
>
> My environment is: Maven 3.0.3, Java 1.7.0_17, Mac OSX - 10.8.3 (x86_64),
>  with java.source=1.6 and java.target=1.6 POM settings.
>
> Building Crunch with both profiles (hadoop-1 and hadoop-2) shows the
> following two issues:
>
> 1.  o.a.c.l.j.MapsideJoinIT.testMapsideJoin() fails because the
> System.setProperty("java.io.tmpdir", "/tmp") does not work in this
> environment.
> The workaround is to set -Djava.io.tmpdir=/tmp on the command line.
>
> 2.  o.a.c.i.h.WordCountHBaseIT fails due to some internal HBase issue. Same
> problem affects both hadoop-1/hbase-0.90.4 and
> hadoop-2/hbase-0.94.1 profiles. Not sure if there is a workaround for this
> problem. It might be related to HBASE-5261.
>
>
> Has someone experimented with Crunch and Java7? any insides?
>
> Cheers,
> Chris
>