You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Debasish Das <de...@gmail.com> on 2014/04/05 23:04:22 UTC
Master compilation
I am synced with apache/spark master but getting error in spark/sql
compilation...
Is the master broken ?
[info] Compiling 34 Scala sources to
/home/debasish/spark_deploy/sql/core/target/scala-2.10/classes...
[error]
/home/debasish/spark_deploy/sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetRelation.scala:106:
value getGlobal is not a member of object java.util.logging.Logger
[error] logger.setParent(Logger.getGlobal)
[error] ^
[error] one error found
[error] (sql/compile:compile) Compilation failed
[error] Total time: 171 s, completed Apr 5, 2014 4:58:41 PM
Thanks.
Deb
Re: Master compilation
Posted by Koert Kuipers <ko...@tresata.com>.
thats confusing. it seems to me the breeze dependency has been compiled
with java 6, since the mllib tests passed fine for me with java 6
On Sun, Apr 6, 2014 at 12:00 PM, Debasish Das <de...@gmail.com>wrote:
> Hi Koert,
>
> How do I specify that in sbt ?
>
> Is this the correct way ?
> javacOptions ++= Seq("-target", "1.6", "-source","1.6")
>
> Breeze project for examples compiles fine with jdk7, fails with jdk6 and
> the function it fails on:
>
> error] /home/debasish/github/breeze/
> src/main/scala/breeze/util/package.scala:200: value valueOf is not a member
> of object java.util.BitSet
> [error] java.util.BitSet.valueOf(bs.toBitMask)
>
> is not available in jdk6...
>
> http://docs.oracle.com/javase/6/docs/api/java/util/BitSet.html
>
> I have no clue how with target 1.6 solves the issue...are you saying jdk7
> will put a function that's closest to java.util.BitSet.valueOf ?
>
> Thanks.
> Deb
>
>
>
> On Sun, Apr 6, 2014 at 8:41 AM, Koert Kuipers <ko...@tresata.com> wrote:
>
> > classes compiled with java7 run fine on java6 if you specified "-target
> > 1.6". however if thats the case generally you should also be able to also
> > then compile it with java 6 just fine.
> >
> > something compiled with java7 with "-target 1.7" will not run on java 6
> >
> >
> >
> > On Sat, Apr 5, 2014 at 9:10 PM, Debasish Das <debasish.das83@gmail.com
> > >wrote:
> >
> > > With jdk7 I could compile it fine:
> > >
> > > java version "1.7.0_51"
> > > Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
> > > Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)
> > >
> > > What happens if I say take the jar and try to deploy it on ancient
> > centos6
> > > default on cluster ?
> > >
> > > java -version
> > > java version "1.6.0_31"
> > > Java(TM) SE Runtime Environment (build 1.6.0_31-b04)
> > > Java HotSpot(TM) 64-Bit Server VM (build 20.6-b01, mixed mode)
> > >
> > > Breeze compilation also fails with jdk6, runs fine with jdk7 and breeze
> > jar
> > > is already included in spark mllib with Xiangrui's Sparse vector
> > > checkin....
> > >
> > > Does that mean that classes compiled and generated using jdk7 will run
> > fine
> > > on jre6 ?
> > >
> > > I am confused
> > >
> > >
> > > On Sat, Apr 5, 2014 at 3:09 PM, Sean Owen <so...@cloudera.com> wrote:
> > >
> > > > Will do. I'm just finishing a recompile to check for anything else
> like
> > > > this.
> > > >
> > > > The reason is because the tests run with Java 7 (like lots of us do
> > > > including me) so it used the Java 7 classpath and found the class.
> > > > It's possible to use Java 7 with the Java 6 -bootclasspath. Or just
> > > > use Java 6.
> > > > --
> > > > Sean Owen | Director, Data Science | London
> > > >
> > > >
> > > > On Sat, Apr 5, 2014 at 11:06 PM, Patrick Wendell <pwendell@gmail.com
> >
> > > > wrote:
> > > > > If you want to submit a hot fix for this issue specifically please
> > do.
> > > > I'm
> > > > > not sure why it didn't fail our build...
> > > > >
> > > > >
> > > > > On Sat, Apr 5, 2014 at 2:30 PM, Debasish Das <
> > debasish.das83@gmail.com
> > > > >wrote:
> > > > >
> > > > >> I verified this is happening for both CDH4.5 and 1.0.4...My deploy
> > > > >> environment is Java 6...so Java 7 compilation is not going to
> > help...
> > > > >>
> > > > >> Is this the PR which caused it ?
> > > > >>
> > > > >> Andre Schumacher
> > > > >>
> > > > >> fbebaed Spark parquet improvements A few improvements to
> the
> > > > Parquet
> > > > >> support for SQL queries: - Instead of files a ParquetRelation is
> now
> > > > backed
> > > > >> by a directory, which simplifies importing data from other
> sources -
> > > > >> InsertIntoParquetTable operation now supports switching between
> > > > overwriting
> > > > >> or appending (at least in HiveQL) - tests now use the new API -
> > > Parquet
> > > > >> logging can be set to WARNING level (Default) - Default
> compression
> > > for
> > > > >> Parquet files (GZIP, as in parquet-mr) Author: Andre Schumacher
> &...
> > > > 2
> > > > >> days ago SPARK-1383
> > > > >>
> > > > >> I will go to a stable checkin before this
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Sat, Apr 5, 2014 at 2:22 PM, Debasish Das <
> > > debasish.das83@gmail.com
> > > > >> >wrote:
> > > > >>
> > > > >> > I can compile with Java 7...let me try that...
> > > > >> >
> > > > >> >
> > > > >> > On Sat, Apr 5, 2014 at 2:19 PM, Sean Owen <so...@cloudera.com>
> > > wrote:
> > > > >> >
> > > > >> >> That method was added in Java 7. The project is on Java 6, so I
> > > think
> > > > >> >> this was just an inadvertent error in a recent PR (it was the
> > > 'Spark
> > > > >> >> parquet improvements' one).
> > > > >> >>
> > > > >> >> I'll open a hot-fix PR after looking for other stuff like this
> > that
> > > > >> >> might have snuck in.
> > > > >> >> --
> > > > >> >> Sean Owen | Director, Data Science | London
> > > > >> >>
> > > > >> >>
> > > > >> >> On Sat, Apr 5, 2014 at 10:04 PM, Debasish Das <
> > > > debasish.das83@gmail.com
> > > > >> >
> > > > >> >> wrote:
> > > > >> >> > I am synced with apache/spark master but getting error in
> > > spark/sql
> > > > >> >> > compilation...
> > > > >> >> >
> > > > >> >> > Is the master broken ?
> > > > >> >> >
> > > > >> >> > [info] Compiling 34 Scala sources to
> > > > >> >> >
> > /home/debasish/spark_deploy/sql/core/target/scala-2.10/classes...
> > > > >> >> > [error]
> > > > >> >> >
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> /home/debasish/spark_deploy/sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetRelation.scala:106:
> > > > >> >> > value getGlobal is not a member of object
> > > java.util.logging.Logger
> > > > >> >> > [error] logger.setParent(Logger.getGlobal)
> > > > >> >> > [error] ^
> > > > >> >> > [error] one error found
> > > > >> >> > [error] (sql/compile:compile) Compilation failed
> > > > >> >> > [error] Total time: 171 s, completed Apr 5, 2014 4:58:41 PM
> > > > >> >> >
> > > > >> >> > Thanks.
> > > > >> >> > Deb
> > > > >> >>
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
>
Re: Master compilation
Posted by Debasish Das <de...@gmail.com>.
Yeah spark builds are fine...
For solvers we are planning to use breeze optimization since it has most of
the core functions we will need and we can enhance it further (QP solver
for example)
Right now sparse kmeans in spark mllib uses breeze and that might not even
need this line of code....But still I thought Xiangrui should be aware of
this issue...
On Sun, Apr 6, 2014 at 9:07 AM, Sean Owen <so...@cloudera.com> wrote:
> That's a Breeze question, no? you should not need to compile Breeze
> yourself to compile Spark -- why do that?
>
> That method indeed only exists in Java 7. But Breeze seems to target
> Java 6 as expected:
>
> https://github.com/scalanlp/breeze/blob/master/build.sbt#L59
>
> I see this particular line of code was added after the last release:
>
>
> https://github.com/scalanlp/breeze/commit/ff46ddfa66f98b8c8b0ef5b65a6e7a9f86b5a5c4
>
> So it's possible it's an issue lurking in Breeze of just the same form
> we just saw. It's worth opening an issue since, indeed, I would expect
> exactly the compile error you see with Java 6.
>
> But it should not stop you from building Spark.
>
>
> On Sun, Apr 6, 2014 at 5:00 PM, Debasish Das <de...@gmail.com>
> wrote:
> > Hi Koert,
> >
> > How do I specify that in sbt ?
> >
> > Is this the correct way ?
> > javacOptions ++= Seq("-target", "1.6", "-source","1.6")
> >
> > Breeze project for examples compiles fine with jdk7, fails with jdk6 and
> > the function it fails on:
> >
> > error] /home/debasish/github/breeze/
> > src/main/scala/breeze/util/package.scala:200: value valueOf is not a
> member
> > of object java.util.BitSet
> > [error] java.util.BitSet.valueOf(bs.toBitMask)
> >
> > is not available in jdk6...
> >
> > http://docs.oracle.com/javase/6/docs/api/java/util/BitSet.html
> >
> > I have no clue how with target 1.6 solves the issue...are you saying jdk7
> > will put a function that's closest to java.util.BitSet.valueOf ?
> >
> > Thanks.
> > Deb
> >
>
Re: Master compilation
Posted by Sean Owen <so...@cloudera.com>.
That's a Breeze question, no? you should not need to compile Breeze
yourself to compile Spark -- why do that?
That method indeed only exists in Java 7. But Breeze seems to target
Java 6 as expected:
https://github.com/scalanlp/breeze/blob/master/build.sbt#L59
I see this particular line of code was added after the last release:
https://github.com/scalanlp/breeze/commit/ff46ddfa66f98b8c8b0ef5b65a6e7a9f86b5a5c4
So it's possible it's an issue lurking in Breeze of just the same form
we just saw. It's worth opening an issue since, indeed, I would expect
exactly the compile error you see with Java 6.
But it should not stop you from building Spark.
On Sun, Apr 6, 2014 at 5:00 PM, Debasish Das <de...@gmail.com> wrote:
> Hi Koert,
>
> How do I specify that in sbt ?
>
> Is this the correct way ?
> javacOptions ++= Seq("-target", "1.6", "-source","1.6")
>
> Breeze project for examples compiles fine with jdk7, fails with jdk6 and
> the function it fails on:
>
> error] /home/debasish/github/breeze/
> src/main/scala/breeze/util/package.scala:200: value valueOf is not a member
> of object java.util.BitSet
> [error] java.util.BitSet.valueOf(bs.toBitMask)
>
> is not available in jdk6...
>
> http://docs.oracle.com/javase/6/docs/api/java/util/BitSet.html
>
> I have no clue how with target 1.6 solves the issue...are you saying jdk7
> will put a function that's closest to java.util.BitSet.valueOf ?
>
> Thanks.
> Deb
>
Re: Master compilation
Posted by Debasish Das <de...@gmail.com>.
Hi Koert,
How do I specify that in sbt ?
Is this the correct way ?
javacOptions ++= Seq("-target", "1.6", "-source","1.6")
Breeze project for examples compiles fine with jdk7, fails with jdk6 and
the function it fails on:
error] /home/debasish/github/breeze/
src/main/scala/breeze/util/package.scala:200: value valueOf is not a member
of object java.util.BitSet
[error] java.util.BitSet.valueOf(bs.toBitMask)
is not available in jdk6...
http://docs.oracle.com/javase/6/docs/api/java/util/BitSet.html
I have no clue how with target 1.6 solves the issue...are you saying jdk7
will put a function that's closest to java.util.BitSet.valueOf ?
Thanks.
Deb
On Sun, Apr 6, 2014 at 8:41 AM, Koert Kuipers <ko...@tresata.com> wrote:
> classes compiled with java7 run fine on java6 if you specified "-target
> 1.6". however if thats the case generally you should also be able to also
> then compile it with java 6 just fine.
>
> something compiled with java7 with "-target 1.7" will not run on java 6
>
>
>
> On Sat, Apr 5, 2014 at 9:10 PM, Debasish Das <debasish.das83@gmail.com
> >wrote:
>
> > With jdk7 I could compile it fine:
> >
> > java version "1.7.0_51"
> > Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
> > Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)
> >
> > What happens if I say take the jar and try to deploy it on ancient
> centos6
> > default on cluster ?
> >
> > java -version
> > java version "1.6.0_31"
> > Java(TM) SE Runtime Environment (build 1.6.0_31-b04)
> > Java HotSpot(TM) 64-Bit Server VM (build 20.6-b01, mixed mode)
> >
> > Breeze compilation also fails with jdk6, runs fine with jdk7 and breeze
> jar
> > is already included in spark mllib with Xiangrui's Sparse vector
> > checkin....
> >
> > Does that mean that classes compiled and generated using jdk7 will run
> fine
> > on jre6 ?
> >
> > I am confused
> >
> >
> > On Sat, Apr 5, 2014 at 3:09 PM, Sean Owen <so...@cloudera.com> wrote:
> >
> > > Will do. I'm just finishing a recompile to check for anything else like
> > > this.
> > >
> > > The reason is because the tests run with Java 7 (like lots of us do
> > > including me) so it used the Java 7 classpath and found the class.
> > > It's possible to use Java 7 with the Java 6 -bootclasspath. Or just
> > > use Java 6.
> > > --
> > > Sean Owen | Director, Data Science | London
> > >
> > >
> > > On Sat, Apr 5, 2014 at 11:06 PM, Patrick Wendell <pw...@gmail.com>
> > > wrote:
> > > > If you want to submit a hot fix for this issue specifically please
> do.
> > > I'm
> > > > not sure why it didn't fail our build...
> > > >
> > > >
> > > > On Sat, Apr 5, 2014 at 2:30 PM, Debasish Das <
> debasish.das83@gmail.com
> > > >wrote:
> > > >
> > > >> I verified this is happening for both CDH4.5 and 1.0.4...My deploy
> > > >> environment is Java 6...so Java 7 compilation is not going to
> help...
> > > >>
> > > >> Is this the PR which caused it ?
> > > >>
> > > >> Andre Schumacher
> > > >>
> > > >> fbebaed Spark parquet improvements A few improvements to the
> > > Parquet
> > > >> support for SQL queries: - Instead of files a ParquetRelation is now
> > > backed
> > > >> by a directory, which simplifies importing data from other sources -
> > > >> InsertIntoParquetTable operation now supports switching between
> > > overwriting
> > > >> or appending (at least in HiveQL) - tests now use the new API -
> > Parquet
> > > >> logging can be set to WARNING level (Default) - Default compression
> > for
> > > >> Parquet files (GZIP, as in parquet-mr) Author: Andre Schumacher &...
> > > 2
> > > >> days ago SPARK-1383
> > > >>
> > > >> I will go to a stable checkin before this
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> On Sat, Apr 5, 2014 at 2:22 PM, Debasish Das <
> > debasish.das83@gmail.com
> > > >> >wrote:
> > > >>
> > > >> > I can compile with Java 7...let me try that...
> > > >> >
> > > >> >
> > > >> > On Sat, Apr 5, 2014 at 2:19 PM, Sean Owen <so...@cloudera.com>
> > wrote:
> > > >> >
> > > >> >> That method was added in Java 7. The project is on Java 6, so I
> > think
> > > >> >> this was just an inadvertent error in a recent PR (it was the
> > 'Spark
> > > >> >> parquet improvements' one).
> > > >> >>
> > > >> >> I'll open a hot-fix PR after looking for other stuff like this
> that
> > > >> >> might have snuck in.
> > > >> >> --
> > > >> >> Sean Owen | Director, Data Science | London
> > > >> >>
> > > >> >>
> > > >> >> On Sat, Apr 5, 2014 at 10:04 PM, Debasish Das <
> > > debasish.das83@gmail.com
> > > >> >
> > > >> >> wrote:
> > > >> >> > I am synced with apache/spark master but getting error in
> > spark/sql
> > > >> >> > compilation...
> > > >> >> >
> > > >> >> > Is the master broken ?
> > > >> >> >
> > > >> >> > [info] Compiling 34 Scala sources to
> > > >> >> >
> /home/debasish/spark_deploy/sql/core/target/scala-2.10/classes...
> > > >> >> > [error]
> > > >> >> >
> > > >> >>
> > > >>
> > >
> >
> /home/debasish/spark_deploy/sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetRelation.scala:106:
> > > >> >> > value getGlobal is not a member of object
> > java.util.logging.Logger
> > > >> >> > [error] logger.setParent(Logger.getGlobal)
> > > >> >> > [error] ^
> > > >> >> > [error] one error found
> > > >> >> > [error] (sql/compile:compile) Compilation failed
> > > >> >> > [error] Total time: 171 s, completed Apr 5, 2014 4:58:41 PM
> > > >> >> >
> > > >> >> > Thanks.
> > > >> >> > Deb
> > > >> >>
> > > >> >
> > > >> >
> > > >>
> > >
> >
>
Re: Master compilation
Posted by Koert Kuipers <ko...@tresata.com>.
classes compiled with java7 run fine on java6 if you specified "-target
1.6". however if thats the case generally you should also be able to also
then compile it with java 6 just fine.
something compiled with java7 with "-target 1.7" will not run on java 6
On Sat, Apr 5, 2014 at 9:10 PM, Debasish Das <de...@gmail.com>wrote:
> With jdk7 I could compile it fine:
>
> java version "1.7.0_51"
> Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
> Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)
>
> What happens if I say take the jar and try to deploy it on ancient centos6
> default on cluster ?
>
> java -version
> java version "1.6.0_31"
> Java(TM) SE Runtime Environment (build 1.6.0_31-b04)
> Java HotSpot(TM) 64-Bit Server VM (build 20.6-b01, mixed mode)
>
> Breeze compilation also fails with jdk6, runs fine with jdk7 and breeze jar
> is already included in spark mllib with Xiangrui's Sparse vector
> checkin....
>
> Does that mean that classes compiled and generated using jdk7 will run fine
> on jre6 ?
>
> I am confused
>
>
> On Sat, Apr 5, 2014 at 3:09 PM, Sean Owen <so...@cloudera.com> wrote:
>
> > Will do. I'm just finishing a recompile to check for anything else like
> > this.
> >
> > The reason is because the tests run with Java 7 (like lots of us do
> > including me) so it used the Java 7 classpath and found the class.
> > It's possible to use Java 7 with the Java 6 -bootclasspath. Or just
> > use Java 6.
> > --
> > Sean Owen | Director, Data Science | London
> >
> >
> > On Sat, Apr 5, 2014 at 11:06 PM, Patrick Wendell <pw...@gmail.com>
> > wrote:
> > > If you want to submit a hot fix for this issue specifically please do.
> > I'm
> > > not sure why it didn't fail our build...
> > >
> > >
> > > On Sat, Apr 5, 2014 at 2:30 PM, Debasish Das <debasish.das83@gmail.com
> > >wrote:
> > >
> > >> I verified this is happening for both CDH4.5 and 1.0.4...My deploy
> > >> environment is Java 6...so Java 7 compilation is not going to help...
> > >>
> > >> Is this the PR which caused it ?
> > >>
> > >> Andre Schumacher
> > >>
> > >> fbebaed Spark parquet improvements A few improvements to the
> > Parquet
> > >> support for SQL queries: - Instead of files a ParquetRelation is now
> > backed
> > >> by a directory, which simplifies importing data from other sources -
> > >> InsertIntoParquetTable operation now supports switching between
> > overwriting
> > >> or appending (at least in HiveQL) - tests now use the new API -
> Parquet
> > >> logging can be set to WARNING level (Default) - Default compression
> for
> > >> Parquet files (GZIP, as in parquet-mr) Author: Andre Schumacher &...
> > 2
> > >> days ago SPARK-1383
> > >>
> > >> I will go to a stable checkin before this
> > >>
> > >>
> > >>
> > >>
> > >> On Sat, Apr 5, 2014 at 2:22 PM, Debasish Das <
> debasish.das83@gmail.com
> > >> >wrote:
> > >>
> > >> > I can compile with Java 7...let me try that...
> > >> >
> > >> >
> > >> > On Sat, Apr 5, 2014 at 2:19 PM, Sean Owen <so...@cloudera.com>
> wrote:
> > >> >
> > >> >> That method was added in Java 7. The project is on Java 6, so I
> think
> > >> >> this was just an inadvertent error in a recent PR (it was the
> 'Spark
> > >> >> parquet improvements' one).
> > >> >>
> > >> >> I'll open a hot-fix PR after looking for other stuff like this that
> > >> >> might have snuck in.
> > >> >> --
> > >> >> Sean Owen | Director, Data Science | London
> > >> >>
> > >> >>
> > >> >> On Sat, Apr 5, 2014 at 10:04 PM, Debasish Das <
> > debasish.das83@gmail.com
> > >> >
> > >> >> wrote:
> > >> >> > I am synced with apache/spark master but getting error in
> spark/sql
> > >> >> > compilation...
> > >> >> >
> > >> >> > Is the master broken ?
> > >> >> >
> > >> >> > [info] Compiling 34 Scala sources to
> > >> >> > /home/debasish/spark_deploy/sql/core/target/scala-2.10/classes...
> > >> >> > [error]
> > >> >> >
> > >> >>
> > >>
> >
> /home/debasish/spark_deploy/sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetRelation.scala:106:
> > >> >> > value getGlobal is not a member of object
> java.util.logging.Logger
> > >> >> > [error] logger.setParent(Logger.getGlobal)
> > >> >> > [error] ^
> > >> >> > [error] one error found
> > >> >> > [error] (sql/compile:compile) Compilation failed
> > >> >> > [error] Total time: 171 s, completed Apr 5, 2014 4:58:41 PM
> > >> >> >
> > >> >> > Thanks.
> > >> >> > Deb
> > >> >>
> > >> >
> > >> >
> > >>
> >
>
Re: Master compilation
Posted by Debasish Das <de...@gmail.com>.
With jdk7 I could compile it fine:
java version "1.7.0_51"
Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)
What happens if I say take the jar and try to deploy it on ancient centos6
default on cluster ?
java -version
java version "1.6.0_31"
Java(TM) SE Runtime Environment (build 1.6.0_31-b04)
Java HotSpot(TM) 64-Bit Server VM (build 20.6-b01, mixed mode)
Breeze compilation also fails with jdk6, runs fine with jdk7 and breeze jar
is already included in spark mllib with Xiangrui's Sparse vector checkin....
Does that mean that classes compiled and generated using jdk7 will run fine
on jre6 ?
I am confused
On Sat, Apr 5, 2014 at 3:09 PM, Sean Owen <so...@cloudera.com> wrote:
> Will do. I'm just finishing a recompile to check for anything else like
> this.
>
> The reason is because the tests run with Java 7 (like lots of us do
> including me) so it used the Java 7 classpath and found the class.
> It's possible to use Java 7 with the Java 6 -bootclasspath. Or just
> use Java 6.
> --
> Sean Owen | Director, Data Science | London
>
>
> On Sat, Apr 5, 2014 at 11:06 PM, Patrick Wendell <pw...@gmail.com>
> wrote:
> > If you want to submit a hot fix for this issue specifically please do.
> I'm
> > not sure why it didn't fail our build...
> >
> >
> > On Sat, Apr 5, 2014 at 2:30 PM, Debasish Das <debasish.das83@gmail.com
> >wrote:
> >
> >> I verified this is happening for both CDH4.5 and 1.0.4...My deploy
> >> environment is Java 6...so Java 7 compilation is not going to help...
> >>
> >> Is this the PR which caused it ?
> >>
> >> Andre Schumacher
> >>
> >> fbebaed Spark parquet improvements A few improvements to the
> Parquet
> >> support for SQL queries: - Instead of files a ParquetRelation is now
> backed
> >> by a directory, which simplifies importing data from other sources -
> >> InsertIntoParquetTable operation now supports switching between
> overwriting
> >> or appending (at least in HiveQL) - tests now use the new API - Parquet
> >> logging can be set to WARNING level (Default) - Default compression for
> >> Parquet files (GZIP, as in parquet-mr) Author: Andre Schumacher &...
> 2
> >> days ago SPARK-1383
> >>
> >> I will go to a stable checkin before this
> >>
> >>
> >>
> >>
> >> On Sat, Apr 5, 2014 at 2:22 PM, Debasish Das <debasish.das83@gmail.com
> >> >wrote:
> >>
> >> > I can compile with Java 7...let me try that...
> >> >
> >> >
> >> > On Sat, Apr 5, 2014 at 2:19 PM, Sean Owen <so...@cloudera.com> wrote:
> >> >
> >> >> That method was added in Java 7. The project is on Java 6, so I think
> >> >> this was just an inadvertent error in a recent PR (it was the 'Spark
> >> >> parquet improvements' one).
> >> >>
> >> >> I'll open a hot-fix PR after looking for other stuff like this that
> >> >> might have snuck in.
> >> >> --
> >> >> Sean Owen | Director, Data Science | London
> >> >>
> >> >>
> >> >> On Sat, Apr 5, 2014 at 10:04 PM, Debasish Das <
> debasish.das83@gmail.com
> >> >
> >> >> wrote:
> >> >> > I am synced with apache/spark master but getting error in spark/sql
> >> >> > compilation...
> >> >> >
> >> >> > Is the master broken ?
> >> >> >
> >> >> > [info] Compiling 34 Scala sources to
> >> >> > /home/debasish/spark_deploy/sql/core/target/scala-2.10/classes...
> >> >> > [error]
> >> >> >
> >> >>
> >>
> /home/debasish/spark_deploy/sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetRelation.scala:106:
> >> >> > value getGlobal is not a member of object java.util.logging.Logger
> >> >> > [error] logger.setParent(Logger.getGlobal)
> >> >> > [error] ^
> >> >> > [error] one error found
> >> >> > [error] (sql/compile:compile) Compilation failed
> >> >> > [error] Total time: 171 s, completed Apr 5, 2014 4:58:41 PM
> >> >> >
> >> >> > Thanks.
> >> >> > Deb
> >> >>
> >> >
> >> >
> >>
>
Re: Master compilation
Posted by Sean Owen <so...@cloudera.com>.
Will do. I'm just finishing a recompile to check for anything else like this.
The reason is because the tests run with Java 7 (like lots of us do
including me) so it used the Java 7 classpath and found the class.
It's possible to use Java 7 with the Java 6 -bootclasspath. Or just
use Java 6.
--
Sean Owen | Director, Data Science | London
On Sat, Apr 5, 2014 at 11:06 PM, Patrick Wendell <pw...@gmail.com> wrote:
> If you want to submit a hot fix for this issue specifically please do. I'm
> not sure why it didn't fail our build...
>
>
> On Sat, Apr 5, 2014 at 2:30 PM, Debasish Das <de...@gmail.com>wrote:
>
>> I verified this is happening for both CDH4.5 and 1.0.4...My deploy
>> environment is Java 6...so Java 7 compilation is not going to help...
>>
>> Is this the PR which caused it ?
>>
>> Andre Schumacher
>>
>> fbebaed Spark parquet improvements A few improvements to the Parquet
>> support for SQL queries: - Instead of files a ParquetRelation is now backed
>> by a directory, which simplifies importing data from other sources -
>> InsertIntoParquetTable operation now supports switching between overwriting
>> or appending (at least in HiveQL) - tests now use the new API - Parquet
>> logging can be set to WARNING level (Default) - Default compression for
>> Parquet files (GZIP, as in parquet-mr) Author: Andre Schumacher &... 2
>> days ago SPARK-1383
>>
>> I will go to a stable checkin before this
>>
>>
>>
>>
>> On Sat, Apr 5, 2014 at 2:22 PM, Debasish Das <debasish.das83@gmail.com
>> >wrote:
>>
>> > I can compile with Java 7...let me try that...
>> >
>> >
>> > On Sat, Apr 5, 2014 at 2:19 PM, Sean Owen <so...@cloudera.com> wrote:
>> >
>> >> That method was added in Java 7. The project is on Java 6, so I think
>> >> this was just an inadvertent error in a recent PR (it was the 'Spark
>> >> parquet improvements' one).
>> >>
>> >> I'll open a hot-fix PR after looking for other stuff like this that
>> >> might have snuck in.
>> >> --
>> >> Sean Owen | Director, Data Science | London
>> >>
>> >>
>> >> On Sat, Apr 5, 2014 at 10:04 PM, Debasish Das <debasish.das83@gmail.com
>> >
>> >> wrote:
>> >> > I am synced with apache/spark master but getting error in spark/sql
>> >> > compilation...
>> >> >
>> >> > Is the master broken ?
>> >> >
>> >> > [info] Compiling 34 Scala sources to
>> >> > /home/debasish/spark_deploy/sql/core/target/scala-2.10/classes...
>> >> > [error]
>> >> >
>> >>
>> /home/debasish/spark_deploy/sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetRelation.scala:106:
>> >> > value getGlobal is not a member of object java.util.logging.Logger
>> >> > [error] logger.setParent(Logger.getGlobal)
>> >> > [error] ^
>> >> > [error] one error found
>> >> > [error] (sql/compile:compile) Compilation failed
>> >> > [error] Total time: 171 s, completed Apr 5, 2014 4:58:41 PM
>> >> >
>> >> > Thanks.
>> >> > Deb
>> >>
>> >
>> >
>>
Re: Master compilation
Posted by Debasish Das <de...@gmail.com>.
@patrick our cluster still has java6 deployed...and I compiled using jdk6...
Sean is looking into it...this api is in java7 but not java6...
On Sat, Apr 5, 2014 at 3:06 PM, Patrick Wendell <pw...@gmail.com> wrote:
> If you want to submit a hot fix for this issue specifically please do. I'm
> not sure why it didn't fail our build...
>
>
> On Sat, Apr 5, 2014 at 2:30 PM, Debasish Das <debasish.das83@gmail.com
> >wrote:
>
> > I verified this is happening for both CDH4.5 and 1.0.4...My deploy
> > environment is Java 6...so Java 7 compilation is not going to help...
> >
> > Is this the PR which caused it ?
> >
> > Andre Schumacher
> >
> > fbebaed Spark parquet improvements A few improvements to the
> Parquet
> > support for SQL queries: - Instead of files a ParquetRelation is now
> backed
> > by a directory, which simplifies importing data from other sources -
> > InsertIntoParquetTable operation now supports switching between
> overwriting
> > or appending (at least in HiveQL) - tests now use the new API - Parquet
> > logging can be set to WARNING level (Default) - Default compression for
> > Parquet files (GZIP, as in parquet-mr) Author: Andre Schumacher &... 2
> > days ago SPARK-1383
> >
> > I will go to a stable checkin before this
> >
> >
> >
> >
> > On Sat, Apr 5, 2014 at 2:22 PM, Debasish Das <debasish.das83@gmail.com
> > >wrote:
> >
> > > I can compile with Java 7...let me try that...
> > >
> > >
> > > On Sat, Apr 5, 2014 at 2:19 PM, Sean Owen <so...@cloudera.com> wrote:
> > >
> > >> That method was added in Java 7. The project is on Java 6, so I think
> > >> this was just an inadvertent error in a recent PR (it was the 'Spark
> > >> parquet improvements' one).
> > >>
> > >> I'll open a hot-fix PR after looking for other stuff like this that
> > >> might have snuck in.
> > >> --
> > >> Sean Owen | Director, Data Science | London
> > >>
> > >>
> > >> On Sat, Apr 5, 2014 at 10:04 PM, Debasish Das <
> debasish.das83@gmail.com
> > >
> > >> wrote:
> > >> > I am synced with apache/spark master but getting error in spark/sql
> > >> > compilation...
> > >> >
> > >> > Is the master broken ?
> > >> >
> > >> > [info] Compiling 34 Scala sources to
> > >> > /home/debasish/spark_deploy/sql/core/target/scala-2.10/classes...
> > >> > [error]
> > >> >
> > >>
> >
> /home/debasish/spark_deploy/sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetRelation.scala:106:
> > >> > value getGlobal is not a member of object java.util.logging.Logger
> > >> > [error] logger.setParent(Logger.getGlobal)
> > >> > [error] ^
> > >> > [error] one error found
> > >> > [error] (sql/compile:compile) Compilation failed
> > >> > [error] Total time: 171 s, completed Apr 5, 2014 4:58:41 PM
> > >> >
> > >> > Thanks.
> > >> > Deb
> > >>
> > >
> > >
> >
>
Re: Master compilation
Posted by Koert Kuipers <ko...@tresata.com>.
see here for similar issue
http://mail-archives.apache.org/mod_mbox/spark-user/201401.mbox/%3CCALNFXi2hBSyCkPpnBJBYJnPv3dSLNw8VpL_6caEn3yfXCykO=w@mail.gmail.com%3E
On Apr 6, 2014 4:10 PM, "Sean Owen" <so...@cloudera.com> wrote:
> scala.None certainly isn't new in 2.10.4; it's ancient :
> http://www.scala-lang.org/api/2.10.3/index.html#scala.None$
>
> Surely this is some other problem?
>
> On Sun, Apr 6, 2014 at 6:46 PM, Koert Kuipers <ko...@tresata.com> wrote:
> > also, i thought scala 2.10 was binary compatible, but does not seem to be
> > the case. the spark artifacts for scala 2.10.4 dont work for me, since we
> > are still on scala 2.10.3, but when i recompiled and published spark with
> > scala 2.10.3 everything was fine again.
> >
> > errors i see:
> > java.lang.ClassNotFoundException: scala.None$
> >
> > fun stuff!
>
Re: Master compilation
Posted by Sean Owen <so...@cloudera.com>.
scala.None certainly isn't new in 2.10.4; it's ancient :
http://www.scala-lang.org/api/2.10.3/index.html#scala.None$
Surely this is some other problem?
On Sun, Apr 6, 2014 at 6:46 PM, Koert Kuipers <ko...@tresata.com> wrote:
> also, i thought scala 2.10 was binary compatible, but does not seem to be
> the case. the spark artifacts for scala 2.10.4 dont work for me, since we
> are still on scala 2.10.3, but when i recompiled and published spark with
> scala 2.10.3 everything was fine again.
>
> errors i see:
> java.lang.ClassNotFoundException: scala.None$
>
> fun stuff!
Re: Master compilation
Posted by Koert Kuipers <ko...@tresata.com>.
i suggest we stick to 2.10.3, since otherwise it seems that (surprisingly)
you force everyone to upgrade
On Sun, Apr 6, 2014 at 1:46 PM, Koert Kuipers <ko...@tresata.com> wrote:
> also, i thought scala 2.10 was binary compatible, but does not seem to be
> the case. the spark artifacts for scala 2.10.4 dont work for me, since we
> are still on scala 2.10.3, but when i recompiled and published spark with
> scala 2.10.3 everything was fine again.
>
> errors i see:
> java.lang.ClassNotFoundException: scala.None$
>
> fun stuff!
>
>
> On Sun, Apr 6, 2014 at 12:13 PM, Koert Kuipers <ko...@tresata.com> wrote:
>
>> patrick,
>> this has happened before, that a commit introduced java 7
>> code/dependencies and your build didnt fail, i think it was when reynold
>> upgraded to jetty 9. must be that your entire build infrastructure runs
>> java 7...
>>
>>
>> On Sat, Apr 5, 2014 at 6:06 PM, Patrick Wendell <pw...@gmail.com>wrote:
>>
>>> If you want to submit a hot fix for this issue specifically please do.
>>> I'm
>>> not sure why it didn't fail our build...
>>>
>>>
>>> On Sat, Apr 5, 2014 at 2:30 PM, Debasish Das <debasish.das83@gmail.com
>>> >wrote:
>>>
>>> > I verified this is happening for both CDH4.5 and 1.0.4...My deploy
>>> > environment is Java 6...so Java 7 compilation is not going to help...
>>> >
>>> > Is this the PR which caused it ?
>>> >
>>> > Andre Schumacher
>>> >
>>> > fbebaed Spark parquet improvements A few improvements to the
>>> Parquet
>>> > support for SQL queries: - Instead of files a ParquetRelation is now
>>> backed
>>> > by a directory, which simplifies importing data from other sources -
>>> > InsertIntoParquetTable operation now supports switching between
>>> overwriting
>>> > or appending (at least in HiveQL) - tests now use the new API - Parquet
>>> > logging can be set to WARNING level (Default) - Default compression for
>>> > Parquet files (GZIP, as in parquet-mr) Author: Andre Schumacher &...
>>> 2
>>> > days ago SPARK-1383
>>> >
>>> > I will go to a stable checkin before this
>>> >
>>> >
>>> >
>>> >
>>> > On Sat, Apr 5, 2014 at 2:22 PM, Debasish Das <debasish.das83@gmail.com
>>> > >wrote:
>>> >
>>> > > I can compile with Java 7...let me try that...
>>> > >
>>> > >
>>> > > On Sat, Apr 5, 2014 at 2:19 PM, Sean Owen <so...@cloudera.com>
>>> wrote:
>>> > >
>>> > >> That method was added in Java 7. The project is on Java 6, so I
>>> think
>>> > >> this was just an inadvertent error in a recent PR (it was the 'Spark
>>> > >> parquet improvements' one).
>>> > >>
>>> > >> I'll open a hot-fix PR after looking for other stuff like this that
>>> > >> might have snuck in.
>>> > >> --
>>> > >> Sean Owen | Director, Data Science | London
>>> > >>
>>> > >>
>>> > >> On Sat, Apr 5, 2014 at 10:04 PM, Debasish Das <
>>> debasish.das83@gmail.com
>>> > >
>>> > >> wrote:
>>> > >> > I am synced with apache/spark master but getting error in
>>> spark/sql
>>> > >> > compilation...
>>> > >> >
>>> > >> > Is the master broken ?
>>> > >> >
>>> > >> > [info] Compiling 34 Scala sources to
>>> > >> > /home/debasish/spark_deploy/sql/core/target/scala-2.10/classes...
>>> > >> > [error]
>>> > >> >
>>> > >>
>>> >
>>> /home/debasish/spark_deploy/sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetRelation.scala:106:
>>> > >> > value getGlobal is not a member of object java.util.logging.Logger
>>> > >> > [error] logger.setParent(Logger.getGlobal)
>>> > >> > [error] ^
>>> > >> > [error] one error found
>>> > >> > [error] (sql/compile:compile) Compilation failed
>>> > >> > [error] Total time: 171 s, completed Apr 5, 2014 4:58:41 PM
>>> > >> >
>>> > >> > Thanks.
>>> > >> > Deb
>>> > >>
>>> > >
>>> > >
>>> >
>>>
>>
>>
>
Re: Master compilation
Posted by Koert Kuipers <ko...@tresata.com>.
also, i thought scala 2.10 was binary compatible, but does not seem to be
the case. the spark artifacts for scala 2.10.4 dont work for me, since we
are still on scala 2.10.3, but when i recompiled and published spark with
scala 2.10.3 everything was fine again.
errors i see:
java.lang.ClassNotFoundException: scala.None$
fun stuff!
On Sun, Apr 6, 2014 at 12:13 PM, Koert Kuipers <ko...@tresata.com> wrote:
> patrick,
> this has happened before, that a commit introduced java 7
> code/dependencies and your build didnt fail, i think it was when reynold
> upgraded to jetty 9. must be that your entire build infrastructure runs
> java 7...
>
>
> On Sat, Apr 5, 2014 at 6:06 PM, Patrick Wendell <pw...@gmail.com>wrote:
>
>> If you want to submit a hot fix for this issue specifically please do. I'm
>> not sure why it didn't fail our build...
>>
>>
>> On Sat, Apr 5, 2014 at 2:30 PM, Debasish Das <debasish.das83@gmail.com
>> >wrote:
>>
>> > I verified this is happening for both CDH4.5 and 1.0.4...My deploy
>> > environment is Java 6...so Java 7 compilation is not going to help...
>> >
>> > Is this the PR which caused it ?
>> >
>> > Andre Schumacher
>> >
>> > fbebaed Spark parquet improvements A few improvements to the
>> Parquet
>> > support for SQL queries: - Instead of files a ParquetRelation is now
>> backed
>> > by a directory, which simplifies importing data from other sources -
>> > InsertIntoParquetTable operation now supports switching between
>> overwriting
>> > or appending (at least in HiveQL) - tests now use the new API - Parquet
>> > logging can be set to WARNING level (Default) - Default compression for
>> > Parquet files (GZIP, as in parquet-mr) Author: Andre Schumacher &...
>> 2
>> > days ago SPARK-1383
>> >
>> > I will go to a stable checkin before this
>> >
>> >
>> >
>> >
>> > On Sat, Apr 5, 2014 at 2:22 PM, Debasish Das <debasish.das83@gmail.com
>> > >wrote:
>> >
>> > > I can compile with Java 7...let me try that...
>> > >
>> > >
>> > > On Sat, Apr 5, 2014 at 2:19 PM, Sean Owen <so...@cloudera.com> wrote:
>> > >
>> > >> That method was added in Java 7. The project is on Java 6, so I think
>> > >> this was just an inadvertent error in a recent PR (it was the 'Spark
>> > >> parquet improvements' one).
>> > >>
>> > >> I'll open a hot-fix PR after looking for other stuff like this that
>> > >> might have snuck in.
>> > >> --
>> > >> Sean Owen | Director, Data Science | London
>> > >>
>> > >>
>> > >> On Sat, Apr 5, 2014 at 10:04 PM, Debasish Das <
>> debasish.das83@gmail.com
>> > >
>> > >> wrote:
>> > >> > I am synced with apache/spark master but getting error in spark/sql
>> > >> > compilation...
>> > >> >
>> > >> > Is the master broken ?
>> > >> >
>> > >> > [info] Compiling 34 Scala sources to
>> > >> > /home/debasish/spark_deploy/sql/core/target/scala-2.10/classes...
>> > >> > [error]
>> > >> >
>> > >>
>> >
>> /home/debasish/spark_deploy/sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetRelation.scala:106:
>> > >> > value getGlobal is not a member of object java.util.logging.Logger
>> > >> > [error] logger.setParent(Logger.getGlobal)
>> > >> > [error] ^
>> > >> > [error] one error found
>> > >> > [error] (sql/compile:compile) Compilation failed
>> > >> > [error] Total time: 171 s, completed Apr 5, 2014 4:58:41 PM
>> > >> >
>> > >> > Thanks.
>> > >> > Deb
>> > >>
>> > >
>> > >
>> >
>>
>
>
Re: Master compilation
Posted by Koert Kuipers <ko...@tresata.com>.
patrick,
this has happened before, that a commit introduced java 7 code/dependencies
and your build didnt fail, i think it was when reynold upgraded to jetty 9.
must be that your entire build infrastructure runs java 7...
On Sat, Apr 5, 2014 at 6:06 PM, Patrick Wendell <pw...@gmail.com> wrote:
> If you want to submit a hot fix for this issue specifically please do. I'm
> not sure why it didn't fail our build...
>
>
> On Sat, Apr 5, 2014 at 2:30 PM, Debasish Das <debasish.das83@gmail.com
> >wrote:
>
> > I verified this is happening for both CDH4.5 and 1.0.4...My deploy
> > environment is Java 6...so Java 7 compilation is not going to help...
> >
> > Is this the PR which caused it ?
> >
> > Andre Schumacher
> >
> > fbebaed Spark parquet improvements A few improvements to the
> Parquet
> > support for SQL queries: - Instead of files a ParquetRelation is now
> backed
> > by a directory, which simplifies importing data from other sources -
> > InsertIntoParquetTable operation now supports switching between
> overwriting
> > or appending (at least in HiveQL) - tests now use the new API - Parquet
> > logging can be set to WARNING level (Default) - Default compression for
> > Parquet files (GZIP, as in parquet-mr) Author: Andre Schumacher &... 2
> > days ago SPARK-1383
> >
> > I will go to a stable checkin before this
> >
> >
> >
> >
> > On Sat, Apr 5, 2014 at 2:22 PM, Debasish Das <debasish.das83@gmail.com
> > >wrote:
> >
> > > I can compile with Java 7...let me try that...
> > >
> > >
> > > On Sat, Apr 5, 2014 at 2:19 PM, Sean Owen <so...@cloudera.com> wrote:
> > >
> > >> That method was added in Java 7. The project is on Java 6, so I think
> > >> this was just an inadvertent error in a recent PR (it was the 'Spark
> > >> parquet improvements' one).
> > >>
> > >> I'll open a hot-fix PR after looking for other stuff like this that
> > >> might have snuck in.
> > >> --
> > >> Sean Owen | Director, Data Science | London
> > >>
> > >>
> > >> On Sat, Apr 5, 2014 at 10:04 PM, Debasish Das <
> debasish.das83@gmail.com
> > >
> > >> wrote:
> > >> > I am synced with apache/spark master but getting error in spark/sql
> > >> > compilation...
> > >> >
> > >> > Is the master broken ?
> > >> >
> > >> > [info] Compiling 34 Scala sources to
> > >> > /home/debasish/spark_deploy/sql/core/target/scala-2.10/classes...
> > >> > [error]
> > >> >
> > >>
> >
> /home/debasish/spark_deploy/sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetRelation.scala:106:
> > >> > value getGlobal is not a member of object java.util.logging.Logger
> > >> > [error] logger.setParent(Logger.getGlobal)
> > >> > [error] ^
> > >> > [error] one error found
> > >> > [error] (sql/compile:compile) Compilation failed
> > >> > [error] Total time: 171 s, completed Apr 5, 2014 4:58:41 PM
> > >> >
> > >> > Thanks.
> > >> > Deb
> > >>
> > >
> > >
> >
>
Re: Master compilation
Posted by Patrick Wendell <pw...@gmail.com>.
If you want to submit a hot fix for this issue specifically please do. I'm
not sure why it didn't fail our build...
On Sat, Apr 5, 2014 at 2:30 PM, Debasish Das <de...@gmail.com>wrote:
> I verified this is happening for both CDH4.5 and 1.0.4...My deploy
> environment is Java 6...so Java 7 compilation is not going to help...
>
> Is this the PR which caused it ?
>
> Andre Schumacher
>
> fbebaed Spark parquet improvements A few improvements to the Parquet
> support for SQL queries: - Instead of files a ParquetRelation is now backed
> by a directory, which simplifies importing data from other sources -
> InsertIntoParquetTable operation now supports switching between overwriting
> or appending (at least in HiveQL) - tests now use the new API - Parquet
> logging can be set to WARNING level (Default) - Default compression for
> Parquet files (GZIP, as in parquet-mr) Author: Andre Schumacher &... 2
> days ago SPARK-1383
>
> I will go to a stable checkin before this
>
>
>
>
> On Sat, Apr 5, 2014 at 2:22 PM, Debasish Das <debasish.das83@gmail.com
> >wrote:
>
> > I can compile with Java 7...let me try that...
> >
> >
> > On Sat, Apr 5, 2014 at 2:19 PM, Sean Owen <so...@cloudera.com> wrote:
> >
> >> That method was added in Java 7. The project is on Java 6, so I think
> >> this was just an inadvertent error in a recent PR (it was the 'Spark
> >> parquet improvements' one).
> >>
> >> I'll open a hot-fix PR after looking for other stuff like this that
> >> might have snuck in.
> >> --
> >> Sean Owen | Director, Data Science | London
> >>
> >>
> >> On Sat, Apr 5, 2014 at 10:04 PM, Debasish Das <debasish.das83@gmail.com
> >
> >> wrote:
> >> > I am synced with apache/spark master but getting error in spark/sql
> >> > compilation...
> >> >
> >> > Is the master broken ?
> >> >
> >> > [info] Compiling 34 Scala sources to
> >> > /home/debasish/spark_deploy/sql/core/target/scala-2.10/classes...
> >> > [error]
> >> >
> >>
> /home/debasish/spark_deploy/sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetRelation.scala:106:
> >> > value getGlobal is not a member of object java.util.logging.Logger
> >> > [error] logger.setParent(Logger.getGlobal)
> >> > [error] ^
> >> > [error] one error found
> >> > [error] (sql/compile:compile) Compilation failed
> >> > [error] Total time: 171 s, completed Apr 5, 2014 4:58:41 PM
> >> >
> >> > Thanks.
> >> > Deb
> >>
> >
> >
>
Re: Master compilation
Posted by Debasish Das <de...@gmail.com>.
I verified this is happening for both CDH4.5 and 1.0.4...My deploy
environment is Java 6...so Java 7 compilation is not going to help...
Is this the PR which caused it ?
Andre Schumacher
fbebaed Spark parquet improvements A few improvements to the Parquet
support for SQL queries: - Instead of files a ParquetRelation is now backed
by a directory, which simplifies importing data from other sources -
InsertIntoParquetTable operation now supports switching between overwriting
or appending (at least in HiveQL) - tests now use the new API - Parquet
logging can be set to WARNING level (Default) - Default compression for
Parquet files (GZIP, as in parquet-mr) Author: Andre Schumacher &... 2
days ago SPARK-1383
I will go to a stable checkin before this
On Sat, Apr 5, 2014 at 2:22 PM, Debasish Das <de...@gmail.com>wrote:
> I can compile with Java 7...let me try that...
>
>
> On Sat, Apr 5, 2014 at 2:19 PM, Sean Owen <so...@cloudera.com> wrote:
>
>> That method was added in Java 7. The project is on Java 6, so I think
>> this was just an inadvertent error in a recent PR (it was the 'Spark
>> parquet improvements' one).
>>
>> I'll open a hot-fix PR after looking for other stuff like this that
>> might have snuck in.
>> --
>> Sean Owen | Director, Data Science | London
>>
>>
>> On Sat, Apr 5, 2014 at 10:04 PM, Debasish Das <de...@gmail.com>
>> wrote:
>> > I am synced with apache/spark master but getting error in spark/sql
>> > compilation...
>> >
>> > Is the master broken ?
>> >
>> > [info] Compiling 34 Scala sources to
>> > /home/debasish/spark_deploy/sql/core/target/scala-2.10/classes...
>> > [error]
>> >
>> /home/debasish/spark_deploy/sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetRelation.scala:106:
>> > value getGlobal is not a member of object java.util.logging.Logger
>> > [error] logger.setParent(Logger.getGlobal)
>> > [error] ^
>> > [error] one error found
>> > [error] (sql/compile:compile) Compilation failed
>> > [error] Total time: 171 s, completed Apr 5, 2014 4:58:41 PM
>> >
>> > Thanks.
>> > Deb
>>
>
>
Re: Master compilation
Posted by Debasish Das <de...@gmail.com>.
I can compile with Java 7...let me try that...
On Sat, Apr 5, 2014 at 2:19 PM, Sean Owen <so...@cloudera.com> wrote:
> That method was added in Java 7. The project is on Java 6, so I think
> this was just an inadvertent error in a recent PR (it was the 'Spark
> parquet improvements' one).
>
> I'll open a hot-fix PR after looking for other stuff like this that
> might have snuck in.
> --
> Sean Owen | Director, Data Science | London
>
>
> On Sat, Apr 5, 2014 at 10:04 PM, Debasish Das <de...@gmail.com>
> wrote:
> > I am synced with apache/spark master but getting error in spark/sql
> > compilation...
> >
> > Is the master broken ?
> >
> > [info] Compiling 34 Scala sources to
> > /home/debasish/spark_deploy/sql/core/target/scala-2.10/classes...
> > [error]
> >
> /home/debasish/spark_deploy/sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetRelation.scala:106:
> > value getGlobal is not a member of object java.util.logging.Logger
> > [error] logger.setParent(Logger.getGlobal)
> > [error] ^
> > [error] one error found
> > [error] (sql/compile:compile) Compilation failed
> > [error] Total time: 171 s, completed Apr 5, 2014 4:58:41 PM
> >
> > Thanks.
> > Deb
>
Re: Master compilation
Posted by Sean Owen <so...@cloudera.com>.
That method was added in Java 7. The project is on Java 6, so I think
this was just an inadvertent error in a recent PR (it was the 'Spark
parquet improvements' one).
I'll open a hot-fix PR after looking for other stuff like this that
might have snuck in.
--
Sean Owen | Director, Data Science | London
On Sat, Apr 5, 2014 at 10:04 PM, Debasish Das <de...@gmail.com> wrote:
> I am synced with apache/spark master but getting error in spark/sql
> compilation...
>
> Is the master broken ?
>
> [info] Compiling 34 Scala sources to
> /home/debasish/spark_deploy/sql/core/target/scala-2.10/classes...
> [error]
> /home/debasish/spark_deploy/sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetRelation.scala:106:
> value getGlobal is not a member of object java.util.logging.Logger
> [error] logger.setParent(Logger.getGlobal)
> [error] ^
> [error] one error found
> [error] (sql/compile:compile) Compilation failed
> [error] Total time: 171 s, completed Apr 5, 2014 4:58:41 PM
>
> Thanks.
> Deb