You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by scwf <wa...@huawei.com> on 2014/08/15 05:01:17 UTC
mvn test error
env: ubuntu 14.04 + spark master buranch
mvn -Pyarn -Phive -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean package
mvn -Pyarn -Phadoop-2.4 -Phive test
test error:
DriverSuite:
Spark assembly has been built with Hive, including Datanucleus jars on classpath
- driver should exit after finishing *** FAILED ***
SparkException was thrown during property evaluation. (DriverSuite.scala:40)
Message: Process List(./bin/spark-class, org.apache.spark.DriverWithoutCleanup, local) exited with code 1
Occurred at table row 0 (zero based, not counting headings), which had values (
master = local
)
SparkSubmitSuite:
Spark assembly has been built with Hive, including Datanucleus jars on classpath
- launch simple application with spark-submit *** FAILED ***
org.apache.spark.SparkException: Process List(./bin/spark-submit, --class, org.apache.spark.deploy.SimpleApplicationTest, --name, testApp, --master, local, file:/tmp/1408015655220-0/testJar-1408015655220.jar) exited with code 1
at org.apache.spark.util.Utils$.executeAndGetOutput(Utils.scala:810)
at org.apache.spark.deploy.SparkSubmitSuite.runSparkSubmit(SparkSubmitSuite.scala:311)
at org.apache.spark.deploy.SparkSubmitSuite$$anonfun$14.apply$mcV$sp(SparkSubmitSuite.scala:291)
at org.apache.spark.deploy.SparkSubmitSuite$$anonfun$14.apply(SparkSubmitSuite.scala:284)
at org.apache.spark.deploy.SparkSubmitSuite$$anonfun$14.apply(SparkSubmitSuite.scala:284)
at org.scalatest.Transformer$$anonfun$apply$1.apply(Transformer.scala:22)
at org.scalatest.Transformer$$anonfun$apply$1.apply(Transformer.scala:22)
at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
at org.scalatest.Transformer.apply(Transformer.scala:22)
...
Spark assembly has been built with Hive, including Datanucleus jars on classpath
- spark submit includes jars passed in through --jar *** FAILED ***
org.apache.spark.SparkException: Process List(./bin/spark-submit, --class, org.apache.spark.deploy.JarCreationTest, --name, testApp, --master, local-cluster[2,1,512], --jars, file:/tmp/1408015659416-0/testJar-1408015659471.jar,fi
le:/tmp/1408015659472-0/testJar-1408015659513.jar, file:/tmp/1408015659415-0/testJar-1408015659416.jar) exited with code 1
at org.apache.spark.util.Utils$.executeAndGetOutput(Utils.scala:810)
at org.apache.spark.deploy.SparkSubmitSuite.runSparkSubmit(SparkSubmitSuite.scala:311)
at org.apache.spark.deploy.SparkSubmitSuite$$anonfun$15.apply$mcV$sp(SparkSubmitSuite.scala:305)
at org.apache.spark.deploy.SparkSubmitSuite$$anonfun$15.apply(SparkSubmitSuite.scala:294)
at org.apache.spark.deploy.SparkSubmitSuite$$anonfun$15.apply(SparkSubmitSuite.scala:294)
at org.scalatest.Transformer$$anonfun$apply$1.apply(Transformer.scala:22)
at org.scalatest.Transformer$$anonfun$apply$1.apply(Transformer.scala:22)
at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
at org.scalatest.Transformer.apply(Transformer.scala:22)
...
but only test the specific suite as follows will be ok:
mvn -Pyarn -Phadoop-2.4 -Phive -DwildcardSuites=org.apache.spark.DriverSuite test
it seems when run with "mvn -Pyarn -Phadoop-2.4 -Phive test",the process with Utils.executeAndGetOutput started can not exited successfully (exitcode is not zero)
anyone has idea for this?
--
Best Regards
Fei Wang
--------------------------------------------------------------------------------
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: mvn test error
Posted by Cheng Lian <li...@gmail.com>.
Just FYI, thought this might be helpful, I'm refactoring Hive Thrift server
test suites. These suites also fork new processes and suffer similar
issues. Stdout and stderr of forked processes are logged in the new version
of test suites with utilities under scala.sys.process package
https://github.com/apache/spark/pull/1856/files
On Tue, Aug 19, 2014 at 2:55 PM, scwf <wa...@huawei.com> wrote:
> hi,Cheng Lian
> thanks, printing stdout/stderr of the forked process is more reasonable.
>
> On 2014/8/19 13:35, Cheng Lian wrote:
>
>> The exception indicates that the forked process doesn’t executed as
>> expected, thus the test case /should/ fail.
>>
>> Instead of replacing the exception with a |logWarning|, capturing and
>> printing stdout/stderr of the forked process can be helpful for diagnosis.
>> Currently the only information we have at hand is the process exit code,
>> it’s hard to determine the reason why the forked process fails.
>>
>>
>>
>>
>>
>> On Tue, Aug 19, 2014 at 1:27 PM, scwf <wangfei1@huawei.com <mailto:
>> wangfei1@huawei.com>> wrote:
>>
>> hi, all
>> I notice that jenkins may also throw this error when running tests(
>> https://amplab.cs.__berkeley.edu/jenkins/job/__
>> SparkPullRequestBuilder/18688/__consoleFull <https://amplab.cs.berkeley.
>> edu/jenkins/job/SparkPullRequestBuilder/18688/consoleFull>).
>>
>>
>>
>> This is because in Utils.executeAndGetOutput our progress exitCode is
>> not 0, may be we should logWarning here rather than throw a exception?
>>
>> Utils.executeAndGetOutput {
>> val exitCode = process.waitFor()
>> stdoutThread.join() // Wait for it to finish reading output
>> if (exitCode != 0) {
>> throw new SparkException("Process " + command + " exited with
>> code " + exitCode)
>> }
>> }
>>
>> any idea?
>>
>>
>>
>> On 2014/8/15 11:01, scwf wrote:
>>
>> env: ubuntu 14.04 + spark master buranch
>>
>> mvn -Pyarn -Phive -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests
>> clean package
>>
>> mvn -Pyarn -Phadoop-2.4 -Phive test
>>
>> test error:
>>
>> DriverSuite:
>> Spark assembly has been built with Hive, including Datanucleus
>> jars on classpath
>> - driver should exit after finishing *** FAILED ***
>> SparkException was thrown during property evaluation.
>> (DriverSuite.scala:40)
>> Message: Process List(./bin/spark-class, org.apache.spark.__DriverWithoutCleanup,
>> local) exited with code 1
>>
>> Occurred at table row 0 (zero based, not counting
>> headings), which had values (
>> master = local
>> )
>>
>> SparkSubmitSuite:
>> Spark assembly has been built with Hive, including Datanucleus
>> jars on classpath
>> - launch simple application with spark-submit *** FAILED ***
>> org.apache.spark.__SparkException: Process
>> List(./bin/spark-submit, --class, org.apache.spark.deploy.__SimpleApplicationTest,
>> --name, testApp, --master, local, file:/tmp/1408015655220-0/__testJar-1408015655220.jar)
>> exited with code 1
>>
>> at org.apache.spark.util.Utils$._
>> _executeAndGetOutput(Utils.__scala:810)
>> at org.apache.spark.deploy.__SparkSubmitSuite.__
>> runSparkSubmit(__SparkSubmitSuite.scala:311)
>> at org.apache.spark.deploy.__SparkSubmitSuite$$anonfun$14._
>> _apply$mcV$sp(SparkSubmitSuite.__scala:291)
>> at org.apache.spark.deploy.__SparkSubmitSuite$$anonfun$14._
>> _apply(SparkSubmitSuite.scala:__284)
>> at org.apache.spark.deploy.__SparkSubmitSuite$$anonfun$14._
>> _apply(SparkSubmitSuite.scala:__284)
>> at org.scalatest.Transformer$$__anonfun$apply$1.apply(__
>> Transformer.scala:22)
>> at org.scalatest.Transformer$$__anonfun$apply$1.apply(__
>> Transformer.scala:22)
>> at org.scalatest.OutcomeOf$class.__outcomeOf(OutcomeOf.scala:
>> 85)
>> at org.scalatest.OutcomeOf$.__outcomeOf(OutcomeOf.scala:104)
>> at org.scalatest.Transformer.__apply(Transformer.scala:22)
>>
>> ...
>> Spark assembly has been built with Hive, including Datanucleus
>> jars on classpath
>> - spark submit includes jars passed in through --jar *** FAILED
>> ***
>> org.apache.spark.__SparkException: Process
>> List(./bin/spark-submit, --class, org.apache.spark.deploy.__JarCreationTest,
>> --name, testApp, --master, local-cluster[2,1,512], --jars,
>> file:/tmp/1408015659416-0/__testJar-1408015659471.jar,fi
>> le:/tmp/1408015659472-0/__testJar-1408015659513.jar,
>> file:/tmp/1408015659415-0/__testJar-1408015659416.jar) exited with code 1
>> at org.apache.spark.util.Utils$._
>> _executeAndGetOutput(Utils.__scala:810)
>> at org.apache.spark.deploy.__SparkSubmitSuite.__
>> runSparkSubmit(__SparkSubmitSuite.scala:311)
>> at org.apache.spark.deploy.__SparkSubmitSuite$$anonfun$15._
>> _apply$mcV$sp(SparkSubmitSuite.__scala:305)
>> at org.apache.spark.deploy.__SparkSubmitSuite$$anonfun$15._
>> _apply(SparkSubmitSuite.scala:__294)
>> at org.apache.spark.deploy.__SparkSubmitSuite$$anonfun$15._
>> _apply(SparkSubmitSuite.scala:__294)
>> at org.scalatest.Transformer$$__anonfun$apply$1.apply(__
>> Transformer.scala:22)
>> at org.scalatest.Transformer$$__anonfun$apply$1.apply(__
>> Transformer.scala:22)
>> at org.scalatest.OutcomeOf$class.__outcomeOf(OutcomeOf.scala:
>> 85)
>> at org.scalatest.OutcomeOf$.__outcomeOf(OutcomeOf.scala:104)
>> at org.scalatest.Transformer.__apply(Transformer.scala:22)
>>
>> ...
>>
>>
>> but only test the specific suite as follows will be ok:
>> mvn -Pyarn -Phadoop-2.4 -Phive -DwildcardSuites=org.apache.__spark.DriverSuite
>> test
>>
>>
>> it seems when run with "mvn -Pyarn -Phadoop-2.4 -Phive test",the
>> process with Utils.executeAndGetOutput started can not exited successfully
>> (exitcode is not zero)
>>
>> anyone has idea for this?
>>
>>
>>
>>
>>
>>
>> --
>>
>> Best Regards
>> Fei Wang
>>
>> ------------------------------__----------------------------
>> --__--------------------
>>
>>
>>
>> ------------------------------__----------------------------
>> --__---------
>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.__org <mailto:
>> dev-unsubscribe@spark.apache.org>
>> For additional commands, e-mail: dev-help@spark.apache.org <mailto:
>> dev-help@spark.apache.org>
>>
>>
>>
>
> --
>
> Best Regards
> Fei Wang
>
> ------------------------------------------------------------
> --------------------
>
>
>
Re: mvn test error
Posted by scwf <wa...@huawei.com>.
hi,Cheng Lian
thanks, printing stdout/stderr of the forked process is more reasonable.
On 2014/8/19 13:35, Cheng Lian wrote:
> The exception indicates that the forked process doesn’t executed as expected, thus the test case /should/ fail.
>
> Instead of replacing the exception with a |logWarning|, capturing and printing stdout/stderr of the forked process can be helpful for diagnosis. Currently the only information we have at hand is the process exit code, it’s hard to determine the reason why the forked process fails.
>
>
>
>
> On Tue, Aug 19, 2014 at 1:27 PM, scwf <wangfei1@huawei.com <ma...@huawei.com>> wrote:
>
> hi, all
> I notice that jenkins may also throw this error when running tests(https://amplab.cs.__berkeley.edu/jenkins/job/__SparkPullRequestBuilder/18688/__consoleFull <https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18688/consoleFull>).
>
>
> This is because in Utils.executeAndGetOutput our progress exitCode is not 0, may be we should logWarning here rather than throw a exception?
>
> Utils.executeAndGetOutput {
> val exitCode = process.waitFor()
> stdoutThread.join() // Wait for it to finish reading output
> if (exitCode != 0) {
> throw new SparkException("Process " + command + " exited with code " + exitCode)
> }
> }
>
> any idea?
>
>
>
> On 2014/8/15 11:01, scwf wrote:
>
> env: ubuntu 14.04 + spark master buranch
>
> mvn -Pyarn -Phive -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean package
>
> mvn -Pyarn -Phadoop-2.4 -Phive test
>
> test error:
>
> DriverSuite:
> Spark assembly has been built with Hive, including Datanucleus jars on classpath
> - driver should exit after finishing *** FAILED ***
> SparkException was thrown during property evaluation. (DriverSuite.scala:40)
> Message: Process List(./bin/spark-class, org.apache.spark.__DriverWithoutCleanup, local) exited with code 1
> Occurred at table row 0 (zero based, not counting headings), which had values (
> master = local
> )
>
> SparkSubmitSuite:
> Spark assembly has been built with Hive, including Datanucleus jars on classpath
> - launch simple application with spark-submit *** FAILED ***
> org.apache.spark.__SparkException: Process List(./bin/spark-submit, --class, org.apache.spark.deploy.__SimpleApplicationTest, --name, testApp, --master, local, file:/tmp/1408015655220-0/__testJar-1408015655220.jar) exited with code 1
>
> at org.apache.spark.util.Utils$.__executeAndGetOutput(Utils.__scala:810)
> at org.apache.spark.deploy.__SparkSubmitSuite.__runSparkSubmit(__SparkSubmitSuite.scala:311)
> at org.apache.spark.deploy.__SparkSubmitSuite$$anonfun$14.__apply$mcV$sp(SparkSubmitSuite.__scala:291)
> at org.apache.spark.deploy.__SparkSubmitSuite$$anonfun$14.__apply(SparkSubmitSuite.scala:__284)
> at org.apache.spark.deploy.__SparkSubmitSuite$$anonfun$14.__apply(SparkSubmitSuite.scala:__284)
> at org.scalatest.Transformer$$__anonfun$apply$1.apply(__Transformer.scala:22)
> at org.scalatest.Transformer$$__anonfun$apply$1.apply(__Transformer.scala:22)
> at org.scalatest.OutcomeOf$class.__outcomeOf(OutcomeOf.scala:85)
> at org.scalatest.OutcomeOf$.__outcomeOf(OutcomeOf.scala:104)
> at org.scalatest.Transformer.__apply(Transformer.scala:22)
> ...
> Spark assembly has been built with Hive, including Datanucleus jars on classpath
> - spark submit includes jars passed in through --jar *** FAILED ***
> org.apache.spark.__SparkException: Process List(./bin/spark-submit, --class, org.apache.spark.deploy.__JarCreationTest, --name, testApp, --master, local-cluster[2,1,512], --jars, file:/tmp/1408015659416-0/__testJar-1408015659471.jar,fi
> le:/tmp/1408015659472-0/__testJar-1408015659513.jar, file:/tmp/1408015659415-0/__testJar-1408015659416.jar) exited with code 1
> at org.apache.spark.util.Utils$.__executeAndGetOutput(Utils.__scala:810)
> at org.apache.spark.deploy.__SparkSubmitSuite.__runSparkSubmit(__SparkSubmitSuite.scala:311)
> at org.apache.spark.deploy.__SparkSubmitSuite$$anonfun$15.__apply$mcV$sp(SparkSubmitSuite.__scala:305)
> at org.apache.spark.deploy.__SparkSubmitSuite$$anonfun$15.__apply(SparkSubmitSuite.scala:__294)
> at org.apache.spark.deploy.__SparkSubmitSuite$$anonfun$15.__apply(SparkSubmitSuite.scala:__294)
> at org.scalatest.Transformer$$__anonfun$apply$1.apply(__Transformer.scala:22)
> at org.scalatest.Transformer$$__anonfun$apply$1.apply(__Transformer.scala:22)
> at org.scalatest.OutcomeOf$class.__outcomeOf(OutcomeOf.scala:85)
> at org.scalatest.OutcomeOf$.__outcomeOf(OutcomeOf.scala:104)
> at org.scalatest.Transformer.__apply(Transformer.scala:22)
> ...
>
>
> but only test the specific suite as follows will be ok:
> mvn -Pyarn -Phadoop-2.4 -Phive -DwildcardSuites=org.apache.__spark.DriverSuite test
>
> it seems when run with "mvn -Pyarn -Phadoop-2.4 -Phive test",the process with Utils.executeAndGetOutput started can not exited successfully (exitcode is not zero)
>
> anyone has idea for this?
>
>
>
>
>
>
> --
>
> Best Regards
> Fei Wang
>
> ------------------------------__------------------------------__--------------------
>
>
>
> ------------------------------__------------------------------__---------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.__org <ma...@spark.apache.org>
> For additional commands, e-mail: dev-help@spark.apache.org <ma...@spark.apache.org>
>
>
--
Best Regards
Fei Wang
--------------------------------------------------------------------------------
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: mvn test error
Posted by Cheng Lian <li...@gmail.com>.
The exception indicates that the forked process doesn’t executed as
expected, thus the test case *should* fail.
Instead of replacing the exception with a logWarning, capturing and
printing stdout/stderr of the forked process can be helpful for diagnosis.
Currently the only information we have at hand is the process exit code,
it’s hard to determine the reason why the forked process fails.
On Tue, Aug 19, 2014 at 1:27 PM, scwf <wa...@huawei.com> wrote:
> hi, all
> I notice that jenkins may also throw this error when running tests(
> https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18688/
> consoleFull).
>
>
> This is because in Utils.executeAndGetOutput our progress exitCode is not
> 0, may be we should logWarning here rather than throw a exception?
>
> Utils.executeAndGetOutput {
> val exitCode = process.waitFor()
> stdoutThread.join() // Wait for it to finish reading output
> if (exitCode != 0) {
> throw new SparkException("Process " + command + " exited with code "
> + exitCode)
> }
> }
>
> any idea?
>
>
>
> On 2014/8/15 11:01, scwf wrote:
>
>> env: ubuntu 14.04 + spark master buranch
>>
>> mvn -Pyarn -Phive -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean
>> package
>>
>> mvn -Pyarn -Phadoop-2.4 -Phive test
>>
>> test error:
>>
>> DriverSuite:
>> Spark assembly has been built with Hive, including Datanucleus jars on
>> classpath
>> - driver should exit after finishing *** FAILED ***
>> SparkException was thrown during property evaluation.
>> (DriverSuite.scala:40)
>> Message: Process List(./bin/spark-class, org.apache.spark.DriverWithoutCleanup,
>> local) exited with code 1
>> Occurred at table row 0 (zero based, not counting headings), which
>> had values (
>> master = local
>> )
>>
>> SparkSubmitSuite:
>> Spark assembly has been built with Hive, including Datanucleus jars on
>> classpath
>> - launch simple application with spark-submit *** FAILED ***
>> org.apache.spark.SparkException: Process List(./bin/spark-submit,
>> --class, org.apache.spark.deploy.SimpleApplicationTest, --name, testApp,
>> --master, local, file:/tmp/1408015655220-0/testJar-1408015655220.jar)
>> exited with code 1
>>
>> at org.apache.spark.util.Utils$.executeAndGetOutput(Utils.scala:810)
>> at org.apache.spark.deploy.SparkSubmitSuite.runSparkSubmit(
>> SparkSubmitSuite.scala:311)
>> at org.apache.spark.deploy.SparkSubmitSuite$$anonfun$14.
>> apply$mcV$sp(SparkSubmitSuite.scala:291)
>> at org.apache.spark.deploy.SparkSubmitSuite$$anonfun$14.
>> apply(SparkSubmitSuite.scala:284)
>> at org.apache.spark.deploy.SparkSubmitSuite$$anonfun$14.
>> apply(SparkSubmitSuite.scala:284)
>> at org.scalatest.Transformer$$anonfun$apply$1.apply(
>> Transformer.scala:22)
>> at org.scalatest.Transformer$$anonfun$apply$1.apply(
>> Transformer.scala:22)
>> at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
>> at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
>> at org.scalatest.Transformer.apply(Transformer.scala:22)
>> ...
>> Spark assembly has been built with Hive, including Datanucleus jars on
>> classpath
>> - spark submit includes jars passed in through --jar *** FAILED ***
>> org.apache.spark.SparkException: Process List(./bin/spark-submit,
>> --class, org.apache.spark.deploy.JarCreationTest, --name, testApp,
>> --master, local-cluster[2,1,512], --jars, file:/tmp/1408015659416-0/
>> testJar-1408015659471.jar,fi
>> le:/tmp/1408015659472-0/testJar-1408015659513.jar,
>> file:/tmp/1408015659415-0/testJar-1408015659416.jar) exited with code 1
>> at org.apache.spark.util.Utils$.executeAndGetOutput(Utils.scala:810)
>> at org.apache.spark.deploy.SparkSubmitSuite.runSparkSubmit(
>> SparkSubmitSuite.scala:311)
>> at org.apache.spark.deploy.SparkSubmitSuite$$anonfun$15.
>> apply$mcV$sp(SparkSubmitSuite.scala:305)
>> at org.apache.spark.deploy.SparkSubmitSuite$$anonfun$15.
>> apply(SparkSubmitSuite.scala:294)
>> at org.apache.spark.deploy.SparkSubmitSuite$$anonfun$15.
>> apply(SparkSubmitSuite.scala:294)
>> at org.scalatest.Transformer$$anonfun$apply$1.apply(
>> Transformer.scala:22)
>> at org.scalatest.Transformer$$anonfun$apply$1.apply(
>> Transformer.scala:22)
>> at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
>> at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
>> at org.scalatest.Transformer.apply(Transformer.scala:22)
>> ...
>>
>>
>> but only test the specific suite as follows will be ok:
>> mvn -Pyarn -Phadoop-2.4 -Phive -DwildcardSuites=org.apache.spark.DriverSuite
>> test
>>
>> it seems when run with "mvn -Pyarn -Phadoop-2.4 -Phive test",the process
>> with Utils.executeAndGetOutput started can not exited successfully
>> (exitcode is not zero)
>>
>> anyone has idea for this?
>>
>>
>>
>>
>>
>
> --
>
> Best Regards
> Fei Wang
>
> ------------------------------------------------------------
> --------------------
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>
Re: mvn test error
Posted by scwf <wa...@huawei.com>.
hi, all
I notice that jenkins may also throw this error when running tests(https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18688/consoleFull).
This is because in Utils.executeAndGetOutput our progress exitCode is not 0, may be we should logWarning here rather than throw a exception?
Utils.executeAndGetOutput {
val exitCode = process.waitFor()
stdoutThread.join() // Wait for it to finish reading output
if (exitCode != 0) {
throw new SparkException("Process " + command + " exited with code " + exitCode)
}
}
any idea?
On 2014/8/15 11:01, scwf wrote:
> env: ubuntu 14.04 + spark master buranch
>
> mvn -Pyarn -Phive -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean package
>
> mvn -Pyarn -Phadoop-2.4 -Phive test
>
> test error:
>
> DriverSuite:
> Spark assembly has been built with Hive, including Datanucleus jars on classpath
> - driver should exit after finishing *** FAILED ***
> SparkException was thrown during property evaluation. (DriverSuite.scala:40)
> Message: Process List(./bin/spark-class, org.apache.spark.DriverWithoutCleanup, local) exited with code 1
> Occurred at table row 0 (zero based, not counting headings), which had values (
> master = local
> )
>
> SparkSubmitSuite:
> Spark assembly has been built with Hive, including Datanucleus jars on classpath
> - launch simple application with spark-submit *** FAILED ***
> org.apache.spark.SparkException: Process List(./bin/spark-submit, --class, org.apache.spark.deploy.SimpleApplicationTest, --name, testApp, --master, local, file:/tmp/1408015655220-0/testJar-1408015655220.jar) exited with code 1
>
> at org.apache.spark.util.Utils$.executeAndGetOutput(Utils.scala:810)
> at org.apache.spark.deploy.SparkSubmitSuite.runSparkSubmit(SparkSubmitSuite.scala:311)
> at org.apache.spark.deploy.SparkSubmitSuite$$anonfun$14.apply$mcV$sp(SparkSubmitSuite.scala:291)
> at org.apache.spark.deploy.SparkSubmitSuite$$anonfun$14.apply(SparkSubmitSuite.scala:284)
> at org.apache.spark.deploy.SparkSubmitSuite$$anonfun$14.apply(SparkSubmitSuite.scala:284)
> at org.scalatest.Transformer$$anonfun$apply$1.apply(Transformer.scala:22)
> at org.scalatest.Transformer$$anonfun$apply$1.apply(Transformer.scala:22)
> at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
> at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
> at org.scalatest.Transformer.apply(Transformer.scala:22)
> ...
> Spark assembly has been built with Hive, including Datanucleus jars on classpath
> - spark submit includes jars passed in through --jar *** FAILED ***
> org.apache.spark.SparkException: Process List(./bin/spark-submit, --class, org.apache.spark.deploy.JarCreationTest, --name, testApp, --master, local-cluster[2,1,512], --jars, file:/tmp/1408015659416-0/testJar-1408015659471.jar,fi
> le:/tmp/1408015659472-0/testJar-1408015659513.jar, file:/tmp/1408015659415-0/testJar-1408015659416.jar) exited with code 1
> at org.apache.spark.util.Utils$.executeAndGetOutput(Utils.scala:810)
> at org.apache.spark.deploy.SparkSubmitSuite.runSparkSubmit(SparkSubmitSuite.scala:311)
> at org.apache.spark.deploy.SparkSubmitSuite$$anonfun$15.apply$mcV$sp(SparkSubmitSuite.scala:305)
> at org.apache.spark.deploy.SparkSubmitSuite$$anonfun$15.apply(SparkSubmitSuite.scala:294)
> at org.apache.spark.deploy.SparkSubmitSuite$$anonfun$15.apply(SparkSubmitSuite.scala:294)
> at org.scalatest.Transformer$$anonfun$apply$1.apply(Transformer.scala:22)
> at org.scalatest.Transformer$$anonfun$apply$1.apply(Transformer.scala:22)
> at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
> at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
> at org.scalatest.Transformer.apply(Transformer.scala:22)
> ...
>
>
> but only test the specific suite as follows will be ok:
> mvn -Pyarn -Phadoop-2.4 -Phive -DwildcardSuites=org.apache.spark.DriverSuite test
>
> it seems when run with "mvn -Pyarn -Phadoop-2.4 -Phive test",the process with Utils.executeAndGetOutput started can not exited successfully (exitcode is not zero)
>
> anyone has idea for this?
>
>
>
>
--
Best Regards
Fei Wang
--------------------------------------------------------------------------------
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org