[GitHub] spark issue #21588: [SPARK-24590][BUILD] Make Jenkins tests passed with hado...

[LiehuoChen <gi...@git.apache.org>]

Github user LiehuoChen commented on the issue:

    https://github.com/apache/spark/pull/21588
  
    Hi HyukjinKwon,
    Thanks for all the works to try to make the Jenkin test pass.
    I patched this PR to spark 2.4, and anything works fine but failed in org.apache.spark.deploy.yarn.YarnClusterSuite for following four unit tests:
    1). run Spark in yarn-cluster mode
    2). run Spark in yarn-cluster mode with different configurations, ensuring redaction
    3). run Spark in yarn-client mode
    4). run Spark in yarn-client mode with different configurations, ensuring redaction
    1), 2), failed everytime with really few useful error Msg, like:
    `FAILED did not equal FINISHED  Exception in thread "main" org.apache.spark.SparkException: Application application_1542090777201_0002 finished with failed status
    [info]   	at org.apache.spark.deploy.yarn.Client.run(Client.scala:1149)
    ......
    [info]   	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) (BaseYarnClusterSuite.scala:201)`
    2).4). succeed most of time, but sometimes failed on 
    ` Exception in thread "main" java.io.IOException: Server returned HTTP response code: 500 for URL: http://user-c02wq03ghtdg.corp.uber.com:61313/node/containerlogs/container_1541809642345_0002_01_000002/lhc/stdout?start=-4096`
    and `Fail to invoke HBaseConfiguration
    [info]   java.lang.ClassNotFoundException: org.apache.hadoop.hbase.HBaseConfiguration`
    
    Any you ever see the similars errors before? do you did any other fixes besides this PR to make the all test pass. 
    Thanks for your time.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org