You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Vu Tan (Jira)" <ji...@apache.org> on 2022/01/24 07:32:00 UTC
[jira] [Created] (SPARK-37994) Unable to build spark3.2 with -Dhadoop.version=3.1.4

Vu Tan created SPARK-37994:
------------------------------

             Summary: Unable to build spark3.2 with -Dhadoop.version=3.1.4
                 Key: SPARK-37994
                 URL: https://issues.apache.org/jira/browse/SPARK-37994
             Project: Spark
          Issue Type: Bug
          Components: Build
    Affects Versions: 3.2.0
            Reporter: Vu Tan


I downloaded Spark 3.2 sourcecode from [https://github.com/apache/spark/archive/refs/tags/v3.2.0.zip]

and try building with the below command 
{code:java}
./dev/make-distribution.sh --name without-hadoop --pip --r --tgz -Psparkr -Phive -Phive-thriftserver -Phadoop-provided -Pyarn -Dhadoop.version=3.1.4 -Pkubernetes {code}
Then it gives the below error 
{code:java}
[INFO] --- scala-maven-plugin:4.3.0:compile (scala-compile-first) @ spark-core_2.12 ---
[INFO] Using incremental compilation using Mixed compile order
[INFO] Compiler bridge file: /Users/JP28431/.sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.12-1.3.1-bin_2.12.15__52.0-1.3.1_20191012T045515.jar
[INFO] compiler plugin: BasicArtifact(com.github.ghik,silencer-plugin_2.12.15,1.7.6,null)
[INFO] Compiling 567 Scala sources and 104 Java sources to /Users/JP28431/Downloads/spark-3.2.0-github/core/target/scala-2.12/classes ...
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:38: object io is not a member of package org.apache.hadoop
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:2778: not found: type ArrayWritable
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:2777: not found: type Writable
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SSLOptions.scala:24: object conf is not a member of package org.apache.hadoop
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SSLOptions.scala:174: not found: type Configuration
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SecurityManager.scala:25: object io is not a member of package org.apache.hadoop
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SecurityManager.scala:26: object security is not a member of package org.apache.hadoop
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala:33: object fs is not a member of package org.apache.hadoop
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala:32: object conf is not a member of package org.apache.hadoop
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala:121: not found: type Configuration
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SecurityManager.scala:284: not found: value UserGroupInformation
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:41: object mapreduce is not a member of package org.apache.hadoop
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:40: object mapreduce is not a member of package org.apache.hadoop
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:39: object mapred is not a member of package org.apache.hadoop
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:37: object fs is not a member of package org.apache.hadoop
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:36: object conf is not a member of package org.apache.hadoop
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SecurityManager.scala:348: not found: type Credentials
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SecurityManager.scala:350: not found: value UserGroupInformation
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SecurityManager.scala:402: not found: type Text
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SerializableWritable.scala:22: object conf is not a member of package org.apache.hadoop
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SerializableWritable.scala:23: object io is not a member of package org.apache.hadoop
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SerializableWritable.scala:24: object io is not a member of package org.apache.hadoop
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SerializableWritable.scala:30: not found: type Writable
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SerializableWritable.scala:38: not found: type ObjectWritable
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SerializableWritable.scala:43: not found: type ObjectWritable
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/internal/io/FileCommitProtocol.scala:20: object fs is not a member of package org.apache.hadoop
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/internal/io/FileCommitProtocol.scala:21: object mapreduce is not a member of package org.apache.hadoop
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:210: not found: type Configuration
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:306: not found: type Configuration
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala:26: object conf is not a member of package org.apache.hadoop
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala:55: not found: type Configuration
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1188: not found: type InputFormat
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1161: not found: type InputFormat
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1120: not found: type InputFormat
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:925: not found: type TextInputFormat
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:925: not found: type LongWritable
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:925: not found: type Text
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:968: not found: value NewHadoopJob
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:971: not found: value NewFileInputFormat
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala:31: object mapreduce is not a member of package org.apache.hadoop
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/WholeTextFileRDD.scala:21: object io is not a member of package org.apache.hadoop
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/WholeTextFileRDD.scala:39: not found: type Text
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/WholeTextFileRDD.scala:39: not found: type Text
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/input/WholeTextFileInputFormat.scala:25: object mapreduce is not a member of package org.apache.hadoop
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/input/WholeTextFileInputFormat.scala:34: not found: type CombineFileInputFormat
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/input/WholeTextFileRecordReader.scala:21: object conf is not a member of package org.apache.hadoop
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/input/WholeTextFileRecordReader.scala:32: not found: type HConfigurable
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/WholeTextFileRDD.scala:35: not found: type Text
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/WholeTextFileRDD.scala:36: not found: type Text
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/WholeTextFileRDD.scala:20: object conf is not a member of package org.apache.hadoop
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/WholeTextFileRDD.scala:37: not found: type Configuration
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:976: not found: type Text
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:977: not found: type Text
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1020: not found: value NewHadoopJob
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1023: not found: value NewFileInputFormat
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/BinaryFileRDD.scala:22: object mapreduce is not a member of package org.apache.hadoop
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/input/PortableDataStream.scala:28: object mapreduce is not a member of package org.apache.hadoop
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/input/PortableDataStream.scala:39: not found: type CombineFileInputFormat
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/BinaryFileRDD.scala:20: object conf is not a member of package org.apache.hadoop
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/BinaryFileRDD.scala:34: not found: type Configuration
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1050: not found: type Configuration
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1237: not found: type NewInputFormat
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/input/FixedLengthBinaryInputFormat.scala:23: object mapreduce is not a member of package org.apache.hadoop
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/input/FixedLengthBinaryInputFormat.scala:43: not found: type FileInputFormat
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1210: not found: type NewInputFormat
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1053: not found: type LongWritable
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1053: not found: type BytesWritable
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1055: not found: type LongWritable
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1056: not found: type BytesWritable
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1087: not found: type JobConf
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1088: not found: type InputFormat
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1096: class FileSystem in package io cannot be accessed in package java.io
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala:34: object mapred is not a member of package org.apache.hadoop
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala:131: not found: type JobConf
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala:29: object mapred is not a member of package org.apache.hadoop
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala:112: not found: type JobConf
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala:113: not found: type InputFormat
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala:99: not found: type JobConf
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala:100: not found: type InputFormat
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1128: not found: value FileSystem
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/util/SerializableConfiguration.scala:21: object conf is not a member of package org.apache.hadoop
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/util/SerializableConfiguration.scala:31: not found: type Configuration
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1132: not found: type JobConf
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1132: not found: value FileInputFormat
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1242: not found: type Configuration
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1247: not found: value FileSystem
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1251: not found: value NewHadoopJob
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1254: not found: value NewFileInputFormat
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala:73: not found: type InputFormat
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala:27: object conf is not a member of package org.apache.hadoop
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala:76: not found: type Configuration
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1278: not found: type Configuration
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1277: not found: type NewInputFormat
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1286: not found: value FileSystem
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1289: not found: type JobConf
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1315: not found: type SequenceFileInputFormat
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1375: not found: type SequenceFileInputFormat
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:3045: not found: type Writable
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1377: not found: type Writable
[ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1378: not found: type Writable
[INFO] [Info] : Any <: String?
[INFO] [Info] : false
[ERROR] 100 errors found
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Spark Project Parent POM 3.2.0:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [  4.964 s]
[INFO] Spark Project Tags ................................. SUCCESS [  8.381 s]
[INFO] Spark Project Sketch ............................... SUCCESS [  7.871 s]
[INFO] Spark Project Local DB ............................. SUCCESS [  2.191 s]
[INFO] Spark Project Networking ........................... SUCCESS [  4.879 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [  2.505 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [  8.310 s]
[INFO] Spark Project Launcher ............................. SUCCESS [  1.990 s]
[INFO] Spark Project Core ................................. FAILURE [ 45.343 s]
[INFO] Spark Project ML Local Library ..................... SKIPPED
[INFO] Spark Project GraphX ............................... SKIPPED
[INFO] Spark Project Streaming ............................ SKIPPED
[INFO] Spark Project Catalyst ............................. SKIPPED
[INFO] Spark Project SQL .................................. SKIPPED
[INFO] Spark Project ML Library ........................... SKIPPED
[INFO] Spark Project Tools ................................ SKIPPED
[INFO] Spark Project Hive ................................. SKIPPED
[INFO] Spark Project REPL ................................. SKIPPED
[INFO] Spark Project YARN Shuffle Service ................. SKIPPED
[INFO] Spark Project YARN ................................. SKIPPED
[INFO] Spark Project Kubernetes ........................... SKIPPED
[INFO] Spark Project Hive Thrift Server ................... SKIPPED
[INFO] Spark Project Assembly ............................. SKIPPED
[INFO] Kafka 0.10+ Token Provider for Streaming ........... SKIPPED
[INFO] Spark Integration for Kafka 0.10 ................... SKIPPED
[INFO] Kafka 0.10+ Source for Structured Streaming ........ SKIPPED
[INFO] Spark Project Examples ............................. SKIPPED
[INFO] Spark Integration for Kafka 0.10 Assembly .......... SKIPPED
[INFO] Spark Avro ......................................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:27 min
[INFO] Finished at: 2022-01-24T11:07:30+09:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:4.3.0:compile (scala-compile-first) on project spark-core_2.12: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:4.3.0:compile failed.: CompileFailed -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :spark-core_2.12 {code}

is this a expected behavior or should be considered a bug ?



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org