You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (Jira)" <ji...@apache.org> on 2021/09/17 05:02:00 UTC
[jira] [Commented] (SPARK-36761) spark-examples_2.12-3.0.2.jar
DFSReadWriteTest S3A Implementation
[ https://issues.apache.org/jira/browse/SPARK-36761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17416453#comment-17416453 ]
Hyukjin Kwon commented on SPARK-36761:
--------------------------------------
does other spark application work? seems like it's an issue in your env.
> spark-examples_2.12-3.0.2.jar DFSReadWriteTest S3A Implementation
> -----------------------------------------------------------------
>
> Key: SPARK-36761
> URL: https://issues.apache.org/jira/browse/SPARK-36761
> Project: Spark
> Issue Type: Bug
> Components: Examples
> Affects Versions: 3.0.2
> Reporter: Raj
> Priority: Major
>
> Dear Team,
> I am using Spark3 to test the s3a storage writing. Part of test i am invoking DFSReadWriteTest from spark-examples_2.12-3.0.2.jar file. I am passing the arguments as below
> spark-submit --verbose --driver-java-options "-Dlog4j.configuration=file:/home/myid/log4j.properties" --conf "spark.executor.extraJavaOptions='-Dlog4j.configuration=file:/home/myid/log4j.properties'" --driver-class-path "/usr/hdp/3.1.5.0-152/hadoop/ceph-rgw-sts-auth-6.jar,/opt/spark3/jars/hadoop-aws-3.1.1.3.1.5.0-152.jar" --class org.apache.spark.examples.DFSReadWriteTest --deploy-mode client --executor-memory 1G --num-executors 3 --conf "spark.hadoop.fs.s3a.refreshTokenFile='/home/myid/keycloaktoken/tokenfile'" /opt/spark3/examples/jars/spark-examples_2.12-3.0.2.jar "/home/myid/sparkreadtest.txt" "s3a://d14/import/rajtestsp33"
> The Program fails with the message Wrong FS ( It seems the file system comparison fails in checkpath)
> Exception in thread "main" java.lang.IllegalArgumentException: Wrong FS: s3a://d14/import/rajtestsp33/dfs_read_write_test, expected: hdfs://pphdpException in thread "main" java.lang.IllegalArgumentException: Wrong FS: s3a://d14/import/rajtestsp33/dfs_read_write_test, expected: hdfs://pphdp at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:730) at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:234) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1577) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1574) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1589) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1683) at org.apache.spark.examples.DFSReadWriteTest$.main(DFSReadWriteTest.scala:115) at org.apache.spark.examples.DFSReadWriteTest.main(DFSReadWriteTest.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:928) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>
> And the same program successfully ran with the Spark2 with HDP 3.1.5.0-152 and spark-examples_2.11-2.3.2.3.1.5.0-152.jar file.
> Any inputs appreciated.
>
> Thanks
> Raj
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org