You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by sud_self <85...@qq.com> on 2015/03/28 06:42:50 UTC
rdd.toDF().saveAsParquetFile("tachyon://host:19998/test")
spark version is 1.3.0 with tanhyon-0.6.1
QUESTION DESCRIPTION: rdd.saveAsObjectFile("tachyon://host:19998/test") and
rdd.saveAsTextFile("tachyon://host:19998/test") succeed, but
rdd.toDF().saveAsParquetFile("tachyon://host:19998/test") failure.
ERROR MESSAGE: java.lang.IllegalArgumentException: Wrong FS:
tachyon://host:19998/test, expected: hdfs://host:8020
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:645)
at org.apache.hadoop.fs.FileSystem.makeQualified(FileSystem.java:465)
at
org.apache.spark.sql.parquet.ParquetRelation2$MetadataCache$$anonfun$6.apply(newParquet.scala:252)
at
org.apache.spark.sql.parquet.ParquetRelation2$MetadataCache$$anonfun$6.apply(newParquet.scala:251)
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/rdd-toDF-saveAsParquetFile-tachyon-host-19998-test-tp22264.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: rdd.toDF().saveAsParquetFile("tachyon://host:19998/test")
Posted by Yin Huai <yh...@databricks.com>.
You are hitting https://issues.apache.org/jira/browse/SPARK-6330. It has
been fixed in 1.3.1, which will be released soon.
On Fri, Mar 27, 2015 at 10:42 PM, sud_self <85...@qq.com> wrote:
> spark version is 1.3.0 with tanhyon-0.6.1
>
> QUESTION DESCRIPTION: rdd.saveAsObjectFile("tachyon://host:19998/test")
> and
> rdd.saveAsTextFile("tachyon://host:19998/test") succeed, but
> rdd.toDF().saveAsParquetFile("tachyon://host:19998/test") failure.
>
> ERROR MESSAGE: java.lang.IllegalArgumentException: Wrong FS:
> tachyon://host:19998/test, expected: hdfs://host:8020
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:645)
> at
> org.apache.hadoop.fs.FileSystem.makeQualified(FileSystem.java:465)
> at
>
> org.apache.spark.sql.parquet.ParquetRelation2$MetadataCache$$anonfun$6.apply(newParquet.scala:252)
> at
>
> org.apache.spark.sql.parquet.ParquetRelation2$MetadataCache$$anonfun$6.apply(newParquet.scala:251)
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/rdd-toDF-saveAsParquetFile-tachyon-host-19998-test-tp22264.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>