You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Richard Xin <ri...@yahoo.com.INVALID> on 2017/01/19 05:44:06 UTC
is partitionBy of DataFrameWriter supported in 1.6.x?
I found contradictions in document 1.6.0 and 2.1.x
in http://spark.apache.org/docs/1.6.0/api/scala/index.html#org.apache.spark.sql.DataFrameWriterit says: "This is only applicable for Parquet at the moment."
in http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.DataFrameWriterit says: "This was initially applicable for Parquet but in 1.5+ covers JSON, text, ORC and avro as well."
and I got warning when trying to save in scala:
> df.write.mode("overwrite").format("orc").partitionBy("date").saveAsTable("test.my_test")
17/01/19 13:34:43 WARN hive.HiveContext$$anon$2: Persisting partitioned data source relation `test`.`my_test` into Hive metastore in Spark SQL specific format, which is NOT compatible with Hive. Input path(s):
hdfs://nameservice1/user/hive/warehouse/test.db/my_test
looking at hdfs directories, the folders are there, but not selectable on both presto and hive.
any comments?
Thanks.
Re: is partitionBy of DataFrameWriter supported in 1.6.x?
Posted by Takeshi Yamamuro <li...@gmail.com>.
Hi,
In v1.6.0, it seems spark has supported `partitionBy` for JSON, text, ORC
and avro.
So, this is a bug of documents.
Actually, this bug was fixed in v1.6.1 (See:
https://github.com/apache/spark/commit/1005ee396f74dc4fcf127613b65e1abdb7f1934c
)
Also, AFAIK, this document only describes datasource tables in spark.
So, this is not related to Hive metastore tables, I think.
// maropu
On Thu, Jan 19, 2017 at 2:44 PM, Richard Xin <
richardxin168@yahoo.com.invalid> wrote:
> I found contradictions in document 1.6.0 and 2.1.x
>
> in http://spark.apache.org/docs/1.6.0/api/scala/index.html#
> org.apache.spark.sql.DataFrameWriter
> it says: "This is only applicable for Parquet at the moment."
>
> in http://spark.apache.org/docs/latest/api/scala/index.html#
> org.apache.spark.sql.DataFrameWriter
> it says: "This was initially applicable for Parquet but in 1.5+ covers
> JSON, text, ORC and avro as well."
>
> and I got warning when trying to save in scala:
>
> > df.write.mode("overwrite").format("orc").partitionBy("
> date").saveAsTable("test.my_test")
> 17/01/19 13:34:43 WARN hive.HiveContext$$anon$2: Persisting partitioned
> data source relation `test`.`my_test` into Hive metastore in Spark SQL
> specific format, which is NOT compatible with Hive. Input path(s):
> hdfs://nameservice1/user/hive/warehouse/test.db/my_test
>
> looking at hdfs directories, the folders are there, but not selectable on
> both presto and hive.
>
> any comments?
>
> Thanks.
>
--
---
Takeshi Yamamuro