You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Wenchen Fan (JIRA)" <ji...@apache.org> on 2017/02/16 06:46:41 UTC
[jira] [Resolved] (SPARK-19618) Inconsistency wrt max. buckets
allowed from Dataframe API vs SQL
[ https://issues.apache.org/jira/browse/SPARK-19618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wenchen Fan resolved SPARK-19618.
---------------------------------
Resolution: Fixed
Fix Version/s: 2.2.0
Issue resolved by pull request 16948
[https://github.com/apache/spark/pull/16948]
> Inconsistency wrt max. buckets allowed from Dataframe API vs SQL
> ----------------------------------------------------------------
>
> Key: SPARK-19618
> URL: https://issues.apache.org/jira/browse/SPARK-19618
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.1.0
> Reporter: Tejas Patil
> Fix For: 2.2.0
>
>
> High number of buckets is allowed while creating a table via SQL query:
> {code}
> sparkSession.sql("""
> CREATE TABLE bucketed_table(col1 INT) USING parquet
> CLUSTERED BY (col1) SORTED BY (col1) INTO 147483647 BUCKETS
> """)
> sparkSession.sql("DESC FORMATTED bucketed_table").collect.foreach(println)
> ....
> [Num Buckets:,147483647,]
> [Bucket Columns:,[col1],]
> [Sort Columns:,[col1],]
> ....
> {code}
> Trying the same via dataframe API does not work:
> {code}
> > df.write.format("orc").bucketBy(147483647, "j","k").sortBy("j","k").saveAsTable("bucketed_table")
> java.lang.IllegalArgumentException: requirement failed: Bucket number must be greater than 0 and less than 100000.
> at scala.Predef$.require(Predef.scala:224)
> at org.apache.spark.sql.DataFrameWriter$$anonfun$getBucketSpec$2.apply(DataFrameWriter.scala:293)
> at org.apache.spark.sql.DataFrameWriter$$anonfun$getBucketSpec$2.apply(DataFrameWriter.scala:291)
> at scala.Option.map(Option.scala:146)
> at org.apache.spark.sql.DataFrameWriter.getBucketSpec(DataFrameWriter.scala:291)
> at org.apache.spark.sql.DataFrameWriter.createTable(DataFrameWriter.scala:429)
> at org.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:410)
> at org.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:365)
> ... 50 elided
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org