[jira] [Commented] (SPARK-18202) Spark throws a mysterious system error when a Hive command has at least 100,000 results

["Martin Petricek (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-18202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15677072#comment-15677072 ] 

Martin Petricek commented on SPARK-18202:
-----------------------------------------

I encountered the same problem in 1.6.1 
It seems that there is no easy way to override this hardcoded default. While there is runHive(sql, maxResults) method, it is protected (and on the way from org.apache.spark.sql.SQLContext.sql(sqlText: String): DataFrame to that you encounter lot of protected/hidden under interface methods, so just simply overriding this hardcoded default is not possible).

> Spark throws a mysterious system error when a Hive command has at least 100,000 results
> ---------------------------------------------------------------------------------------
>
>                 Key: SPARK-18202
>                 URL: https://issues.apache.org/jira/browse/SPARK-18202
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.0.1
>            Reporter: Michael Allman
>
> There's a hard coded limit (100,000) to the number of results a hive command can return. If the number of results is greater than or equal this limit, Spark throws a system exception with the somewhat mysterious message "RESULTS POSSIBLY TRUNCATED".
> This problem manifests itself when calling `SHOW PARTITIONS <table>`, where <table> is a Hive table with not less than 100,000 partitions.
> The code in question can be found here:
> https://github.com/apache/spark/blob/ccb11543048dccd4cc590a8db1df1d9d5847d112/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala#L560-L566



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org