You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by 李书明 <al...@126.com> on 2019/07/11 12:01:11 UTC
Help: What's the biggest length of SQL that's supported in
SparkSQL?
I have a question about the limit(biggest) of SQL's length that is supported in SparkSQL. I can't find the answer in the documents of Spark.
Maybe Interger.MAX_VALUE or not ?
Re: Help: What's the biggest length of SQL that's supported in
SparkSQL?
Posted by Reynold Xin <rx...@databricks.com>.
No sorry I'm not at liberty to share other people's code.
On Fri, Jul 12, 2019 at 9:33 AM, Gourav Sengupta < gourav.sengupta@gmail.com > wrote:
>
> Hi Reynold,
>
>
> I am genuinely curious about queries which are more than 1 MB and am
> stunned by tens of MB's. Any samples to share :)
>
>
> Regards,
> Gourav
>
> On Thu, Jul 11, 2019 at 5:03 PM Reynold Xin < rxin@ databricks. com (
> rxin@databricks.com ) > wrote:
>
>
>> There is no explicit limit but a JVM string cannot be bigger than 2G. It
>> will also at some point run out of memory with too big of a query plan
>> tree or become incredibly slow due to query planning complexity. I've seen
>> queries that are tens of MBs in size.
>>
>>
>>
>>
>>
>>
>> On Thu, Jul 11, 2019 at 5:01 AM, 李书明 < alemmontree@ 126. com (
>> alemmontree@126.com ) > wrote:
>>
>>> I have a question about the limit(biggest) of SQL's length that is
>>> supported in SparkSQL. I can't find the answer in the documents of Spark.
>>>
>>>
>>> Maybe Interger.MAX_VALUE or not ?
>>>
>>
>>
>
>
Re: Help: What's the biggest length of SQL that's supported in
SparkSQL?
Posted by Reynold Xin <rx...@databricks.com>.
No sorry I'm not at liberty to share other people's code.
On Fri, Jul 12, 2019 at 9:33 AM, Gourav Sengupta < gourav.sengupta@gmail.com > wrote:
>
> Hi Reynold,
>
>
> I am genuinely curious about queries which are more than 1 MB and am
> stunned by tens of MB's. Any samples to share :)
>
>
> Regards,
> Gourav
>
> On Thu, Jul 11, 2019 at 5:03 PM Reynold Xin < rxin@ databricks. com (
> rxin@databricks.com ) > wrote:
>
>
>> There is no explicit limit but a JVM string cannot be bigger than 2G. It
>> will also at some point run out of memory with too big of a query plan
>> tree or become incredibly slow due to query planning complexity. I've seen
>> queries that are tens of MBs in size.
>>
>>
>>
>>
>>
>>
>> On Thu, Jul 11, 2019 at 5:01 AM, 李书明 < alemmontree@ 126. com (
>> alemmontree@126.com ) > wrote:
>>
>>> I have a question about the limit(biggest) of SQL's length that is
>>> supported in SparkSQL. I can't find the answer in the documents of Spark.
>>>
>>>
>>> Maybe Interger.MAX_VALUE or not ?
>>>
>>
>>
>
>
Re: Help: What's the biggest length of SQL that's supported in SparkSQL?
Posted by Gourav Sengupta <go...@gmail.com>.
Hi Reynold,
I am genuinely curious about queries which are more than 1 MB and am
stunned by tens of MB's. Any samples to share :)
Regards,
Gourav
On Thu, Jul 11, 2019 at 5:03 PM Reynold Xin <rx...@databricks.com> wrote:
> There is no explicit limit but a JVM string cannot be bigger than 2G. It
> will also at some point run out of memory with too big of a query plan tree
> or become incredibly slow due to query planning complexity. I've seen
> queries that are tens of MBs in size.
>
>
>
> On Thu, Jul 11, 2019 at 5:01 AM, 李书明 <al...@126.com> wrote:
>
>> I have a question about the limit(biggest) of SQL's length that is
>> supported in SparkSQL. I can't find the answer in the documents of Spark.
>>
>> Maybe Interger.MAX_VALUE or not ?
>>
>>
>
Re: Help: What's the biggest length of SQL that's supported in
SparkSQL?
Posted by Reynold Xin <rx...@databricks.com>.
There is no explicit limit but a JVM string cannot be bigger than 2G. It will also at some point run out of memory with too big of a query plan tree or become incredibly slow due to query planning complexity. I've seen queries that are tens of MBs in size.
On Thu, Jul 11, 2019 at 5:01 AM, 李书明 < alemmontree@126.com > wrote:
>
> I have a question about the limit(biggest) of SQL's length that is
> supported in SparkSQL. I can't find the answer in the documents of Spark.
>
>
> Maybe Interger.MAX_VALUE or not ?
>
>
>
>
Re: Help: What's the biggest length of SQL that's supported in
SparkSQL?
Posted by Reynold Xin <rx...@databricks.com>.
There is no explicit limit but a JVM string cannot be bigger than 2G. It will also at some point run out of memory with too big of a query plan tree or become incredibly slow due to query planning complexity. I've seen queries that are tens of MBs in size.
On Thu, Jul 11, 2019 at 5:01 AM, 李书明 < alemmontree@126.com > wrote:
>
> I have a question about the limit(biggest) of SQL's length that is
> supported in SparkSQL. I can't find the answer in the documents of Spark.
>
>
> Maybe Interger.MAX_VALUE or not ?
>
>
>
>