[jira] [Created] (SPARK-37821) spark thrift server RDD ID overflow lead sql execute failed

["muhong (Jira)" <ji...@apache.org>]

muhong created SPARK-37821:
------------------------------

             Summary: spark thrift server RDD ID overflow lead sql execute failed
                 Key: SPARK-37821
                 URL: https://issues.apache.org/jira/browse/SPARK-37821
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 3.2.0
            Reporter: muhong


this problem will happen in long run spark application，such as thrift server；

as only one SparkContext instance in thrift server driver size，so if the concurrency of sql request is large or the sql is too complicate（this will create a lot of rdd）, the rdd will be generate too fast , the rdd id （SparkContext.scala#nextRddId:[https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SparkContext.scala] ）will be consume fast, after a few months the nextRddId will overflow。the newRddId will be negative number，but the rdd's block id need to be positive, so this will lead a exception"Failed to parse rdd_-2123452330_2 into block ID"，so can not exchange data during sql execution, and lead sql execute failed



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org