You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by yuanboliu <gi...@git.apache.org> on 2018/07/02 04:03:11 UTC

[GitHub] spark pull request #21690: [SPARK-24713]AppMatser of spark streaming kafka O...

GitHub user yuanboliu opened a pull request:

    https://github.com/apache/spark/pull/21690

    [SPARK-24713]AppMatser of spark streaming kafka OOM if there are hund…

    We have hundreds of kafka topics need to be consumed in one application. The application master will throw OOM exception after hanging for nearly half of an hour.
    
    OOM happens in the env with a lot of topics, and it's not convenient to set up such kind of env in the unit test. So I didn't change/add test case.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/yuanboliu/spark master

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/21690.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #21690
    
----
commit 15d03e63453e72675390c65ddf72492caefdadee
Author: yuanbo <yu...@...>
Date:   2018-07-02T03:58:53Z

    [SPARK-24713]AppMatser of spark streaming kafka OOM if there are hundreds of topics consumed

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

Posted by koeninger <gi...@git.apache.org>.

Github user koeninger commented on the issue:

    https://github.com/apache/spark/pull/21690
  
    @yuanboliu From reading KafkaConsumer code, and from testing, I don't see where consumer.position() alone would un-pause topicpartitions.  See below.  Can you give a counter-example?
    
    I am seeing poll() reset the paused state.  When you are having the problem, are you seeing the info level log messages "poll(0) returned messages"?
    
    If that's what's happening, I think the best we can do is call pause() in only one place, the first line of paranoidPoll, e.g.
    
    `c.pause(c.assignment)
    val msgs = c.poll(0)
    `
    
    
    
    Here's what I saw in testing:
    
    `scala> c.paused
    res34: java.util.Set[org.apache.kafka.common.TopicPartition] = []
    
    scala> c.assignment
    res35: java.util.Set[org.apache.kafka.common.TopicPartition] = [test-0]
    
    scala> c.pause(topics)
    
    scala> c.paused
    res37: java.util.Set[org.apache.kafka.common.TopicPartition] = [test-0]
    
    scala> c.position(tp)
    res38: Long = 248
    
    scala> c.paused
    res39: java.util.Set[org.apache.kafka.common.TopicPartition] = [test-0]
    
    scala> c.poll(0)
    res40: org.apache.kafka.clients.consumer.ConsumerRecords[String,String] = org.apache.kafka.clients.consumer.ConsumerRecords@20d7efbe
    
    scala> c.paused
    res41: java.util.Set[org.apache.kafka.common.TopicPartition] = [test-0]
    
    scala> c.position(tp)
    res42: Long = 248
    
    scala> c.paused
    res43: java.util.Set[org.apache.kafka.common.TopicPartition] = [test-0]
    
    scala> c.poll(1)
    res44: org.apache.kafka.clients.consumer.ConsumerRecords[String,String] = org.apache.kafka.clients.consumer.ConsumerRecords@20d7efbe
    
    scala> c.paused
    res45: java.util.Set[org.apache.kafka.common.TopicPartition] = [test-0]
    
    scala> c.poll(100)
    res46: org.apache.kafka.clients.consumer.ConsumerRecords[String,String] = org.apache.kafka.clients.consumer.ConsumerRecords@28e4439b
    
    scala> c.paused
    res47: java.util.Set[org.apache.kafka.common.TopicPartition] = []
    `


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21690
  
    **[Test build #92709 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92709/testReport)** for PR 21690 at commit [`d1a8c60`](https://github.com/apache/spark/commit/d1a8c605e163bc09d1329cbd90560cc5165de555).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

Posted by yuanboliu <gi...@git.apache.org>.

Github user yuanboliu commented on the issue:

    https://github.com/apache/spark/pull/21690
  
    Thanks very much


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

Posted by yuanboliu <gi...@git.apache.org>.

Github user yuanboliu commented on the issue:

    https://github.com/apache/spark/pull/21690
  
    @koeninger Sorry to interrupt, could you take a look at my patch?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

Posted by yuanboliu <gi...@git.apache.org>.

Github user yuanboliu commented on the issue:

    https://github.com/apache/spark/pull/21690
  
    The first pause is used to stop poll() in the method paranoidPoll
    The second one is attached because of p.partition().
    I'm not sure whether the state of pause will be rewritten after these methods are called, so I use pause repeatedly.  


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21690
  
    **[Test build #92528 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92528/testReport)** for PR 21690 at commit [`15d03e6`](https://github.com/apache/spark/commit/15d03e63453e72675390c65ddf72492caefdadee).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

Posted by koeninger <gi...@git.apache.org>.

Github user koeninger commented on the issue:

    https://github.com/apache/spark/pull/21690
  
    LGTM, merging to master.  Thanks!


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

Posted by yuanboliu <gi...@git.apache.org>.

Github user yuanboliu commented on the issue:

    https://github.com/apache/spark/pull/21690
  
    @koeninger  Thanks for your details.  Sorry quite busy this week. I will delete the last pause, test the patch on my own cluster this weekend and give feedback asap.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

Posted by koeninger <gi...@git.apache.org>.

Github user koeninger commented on the issue:

    https://github.com/apache/spark/pull/21690
  
    What results are you seeing?
    
    On Thu, Jul 12, 2018, 6:53 AM Yuanbo Liu <no...@github.com> wrote:
    
    > @koeninger <https://github.com/koeninger> Sorry to interrupt, could you
    > take a look at my patch?
    >
    > —
    > You are receiving this because you were mentioned.
    > Reply to this email directly, view it on GitHub
    > <https://github.com/apache/spark/pull/21690#issuecomment-404501493>, or mute
    > the thread
    > <https://github.com/notifications/unsubscribe-auth/AAGABxNyVlFhIo7z2D7jXtfeBfsVRsMmks5uF0bXgaJpZM4U-p4M>
    > .
    >



---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

Posted by yuanboliu <gi...@git.apache.org>.

Github user yuanboliu commented on the issue:

    https://github.com/apache/spark/pull/21690
  
    @koeninger Thanks for your reply. Agree with you. there is no need to to use pause repeatedly. 
    This is my test without any pause, and the app master stuck for a long time without any process
    
    
    
    ![wechatworkscreenshot_abb443bd-97db-48f9-88a2-e45a65617f80](https://user-images.githubusercontent.com/5643344/42409693-b324d45e-8210-11e8-96eb-39fc359b1b42.png)
    
    I will update my patch shortly.



---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

Posted by koeninger <gi...@git.apache.org>.

Github user koeninger commented on the issue:

    https://github.com/apache/spark/pull/21690
  
    @yuanboliu What I'm suggesting is more like this:
    
    https://github.com/apache/spark/compare/master...koeninger:SPARK-24713?expand=1


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21690
  
    Can one of the admins verify this patch?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21690
  
    Can one of the admins verify this patch?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21690
  
    **[Test build #92528 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92528/testReport)** for PR 21690 at commit [`15d03e6`](https://github.com/apache/spark/commit/15d03e63453e72675390c65ddf72492caefdadee).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21690
  
    **[Test build #92709 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92709/testReport)** for PR 21690 at commit [`d1a8c60`](https://github.com/apache/spark/commit/d1a8c605e163bc09d1329cbd90560cc5165de555).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21690
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92528/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21690: [SPARK-24713]AppMatser of spark streaming kafka OOM if t...

Posted by koeninger <gi...@git.apache.org>.

Github user koeninger commented on the issue:

    https://github.com/apache/spark/pull/21690
  
    @yuanboliu Can you clarify why repeated pause is necessary?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org