[GitHub] [spark] mridulm edited a comment on pull request #30139: [SPARK-31069][CORE] high cpu caused by chunksBeingTransferred in external shuffle service

[GitBox <gi...@apache.org>]

mridulm edited a comment on pull request #30139:
URL: https://github.com/apache/spark/pull/30139#issuecomment-718528130


   If it will be eventually consistent, that is fine: what we can’t have is
   permanent divergence (positive or to negative).
   The current situation does exhibit eventual consistency (if updates while
   computing, then it diverges from reality - but that is transient).
   
   If the value will never converge, that is a problem.
   The current pr, last I checked, has potential for this and that needs to be
   fixed.
   
   (I have not checked the latest changes, so this might be fixed now)
   
   
   On Thu, Oct 29, 2020 at 2:14 AM Jungtaek Lim <no...@github.com>
   wrote:
   
   > I somehow revisited the related configuration
   > spark.shuffle.maxChunksBeingTransferred and realized the default value is
   > set to Long.MAX_VALUE. (effectively off by default)
   >
   > That leads me to wonder, does the value (numChunksBeingTransferred) need
   > to be accurate at the specific time? Do we really need to make both updates
   > be in same critical section? Can we tolerate the skew and allow eventual
   > consistent of the value?
   >
   > For the concern of being diverged, I guess updates to both would happen
   > anyway unless there's no bug (I agree this is a tricky one to guarantee),
   > as we anyway do the update on both sequentially. Please correct me if I'm
   > missing something.
   >
   > —
   > You are receiving this because you were mentioned.
   > Reply to this email directly, view it on GitHub
   > <https://github.com/apache/spark/pull/30139#issuecomment-718410305>, or
   > unsubscribe
   > <https://github.com/notifications/unsubscribe-auth/AAMETFEYOUBJFR2AAVQL4BLSNEI43ANCNFSM4S4G3CXQ>
   > .
   >
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org