You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/12/30 02:52:32 UTC
[GitHub] [spark] HyukjinKwon opened a new pull request #35065: [SPARK-37785][SQL][CORE] Add Utils.isAtExecutorSide
HyukjinKwon opened a new pull request #35065:
URL: https://github.com/apache/spark/pull/35065
### What changes were proposed in this pull request?
This PR proposes to add `Utils.isAtExecutorSide` to see if the codes are running on Executor (or Driver).
### Why are the changes needed?
There is currently no single call to see if we're on Executor (or Driver). `TaskContext.get == null` is being used for that way. We should better explicitly factor out to `Utils`.
### Does this PR introduce _any_ user-facing change?
No, dev-only.
### How was this patch tested?
Existing unittests should cover this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #35065: [SPARK-37785][SQL][CORE] Add Utils.isInRunningSparkTask
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #35065:
URL: https://github.com/apache/spark/pull/35065#issuecomment-1002952565
With the rename, it looks good to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on pull request #35065: [SPARK-37785][SQL][CORE] Add Utils.isInRunningSparkTask
Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on pull request #35065:
URL: https://github.com/apache/spark/pull/35065#issuecomment-1003046001
Thanks all!
Merged to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] JoshRosen commented on pull request #35065: [SPARK-37785][SQL][CORE] Add Utils.isAtExecutorSide
Posted by GitBox <gi...@apache.org>.
JoshRosen commented on pull request #35065:
URL: https://github.com/apache/spark/pull/35065#issuecomment-1002916194
Just brainstorming aloud here:
Is the name `isAtExecutorSide` potentially confusing in the context of `local` mode (since in that case we don't have separate executor processes)?
I suppose it makes sense if we think of "at the executor side" as meaning
> "in the context of a running Spark task"
instead of meaning
> "not in the driver JVM".
If we wanted to be more precise, maybe we could call this `Utils.inRunningSparkTask` or something like that.
I don't feel super strongly about this. I mostly just want to flag the potential for ambiguity in case a new developer uses this method expecting the "not in the driver JVM" semantic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #35065: [SPARK-37785][SQL][CORE] Add Utils.isInRunningSparkTask
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #35065:
URL: https://github.com/apache/spark/pull/35065
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #35065: [SPARK-37785][SQL][CORE] Add Utils.isAtExecutorSide
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #35065:
URL: https://github.com/apache/spark/pull/35065#issuecomment-1002928037
Yeah, I think it's better to be specific on what it means. Let me change like that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #35065: [SPARK-37785][SQL][CORE] Add Utils.isInRunningSparkTask
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #35065:
URL: https://github.com/apache/spark/pull/35065#issuecomment-1003046001
Merged to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org