You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2023/01/02 08:43:49 UTC
[GitHub] [spark] adrian-wang opened a new pull request, #39343: [SPARK-41816][SQL][ThriftServer] Not close filesystem when log out ThriftServer
adrian-wang opened a new pull request, #39343:
URL: https://github.com/apache/spark/pull/39343
### What changes were proposed in this pull request?
Not close FileSystem instances when log out a sts session. The FileSystem instances will remain in FileSystem#CACHE, and auto close if CACHE is full.
### Why are the changes needed?
Currently when enabled impersonation, Spark Thriftserver will close filesystem instance for the user logged in when logout. If there are two sessions with the same user, the remaining session will become corrupted.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Tested in dev and production environment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #39343: [SPARK-41816][SQL] Not close filesystem when log out ThriftServer
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #39343:
URL: https://github.com/apache/spark/pull/39343#issuecomment-1369410210
cc @srowen @wangyum @yaooqinn @dongjoon-hyun @HyukjinKwon FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] adrian-wang commented on pull request #39343: [SPARK-41816][SQL] Not close filesystem when log out ThriftServer
Posted by GitBox <gi...@apache.org>.
adrian-wang commented on PR #39343:
URL: https://github.com/apache/spark/pull/39343#issuecomment-1374545224
Sorry I forgot the scenario I met that FS should not be closed was result from DeltaLog keeping FileSystem in a `lazy val`. After the session log out, the lazy val keeps the closed FileSystem instance and causing problems. Since current version of Delta has fixed this, I'm closing this pull request.
Thanks for your time!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #39343: [SPARK-41816][SQL][ThriftServer] Not close filesystem when log out ThriftServer
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #39343:
URL: https://github.com/apache/spark/pull/39343#issuecomment-1369393785
Yes, the original close method may close the fs held by other threads. If there is a simple way, I think we can manage fs by ourselves
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] adrian-wang closed pull request #39343: [SPARK-41816][SQL] Not close filesystem when log out ThriftServer
Posted by GitBox <gi...@apache.org>.
adrian-wang closed pull request #39343: [SPARK-41816][SQL] Not close filesystem when log out ThriftServer
URL: https://github.com/apache/spark/pull/39343
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] adrian-wang commented on pull request #39343: [SPARK-41816][SQL][ThriftServer] Not close filesystem when log out ThriftServer
Posted by GitBox <gi...@apache.org>.
adrian-wang commented on PR #39343:
URL: https://github.com/apache/spark/pull/39343#issuecomment-1369382022
@cloud-fan Thanks for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #39343: [SPARK-41816][SQL] Not close filesystem when log out ThriftServer
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on PR #39343:
URL: https://github.com/apache/spark/pull/39343#issuecomment-1371044732
cc @bogdanghit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #39343: [SPARK-41816][SQL][ThriftServer] Not close filesystem when log out ThriftServer
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #39343:
URL: https://github.com/apache/spark/pull/39343#issuecomment-1369344716
> and auto close if CACHE is full.
@adrian-wang Could you point out where this logic is? Both hadoop-2 and hadoop-3, In Spark 3.4.0, we still need to ensure that hadoop-2 is available, although there will be no distribution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #39343: [SPARK-41816][SQL] Not close filesystem when log out ThriftServer
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on PR #39343:
URL: https://github.com/apache/spark/pull/39343#issuecomment-1372103080
Since 2 different sessions from the same user have their own `sessionUgi`, does `FileSystem.closeAllForUGI` only close those instances belonging to the session?
Did a quick test with spark-shell script as follows
```scala
val ugi1 = UserGroupInformation.createRemoteUser("abc")
val fs1 = ugi1.doAs(new PrivilegedExceptionAction[FileSystem]() {
override def run: FileSystem = FileSystem.get(new Configuration)
})
val ugi2 = UserGroupInformation.createRemoteUser("abc")
val fs2 = ugi2.doAs(new PrivilegedExceptionAction[FileSystem]() {
override def run: FileSystem = FileSystem.get(new Configuration)
})
// as ugi1 != ugi2, only fs1 will be closed
FileSystem.closeAllForUGI(ugi1)
val fsNotClosed = ugi2.doAs(new PrivilegedExceptionAction[FileSystem]() {
override def run: FileSystem = FileSystem.get(new Configuration)
})
require(fs2 == fsNotClosed)
val fsRecreated = ugi1.doAs(new PrivilegedExceptionAction[FileSystem]() {
override def run: FileSystem = FileSystem.get(new Configuration)
})
require(fs1 != fsRecreated)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] adrian-wang commented on pull request #39343: [SPARK-41816][SQL][ThriftServer] Not close filesystem when log out ThriftServer
Posted by GitBox <gi...@apache.org>.
adrian-wang commented on PR #39343:
URL: https://github.com/apache/spark/pull/39343#issuecomment-1369385071
@LuciferYang Thanks for your review.
I found that there's no limit from Hadoop side, all fs intances are closed only when JVM shutdown if the we rely on fs auto close. Shall we manage the `FileSystem` instances by ourselves?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bogdanghit commented on pull request #39343: [SPARK-41816][SQL] Not close filesystem when log out ThriftServer
Posted by GitBox <gi...@apache.org>.
bogdanghit commented on PR #39343:
URL: https://github.com/apache/spark/pull/39343#issuecomment-1373499674
When is the filesystem closed after this change? What happens if the client doesn't close the filesystem?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org