[GitHub] [spark] xianyinxin opened a new pull request #25197: [SPARK-21067][SQL] Initialize the HdfsEncryptionShim and the underlying FileSystem(FS) e…

[GitBox <gi...@apache.org>]

xianyinxin opened a new pull request #25197: [SPARK-21067][SQL] Initialize the HdfsEncryptionShim and the underlying FileSystem(FS) e…
URL: https://github.com/apache/spark/pull/25197
 
 
   ## What changes were proposed in this pull request?
   
   This pr fix the "Filesystem Closed" exception that caused by the closing of a hive session in thriftserver.
   
   A `sessionState` holds a `hdfsEncryptionShim`, which holds a `fileSystem`. When a session was closed, the fileSystem may be closed. While the fileSystem is cached, so the subsequent using of the same fileSystem may throw "Filesystem closed" exception if previous session closes the fileSystem.
   
   We can move the "holding of the `fileSystem`" from session to thriftserver by triggering `SessionState#getHdfsEncryptionShim` eagerly at the starting phase of the thriftserver, see below,
   
   ```
   HiveThriftServer2#main()
       hiveClientImpl
           val state = newState
               SessionState.start(state)
                   SessionState.setCurrentSession(state)
           **state.getHdfsEncryptionShim**
   ```
   
   In hive session, the new sessionState will reuse the `fileSystem` created by thriftserver,
   
   ```
   CliService
       openSession
           HiveSessionImpl
               init
                   sessionState = new SessionState
                   SessionState.setCurrentSession(sessionState)
               executeStatementInternal
                   SessionState.setCurrentSession(sessionState)
                   operation.run()
                       hiveSession.getSessionManager().submitBackgroundOperation(operation)
                       **// If some code calls "state.getHdfsEncryptionShim", the cached `fileSystem` is reused.**
   ```
   
   ## How was this patch tested?
   
   manual test
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org