You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "sririshindra (via GitHub)" <gi...@apache.org> on 2025/02/05 16:59:36 UTC

[PR] [DRAFT] [SPARK-51095]. Include caller context for hdfs audit logs for calls from driver [spark]

sririshindra opened a new pull request, #49814:
URL: https://github.com/apache/spark/pull/49814

   <!--
   Thanks for sending a pull request!  Here are some tips for you:
     1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
     2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
     3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
     4. Be sure to keep the PR description updated to reflect all changes.
     5. Please write your PR title to summarize what this PR proposes.
     6. If possible, provide a concise example to reproduce the issue for a faster review.
     7. If you want to add a new configuration, please read the guideline first for naming configurations in
        'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
     8. If you want to add or modify an error type or message, please read the guideline first in
        'common/utils/src/main/resources/error/README.md'.
   -->
   
   ### What changes were proposed in this pull request?
   <!--
   Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue. 
   If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
     1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
     2. If you fix some SQL features, you can provide some references of other DBMSes.
     3. If there is design documentation, please add the link.
     4. If there is a discussion in the mailing list, please add the link.
   -->
   Add the caller context for calls from DRIVER to HDFS.
   
   ### Why are the changes needed?
   HDFS audit logs include the ability to add a "caller context".  Spark already leverages this to set the yarn application id, job id, task id, etc. but only on executors.  The caller context is left empty on the spark driver.  With introductions of Iceberg we have seen multiple scenarios in which files in HDFS are accessed from the driver. But since the caller context is left empty our ability to forensically analyse any issues has diminished. This PR includes sets caller context from the driver as well. 
   
   <!--
   Please clarify why the changes are needed. For instance,
     1. If you propose a new API, clarify the use case for a new API.
     2. If you fix a bug, you can clarify why it is a bug.
   -->
   
   
   ### Does this PR introduce _any_ user-facing change?
   Yes, hdfs audit logs will now have caller context for calls from driver.
   
   <!--
   Note that it means *any* user-facing change including all aspects such as new features, bug fixes, or other behavior changes. Documentation-only updates are not considered user-facing changes.
   
   If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
   If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
   If no, write 'No'.
   -->
   
   
   ### How was this patch tested?
   This patch was tested manually. After this change the hdfs audit logs now contain caller context from the driver.
   
   ```
   2025-02-04 21:45:12,488 INFO FSNamesystem.audit: allowed=true	ugi=iceberg (auth:SIMPLE)	ip=/10.140.144.131	cmd=create	src=/warehouse/tablespace/external/hive/mwies.db/berg/metadata/00534-a14ccf3e-e50c-4a69-84e3-262389720ae3.metadata.json	dst=null	perm=iceberg:hive:rw-rw----	proto=rpc	callerContext=SPARK_DRIVER__application_1738349944329_0054
   2025-02-04 21:45:12,578 INFO FSNamesystem.audit: allowed=true	ugi=iceberg (auth:SIMPLE)	ip=/10.140.144.131	cmd=delete	src=/warehouse/tablespace/external/hive/mwies.db/berg/metadata/00528-7ead607d-fa62-40e8-9508-f6a9c87d60e5.metadata.json	dst=null	perm=null	proto=rpc	callerContext=SPARK_DRIVER__application_1738349944329_0054
   2025-02-04 21:45:12,607 INFO FSNamesystem.audit: allowed=true	ugi=iceberg (auth:SIMPLE)	ip=/10.140.144.131	cmd=open	src=/warehouse/tablespace/external/hive/mwies.db/berg/metadata/00534-a14ccf3e-e50c-4a69-84e3-262389720ae3.metadata.json	dst=null	perm=null	proto=rpc	callerContext=SPARK_DRIVER__application_1738349944329_0054
   2025-02-04 21:45:12,636 INFO FSNamesystem.audit: allowed=true	ugi=iceberg (auth:SIMPLE)	ip=/10.140.144.131	cmd=open	src=/warehouse/tablespace/external/hive/mwies.db/berg/metadata/snap-4283759160550131345-1-4d208d89-e98a-4493-83a9-5421b46b3075.avro	dst=null	perm=null	proto=rpc	callerContext=SPARK_DRIVER__application_1738349944329_0054
   2025-02-04 21:45:12,639 INFO FSNamesystem.audit: allowed=true	ugi=iceberg (auth:SIMPLE)	ip=/10.140.144.131	cmd=getfileinfo	src=/warehouse/tablespace/external/hive/mwies.db/berg/metadata/snap-4283759160550131345-1-4d208d89-e98a-4493-83a9-5421b46b3075.avro	dst=null	perm=null	proto=rpc	callerContext=SPARK_DRIVER__application_1738349944329_0054
   ```
   
   <!--
   If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
   If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
   If tests were not added, please describe why they were not added and/or why it was difficult to add.
   If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
   -->
   
   
   ### Was this patch authored or co-authored using generative AI tooling?
   <!--
   If generative AI tooling has been used in the process of authoring this patch, please include the
   phrase: 'Generated-by: ' followed by the name of the tool and its version.
   If no, write 'No'.
   Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
   -->
   No


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51095][CORE][SQL] Include caller context for hdfs audit logs for calls from driver [spark]

Posted by "sririshindra (via GitHub)" <gi...@apache.org>.
sririshindra commented on PR #49814:
URL: https://github.com/apache/spark/pull/49814#issuecomment-2661324055

   @attilapiros , @dongjoon-hyun Can you please review this PR when you get a chance. Thanks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51095][CORE][SQL] Include caller context for hdfs audit logs for calls from driver [spark]

Posted by "sririshindra (via GitHub)" <gi...@apache.org>.
sririshindra commented on PR #49814:
URL: https://github.com/apache/spark/pull/49814#issuecomment-2638105639

   @attilapiros Could you take a look at this PR. There are some test failures apparently due to some scala style failures. But I couldn't quite figure out what they are and also the modules that seem to be failing don't seem to have anything to do with this PR


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51095][CORE][SQL] Include caller context for hdfs audit logs for calls from driver [spark]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #49814:
URL: https://github.com/apache/spark/pull/49814#discussion_r1955489715


##########
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala:
##########
@@ -171,6 +172,11 @@ private[hive] class HiveClientImpl(
   private def newState(): SessionState = {
     val hiveConf = newHiveConf(sparkConf, hadoopConf, extraConfig, Some(initClassLoader))
     val state = new SessionState(hiveConf)
+    // When SessionState is initialized, the caller context is overridden by hive
+    // so we need to reset it back to the DRIVER

Review Comment:
   @sririshindra Thanks for your effort in testing it! For Iceberg cases, you can use Spark 3.5 to test, AFAIK, there are no changes related to CallerContext in the Spark 4.0 period



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51095][CORE][SQL] Include caller context for hdfs audit logs for calls from driver [spark]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #49814:
URL: https://github.com/apache/spark/pull/49814#discussion_r1947465768


##########
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala:
##########
@@ -171,6 +172,11 @@ private[hive] class HiveClientImpl(
   private def newState(): SessionState = {
     val hiveConf = newHiveConf(sparkConf, hadoopConf, extraConfig, Some(initClassLoader))
     val state = new SessionState(hiveConf)
+    // When SessionState is initialized, the caller context is overridden by hive
+    // so we need to reset it back to the DRIVER

Review Comment:
   Are you using the customized version of Spark/Hive? The stacktrace does not match the Hive 2.3.10.
   
   I searched the Hive 2.3.10 codebase seems there are no places to set the caller context



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51095][CORE][SQL] Include caller context for hdfs audit logs for calls from driver [spark]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #49814:
URL: https://github.com/apache/spark/pull/49814#discussion_r1946442562


##########
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala:
##########
@@ -171,6 +172,11 @@ private[hive] class HiveClientImpl(
   private def newState(): SessionState = {
     val hiveConf = newHiveConf(sparkConf, hadoopConf, extraConfig, Some(initClassLoader))
     val state = new SessionState(hiveConf)
+    // When SessionState is initialized, the caller context is overridden by hive
+    // so we need to reset it back to the DRIVER
+    new CallerContext("DRIVER",
+      sparkConf.get(APP_CALLER_CONTEXT),
+      Option(sparkConf.getOption("spark.app.id").getOrElse(""))).setCurrentContext()

Review Comment:
   similar to above, appId won't be absent here
   ```suggestion
         Some(sparkConf.getAppId)).setCurrentContext()
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51095][CORE][SQL] Include caller context for hdfs audit logs for calls from driver [spark]

Posted by "cnauroth (via GitHub)" <gi...@apache.org>.
cnauroth commented on code in PR #49814:
URL: https://github.com/apache/spark/pull/49814#discussion_r1946824851


##########
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala:
##########
@@ -171,6 +172,11 @@ private[hive] class HiveClientImpl(
   private def newState(): SessionState = {
     val hiveConf = newHiveConf(sparkConf, hadoopConf, extraConfig, Some(initClassLoader))
     val state = new SessionState(hiveConf)
+    // When SessionState is initialized, the caller context is overridden by hive
+    // so we need to reset it back to the DRIVER

Review Comment:
   The caller context is ultimately captured in thread-local storage and propagated to child threads:
   
   https://github.com/apache/hadoop/blob/rel/release-3.4.1/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/CallerContext.java#L265
   
   Anything passing through Hive client initialization would override that with a Hive session ID, hence the need for the Spark code to restore it to "DRIVER":
   
   https://github.com/apache/hive/blob/rel/release-4.0.1/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java#L479
   
   @sririshindra , have you tried testing running multiple queries on Iceberg tables in the same Spark session? It might be interesting to see if Iceberg's Hive client has overridden it at the end, but we don't really see the effects until a second/third query.
   
   If we aren't seeing Iceberg's Hive client override it, then I think potential explanations could be: 1) the restore here is also past Iceberg's Hive client initialization, 2) Iceberg's client is on a separate thread with separate thread-local storage, or 3) Iceberg loads the client in an isolated classloader, so their `CallerContext` is not really the same as Spark's `CallerContext`. I don't know the Iceberg code path well enough to say if any of these theories are true though.
   
   From the perspective of HDFS/S3A auditing, what matters is the `CallerContext` at the time of the `FileSystem` API calls. The testing so far looks good for that.
   
   If we find there are broader problems with Spark's Hive clients leaking state, then an alternative solution might be to isolate `org.apache.hadoop.ipc.CallerContext` via the Hive client `IsolatedClientLoader`. Currently it treats all the Hadoop classes as shared.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51095][CORE][SQL] Include caller context for hdfs audit logs for calls from driver [spark]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #49814:
URL: https://github.com/apache/spark/pull/49814#discussion_r1946439901


##########
core/src/main/scala/org/apache/spark/SparkContext.scala:
##########
@@ -722,6 +722,9 @@ class SparkContext(config: SparkConf) extends Logging {
     }
     appStatusSource.foreach(_env.metricsSystem.registerSource(_))
     _plugins.foreach(_.registerMetrics(applicationId))
+
+    new CallerContext("DRIVER", config.get(APP_CALLER_CONTEXT),
+      Option(applicationId), applicationAttemptId).setCurrentContext()

Review Comment:
   `applicationId` won't be `null` here
   
   ```suggestion
         Some(applicationId), applicationAttemptId).setCurrentContext()
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51095][CORE][SQL] Include caller context for hdfs audit logs for calls from driver [spark]

Posted by "sririshindra (via GitHub)" <gi...@apache.org>.
sririshindra commented on code in PR #49814:
URL: https://github.com/apache/spark/pull/49814#discussion_r1950080047


##########
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala:
##########
@@ -171,6 +172,11 @@ private[hive] class HiveClientImpl(
   private def newState(): SessionState = {
     val hiveConf = newHiveConf(sparkConf, hadoopConf, extraConfig, Some(initClassLoader))
     val state = new SessionState(hiveConf)
+    // When SessionState is initialized, the caller context is overridden by hive
+    // so we need to reset it back to the DRIVER

Review Comment:
   @cnauroth , @pan3793 . So are you suggesting any changes? I am not sure I fully understand the reason why IsolatedClientLoader is not used, but sounds it is possible to use it. So it seems to me that the current Pr does not need any changes. Is my understanding correct?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51095][CORE][SQL] Include caller context for hdfs audit logs for calls from driver [spark]

Posted by "attilapiros (via GitHub)" <gi...@apache.org>.
attilapiros commented on PR #49814:
URL: https://github.com/apache/spark/pull/49814#issuecomment-2638235353

   The '__' in the PR description suggest the example was created with an older version of the PR, right?
   
   I.e. the `SPARK_DRIVER__application` in this line:
   ```
   2025-02-04 21:45:12,488 INFO FSNamesystem.audit: allowed=true	ugi=iceberg (auth:SIMPLE)	ip=/10.140.144.131	cmd=create	src=/warehouse/tablespace/external/hive/mwies.db/berg/metadata/00534-a14ccf3e-e50c-4a69-84e3-262389720ae3.metadata.json	dst=null	perm=iceberg:hive:rw-rw----	proto=rpc	callerContext=SPARK_DRIVER__application_1738349944329_0054
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51095][CORE][SQL] Include caller context for hdfs audit logs for calls from driver [spark]

Posted by "cnauroth (via GitHub)" <gi...@apache.org>.
cnauroth commented on code in PR #49814:
URL: https://github.com/apache/spark/pull/49814#discussion_r1947466506


##########
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala:
##########
@@ -171,6 +172,11 @@ private[hive] class HiveClientImpl(
   private def newState(): SessionState = {
     val hiveConf = newHiveConf(sparkConf, hadoopConf, extraConfig, Some(initClassLoader))
     val state = new SessionState(hiveConf)
+    // When SessionState is initialized, the caller context is overridden by hive
+    // so we need to reset it back to the DRIVER

Review Comment:
   @sririshindra , thanks so much for that additional testing! I remain +1 (non-binding), pending resolution of feedback from other reviewers.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51095][CORE][SQL] Include caller context for hdfs audit logs for calls from driver [spark]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #49814:
URL: https://github.com/apache/spark/pull/49814#discussion_r1947487051


##########
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala:
##########
@@ -171,6 +172,11 @@ private[hive] class HiveClientImpl(
   private def newState(): SessionState = {
     val hiveConf = newHiveConf(sparkConf, hadoopConf, extraConfig, Some(initClassLoader))
     val state = new SessionState(hiveConf)
+    // When SessionState is initialized, the caller context is overridden by hive
+    // so we need to reset it back to the DRIVER

Review Comment:
   > I searched the Hive 2.3.10 codebase seems there are no places to set the caller context
   
   @sririshindra BTW, I guess the change in HiveClientImpl is unnecessary for the current OSS spark



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51095][CORE][SQL] Include caller context for hdfs audit logs for calls from driver [spark]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #49814:
URL: https://github.com/apache/spark/pull/49814#discussion_r1951042904


##########
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala:
##########
@@ -171,6 +172,11 @@ private[hive] class HiveClientImpl(
   private def newState(): SessionState = {
     val hiveConf = newHiveConf(sparkConf, hadoopConf, extraConfig, Some(initClassLoader))
     val state = new SessionState(hiveConf)
+    // When SessionState is initialized, the caller context is overridden by hive
+    // so we need to reset it back to the DRIVER

Review Comment:
   @sririshindra thank you for sharing the progress



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51095][CORE][SQL] Include caller context for hdfs audit logs for calls from driver [spark]

Posted by "sririshindra (via GitHub)" <gi...@apache.org>.
sririshindra commented on PR #49814:
URL: https://github.com/apache/spark/pull/49814#issuecomment-2660017867

   
   
   
   > Note that due to recent changes in #49893 and #49898 , it should be possible to write a reliable unit test for this. Otherwise, I am +1 (non-binding).
   
   Thanks @cnauroth . I now added a unit test as well. 
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51095][CORE][SQL] Include caller context for hdfs audit logs for calls from driver [spark]

Posted by "cnauroth (via GitHub)" <gi...@apache.org>.
cnauroth commented on code in PR #49814:
URL: https://github.com/apache/spark/pull/49814#discussion_r1950100085


##########
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala:
##########
@@ -171,6 +172,11 @@ private[hive] class HiveClientImpl(
   private def newState(): SessionState = {
     val hiveConf = newHiveConf(sparkConf, hadoopConf, extraConfig, Some(initClassLoader))
     val state = new SessionState(hiveConf)
+    // When SessionState is initialized, the caller context is overridden by hive
+    // so we need to reset it back to the DRIVER

Review Comment:
   @sririshindra , I am not requesting any further changes. I remain +1, but my vote is non-binding. A Spark committer will need to complete the review to get the change into the repository.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51095][CORE][SQL] Include caller context for hdfs audit logs for calls from driver [spark]

Posted by "sririshindra (via GitHub)" <gi...@apache.org>.
sririshindra commented on code in PR #49814:
URL: https://github.com/apache/spark/pull/49814#discussion_r1951038298


##########
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala:
##########
@@ -171,6 +172,11 @@ private[hive] class HiveClientImpl(
   private def newState(): SessionState = {
     val hiveConf = newHiveConf(sparkConf, hadoopConf, extraConfig, Some(initClassLoader))
     val state = new SessionState(hiveConf)
+    // When SessionState is initialized, the caller context is overridden by hive
+    // so we need to reset it back to the DRIVER

Review Comment:
   Got it. I am working on testing on OSS Spark and hive. I tried working with the Hive docker image, but I had trouble configuring the HDFS audit logs in that Image. I also tried the https://github.com/awesome-kyuubi/hadoop-testing project. There I am having trouble building the docker images. I will figure out something on how to test it and post the results here once I have something.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51095][CORE][SQL] Include caller context for hdfs audit logs for calls from driver [spark]

Posted by "sririshindra (via GitHub)" <gi...@apache.org>.
sririshindra commented on code in PR #49814:
URL: https://github.com/apache/spark/pull/49814#discussion_r1955487024


##########
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala:
##########
@@ -171,6 +172,11 @@ private[hive] class HiveClientImpl(
   private def newState(): SessionState = {
     val hiveConf = newHiveConf(sparkConf, hadoopConf, extraConfig, Some(initClassLoader))
     val state = new SessionState(hiveConf)
+    // When SessionState is initialized, the caller context is overridden by hive
+    // so we need to reset it back to the DRIVER

Review Comment:
   @pan3793 , @cnauroth I was finally able to properly test on this upstream version in a docker based Cluster with this current branch. Looks like the change in hiveClinetImpl is not needed in Spark 4. I checked if the caller context is being set during the sessionSate initialization in HiveClinetImpl and it looks like it is not. So, Once the CallerContext is set inside the SparkContext class its is not being overridden by anything else from the Driver process. 
   
   ```
   2025-02-14 02:26:23,249 INFO FSNamesystem.audit: allowed=true   ugi=root (auth:SIMPLE)  ip=/192.168.97.4        cmd=getfileinfo src=/warehouse/sample   dst=null        perm=null       proto=rpc       callerContext=SPARK_DRIVER_application_1739496632907_0005
   2025-02-14 02:26:23,265 INFO FSNamesystem.audit: allowed=true   ugi=root (auth:SIMPLE)  ip=/192.168.97.4        cmd=listStatus  src=/warehouse/sample   dst=null        perm=null       proto=rpc       callerContext=SPARK_DRIVER_application_1739496632907_0005
   2025-02-14 02:26:25,519 INFO FSNamesystem.audit: allowed=true   ugi=root (auth:SIMPLE)  ip=/192.168.97.5        cmd=open        src=/warehouse/sample/part-00000-dd473344-76b1-4179-91ae-d15a8da4a888-c000      dst=null        perm=null       proto=rpc       callerContext=SPARK_TASK_application_1739496632907_0005_JId_0_SId_0_0_TId_0_0
   2025-02-14 02:26:26,345 INFO FSNamesystem.audit: allowed=true   ugi=root (auth:SIMPLE)  ip=/192.168.97.5        cmd=open        src=/warehouse/sample/part-00000-dd473344-76b1-4179-91ae-d15a8da4a888-c000      dst=null        perm=null       proto=rpc       callerContext=SPARK_TASK_application_1739496632907_0005_JId_1_SId_1_0_TId_1_0
   ```
   I wasn't able to test with Iceberg though. This is because Iceberg doesn't support Spark4 yet. Once an Iceberg release with Spark 4 support is released, I will retest it and make any changes needed in a separate PR. But For now, I removed the change that was in HiveClientImpl.scala .  



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51095][CORE][SQL] Include caller context for hdfs audit logs for calls from driver [spark]

Posted by "attilapiros (via GitHub)" <gi...@apache.org>.
attilapiros closed pull request #49814: [SPARK-51095][CORE][SQL] Include caller context for hdfs audit logs for calls from driver
URL: https://github.com/apache/spark/pull/49814


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51095][CORE][SQL] Include caller context for hdfs audit logs for calls from driver [spark]

Posted by "sririshindra (via GitHub)" <gi...@apache.org>.
sririshindra commented on PR #49814:
URL: https://github.com/apache/spark/pull/49814#issuecomment-2640817650

   > The '__' in the PR description suggest the example was created with an older version of the PR, right?
   > 
   > I.e. the `SPARK_DRIVER__application` in this line:
   > 
   > ```
   > 2025-02-04 21:45:12,488 INFO FSNamesystem.audit: allowed=true	ugi=iceberg (auth:SIMPLE)	ip=/10.140.144.131	cmd=create	src=/warehouse/tablespace/external/hive/mwies.db/berg/metadata/00534-a14ccf3e-e50c-4a69-84e3-262389720ae3.metadata.json	dst=null	perm=iceberg:hive:rw-rw----	proto=rpc	callerContext=SPARK_DRIVER__application_1738349944329_0054
   > ```
   
   Yes, it was from an older version. I reran the test with the latest version and updated the PR description with the new results. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51095][CORE][SQL] Include caller context for hdfs audit logs for calls from driver [spark]

Posted by "sririshindra (via GitHub)" <gi...@apache.org>.
sririshindra commented on code in PR #49814:
URL: https://github.com/apache/spark/pull/49814#discussion_r1947480397


##########
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala:
##########
@@ -171,6 +172,11 @@ private[hive] class HiveClientImpl(
   private def newState(): SessionState = {
     val hiveConf = newHiveConf(sparkConf, hadoopConf, extraConfig, Some(initClassLoader))
     val state = new SessionState(hiveConf)
+    // When SessionState is initialized, the caller context is overridden by hive
+    // so we need to reset it back to the DRIVER
+    new CallerContext("DRIVER",
+      sparkConf.get(APP_CALLER_CONTEXT),
+      Option(sparkConf.getOption("spark.app.id").getOrElse(""))).setCurrentContext()

Review Comment:
   Done. Sorry I missed it earlier. Fixed it in the latest commit.



##########
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala:
##########
@@ -171,6 +172,11 @@ private[hive] class HiveClientImpl(
   private def newState(): SessionState = {
     val hiveConf = newHiveConf(sparkConf, hadoopConf, extraConfig, Some(initClassLoader))
     val state = new SessionState(hiveConf)
+    // When SessionState is initialized, the caller context is overridden by hive
+    // so we need to reset it back to the DRIVER

Review Comment:
   Thanks @cnauroth .
   
   Yes, @pan3793 . I am using a customized version of Spark/Hive. I don't have a hadoop cluster with upstream versions of spark/hive. I had to test it in a customized version. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51095][CORE][SQL] Include caller context for hdfs audit logs for calls from driver [spark]

Posted by "sririshindra (via GitHub)" <gi...@apache.org>.
sririshindra commented on code in PR #49814:
URL: https://github.com/apache/spark/pull/49814#discussion_r1955508779


##########
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala:
##########
@@ -171,6 +172,11 @@ private[hive] class HiveClientImpl(
   private def newState(): SessionState = {
     val hiveConf = newHiveConf(sparkConf, hadoopConf, extraConfig, Some(initClassLoader))
     val state = new SessionState(hiveConf)
+    // When SessionState is initialized, the caller context is overridden by hive
+    // so we need to reset it back to the DRIVER

Review Comment:
   Thanks for the suggestion @pan3793 . I cherry picked this change to spark3.5.4 and tested it with Iceberg and it works correctly. 
   
   ```
   2025-02-14 03:28:14,596 INFO FSNamesystem.audit: allowed=true   ugi=root (auth:SIMPLE)  ip=/192.168.97.4        cmd=open        src=/warehouse/sample1/metadata/00000-f686c743-a610-4cb0-9fb4-5a3ca65c7696.metadata.json        dst=null        perm=null       proto=rpc       callerContext=SPARK_DRIVER_application_1739496632907_0008
   2025-02-14 03:28:22,558 INFO FSNamesystem.audit: allowed=true   ugi=root (auth:SIMPLE)  ip=/192.168.97.7        cmd=create      src=/warehouse/sample1/data/00000-0-cd51fcf5-b94e-4fb3-9ff8-23db99a03ae4-0-00001.parquet        dst=null        perm=root:hadoop:rw-r--r--      proto=rpc       callerContext=SPARK_TASK_application_1739496632907_0008_JId_0_SId_0_0_TId_0_0
   2025-02-14 03:28:22,984 INFO FSNamesystem.audit: allowed=true   ugi=root (auth:SIMPLE)  ip=/192.168.97.4        cmd=create      src=/warehouse/sample1/metadata/ebc9320e-92b4-40fb-9f64-07e3424c6d75-m0.avro    dst=null        perm=root:hadoop:rw-r--r--      proto=rpc       callerContext=SPARK_DRIVER_application_1739496632907_0008
   2025-02-14 03:28:23,071 INFO FSNamesystem.audit: allowed=true   ugi=root (auth:SIMPLE)  ip=/192.168.97.4        cmd=create      src=/warehouse/sample1/metadata/snap-7578694066380923258-1-ebc9320e-92b4-40fb-9f64-07e3424c6d75.avro    dst=null        perm=root:hadoop:rw-r--r--      proto=rpc       callerContext=SPARK_DRIVER_application_1739496632907_0008
   2025-02-14 03:28:23,113 INFO FSNamesystem.audit: allowed=true   ugi=root (auth:SIMPLE)  ip=/192.168.97.4        cmd=create      src=/warehouse/sample1/metadata/00001-2d907fe8-40df-4618-81d8-ef7fa25c3da1.metadata.json        dst=null        perm=root:hadoop:rw-r--r--      proto=rpc       callerContext=SPARK_DRIVER_application_1739496632907_0008
   2025-02-14 03:28:23,254 INFO FSNamesystem.audit: allowed=true   ugi=root (auth:SIMPLE)  ip=/192.168.97.4        cmd=open        src=/warehouse/sample1/metadata/snap-7578694066380923258-1-ebc9320e-92b4-40fb-9f64-07e3424c6d75.avro    dst=null        perm=null       proto=rpc       callerContext=SPARK_DRIVER_application_1739496632907_0008
   2025-02-14 03:28:23,255 INFO FSNamesystem.audit: allowed=true   ugi=root (auth:SIMPLE)  ip=/192.168.97.4        cmd=getfileinfo src=/warehouse/sample1/metadata/snap-7578694066380923258-1-ebc9320e-92b4-40fb-9f64-07e3424c6d75.avro    dst=null        perm=null       proto=rpc       callerContext=SPARK_DRIVER_application_1739496632907_0008
   2025-02-14 03:28:23,284 INFO FSNamesystem.audit: allowed=true   ugi=root (auth:SIMPLE)  ip=/192.168.97.4        cmd=open        src=/warehouse/sample1/metadata/00001-2d907fe8-40df-4618-81d8-ef7fa25c3da1.metadata.json        dst=null        perm=null       proto=rpc       callerContext=SPARK_DRIVER_application_1739496632907_0008
   2025-02-14 03:28:58,189 INFO FSNamesystem.audit: allowed=true   ugi=root (auth:SIMPLE)  ip=/192.168.97.4        cmd=open        src=/warehouse/sample1/metadata/00001-2d907fe8-40df-4618-81d8-ef7fa25c3da1.metadata.json        dst=null        perm=null       proto=rpc       callerContext=SPARK_DRIVER_application_1739496632907_0008
   2025-02-14 03:28:58,302 INFO FSNamesystem.audit: allowed=true   ugi=root (auth:SIMPLE)  ip=/192.168.97.4        cmd=open        src=/warehouse/sample1/metadata/snap-7578694066380923258-1-ebc9320e-92b4-40fb-9f64-07e3424c6d75.avro    dst=null        perm=null       proto=rpc       callerContext=SPARK_DRIVER_application_1739496632907_0008
   2025-02-14 03:28:58,305 INFO FSNamesystem.audit: allowed=true   ugi=root (auth:SIMPLE)  ip=/192.168.97.4        cmd=getfileinfo src=/warehouse/sample1/metadata/snap-7578694066380923258-1-ebc9320e-92b4-40fb-9f64-07e3424c6d75.avro    dst=null        perm=null       proto=rpc       callerContext=SPARK_DRIVER_application_1739496632907_0008
   2025-02-14 03:28:58,332 INFO FSNamesystem.audit: allowed=true   ugi=root (auth:SIMPLE)  ip=/192.168.97.4        cmd=open        src=/warehouse/sample1/metadata/ebc9320e-92b4-40fb-9f64-07e3424c6d75-m0.avro    dst=null        perm=null       proto=rpc       callerContext=SPARK_DRIVER_application_1739496632907_0008
   2025-02-14 03:28:58,373 INFO FSNamesystem.audit: allowed=true   ugi=root (auth:SIMPLE)  ip=/192.168.97.4        cmd=open        src=/warehouse/sample1/data/00000-0-cd51fcf5-b94e-4fb3-9ff8-23db99a03ae4-0-00001.parquet        dst=null        perm=null       proto=rpc       callerContext=SPARK_DRIVER_application_1739496632907_0008
   2025-02-14 03:28:58,401 INFO FSNamesystem.audit: allowed=true   ugi=root (auth:SIMPLE)  ip=/192.168.97.4        cmd=open        src=/warehouse/sample1/data/00000-0-cd51fcf5-b94e-4fb3-9ff8-23db99a03ae4-0-00001.parquet        dst=null        perm=null       proto=rpc       callerContext=SPARK_DRIVER_application_1739496632907_0008
   2025-02-14 03:28:58,837 INFO FSNamesystem.audit: allowed=true   ugi=root (auth:SIMPLE)  ip=/192.168.97.7        cmd=getfileinfo src=/warehouse/sample1/data/00000-0-cd51fcf5-b94e-4fb3-9ff8-23db99a03ae4-0-00001.parquet        dst=null        perm=null       proto=rpc       callerContext=SPARK_TASK_application_1739496632907_0008_JId_1_SId_1_0_TId_1_0
   2025-02-14 03:28:58,842 INFO FSNamesystem.audit: allowed=true   ugi=root (auth:SIMPLE)  ip=/192.168.97.7        cmd=open        src=/warehouse/sample1/data/00000-0-cd51fcf5-b94e-4fb3-9ff8-23db99a03ae4-0-00001.parquet        dst=null        perm=null       proto=rpc       callerContext=SPARK_TASK_application_1739496632907_0008_JId_1_SId_1_0_TId_1_0
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51095][CORE][SQL] Include caller context for hdfs audit logs for calls from driver [spark]

Posted by "cnauroth (via GitHub)" <gi...@apache.org>.
cnauroth commented on code in PR #49814:
URL: https://github.com/apache/spark/pull/49814#discussion_r1947982262


##########
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala:
##########
@@ -171,6 +172,11 @@ private[hive] class HiveClientImpl(
   private def newState(): SessionState = {
     val hiveConf = newHiveConf(sparkConf, hadoopConf, extraConfig, Some(initClassLoader))
     val state = new SessionState(hiveConf)
+    // When SessionState is initialized, the caller context is overridden by hive
+    // so we need to reset it back to the DRIVER

Review Comment:
   @pan3793 , thank you for pointing out that info on `IsolatedClientLoader`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51095][CORE][SQL] Include caller context for hdfs audit logs for calls from driver [spark]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #49814:
URL: https://github.com/apache/spark/pull/49814#discussion_r1946435692


##########
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala:
##########
@@ -171,6 +172,11 @@ private[hive] class HiveClientImpl(
   private def newState(): SessionState = {
     val hiveConf = newHiveConf(sparkConf, hadoopConf, extraConfig, Some(initClassLoader))
     val state = new SessionState(hiveConf)
+    // When SessionState is initialized, the caller context is overridden by hive
+    // so we need to reset it back to the DRIVER

Review Comment:
   iceberg constructs its own hive client instance, how does the provided caller context propagate?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51095][CORE][SQL] Include caller context for hdfs audit logs for calls from driver [spark]

Posted by "sririshindra (via GitHub)" <gi...@apache.org>.
sririshindra commented on code in PR #49814:
URL: https://github.com/apache/spark/pull/49814#discussion_r1946734535


##########
core/src/main/scala/org/apache/spark/SparkContext.scala:
##########
@@ -722,6 +722,9 @@ class SparkContext(config: SparkConf) extends Logging {
     }
     appStatusSource.foreach(_env.metricsSystem.registerSource(_))
     _plugins.foreach(_.registerMetrics(applicationId))
+
+    new CallerContext("DRIVER", config.get(APP_CALLER_CONTEXT),
+      Option(applicationId), applicationAttemptId).setCurrentContext()

Review Comment:
   Done



##########
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala:
##########
@@ -171,6 +172,11 @@ private[hive] class HiveClientImpl(
   private def newState(): SessionState = {
     val hiveConf = newHiveConf(sparkConf, hadoopConf, extraConfig, Some(initClassLoader))
     val state = new SessionState(hiveConf)
+    // When SessionState is initialized, the caller context is overridden by hive
+    // so we need to reset it back to the DRIVER
+    new CallerContext("DRIVER",
+      sparkConf.get(APP_CALLER_CONTEXT),
+      Option(sparkConf.getOption("spark.app.id").getOrElse(""))).setCurrentContext()

Review Comment:
   Done.



##########
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala:
##########
@@ -171,6 +172,11 @@ private[hive] class HiveClientImpl(
   private def newState(): SessionState = {
     val hiveConf = newHiveConf(sparkConf, hadoopConf, extraConfig, Some(initClassLoader))
     val state = new SessionState(hiveConf)
+    // When SessionState is initialized, the caller context is overridden by hive
+    // so we need to reset it back to the DRIVER

Review Comment:
   I see, I didn't know that iceberg constructs its own hive client instance. I am not quite sure how it is being propagated. But I know it is being propagated properly based on testing. I tested this by running both batch and streaming jobs that do writes and reads from an Iceberg table.  
   
   The way I identified the best location to set the caller context is by first identifying all the places it is being set elsewhere by setting a breakpoint on the setCurrent method in the CallerContext.java file in hadoop library. I then let a spark streaming (and also a batch) job on a cluster write/read from an iceberg table and I recorded the stack traces where setCurrent method is being called by attaching a remote debugger. I then identified the shared execution paths in all these stacktraces and I decided that the the best place to reset the caller context in spark is here in the HiveClientImpl file after the SessionState is initialized.
   
   Like you said may be iceberg constructs its own hive instance but maybe that codepath doesn't set the caller context? I have checked the Iceberg codebase and I have found the location where a hive client is being created, but I am not sure if that in turn calls [newClinet](https://github.com/apache/iceberg/blob/afda8be25652d44d9339a79c6797b6bf20c55bd6/hive-metastore/src/main/java/org/apache/iceberg/hive/HiveClientPool.java#L60) setCurrent method. 
   
   Here are the stacktraces where the method setCurrent is being set in my testing.
   
   ```
   Breakpoint reached
   	at org.apache.hadoop.ipc.CallerContext.setCurrent(CallerContext.java:148)
   	at org.apache.hadoop.hive.shims.Hadoop23Shims.setHadoopCallerContext(Hadoop23Shims.java:546)
   	at org.apache.hadoop.hive.shims.Hadoop23Shims.setHadoopSessionContext(Hadoop23Shims.java:556)
   	at org.apache.hadoop.hive.ql.session.SessionState.<init>(SessionState.java:508)
   	at org.apache.hadoop.hive.ql.session.SessionState.<init>(SessionState.java:475)
   	at org.apache.hadoop.hive.ql.session.SessionState.<init>(SessionState.java:471)
   	at org.apache.spark.sql.hive.client.HiveClientImpl.newState(HiveClientImpl.scala:195)
   	at org.apache.spark.sql.hive.client.HiveClientImpl.<init>(HiveClientImpl.scala:143)
   	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(NativeConstructorAccessorImpl.java:-1)
   	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
   	at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
   	at java.lang.reflect.Constructor.newInstance(Constructor.java:490)
   	at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:316)
   	at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:539)
   	at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:399)
   	at org.apache.spark.sql.hive.HiveExternalCatalog.client$lzycompute(HiveExternalCatalog.scala:70)
   	at org.apache.spark.sql.hive.HiveExternalCatalog.client(HiveExternalCatalog.scala:69)
   	at org.apache.spark.sql.hive.HiveExternalCatalog.$anonfun$databaseExists$1(HiveExternalCatalog.scala:223)
   	at org.apache.spark.sql.hive.HiveExternalCatalog$$Lambda$2058.2002899646.apply$mcZ$sp(Unknown Source:-1)
   	at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23)
   	at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:101)
   	at org.apache.spark.sql.hive.HiveExternalCatalog.databaseExists(HiveExternalCatalog.scala:223)
   	at org.apache.spark.sql.internal.SharedState.externalCatalog$lzycompute(SharedState.scala:150)
   	at org.apache.spark.sql.internal.SharedState.externalCatalog(SharedState.scala:144)
   	at org.apache.spark.sql.internal.SharedState.globalTempViewManager$lzycompute(SharedState.scala:180)
   	at org.apache.spark.sql.internal.SharedState.globalTempViewManager(SharedState.scala:178)
   	at org.apache.spark.sql.hive.HiveSessionStateBuilder.$anonfun$catalog$2(HiveSessionStateBuilder.scala:70)
   	at org.apache.spark.sql.hive.HiveSessionStateBuilder$$Lambda$1492.867369720.apply(Unknown Source:-1)
   	at org.apache.spark.sql.catalyst.catalog.SessionCatalog.globalTempViewManager$lzycompute(SessionCatalog.scala:123)
   	at org.apache.spark.sql.catalyst.catalog.SessionCatalog.globalTempViewManager(SessionCatalog.scala:123)
   	at org.apache.spark.sql.catalyst.catalog.SessionCatalog.isGlobalTempViewDB(SessionCatalog.scala:1001)
   	at org.apache.spark.sql.catalyst.catalog.SessionCatalog.getRawLocalOrGlobalTempView(SessionCatalog.scala:716)
   	at org.apache.spark.sql.catalyst.catalog.SessionCatalog.isTempView(SessionCatalog.scala:1008)
   	at org.apache.spark.sql.streaming.DataStreamWriter.toTable(DataStreamWriter.scala:282)
   	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-1)
   	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   	at java.lang.reflect.Method.invoke(Method.java:566)
   	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
   	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
   	at py4j.Gateway.invoke(Gateway.java:282)
   	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
   	at py4j.commands.CallCommand.execute(CallCommand.java:79)
   	at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
   	at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
   	at java.lang.Thread.run(Thread.java:834)
   Breakpoint reached at org.apache.hadoop.ipc.CallerContext.setCurrent(CallerContext.java:148)
   Breakpoint reached
   	at org.apache.hadoop.ipc.CallerContext.setCurrent(CallerContext.java:148)
   	at org.apache.hadoop.hive.shims.Hadoop23Shims.setHadoopCallerContext(Hadoop23Shims.java:546)
   	at org.apache.hadoop.hive.shims.Hadoop23Shims.setHadoopSessionContext(Hadoop23Shims.java:556)
   	at org.apache.hadoop.hive.ql.session.SessionState.<init>(SessionState.java:510)
   	at org.apache.hadoop.hive.ql.session.SessionState.<init>(SessionState.java:475)
   	at org.apache.hadoop.hive.ql.session.SessionState.<init>(SessionState.java:471)
   	at org.apache.spark.sql.hive.client.HiveClientImpl.newState(HiveClientImpl.scala:195)
   	at org.apache.spark.sql.hive.client.HiveClientImpl.<init>(HiveClientImpl.scala:143)
   	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(NativeConstructorAccessorImpl.java:-1)
   	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
   	at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
   	at java.lang.reflect.Constructor.newInstance(Constructor.java:490)
   	at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:316)
   	at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:539)
   	at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:399)
   	at org.apache.spark.sql.hive.HiveExternalCatalog.client$lzycompute(HiveExternalCatalog.scala:70)
   	at org.apache.spark.sql.hive.HiveExternalCatalog.client(HiveExternalCatalog.scala:69)
   	at org.apache.spark.sql.hive.HiveExternalCatalog.$anonfun$databaseExists$1(HiveExternalCatalog.scala:223)
   	at org.apache.spark.sql.hive.HiveExternalCatalog$$Lambda$2058.2002899646.apply$mcZ$sp(Unknown Source:-1)
   	at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23)
   	at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:101)
   	at org.apache.spark.sql.hive.HiveExternalCatalog.databaseExists(HiveExternalCatalog.scala:223)
   	at org.apache.spark.sql.internal.SharedState.externalCatalog$lzycompute(SharedState.scala:150)
   	at org.apache.spark.sql.internal.SharedState.externalCatalog(SharedState.scala:144)
   	at org.apache.spark.sql.internal.SharedState.globalTempViewManager$lzycompute(SharedState.scala:180)
   	at org.apache.spark.sql.internal.SharedState.globalTempViewManager(SharedState.scala:178)
   	at org.apache.spark.sql.hive.HiveSessionStateBuilder.$anonfun$catalog$2(HiveSessionStateBuilder.scala:70)
   	at org.apache.spark.sql.hive.HiveSessionStateBuilder$$Lambda$1492.867369720.apply(Unknown Source:-1)
   	at org.apache.spark.sql.catalyst.catalog.SessionCatalog.globalTempViewManager$lzycompute(SessionCatalog.scala:123)
   	at org.apache.spark.sql.catalyst.catalog.SessionCatalog.globalTempViewManager(SessionCatalog.scala:123)
   	at org.apache.spark.sql.catalyst.catalog.SessionCatalog.isGlobalTempViewDB(SessionCatalog.scala:1001)
   	at org.apache.spark.sql.catalyst.catalog.SessionCatalog.getRawLocalOrGlobalTempView(SessionCatalog.scala:716)
   	at org.apache.spark.sql.catalyst.catalog.SessionCatalog.isTempView(SessionCatalog.scala:1008)
   	at org.apache.spark.sql.streaming.DataStreamWriter.toTable(DataStreamWriter.scala:282)
   	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-1)
   	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   	at java.lang.reflect.Method.invoke(Method.java:566)
   	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
   	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
   	at py4j.Gateway.invoke(Gateway.java:282)
   	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
   	at py4j.commands.CallCommand.execute(CallCommand.java:79)
   	at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
   	at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
   	at java.lang.Thread.run(Thread.java:834)
   Breakpoint reached at org.apache.hadoop.ipc.CallerContext.setCurrent(CallerContext.java:148)
   Breakpoint reached
   	at org.apache.hadoop.ipc.CallerContext.setCurrent(CallerContext.java:148)
   	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-1)
   	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   	at java.lang.reflect.Method.invoke(Method.java:566)
   	at org.apache.spark.util.CallerContext.setCurrentContext(Utils.scala:3414)
   	at org.apache.spark.sql.hive.client.HiveClientImpl.newState(HiveClientImpl.scala:202)
   	at org.apache.spark.sql.hive.client.HiveClientImpl.<init>(HiveClientImpl.scala:143)
   	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(NativeConstructorAccessorImpl.java:-1)
   	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
   	at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
   	at java.lang.reflect.Constructor.newInstance(Constructor.java:490)
   	at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:316)
   	at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:539)
   	at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:399)
   	at org.apache.spark.sql.hive.HiveExternalCatalog.client$lzycompute(HiveExternalCatalog.scala:70)
   	at org.apache.spark.sql.hive.HiveExternalCatalog.client(HiveExternalCatalog.scala:69)
   	at org.apache.spark.sql.hive.HiveExternalCatalog.$anonfun$databaseExists$1(HiveExternalCatalog.scala:223)
   	at org.apache.spark.sql.hive.HiveExternalCatalog$$Lambda$2058.2002899646.apply$mcZ$sp(Unknown Source:-1)
   	at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23)
   	at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:101)
   	at org.apache.spark.sql.hive.HiveExternalCatalog.databaseExists(HiveExternalCatalog.scala:223)
   	at org.apache.spark.sql.internal.SharedState.externalCatalog$lzycompute(SharedState.scala:150)
   	at org.apache.spark.sql.internal.SharedState.externalCatalog(SharedState.scala:144)
   	at org.apache.spark.sql.internal.SharedState.globalTempViewManager$lzycompute(SharedState.scala:180)
   	at org.apache.spark.sql.internal.SharedState.globalTempViewManager(SharedState.scala:178)
   	at org.apache.spark.sql.hive.HiveSessionStateBuilder.$anonfun$catalog$2(HiveSessionStateBuilder.scala:70)
   	at org.apache.spark.sql.hive.HiveSessionStateBuilder$$Lambda$1492.867369720.apply(Unknown Source:-1)
   	at org.apache.spark.sql.catalyst.catalog.SessionCatalog.globalTempViewManager$lzycompute(SessionCatalog.scala:123)
   	at org.apache.spark.sql.catalyst.catalog.SessionCatalog.globalTempViewManager(SessionCatalog.scala:123)
   	at org.apache.spark.sql.catalyst.catalog.SessionCatalog.isGlobalTempViewDB(SessionCatalog.scala:1001)
   	at org.apache.spark.sql.catalyst.catalog.SessionCatalog.getRawLocalOrGlobalTempView(SessionCatalog.scala:716)
   	at org.apache.spark.sql.catalyst.catalog.SessionCatalog.isTempView(SessionCatalog.scala:1008)
   	at org.apache.spark.sql.streaming.DataStreamWriter.toTable(DataStreamWriter.scala:282)
   	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-1)
   	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   	at java.lang.reflect.Method.invoke(Method.java:566)
   	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
   	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
   	at py4j.Gateway.invoke(Gateway.java:282)
   	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
   	at py4j.commands.CallCommand.execute(CallCommand.java:79)
   	at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
   	at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
   	at java.lang.Thread.run(Thread.java:834)
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51095][CORE][SQL] Include caller context for hdfs audit logs for calls from driver [spark]

Posted by "sririshindra (via GitHub)" <gi...@apache.org>.
sririshindra commented on code in PR #49814:
URL: https://github.com/apache/spark/pull/49814#discussion_r1944966711


##########
core/src/main/scala/org/apache/spark/SparkContext.scala:
##########
@@ -722,6 +722,10 @@ class SparkContext(config: SparkConf) extends Logging {
     }
     appStatusSource.foreach(_env.metricsSystem.registerSource(_))
     _plugins.foreach(_.registerMetrics(applicationId))
+
+    logDebug("Adding driver caller context")

Review Comment:
   Yes, sorry I somehow missed from commit. I have messed my commit history while trying to create a rebase so that there is a clean commit when I raise the PR. Fixed in latest commit. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51095][CORE][SQL] Include caller context for hdfs audit logs for calls from driver [spark]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #49814:
URL: https://github.com/apache/spark/pull/49814#discussion_r1950146770


##########
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala:
##########
@@ -171,6 +172,11 @@ private[hive] class HiveClientImpl(
   private def newState(): SessionState = {
     val hiveConf = newHiveConf(sparkConf, hadoopConf, extraConfig, Some(initClassLoader))
     val state = new SessionState(hiveConf)
+    // When SessionState is initialized, the caller context is overridden by hive
+    // so we need to reset it back to the DRIVER

Review Comment:
   @sririshindra my point is
   
   >> I searched the Hive 2.3.10 codebase seems there are no places to set the caller context
   >
   > @sririshindra BTW, I guess the change in HiveClientImpl is unnecessary for the current OSS spark
   
   would be great if you could test with the OSS Spark/Hive



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51095][CORE][SQL] Include caller context for hdfs audit logs for calls from driver [spark]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #49814:
URL: https://github.com/apache/spark/pull/49814#discussion_r1947485021


##########
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala:
##########
@@ -171,6 +172,11 @@ private[hive] class HiveClientImpl(
   private def newState(): SessionState = {
     val hiveConf = newHiveConf(sparkConf, hadoopConf, extraConfig, Some(initClassLoader))
     val state = new SessionState(hiveConf)
+    // When SessionState is initialized, the caller context is overridden by hive
+    // so we need to reset it back to the DRIVER

Review Comment:
   @sririshindra the usage of Hive in Spark is a little bit complex, and the isolated classloader does not always take effect. For example, you could set the following configuration to let the Spark session catalog use a different version of HMS client
   
   ```
   spark.sql.hive.metastore.version
   spark.sql.hive.metastore.jars
   ```
   
   But Iceberg's HMS client does not respect that, it always uses the compiled Hive classes.
   
   `spark-sql` and Spark Thrift Server also have different behaviors ...
   
   > I don't have a hadoop cluster with upstream versions of spark/hive.
   
   https://github.com/awesome-kyuubi/hadoop-testing this project might be helpful



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51095][CORE][SQL] Include caller context for hdfs audit logs for calls from driver [spark]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #49814:
URL: https://github.com/apache/spark/pull/49814#discussion_r1947554775


##########
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala:
##########
@@ -171,6 +172,11 @@ private[hive] class HiveClientImpl(
   private def newState(): SessionState = {
     val hiveConf = newHiveConf(sparkConf, hadoopConf, extraConfig, Some(initClassLoader))
     val state = new SessionState(hiveConf)
+    // When SessionState is initialized, the caller context is overridden by hive
+    // so we need to reset it back to the DRIVER

Review Comment:
   > If we find there are broader problems with Spark's Hive clients leaking state, then an alternative solution might be to isolate org.apache.hadoop.ipc.CallerContext via the Hive client IsolatedClientLoader. Currently it treats all the Hadoop classes as shared.
   
   @cnauroth after SPARK-42539, Spark does not use IsolatedClientLoader by default, the current Hive 2.3.10 is compatible with HMS 1.2 to 4.0, and in practice, IsolatedClientLoader is rare to use due to setup complexity and some known issues(see SPARK-42539)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51095][CORE][SQL] Include caller context for hdfs audit logs for calls from driver [spark]

Posted by "cnauroth (via GitHub)" <gi...@apache.org>.
cnauroth commented on code in PR #49814:
URL: https://github.com/apache/spark/pull/49814#discussion_r1955592496


##########
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala:
##########
@@ -171,6 +172,11 @@ private[hive] class HiveClientImpl(
   private def newState(): SessionState = {
     val hiveConf = newHiveConf(sparkConf, hadoopConf, extraConfig, Some(initClassLoader))
     val state = new SessionState(hiveConf)
+    // When SessionState is initialized, the caller context is overridden by hive
+    // so we need to reset it back to the DRIVER

Review Comment:
   Awesome! I am +1 (non-binding).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51095][CORE][SQL] Include caller context for hdfs audit logs for calls from driver [spark]

Posted by "attilapiros (via GitHub)" <gi...@apache.org>.
attilapiros commented on PR #49814:
URL: https://github.com/apache/spark/pull/49814#issuecomment-2638107602

   This must be:
   https://github.com/sririshindra/spark/actions/runs/13162571866/job/36749883818#step:13:5783
   
   ```
   [error] /__w/spark/spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala:53:0: org.apache.spark.internal.config.APP_CALLER_CONTEXT is in wrong order relative to org.apache.spark.internal.config.Tests.IS_TESTING.
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51095][CORE][SQL] Include caller context for hdfs audit logs for calls from driver [spark]

Posted by "sririshindra (via GitHub)" <gi...@apache.org>.
sririshindra commented on code in PR #49814:
URL: https://github.com/apache/spark/pull/49814#discussion_r1947398927


##########
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala:
##########
@@ -171,6 +172,11 @@ private[hive] class HiveClientImpl(
   private def newState(): SessionState = {
     val hiveConf = newHiveConf(sparkConf, hadoopConf, extraConfig, Some(initClassLoader))
     val state = new SessionState(hiveConf)
+    // When SessionState is initialized, the caller context is overridden by hive
+    // so we need to reset it back to the DRIVER

Review Comment:
   @cnauroth , I now tested with multiple different queries touching different iceberg and hive tables and it works as expected from the beginning. I have done these tests both within the same spark session and seperate sessions as well. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51095][CORE][SQL] Include caller context for hdfs audit logs for calls from driver [spark]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #49814:
URL: https://github.com/apache/spark/pull/49814#discussion_r1947466059


##########
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala:
##########
@@ -171,6 +172,11 @@ private[hive] class HiveClientImpl(
   private def newState(): SessionState = {
     val hiveConf = newHiveConf(sparkConf, hadoopConf, extraConfig, Some(initClassLoader))
     val state = new SessionState(hiveConf)
+    // When SessionState is initialized, the caller context is overridden by hive
+    // so we need to reset it back to the DRIVER
+    new CallerContext("DRIVER",
+      sparkConf.get(APP_CALLER_CONTEXT),
+      Option(sparkConf.getOption("spark.app.id").getOrElse(""))).setCurrentContext()

Review Comment:
   also `sparkConf.getOption("spark.app.id").getOrElse("")` => `sparkConf.getAppId`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51095][CORE][SQL] Include caller context for hdfs audit logs for calls from driver [spark]

Posted by "attilapiros (via GitHub)" <gi...@apache.org>.
attilapiros commented on code in PR #49814:
URL: https://github.com/apache/spark/pull/49814#discussion_r1943828048


##########
core/src/main/scala/org/apache/spark/SparkContext.scala:
##########
@@ -722,6 +722,10 @@ class SparkContext(config: SparkConf) extends Logging {
     }
     appStatusSource.foreach(_env.metricsSystem.registerSource(_))
     _plugins.foreach(_.registerMetrics(applicationId))
+
+    logDebug("Adding driver caller context")

Review Comment:
   I do not think we need this debug line. Please remove it!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org