You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "steveloughran (via GitHub)" <gi...@apache.org> on 2025/02/03 19:19:04 UTC
[PR] [SPARK-51072][CORE] CallerContext to set Hadoop cloud audit context [spark]
steveloughran opened a new pull request, #49779:
URL: https://github.com/apache/spark/pull/49779
### What changes were proposed in this pull request?
When enabled, cloud store client audit context is set to the
same context string as the Hadoop IPC context.
### Why are the changes needed?
CallerContext adds information about the spark task to hadoop IPC context and then to HDFS, YARN and HBase server logs.
It is also possible to update the cloud storage "audit context".
Storage clients can attach the audit information to requests to be stored in the service's own logs, where it can be retrieved, parsed and used for analysis.
It is currently supported by the S3A connector, which adds the information to a synthetic referrer header, which is then stored in the S3 Server logs. (Not cloudtrail, sadly).
See [S3A Auditing](https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/auditing.html)
### Does this PR introduce _any_ user-facing change?
If enabled, it adds extra entries in cloud storage server logs through cloud
storage clients which support it.
### How was this patch tested?
Expanded existing test `"Set Spark CallerContext"` to verify
full setting of passed down parameters to caller and audit contexts.
This required extracting the functional code of `CallerContext.setCurrentContext`
into a `@VisibleForTesting private[util]` method `setCurrentContext(Boolean)`
Without this, the test suite only ran if the process had been launched
with the configuration option `"hadoop.caller.context.enabled` being set
to true -this is not the default, so the existing test suite code
was probably never executed.
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51072][CORE] CallerContext to set Hadoop cloud audit context [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #49779: [SPARK-51072][CORE] CallerContext to set Hadoop cloud audit context
URL: https://github.com/apache/spark/pull/49779
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51072][CORE] CallerContext to set Hadoop cloud audit context [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #49779:
URL: https://github.com/apache/spark/pull/49779#discussion_r1939937954
##########
core/src/main/scala/org/apache/spark/util/Utils.scala:
##########
@@ -3214,12 +3219,28 @@ private[spark] class CallerContext(
/**
* Set up the caller context [[context]] by invoking Hadoop CallerContext API of
- * [[HadoopCallerContext]].
+ * [[HadoopCallerContext]], which is included in IPC calls.
+ * and the hadoop audit context, which may be included in cloud storage
+ * requests for collection in cloud service logs.
*/
- def setCurrentContext(): Unit = if (CallerContext.callerContextEnabled) {
- val hdfsContext = new HadoopCallerContextBuilder(context).build()
- HadoopCallerContext.setCurrent(hdfsContext)
+ def setCurrentContext(): Unit = {
+ setCurrentContext(CallerContext.callerContextEnabled)
}
+
+ /**
+ * Inner method to set the context.
+ * @param enabled should the thread-level contexts be updated?
+ */
+ @VisibleForTesting
+ private[util] def setCurrentContext(enabled: Boolean): Unit = {
+ if (enabled) {
+ val hdfsContext = new HadoopCallerContextBuilder(context).build()
+ HadoopCallerContext.setCurrent(hdfsContext)
+ // audit context as passed down to object stores, use prefix "spark"
+ currentAuditContext.put("spark", context)
Review Comment:
Thank you for making a PR. This one line seems to be the actual change. Did I understand correctly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51072][CORE] CallerContext to set Hadoop cloud audit context [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #49779:
URL: https://github.com/apache/spark/pull/49779#issuecomment-2632302741
Ya, IIUC, without this PR, S3A audit has been working already as designed, doesn't it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51072][CORE] CallerContext to set Hadoop cloud audit context [spark]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on PR #49779:
URL: https://github.com/apache/spark/pull/49779#issuecomment-2636482196
@cnauroth well, you should -if you can get it anywhere into your logs, possibly as a new http header. s3afs attaches as an http referrer as it is the sole entry other than UA which goes into the standard S3 logs -and other things like to set that UA field.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51072][CORE] CallerContext to set Hadoop cloud audit context [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #49779:
URL: https://github.com/apache/spark/pull/49779#issuecomment-2640286925
Got it. Please let me know when the PR is ready, @steveloughran .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51072][CORE] CallerContext to set Hadoop cloud audit context [spark]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on code in PR #49779:
URL: https://github.com/apache/spark/pull/49779#discussion_r1942702521
##########
core/src/main/scala/org/apache/spark/util/Utils.scala:
##########
@@ -3214,12 +3219,28 @@ private[spark] class CallerContext(
/**
* Set up the caller context [[context]] by invoking Hadoop CallerContext API of
- * [[HadoopCallerContext]].
+ * [[HadoopCallerContext]], which is included in IPC calls.
+ * and the hadoop audit context, which may be included in cloud storage
+ * requests for collection in cloud service logs.
*/
- def setCurrentContext(): Unit = if (CallerContext.callerContextEnabled) {
- val hdfsContext = new HadoopCallerContextBuilder(context).build()
- HadoopCallerContext.setCurrent(hdfsContext)
+ def setCurrentContext(): Unit = {
+ setCurrentContext(CallerContext.callerContextEnabled)
}
+
+ /**
+ * Inner method to set the context.
+ * @param enabled should the thread-level contexts be updated?
+ */
+ @VisibleForTesting
+ private[util] def setCurrentContext(enabled: Boolean): Unit = {
+ if (enabled) {
+ val hdfsContext = new HadoopCallerContextBuilder(context).build()
+ HadoopCallerContext.setCurrent(hdfsContext)
+ // audit context as passed down to object stores, use prefix "spark"
+ currentAuditContext.put("spark", context)
Review Comment:
yes! Rest of it is test related
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51072][CORE] CallerContext to set Hadoop cloud audit context [spark]
Posted by "cnauroth (via GitHub)" <gi...@apache.org>.
cnauroth commented on PR #49779:
URL: https://github.com/apache/spark/pull/49779#issuecomment-2632122269
> According to the PR title, do you mean S3 Audit Context feature has been broken until now ? Otherwise, could you revise the PR title by narrowing down the scope more specifically?
>
> > [SPARK-51072][CORE] CallerContext to set Hadoop cloud audit context
From the perspective of Spark or other clients of the file system, they are interacting with a general auditing feature defined in the Hadoop Common module. In theory, multiple file systems could implement support for actually recording this audit information. AFAIK, only S3A implements it right now. (We don't have it in the GCS file system.) Other file systems could eventually choose to implement it though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51072][CORE] CallerContext to set Hadoop cloud audit context [spark]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on PR #49779:
URL: https://github.com/apache/spark/pull/49779#issuecomment-2636477498
@dongjoon-hyun audit context has been working properly, but spark info not wired up. Other things get in (process Id, UGI, filesystem id, underlying operation for a sequence), but the actual spark job didn't get in. Once an s3a or manifest committer was started, it'd set the app and job ID values -but they only get involved during the write phase -and they didn't get the full spark context info
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51072][CORE] CallerContext to set Hadoop cloud audit context [spark]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on PR #49779:
URL: https://github.com/apache/spark/pull/49779#issuecomment-2663444170
I can't think of any changes, unless we want to set that audit stuff even if caller context is not being set.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org