You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/05/18 12:08:53 UTC
[GitHub] [spark] AngersZhuuuu opened a new pull request, #36596: [SPARK-39224][CORE] Ignore noisy warning message in ProcfsMetricsGetter
AngersZhuuuu opened a new pull request, #36596:
URL: https://github.com/apache/spark/pull/36596
### What changes were proposed in this pull request?
There are many noisy warning logs such as
```
22/05/18 16:48:58 WARN ProcfsMetricsGetter: There was a problem with reading the stat file of the process.
java.io.FileNotFoundException: /proc/50371/stat (No such file or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at org.apache.spark.executor.ProcfsMetricsGetter.openReader$1(ProcfsMetricsGetter.scala:174)
at org.apache.spark.executor.ProcfsMetricsGetter.$anonfun$addProcfsMetricsFromOneProcess$1(ProcfsMetricsGetter.scala:176)
at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2647)
at org.apache.spark.executor.ProcfsMetricsGetter.addProcfsMetricsFromOneProcess(ProcfsMetricsGetter.scala:176)
at org.apache.spark.executor.ProcfsMetricsGetter.$anonfun$computeAllMetrics$1(ProcfsMetricsGetter.scala:216)
at scala.runtime.java8.JFunction1$mcVI$sp.apply(JFunction1$mcVI$sp.java:23)
at scala.collection.immutable.Set$Set2.foreach(Set.scala:132)
at org.apache.spark.executor.ProcfsMetricsGetter.computeAllMetrics(ProcfsMetricsGetter.scala:214)
at org.apache.spark.metrics.ProcessTreeMetrics$.getMetricValues(ExecutorMetricType.scala:93)
at org.apache.spark.executor.ExecutorMetrics$.$anonfun$getCurrentMetrics$1(ExecutorMetrics.scala:103)
at org.apache.spark.executor.ExecutorMetrics$.$anonfun$getCurrentMetrics$1$adapted(ExecutorMetrics.scala:102)
at scala.collection.Iterator.foreach(Iterator.scala:941)
at scala.collection.Iterator.foreach$(Iterator.scala:941)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
at scala.collection.IterableLike.foreach(IterableLike.scala:74)
at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
at org.apache.spark.executor.ExecutorMetrics$.getCurrentMetrics(ExecutorMetrics.scala:102)
at org.apache.spark.SparkContext.reportHeartBeat(SparkContext.scala:2578)
at org.apache.spark.SparkContext.$anonfun$new$30(SparkContext.scala:587)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:2022)
at org.apache.spark.Heartbeater$$anon$1.run(Heartbeater.scala:46)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
```
Many user complain about this. We can ignore this by default.
### Why are the changes needed?
Ignore noisy logs
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
MT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #36596: [SPARK-39224][CORE] Ignore noisy warning message in ProcfsMetricsGetter
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on PR #36596:
URL: https://github.com/apache/spark/pull/36596#issuecomment-1131014277
> Thank you for pinging me, @AngersZhuuuu . I'm also +1 for @srowen 's advice to lower the log level instead of having another configuration for log.
How about current?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #36596: [SPARK-39224][CORE] Ignore noisy warning message in ProcfsMetricsGetter
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on PR #36596:
URL: https://github.com/apache/spark/pull/36596#issuecomment-1129927080
ping @dongjoon-hyun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #36596: [SPARK-39224][CORE] Lower general ProcfsMetricsGetter error log levels except `/proc/` lookup error
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #36596: [SPARK-39224][CORE] Lower general ProcfsMetricsGetter error log levels except `/proc/` lookup error
URL: https://github.com/apache/spark/pull/36596
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a diff in pull request #36596: [SPARK-39224][CORE] Ignore noisy warning message in ProcfsMetricsGetter
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on code in PR #36596:
URL: https://github.com/apache/spark/pull/36596#discussion_r876524717
##########
core/src/main/scala/org/apache/spark/internal/config/package.scala:
##########
@@ -278,6 +278,13 @@ package object config {
.booleanConf
.createWithDefault(false)
+ private[spark] val IGNORE_PROCFS_TREE_METRICS_GETTER_WARNING =
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on a diff in pull request #36596: [SPARK-39224][CORE] Ignore noisy warning message in ProcfsMetricsGetter
Posted by GitBox <gi...@apache.org>.
srowen commented on code in PR #36596:
URL: https://github.com/apache/spark/pull/36596#discussion_r875905175
##########
core/src/main/scala/org/apache/spark/internal/config/package.scala:
##########
@@ -278,6 +278,13 @@ package object config {
.booleanConf
.createWithDefault(false)
+ private[spark] val IGNORE_PROCFS_TREE_METRICS_GETTER_WARNING =
Review Comment:
Let's not make a flag for this. Reduce the log level
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #36596: [SPARK-39224][CORE] Ignore noisy warning message in ProcfsMetricsGetter
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on PR #36596:
URL: https://github.com/apache/spark/pull/36596#issuecomment-1130224552
Thank you for pinging me, @AngersZhuuuu . I'm also +1 for @srowen 's advice to lower the log level instead of having another configuration for log.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org