You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/09/01 09:38:11 UTC
[GitHub] [spark] allendang001 opened a new pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
allendang001 opened a new pull request #33874:
URL: https://github.com/apache/spark/pull/33874
What changes were proposed in this pull request?
add inputBytes and inputRecords interface in SparkStageInfo
Why are the changes needed?
One of our projects needs to count the amount of data scanned and the number of scanned data rows during the execution of sparksql statements, but the current version of spark does not provide an interface to view these data, so I want to obtain this type of data through the spark context interface
Does this PR introduce any user-facing change?
no
How was this patch tested?
no
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-916781006
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143143/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-917539491
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143176/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-915845806
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47615/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-910016920
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47410/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-915845806
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47615/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-917537196
@allendang001 mind enabling GitHub Actions in your forked repository? See also https://github.com/apache/spark/pull/33874/checks?check_run_id=3565581644
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-909960681
**[Test build #142907 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142907/testReport)** for PR 33874 at commit [`fb66ab3`](https://github.com/apache/spark/commit/fb66ab3a7c3fcea2b9481ed62623b404d47cb41a).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] closed pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #33874:
URL: https://github.com/apache/spark/pull/33874
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-915815530
**[Test build #143112 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143112/testReport)** for PR 33874 at commit [`c67a9dd`](https://github.com/apache/spark/commit/c67a9ddd4e49c29e0d5e376b3f93638a7c2a5ff3).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-909950202
ok to test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-909955044
**[Test build #142907 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142907/testReport)** for PR 33874 at commit [`fb66ab3`](https://github.com/apache/spark/commit/fb66ab3a7c3fcea2b9481ed62623b404d47cb41a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-915811303
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143111/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-915815530
**[Test build #143112 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143112/testReport)** for PR 33874 at commit [`c67a9dd`](https://github.com/apache/spark/commit/c67a9ddd4e49c29e0d5e376b3f93638a7c2a5ff3).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-916780950
**[Test build #143143 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143143/testReport)** for PR 33874 at commit [`16e6323`](https://github.com/apache/spark/commit/16e632316f6727f6bfe7c51b3990924756c6671f).
* This patch **fails MiMa tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-909950202
ok to test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-909096909
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-909987836
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142907/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-916773798
**[Test build #143143 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143143/testReport)** for PR 33874 at commit [`16e6323`](https://github.com/apache/spark/commit/16e632316f6727f6bfe7c51b3990924756c6671f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] allendang001 edited a comment on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
allendang001 edited a comment on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-915800477
> StatusTrackerSuite
@HyukjinKwon thanks a lot
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-909987836
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142907/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-909805603
+CC @cloud-fan who took a look at this last.
I am fine with the change in general.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-915849727
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47617/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-917543968
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47679/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-916804031
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47647/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-914962534
Just adding these fields into `status.py` should be enough. e.g.)
```diff
diff --git a/python/pyspark/status.py b/python/pyspark/status.py
index a6fa7dd3144..f342ee38a2d 100644
--- a/python/pyspark/status.py
+++ b/python/pyspark/status.py
@@ -28,7 +28,7 @@ class SparkJobInfo(namedtuple("SparkJobInfo", "jobId stageIds status")):
class SparkStageInfo(namedtuple("SparkStageInfo",
"stageId currentAttemptId name numTasks numActiveTasks "
- "numCompletedTasks numFailedTasks")):
+ "numCompletedTasks numFailedTasks inputBytes inputRecords")):
"""
Exposes information about Spark Stages.
"""
diff --git a/python/pyspark/status.pyi b/python/pyspark/status.pyi
index 0558e245f49..8ea885693bb 100644
--- a/python/pyspark/status.pyi
+++ b/python/pyspark/status.pyi
@@ -32,6 +32,8 @@ class SparkStageInfo(NamedTuple):
numActiveTasks: int
numCompletedTasks: int
numFailedTasks: int
+ inputBytes: int
+ inputRecords: int
class StatusTracker:
def __init__(self, jtracker: JavaObject) -> None: ...
diff --git a/python/pyspark/tests/test_context.py b/python/pyspark/tests/test_context.py
index 4611d038f96..2c28fbabcc8 100644
--- a/python/pyspark/tests/test_context.py
+++ b/python/pyspark/tests/test_context.py
@@ -239,6 +239,8 @@ class ContextTests(unittest.TestCase):
self.assertEqual(1, len(job.stageIds))
stage = tracker.getStageInfo(job.stageIds[0])
self.assertEqual(rdd.getNumPartitions(), stage.numTasks)
+ self.assertGreater(0, stage.inputBytes)
+ self.assertEqual(10, stage.inputRecords)
sc.cancelAllJobs()
t.join()
```
BTW, please keep the Github PR template as is (https://github.com/apache/spark/blob/master/.github/PULL_REQUEST_TEMPLATE), and describe, at "Does this PR introduce any user-facing change?", which interface it adds with an example preferably.
Also, please add a test at `StatusTrackerSuite`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-916773798
**[Test build #143143 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143143/testReport)** for PR 33874 at commit [`16e6323`](https://github.com/apache/spark/commit/16e632316f6727f6bfe7c51b3990924756c6671f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-915840615
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47615/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-909096909
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-909096909
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-910016880
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47410/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-910016920
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47410/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] allendang001 edited a comment on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
allendang001 edited a comment on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-913494624
> @allendang001 have you checked the python codes?
>
> Especially the `status.py`:
> https://github.com/apache/spark/blob/387a251a682a596ba4156b7d12e6025762ebac85/python/pyspark/status.py#L29-L31
I am not familiar with python codes, can you help me to complete this part?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-915836344
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143112/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-915849673
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47617/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-916796427
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47647/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-917542534
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47679/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-909955044
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-915809612
**[Test build #143111 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143111/testReport)** for PR 33874 at commit [`ebe9447`](https://github.com/apache/spark/commit/ebe9447e30570cbf3a6d3871cb0acc53579e0de3).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] allendang001 commented on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
allendang001 commented on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-915819263
i have submit this codes, @HyukjinKwon PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-914964453
Oh, and we should fix Python example here: https://github.com/apache/spark/blob/0494dc90af48ce7da0625485a4dc6917a244d580/examples/src/main/python/status_api_demo.py#L59-L60 just like you did in Java.
Lastly, Apache Spark leverages the resources of GitHub Actions in your forked repository to test your PR. Please enable it, see also https://github.com/apache/spark/pull/33874/checks?check_run_id=3471668214.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] allendang001 commented on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
allendang001 commented on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-913494624
> @allendang001 have you checked the python codes?
>
> Especially the `status.py`:
> https://github.com/apache/spark/blob/387a251a682a596ba4156b7d12e6025762ebac85/python/pyspark/status.py#L29-L31
I am not familiar with python codes, can you complete this part?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-915836344
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143112/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-917543968
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47679/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-917537164
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] attilapiros commented on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
attilapiros commented on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-911957166
@allendang001 have you checked the python codes?
Especially the `status.py`:
https://github.com/apache/spark/blob/387a251a682a596ba4156b7d12e6025762ebac85/python/pyspark/status.py#L29-L31
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm edited a comment on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
mridulm edited a comment on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-909805603
+CC @cloud-fan, @attilapiros who took a look at this last.
I am fine with the change in general.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] allendang001 commented on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
allendang001 commented on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-909115282
@cloud-fan hi, please take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] allendang001 commented on a change in pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
allendang001 commented on a change in pull request #33874:
URL: https://github.com/apache/spark/pull/33874#discussion_r706025945
##########
File path: examples/src/main/python/status_api_demo.py
##########
@@ -56,8 +56,8 @@ def run():
for sid in job.stageIds:
info = status.getStageInfo(sid)
if info:
- print("Stage %d: %d tasks total (%d active, %d complete)" %
- (sid, info.numTasks, info.numActiveTasks, info.numCompletedTasks))
+ print("Stage %d: %d tasks total (%d active, %d complete), %s inputBytes %s inputRecords" %
+ (sid, info.numTasks, info.numActiveTasks, info.numCompletedTasks, info.inputBytes, info.inputRecords))
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] allendang001 commented on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
allendang001 commented on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-909115282
@cloud-fan hi, please take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-916801538
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47647/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-909096909
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-917543488
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47679/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-909955044
**[Test build #142907 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142907/testReport)** for PR 33874 at commit [`fb66ab3`](https://github.com/apache/spark/commit/fb66ab3a7c3fcea2b9481ed62623b404d47cb41a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-915845777
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47615/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-917539491
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143176/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-999180358
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] allendang001 commented on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
allendang001 commented on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-915800477
> StatusTrackerSuite
thanks a lot
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-914131873
cc @HyukjinKwon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-917538117
**[Test build #143176 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143176/testReport)** for PR 33874 at commit [`16e6323`](https://github.com/apache/spark/commit/16e632316f6727f6bfe7c51b3990924756c6671f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-915811288
**[Test build #143111 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143111/testReport)** for PR 33874 at commit [`ebe9447`](https://github.com/apache/spark/commit/ebe9447e30570cbf3a6d3871cb0acc53579e0de3).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-909805603
+CC @cloud-fan who took a look at this last.
I am fine with the change in general.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-915849727
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47617/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33874:
URL: https://github.com/apache/spark/pull/33874#discussion_r705816774
##########
File path: examples/src/main/python/status_api_demo.py
##########
@@ -56,8 +56,8 @@ def run():
for sid in job.stageIds:
info = status.getStageInfo(sid)
if info:
- print("Stage %d: %d tasks total (%d active, %d complete)" %
- (sid, info.numTasks, info.numActiveTasks, info.numCompletedTasks))
+ print("Stage %d: %d tasks total (%d active, %d complete), %s inputBytes %s inputRecords" %
+ (sid, info.numTasks, info.numActiveTasks, info.numCompletedTasks, info.inputBytes, info.inputRecords))
Review comment:
Can you fix the style here?
```
./examples/src/main/python/status_api_demo.py:59:101: E501 line too long (110 > 100 characters)
./examples/src/main/python/status_api_demo.py:60:101: E501 line too long (128 > 100 characters)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm edited a comment on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
mridulm edited a comment on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-909805603
+CC @cloud-fan, @attilapiros who took a look at this last.
I am fine with the change in general.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-917539480
**[Test build #143176 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143176/testReport)** for PR 33874 at commit [`16e6323`](https://github.com/apache/spark/commit/16e632316f6727f6bfe7c51b3990924756c6671f).
* This patch **fails MiMa tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-915811303
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143111/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-917538117
**[Test build #143176 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143176/testReport)** for PR 33874 at commit [`16e6323`](https://github.com/apache/spark/commit/16e632316f6727f6bfe7c51b3990924756c6671f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-915817045
**[Test build #143112 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143112/testReport)** for PR 33874 at commit [`c67a9dd`](https://github.com/apache/spark/commit/c67a9ddd4e49c29e0d5e376b3f93638a7c2a5ff3).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-916804031
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47647/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-916781006
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143143/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-909955044
**[Test build #142907 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142907/testReport)** for PR 33874 at commit [`fb66ab3`](https://github.com/apache/spark/commit/fb66ab3a7c3fcea2b9481ed62623b404d47cb41a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-915809612
**[Test build #143111 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143111/testReport)** for PR 33874 at commit [`ebe9447`](https://github.com/apache/spark/commit/ebe9447e30570cbf3a6d3871cb0acc53579e0de3).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-910005430
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47410/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-909096909
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] allendang001 commented on pull request #33874: [SPARK-35624][CORE]Support reading inputbytes and inputrecords of the…
Posted by GitBox <gi...@apache.org>.
allendang001 commented on pull request #33874:
URL: https://github.com/apache/spark/pull/33874#issuecomment-909115282
@cloud-fan hi, please take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org