You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by yucai <gi...@git.apache.org> on 2018/07/17 10:44:31 UTC
[GitHub] spark pull request #21791: [SPARK-24832][SQL] Improve inputMetrics's bytesRe...
GitHub user yucai opened a pull request:
https://github.com/apache/spark/pull/21791
[SPARK-24832][SQL] Improve inputMetrics's bytesRead update for ColumnarBatch
## What changes were proposed in this pull request?
Currently, ColumnarBatch's bytesRead need to be updated every 4096 * 1000 rows, which makes the metrics out of date. This PR makes it update for each batch.
## How was this patch tested?
Existing UTs.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/yucai/spark SPARK-24832
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21791.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21791
----
commit b9320c8d8a3735d4569709b51a0d66a7121e23cb
Author: yucai <yy...@...>
Date: 2018-07-17T10:20:18Z
[SPARK-24832][SQL] Improve inputMetrics's bytesRead update for ColumnarBatch
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21791: [SPARK-24832][SQL] Improve inputMetrics's bytesRead upda...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21791
**[Test build #93168 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93168/testReport)** for PR 21791 at commit [`b9320c8`](https://github.com/apache/spark/commit/b9320c8d8a3735d4569709b51a0d66a7121e23cb).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21791: [SPARK-24925][SQL] input bytesRead metrics fluctuate fro...
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/21791
gentle ping @yucai
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21791: [SPARK-24832][SQL] Improve inputMetrics's bytesRead upda...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21791
**[Test build #93168 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93168/testReport)** for PR 21791 at commit [`b9320c8`](https://github.com/apache/spark/commit/b9320c8d8a3735d4569709b51a0d66a7121e23cb).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21791: [SPARK-24925][SQL] input bytesRead metrics fluctuate fro...
Posted by yucai <gi...@git.apache.org>.
Github user yucai commented on the issue:
https://github.com/apache/spark/pull/21791
@kiszk I see #22324 has been solved, which is one of my PR's dependency actually (see https://issues.apache.org/jira/browse/SPARK-24925?focusedCommentId=16556818&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16556818) , I am stuck for that for a long time, I will update this one recently. Thanks!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21791: [SPARK-24832][SQL] Improve inputMetrics's bytesRead upda...
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/21791
Is it better to add a test to `InputOutputMetricsSuite`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21791: [SPARK-24832][SQL] Improve inputMetrics's bytesRead upda...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21791
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93168/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21791: [SPARK-24832][SQL] Improve inputMetrics's bytesRead upda...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21791
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21791: [SPARK-24832][SQL] Improve inputMetrics's bytesRead upda...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21791
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21791: [SPARK-24925][SQL] input bytesRead metrics fluctuate fro...
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/21791
I see. thank you for sharing the dependency and status.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21791: [SPARK-24925][SQL] input bytesRead metrics fluctuate fro...
Posted by yucai <gi...@git.apache.org>.
Github user yucai commented on the issue:
https://github.com/apache/spark/pull/21791
@kiszk I feel it is hard to add UT, do you have any idea?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21791: [SPARK-24832][SQL] Improve inputMetrics's bytesRead upda...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21791
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org