You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sr...@apache.org on 2016/09/15 08:37:16 UTC
spark git commit: [SPARK-17524][TESTS] Use specified
spark.buffer.pageSize
Repository: spark
Updated Branches:
refs/heads/master d15b4f90e -> f893e2625
[SPARK-17524][TESTS] Use specified spark.buffer.pageSize
## What changes were proposed in this pull request?
This PR has the appendRowUntilExceedingPageSize test in RowBasedKeyValueBatchSuite use whatever spark.buffer.pageSize value a user has specified to prevent a test failure for anyone testing Apache Spark on a box with a reduced page size. The test is currently hardcoded to use the default page size which is 64 MB so this minor PR is a test improvement
## How was this patch tested?
Existing unit tests with 1 MB page size and with 64 MB (the default) page size
Author: Adam Roberts <ar...@uk.ibm.com>
Closes #15079 from a-roberts/patch-5.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/f893e262
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/f893e262
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/f893e262
Branch: refs/heads/master
Commit: f893e262500e2f183de88e984300dd5b085e1f71
Parents: d15b4f9
Author: Adam Roberts <ar...@uk.ibm.com>
Authored: Thu Sep 15 09:37:12 2016 +0100
Committer: Sean Owen <so...@cloudera.com>
Committed: Thu Sep 15 09:37:12 2016 +0100
----------------------------------------------------------------------
.../sql/catalyst/expressions/RowBasedKeyValueBatchSuite.java | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/f893e262/sql/catalyst/src/test/java/org/apache/spark/sql/catalyst/expressions/RowBasedKeyValueBatchSuite.java
----------------------------------------------------------------------
diff --git a/sql/catalyst/src/test/java/org/apache/spark/sql/catalyst/expressions/RowBasedKeyValueBatchSuite.java b/sql/catalyst/src/test/java/org/apache/spark/sql/catalyst/expressions/RowBasedKeyValueBatchSuite.java
index 0dd129c..fb3dbe8 100644
--- a/sql/catalyst/src/test/java/org/apache/spark/sql/catalyst/expressions/RowBasedKeyValueBatchSuite.java
+++ b/sql/catalyst/src/test/java/org/apache/spark/sql/catalyst/expressions/RowBasedKeyValueBatchSuite.java
@@ -338,15 +338,17 @@ public class RowBasedKeyValueBatchSuite {
@Test
public void appendRowUntilExceedingPageSize() throws Exception {
+ // Use default size or spark.buffer.pageSize if specified
+ int pageSizeToUse = (int) memoryManager.pageSizeBytes();
RowBasedKeyValueBatch batch = RowBasedKeyValueBatch.allocate(keySchema,
- valueSchema, taskMemoryManager, 64 * 1024 * 1024); //enough capacity
+ valueSchema, taskMemoryManager, pageSizeToUse); //enough capacity
try {
UnsafeRow key = makeKeyRow(1, "A");
UnsafeRow value = makeValueRow(1, 1);
int recordLength = 8 + key.getSizeInBytes() + value.getSizeInBytes() + 8;
int totalSize = 4;
int numRows = 0;
- while (totalSize + recordLength < 64 * 1024 * 1024) { // default page size
+ while (totalSize + recordLength < pageSizeToUse) {
appendRow(batch, key, value);
totalSize += recordLength;
numRows++;
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org