You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by hv...@apache.org on 2018/04/06 16:42:19 UTC
spark git commit: [SPARK-23882][CORE]
UTF8StringSuite.writeToOutputStreamUnderflow() is not expected to be
supported
Repository: spark
Updated Branches:
refs/heads/master d766ea2ff -> c926acf71
[SPARK-23882][CORE] UTF8StringSuite.writeToOutputStreamUnderflow() is not expected to be supported
## What changes were proposed in this pull request?
This PR excludes an existing UT [`writeToOutputStreamUnderflow()`](https://github.com/apache/spark/blob/master/common/unsafe/src/test/java/org/apache/spark/unsafe/types/UTF8StringSuite.java#L519-L532) in `UTF8StringSuite`.
As discussed [here](https://github.com/apache/spark/pull/19222#discussion_r177692142), the behavior of this test looks surprising. This test seems to access metadata area of the JVM object where is reserved by `Platform.BYTE_ARRAY_OFFSET`.
This test is introduced thru #16089 by NathanHowell. More specifically, [the commit](https://github.com/apache/spark/pull/16089/commits/27c102deb1701fe62f776fe4da61dac959270b73) `Improve test coverage of UTFString.write` introduced this UT. However, I cannot find any discussion about this UT.
I think that it would be good to exclude this UT.
```java
public void writeToOutputStreamUnderflow() throws IOException {
// offset underflow is apparently supported?
final ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
final byte[] test = "01234567".getBytes(StandardCharsets.UTF_8);
for (int i = 1; i <= Platform.BYTE_ARRAY_OFFSET; ++i) {
new UTF8String(
new ByteArrayMemoryBlock(test, Platform.BYTE_ARRAY_OFFSET - i, test.length + i))
.writeTo(outputStream);
final ByteBuffer buffer = ByteBuffer.wrap(outputStream.toByteArray(), i, test.length);
assertEquals("01234567", StandardCharsets.UTF_8.decode(buffer).toString());
outputStream.reset();
}
}
```
## How was this patch tested?
Existing UTs
Author: Kazuaki Ishizaki <is...@jp.ibm.com>
Closes #20995 from kiszk/SPARK-23882.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/c926acf7
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/c926acf7
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/c926acf7
Branch: refs/heads/master
Commit: c926acf719a6deb9d884a0f19bde075c312bfe5a
Parents: d766ea2
Author: Kazuaki Ishizaki <is...@jp.ibm.com>
Authored: Fri Apr 6 18:42:14 2018 +0200
Committer: Herman van Hovell <hv...@databricks.com>
Committed: Fri Apr 6 18:42:14 2018 +0200
----------------------------------------------------------------------
.../apache/spark/unsafe/types/UTF8StringSuite.java | 16 ----------------
1 file changed, 16 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/c926acf7/common/unsafe/src/test/java/org/apache/spark/unsafe/types/UTF8StringSuite.java
----------------------------------------------------------------------
diff --git a/common/unsafe/src/test/java/org/apache/spark/unsafe/types/UTF8StringSuite.java b/common/unsafe/src/test/java/org/apache/spark/unsafe/types/UTF8StringSuite.java
index bad908f..652c40a 100644
--- a/common/unsafe/src/test/java/org/apache/spark/unsafe/types/UTF8StringSuite.java
+++ b/common/unsafe/src/test/java/org/apache/spark/unsafe/types/UTF8StringSuite.java
@@ -516,22 +516,6 @@ public class UTF8StringSuite {
}
@Test
- public void writeToOutputStreamUnderflow() throws IOException {
- // offset underflow is apparently supported?
- final ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
- final byte[] test = "01234567".getBytes(StandardCharsets.UTF_8);
-
- for (int i = 1; i <= Platform.BYTE_ARRAY_OFFSET; ++i) {
- new UTF8String(
- new ByteArrayMemoryBlock(test, Platform.BYTE_ARRAY_OFFSET - i, test.length + i))
- .writeTo(outputStream);
- final ByteBuffer buffer = ByteBuffer.wrap(outputStream.toByteArray(), i, test.length);
- assertEquals("01234567", StandardCharsets.UTF_8.decode(buffer).toString());
- outputStream.reset();
- }
- }
-
- @Test
public void writeToOutputStreamSlice() throws IOException {
final ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
final byte[] test = "01234567".getBytes(StandardCharsets.UTF_8);
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org