You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/10/14 08:24:12 UTC
[GitHub] [spark] LuciferYang opened a new pull request #34284: [SPARK-36900][CORE][TEST] Refactor `ChunkedByteBufferOutputStreamSuite` to pass `SPARK-36464: size returns correct positive number even with over 2GB data` with Java 8, 11 and 17
LuciferYang opened a new pull request #34284:
URL: https://github.com/apache/spark/pull/34284
### What changes were proposed in this pull request?
Refactor `SPARK-36464: size returns correct positive number even with over 2GB data` in `ChunkedByteBufferOutputStreamSuite` to reduce the total use of memory for this test case, then this case can pass with Java 8, Java 11 and Java 17 use `-Xmx4g`.
### Why are the changes needed?
`SPARK-36464: size returns correct positive number even with over 2GB data` pass with Java 8 but OOM with Java 11 and Java 17.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
- Pass the Jenkins or GitHub Action
- Manual test
```
mvn clean install -pl core -am -Dtest=none -DwildcardSuites=org.apache.spark.util.io.ChunkedByteBufferOutputStreamSuite
```
with Java 8, Java 11 and Java 17, all tests passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #34284: [SPARK-36900][CORE][TEST] Refactor `SPARK-36464: size returns correct positive number even with over 2GB data` to pass with Java 8, 11 and 17
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34284:
URL: https://github.com/apache/spark/pull/34284#discussion_r728984888
##########
File path: core/src/test/scala/org/apache/spark/util/io/ChunkedByteBufferOutputStreamSuite.scala
##########
@@ -121,12 +121,10 @@ class ChunkedByteBufferOutputStreamSuite extends SparkFunSuite {
}
test("SPARK-36464: size returns correct positive number even with over 2GB data") {
- val ref = new Array[Byte](1024 * 1024 * 1024)
- val o = new ChunkedByteBufferOutputStream(1024 * 1024, ByteBuffer.allocate)
- o.write(ref)
- o.write(ref)
- o.close()
+ val ref = new Array[Byte](1024 * 1024 * 4)
+ val o = new ChunkedByteBufferOutputStream(1024 * 1024 * 4, ByteBuffer.allocate)
+ (0 to 512).foreach(_ => o.write(ref))
Review comment:
Yes，write 513 times.
It was my mistake to lose call of close， I'll fix it later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #34284: [SPARK-36900][CORE][TEST] Refactor `SPARK-36464: size returns correct positive number even with over 2GB data` to pass with Java 8, 11 and 17
Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #34284:
URL: https://github.com/apache/spark/pull/34284#discussion_r728979347
##########
File path: core/src/test/scala/org/apache/spark/util/io/ChunkedByteBufferOutputStreamSuite.scala
##########
@@ -121,12 +121,10 @@ class ChunkedByteBufferOutputStreamSuite extends SparkFunSuite {
}
test("SPARK-36464: size returns correct positive number even with over 2GB data") {
- val ref = new Array[Byte](1024 * 1024 * 1024)
- val o = new ChunkedByteBufferOutputStream(1024 * 1024, ByteBuffer.allocate)
- o.write(ref)
- o.write(ref)
- o.close()
+ val ref = new Array[Byte](1024 * 1024 * 4)
+ val o = new ChunkedByteBufferOutputStream(1024 * 1024 * 4, ByteBuffer.allocate)
+ (0 to 512).foreach(_ => o.write(ref))
Review comment:
OK this writes 513 times right? sounds good. is o.close() important? probably not but could keep it.
If you change again, maybe introduce a constant for "4" and some expressions to demonstrate the logic here of writing just over 2G, but, it doesn't matter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34284: [SPARK-36900][CORE][TEST] Refactor `SPARK-36464: size returns correct positive number even with over 2GB data` to pass with Java 8, 11 and 17
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34284:
URL: https://github.com/apache/spark/pull/34284#issuecomment-943259450
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144247/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34284: [SPARK-36900][CORE][TEST] Refactor `SPARK-36464: size returns correct positive number even with over 2GB data` to pass with Java 8, 11 and 17
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34284:
URL: https://github.com/apache/spark/pull/34284#issuecomment-944437643
**[Test build #144305 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144305/testReport)** for PR 34284 at commit [`5cfc955`](https://github.com/apache/spark/commit/5cfc9552b10887fadc3a77a8d6bc04535d9ecf3e).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #34284: [SPARK-36900][CORE][TEST] Refactor `ChunkedByteBufferOutputStreamSuite` to pass `SPARK-36464: size returns correct positive number even with over 2GB data` with Java 8, 11 and 17
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #34284:
URL: https://github.com/apache/spark/pull/34284#issuecomment-943134611
cc @srowen this case can pass with Java 8, 11 and 17 now, although I still haven't found the root cause.
I found that there are a large number of `int []` (about 2g memory usage) in the oom memory dump, but these `int []` do not have GC root, so I haven't found out who created them...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #34284: [SPARK-36900][CORE][TEST] Refactor `SPARK-36464: size returns correct positive number even with over 2GB data` to pass with Java 8, 11 and 17
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34284:
URL: https://github.com/apache/spark/pull/34284#discussion_r728984888
##########
File path: core/src/test/scala/org/apache/spark/util/io/ChunkedByteBufferOutputStreamSuite.scala
##########
@@ -121,12 +121,10 @@ class ChunkedByteBufferOutputStreamSuite extends SparkFunSuite {
}
test("SPARK-36464: size returns correct positive number even with over 2GB data") {
- val ref = new Array[Byte](1024 * 1024 * 1024)
- val o = new ChunkedByteBufferOutputStream(1024 * 1024, ByteBuffer.allocate)
- o.write(ref)
- o.write(ref)
- o.close()
+ val ref = new Array[Byte](1024 * 1024 * 4)
+ val o = new ChunkedByteBufferOutputStream(1024 * 1024 * 4, ByteBuffer.allocate)
+ (0 to 512).foreach(_ => o.write(ref))
Review comment:
Yes，write 513 times.
It was my mistake to lose ｀close（）｀， I'll fix it later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34284: [SPARK-36900][CORE][TEST] Refactor `ChunkedByteBufferOutputStreamSuite` to pass `SPARK-36464: size returns correct positive number even with over 2GB data` with Java 8, 11 and 17
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34284:
URL: https://github.com/apache/spark/pull/34284#issuecomment-943133077
**[Test build #144247 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144247/testReport)** for PR 34284 at commit [`05835e3`](https://github.com/apache/spark/commit/05835e32c4f57f44047b9ac780b715f7bfbeb3c8).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #34284: [SPARK-36900][CORE][TEST] Refactor `SPARK-36464: size returns correct positive number even with over 2GB data` to pass with Java 8, 11 and 17
Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #34284:
URL: https://github.com/apache/spark/pull/34284#issuecomment-944435193
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen closed pull request #34284: [SPARK-36900][CORE][TEST] Refactor `SPARK-36464: size returns correct positive number even with over 2GB data` to pass with Java 8, 11 and 17
Posted by GitBox <gi...@apache.org>.
srowen closed pull request #34284:
URL: https://github.com/apache/spark/pull/34284
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34284: [SPARK-36900][CORE][TEST] Refactor `SPARK-36464: size returns correct positive number even with over 2GB data` to pass with Java 8, 11 and 17
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34284:
URL: https://github.com/apache/spark/pull/34284#issuecomment-943257078
**[Test build #144247 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144247/testReport)** for PR 34284 at commit [`05835e3`](https://github.com/apache/spark/commit/05835e32c4f57f44047b9ac780b715f7bfbeb3c8).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #34284: [SPARK-36900][CORE][TEST] Refactor `SPARK-36464: size returns correct positive number even with over 2GB data` to pass with Java 8, 11 and 17
Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #34284:
URL: https://github.com/apache/spark/pull/34284#discussion_r729964418
##########
File path: core/src/test/scala/org/apache/spark/util/io/ChunkedByteBufferOutputStreamSuite.scala
##########
@@ -121,12 +121,13 @@ class ChunkedByteBufferOutputStreamSuite extends SparkFunSuite {
}
test("SPARK-36464: size returns correct positive number even with over 2GB data") {
- val ref = new Array[Byte](1024 * 1024 * 1024)
- val o = new ChunkedByteBufferOutputStream(1024 * 1024, ByteBuffer.allocate)
- o.write(ref)
- o.write(ref)
+ val data4M = 1024 * 1024 * 4
Review comment:
This all seems fine. I was thinking something more like this but I really don't think it matters.
```
val writeTimes = 512
val chunkSize = 2 * 1024 * 1024 / writeTimes
...
(0 until writeTimes + 1)....
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34284: [SPARK-36900][CORE][TEST] Refactor `SPARK-36464: size returns correct positive number even with over 2GB data` to pass with Java 8, 11 and 17
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34284:
URL: https://github.com/apache/spark/pull/34284#issuecomment-943259450
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144247/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #34284: [SPARK-36900][CORE][TEST] Refactor `SPARK-36464: size returns correct positive number even with over 2GB data` to pass with Java 8, 11 and 17
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #34284:
URL: https://github.com/apache/spark/pull/34284#issuecomment-943307619
cc @dongjoon-hyun @srowen
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34284: [SPARK-36900][CORE][TEST] Refactor `SPARK-36464: size returns correct positive number even with over 2GB data` to pass with Java 8, 11 and 17
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34284:
URL: https://github.com/apache/spark/pull/34284#issuecomment-943205760
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48727/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34284: [SPARK-36900][CORE][TEST] Refactor `SPARK-36464: size returns correct positive number even with over 2GB data` to pass with Java 8, 11 and 17
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34284:
URL: https://github.com/apache/spark/pull/34284#issuecomment-943217319
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48727/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #34284: [SPARK-36900][CORE][TEST] Refactor `SPARK-36464: size returns correct positive number even with over 2GB data` to pass with Java 8, 11 and 17
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #34284:
URL: https://github.com/apache/spark/pull/34284#issuecomment-944481456
Thank you for pinging me, @LuciferYang .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #34284: [SPARK-36900][CORE][TEST] Refactor `SPARK-36464: size returns correct positive number even with over 2GB data` to pass with Java 8, 11 and 17
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #34284:
URL: https://github.com/apache/spark/pull/34284#issuecomment-974529623
Hi, All.
I'll cherry-pick this test PR to branch-3.2 to improve `branch-3.2` stability.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34284: [SPARK-36900][CORE][TEST] Refactor `SPARK-36464: size returns correct positive number even with over 2GB data` to pass with Java 8, 11 and 17
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34284:
URL: https://github.com/apache/spark/pull/34284#issuecomment-943133077
**[Test build #144247 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144247/testReport)** for PR 34284 at commit [`05835e3`](https://github.com/apache/spark/commit/05835e32c4f57f44047b9ac780b715f7bfbeb3c8).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34284: [SPARK-36900][CORE][TEST] Refactor `SPARK-36464: size returns correct positive number even with over 2GB data` to pass with Java 8, 11 and 17
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34284:
URL: https://github.com/apache/spark/pull/34284#issuecomment-944437643
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] kazuyukitanimura commented on a change in pull request #34284: [SPARK-36900][CORE][TEST] Refactor `SPARK-36464: size returns correct positive number even with over 2GB data` to pass with Java 8, 11 and 17
Posted by GitBox <gi...@apache.org>.
kazuyukitanimura commented on a change in pull request #34284:
URL: https://github.com/apache/spark/pull/34284#discussion_r729986884
##########
File path: core/src/test/scala/org/apache/spark/util/io/ChunkedByteBufferOutputStreamSuite.scala
##########
@@ -121,12 +121,13 @@ class ChunkedByteBufferOutputStreamSuite extends SparkFunSuite {
}
test("SPARK-36464: size returns correct positive number even with over 2GB data") {
- val ref = new Array[Byte](1024 * 1024 * 1024)
- val o = new ChunkedByteBufferOutputStream(1024 * 1024, ByteBuffer.allocate)
- o.write(ref)
- o.write(ref)
+ val data4M = 1024 * 1024 * 4
Review comment:
LGTM (non-binding)
Or alternatively, .... (twisting Sean's point a bit)
```
val chunkSize = 1024 * 1024 * 4
val writeTimes = 2L * 1024 * 1024 * 1024 / chunkSize + 1 // = 513
...
(0 until writeTimes)....
```
##########
File path: core/src/test/scala/org/apache/spark/util/io/ChunkedByteBufferOutputStreamSuite.scala
##########
@@ -121,12 +121,10 @@ class ChunkedByteBufferOutputStreamSuite extends SparkFunSuite {
}
test("SPARK-36464: size returns correct positive number even with over 2GB data") {
- val ref = new Array[Byte](1024 * 1024 * 1024)
- val o = new ChunkedByteBufferOutputStream(1024 * 1024, ByteBuffer.allocate)
- o.write(ref)
- o.write(ref)
- o.close()
+ val ref = new Array[Byte](1024 * 1024 * 4)
+ val o = new ChunkedByteBufferOutputStream(1024 * 1024 * 4, ByteBuffer.allocate)
Review comment:
Nice finding. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34284: [SPARK-36900][CORE][TEST] Refactor `SPARK-36464: size returns correct positive number even with over 2GB data` to pass with Java 8, 11 and 17
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34284:
URL: https://github.com/apache/spark/pull/34284#issuecomment-944505733
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34284: [SPARK-36900][CORE][TEST] Refactor `SPARK-36464: size returns correct positive number even with over 2GB data` to pass with Java 8, 11 and 17
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34284:
URL: https://github.com/apache/spark/pull/34284#issuecomment-944505733
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34284: [SPARK-36900][CORE][TEST] Refactor `SPARK-36464: size returns correct positive number even with over 2GB data` to pass with Java 8, 11 and 17
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34284:
URL: https://github.com/apache/spark/pull/34284#issuecomment-943171094
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48727/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34284: [SPARK-36900][CORE][TEST] Refactor `SPARK-36464: size returns correct positive number even with over 2GB data` to pass with Java 8, 11 and 17
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34284:
URL: https://github.com/apache/spark/pull/34284#issuecomment-943217319
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48727/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org