You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "10183974 (via GitHub)" <gi...@apache.org> on 2026/03/09 08:27:34 UTC
[I] [Java][Dataset] OOM Killer triggered by unbounded native memory usage during Parquet read; NativeMemoryPool.createListenable crashes with JNI error #49472 [arrow-java]
10183974 opened a new issue, #1057:
URL: https://github.com/apache/arrow-java/issues/1057
### Describe the bug, including details regarding any error messages, version, and platform.
We are encountering a critical issue when using the Apache Arrow Java Dataset API to read a large local Parquet file (15GB). The JVM process is killed by the OS OOM Killer despite having sufficient Docker container memory (15GB) and conservative JVM heap/direct memory settings.
Additionally, attempting to mitigate this by using a listenable NativeMemoryPool results in a fatal JNI error (JNIEnv was not attached to current thread).
Environment
Apache Arrow Version: 17.0.0
OS: Linux
JDK: jdk8
File Format: Parquet (15GB, generated by Pandas/PyArrow default config)
Hardware: Docker container limited to 15GB RAM.
Reproduction Steps
Configuration
Docker Memory Limit: 15GB
JVM Arguments:
java
-Xms1g -Xmx3g
-XX:MaxDirectMemorySize=3g
-XX:MaxMetaspaceSize=256m
-XX:+UseG1GC
-XX:+PrintGCDetails
-XX:+PrintGCDateStamps
-Xloggc:gc.log
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=./heap_dump.hprof
-Darrow.memory.debug.allocator=true
-jar arrow-test-1.0-SNAPSHOT.jar
Code Logic: Using FileSystemDatasetFactory to read a single 15GB Parquet file with a batch size of 65,536.
2. Code Snippet
java:
>
Runtime runtime = Runtime.getRuntime();
// 单位转换辅助
long mb = 1024 * 1024;
System.out.println("===== JVM 内存概况 =====");
System.out.println("最大可用内存 (Max Memory): " + runtime.maxMemory() / mb + " MB");
System.out.println("已分配内存 (Total Memory): " + runtime.totalMemory() / mb + " MB");
System.out.println("空闲内存 (Free Memory): " + runtime.freeMemory() / mb + " MB");
System.out.println("已使用内存 (Used Memory): " + (runtime.totalMemory() - runtime.freeMemory()) / mb + " MB");
// fileSize: 15G
String hdfsUri = "hdfs://node01.cdh5:8020/user/hive/warehouse/perf_test_200col_500w_nopk_parquet/ds=1/perf_test_200col_500w_nopk_parquet";
ScanOptions options = new ScanOptions(100000);
try (RootAllocator allocator = new RootAllocator(512 * 1024 * 1024);
NativeMemoryPool nativePool = NativeMemoryPool.createListenable(DirectReservationListener.instance());
FileSystemDatasetFactory datasetFactory = new FileSystemDatasetFactory(allocator, nativePool, FileFormat.PARQUET, hdfsUri);
Dataset dataset = datasetFactory.finish();
Scanner scanner = dataset.newScan(options);
ArrowReader reader = scanner.scanBatches()) {
long totalRows = 0;
int batchCount = 0;
while (reader.loadNextBatch()) {
try (VectorSchemaRoot root = reader.getVectorSchemaRoot()) {
int rowCount = root.getRowCount();
totalRows += rowCount;
batchCount++;
System.out.println("Batch " + batchCount + " - rows: " + rowCount + ", total: " + totalRows);
}
}
System.out.println("\n===== Processing Complete =====");
System.out.println("Total batches: " + batchCount);
System.out.println("Total rows: " + totalRows);
} catch (Exception e) {
logger.error("Error processing parquet file", e);
e.printStackTrace();
}
>
Observed Behavior:
Batch 1 - rows: 41547, total: 41547
/mnt/executor/sandbox/shell-0000001022165.sh: line 21: 92 Killed java -Xms1g -Xmx3g -XX:MaxDirectMemorySize=3g -XX:MaxMetaspaceSize=256m -XX:+UseG1GC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:gc.log -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=./heap_dump.hprof -Darrow.memory.debug.allocator=true -jar /mnt/executor/sandbox/resources/0000001022165/resource.arrow-test-1.0-SNAPSHOT.jar
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java][Dataset] OOM Killer triggered by unbounded native memory usage during Parquet read; NativeMemoryPool.createListenable crashes with JNI error #49472 [arrow-java]
Posted by "10183974 (via GitHub)" <gi...@apache.org>.
10183974 commented on issue #1057:
URL: https://github.com/apache/arrow-java/issues/1057#issuecomment-4028022541
> ScanOptions options = new ScanOptions(/_batchSize=_/ 65_536);
>
> try (RootAllocator allocator = new RootAllocator(512 * 1024 * 1024); NativeMemoryPool nativePool = NativeMemoryPool.createListenable( DirectReservationListener.instance()); FileSystemDatasetFactory datasetFactory = new FileSystemDatasetFactory( allocator, nativePool, FileFormat.PARQUET, hdfsUri); Dataset dataset = datasetFactory.finish(); Scanner scanner = dataset.newScan(options); ArrowReader reader = scanner.scanBatches()) {
>
> ```
> long totalRows = 0;
> int batchCount = 0;
>
>
> VectorSchemaRoot root = reader.getVectorSchemaRoot();
>
> while (reader.loadNextBatch()) {
> int rowCount = root.getRowCount();
> totalRows += rowCount;
> batchCount++;
> System.out.println("Batch " + batchCount + " - rows: " + rowCount + ", total: " + totalRows);
> }
>
> System.out.println("Total batches: " + batchCount);
> System.out.println("Total rows: " + totalRows);
> ```
>
> } catch (Exception e) { logger.error("Error processing parquet file", e); }
Thank you for the suggestions. I have re-run the application incorporating your recommended code changes. While the process was able to consume more batches compared to the previous attempt, it ultimately failed with the same underlying issue.
Here are the detailed results and environment configurations:
1. Environment Configuration
Container Memory Limit: 15 GB
JVM Arguments:
bash
java \
-Xms1g -Xmx3g \
-XX:MaxDirectMemorySize=3g \
-XX:MaxMetaspaceSize=256m \
-XX:+UseG1GC \
-XX:+PrintGCDetails \
-XX:+PrintGCDateStamps \
-Xloggc:gc.log \
-XX:+HeapDumpOnOutOfMemoryError \
-XX:HeapDumpPath=./heap_dump.hprof \
-Darrow.memory.debug.allocator=true \
-jar arrow-test-1.0-SNAPSHOT.jar
2. Execution Behavior
The application successfully processed more batches before crashing, indicating some improvement, but it still hit a hard limit:
Successfully processed up to Batch 26 (Total rows: ~1,007,544).
The crash occurred consistently when attempting to reserve memory for subsequent operations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java][Dataset] OOM Killer triggered by unbounded native memory usage during Parquet read; NativeMemoryPool.createListenable crashes with JNI error #49472 [arrow-java]
Posted by "10183974 (via GitHub)" <gi...@apache.org>.
10183974 closed issue #1057: [Java][Dataset] OOM Killer triggered by unbounded native memory usage during Parquet read; NativeMemoryPool.createListenable crashes with JNI error #49472
URL: https://github.com/apache/arrow-java/issues/1057
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java][Dataset] OOM Killer triggered by unbounded native memory usage during Parquet read; NativeMemoryPool.createListenable crashes with JNI error #49472 [arrow-java]
Posted by "10183974 (via GitHub)" <gi...@apache.org>.
10183974 commented on issue #1057:
URL: https://github.com/apache/arrow-java/issues/1057#issuecomment-4053641917
this is indeed a bug. I was able to identify and resolve the issue with the assistance of AI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java][Dataset] OOM Killer triggered by unbounded native memory usage during Parquet read; NativeMemoryPool.createListenable crashes with JNI error #49472 [arrow-java]
Posted by "Rahul-324 (via GitHub)" <gi...@apache.org>.
Rahul-324 commented on issue #1057:
URL: https://github.com/apache/arrow-java/issues/1057#issuecomment-4025991897
ScanOptions options = new ScanOptions(/*batchSize=*/ 65_536);
try (RootAllocator allocator = new RootAllocator(512 * 1024 * 1024);
NativeMemoryPool nativePool = NativeMemoryPool.createListenable(
DirectReservationListener.instance());
FileSystemDatasetFactory datasetFactory = new FileSystemDatasetFactory(
allocator, nativePool, FileFormat.PARQUET, hdfsUri);
Dataset dataset = datasetFactory.finish();
Scanner scanner = dataset.newScan(options);
ArrowReader reader = scanner.scanBatches()) {
long totalRows = 0;
int batchCount = 0;
VectorSchemaRoot root = reader.getVectorSchemaRoot();
while (reader.loadNextBatch()) {
int rowCount = root.getRowCount();
totalRows += rowCount;
batchCount++;
System.out.println("Batch " + batchCount + " - rows: " + rowCount + ", total: " + totalRows);
}
System.out.println("Total batches: " + batchCount);
System.out.println("Total rows: " + totalRows);
} catch (Exception e) {
logger.error("Error processing parquet file", e);
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java][Dataset] OOM Killer triggered by unbounded native memory usage during Parquet read; NativeMemoryPool.createListenable crashes with JNI error #49472 [arrow-java]
Posted by "10183974 (via GitHub)" <gi...@apache.org>.
10183974 commented on issue #1057:
URL: https://github.com/apache/arrow-java/issues/1057#issuecomment-4044190976
@amoeba @mrkn @toddlipcon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org