You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "fuhaiq (via GitHub)" <gi...@apache.org> on 2023/08/10 09:18:26 UTC
[GitHub] [arrow] fuhaiq opened a new issue, #37104: java.lang.IndexOutOfBoundsException when using VectorSchemaRootAppender.append
fuhaiq opened a new issue, #37104:
URL: https://github.com/apache/arrow/issues/37104
### Describe the bug, including details regarding any error messages, version, and platform.
I'm using `VectorSchemaRootAppender.append` append batch data into a final VectorSchemaRoot
```
VectorSchemaRoot result = null;
ScanOptions options = new ScanOptions(/*batchSize*/ 32768, projection);
try (
var allocator = new RootAllocator();
var datasetFactory = new FileSystemDatasetFactory(allocator,
NativeMemoryPool.getDefault(), FileFormat.PARQUET, uri);
Dataset dataset = datasetFactory.finish();
Scanner scanner = dataset.newScan(options);
ArrowReader reader = scanner.scanBatches()
) {
while (reader.loadNextBatch()) {
try (VectorSchemaRoot root = reader.getVectorSchemaRoot()) {
if(result == null) {
result = VectorSchemaRoot.create(root.getSchema(), allocator);
}
VectorSchemaRootAppender.append(result, root);
}
}
return result;
} catch (Exception e) {
throw new RuntimeException(e);
}
```
But exception as bellowed
```
Caused by: java.lang.IndexOutOfBoundsException: index: 0, length: 4 (expected: range(0, 0))
at org.apache.arrow.memory.ArrowBuf.checkIndexD(ArrowBuf.java:319)
at org.apache.arrow.memory.ArrowBuf.chk(ArrowBuf.java:306)
at org.apache.arrow.memory.ArrowBuf.getInt(ArrowBuf.java:425)
at org.apache.arrow.vector.util.VectorAppender.visit(VectorAppender.java:111)
at org.apache.arrow.vector.util.VectorAppender.visit(VectorAppender.java:46)
at org.apache.arrow.vector.BaseVariableWidthVector.accept(BaseVariableWidthVector.java:1424)
at org.apache.arrow.vector.util.VectorSchemaRootAppender.append(VectorSchemaRootAppender.java:67)
at org.apache.arrow.vector.util.VectorSchemaRootAppender.append(VectorSchemaRootAppender.java:81)
```
### Component(s)
Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] fuhaiq closed issue #37104: [Java] java.lang.IndexOutOfBoundsException when using VectorSchemaRootAppender.append
Posted by "fuhaiq (via GitHub)" <gi...@apache.org>.
fuhaiq closed issue #37104: [Java] java.lang.IndexOutOfBoundsException when using VectorSchemaRootAppender.append
URL: https://github.com/apache/arrow/issues/37104
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java] java.lang.IndexOutOfBoundsException when using VectorSchemaRootAppender.append [arrow]
Posted by "muyihao (via GitHub)" <gi...@apache.org>.
muyihao commented on issue #37104:
URL: https://github.com/apache/arrow/issues/37104#issuecomment-2119188786
when I allocateNew before append data, then the data of result before append was cleaned, what I want to do is to append two or even more batch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] fuhaiq commented on issue #37104: [Java] java.lang.IndexOutOfBoundsException when using VectorSchemaRootAppender.append
Posted by "fuhaiq (via GitHub)" <gi...@apache.org>.
fuhaiq commented on issue #37104:
URL: https://github.com/apache/arrow/issues/37104#issuecomment-1679882480
`allocateNew` before append data
```
if(result == null) {
result = VectorSchemaRoot.create(root.getSchema(), ExecutionContext.instance().allocator());
result.allocateNew();
}
VectorSchemaRootAppender.append(result, root);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java] java.lang.IndexOutOfBoundsException when using VectorSchemaRootAppender.append [arrow]
Posted by "muyihao (via GitHub)" <gi...@apache.org>.
muyihao commented on issue #37104:
URL: https://github.com/apache/arrow/issues/37104#issuecomment-2119190890
> when I allocateNew before append data, then the data of result before append was cleaned, what I want to do is to append two or even more batch.
` public static void newMerge() {
VectorSchemaRoot root = null;
for (int i = 0; i < 100; i++) {
VectorSchemaRoot vectorSchemaRoot = createVectorSchemaRoot(ArrowUtil.rootAllocator);
if (root == null) {
root = vectorSchemaRoot;
} else {
if (root.getRowCount() <= 10) {
VectorSchemaRootAppender.append(root, vectorSchemaRoot);
} else {
System.out.println("merged:" + root.contentToTSVString());
root = vectorSchemaRoot;
root.allocateNew();
}
}
vectorSchemaRoot.close();
}
}`
Writing this way will report an error: java.lang.IndexOutOfBoundsException
while write as bellow will result null data
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on issue #37104: java.lang.IndexOutOfBoundsException when using VectorSchemaRootAppender.append
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #37104:
URL: https://github.com/apache/arrow/issues/37104#issuecomment-1673042161
CC @davisusanibar
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] davisusanibar commented on issue #37104: [Java] java.lang.IndexOutOfBoundsException when using VectorSchemaRootAppender.append
Posted by "davisusanibar (via GitHub)" <gi...@apache.org>.
davisusanibar commented on issue #37104:
URL: https://github.com/apache/arrow/issues/37104#issuecomment-1680267441
Thank you @fuhaiq to share that. Just added your solution as part of the Arrow Java Cookbooks - https://github.com/apache/arrow-cookbook/pull/318
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org