You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2020/11/05 07:48:59 UTC
[GitHub] [iceberg] paulzanmei opened a new issue #1724: The row-level deletion is complete, and the conditional filter query null pointer exception
paulzanmei opened a new issue #1724:
URL: https://github.com/apache/iceberg/issues/1724
I call the row-level delete API to delete a single record, and a null pointer exception occurs when calling the filter condition query API
Step#1: Delete data based on conditions
```
HadoopCatalog catalog = new HadoopCatalog(conf, warehousePath);
Table table = catalog.loadTable(TableIdentifier.of("test","customers_v2_5"));
Schema deleteRowSchema = table.schema().select("customer_id");
Record dataDelete = GenericRecord.create(deleteRowSchema);
Long id = 16211L;
List<Record> dataDeletes = Lists.newArrayList(
dataDelete.copy("customer_id", id)
);
Map<String, String> props = table.properties();
FileFormat fileFormat = getFileFormat(props);
OutputFileFactory fileFactory = new OutputFileFactory(table.spec(), fileFormat, table.locationProvider(), table.io(), table.encryption(), 1, 1);
OutputFile temp = fileFactory.newOutputFile().encryptingOutputFile();
EqualityDeleteWriter<Record> writer = Parquet.writeDeletes(temp)
.forTable(table)
.withPartition(null)
.rowSchema(deleteRowSchema)
.createWriterFunc(GenericParquetWriter::buildWriter)
.overwrite()
.equalityFieldIds(deleteRowSchema.columns().stream().mapToInt(Types.NestedField::fieldId).toArray())
.buildEqualityWriter();
try (Closeable toClose = writer) {
writer.writeAll(dataDeletes);
}
DeleteFile eqDeletes = writer.toContentFile();
table.newRowDelta()
.addDeletes(eqDeletes)
.commit();
```
execution succeed
Step#2: Conditional query failed
```
HadoopCatalog catalog = new HadoopCatalog(conf, warehousePath);
Table table = catalog.loadTable(TableIdentifier.of("test","customers_v2_5"));
Iterable<Record> results = IcebergGenerics.read(table).where(Expressions.equal("customer_id", 16219)).build();
for (Record record : results) {
System.out.println(record.toString());
}
```
```
java.lang.NullPointerException
at org.apache.iceberg.ManifestReader.requireStatsProjection(ManifestReader.java:274)
at org.apache.iceberg.ManifestReader.entries(ManifestReader.java:175)
at org.apache.iceberg.ManifestReader.liveEntries(ManifestReader.java:215)
at org.apache.iceberg.DeleteFileIndex$Builder.lambda$deleteManifestReaders$11(DeleteFileIndex.java:452)
at org.apache.iceberg.relocated.com.google.common.collect.Iterators$6.transform(Iterators.java:783)
at org.apache.iceberg.relocated.com.google.common.collect.TransformedIterator.next(TransformedIterator.java:47)
at org.apache.iceberg.util.Tasks$Builder.runParallel(Tasks.java:301)
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:195)
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:189)
at org.apache.iceberg.DeleteFileIndex$Builder.build(DeleteFileIndex.java:357)
at org.apache.iceberg.ManifestGroup.planFiles(ManifestGroup.java:165)
at org.apache.iceberg.DataTableScan.planFiles(DataTableScan.java:89)
at org.apache.iceberg.BaseTableScan.planFiles(BaseTableScan.java:211)
at org.apache.iceberg.DataTableScan.planFiles(DataTableScan.java:28)
at org.apache.iceberg.BaseTableScan.planTasks(BaseTableScan.java:244)
at org.apache.iceberg.DataTableScan.planTasks(DataTableScan.java:28)
at org.apache.iceberg.data.TableScanIterable.<init>(TableScanIterable.java:36)
at org.apache.iceberg.data.IcebergGenerics$ScanBuilder.build(IcebergGenerics.java:91)
at org.datacloud.flinksql.hive.HiveTest.testFilter(HiveTest.java:97)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:89)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:41)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:541)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:763)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:463)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:209)
```
If query without conditions:
```
Iterable<Record> results = IcebergGenerics.read(table).build();
```
Can query all data and filter the deleted data
May I ask what is the problem, am I using it wrong?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] paulzanmei closed issue #1724: The row-level deletion is complete, and the conditional filter query null pointer exception
Posted by GitBox <gi...@apache.org>.
paulzanmei closed issue #1724:
URL: https://github.com/apache/iceberg/issues/1724
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] liubo1022126 commented on issue #1724: The row-level deletion is complete, and the conditional filter query null pointer exception
Posted by GitBox <gi...@apache.org>.
liubo1022126 commented on issue #1724:
URL: https://github.com/apache/iceberg/issues/1724#issuecomment-751335603
I have a test like Step#1, but get exception : IllegalArgumentException: Cannot write delete files in a v1 table.
I look over https://iceberg.apache.org/spec/#table-metadata for format-version, it said that An integer version number for the format. Currently, this is always 1. Implementations must throw an exception if a table’s version is higher than the supported version. so what can i do to test row-level deletion ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org