You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "leaves12138 (via GitHub)" <gi...@apache.org> on 2026/08/10 12:56:53 UTC
[PR] [core] Remove configurable manifest format [paimon]
leaves12138 opened a new pull request, #9154:
URL: https://github.com/apache/paimon/pull/9154
## Purpose
Manifest files are standardized on Avro, and the Python implementation already supports Avro manifests only. Keeping `manifest.format` exposes unsupported alternative formats and can make manifests unreadable across implementations.
## Changes
- Remove the `manifest.format` core option.
- Always create manifest readers and writers with the Avro file format.
- Remove ORC-specific manifest tests and update affected test fixtures.
- Remove the option from the generated configuration documentation.
- Keep `manifest.compression` configurable.
## Tests
- `IndexManifestFileHandlerTest`, `ManifestsTableTest`, and `DataEvolutionRowIdReassignerTest` (69 tests)
- `FileFormatTest#testManifestFormatIsAlwaysAvro`
- `CatalogTableITCase#testSchemasTable`
- `mvn -pl paimon-docs -am -DskipTests package`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Remove configurable manifest format [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #9154:
URL: https://github.com/apache/paimon/pull/9154
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Remove configurable manifest format [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #9154:
URL: https://github.com/apache/paimon/pull/9154#issuecomment-5240794360
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Remove configurable manifest format [paimon]
Posted by "huangxiaopingRD (via GitHub)" <gi...@apache.org>.
huangxiaopingRD commented on PR #9154:
URL: https://github.com/apache/paimon/pull/9154#issuecomment-5274934234
This change will render some tables that didn't use the Avro manifest format unreadable. Do we need to ensure compatibility when reading historical tables?
Below is an exception that occurred when reading a table with an ORC manifest format.
```
Caused by: org.apache.paimon.shade.org.apache.avro.InvalidAvroMagicException: Not an Avro data file
at org.apache.paimon.shade.org.apache.avro.file.DataFileReader.openReader(DataFileReader.java:79)
at org.apache.paimon.format.avro.AvroBulkFormat$AvroReader.createReaderFromPath(AvroBulkFormat.java:87)
at org.apache.paimon.format.avro.AvroBulkFormat$AvroReader.<init>(AvroBulkFormat.java:71)
at org.apache.paimon.format.avro.AvroBulkFormat.createReader(AvroBulkFormat.java:55)
at org.apache.paimon.utils.FileUtils.createFormatReader(FileUtils.java:144)
at org.apache.paimon.utils.ObjectsFile.lambda$new$0(ObjectsFile.java:71)
at org.apache.paimon.utils.ObjectsFile.createIterator(ObjectsFile.java:226)
at org.apache.paimon.utils.SimpleObjectsCache.createSegments(SimpleObjectsCache.java:71)
at org.apache.paimon.utils.ObjectsCache.read(ObjectsCache.java:88)
at org.apache.paimon.utils.ObjectsFile.readWithIOException(ObjectsFile.java:179)
at org.apache.paimon.utils.ObjectsFile.read(ObjectsFile.java:156)
at org.apache.paimon.utils.ObjectsFile.read(ObjectsFile.java:127)
at org.apache.paimon.manifest.ManifestList.readDataManifests(ManifestList.java:89)
at org.apache.paimon.operation.ManifestsReader.readManifests(ManifestsReader.java:161)
at org.apache.paimon.operation.ManifestsReader.read(ManifestsReader.java:144)
at org.apache.paimon.operation.AbstractFileStoreScan.readManifests(AbstractFileStoreScan.java:462)
at org.apache.paimon.operation.AbstractFileStoreScan.plan(AbstractFileStoreScan.java:299)
at org.apache.paimon.table.source.snapshot.SnapshotReaderImpl.read(SnapshotReaderImpl.java:394)
at org.apache.paimon.table.source.snapshot.ReadPlanStartingScanner.scan(ReadPlanStartingScanner.java:45)
at org.apache.paimon.table.source.AbstractBatchTableScan.lambda$planWithoutAuth$1(AbstractBatchTableScan.java:134)
at java.base/java.util.Optional.orElseGet(Optional.java:364)
at org.apache.paimon.table.source.AbstractBatchTableScan.planWithoutAuth(AbstractBatchTableScan.java:134)
at org.apache.paimon.table.source.AbstractDataTableScan.plan(AbstractDataTableScan.java:113)
at org.apache.paimon.spark.PaimonBaseScan.getInputSplits(PaimonBaseScan.scala:58)
at org.apache.paimon.spark.read.PaimonSupportsRuntimeFiltering.inputSplits(PaimonSupportsRuntimeFiltering.scala:43)
at org.apache.paimon.spark.read.PaimonSupportsRuntimeFiltering.inputSplits$(PaimonSupportsRuntimeFiltering.scala:41)
at org.apache.paimon.spark.PaimonBaseScan.inputSplits(PaimonBaseScan.scala:42)
at org.apache.paimon.spark.read.BaseScan.estimateStatistics(BaseScan.scala:152)
at org.apache.paimon.spark.read.BaseScan.estimateStatistics$(BaseScan.scala:150)
at org.apache.paimon.spark.PaimonBaseScan.estimateStatistics(PaimonBaseScan.scala:167)
at org.apache.spark.sql.execution.datasources.v2.DescribeTableExec.addTableStats(DescribeTableExec.scala:146)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org