You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "TejMakode1523 (via GitHub)" <gi...@apache.org> on 2026/02/02 06:26:57 UTC
[I] [Bug] Adding external Parquet data files to an existing Paimon table [paimon]
TejMakode1523 opened a new issue, #7173:
URL: https://github.com/apache/paimon/issues/7173
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Paimon version
I’m evaluating Apache Paimon for migrating an existing data lake and would like to clarify the intended design and supported patterns around adding external Parquet data files into a Paimon table.
Any solution on it?
### Compute Engine
NA
### Minimal reproduce step
NA
### What doesn't meet your expectations?
NA
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Adding external Parquet data files to an existing Paimon table [paimon]
Posted by "TejMakode1523 (via GitHub)" <gi...@apache.org>.
TejMakode1523 commented on issue #7173:
URL: https://github.com/apache/paimon/issues/7173#issuecomment-3834001402
@steFaiz , Thanks for the response.
To clarify, my question is not about migration documentation, but specifically about loading data that already exists as Parquet files into a Paimon table.
I want to understand whether Apache Paimon provides any supported mechanism to add or register external Parquet files into an existing Paimon table (for example, similar to Hive external tables or Hudi bootstrap), without rewriting the data through a full Flink INSERT job.
If this is intentionally unsupported, it would be helpful to confirm explicitly that:
External Parquet files cannot be added or referenced directly by a Paimon table
The only supported way to load data is by reading the Parquet files and rewriting them through Paimon writers (Flink SQL / Table API / DataStream)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Adding external Parquet data files to an existing Paimon table [paimon]
Posted by "steFaiz (via GitHub)" <gi...@apache.org>.
steFaiz commented on issue #7173:
URL: https://github.com/apache/paimon/issues/7173#issuecomment-3833532740
Hello, here's the [docs](https://paimon.apache.org/docs/1.3/migration/migration-from-iceberg/) about migration. You can also find how to migrate from hive there. Could you explain why you consider this a bug?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Adding external Parquet data files to an existing Paimon table [paimon]
Posted by "steFaiz (via GitHub)" <gi...@apache.org>.
steFaiz commented on issue #7173:
URL: https://github.com/apache/paimon/issues/7173#issuecomment-3834194423
@TejMakode1523 Thanks for your explaination!
> whether Apache Paimon provides any supported mechanism to add or register external Parquet files into an existing Paimon table (for example, similar to Hive external tables or Hudi bootstrap), without rewriting the data through a full Flink INSERT job.
The answer is: yes! Hive Migration and Iceberge Migration will only involves metadata copy and data file rename, which is much more lighter than rewriting full-table. You can refer to implementations of `org.apache.paimon.migrate.Migrator` if you are interested. But there are some limitations:
1. only support migrating to an APPEND table in paimon
2. the original hive/iceberg table will disapper
3. more
You can refer to the docs for details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Adding external Parquet data files to an existing Paimon table [paimon]
Posted by "Zerolzj (via GitHub)" <gi...@apache.org>.
Zerolzj commented on issue #7173:
URL: https://github.com/apache/paimon/issues/7173#issuecomment-5552907234
Thanks for clarifying that the current Hive and Iceberg migrators can perform metadata-only
migration, while generic files do not have an `add_files`-style API.
I would like to ask about the supported API boundary for a third-party migrator. The target use case
is a fixed-bucket primary-key table whose immutable files already have a registered Paimon file
format implementation. The migrator can provide partition and bucket identity, schema ID, row count,
key statistics, sequence range, sort-order proof, file size, and a content fingerprint. It wants to
create a normal Paimon snapshot without decoding and rewriting every row.
`DataFileMeta`, `CommitMessage`, and table commit interfaces are public, but the community migration
implementations appear to construct the actual data increment through classes such as
`FileMetaUtils`, `CommitMessageImpl`, and `DataIncrement`, which are not marked as public APIs.
Could maintainers clarify the following?
1. Is using `FileMetaUtils.createCommitMessage` directly an intended and compatibility-supported
extension point for third-party migrators?
2. If not, would the community consider a stable `ExistingFile`/`ImportFile` builder or an
`add_files`-style procedure that validates and commits existing files?
3. Which validation inputs should be mandatory for a primary-key table so an importer cannot create
a snapshot with incorrect bucket placement, key range, sequence semantics, or sort order?
4. Is metadata-only registration expected to be limited to Paimon-owned files, or is a
referenced-but-not-owned file model in scope? `externalPath` expresses location, while normal
snapshot cleanup still owns the referenced data file and its `extraFiles`.
5. If borrowed-file ownership is intentionally out of scope, is a Paimon-owned descriptor plus an
external lifecycle manager the recommended integration boundary?
This does not require a generic file parser in Paimon: format-specific inspection and validation can
remain in the migrator. The requested public surface would only define the metadata and commit
contract. I can provide a prototype and negative tests once the intended boundary is clear.
References:
- Existing-file question: https://github.com/apache/paimon/issues/7173
- Multi-location and `add_files` discussion:
https://github.com/apache/paimon/issues/3627#issuecomment-2664980472
- `DataFileMeta`:
https://github.com/apache/paimon/blob/master/paimon-core/src/main/java/org/apache/paimon/io/DataFileMeta.java
- `FileMetaUtils`:
https://github.com/apache/paimon/blob/master/paimon-core/src/main/java/org/apache/paimon/migrate/FileMetaUtils.java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Adding external Parquet data files to an existing Paimon table [paimon]
Posted by "steFaiz (via GitHub)" <gi...@apache.org>.
steFaiz commented on issue #7173:
URL: https://github.com/apache/paimon/issues/7173#issuecomment-3839519232
@TejMakode1523 As far as I know, yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Adding external Parquet data files to an existing Paimon table [paimon]
Posted by "TejMakode1523 (via GitHub)" <gi...@apache.org>.
TejMakode1523 commented on issue #7173:
URL: https://github.com/apache/paimon/issues/7173#issuecomment-3845496480
@steFaiz I didn't see any proper documentation for reference.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Adding external Parquet data files to an existing Paimon table [paimon]
Posted by "TejMakode1523 (via GitHub)" <gi...@apache.org>.
TejMakode1523 commented on issue #7173:
URL: https://github.com/apache/paimon/issues/7173#issuecomment-3834316520
Thanks for the clarification.
To confirm my understanding: for generic external Parquet files (not managed by Hive or Iceberg metadata), Apache Paimon does not provide any built-in configuration or functionality to attach or register those files into an existing table.
The only supported way to load such data is to read the Parquet files using an engine (e.g. Flink SQL / Table API / DataStream) and rewrite the data through Paimon writers, so that snapshots and manifests are generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org