You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@amoro.apache.org by "wangmingjin163 (via GitHub)" <gi...@apache.org> on 2024/08/20 08:43:45 UTC
[I] [Bug]: Issue with Merging Parquet Files Without Field ID Leading to Misaligned Columns [amoro]
wangmingjin163 opened a new issue, #3120:
URL: https://github.com/apache/amoro/issues/3120
### What happened?
I encountered an issue when working with Parquet files in the Amoro project. The problem arises when Parquet files are written by using Arrow Schema without Field IDs, which later causes issues during file merging operations. Specifically, the columns in the merged files become misaligned, resulting in incorrect data projections.
### Affects Versions
0.7.0
### What table formats are you seeing the problem on?
Iceberg
### What engines are you seeing the problem on?
Optimizer
### How to reproduce
1.Create Parquet files using Iceberg schema without including Field IDs.
2.Attempt to merge these Parquet files using Iceberg’s rewriteDataFiles method.
3.Observe that the columns in the merged files are misaligned.
### Relevant log output
_No response_
### Anything else
Proposed Solution:
I added a check to apply NameMapping during the Parquet file reading process. This ensures that fields are correctly mapped by name to their corresponding IDs, preventing misalignment during merging.
The key part of the solution involves using withNameMapping(NameMappingParser.fromJson(nameMapping)) in the Parquet.ReadBuilder when opening Parquet files. This ensures that the schema mapping is handled correctly, even in the absence of Field IDs.
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's Code of Conduct
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Issue with Merging Parquet Files Without Field ID Leading to Misaligned Columns [amoro]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #3120:
URL: https://github.com/apache/amoro/issues/3120#issuecomment-2695847552
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Issue with Merging Parquet Files Without Field ID Leading to Misaligned Columns [amoro]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #3120: [Bug]: Issue with Merging Parquet Files Without Field ID Leading to Misaligned Columns
URL: https://github.com/apache/amoro/issues/3120
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Issue with Merging Parquet Files Without Field ID Leading to Misaligned Columns [amoro]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #3120:
URL: https://github.com/apache/amoro/issues/3120#issuecomment-2661690510
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org