You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "ahmedabu98 (via GitHub)" <gi...@apache.org> on 2026/04/14 15:45:41 UTC
[PR] [Iceberg AddFiles] Attach name-mapping to table [beam]
ahmedabu98 opened a new pull request, #38189:
URL: https://github.com/apache/beam/pull/38189
Helps engines to read legacy parquet files that don't have schema-id mapping present.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Iceberg AddFiles] Attach name-mapping to table [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on PR #38189:
URL: https://github.com/apache/beam/pull/38189#issuecomment-4245534878
Added to existing tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Iceberg AddFiles] Attach name-mapping to table [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38189:
URL: https://github.com/apache/beam/pull/38189#issuecomment-4246615236
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Iceberg AddFiles] Attach name-mapping to table [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on PR #38189:
URL: https://github.com/apache/beam/pull/38189#issuecomment-4269648285
Fix: #38232
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Iceberg AddFiles] Attach name-mapping to table [beam]
Posted by "gemini-code-assist[bot] (via GitHub)" <gi...@apache.org>.
gemini-code-assist[bot] commented on PR #38189:
URL: https://github.com/apache/beam/pull/38189#issuecomment-4245252984
## Summary of Changes
Hello, I'm Gemini Code Assist[^1]! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request updates the Iceberg table management logic to automatically attach a name mapping to tables if one is not already defined. By forcing name-based resolution, it ensures better interoperability with legacy Parquet files that do not contain explicit schema-id mappings, thereby preventing potential read failures in downstream engines.
### Highlights
* **Name Mapping Integration**: Added logic to ensure that a default name mapping is present in the Iceberg table properties upon table creation or loading.
* **Compatibility Improvement**: Enabled name-based resolution for legacy Parquet files that lack schema-id mapping, improving read compatibility for various engines.
🧠**New Feature in Public Preview:** You can now enable **Memory** to help **Gemini Code Assist** learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. **Click [here](https://codeassist.google/code-review/login) to enable Memory in your admin console.**
<details>
<summary><b>Using Gemini Code Assist</b></summary>
<br>
The full guide for Gemini Code Assist can be found on our [documentation page](https://developers.google.com/gemini-code-assist/docs/review-github-code), here are some quick tips.
<b>Invoking Gemini</b>
You can request assistance from Gemini at any point by creating a comment using either `/gemini <command>` or `@gemini-code-assist <command>`. Below is a summary of the supported commands on the current page.
Feature | Command | Description
--- | --- | ---
Code Review | `/gemini review` | Performs a code review for the current pull request in its current state.
Pull Request Summary | `/gemini summary` | Provides a summary of the current pull request in its current state.
Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help | `/gemini help` | Displays a list of available commands.
<b>Customization</b>
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a `.gemini/` folder in the base of the repository. Detailed instructions can be found [here](https://developers.google.com/gemini-code-assist/docs/customize-gemini-behavior-github).
<b>Limitations & Feedback</b>
Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with :thumbsup: and :thumbsdown: on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up [here](https://google.qualtrics.com/jfe/form/SV_2cyuGuTWsEw84yG).
</details>
[^1]: Review the [Privacy Notices](https://policies.google.com/privacy), [Generative AI Prohibited Use Policy](https://policies.google.com/terms/generative-ai/use-policy), [Terms of Service](https://policies.google.com/terms), and learn how to configure Gemini Code Assist in GitHub [here](https://developers.google.com/gemini-code-assist/docs/customize-gemini-behavior-github). Gemini can make mistakes, so double check it and [use code with caution](https://support.google.com/legal/answer/13505487).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Iceberg AddFiles] Attach name-mapping to table [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 merged PR #38189:
URL: https://github.com/apache/beam/pull/38189
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Iceberg AddFiles] Attach name-mapping to table [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #38189:
URL: https://github.com/apache/beam/pull/38189#issuecomment-4269429320
Actually this one likely breaks Iceberg integration test: , by comparing the history betwwen https://github.com/apache/beam/commits/ae536b6f0f83b978854f1d6852fbacb62a3e5ec0 https://github.com/apache/beam/commits/4ee32cddb4c10dcf738558e4ffcafc50255a1bd0
#31931
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Iceberg AddFiles] Attach name-mapping to table [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #38189:
URL: https://github.com/apache/beam/pull/38189#issuecomment-4269515034
```
2026-04-17T10:48:14.7626797Z AddFilesIT > testStreamingParquetImport FAILED
2026-04-17T10:48:14.7630365Z org.awaitility.core.ConditionTimeoutException: Condition with lambda expression in org.apache.beam.sdk.io.iceberg.AddFilesIT was not fulfilled within 2 minutes.
2026-04-17T10:48:14.7633163Z at app//org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:167)
2026-04-17T10:48:14.7635247Z at app//org.awaitility.core.CallableCondition.await(CallableCondition.java:78)
2026-04-17T10:48:14.7637449Z at app//org.awaitility.core.CallableCondition.await(CallableCondition.java:26)
2026-04-17T10:48:14.7639404Z at app//org.awaitility.core.ConditionFactory.until(ConditionFactory.java:985)
2026-04-17T10:48:14.7641492Z at app//org.awaitility.core.ConditionFactory.until(ConditionFactory.java:954)
2026-04-17T10:48:14.7643616Z at app//org.apache.beam.sdk.io.iceberg.AddFilesIT.testStreamingParquetImport(AddFilesIT.java:394)
2026-04-17T10:50:55.2625870Z AddFilesIT > testStreamingImportFromExistingIcebergTable FAILED
2026-04-17T10:50:55.2628153Z org.awaitility.core.ConditionTimeoutException: Condition with lambda expression in org.apache.beam.sdk.io.iceberg.AddFilesIT was not fulfilled within 2 minutes.
2026-04-17T10:50:55.2630416Z at app//org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:167)
2026-04-17T10:50:55.2631516Z at app//org.awaitility.core.CallableCondition.await(CallableCondition.java:78)
2026-04-17T10:50:55.2632532Z at app//org.awaitility.core.CallableCondition.await(CallableCondition.java:26)
2026-04-17T10:50:55.2633662Z at app//org.awaitility.core.ConditionFactory.until(ConditionFactory.java:985)
2026-04-17T10:50:55.2634634Z at app//org.awaitility.core.ConditionFactory.until(ConditionFactory.java:954)
2026-04-17T10:50:55.2635943Z at app//org.apache.beam.sdk.io.iceberg.AddFilesIT.testStreamingImportFromExistingIcebergTable(AddFilesIT.java:280)
2026-04-17T10:45:45.6627970Z AddFilesIT > testBatchParquetImport FAILED
2026-04-17T10:45:45.6629898Z org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.util.concurrent.ExecutionException: org.apache.iceberg.exceptions.RESTException: Unable to process: Resource has been exhausted (e.g. check quota).
2026-04-17T10:45:45.6631898Z at app//org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:382)
2026-04-17T10:45:45.6633373Z at app//org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:350)
2026-04-17T10:45:45.6634632Z at app//org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:223)
2026-04-17T10:45:45.6636160Z at app//org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:69)
2026-04-17T10:45:45.6638080Z at app//org.apache.beam.sdk.Pipeline.run(Pipeline.java:325)
2026-04-17T10:45:45.6639132Z at app//org.apache.beam.sdk.Pipeline.run(Pipeline.java:310)
2026-04-17T10:45:45.6640200Z at app//org.apache.beam.sdk.io.iceberg.AddFilesIT.testBatchParquetImport(AddFilesIT.java:462)
2026-04-17T10:45:45.6641335Z
2026-04-17T10:45:45.6641481Z Caused by:
2026-04-17T10:45:45.6642601Z java.util.concurrent.ExecutionException: org.apache.iceberg.exceptions.RESTException: Unable to process: Resource has been exhausted (e.g. check quota).
```
also there is a log suggesting resource leaking (might be pre-exising issue)
```
2026-04-17T11:31:51.3624913Z Apr 17, 2026 11:31:51 AM org.apache.beam.sdk.io.iceberg.catalog.IcebergCatalogBaseIT cleanUp
2026-04-17T11:31:51.3626195Z WARNING: Failed to clean up GCS files.
2026-04-17T11:31:51.3628967Z java.lang.UnsupportedOperationException: Can't get filename from root path in the bucket: gs://managed-iceberg-biglake-its/
2026-04-17T11:31:51.3630467Z at org.apache.beam.sdk.extensions.gcp.util.gcsfs.GcsPath.getFileName(GcsPath.java:255)
2026-04-17T11:31:51.3631698Z at org.apache.beam.sdk.io.iceberg.catalog.IcebergCatalogBaseIT.cleanUp(IcebergCatalogBaseIT.java:214)
2026-04-17T11:31:51.3633195Z at jdk.internal.reflect.GeneratedMethodAccessor363.invoke(Unknown Source)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org