You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "manuzhang (via GitHub)" <gi...@apache.org> on 2026/09/23 16:38:36 UTC

[PR] Spark 4.2: Replace unknown collection element test overrides with a hook [iceberg]

manuzhang opened a new pull request, #18223:
URL: https://github.com/apache/iceberg/pull/18223

   ### Rationale
   
   `AvroDataTestBase.testUnknownListType` and `testUnknownMapType` write a `list<unknown>` and a `map<string, unknown>` and expect a clean round trip, which only Avro can do.
   
   Unknown columns are omitted from data files. A struct drops the field — `TypeToMessageType.struct` skips children whose `field()` returns null — but a list or map would be left with no element or value type, so `TypeToMessageType.list`/`map` reject the schema, and the ORC schema conversion does the same.
   
   Every Parquet and ORC subclass works around this by overriding both tests to assert the rejection, so any new subclass with nested-type support has to rediscover the limitation and copy them again.
   
   ### Changes
   
   - `AvroDataTestBase`: add `supportsUnknownCollectionElements()`, defaulting to `true`, and route both tests through one helper that either round-trips or asserts the rejection.
   - `TestSparkOrcReader`, `TestSparkParquetReader`, `TestSparkRecordOrcReaderWriter`, `TestORCDataFrameWrite`, `TestParquetDataFrameWrite`, `TestParquetScan`: drop both overrides and return `false` from the hook.
   
   Spark 4.2 only. The same change applies to 3.5, 4.0 and 4.1, and can follow once the approach is agreed.
   
   ### Tests
   
   The seven affected classes run 393 tests with no failures. All 14 unknown-collection cases execute and none are skipped: `TestSparkAvroReader` covers the round-trip branch and the other six cover the rejection branch.
   
   ---
   **AI Disclosure**
   - Model: Claude Opus 5
   - Platform/Tool: Claude Code
   - Human Oversight: partially reviewed
   - Prompt Summary: extract the duplicated `testUnknownListType`/`testUnknownMapType` overrides in the Spark test hierarchy into a capability hook on `AvroDataTestBase`.
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 4.2: Replace unknown collection element test overrides with a hook [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18223:
URL: https://github.com/apache/iceberg/pull/18223#issuecomment-6391415729

   This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 4.2: Replace unknown collection element test overrides with a hook [iceberg]

Posted by "github-actions[bot] (via GitHub) via issues" <is...@iceberg.apache.org>.
github-actions[bot] commented on PR #18223:
URL: https://github.com/apache/iceberg/pull/18223#issuecomment-6566045854

   This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 4.2: Replace unknown collection element test overrides with a hook [iceberg]

Posted by "github-actions[bot] (via GitHub) via issues" <is...@iceberg.apache.org>.
github-actions[bot] closed pull request #18223: Spark 4.2: Replace unknown collection element test overrides with a hook
URL: https://github.com/apache/iceberg/pull/18223


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org