You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "fightBoxing (via GitHub)" <gi...@apache.org> on 2026/01/30 06:22:02 UTC
[PR] [FLINK] Implement Iceberg lookup join functionality, and source code and junit test code. [iceberg]
fightBoxing opened a new pull request, #15183:
URL: https://github.com/apache/iceberg/pull/15183
Problem
In production environments, there is a common need to join streaming data with dimension data stored in Iceberg tables. The dimension data needs to be periodically refreshed to ensure join accuracy. Currently, Flink lacks native support for Iceberg lookup joins, forcing users to work around this limitation or use alternative solutions.
Solution
This PR implements Iceberg lookup join functionality for Flink, enabling efficient joins between streaming data and Iceberg dimension tables. The implementation includes:
IcebergLookupCache: A cache mechanism for storing and managing lookup data with TTL support
IcebergLookupReader: A reader component for loading and refreshing lookup data from Iceberg tables
IcebergTableSource enhancement: Updated to support lookup join operations
Configuration options: New config options for customizing lookup join behavior (cache size, refresh interval, etc.)
Integration tests: Comprehensive test coverage (IcebergLookupJoinITCase)
Changes
Added IcebergLookupCache for efficient caching of lookup data
Added IcebergLookupReader for reading lookup data from Iceberg tables
Added IcebergLookupJoinITCase for integration testing
Updated IcebergTableSource to support lookup join operations
Added configuration options in FlinkConfigOptions for lookup join settings
Updated build.gradle files for v1.16, v1.17, and v1.18
Benefits
Enables real-time joins with Iceberg dimension tables
Reduces data latency by avoiding frequent full table scans
Improves performance through intelligent caching strategies
Seamlessly integrates with existing Flink lookup join framework
Supports periodic data refresh to ensure data freshness
Testing
Added integration tests to validate lookup join functionality
Tested cache refresh mechanisms
Verified correctness of join results
Ensures backward compatibility
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [FLINK] Implement Iceberg lookup join functionality, and source code and junit test code. [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #15183: [FLINK] Implement Iceberg lookup join functionality, and source code and junit test code.
URL: https://github.com/apache/iceberg/pull/15183
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [FLINK] Implement Iceberg lookup join functionality, and source code and junit test code. [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #15183:
URL: https://github.com/apache/iceberg/pull/15183#issuecomment-4020366454
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [FLINK] Implement Iceberg lookup join functionality, and source code and junit test code. [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #15183:
URL: https://github.com/apache/iceberg/pull/15183#issuecomment-3981394878
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org