You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2022/12/09 04:09:20 UTC
[GitHub] [flink-table-store] JingsongLi opened a new pull request, #430: [FLINK-30333] Supports lookup a partial-update table
JingsongLi opened a new pull request, #430:
URL: https://github.com/apache/flink-table-store/pull/430
The lookup uses streaming read for reading table. (In TableStreamingReader)
- We should support lookup a partial-update table with full compaction.
- But partial-update table without full compaction, we should throw exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-table-store] tsreaper merged pull request #430: [FLINK-30333] Supports lookup a partial-update table
Posted by GitBox <gi...@apache.org>.
tsreaper merged PR #430:
URL: https://github.com/apache/flink-table-store/pull/430
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-table-store] tsreaper commented on a diff in pull request #430: [FLINK-30333] Supports lookup a partial-update table
Posted by GitBox <gi...@apache.org>.
tsreaper commented on code in PR #430:
URL: https://github.com/apache/flink-table-store/pull/430#discussion_r1044204129
##########
flink-table-store-core/src/main/java/org/apache/flink/table/store/table/source/snapshot/SnapshotEnumerator.java:
##########
@@ -36,4 +44,24 @@ public interface SnapshotEnumerator {
*/
@Nullable
DataTableScan.DataFilePlan enumerate();
+
+ static void validateContinuous(TableSchema schema) {
+ CoreOptions options = new CoreOptions(schema.options());
+ MergeEngine mergeEngine = options.mergeEngine();
+ HashMap<MergeEngine, String> mergeEngineDesc =
+ new HashMap<MergeEngine, String>() {
+ {
+ put(MergeEngine.PARTIAL_UPDATE, "Partial update");
+ put(MergeEngine.AGGREGATE, "Pre-aggregate");
+ }
+ };
+ if (schema.primaryKeys().size() > 0
+ && mergeEngineDesc.containsKey(mergeEngine)
+ && options.changelogProducer() != FULL_COMPACTION) {
+ throw new ValidationException(
+ mergeEngineDesc.get(mergeEngine)
+ + " continuous reading is not supported. "
+ + "You can use full compaction changelog producer to support streaming reading.");
+ }
+ }
Review Comment:
Why not move this to `ContinuousDataFileSnapshotEnumerator` and rename it as `validate`? This validation is only about streaming reads of data files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-table-store] tsreaper commented on a diff in pull request #430: [FLINK-30333] Supports lookup a partial-update table
Posted by GitBox <gi...@apache.org>.
tsreaper commented on code in PR #430:
URL: https://github.com/apache/flink-table-store/pull/430#discussion_r1044089189
##########
flink-table-store-core/src/main/java/org/apache/flink/table/store/table/source/TableStreamingReader.java:
##########
@@ -88,13 +86,7 @@ public TableStreamingReader(
if (predicate != null) {
scan.withFilter(predicate);
}
- enumerator =
- new ContinuousDataFileSnapshotEnumerator(
- table.location(),
- scan,
- new FullStartingScanner(),
- new DeltaFollowUpScanner(),
- null);
+ enumerator = ContinuousDataFileSnapshotEnumerator.create(table, scan, null);
Review Comment:
What if user set 'scan.mode' = 'from-timestamp' or 'latest' for this table? The base records wont' be read.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org