You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2022/04/27 02:13:24 UTC
[GitHub] [flink] luoyuxia commented on a diff in pull request #19573: [FLINK-27384] solve the problem that the latest data cannot be read under the creat…
luoyuxia commented on code in PR #19573:
URL: https://github.com/apache/flink/pull/19573#discussion_r859313505
##########
flink-connectors/flink-connector-hive/src/main/java/org/apache/flink/connectors/hive/read/HivePartitionFetcherContextBase.java:
##########
@@ -139,16 +138,9 @@ public List<ComparablePartitionValue> getComparablePartitionValueList() throws E
tablePath.getDatabaseName(),
tablePath.getObjectName(),
Short.MAX_VALUE);
- List<String> newNames =
- partitionNames.stream()
- .filter(
- n ->
- !partValuesToCreateTime.containsKey(
- extractPartitionValues(n)))
- .collect(Collectors.toList());
List<Partition> newPartitions =
metaStoreClient.getPartitionsByNames(
- tablePath.getDatabaseName(), tablePath.getObjectName(), newNames);
+ tablePath.getDatabaseName(), tablePath.getObjectName(), partitionNames);
Review Comment:
Then it will always contain the all paritions, finally return all partitions with create time, right?
I prefer to only return the partitions that dose changes. Maybe you can compare the modifcation time for all partitions to decide which partition is updated and should be returned.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org