[GitHub] [hudi] danny0405 commented on a diff in pull request #8668: [HUDI-3639] Add Proper Incremental Records FIltering support into Hudi's custom RDD

["danny0405 (via GitHub)" <gi...@apache.org>]

danny0405 commented on code in PR #8668:
URL: https://github.com/apache/hudi/pull/8668#discussion_r1189436153


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/MergeOnReadIncrementalRelation.scala:
##########
@@ -181,25 +181,25 @@ trait HoodieIncrementalRelationTrait extends HoodieBaseRelation {
     listAffectedFilesForCommits(conf, new Path(metaClient.getBasePath), commitsMetadata)
   }
 
-  protected lazy val (shouldEqualStartTimestamp, filterStartTimestamp) = if (startInstantArchived) {
+  protected lazy val (includeStartTime, startTs) = if (startInstantArchived) {
     (false, startTimestamp)
   } else {
     (true, includedCommits.head.getTimestamp)
   }
-  protected lazy val filterEndTimestamp = if (endInstantArchived) endTimestamp else includedCommits.last.getTimestamp
+  protected lazy val endTs: String = if (endInstantArchived) endTimestamp else includedCommits.last.getTimestamp
 
   // Record filters making sure that only records w/in the requested bounds are being fetched as part of the
   // scan collected by this relation
   protected lazy val incrementalSpanRecordFilters: Seq[Filter] = {
     val isNotNullFilter = IsNotNull(HoodieRecord.COMMIT_TIME_METADATA_FIELD)
 
-    val largerThanFilter = if (shouldEqualStartTimestamp) {
-      GreaterThan(HoodieRecord.COMMIT_TIME_METADATA_FIELD, filterStartTimestamp)
+    val largerThanFilter = if (includeStartTime) {

Review Comment:
   Do we still need to keep the filters?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org