You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "sivabalan narayanan (Jira)" <ji...@apache.org> on 2022/01/02 22:39:00 UTC
[jira] [Commented] (HUDI-3137) Hudi incremental read fails when there is no commit between start and end time
[ https://issues.apache.org/jira/browse/HUDI-3137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17467724#comment-17467724 ]
sivabalan narayanan commented on HUDI-3137:
-------------------------------------------
with COW, I don't see any issues.
{code:java}
val tripsIncrementalDF = spark.read.format("hudi").
| | option(QUERY_TYPE_OPT_KEY, QUERY_TYPE_INCREMENTAL_OPT_VAL).
| | option(BEGIN_INSTANTTIME_OPT_KEY, "20211230071507597").
| | option(END_INSTANTTIME_OPT_KEY, "20211230071507600").
| load(basePath)
22/01/02 17:38:00 WARN HoodieBackedTableMetadata: Metadata table was not found at path file:/tmp/hudi_trips_cow/.hoodie/metadata
tripsIncrementalDF: org.apache.spark.sql.DataFrame = [_hoodie_commit_time: string, _hoodie_commit_seqno: string ... 13 more fields]
scala> tripsIncrementalDF.createOrReplaceTempView("hudi_tbl1")
scala> spark.sql("select `_hoodie_commit_time`, fare, begin_lon, begin_lat, ts from hudi_tbl1 ").show()
+-------------------+----+---------+---------+---+
|_hoodie_commit_time|fare|begin_lon|begin_lat| ts|
+-------------------+----+---------+---------+---+
+-------------------+----+---------+---------+---+
{code}
> Hudi incremental read fails when there is no commit between start and end time
> ------------------------------------------------------------------------------
>
> Key: HUDI-3137
> URL: https://issues.apache.org/jira/browse/HUDI-3137
> Project: Apache Hudi
> Issue Type: Task
> Components: Incremental Pull
> Reporter: sivabalan narayanan
> Priority: Major
>
> when there is no valid commits found between begin time and end time, hudi may throw an exception rather than silently returning empty rows.
> Ref issue:
> [https://github.com/apache/hudi/issues/4340]
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)