You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Hans-Raintree (via GitHub)" <gi...@apache.org> on 2025/09/03 07:31:40 UTC
[I] [SUPPORT] Incremental reads broken with Flink NBCC and SIMPLE BUCKET index for MOR - 1.0.2 [hudi]
Hans-Raintree opened a new issue, #13827:
URL: https://github.com/apache/hudi/issues/13827
**Describe the problem you faced**
Incremental reads are broken with Flink NBCC + Simple Bucket Index + MoR.
**To Reproduce**
Flink:
```
SET 'execution.runtime-mode' = 'batch';
SET 'sql-client.execution.result-mode' = 'tableau';
DROP TABLE IF EXISTS t_mor;
CREATE TABLE t_mor (
ts BIGINT,
uuid STRING PRIMARY KEY NOT ENFORCED,
rider STRING,
city STRING
) PARTITIONED BY (city)
WITH (
'connector' = 'hudi',
'path' = 'file:///data/hudi_mor_test',
'table.type' = 'MERGE_ON_READ',
'hoodie.write.concurrency.mode' = 'NON_BLOCKING_CONCURRENCY_CONTROL',
'hoodie.write.lock.provider' = 'org.apache.hudi.client.transaction.lock.InProcessLockProvider',
'record.key.field' = 'uuid',
'precombine.field' = 'ts',
'metadata.enabled' = 'false',
'index.key.field' = 'uuid',
'clean.retain_commits' = '2',
'compaction.delta_commits' = '3',
'hoodie.index.type' = 'BUCKET',
'hoodie.index.bucket.engine' = 'SIMPLE',
'hoodie.bucket.index.num.buckets' = '1',
'hoodie.clean.failed.writes.policy' = 'LAZY'
);
INSERT INTO t_mor VALUES
(1,'id-0001', 'r11', 'sf'),
(2,'id-0002', 'r21', 'nyc'),
(3,'id-0003', 'r31', 'la');
```
Spark:
```
PATH = "/data/hudi_mor_test"
spark = (
SparkSession.builder
.appName("hudi-incremental-repro")
.config("spark.serializer","org.apache.spark.serializer.KryoSerializer")
.config("spark.sql.extensions","org.apache.spark.sql.hudi.HoodieSparkSessionExtension")
.config("spark.sql.catalog.spark_catalog","org.apache.spark.sql.hudi.catalog.HoodieCatalog")
.getOrCreate()
)
snap = spark.read.format("hudi").load(PATH).cache()
snap_cnt = snap.count()
incr = (
spark.read.format("hudi")
.option("hoodie.datasource.query.type", "incremental")
.option("hoodie.datasource.read.begin.instanttime", "000")
.option("hoodie.datasource.read.incr.fallback.fulltablescan.enable", "false")
.load(PATH)
).cache()
incr_cnt = incr.count()
print(f"SNAPSHOT_COUNT={snap_cnt}")
print(f"INCREMENTAL_COUNT={incr_cnt}")
snap.orderBy("ts").show(100, truncate=False)
incr.orderBy("ts").show(100, truncate=False)
spark.stop()
SNAPSHOT_COUNT=3
INCREMENTAL_COUNT=1
+-------------------+----------------------+------------------+----------------------+--------------------------------------+---+-------+-----+----+
|_hoodie_commit_time|_hoodie_commit_seqno |_hoodie_record_key|_hoodie_partition_path|_hoodie_file_name |ts |uuid |rider|city|
+-------------------+----------------------+------------------+----------------------+--------------------------------------+---+-------+-----+----+
|20250903072719965 |20250903072719965_0_16|id-0001 |sf |00000000-0000-0000-0000-000000000000-0|1 |id-0001|r11 |sf |
|20250903072719965 |20250903072719965_0_17|id-0002 |nyc |00000000-0000-0000-0000-000000000000-0|2 |id-0002|r21 |nyc |
|20250903072719965 |20250903072719965_0_18|id-0003 |la |00000000-0000-0000-0000-000000000000-0|3 |id-0003|r31 |la |
+-------------------+----------------------+------------------+----------------------+--------------------------------------+---+-------+-----+----+
+-------------------+----------------------+------------------+----------------------+--------------------------------------+---+-------+-----+----+
|_hoodie_commit_time|_hoodie_commit_seqno |_hoodie_record_key|_hoodie_partition_path|_hoodie_file_name |ts |uuid |rider|city|
+-------------------+----------------------+------------------+----------------------+--------------------------------------+---+-------+-----+----+
|20250903072719965 |20250903072719965_0_16|id-0001 |sf |00000000-0000-0000-0000-000000000000-0|1 |id-0001|r11 |sf |
+-------------------+----------------------+------------------+----------------------+--------------------------------------+---+-------+-----+----+
It also gives these warnings:
25/09/03 07:27:29 WARN HoodieTableFileSystemView: Partition: la is not available in store
25/09/03 07:27:29 WARN HoodieTableFileSystemView: Partition: nyc is not available in store
```
**Expected behavior**
Incremental read shows all 3 inserts.
**Environment Description**
Hudi version : 1.0.2
Spark version : 3.5
Storage (HDFS/S3/GCS..) : HDFS
Running on Docker? (yes/no) : yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Incremental reads broken with Flink NBCC and SIMPLE BUCKET index for MOR - 1.0.2 [hudi]
Posted by "Hans-Raintree (via GitHub)" <gi...@apache.org>.
Hans-Raintree commented on issue #13827:
URL: https://github.com/apache/hudi/issues/13827#issuecomment-3248146410
It also doesn't work when metadata is enabled.
When I increase the bucket count to 2, it's missing the the third record:
```
SNAPSHOT_COUNT=3
INCREMENTAL_COUNT=2
+-------------------+----------------------+------------------+----------------------+--------------------------------------+---+-------+-----+----+
|_hoodie_commit_time|_hoodie_commit_seqno |_hoodie_record_key|_hoodie_partition_path|_hoodie_file_name |ts |uuid |rider|city|
+-------------------+----------------------+------------------+----------------------+--------------------------------------+---+-------+-----+----+
|20250903080111086 |20250903080111086_0_10|id-0001 |sf |00000000-0000-0000-0000-000000000000-0|1 |id-0001|r11 |sf |
|20250903080111086 |20250903080111086_0_11|id-0002 |nyc |00000001-0000-0000-0000-000000000000-0|2 |id-0002|r21 |nyc |
|20250903080111086 |20250903080111086_0_12|id-0003 |la |00000000-0000-0000-0000-000000000000-0|3 |id-0003|r31 |la |
+-------------------+----------------------+------------------+----------------------+--------------------------------------+---+-------+-----+----+
+-------------------+----------------------+------------------+----------------------+--------------------------------------+---+-------+-----+----+
|_hoodie_commit_time|_hoodie_commit_seqno |_hoodie_record_key|_hoodie_partition_path|_hoodie_file_name |ts |uuid |rider|city|
+-------------------+----------------------+------------------+----------------------+--------------------------------------+---+-------+-----+----+
|20250903080111086 |20250903080111086_0_10|id-0001 |sf |00000000-0000-0000-0000-000000000000-0|1 |id-0001|r11 |sf |
|20250903080111086 |20250903080111086_0_11|id-0002 |nyc |00000001-0000-0000-0000-000000000000-0|2 |id-0002|r21 |nyc |
+-------------------+----------------------+------------------+----------------------+--------------------------------------+---+-------+-----+----+
nyc\.00000001-0000-0000-0000-000000000000-0_20250903080346023.log.1_0-1-0
la\.00000000-0000-0000-0000-000000000000-0_20250903080346023.log.1_0-1-0
sf\.00000000-0000-0000-0000-000000000000-0_20250903080346023.log.1_0-1-0
nyc gets '00000001' fileid, so that also makes it seem like the issue is related to the identical fileids in different partitions.
```
When I remove NON_BLOCKING_CONCURRENCY_CONTROL everything works fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Incremental reads broken with Flink NBCC and SIMPLE BUCKET index for MOR - 1.0.2 [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on issue #13827:
URL: https://github.com/apache/hudi/issues/13827#issuecomment-3248190576
cc @danny0405
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Incremental reads broken with Flink NBCC and SIMPLE BUCKET index for MOR - 1.0.2 [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on issue #13827:
URL: https://github.com/apache/hudi/issues/13827#issuecomment-3253273158
> Hey [@cshuo](https://github.com/cshuo), thank you!!!
>
> One question, does this issue affect the writer at all? I have Flink writing into Hudi sinks, compaction/clean/read happens via Spark, do I need to update the writer as well or updating the package Spark uses is enough? (ie. the writer does some type of file listings or whatever that are affected by this as well?)
@Hans-Raintree the bug only affects incremental query (flink/spark/hive), and nothing needed to update in the write side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Incremental reads broken with Flink NBCC and SIMPLE BUCKET index for MOR - 1.0.2 [hudi]
Posted by "Hans-Raintree (via GitHub)" <gi...@apache.org>.
Hans-Raintree commented on issue #13827:
URL: https://github.com/apache/hudi/issues/13827#issuecomment-3248511721
This is one way I tested in spark:
```
PATH = "/data/hudi_mor_test"
spark = (
SparkSession.builder
.appName("hudi-incremental-repro")
.config("spark.serializer","org.apache.spark.serializer.KryoSerializer")
.config("spark.sql.extensions","org.apache.spark.sql.hudi.HoodieSparkSessionExtension")
.config("spark.sql.catalog.spark_catalog","org.apache.spark.sql.hudi.catalog.HoodieCatalog")
.getOrCreate()
)
def upsert(rows):
opts = {
"hoodie.table.name":"t_bug",
"hoodie.write.concurrency.mode": "NON_BLOCKING_CONCURRENCY_CONTROL",
"hoodie.table.type":"MERGE_ON_READ",
"hoodie.datasource.write.table.type":"MERGE_ON_READ",
"hoodie.datasource.write.recordkey.field":"id",
"hoodie.datasource.write.precombine.field":"ts",
"hoodie.datasource.write.partitionpath.field":"city",
"hoodie.index.type":"BUCKET",
"hoodie.write.lock.provider": "org.apache.hudi.client.transaction.lock.FileSystemBasedLockProvider",
"hoodie.index.bucket.engine":"SIMPLE",
"hoodie.bucket.index.num.buckets":"1",
"hoodie.metadata.enable": "false",
"hoodie.clean.failed.writes.policy": "LAZY"
}
df = spark.createDataFrame([Row(**r) for r in rows])
(df.write
.format("hudi")
.option('hoodie.datasource.write.operation', 'upsert')
.options(**opts)
.mode("append")
.save(PATH)
)
for i in range(1, 3):
upsert(
[
{"id": "id-1", "ts": i, "rider": f"r1{i}", "city": "la"},
{"id": "id-1", "ts": i, "rider": f"r2{i}", "city": "ny"},
{"id": "id-1", "ts": i, "rider": f"r3{i}", "city": "sf"},
]
)
snap = spark.read.format("hudi").load(PATH)
incr = (spark.read.format("hudi")
.option("hoodie.datasource.query.type","incremental")
.option("hoodie.datasource.read.begin.instanttime","000")
.option("hoodie.datasource.read.incr.fallback.fulltablescan.enable", "false")
.load(PATH))
snap.show(truncate=False)
incr.show(truncate=False)
spark.stop()
+-------------------+---------------------+------------------+----------------------+--------------------------------------+----+---+-----+----+
|_hoodie_commit_time|_hoodie_commit_seqno |_hoodie_record_key|_hoodie_partition_path|_hoodie_file_name |id |ts |rider|city|
+-------------------+---------------------+------------------+----------------------+--------------------------------------+----+---+-----+----+
|20250903093733351 |20250903093733351_0_4|id-1 |sf |00000000-0000-0000-0000-000000000000-0|id-1|2 |r32 |sf |
|20250903093733351 |20250903093733351_2_6|id-1 |ny |00000000-0000-0000-0000-000000000000-0|id-1|2 |r22 |ny |
|20250903093733351 |20250903093733351_1_5|id-1 |la |00000000-0000-0000-0000-000000000000-0|id-1|2 |r12 |la |
+-------------------+---------------------+------------------+----------------------+--------------------------------------+----+---+-----+----+
+-------------------+---------------------+------------------+----------------------+--------------------------------------+----+---+-----+----+
|_hoodie_commit_time|_hoodie_commit_seqno |_hoodie_record_key|_hoodie_partition_path|_hoodie_file_name |id |ts |rider|city|
+-------------------+---------------------+------------------+----------------------+--------------------------------------+----+---+-----+----+
|20250903093733351 |20250903093733351_1_5|id-1 |la |00000000-0000-0000-0000-000000000000-0|id-1|2 |r12 |la |
|20250903093733351 |20250903093733351_2_6|id-1 |ny |00000000-0000-0000-0000-000000000000-0|id-1|2 |r22 |ny |
|20250903093733351 |20250903093733351_0_4|id-1 |sf |00000000-0000-0000-0000-000000000000-0|id-1|2 |r32 |sf |
+-------------------+---------------------+------------------+----------------------+--------------------------------------+----+---+-----+----+
la\.00000000-0000-0000-0000-000000000000-0_20250903093726835.log.1_1-5-97
la\.00000000-0000-0000-0000-000000000000-0_20250903093733351.log.1_1-15-201
ny\.00000000-0000-0000-0000-000000000000-0_20250903093726835.log.1_2-5-98
ny\.00000000-0000-0000-0000-000000000000-0_20250903093733351.log.1_2-15-202
sf\.00000000-0000-0000-0000-000000000000-0_20250903093726835.log.1_0-5-96
sf\.00000000-0000-0000-0000-000000000000-0_20250903093733351.log.1_0-15-200
Wasn't able to reproduce in Spark, note how the file extensions are unique though, so might be related to that?
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Incremental reads broken with Flink NBCC and SIMPLE BUCKET index for MOR - 1.0.2 [hudi]
Posted by "Hans-Raintree (via GitHub)" <gi...@apache.org>.
Hans-Raintree commented on issue #13827:
URL: https://github.com/apache/hudi/issues/13827#issuecomment-3248052710
Also, I'm not sure if this is relevant, but the file names are identical in the partitions:
la\.00000000-0000-0000-0000-000000000000-0_20250903072719965.log.1_0-1-0
nyc\.00000000-0000-0000-0000-000000000000-0_20250903072719965.log.1_0-1-0
sf\.00000000-0000-0000-0000-000000000000-0_20250903072719965.log.1_0-1-0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Incremental reads broken with Flink NBCC and SIMPLE BUCKET index for MOR - 1.0.2 [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on issue #13827:
URL: https://github.com/apache/hudi/issues/13827#issuecomment-3252122461
hi @Hans-Raintree the problem is indeed related to the suffix, and there is a bug in file listing of incremental query, here is the fix https://github.com/apache/hudi/pull/13838.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Incremental reads broken with Flink NBCC and SIMPLE BUCKET index for MOR - 1.0.2 [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 closed issue #13827: [SUPPORT] Incremental reads broken with Flink NBCC and SIMPLE BUCKET index for MOR - 1.0.2
URL: https://github.com/apache/hudi/issues/13827
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Incremental reads broken with Flink NBCC and SIMPLE BUCKET index for MOR - 1.0.2 [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on issue #13827:
URL: https://github.com/apache/hudi/issues/13827#issuecomment-3249196931
> la\.00000000-0000-0000-0000-000000000000-0_20250903072719965.log.1_0-1-0
nyc\.00000000-0000-0000-0000-000000000000-0_20250903072719965.log.1_0-1-0
sf\.00000000-0000-0000-0000-000000000000-0_20250903072719965.log.1_0-1-0
@Hans-Raintree The suffix `0-1-0` denotes `subtask_idx-parallelism-attemp_id` of the flink job, it seems you are setting the parallelism of the flink job as 1. Can you make a quick test by setting the parallelism more than 1, for example 4, to help determine whether the problem is related to the suffix?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Incremental reads broken with Flink NBCC and SIMPLE BUCKET index for MOR - 1.0.2 [hudi]
Posted by "Hans-Raintree (via GitHub)" <gi...@apache.org>.
Hans-Raintree commented on issue #13827:
URL: https://github.com/apache/hudi/issues/13827#issuecomment-3249465200
Hey @cshuo,
I tried with parallelism 3:
```
SNAPSHOT_COUNT=3
INCREMENTAL_COUNT=2
+-------------------+---------------------+------------------+----------------------+--------------------------------------+---+-------+-----+----+
|_hoodie_commit_time|_hoodie_commit_seqno |_hoodie_record_key|_hoodie_partition_path|_hoodie_file_name |ts |uuid |rider|city|
+-------------------+---------------------+------------------+----------------------+--------------------------------------+---+-------+-----+----+
|20250903135712613 |20250903135712613_1_8|id-0001 |sf |00000000-0000-0000-0000-000000000000-0|1 |id-0001|r11 |sf |
|20250903135712613 |20250903135712613_0_7|id-0002 |nyc |00000000-0000-0000-0000-000000000000-0|2 |id-0002|r21 |nyc |
|20250903135712613 |20250903135712613_1_9|id-0003 |la |00000000-0000-0000-0000-000000000000-0|3 |id-0003|r31 |la |
+-------------------+---------------------+------------------+----------------------+--------------------------------------+---+-------+-----+----+
+-------------------+---------------------+------------------+----------------------+--------------------------------------+---+-------+-----+----+
|_hoodie_commit_time|_hoodie_commit_seqno |_hoodie_record_key|_hoodie_partition_path|_hoodie_file_name |ts |uuid |rider|city|
+-------------------+---------------------+------------------+----------------------+--------------------------------------+---+-------+-----+----+
|20250903135712613 |20250903135712613_1_8|id-0001 |sf |00000000-0000-0000-0000-000000000000-0|1 |id-0001|r11 |sf |
|20250903135712613 |20250903135712613_0_7|id-0002 |nyc |00000000-0000-0000-0000-000000000000-0|2 |id-0002|r21 |nyc |
+-------------------+---------------------+------------------+----------------------+--------------------------------------+---+-------+-----+----+
sf\.00000000-0000-0000-0000-000000000000-0_20250903135712613.log.1_1-3-0
nyc\.00000000-0000-0000-0000-000000000000-0_20250903135712613.log.1_0-3-0
la\.00000000-0000-0000-0000-000000000000-0_20250903135712613.log.1_1-3-0
It looks like although the parallelism and write tasks etc were set to 3, 2 records went to the single subtask and one subtask got zero records.
I tried with 5 values as well:
INSERT INTO t_mor VALUES
(1,'id-0001', 'r11', 'sf'),
(2,'id-0002', 'r21', 'nyc'),
(3,'id-0003', 'r31', 'la'),
(4,'id-0004', 'r41', 'hu'),
(5,'id-0005', 'r51', 'wd');
SNAPSHOT_COUNT=5
INCREMENTAL_COUNT=3
+-------------------+----------------------+------------------+----------------------+--------------------------------------+---+-------+-----+----+
|_hoodie_commit_time|_hoodie_commit_seqno |_hoodie_record_key|_hoodie_partition_path|_hoodie_file_name |ts |uuid |rider|city|
+-------------------+----------------------+------------------+----------------------+--------------------------------------+---+-------+-----+----+
|20250903141316770 |20250903141316770_1_11|id-0001 |sf |00000000-0000-0000-0000-000000000000-0|1 |id-0001|r11 |sf |
|20250903141316770 |20250903141316770_0_12|id-0002 |nyc |00000000-0000-0000-0000-000000000000-0|2 |id-0002|r21 |nyc |
|20250903141316770 |20250903141316770_1_14|id-0003 |la |00000000-0000-0000-0000-000000000000-0|3 |id-0003|r31 |la |
|20250903141316770 |20250903141316770_2_10|id-0004 |hu |00000000-0000-0000-0000-000000000000-0|4 |id-0004|r41 |hu |
|20250903141316770 |20250903141316770_0_13|id-0005 |wd |00000000-0000-0000-0000-000000000000-0|5 |id-0005|r51 |wd |
+-------------------+----------------------+------------------+----------------------+--------------------------------------+---+-------+-----+----+
+-------------------+----------------------+------------------+----------------------+--------------------------------------+---+-------+-----+----+
|_hoodie_commit_time|_hoodie_commit_seqno |_hoodie_record_key|_hoodie_partition_path|_hoodie_file_name |ts |uuid |rider|city|
+-------------------+----------------------+------------------+----------------------+--------------------------------------+---+-------+-----+----+
|20250903141316770 |20250903141316770_1_14|id-0003 |la |00000000-0000-0000-0000-000000000000-0|3 |id-0003|r31 |la |
|20250903141316770 |20250903141316770_2_10|id-0004 |hu |00000000-0000-0000-0000-000000000000-0|4 |id-0004|r41 |hu |
|20250903141316770 |20250903141316770_0_13|id-0005 |wd |00000000-0000-0000-0000-000000000000-0|5 |id-0005|r51 |wd |
+-------------------+----------------------+------------------+----------------------+--------------------------------------+---+-------+-----+----+
hu\.00000000-0000-0000-0000-000000000000-0_20250903141316770.log.1_2-3-0
la\.00000000-0000-0000-0000-000000000000-0_20250903141316770.log.1_1-3-0
nyc\.00000000-0000-0000-0000-000000000000-0_20250903141316770.log.1_0-3-0
sf\.00000000-0000-0000-0000-000000000000-0_20250903141316770.log.1_1-3-0
wd\.00000000-0000-0000-0000-000000000000-0_20250903141316770.log.1_0-3-0
```
It does seem like it's related to the suffix, as only the unique records show up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Incremental reads broken with Flink NBCC and SIMPLE BUCKET index for MOR - 1.0.2 [hudi]
Posted by "Hans-Raintree (via GitHub)" <gi...@apache.org>.
Hans-Raintree commented on issue #13827:
URL: https://github.com/apache/hudi/issues/13827#issuecomment-3253002332
Hey @cshuo, thank you!!!
One question, does this issue affect the writer at all? I have Flink writing into Hudi sinks, compaction/clean/read happens via Spark, do I need to update the writer as well or updating the package Spark uses is enough? (ie. the writer does some type of file listings or whatever that are affected by this as well?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Incremental reads broken with Flink NBCC and SIMPLE BUCKET index for MOR - 1.0.2 [hudi]
Posted by "Hans-Raintree (via GitHub)" <gi...@apache.org>.
Hans-Raintree commented on issue #13827:
URL: https://github.com/apache/hudi/issues/13827#issuecomment-3248210148
I was not able to reproduce this via Spark, since Spark writes inserts into the base file I tried seeing if the 'ts' is not the latest, but it was correctly pulling the latest records in my Spark testing, so one aspect of this issue seems to be inserts going to log files with Flink.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org