You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "mansipp (via GitHub)" <gi...@apache.org> on 2025/08/05 19:14:01 UTC
[I] [SUPPORT] Hudi 1.0.2 SQL Global_Bloom index not returning newly inserted data [hudi]
mansipp opened a new issue, #13680:
URL: https://github.com/apache/hudi/issues/13680
**_Tips before filing an issue_**
- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
**Describe the problem you faced**
Hudi 1.0.2 SQL Global_Bloom index not returning newly inserted data. However, dataframe works as expected.
**To Reproduce**
Steps to reproduce the behavior:
```
spark-shell --jars /home/hadoop/hudi-spark3-bundle_2.12-1.0.2-amzn-0-SNAPSHOT.jar \
--conf "spark.serializer=org.apache.spark.serializer.KryoSerializer" \
--conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog" \
--conf "spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension"
import org.apache.hudi.DataSourceWriteOptions
import org.apache.spark.sql.SaveMode
val df1 = Seq(
("100", "2015-01-01", "event_name_900", "2015-01-01T13:51:39.340396Z", "type1"),
("101", "2015-01-01", "event_name_546", "2015-01-01T12:14:58.597216Z", "type2"),
("102", "2015-01-01", "event_name_345", "2015-01-01T13:51:40.417052Z", "type3"),
("103", "2015-01-01", "event_name_234", "2015-01-01T13:51:40.519832Z", "type4"),
("104", "2015-01-01", "event_name_123", "2015-01-01T12:15:00.512679Z", "type1"),
("105", "2015-01-01", "event_name_678", "2015-01-01T13:51:42.248818Z", "type2"),
("106", "2015-01-01", "event_name_890", "2015-01-01T13:51:44.735360Z", "type3"),
("107", "2015-01-01", "event_name_944", "2015-01-01T13:51:45.019544Z", "type4"),
("108", "2015-01-01", "event_name_456", "2015-01-01T13:51:45.208007Z", "type1"),
("109", "2015-01-01", "event_name_567", "2015-01-01T13:51:45.369689Z", "type2"),
("110", "2015-01-01", "event_name_789", "2015-01-01T12:15:05.664947Z", "type3"),
("111", "2015-01-01", "event_name_322", "2015-01-01T13:51:47.388239Z", "type4")
).toDF("event_id", "event_date", "event_name", "event_ts", "event_type")
var tableName = "mansipp_hudi_102_cow_fta_write_lf_table_update_test"
var tablePath = "s3://<path>/" + tableName + "/"
df1.write.format("hudi")
.option("hoodie.metadata.enable", "true")
.option("hoodie.table.name", tableName)
.option("hoodie.database.name", "default")
.option("hoodie.datasource.write.operation", "upsert")
.option("hoodie.datasource.write.table.type", "COPY_ON_WRITE")
.option("hoodie.datasource.write.recordkey.field", "event_id,event_date")
.option("hoodie.datasource.write.partitionpath.field", "event_type")
.option("hoodie.datasource.write.precombine.field", "event_ts")
.option("hoodie.datasource.write.keygenerator.class", "org.apache.hudi.keygen.ComplexKeyGenerator")
.option("hoodie.datasource.hive_sync.enable", "true")
.option("hoodie.datasource.meta.sync.enable", "true")
.option("hoodie.index.type", "GLOBAL_BLOOM")
.option("hoodie.datasource.hive_sync.mode", "hms")
.option("hoodie.datasource.hive_sync.database", "default")
.option("hoodie.datasource.hive_sync.table", tableName)
.option("hoodie.datasource.hive_sync.partition_fields", "event_type")
.option("hoodie.datasource.hive_sync.partition_extractor_class", "org.apache.hudi.hive.MultiPartKeysValueExtractor")
.mode(SaveMode.Overwrite)
.save(tablePath)
# SELECT
spark.sql("select * from mansipp_hudi_102_cow_fta_write_lf_table_update_test order by event_id").show();
+-------------------+--------------------+--------------------+----------------------+--------------------+--------+----------+--------------+--------------------+----------+
|_hoodie_commit_time|_hoodie_commit_seqno| _hoodie_record_key|_hoodie_partition_path| _hoodie_file_name|event_id|event_date| event_name| event_ts|event_type|
+-------------------+--------------------+--------------------+----------------------+--------------------+--------+----------+--------------+--------------------+----------+
| 20250805000927187|20250805000927187...|event_id:100,even...| type1|db420273-0543-4c0...| 100|2015-01-01|event_name_900|2015-01-01T13:51:...| type1|
| 20250805000927187|20250805000927187...|event_id:101,even...| type2|d888bc7e-09c8-44a...| 101|2015-01-01|event_name_546|2015-01-01T12:14:...| type2|
| 20250805000927187|20250805000927187...|event_id:102,even...| type3|198de9d0-7df0-4c5...| 102|2015-01-01|event_name_345|2015-01-01T13:51:...| type3|
| 20250805000927187|20250805000927187...|event_id:103,even...| type4|72c26c48-802d-473...| 103|2015-01-01|event_name_234|2015-01-01T13:51:...| type4|
| 20250805000927187|20250805000927187...|event_id:104,even...| type1|db420273-0543-4c0...| 104|2015-01-01|event_name_123|2015-01-01T12:15:...| type1|
| 20250805000927187|20250805000927187...|event_id:105,even...| type2|d888bc7e-09c8-44a...| 105|2015-01-01|event_name_678|2015-01-01T13:51:...| type2|
| 20250805000927187|20250805000927187...|event_id:106,even...| type3|198de9d0-7df0-4c5...| 106|2015-01-01|event_name_890|2015-01-01T13:51:...| type3|
| 20250805000927187|20250805000927187...|event_id:107,even...| type4|72c26c48-802d-473...| 107|2015-01-01|event_name_944|2015-01-01T13:51:...| type4|
| 20250805000927187|20250805000927187...|event_id:108,even...| type1|db420273-0543-4c0...| 108|2015-01-01|event_name_456|2015-01-01T13:51:...| type1|
| 20250805000927187|20250805000927187...|event_id:109,even...| type2|d888bc7e-09c8-44a...| 109|2015-01-01|event_name_567|2015-01-01T13:51:...| type2|
| 20250805000927187|20250805000927187...|event_id:110,even...| type3|198de9d0-7df0-4c5...| 110|2015-01-01|event_name_789|2015-01-01T12:15:...| type3|
| 20250805000927187|20250805000927187...|event_id:111,even...| type4|72c26c48-802d-473...| 111|2015-01-01|event_name_322|2015-01-01T13:51:...| type4|
+-------------------+--------------------+--------------------+----------------------+--------------------+--------+----------+--------------+--------------------+----------+
# INSERT new data
spark.sql("INSERT INTO mansipp_hudi_102_cow_fta_write_lf_table_update_test (event_id, event_date, event_name, event_ts, event_type) VALUES('112', DATE('2015-01-01'), 'event_name_123', TIMESTAMP('2015-01-01 13:51:45'), 'type5')")
# SELECT - Not showing new record
spark.sql("select * from mansipp_hudi_102_cow_fta_write_lf_table_update_test order by event_id").show();
+-------------------+--------------------+--------------------+----------------------+--------------------+--------+----------+--------------+--------------------+----------+
|_hoodie_commit_time|_hoodie_commit_seqno| _hoodie_record_key|_hoodie_partition_path| _hoodie_file_name|event_id|event_date| event_name| event_ts|event_type|
+-------------------+--------------------+--------------------+----------------------+--------------------+--------+----------+--------------+--------------------+----------+
| 20250805000927187|20250805000927187...|event_id:100,even...| type1|db420273-0543-4c0...| 100|2015-01-01|event_name_900|2015-01-01T13:51:...| type1|
| 20250805000927187|20250805000927187...|event_id:101,even...| type2|d888bc7e-09c8-44a...| 101|2015-01-01|event_name_546|2015-01-01T12:14:...| type2|
| 20250805000927187|20250805000927187...|event_id:102,even...| type3|198de9d0-7df0-4c5...| 102|2015-01-01|event_name_345|2015-01-01T13:51:...| type3|
| 20250805000927187|20250805000927187...|event_id:103,even...| type4|72c26c48-802d-473...| 103|2015-01-01|event_name_234|2015-01-01T13:51:...| type4|
| 20250805000927187|20250805000927187...|event_id:104,even...| type1|db420273-0543-4c0...| 104|2015-01-01|event_name_123|2015-01-01T12:15:...| type1|
| 20250805000927187|20250805000927187...|event_id:105,even...| type2|d888bc7e-09c8-44a...| 105|2015-01-01|event_name_678|2015-01-01T13:51:...| type2|
| 20250805000927187|20250805000927187...|event_id:106,even...| type3|198de9d0-7df0-4c5...| 106|2015-01-01|event_name_890|2015-01-01T13:51:...| type3|
| 20250805000927187|20250805000927187...|event_id:107,even...| type4|72c26c48-802d-473...| 107|2015-01-01|event_name_944|2015-01-01T13:51:...| type4|
| 20250805000927187|20250805000927187...|event_id:108,even...| type1|db420273-0543-4c0...| 108|2015-01-01|event_name_456|2015-01-01T13:51:...| type1|
| 20250805000927187|20250805000927187...|event_id:109,even...| type2|d888bc7e-09c8-44a...| 109|2015-01-01|event_name_567|2015-01-01T13:51:...| type2|
| 20250805000927187|20250805000927187...|event_id:110,even...| type3|198de9d0-7df0-4c5...| 110|2015-01-01|event_name_789|2015-01-01T12:15:...| type3|
| 20250805000927187|20250805000927187...|event_id:111,even...| type4|72c26c48-802d-473...| 111|2015-01-01|event_name_322|2015-01-01T13:51:...| type4|
+-------------------+--------------------+--------------------+----------------------+--------------------+--------+----------+--------------+--------------------+----------+
# df works
import org.apache.hudi.DataSourceWriteOptions
import org.apache.spark.sql.SaveMode
var tableName = "mansipp_hudi_102_cow_fta_write_lf_table_update_test"
var tablePath = "s3://<path>/" + tableName + "/"
val newData = Seq(
("113", "2015-01-01", "event_name_999", "2015-01-01T14:00:00.000000Z", "type5")
).toDF("event_id", "event_date", "event_name", "event_ts", "event_type")
newData.write.format("hudi")
.option("hoodie.metadata.enable", "true")
.option("hoodie.table.name", tableName)
.option("hoodie.database.name", "default")
.option("hoodie.datasource.write.operation", "upsert")
.option("hoodie.datasource.write.table.type", "COPY_ON_WRITE")
.option("hoodie.datasource.write.recordkey.field", "event_id,event_date")
.option("hoodie.datasource.write.partitionpath.field", "event_type")
.option("hoodie.datasource.write.precombine.field", "event_ts")
.option("hoodie.datasource.write.keygenerator.class", "org.apache.hudi.keygen.ComplexKeyGenerator")
.option("hoodie.index.type", "GLOBAL_BLOOM")
.option("hoodie.datasource.hive_sync.enable", "true")
.option("hoodie.datasource.meta.sync.enable", "true")
.option("hoodie.datasource.hive_sync.mode", "hms")
.option("hoodie.datasource.hive_sync.database", "default")
.option("hoodie.datasource.hive_sync.table", tableName)
.option("hoodie.datasource.hive_sync.partition_fields", "event_type")
.option("hoodie.datasource.hive_sync.partition_extractor_class", "org.apache.hudi.hive.MultiPartKeysValueExtractor")
.mode(SaveMode.Append)
.save(tablePath)
# SELECT
spark.sql("select * from mansipp_hudi_102_cow_fta_write_lf_table_update_test order by event_id").show();
+-------------------+--------------------+--------------------+----------------------+--------------------+--------+----------+--------------+--------------------+----------+
|_hoodie_commit_time|_hoodie_commit_seqno| _hoodie_record_key|_hoodie_partition_path| _hoodie_file_name|event_id|event_date| event_name| event_ts|event_type|
+-------------------+--------------------+--------------------+----------------------+--------------------+--------+----------+--------------+--------------------+----------+
| 20250805000927187|20250805000927187...|event_id:100,even...| type1|db420273-0543-4c0...| 100|2015-01-01|event_name_900|2015-01-01T13:51:...| type1|
| 20250805000927187|20250805000927187...|event_id:101,even...| type2|d888bc7e-09c8-44a...| 101|2015-01-01|event_name_546|2015-01-01T12:14:...| type2|
| 20250805000927187|20250805000927187...|event_id:102,even...| type3|198de9d0-7df0-4c5...| 102|2015-01-01|event_name_345|2015-01-01T13:51:...| type3|
| 20250805000927187|20250805000927187...|event_id:103,even...| type4|72c26c48-802d-473...| 103|2015-01-01|event_name_234|2015-01-01T13:51:...| type4|
| 20250805000927187|20250805000927187...|event_id:104,even...| type1|db420273-0543-4c0...| 104|2015-01-01|event_name_123|2015-01-01T12:15:...| type1|
| 20250805000927187|20250805000927187...|event_id:105,even...| type2|d888bc7e-09c8-44a...| 105|2015-01-01|event_name_678|2015-01-01T13:51:...| type2|
| 20250805000927187|20250805000927187...|event_id:106,even...| type3|198de9d0-7df0-4c5...| 106|2015-01-01|event_name_890|2015-01-01T13:51:...| type3|
| 20250805000927187|20250805000927187...|event_id:107,even...| type4|72c26c48-802d-473...| 107|2015-01-01|event_name_944|2015-01-01T13:51:...| type4|
| 20250805000927187|20250805000927187...|event_id:108,even...| type1|db420273-0543-4c0...| 108|2015-01-01|event_name_456|2015-01-01T13:51:...| type1|
| 20250805000927187|20250805000927187...|event_id:109,even...| type2|d888bc7e-09c8-44a...| 109|2015-01-01|event_name_567|2015-01-01T13:51:...| type2|
| 20250805000927187|20250805000927187...|event_id:110,even...| type3|198de9d0-7df0-4c5...| 110|2015-01-01|event_name_789|2015-01-01T12:15:...| type3|
| 20250805000927187|20250805000927187...|event_id:111,even...| type4|72c26c48-802d-473...| 111|2015-01-01|event_name_322|2015-01-01T13:51:...| type4|
| 20250805001335884|20250805001335884...|event_id:113,even...| type5|6f74bbeb-e7cd-4e2...| 113|2015-01-01|event_name_999|2015-01-01T14:00:...| type5|
+-------------------+--------------------+--------------------+----------------------+--------------------+--------+----------+--------------+--------------------+----------+
```
**Expected behavior**
Inserted data should be returned while performing insert and select using SQL.
**Environment Description**
* Hudi version : 1.0.2
* Spark version : 3.5.5/3.5.6
* Hive version :
* Hadoop version :
* Storage (HDFS/S3/GCS..) : s3
* Running on Docker? (yes/no) :
**Additional context**
Add any other context about the problem here.
**Stacktrace**
```Add the stacktrace of the error.```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Hudi 1.0.2 SQL Global_Bloom index not returning newly inserted data [hudi]
Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy commented on issue #13680:
URL: https://github.com/apache/hudi/issues/13680#issuecomment-3162485322
Hi @mansipp
Thanks for raising this issue; I've successfully reproduced it internally using the Hudi Spark Bundle version 1.0.2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Hudi 1.0.2 SQL Global_Bloom index not returning newly inserted data [hudi]
Posted by "mansipp (via GitHub)" <gi...@apache.org>.
mansipp commented on issue #13680:
URL: https://github.com/apache/hudi/issues/13680#issuecomment-3156297407
cc: @yihua
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Hudi 1.0.2 SQL Global_Bloom index not returning newly inserted data [hudi]
Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy commented on issue #13680:
URL: https://github.com/apache/hudi/issues/13680#issuecomment-3451980096
Hi @mansipp
Re-testing today confirms I cannot reproduce the issue. Since I've lost the previous reproduction steps, could you please retry the operation, starting with the open source Hudi Spark bundle?
```scala
scala> import org.apache.hudi.DataSourceWriteOptions
import org.apache.hudi.DataSourceWriteOptions
scala> import org.apache.spark.sql.SaveMode
import org.apache.spark.sql.SaveMode
scala> val df1 = Seq(
| ("100", "2015-01-01", "event_name_900", "2015-01-01T13:51:39.340396Z", "type1"),
| ("101", "2015-01-01", "event_name_546", "2015-01-01T12:14:58.597216Z", "type2"),
| ("102", "2015-01-01", "event_name_345", "2015-01-01T13:51:40.417052Z", "type3"),
| ("103", "2015-01-01", "event_name_234", "2015-01-01T13:51:40.519832Z", "type4"),
| ("104", "2015-01-01", "event_name_123", "2015-01-01T12:15:00.512679Z", "type1"),
| ("105", "2015-01-01", "event_name_678", "2015-01-01T13:51:42.248818Z", "type2"),
| ("106", "2015-01-01", "event_name_890", "2015-01-01T13:51:44.735360Z", "type3"),
| ("107", "2015-01-01", "event_name_944", "2015-01-01T13:51:45.019544Z", "type4"),
| ("108", "2015-01-01", "event_name_456", "2015-01-01T13:51:45.208007Z", "type1"),
| ("109", "2015-01-01", "event_name_567", "2015-01-01T13:51:45.369689Z", "type2"),
| ("110", "2015-01-01", "event_name_789", "2015-01-01T12:15:05.664947Z", "type3"),
| ("111", "2015-01-01", "event_name_322", "2015-01-01T13:51:47.388239Z", "type4")
| ).toDF("event_id", "event_date", "event_name", "event_ts", "event_type")
25/10/27 21:00:46 WARN DFSPropertiesConfiguration: Properties file file:/etc/hudi/conf/hudi-defaults.conf not found. Ignoring to load props file
25/10/27 21:00:46 WARN DFSPropertiesConfiguration: Cannot find HUDI_CONF_DIR, please set it as the dir of hudi-defaults.conf
df1: org.apache.spark.sql.DataFrame = [event_id: string, event_date: string ... 3 more fields]
scala> :paste
// Entering paste mode (ctrl-D to finish)
df1.write.format("hudi")
.option("hoodie.metadata.enable", "true")
.option("hoodie.table.name", tableName)
.option("hoodie.database.name", "default")
.option("hoodie.datasource.write.operation", "upsert")
.option("hoodie.datasource.write.table.type", "COPY_ON_WRITE")
.option("hoodie.datasource.write.recordkey.field", "event_id,event_date")
.option("hoodie.datasource.write.partitionpath.field", "event_type")
.option("hoodie.datasource.write.precombine.field", "event_ts")
.option("hoodie.datasource.write.keygenerator.class", "org.apache.hudi.keygen.ComplexKeyGenerator")
.option("hoodie.datasource.hive_sync.enable", "true")
.option("hoodie.datasource.meta.sync.enable", "true")
.option("hoodie.index.type", "GLOBAL_BLOOM")
.option("hoodie.datasource.hive_sync.mode", "hms")
.option("hoodie.datasource.hive_sync.database", "default")
.option("hoodie.datasource.hive_sync.table", tableName)
.option("hoodie.datasource.hive_sync.partition_fields", "event_type")
.option("hoodie.datasource.hive_sync.partition_extractor_class", "org.apache.hudi.hive.MultiPartKeysValueExtractor")
.mode(SaveMode.Overwrite)
.save(tablePath)
scala> spark.sql("select * from hudi_issue_13680 order by event_id").show();
+-------------------+--------------------+--------------------+----------------------+--------------------+--------+----------+--------------+--------------------+----------+
|_hoodie_commit_time|_hoodie_commit_seqno| _hoodie_record_key|_hoodie_partition_path| _hoodie_file_name|event_id|event_date| event_name| event_ts|event_type|
+-------------------+--------------------+--------------------+----------------------+--------------------+--------+----------+--------------+--------------------+----------+
| 20251027210113487|20251027210113487...|event_id:100,even...| type1|18755352-0d8f-463...| 100|2015-01-01|event_name_900|2015-01-01T13:51:...| type1|
| 20251027210113487|20251027210113487...|event_id:101,even...| type2|bf9cbd75-c800-40f...| 101|2015-01-01|event_name_546|2015-01-01T12:14:...| type2|
| 20251027210113487|20251027210113487...|event_id:102,even...| type3|1f11a59f-ba60-418...| 102|2015-01-01|event_name_345|2015-01-01T13:51:...| type3|
| 20251027210113487|20251027210113487...|event_id:103,even...| type4|ee267944-d497-45f...| 103|2015-01-01|event_name_234|2015-01-01T13:51:...| type4|
| 20251027210113487|20251027210113487...|event_id:104,even...| type1|18755352-0d8f-463...| 104|2015-01-01|event_name_123|2015-01-01T12:15:...| type1|
| 20251027210113487|20251027210113487...|event_id:105,even...| type2|bf9cbd75-c800-40f...| 105|2015-01-01|event_name_678|2015-01-01T13:51:...| type2|
| 20251027210113487|20251027210113487...|event_id:106,even...| type3|1f11a59f-ba60-418...| 106|2015-01-01|event_name_890|2015-01-01T13:51:...| type3|
| 20251027210113487|20251027210113487...|event_id:107,even...| type4|ee267944-d497-45f...| 107|2015-01-01|event_name_944|2015-01-01T13:51:...| type4|
| 20251027210113487|20251027210113487...|event_id:108,even...| type1|18755352-0d8f-463...| 108|2015-01-01|event_name_456|2015-01-01T13:51:...| type1|
| 20251027210113487|20251027210113487...|event_id:109,even...| type2|bf9cbd75-c800-40f...| 109|2015-01-01|event_name_567|2015-01-01T13:51:...| type2|
| 20251027210113487|20251027210113487...|event_id:110,even...| type3|1f11a59f-ba60-418...| 110|2015-01-01|event_name_789|2015-01-01T12:15:...| type3|
| 20251027210113487|20251027210113487...|event_id:111,even...| type4|ee267944-d497-45f...| 111|2015-01-01|event_name_322|2015-01-01T13:51:...| type4|
+-------------------+--------------------+--------------------+----------------------+--------------------+--------+----------+--------------+--------------------+----------+
scala> spark.sql("INSERT INTO hudi_issue_13680 (event_id, event_date, event_name, event_ts, event_type) VALUES('112', DATE('2015-01-01'), 'event_name_123', TIMESTAMP('2015-01-01 13:51:45'), 'type5')")
25/10/27 21:03:17 WARN HoodieTableFileSystemView: Partition: type5 is not available in store
25/10/27 21:03:17 WARN HoodieTableFileSystemView: Partition: type5 is not available in store
res5: org.apache.spark.sql.DataFrame = []
scala> spark.sql("select * from hudi_issue_13680 order by event_id").show();
+-------------------+--------------------+--------------------+----------------------+--------------------+--------+----------+--------------+--------------------+----------+
|_hoodie_commit_time|_hoodie_commit_seqno| _hoodie_record_key|_hoodie_partition_path| _hoodie_file_name|event_id|event_date| event_name| event_ts|event_type|
+-------------------+--------------------+--------------------+----------------------+--------------------+--------+----------+--------------+--------------------+----------+
| 20251027210113487|20251027210113487...|event_id:100,even...| type1|18755352-0d8f-463...| 100|2015-01-01|event_name_900|2015-01-01T13:51:...| type1|
| 20251027210113487|20251027210113487...|event_id:101,even...| type2|bf9cbd75-c800-40f...| 101|2015-01-01|event_name_546|2015-01-01T12:14:...| type2|
| 20251027210113487|20251027210113487...|event_id:102,even...| type3|1f11a59f-ba60-418...| 102|2015-01-01|event_name_345|2015-01-01T13:51:...| type3|
| 20251027210113487|20251027210113487...|event_id:103,even...| type4|ee267944-d497-45f...| 103|2015-01-01|event_name_234|2015-01-01T13:51:...| type4|
| 20251027210113487|20251027210113487...|event_id:104,even...| type1|18755352-0d8f-463...| 104|2015-01-01|event_name_123|2015-01-01T12:15:...| type1|
| 20251027210113487|20251027210113487...|event_id:105,even...| type2|bf9cbd75-c800-40f...| 105|2015-01-01|event_name_678|2015-01-01T13:51:...| type2|
| 20251027210113487|20251027210113487...|event_id:106,even...| type3|1f11a59f-ba60-418...| 106|2015-01-01|event_name_890|2015-01-01T13:51:...| type3|
| 20251027210113487|20251027210113487...|event_id:107,even...| type4|ee267944-d497-45f...| 107|2015-01-01|event_name_944|2015-01-01T13:51:...| type4|
| 20251027210113487|20251027210113487...|event_id:108,even...| type1|18755352-0d8f-463...| 108|2015-01-01|event_name_456|2015-01-01T13:51:...| type1|
| 20251027210113487|20251027210113487...|event_id:109,even...| type2|bf9cbd75-c800-40f...| 109|2015-01-01|event_name_567|2015-01-01T13:51:...| type2|
| 20251027210113487|20251027210113487...|event_id:110,even...| type3|1f11a59f-ba60-418...| 110|2015-01-01|event_name_789|2015-01-01T12:15:...| type3|
| 20251027210113487|20251027210113487...|event_id:111,even...| type4|ee267944-d497-45f...| 111|2015-01-01|event_name_322|2015-01-01T13:51:...| type4|
| 20251027210316907|20251027210316907...|event_id:112,even...| type5|3a3e838b-7b34-494...| 112|2015-01-01|event_name_123| 2015-01-01 13:51:45| type5|
+-------------------+--------------------+--------------------+----------------------+--------------------+--------+----------+--------------+--------------------+----------+
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Hudi 1.0.2 SQL Global_Bloom index not returning newly inserted data [hudi]
Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy commented on issue #13680:
URL: https://github.com/apache/hudi/issues/13680#issuecomment-3491186087
Hi @mansipp
Are you able to re-test this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Hudi 1.0.2 SQL Global_Bloom index not returning newly inserted data [hudi]
Posted by "mansipp (via GitHub)" <gi...@apache.org>.
mansipp commented on issue #13680:
URL: https://github.com/apache/hudi/issues/13680#issuecomment-3559925752
@rangareddy Are you using Hudi 1.1.0 for your latest successful run?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Hudi 1.0.2 SQL Global_Bloom index not returning newly inserted data [hudi]
Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy commented on issue #13680:
URL: https://github.com/apache/hudi/issues/13680#issuecomment-3195017329
Hi @linliu-code
Yes, I used the open-source Hudi Spark bundle to test the above scenario.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Hudi 1.0.2 SQL Global_Bloom index not returning newly inserted data [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on issue #13680:
URL: https://github.com/apache/hudi/issues/13680#issuecomment-3164615595
@rangareddy , did you mean there are issues with Vinilla hudi-spark-bundle version 1.0.2?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Hudi 1.0.2 SQL Global_Bloom index not returning newly inserted data [hudi]
Posted by "deepakpanda93 (via GitHub)" <gi...@apache.org>.
deepakpanda93 commented on issue #13680:
URL: https://github.com/apache/hudi/issues/13680#issuecomment-3569330006
Hello @mansipp
I just tested with Hudi 1.0.2 open source bundle and not able to reproduce the issue.
```
scala> import org.apache.hudi.DataSourceWriteOptions
import org.apache.hudi.DataSourceWriteOptions
scala> import org.apache.spark.sql.SaveMode
import org.apache.spark.sql.SaveMode
scala> val tableName = "hudi_issue_13680"
tableName: String = hudi_issue_13680
scala> var tablePath = "file:///tmp/hudi_issue_13680"
tablePath: String = file:///tmp/hudi_issue_13680
scala> val df1 = Seq(
| ("100", "2015-01-01", "event_name_900", "2015-01-01T13:51:39.340396Z", "type1"),
| ("101", "2015-01-01", "event_name_546", "2015-01-01T12:14:58.597216Z", "type2"),
| ("102", "2015-01-01", "event_name_345", "2015-01-01T13:51:40.417052Z", "type3"),
| ("103", "2015-01-01", "event_name_234", "2015-01-01T13:51:40.519832Z", "type4"),
| ("104", "2015-01-01", "event_name_123", "2015-01-01T12:15:00.512679Z", "type1"),
| ("105", "2015-01-01", "event_name_678", "2015-01-01T13:51:42.248818Z", "type2"),
| ("106", "2015-01-01", "event_name_890", "2015-01-01T13:51:44.735360Z", "type3"),
| ("107", "2015-01-01", "event_name_944", "2015-01-01T13:51:45.019544Z", "type4"),
| ("108", "2015-01-01", "event_name_456", "2015-01-01T13:51:45.208007Z", "type1"),
| ("109", "2015-01-01", "event_name_567", "2015-01-01T13:51:45.369689Z", "type2"),
| ("110", "2015-01-01", "event_name_789", "2015-01-01T12:15:05.664947Z", "type3"),
| ("111", "2015-01-01", "event_name_322", "2015-01-01T13:51:47.388239Z", "type4")
| ).toDF("event_id", "event_date", "event_name", "event_ts", "event_type")
25/11/24 07:21:49 WARN MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties
df1: org.apache.spark.sql.DataFrame = [event_id: string, event_date: string ... 3 more fields]
scala> :paste
// Entering paste mode (ctrl-D to finish)
df1.write.format("hudi")
.option("hoodie.metadata.enable", "true")
.option("hoodie.table.name", tableName)
.option("hoodie.database.name", "default")
.option("hoodie.datasource.write.operation", "upsert")
.option("hoodie.datasource.write.table.type", "COPY_ON_WRITE")
.option("hoodie.datasource.write.recordkey.field", "event_id,event_date")
.option("hoodie.datasource.write.partitionpath.field", "event_type")
.option("hoodie.datasource.write.precombine.field", "event_ts")
.option("hoodie.datasource.write.keygenerator.class", "org.apache.hudi.keygen.ComplexKeyGenerator")
.option("hoodie.datasource.hive_sync.enable", "true")
.option("hoodie.datasource.meta.sync.enable", "true")
.option("hoodie.index.type", "GLOBAL_BLOOM")
.option("hoodie.datasource.hive_sync.mode", "hms")
.option("hoodie.datasource.hive_sync.database", "default")
.option("hoodie.datasource.hive_sync.table", tableName)
.option("hoodie.datasource.hive_sync.partition_fields", "event_type")
.option("hoodie.datasource.hive_sync.partition_extractor_class", "org.apache.hudi.hive.MultiPartKeysValueExtractor")
.mode(SaveMode.Overwrite)
.save(tablePath)
// Exiting paste mode, now interpreting.
25/11/24 07:22:09 WARN MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
# WARNING: Unable to get Instrumentation. Dynamic Attach failed. You may add this JAR as -javaagent manually, or supply -Djdk.attach.allowAttachSelf
# WARNING: Unable to attach Serviceability Agent. sun.jvm.hotspot.memory.Universe.getNarrowOopBase()
25/11/24 07:22:11 WARN HoodieBloomIndex: fallback to loading column ranges from files
scala> spark.sql("select * from hudi_issue_13680 order by event_id").show();
+-------------------+--------------------+--------------------+----------------------+--------------------+--------+----------+--------------+--------------------+----------+
|_hoodie_commit_time|_hoodie_commit_seqno| _hoodie_record_key|_hoodie_partition_path| _hoodie_file_name|event_id|event_date| event_name| event_ts|event_type|
+-------------------+--------------------+--------------------+----------------------+--------------------+--------+----------+--------------+--------------------+----------+
| 20251124072209151|20251124072209151...|event_id:100,even...| type1|91dd24f5-7abd-4f2...| 100|2015-01-01|event_name_900|2015-01-01T13:51:...| type1|
| 20251124072209151|20251124072209151...|event_id:101,even...| type2|24a420cc-ab4b-4eb...| 101|2015-01-01|event_name_546|2015-01-01T12:14:...| type2|
| 20251124072209151|20251124072209151...|event_id:102,even...| type3|11828ac2-24b2-437...| 102|2015-01-01|event_name_345|2015-01-01T13:51:...| type3|
| 20251124072209151|20251124072209151...|event_id:103,even...| type4|79c3f13f-a5ec-4d3...| 103|2015-01-01|event_name_234|2015-01-01T13:51:...| type4|
| 20251124072209151|20251124072209151...|event_id:104,even...| type1|91dd24f5-7abd-4f2...| 104|2015-01-01|event_name_123|2015-01-01T12:15:...| type1|
| 20251124072209151|20251124072209151...|event_id:105,even...| type2|24a420cc-ab4b-4eb...| 105|2015-01-01|event_name_678|2015-01-01T13:51:...| type2|
| 20251124072209151|20251124072209151...|event_id:106,even...| type3|11828ac2-24b2-437...| 106|2015-01-01|event_name_890|2015-01-01T13:51:...| type3|
| 20251124072209151|20251124072209151...|event_id:107,even...| type4|79c3f13f-a5ec-4d3...| 107|2015-01-01|event_name_944|2015-01-01T13:51:...| type4|
| 20251124072209151|20251124072209151...|event_id:108,even...| type1|91dd24f5-7abd-4f2...| 108|2015-01-01|event_name_456|2015-01-01T13:51:...| type1|
| 20251124072209151|20251124072209151...|event_id:109,even...| type2|24a420cc-ab4b-4eb...| 109|2015-01-01|event_name_567|2015-01-01T13:51:...| type2|
| 20251124072209151|20251124072209151...|event_id:110,even...| type3|11828ac2-24b2-437...| 110|2015-01-01|event_name_789|2015-01-01T12:15:...| type3|
| 20251124072209151|20251124072209151...|event_id:111,even...| type4|79c3f13f-a5ec-4d3...| 111|2015-01-01|event_name_322|2015-01-01T13:51:...| type4|
+-------------------+--------------------+--------------------+----------------------+--------------------+--------+----------+--------------+--------------------+----------+
scala> spark.sql("INSERT INTO hudi_issue_13680 (event_id, event_date, event_name, event_ts, event_type) VALUES('112', DATE('2015-01-01'), 'event_name_123', TIMESTAMP('2015-01-01 13:51:45'), 'type5')")
25/11/24 07:22:27 WARN HoodieTableFileSystemView: Partition: type5 is not available in store
25/11/24 07:22:27 WARN HoodieTableFileSystemView: Partition: type5 is not available in store
res2: org.apache.spark.sql.DataFrame = []
scala> spark.sql("select * from hudi_issue_13680 order by event_id").show();
+-------------------+--------------------+--------------------+----------------------+--------------------+--------+----------+--------------+--------------------+----------+
|_hoodie_commit_time|_hoodie_commit_seqno| _hoodie_record_key|_hoodie_partition_path| _hoodie_file_name|event_id|event_date| event_name| event_ts|event_type|
+-------------------+--------------------+--------------------+----------------------+--------------------+--------+----------+--------------+--------------------+----------+
| 20251124072209151|20251124072209151...|event_id:100,even...| type1|91dd24f5-7abd-4f2...| 100|2015-01-01|event_name_900|2015-01-01T13:51:...| type1|
| 20251124072209151|20251124072209151...|event_id:101,even...| type2|24a420cc-ab4b-4eb...| 101|2015-01-01|event_name_546|2015-01-01T12:14:...| type2|
| 20251124072209151|20251124072209151...|event_id:102,even...| type3|11828ac2-24b2-437...| 102|2015-01-01|event_name_345|2015-01-01T13:51:...| type3|
| 20251124072209151|20251124072209151...|event_id:103,even...| type4|79c3f13f-a5ec-4d3...| 103|2015-01-01|event_name_234|2015-01-01T13:51:...| type4|
| 20251124072209151|20251124072209151...|event_id:104,even...| type1|91dd24f5-7abd-4f2...| 104|2015-01-01|event_name_123|2015-01-01T12:15:...| type1|
| 20251124072209151|20251124072209151...|event_id:105,even...| type2|24a420cc-ab4b-4eb...| 105|2015-01-01|event_name_678|2015-01-01T13:51:...| type2|
| 20251124072209151|20251124072209151...|event_id:106,even...| type3|11828ac2-24b2-437...| 106|2015-01-01|event_name_890|2015-01-01T13:51:...| type3|
| 20251124072209151|20251124072209151...|event_id:107,even...| type4|79c3f13f-a5ec-4d3...| 107|2015-01-01|event_name_944|2015-01-01T13:51:...| type4|
| 20251124072209151|20251124072209151...|event_id:108,even...| type1|91dd24f5-7abd-4f2...| 108|2015-01-01|event_name_456|2015-01-01T13:51:...| type1|
| 20251124072209151|20251124072209151...|event_id:109,even...| type2|24a420cc-ab4b-4eb...| 109|2015-01-01|event_name_567|2015-01-01T13:51:...| type2|
| 20251124072209151|20251124072209151...|event_id:110,even...| type3|11828ac2-24b2-437...| 110|2015-01-01|event_name_789|2015-01-01T12:15:...| type3|
| 20251124072209151|20251124072209151...|event_id:111,even...| type4|79c3f13f-a5ec-4d3...| 111|2015-01-01|event_name_322|2015-01-01T13:51:...| type4|
| 20251124072227182|20251124072227182...|event_id:112,even...| type5|44c71743-983e-4fc...| 112|2015-01-01|event_name_123| 2015-01-01 13:51:45| type5|
+-------------------+--------------------+--------------------+----------------------+--------------------+--------+----------+--------------+--------------------+----------+
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Hudi 1.0.2 SQL Global_Bloom index not returning newly inserted data [hudi]
Posted by "mansipp (via GitHub)" <gi...@apache.org>.
mansipp commented on issue #13680:
URL: https://github.com/apache/hudi/issues/13680#issuecomment-3387728615
@rangareddy any updates?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Hudi 1.0.2 SQL Global_Bloom index not returning newly inserted data [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on issue #13680:
URL: https://github.com/apache/hudi/issues/13680#issuecomment-3160388655
@mansipp , I followed your scrip but using the vanilla hudi 1.0.2, which shows the newly added data correctly.
`export SPARK_VERSION=3.5 # or 3.4, 3.3
spark-shell --packages org.apache.hudi:hudi-spark$SPARK_VERSION-bundle_2.12:1.0.2 \
--conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer' \
--conf 'spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog' \
--conf 'spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension' \
--conf 'spark.kryo.registrator=org.apache.spark.HoodieSparkKryoRegistrar'`
`scala> import org.apache.hudi.DataSourceWriteOptions
import org.apache.hudi.DataSourceWriteOptions
scala> import org.apache.spark.sql.SaveMode
import org.apache.spark.sql.SaveMode
scala> val df1 = Seq(
| ("100", "2015-01-01", "event_name_900", "2015-01-01T13:51:39.340396Z", "type1"),
| ("101", "2015-01-01", "event_name_546", "2015-01-01T12:14:58.597216Z", "type2"),
| ("102", "2015-01-01", "event_name_345", "2015-01-01T13:51:40.417052Z", "type3"),
| ("103", "2015-01-01", "event_name_234", "2015-01-01T13:51:40.519832Z", "type4"),
| ("104", "2015-01-01", "event_name_123", "2015-01-01T12:15:00.512679Z", "type1"),
| ("105", "2015-01-01", "event_name_678", "2015-01-01T13:51:42.248818Z", "type2"),
| ("106", "2015-01-01", "event_name_890", "2015-01-01T13:51:44.735360Z", "type3"),
| ("107", "2015-01-01", "event_name_944", "2015-01-01T13:51:45.019544Z", "type4"),
| ("108", "2015-01-01", "event_name_456", "2015-01-01T13:51:45.208007Z", "type1"),
| ("109", "2015-01-01", "event_name_567", "2015-01-01T13:51:45.369689Z", "type2"),
| ("110", "2015-01-01", "event_name_789", "2015-01-01T12:15:05.664947Z", "type3"),
| ("111", "2015-01-01", "event_name_322", "2015-01-01T13:51:47.388239Z", "type4")
| ).toDF("event_id", "event_date", "event_name", "event_ts", "event_type")
25/08/06 07:15:38 WARN DFSPropertiesConfiguration: Properties file file:/etc/hudi/conf/hudi-defaults.conf not found. Ignoring to load props file
25/08/06 07:15:38 WARN DFSPropertiesConfiguration: Cannot find HUDI_CONF_DIR, please set it as the dir of hudi-defaults.conf
df1: org.apache.spark.sql.DataFrame = [event_id: string, event_date: string ... 3 more fields]
scala> var tableName = "mansipp_hudi_102_cow_fta_write_lf_table_update_test"
tableName: String = mansipp_hudi_102_cow_fta_write_lf_table_update_test
scala> var tablePath = /tmp/mansipp_hudi_102_cow_fta_write_lf_table_update_test
<console>:24: error: not found: value /
var tablePath = /tmp/mansipp_hudi_102_cow_fta_write_lf_table_update_test
^
<console>:24: error: not found: value /
var tablePath = /tmp/mansipp_hudi_102_cow_fta_write_lf_table_update_test
^
scala> var tablePath = "/tmp/mansipp_hudi_102_cow_fta_write_lf_table_update_test"
tablePath: String = /tmp/mansipp_hudi_102_cow_fta_write_lf_table_update_test
scala> df1.write.format("hudi")
res0: org.apache.spark.sql.DataFrameWriter[org.apache.spark.sql.Row] = org.apache.spark.sql.DataFrameWriter@d6ee83b
scala> .option("hoodie.metadata.enable", "true")
res1: org.apache.spark.sql.DataFrameWriter[org.apache.spark.sql.Row] = org.apache.spark.sql.DataFrameWriter@d6ee83b
scala> .option("hoodie.table.name", tableName)
res2: org.apache.spark.sql.DataFrameWriter[org.apache.spark.sql.Row] = org.apache.spark.sql.DataFrameWriter@d6ee83b
scala> .option("hoodie.database.name", "default")
res3: org.apache.spark.sql.DataFrameWriter[org.apache.spark.sql.Row] = org.apache.spark.sql.DataFrameWriter@d6ee83b
scala> .option("hoodie.datasource.write.operation", "upsert")
res4: org.apache.spark.sql.DataFrameWriter[org.apache.spark.sql.Row] = org.apache.spark.sql.DataFrameWriter@d6ee83b
scala> .option("hoodie.datasource.write.table.type", "COPY_ON_WRITE")
res5: org.apache.spark.sql.DataFrameWriter[org.apache.spark.sql.Row] = org.apache.spark.sql.DataFrameWriter@d6ee83b
scala> .option("hoodie.datasource.write.recordkey.field", "event_id,event_date")
res6: org.apache.spark.sql.DataFrameWriter[org.apache.spark.sql.Row] = org.apache.spark.sql.DataFrameWriter@d6ee83b
scala> .option("hoodie.datasource.write.partitionpath.field", "event_type")
res7: org.apache.spark.sql.DataFrameWriter[org.apache.spark.sql.Row] = org.apache.spark.sql.DataFrameWriter@d6ee83b
scala> .option("hoodie.datasource.write.precombine.field", "event_ts")
res8: org.apache.spark.sql.DataFrameWriter[org.apache.spark.sql.Row] = org.apache.spark.sql.DataFrameWriter@d6ee83b
scala> .option("hoodie.datasource.write.keygenerator.class", "org.apache.hudi.keygen.ComplexKeyGenerator")
res9: org.apache.spark.sql.DataFrameWriter[org.apache.spark.sql.Row] = org.apache.spark.sql.DataFrameWriter@d6ee83b
scala> .option("hoodie.datasource.hive_sync.enable", "true")
res10: org.apache.spark.sql.DataFrameWriter[org.apache.spark.sql.Row] = org.apache.spark.sql.DataFrameWriter@d6ee83b
scala> .option("hoodie.datasource.meta.sync.enable", "true")
res11: org.apache.spark.sql.DataFrameWriter[org.apache.spark.sql.Row] = org.apache.spark.sql.DataFrameWriter@d6ee83b
scala> .option("hoodie.index.type", "GLOBAL_BLOOM")
res12: org.apache.spark.sql.DataFrameWriter[org.apache.spark.sql.Row] = org.apache.spark.sql.DataFrameWriter@d6ee83b
scala> .option("hoodie.datasource.hive_sync.mode", "hms")
res13: org.apache.spark.sql.DataFrameWriter[org.apache.spark.sql.Row] = org.apache.spark.sql.DataFrameWriter@d6ee83b
scala> .option("hoodie.datasource.hive_sync.database", "default")
res14: org.apache.spark.sql.DataFrameWriter[org.apache.spark.sql.Row] = org.apache.spark.sql.DataFrameWriter@d6ee83b
scala> .option("hoodie.datasource.hive_sync.table", tableName)
res15: org.apache.spark.sql.DataFrameWriter[org.apache.spark.sql.Row] = org.apache.spark.sql.DataFrameWriter@d6ee83b
scala> .option("hoodie.datasource.hive_sync.partition_fields", "event_type")
res16: org.apache.spark.sql.DataFrameWriter[org.apache.spark.sql.Row] = org.apache.spark.sql.DataFrameWriter@d6ee83b
scala> .option("hoodie.datasource.hive_sync.partition_extractor_class", "org.apache.hudi.hive.MultiPartKeysValueExtractor")
res17: org.apache.spark.sql.DataFrameWriter[org.apache.spark.sql.Row] = org.apache.spark.sql.DataFrameWriter@d6ee83b
scala> .mode(SaveMode.Overwrite)
res18: org.apache.spark.sql.DataFrameWriter[org.apache.spark.sql.Row] = org.apache.spark.sql.DataFrameWriter@d6ee83b
scala> .save(tablePath)
25/08/06 07:16:33 WARN HiveConf: HiveConf of name hive.stats.jdbc.timeout does not exist
25/08/06 07:16:33 WARN HiveConf: HiveConf of name hive.stats.retries.wait does not exist
25/08/06 07:16:36 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 2.3.0
25/08/06 07:16:36 WARN ObjectStore: setMetaStoreSchemaVersion called but recording version is disabled: version = 2.3.0, comment = Set by MetaStore linliu@127.0.0.1
25/08/06 07:16:42 WARN MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
# WARNING: Unable to attach Serviceability Agent. Unable to attach even with module exceptions: [org.apache.hudi.org.openjdk.jol.vm.sa.SASupportException: Sense failed., org.apache.hudi.org.openjdk.jol.vm.sa.SASupportException: Sense failed., org.apache.hudi.org.openjdk.jol.vm.sa.SASupportException: Sense failed.]
25/08/06 07:16:47 WARN HoodieBloomIndex: fallback to loading column ranges from files
25/08/06 07:16:55 WARN log: Updating partition stats fast for: mansipp_hudi_102_cow_fta_write_lf_table_update_test
25/08/06 07:16:55 WARN log: Updated size to 436099
25/08/06 07:16:55 WARN log: Updating partition stats fast for: mansipp_hudi_102_cow_fta_write_lf_table_update_test
25/08/06 07:16:55 WARN log: Updated size to 436097
25/08/06 07:16:55 WARN log: Updating partition stats fast for: mansipp_hudi_102_cow_fta_write_lf_table_update_test
25/08/06 07:16:55 WARN log: Updated size to 436098
25/08/06 07:16:55 WARN log: Updating partition stats fast for: mansipp_hudi_102_cow_fta_write_lf_table_update_test
25/08/06 07:16:55 WARN log: Updated size to 436090
25/08/06 07:16:56 WARN HiveConf: HiveConf of name hive.stats.jdbc.timeout does not exist
25/08/06 07:16:56 WARN HiveConf: HiveConf of name hive.stats.retries.wait does not exist
25/08/06 07:16:58 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
scala> spark.sql("select * from mansipp_hudi_102_cow_fta_write_lf_table_update_test order by event_id").show();
+-------------------+--------------------+--------------------+----------------------+--------------------+--------+----------+--------------+--------------------+----------+
|_hoodie_commit_time|_hoodie_commit_seqno| _hoodie_record_key|_hoodie_partition_path| _hoodie_file_name|event_id|event_date| event_name| event_ts|event_type|
+-------------------+--------------------+--------------------+----------------------+--------------------+--------+----------+--------------+--------------------+----------+
| 20250806071637700|20250806071637700...|event_id:100,even...| type1|b282bd3f-41e6-4df...| 100|2015-01-01|event_name_900|2015-01-01T13:51:...| type1|
| 20250806071637700|20250806071637700...|event_id:101,even...| type2|2677a131-5502-4b6...| 101|2015-01-01|event_name_546|2015-01-01T12:14:...| type2|
| 20250806071637700|20250806071637700...|event_id:102,even...| type3|1d8fd795-117f-40f...| 102|2015-01-01|event_name_345|2015-01-01T13:51:...| type3|
| 20250806071637700|20250806071637700...|event_id:103,even...| type4|48336739-a555-415...| 103|2015-01-01|event_name_234|2015-01-01T13:51:...| type4|
| 20250806071637700|20250806071637700...|event_id:104,even...| type1|b282bd3f-41e6-4df...| 104|2015-01-01|event_name_123|2015-01-01T12:15:...| type1|
| 20250806071637700|20250806071637700...|event_id:105,even...| type2|2677a131-5502-4b6...| 105|2015-01-01|event_name_678|2015-01-01T13:51:...| type2|
| 20250806071637700|20250806071637700...|event_id:106,even...| type3|1d8fd795-117f-40f...| 106|2015-01-01|event_name_890|2015-01-01T13:51:...| type3|
| 20250806071637700|20250806071637700...|event_id:107,even...| type4|48336739-a555-415...| 107|2015-01-01|event_name_944|2015-01-01T13:51:...| type4|
| 20250806071637700|20250806071637700...|event_id:108,even...| type1|b282bd3f-41e6-4df...| 108|2015-01-01|event_name_456|2015-01-01T13:51:...| type1|
| 20250806071637700|20250806071637700...|event_id:109,even...| type2|2677a131-5502-4b6...| 109|2015-01-01|event_name_567|2015-01-01T13:51:...| type2|
| 20250806071637700|20250806071637700...|event_id:110,even...| type3|1d8fd795-117f-40f...| 110|2015-01-01|event_name_789|2015-01-01T12:15:...| type3|
| 20250806071637700|20250806071637700...|event_id:111,even...| type4|48336739-a555-415...| 111|2015-01-01|event_name_322|2015-01-01T13:51:...| type4|
+-------------------+--------------------+--------------------+----------------------+--------------------+--------+----------+--------------+--------------------+----------+
scala> spark.sql("INSERT INTO mansipp_hudi_102_cow_fta_write_lf_table_update_test (event_id, event_date, event_name, event_ts, event_type) VALUES('112', DATE('2015-01-01'), 'event_name_123', TIMESTAMP('2015-01-01 13:51:45'), 'type5')")
25/08/06 07:17:23 WARN HoodieTableFileSystemView: Partition: type5 is not available in store
25/08/06 07:17:23 WARN HoodieTableFileSystemView: Partition: type5 is not available in store
25/08/06 07:17:26 WARN log: Updating partition stats fast for: mansipp_hudi_102_cow_fta_write_lf_table_update_test
25/08/06 07:17:26 WARN log: Updated size to 435924
25/08/06 07:17:26 WARN HiveConf: HiveConf of name hive.stats.jdbc.timeout does not exist
25/08/06 07:17:26 WARN HiveConf: HiveConf of name hive.stats.retries.wait does not exist
res21: org.apache.spark.sql.DataFrame = []
scala>
scala> spark.sql("select * from mansipp_hudi_102_cow_fta_write_lf_table_update_test order by event_id").show();
+-------------------+--------------------+--------------------+----------------------+--------------------+--------+----------+--------------+--------------------+----------+
|_hoodie_commit_time|_hoodie_commit_seqno| _hoodie_record_key|_hoodie_partition_path| _hoodie_file_name|event_id|event_date| event_name| event_ts|event_type|
+-------------------+--------------------+--------------------+----------------------+--------------------+--------+----------+--------------+--------------------+----------+
| 20250806071637700|20250806071637700...|event_id:100,even...| type1|b282bd3f-41e6-4df...| 100|2015-01-01|event_name_900|2015-01-01T13:51:...| type1|
| 20250806071637700|20250806071637700...|event_id:101,even...| type2|2677a131-5502-4b6...| 101|2015-01-01|event_name_546|2015-01-01T12:14:...| type2|
| 20250806071637700|20250806071637700...|event_id:102,even...| type3|1d8fd795-117f-40f...| 102|2015-01-01|event_name_345|2015-01-01T13:51:...| type3|
| 20250806071637700|20250806071637700...|event_id:103,even...| type4|48336739-a555-415...| 103|2015-01-01|event_name_234|2015-01-01T13:51:...| type4|
| 20250806071637700|20250806071637700...|event_id:104,even...| type1|b282bd3f-41e6-4df...| 104|2015-01-01|event_name_123|2015-01-01T12:15:...| type1|
| 20250806071637700|20250806071637700...|event_id:105,even...| type2|2677a131-5502-4b6...| 105|2015-01-01|event_name_678|2015-01-01T13:51:...| type2|
| 20250806071637700|20250806071637700...|event_id:106,even...| type3|1d8fd795-117f-40f...| 106|2015-01-01|event_name_890|2015-01-01T13:51:...| type3|
| 20250806071637700|20250806071637700...|event_id:107,even...| type4|48336739-a555-415...| 107|2015-01-01|event_name_944|2015-01-01T13:51:...| type4|
| 20250806071637700|20250806071637700...|event_id:108,even...| type1|b282bd3f-41e6-4df...| 108|2015-01-01|event_name_456|2015-01-01T13:51:...| type1|
| 20250806071637700|20250806071637700...|event_id:109,even...| type2|2677a131-5502-4b6...| 109|2015-01-01|event_name_567|2015-01-01T13:51:...| type2|
| 20250806071637700|20250806071637700...|event_id:110,even...| type3|1d8fd795-117f-40f...| 110|2015-01-01|event_name_789|2015-01-01T12:15:...| type3|
| 20250806071637700|20250806071637700...|event_id:111,even...| type4|48336739-a555-415...| 111|2015-01-01|event_name_322|2015-01-01T13:51:...| type4|
| 20250806071722764|20250806071722764...|event_id:112,even...| type5|2566b8a7-59cb-485...| 112|2015-01-01|event_name_123| 2015-01-01 13:51:45| type5|
+-------------------+--------------------+--------------------+----------------------+--------------------+--------+----------+--------------+--------------------+----------+
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Hudi 1.0.2 SQL Global_Bloom index not returning newly inserted data [hudi]
Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy commented on issue #13680:
URL: https://github.com/apache/hudi/issues/13680#issuecomment-3612756951
Hi @mansipp
This issue was resolved in Hudi 1.0.2 (or an earlier version), so we are closing this ticket. Please feel free to reopen it if you still encounter the problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Hudi 1.0.2 SQL Global_Bloom index not returning newly inserted data [hudi]
Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy closed issue #13680: [SUPPORT] Hudi 1.0.2 SQL Global_Bloom index not returning newly inserted data
URL: https://github.com/apache/hudi/issues/13680
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org