You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "nareshbab (via GitHub)" <gi...@apache.org> on 2025/06/30 14:30:12 UTC
[I] Spark Structured Streaming: Data not immediately visible after merge/write operations [iceberg]
nareshbab opened a new issue, #13431:
URL: https://github.com/apache/iceberg/issues/13431
### Query engine
Apache Spark
### Question
When running Spark Structured Streaming with Apache Iceberg on local, data written in one micro-batch is not immediately visible for subsequent queries until the next micro-batch is processed. This is only applicable in the cases where MERGE is done in the pipeline. For append, replace/overwrite scenarios data is visible as soon as a microbatch is processed.
```
**Pipeline flow:**
Kafka -> Microbatch (30s) -> foreachbatch -> write to snapshot table -> merge to main table using snapshot table -> process new batch
```
```
Environment
Java: 17
Scala: 2.13.16
Spark Version: 3.5.1
Iceberg Version: 1.9.1
Catalog Type: Hadoop Catalog
Storage: Local filesystem
Write Mode: Structured Streaming with foreachBatch
```
**Expected Behavior**
After a merge/write operation completes in a micro-batch, the data should be immediately visible to subsequent queries without waiting for the next micro-batch to trigger.
Actual Behavior
Data written in micro-batch N is only visible after micro-batch N+1 starts processing, causing visibility delays ranging from seconds to minutes depending on trigger interval.
**Questions for Iceberg Team**
- Is this expected behavior for Iceberg with Spark Structured Streaming?
- What's the recommended pattern for achieving immediate visibility in streaming scenarios?
- Should we always use REFRESH TABLE after write operations in streaming contexts? NOTE: REFRESH Table is also not working for this scenario
- Are there Iceberg-specific configurations that can improve metadata refresh frequency for streaming workloads?
- Is there a way to flush/commit transactions explicitly within a micro-batch to ensure immediate visibility?
Additional Context
This issue is consistently reproducible across different environments (local, Biglake) and appears to be a fundamental interaction between Spark's streaming execution model and Iceberg's transaction/metadata management.
Any guidance on best practices or potential improvements would be greatly appreciated!
PS: Same code base works with Bigquery MERGE scenarios and data is available immediately post MERGE statement execution
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Spark Structured Streaming: Data not immediately visible after merge operations [iceberg]
Posted by "nareshbab (via GitHub)" <gi...@apache.org>.
nareshbab commented on issue #13431:
URL: https://github.com/apache/iceberg/issues/13431#issuecomment-3019433000
Additional Detail: This behaviour is only seen in Spark Structured Streaming scenario, batch pipeline to execute MERGE on Iceberg is working as expected and data is also visible immediately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Spark Structured Streaming: Data not immediately visible after merge operations [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #13431:
URL: https://github.com/apache/iceberg/issues/13431#issuecomment-3697882698
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Spark Structured Streaming: Data not immediately visible after merge operations [iceberg]
Posted by "nareshbab (via GitHub)" <gi...@apache.org>.
nareshbab commented on issue #13431:
URL: https://github.com/apache/iceberg/issues/13431#issuecomment-3021744943
@RussellSpitzer Here's the rudimentary code to replicate this. I hope this provides the clarity on the process
Overall steps executed:
- Start streaming pipeline
- post a message in kafka topic with schema as shared below
- Batch N Starts
- Write to Audit table
- Merge to snapshot table
- Batch N ends
- Run another spark script with iceberg configurations to same db
- query from spark script "select * from snapshot" & "select * from temp_audit_table" to see current rows reflected in the shell. Data is visible in audit table but not in snapshot table
- Send another message in kafka whenever user desires
- Batch N+1 starts
- Write to Audit table
- Merge to snapshot table
- Batch N ends
- Run another spark script to query data "select * from snapshot" & "select * from temp_audit_table" to see current rows reflected in the shell. Data from Batch N is visible in snapshot. But data from Batch N+1 is only visible in audit table
`
// Required imports
import org.apache.spark.sql.streaming.Trigger
import org.apache.spark.sql.{Dataset, Row, SaveMode, SparkSession}
import org.apache.spark.internal.Logging
import org.apache.spark.sql.functions.{col, current_date, from_json}
import org.apache.spark.sql.types.{DoubleType, IntegerType, StringType, StructField, StructType, TimestampType}
object MergeTest extends Logging {
// Configuration variables
val catalogName = "local"
val dbName = "biglake_db_us_central1
[batchN_logs.txt](https://github.com/user-attachments/files/20990963/batchN_logs.txt)
[batchN+1_logs.txt](https://github.com/user-attachments/files/20990962/batchN%2B1_logs.txt)
"
val tableName = "snapshot_table"
val tempTableName = "temp_audit_table"
val warehouse = "<>/data/warehouse"
val checkpointLocation = "<>/merge-test-checkpoint"
def main(args: Array[String]): Unit = {
// Spark session with Iceberg Local catalog configuration
implicit val spark = SparkSession.builder()
.appName("Iceberg to BigLake")
.master("local[1]")
.config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")
.config(s"spark.sql.catalog.${catalogName}", "org.apache.iceberg.spark.SparkCatalog")
.config(s"spark.sql.catalog.${catalogName}.type", "hadoop")
.config(s"spark.sql.catalog.${catalogName}.warehouse", warehouse)
.getOrCreate()
val kafkaDF = spark
.readStream
.format("kafka")
.option("kafka.bootstrap.servers", "localhost:9092")
.option("subscribe", "user-updates")
.option("startingOffsets", "earliest")
.option("maxOffsetsPerTrigger", 1000)
.option("kafka.security.protocol", "PLAINTEXT")
.load()
val schema = StructType(Array(
StructField("record_id", StringType, nullable = false),
StructField("field1", StringType, nullable = true),
StructField("field2", IntegerType, nullable = true),
StructField("field3", DoubleType, nullable = true),
StructField("operation", StringType, nullable = false), // INSERT, UPDATE, DELETE
StructField("event_timestamp", TimestampType, nullable = false)
))
val enrichedDF = kafkaDF
.select(
from_json(col("value").cast("string"), schema).as("update_data"),
current_date().as("partition_date")
)
.select(
col("update_data.*"),
col("partition_date")
)
.filter(col("record_id").isNotNull)
enrichedDF
.writeStream
.foreachBatch((batchDF: Dataset[Row], batchId: Long) => {
writeAuditTable(batchDF, batchId)
executeMerge(batchDF, batchId)
})
.option("checkpointLocation", checkpointLocation)
.trigger(Trigger.ProcessingTime("30 seconds"))
.queryName("KafkaToSnapshotPipeline")
.start()
.awaitTermination()
}
def writeAuditTable(batchDF: Dataset[Row], batchID: Long)(implicit spark: SparkSession): Unit = {
logInfo(s"Writing audit table for batch $batchID")
ensureTableExists()
try {
// Write the batch to the temp audit table
batchDF
.writeTo(s"$catalogName.$dbName.$tempTableName")
.replace()
logInfo(s"Successfully wrote audit table for batch $batchID")
} catch {
case e: Exception =>
logError(s"Failed to write audit table for batch $batchID", e)
throw e
}
}
def executeMerge(batchDF: Dataset[Row], batchId: Long)(implicit spark: SparkSession): Unit = {
logInfo(s"Executing Iceberg merge operation for batch $batchId")
ensureTableExists()
try {
// Execute Iceberg MERGE INTO statement
val mergeSQL = s"""
MERGE INTO $catalogName.$dbName.$tableName AS target
USING $catalogName.$dbName.$tempTableName AS source
ON target.record_id = source.record_id
WHEN MATCHED AND source.operation = 'DELETE' THEN
DELETE
WHEN MATCHED AND source.operation IN ('UPDATE', 'INSERT') THEN
UPDATE SET *
WHEN NOT MATCHED AND source.operation IN ('INSERT', 'UPDATE') THEN
INSERT *
"""
logInfo(s"Executing Iceberg MERGE: ${mergeSQL.take(600)}...")
spark.sql(mergeSQL)
logInfo(s"Successfully executed Iceberg merge for batch $batchId")
} catch {
case e: Exception =>
logError(s"Failed to execute Iceberg merge for batch $batchId", e)
throw e
}
}
def ensureTableExists()(implicit spark: SparkSession): Unit = {
val tableExists = spark.catalog.tableExists(s"$catalogName.$dbName.$tableName")
val tempTableExists = spark.catalog.tableExists(s"$catalogName.$dbName.$tempTableName")
if (!tableExists) {
logInfo(s"Table $catalogName.$dbName.$tableName does not exist, creating it")
val createTableSQL = s"""
CREATE TABLE IF NOT EXISTS $catalogName.$dbName.$tableName (
record_id STRING,
field1 STRING,
field2 INT,
field3 DOUBLE,
operation STRING,
event_timestamp TIMESTAMP,
partition_date DATE
)
USING iceberg
PARTITIONED BY (partition_date)
"""
spark.sql(createTableSQL)
logInfo(s"Created Iceberg Table $catalogName.$dbName.$tableName")
} else {
logInfo(s"Table $catalogName.$dbName.$tableName already exists")
}
//Creating temp table
val createTempTableSQL = s"""
CREATE TABLE $catalogName.$dbName.$tempTableName (
record_id STRING,
field1 STRING,
field2 INT,
field3 DOUBLE,
operation STRING,
event_timestamp TIMESTAMP,
partition_date DATE
)
USING iceberg
"""
if (!tempTableExists) {
logInfo(s"Temp table $catalogName.$dbName.$tempTableName does not exist, creating it")
spark.sql(createTempTableSQL)
logInfo(s"Created Temp Iceberg Table $catalogName.$dbName.$tempTableName")
}
}
}
`
Kafka Message
`
{
"record_id": "user_001",
"field1": "Carol Davis",
"field2": 25,
"field3": 88.7,
"operation": "INSERT",
"event_timestamp": "2024-06-24T09:02:00.000Z"
}
`
Attaching below are the logs for batch N and batch N+1 from another spark script to check table state between batches
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Spark Structured Streaming: Data not immediately visible after merge operations [iceberg]
Posted by "nareshbab (via GitHub)" <gi...@apache.org>.
nareshbab commented on issue #13431:
URL: https://github.com/apache/iceberg/issues/13431#issuecomment-3026603432
Update: Figured out the behaviour happening here. Spark is committing the changes after the end of a batch. So during the execution of a specific batch, though data is written in temp table its reflecting only after batch end. This is leading to scenario where merge is reading empty stale or empty data from temp table and hence the data is not reflecting to the snapshot table.
Workaround: Explicitly commit the write for temp table post writing. Modified the below method to make it work
```
def writeAuditTable(batchDF: Dataset[Row], batchID: Long)(implicit spark: SparkSession): Unit = {
logInfo(s"Writing audit table for batch $batchID")
ensureTableExists()
try {
// Write the batch to the temp audit table
batchDF
.writeTo(s"$catalogName.$dbName.$tempTableName")
.replace()
// Explicitly refresh the temp table to commit the data
spark.catalog.refreshTable(s"$catalogName.$dbName.$tempTableName")
logInfo(s"Successfully wrote audit table for batch $batchID")
} catch {
case e: Exception =>
logError(s"Failed to write audit table for batch $batchID", e)
throw e
}
```
Thank you for the support here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Spark Structured Streaming: Data not immediately visible after merge operations [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #13431:
URL: https://github.com/apache/iceberg/issues/13431#issuecomment-3747120510
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Spark Structured Streaming: Data not immediately visible after merge operations [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on issue #13431:
URL: https://github.com/apache/iceberg/issues/13431#issuecomment-3020082336
Could you elaborate a bit more? It sounds like the behavior is
Batch N Starts
Batch N Stops
ForEach Executes and Writes
Some Gap Time Here?
Batch N+1 Starts
Data Visible
Is that correct?
Could you also elaborate on how you are measuring data visibility? What queries are you running and how are you running them?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Spark Structured Streaming: Data not immediately visible after merge operations [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #13431: Spark Structured Streaming: Data not immediately visible after merge operations
URL: https://github.com/apache/iceberg/issues/13431
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org