You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "LuciferYang (via GitHub)" <gi...@apache.org> on 2025/07/31 15:22:38 UTC
[PR] Revert "[SPARK-52610][BUILD] Upgrade rocksdbjni to 10.2.1" [spark]
LuciferYang opened a new pull request, #51750:
URL: https://github.com/apache/spark/pull/51750
### What changes were proposed in this pull request?
This pr reverts commit 2d4ff6eeeffd74ba6126936849cf6d3a71de7f70 to downgrade rocksdbjni from 10.2.1 to 9.8.4.
### Why are the changes needed?
Rocksdbjni 10.2.1 crashed during testing:
- https://github.com/apache/spark/actions/runs/16638741674/job/47084788641
```
#
# A fatal error has been detected by the Java Runtime Environment:
#
03:55:47.848 WARN org.apache.spark.scheduler.DAGScheduler: Failed to cancel job group 2d75e504-d47d-4eb3-ac5b-1ce066ae486b. Cannot find active jobs for it.
# SIGSEGV (0xb) at pc=0x00007f8d46c7eed9, pid=25508, tid=653254
#
# JRE version: OpenJDK Runtime Environment Zulu17.60+17-CRaC-CA (17.0.16+8) (build 17.0.16+8-LTS)
# Java VM: OpenJDK 64-Bit Server VM Zulu17.60+17-CRaC-CA (17.0.16+8-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# C03:55:47.850 ERROR org.apache.spark.util.Utils: Aborting task
org.apache.spark.SparkException: [CANNOT_WRITE_STATE_STORE.CANNOT_COMMIT] Error writing state store files for provider RocksDBStateStore[stateStoreId=StateStoreId[ operatorId=0, partitionId=2, storeName=default ], version=18]. Cannot perform commit during state checkpoint. SQLSTATE: 58030
at org.apache.spark.sql.errors.QueryExecutionErrors$.failedToCommitStateFileError(QueryExecutionErrors.scala:2249)
at org.apache.spark.sql.execution.streaming.state.RocksDBStateStoreProvider$RocksDBStateStore.commit(RocksDBStateStoreProvider.scala:392)
at org.apache.spark.sql.execution.streaming.TransformWithStateExec.$anonfun$processDataWithPartition$5(TransformWithStateExec.scala:341)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
at org.apache.spark.util.Utils$.timeTakenMs(Utils.scala:482)
at org.apache.spark.sql.execution.streaming.StateStoreWriter.timeTakenMs(statefulOperators.scala:406)
at org.apache.spark.sql.execution.streaming.StateStoreWriter.timeTakenMs$(statefulOperators.scala:406)
[librocksdbjni3950065922734541121.so+0x47eed9]03:55:47.851 ERROR org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask: Aborting commit for partition 2 (task 465, attempt 0, stage 161.0)
03:55:47.851 ERROR org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask: Aborted commit for partition 2 (task 465, attempt 0, stage 161.0)
rocksdb::DBImpl::FailIfCfHasTs(rocksdb::ColumnFamilyHandle const*) const+0x29
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/lib/systemd/systemd-coredump %P %u %g %s %t 9223372036854775808 %h %d" (or dumping to /home/runner/work/spark/spark/sql/core/core.25508)
#
# An error report file with more information is saved as:
# /home/runner/work/spark/spark/sql/core/hs_err_pid25508.log
03:55:47.852 WARN org.apache.spark.scheduler.TaskSetManager: Lost task 2.0 in stage 161.0 (TID 465) (localhost executor driver): TaskKilled (Stage cancelled: [SPARK_JOB_CANCELLED] Job 89 cancelled Query [id = a4a0ad54-035d-4049-961c-a36ee70e3d7c, runId = 2d75e504-d47d-4eb3-ac5b-1ce066ae486b] was stopped SQLSTATE: XXKDA)
#
# If you would like to submit a bug report, please visit:
# http://www.azul.com/support/
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
```
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Pass Github Actions
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] Revert "[SPARK-52610][BUILD] Upgrade rocksdbjni to 10.2.1" [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #51750: Revert "[SPARK-52610][BUILD] Upgrade rocksdbjni to 10.2.1"
URL: https://github.com/apache/spark/pull/51750
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] Revert "[SPARK-52610][BUILD] Upgrade rocksdbjni to 10.2.1" [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #51750:
URL: https://github.com/apache/spark/pull/51750#issuecomment-3155990208
I saw it's still happening frequently, @LuciferYang . I'm curious if that's independent from RocksDB itself because there are some RocksDB patches in Apache Spark code. It might introduce some misuse of RocksDB.
If the root cause is not RocksDB, we can bring SPARK-52610 back to the master branch. BTW, until now, I didn't get a chance to look at this in depth due to the other priority. I can re-trigger it but I'm not sure I can make some time for this in next two weeks. Sorry about that and thank you for investing your time on RocksDB stuff.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] Revert "[SPARK-52610][BUILD] Upgrade rocksdbjni to 10.2.1" [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #51750:
URL: https://github.com/apache/spark/pull/51750#issuecomment-3140390476
cc @dongjoon-hyun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] Revert "[SPARK-52610][BUILD] Upgrade rocksdbjni to 10.2.1" [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #51750:
URL: https://github.com/apache/spark/pull/51750#issuecomment-3146526956
Thank you @dongjoon-hyun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] Revert "[SPARK-52610][BUILD] Upgrade rocksdbjni to 10.2.1" [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #51750:
URL: https://github.com/apache/spark/pull/51750#issuecomment-3150590638
- https://github.com/apache/spark/actions/runs/16719761548/job/47321326271
- https://github.com/apache/spark/actions/runs/16715175590/job/47307181449
- https://github.com/apache/spark/actions/runs/16664111789/job/47228601789
- https://github.com/apache/spark/actions/runs/16712340318/job/47299359259
Do we need to further downgrade rocksdbjni? Judging from the submission pipeline data of the past three days, rocksdbjni still experiences crashes, primarily in the `sql-slow` and `sql-others` modules. @dongjoon-hyun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] Revert "[SPARK-52610][BUILD] Upgrade rocksdbjni to 10.2.1" [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #51750:
URL: https://github.com/apache/spark/pull/51750#issuecomment-3140836015
Merged to master for Apache Spark 4.1.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org