You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "LuciferYang (via GitHub)" <gi...@apache.org> on 2025/03/03 03:25:05 UTC
[PR] Revert "[SPARK-51309][BUILD] Upgrade rocksdbjni to 9.10.0" [spark]
LuciferYang opened a new pull request, #50127:
URL: https://github.com/apache/spark/pull/50127
### What changes were proposed in this pull request?
This pr reverts commit 971b9a40c42a1d8c050941ce8088fdfc6ac442bc.
### Why are the changes needed?
A crash related to RocksDB has occurred in the macOS daily test after upgrade to 9.10.0:
- https://github.com/apache/spark/actions/runs/13608353684/job/38042505327
- https://github.com/apache/spark/actions/runs/13464386260/job/37626883152
```
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x0000000137105814, pid=32170, tid=707595
#
# JRE version: OpenJDK Runtime Environment Zulu21.40+17-CRaC-CA (21.0.6+7) (build 21.0.6+7-LTS)
# Java VM: OpenJDK 64-Bit Server VM Zulu21.40+17-CRaC-CA (21.0.6+7-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, bsd-aarch64)
# Problematic frame:
# C [librocksdbjni16177160674801731666.jnilib+0x1d814] Java_org_rocksdb_FlushOptions_disposeInternalJni+0x360
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /Users/runner/work/spark/spark/sql/core/hs_err_pid32170.log
21:02:35.349 ERROR org.apache.spark.util.Utils: Aborting task
org.apache.spark.SparkException: [CANNOT_WRITE_STATE_STORE.CANNOT_COMMIT] Error writing state store files for provider RocksDBStateStore[id=(op=0,part=1),dir=file:/Users/runner/work/spark/spark/sql/core/target/tmp/streaming.metadata-aa0706db-1b92-485b-b2e6-f356f91ffda6/state/0/1]. Cannot perform commit during state checkpoint. SQLSTATE: 58030
at org.apache.spark.sql.errors.QueryExecutionErrors$.failedToCommitStateFileError(QueryExecutionErrors.scala:2203)
at org.apache.spark.sql.execution.streaming.state.RocksDBStateStoreProvider$RocksDBStateStore.commit(RocksDBStateStoreProvider.scala:243)
at org.apache.spark.sql.execution.streaming.TransformWithStateExec.$anonfun$processDataWithPartition$5(TransformWithStateExec.scala:431)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
at org.apache.spark.util.Utils$.timeTakenMs(Utils.scala:481)
at org.apache.spark.sql.execution.streaming.StateStoreWriter.timeTakenMs(statefulOperators.scala:384)
at org.apache.spark.sql.execution.streaming.StateStoreWriter.timeTakenMs$(statefulOperators.scala:384)
21:02:35.351 ERROR org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask: Aborting commit for partition 1 (task 6, attempt 0, stage 3.0)
21:02:35.351 ERROR org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask: Aborted commit for partition 1 (task 6, attempt 0, stage 3.0)
- transformWithState - streaming with rocksdb and invalid processor should fail (encoding = Avro) (without changelog checkpointing)
21:02:35.352 WARN org.apache.spark.scheduler.TaskSetManager: Lost task 1.0 in stage 3.0 (TID 6) (localhost executor driver): TaskKilled (Stage cancelled: [STATEFUL_PROCESSOR_CANNOT_PERFORM_OPERATION_WITH_INVALID_HANDLE_STATE] Failed to perform stateful processor operation=get_value_state with invalid handle state=INITIALIZED. SQLSTATE: 42802)
21:02:35.372 WARN org.apache.spark.sql.execution.streaming.ResolveWriteToStream: spark.sql.adaptive.enabled is not supported in streaming DataFrames/Datasets and will be disabled.
Warning: [1237.706s][warning][os] Loading hsdis library failed
#
# If you would like to submit a bug report, please visit:
# http://www.azul.com/support/
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
```
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Pass GitHub Actions
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] Revert "[SPARK-51309][BUILD] Upgrade rocksdbjni to 9.10.0" [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #50127:
URL: https://github.com/apache/spark/pull/50127#issuecomment-2693191130
cc @wayneguow @yaooqinn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] Revert "[SPARK-51309][BUILD] Upgrade rocksdbjni to 9.10.0" [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang closed pull request #50127: Revert "[SPARK-51309][BUILD] Upgrade rocksdbjni to 9.10.0"
URL: https://github.com/apache/spark/pull/50127
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] Revert "[SPARK-51309][BUILD] Upgrade rocksdbjni to 9.10.0" [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #50127:
URL: https://github.com/apache/spark/pull/50127#issuecomment-2693193103
Thanks @yaooqinn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org