You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/02/09 14:35:28 UTC
[GitHub] [spark] kuwii opened a new pull request #35463: [SPARK-38056][Web UI] Fix issue of Structured streaming not working in history server when using LevelDB
kuwii opened a new pull request #35463:
URL: https://github.com/apache/spark/pull/35463
### What changes were proposed in this pull request?
Change type of `org.apache.spark.sql.streaming.ui.StreamingQueryData.runId` from `UUID` to `String`.
### Why are the changes needed?
In [SPARK-31953](https://github.com/apache/spark/commit/4f9667035886a67e6c9a4e8fad2efa390e87ca68), structured streaming support is added in history server. However this does not work when history server is using LevelDB instead of in-memory KV store.
- Level DB does not support `UUID` as key.
- If `spark.history.store.path` is set in history server to use Level DB, when writing info to the store during replaying events, error will occur.
- `StreamingQueryStatusListener` will throw exceptions when writing info, saying `java.lang.IllegalArgumentException: Type java.util.UUID not allowed as key.`.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Added tests in `StreamingQueryStatusListenerSuite` to test whether `StreamingQueryData` can be successfully written to in-memory store and LevelDB.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #35463: [SPARK-38056][Web UI][3.1] Fix issue of Structured streaming not working in history server when using LevelDB
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #35463:
URL: https://github.com/apache/spark/pull/35463
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org