You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Rajesh (Jira)" <ji...@apache.org> on 2023/04/05 13:17:00 UTC
[jira] [Updated] (SPARK-43036) Spark structuring streaming with kinesis and rocksdb integration
[ https://issues.apache.org/jira/browse/SPARK-43036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rajesh updated SPARK-43036:
---------------------------
Description:
Hi Spark Team,
We need to read/write the kinesis streams using spark streaming.
We checked the official documentation - [https://spark.apache.org/docs/latest/streaming-kinesis-integration.html]
It does not mention about kinesis connector. Alternative is - [https://github.com/qubole/kinesis-sql] which is not active now. This is now handed over here - [https://github.com/roncemer/spark-sql-kinesis]
We have few below questions , It would be great if you can answer
# Does Spark provides officially any kinesis connector which have readstream/writestream and endorse any connector for production use cases ?
# [https://spark.apache.org/docs/latest/streaming-kinesis-integration.html] This documentation does not mention about how to write to kinesis. This method have default dynamodb as checkpoint, can we override it ?
# We have rocksdb as state store but when we ran application using official [https://spark.apache.org/docs/latest/streaming-kinesis-integration.html] rocksdb configurations were not effective. Can you please confirm if rocksdb is not applicable in these cases?
# rocksdb however works with qubole connector , do you have any plan to release kinesis connector?
# Please help/recommend us for any good stable kinesis connector or some pointers around it
> Spark structuring streaming with kinesis and rocksdb integration
> ----------------------------------------------------------------
>
> Key: SPARK-43036
> URL: https://issues.apache.org/jira/browse/SPARK-43036
> Project: Spark
> Issue Type: Question
> Components: Structured Streaming
> Affects Versions: 3.3.1
> Reporter: Rajesh
> Priority: Blocker
>
> Hi Spark Team,
> We need to read/write the kinesis streams using spark streaming.
> We checked the official documentation - [https://spark.apache.org/docs/latest/streaming-kinesis-integration.html]
> It does not mention about kinesis connector. Alternative is - [https://github.com/qubole/kinesis-sql] which is not active now. This is now handed over here - [https://github.com/roncemer/spark-sql-kinesis]
> We have few below questions , It would be great if you can answer
> # Does Spark provides officially any kinesis connector which have readstream/writestream and endorse any connector for production use cases ?
> # [https://spark.apache.org/docs/latest/streaming-kinesis-integration.html] This documentation does not mention about how to write to kinesis. This method have default dynamodb as checkpoint, can we override it ?
> # We have rocksdb as state store but when we ran application using official [https://spark.apache.org/docs/latest/streaming-kinesis-integration.html] rocksdb configurations were not effective. Can you please confirm if rocksdb is not applicable in these cases?
> # rocksdb however works with qubole connector , do you have any plan to release kinesis connector?
> # Please help/recommend us for any good stable kinesis connector or some pointers around it
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org