You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Akash Vellukai <ak...@gmail.com> on 2022/08/17 17:40:32 UTC
Data ingestion
Dear Sir,
How we could do data ingestion from MySQL to Hive with the help of Spark
streaming and not with Kafka
Thanks and regards
Akash
Re: Data ingestion
Posted by Pasha Finkelshtein <pa...@gmail.com>.
But not in streaming, right? It will be a usual batch approach, but initial
question was about streaming.
[image: facebook] <https://fb.com/asm0dey>
[image: twitter] <https://twitter.com/asm0di0>
[image: linkedin] <https://linkedin.com/in/asm0dey>
[image: instagram] <https://instagram.com/asm0dey>
Pasha Finkelshteyn
Developer Advocate for Data Engineering
JetBrains
asm0dey@jetbrains.com
https://linktr.ee/asm0dey
Find out more <https://jetbrains.com>
чт, 18 авг. 2022 г. в 03:12, pengyh <pe...@mail.de>:
> from my experience, spark can read/write from/to both mysql and hive
> fluently.
>
> regards.
>
>
> Akash Vellukai wrote:
> > How we could do data ingestion from MySQL to Hive with the help of Spark
> > streaming and not with Kafka
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>
Re: Data ingestion
Posted by pengyh <pe...@mail.de>.
from my experience, spark can read/write from/to both mysql and hive
fluently.
regards.
Akash Vellukai wrote:
> How we could do data ingestion from MySQL to Hive with the help of Spark
> streaming and not with Kafka
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: Data ingestion
Posted by "\"Yuri Oleynikov (יורי אולייניקו ב)\"" <yu...@gmail.com>.
If you are on aws, you can use RDS + AWS DMS to save data to s3 and then read streaming data with spark structured streaming from s3 into hive
Best regards
> On 17 Aug 2022, at 20:51, Akash Vellukai <ak...@gmail.com> wrote:
>
>
> Dear Sir,
>
>
> How we could do data ingestion from MySQL to Hive with the help of Spark streaming and not with Kafka
>
> Thanks and regards
> Akash
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: Data ingestion
Posted by Pasha Finkelshtein <pa...@gmail.com>.
Hello
Spark does not have any built-in solution for this problem. Most probably
you will want to use Debezium+Kafka and read with Spark from Kafka
[image: facebook] <https://fb.com/asm0dey>
[image: twitter] <https://twitter.com/asm0di0>
[image: linkedin] <https://linkedin.com/in/asm0dey>
[image: instagram] <https://instagram.com/asm0dey>
Pasha Finkelshteyn
Developer Advocate for Data Engineering
JetBrains
asm0dey@jetbrains.com
https://linktr.ee/asm0dey
Find out more <https://jetbrains.com>
ср, 17 авг. 2022 г. в 19:51, Akash Vellukai <ak...@gmail.com>:
> Dear Sir,
>
>
> How we could do data ingestion from MySQL to Hive with the help of Spark
> streaming and not with Kafka
>
> Thanks and regards
> Akash
>