You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Kiran Biswal <bi...@gmail.com> on 2022/04/05 21:43:19 UTC
protobuf data as input to spark streaming
Hello Experts
Has anyone used protobuf (proto3) encoded data (from kafka) as input source
and been able to do spark structured streaming?
I would appreciate if you can share any sample code/example
Regards
Kiran
>
Re: protobuf data as input to spark streaming
Posted by Kiran Biswal <bi...@gmail.com>.
Hello Stelios, friendly reminder if you could share any sample code/repo
Are you using a schema registry?
Thanks
Kiran
On Fri, Apr 8, 2022 at 4:37 PM Kiran Biswal <bi...@gmail.com> wrote:
> Hello Stelios
>
> Just a gentle follow up if you can share any sample code/repo
>
> Regards
> Kiran
>
> On Wed, Apr 6, 2022 at 3:19 PM Kiran Biswal <bi...@gmail.com> wrote:
>
>> Hello Stelios
>>
>> Preferred language would have been Scala or pyspark but if Java is proven
>> I am open to using it
>>
>> Any sample reference or example code link?
>>
>> How are you handling the peotobuf to spark dataframe conversion
>> (serialization federalization)?
>>
>> Thanks
>> Kiran
>>
>> On Wed, Apr 6, 2022, 2:38 PM Stelios Philippou <st...@gmail.com>
>> wrote:
>>
>>> Yes we are currently using it as such.
>>> Code is in java. Will that work?
>>>
>>> On Wed, 6 Apr 2022 at 00:51, Kiran Biswal <bi...@gmail.com> wrote:
>>>
>>>> Hello Experts
>>>>
>>>> Has anyone used protobuf (proto3) encoded data (from kafka) as input
>>>> source and been able to do spark structured streaming?
>>>>
>>>> I would appreciate if you can share any sample code/example
>>>>
>>>> Regards
>>>> Kiran
>>>>
>>>>>
Re: protobuf data as input to spark streaming
Posted by Kiran Biswal <bi...@gmail.com>.
Hello Stelios
Just a gentle follow up if you can share any sample code/repo
Regards
Kiran
On Wed, Apr 6, 2022 at 3:19 PM Kiran Biswal <bi...@gmail.com> wrote:
> Hello Stelios
>
> Preferred language would have been Scala or pyspark but if Java is proven
> I am open to using it
>
> Any sample reference or example code link?
>
> How are you handling the peotobuf to spark dataframe conversion
> (serialization federalization)?
>
> Thanks
> Kiran
>
> On Wed, Apr 6, 2022, 2:38 PM Stelios Philippou <st...@gmail.com> wrote:
>
>> Yes we are currently using it as such.
>> Code is in java. Will that work?
>>
>> On Wed, 6 Apr 2022 at 00:51, Kiran Biswal <bi...@gmail.com> wrote:
>>
>>> Hello Experts
>>>
>>> Has anyone used protobuf (proto3) encoded data (from kafka) as input
>>> source and been able to do spark structured streaming?
>>>
>>> I would appreciate if you can share any sample code/example
>>>
>>> Regards
>>> Kiran
>>>
>>>>
Re: protobuf data as input to spark streaming
Posted by Kiran Biswal <bi...@gmail.com>.
Hello Stelios
Preferred language would have been Scala or pyspark but if Java is proven I
am open to using it
Any sample reference or example code link?
How are you handling the peotobuf to spark dataframe conversion
(serialization federalization)?
Thanks
Kiran
On Wed, Apr 6, 2022, 2:38 PM Stelios Philippou <st...@gmail.com> wrote:
> Yes we are currently using it as such.
> Code is in java. Will that work?
>
> On Wed, 6 Apr 2022 at 00:51, Kiran Biswal <bi...@gmail.com> wrote:
>
>> Hello Experts
>>
>> Has anyone used protobuf (proto3) encoded data (from kafka) as input
>> source and been able to do spark structured streaming?
>>
>> I would appreciate if you can share any sample code/example
>>
>> Regards
>> Kiran
>>
>>>
Re: protobuf data as input to spark streaming
Posted by Stelios Philippou <st...@gmail.com>.
Yes we are currently using it as such.
Code is in java. Will that work?
On Wed, 6 Apr 2022 at 00:51, Kiran Biswal <bi...@gmail.com> wrote:
> Hello Experts
>
> Has anyone used protobuf (proto3) encoded data (from kafka) as input
> source and been able to do spark structured streaming?
>
> I would appreciate if you can share any sample code/example
>
> Regards
> Kiran
>
>>