You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by eu...@gmail.com on 2020/12/03 17:21:14 UTC

Kafka structured straming - how to read headers

Hi folks,
  I am trying to read the message headers from a Kafka structured
stream which should be stored in a column named ``headers``. 
I try something like this:

val stream = sparkSession.readStream.format("kafka")......load()
stream.map(row => { 
 ...
 val headers = row.getAs[HeaderT]("headers")
....
})

My question is: what would be HeaderT?

Thanks in advance

 Eugen

Re: Kafka structured straming - how to read headers

Posted by German Schiavon <gs...@gmail.com>.
Yep I've tried this morning:

[image: Captura de pantalla 2020-12-04 a las 15.42.09.png]

On Fri, 4 Dec 2020 at 13:25, Jungtaek Lim <ka...@gmail.com>
wrote:

> Yeah that would be great, once you manually verified it. Thanks!
>
> On Fri, Dec 4, 2020 at 5:54 PM German Schiavon <gs...@gmail.com>
> wrote:
>
>> I could change it , to something more generic like ?
>>
>> .selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)", "headers")
>>   .as[(String, String, Array[(String, Array[Byte])])]
>>
>>
>> On Fri, 4 Dec 2020 at 09:18, Jungtaek Lim <ka...@gmail.com>
>> wrote:
>>
>>> Probably it's missing - that was Map from the initial proposal and
>>> changed to Array during review. We look to miss updating the doc in sync
>>> during review.
>>>
>>> I'd love to see a volunteer to fix the doc. If there's no volunteer I
>>> might be able to take a look.
>>>
>>> On Fri, Dec 4, 2020 at 4:09 PM German Schiavon <gs...@gmail.com>
>>> wrote:
>>>
>>>> oh, I didn't see that Map, it is weird :/
>>>>
>>>> I did this and it works :
>>>>
>>>> case class Header(key: String, value: Array[Byte])
>>>>
>>>> import spark.implicits._
>>>>   val kafkaDF = spark
>>>>     .readStream
>>>>     .format("kafka")
>>>>     .option("kafka.bootstrap.servers", "localhost:9092")
>>>>     .option("subscribe", "test")
>>>>     .option("includeHeaders", "true")
>>>>     .option("startingOffsets", "earliest")
>>>>     .load()
>>>>
>>>>
>>>> kafkaDF
>>>>   .select('key.cast("string"), 'value.cast("string"), 'headers.as[Header])
>>>>   .writeStream
>>>>   .format("console")
>>>>   .option("truncate", false)
>>>>   .start()
>>>>   .awaitTermination()
>>>>
>>>>
>>>> On Thu, 3 Dec 2020 at 23:11, <eu...@gmail.com> wrote:
>>>>
>>>>> Hi German,
>>>>>   unfortunately the documentation does not help.
>>>>>
>>>>> On Thu, 2020-12-03 at 18:41 +0100, German Schiavon wrote:
>>>>>
>>>>> Hello,
>>>>>
>>>>> see if this works, from the documentation:
>>>>>
>>>>>
>>>>> // Subscribe to 1 topic, with headersval df = spark
>>>>>   .readStream
>>>>>   .format("kafka")
>>>>>   .option("kafka.bootstrap.servers", "host1:port1,host2:port2")
>>>>>   .option("subscribe", "topic1")
>>>>>   .option("includeHeaders", "true")
>>>>>   .load()df.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)", "headers")
>>>>>   .as[(String, String, Map)
>>>>>
>>>>> I am actually wondering what this code is doing since using Map
>>>>> without key and value type is rather strange. And this is actually what I
>>>>> get when I am trying to do this
>>>>>
>>>>>
>>>>> The error messages make totally sense. Now I can try to add the type
>>>>> parameters to Map, assuming that the key is a String type and the value a
>>>>> byte array, in this case I get
>>>>>
>>>>> What seems to work is this.
>>>>>
>>>>> I will try to continue in this direction. I'll let you know about my
>>>>> findings. The documentation is in any case wrong.
>>>>>
>>>>>
>>>>> On Thu, 3 Dec 2020 at 18:22, <eu...@gmail.com> wrote:
>>>>>
>>>>> Hi folks,
>>>>>   I am trying to read the message headers from a Kafka structured
>>>>> stream which should be stored in a column named ``headers``.
>>>>> I try something like this:
>>>>>
>>>>> val stream = sparkSession.readStream.format("kafka")......load()
>>>>>
>>>>> stream.map(row => {
>>>>>
>>>>>  ...
>>>>>
>>>>>  val headers = row.getAs[HeaderT]("headers")
>>>>>
>>>>> ....
>>>>>
>>>>> })
>>>>>
>>>>>
>>>>> My question is: what would be *HeaderT*?
>>>>>
>>>>> Thanks in advance
>>>>>
>>>>>  Eugen
>>>>>
>>>>>
>>>>>

Re: Kafka structured straming - how to read headers

Posted by Jungtaek Lim <ka...@gmail.com>.
Yeah that would be great, once you manually verified it. Thanks!

On Fri, Dec 4, 2020 at 5:54 PM German Schiavon <gs...@gmail.com>
wrote:

> I could change it , to something more generic like ?
>
> .selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)", "headers")
>   .as[(String, String, Array[(String, Array[Byte])])]
>
>
> On Fri, 4 Dec 2020 at 09:18, Jungtaek Lim <ka...@gmail.com>
> wrote:
>
>> Probably it's missing - that was Map from the initial proposal and
>> changed to Array during review. We look to miss updating the doc in sync
>> during review.
>>
>> I'd love to see a volunteer to fix the doc. If there's no volunteer I
>> might be able to take a look.
>>
>> On Fri, Dec 4, 2020 at 4:09 PM German Schiavon <gs...@gmail.com>
>> wrote:
>>
>>> oh, I didn't see that Map, it is weird :/
>>>
>>> I did this and it works :
>>>
>>> case class Header(key: String, value: Array[Byte])
>>>
>>> import spark.implicits._
>>>   val kafkaDF = spark
>>>     .readStream
>>>     .format("kafka")
>>>     .option("kafka.bootstrap.servers", "localhost:9092")
>>>     .option("subscribe", "test")
>>>     .option("includeHeaders", "true")
>>>     .option("startingOffsets", "earliest")
>>>     .load()
>>>
>>>
>>> kafkaDF
>>>   .select('key.cast("string"), 'value.cast("string"), 'headers.as[Header])
>>>   .writeStream
>>>   .format("console")
>>>   .option("truncate", false)
>>>   .start()
>>>   .awaitTermination()
>>>
>>>
>>> On Thu, 3 Dec 2020 at 23:11, <eu...@gmail.com> wrote:
>>>
>>>> Hi German,
>>>>   unfortunately the documentation does not help.
>>>>
>>>> On Thu, 2020-12-03 at 18:41 +0100, German Schiavon wrote:
>>>>
>>>> Hello,
>>>>
>>>> see if this works, from the documentation:
>>>>
>>>>
>>>> // Subscribe to 1 topic, with headersval df = spark
>>>>   .readStream
>>>>   .format("kafka")
>>>>   .option("kafka.bootstrap.servers", "host1:port1,host2:port2")
>>>>   .option("subscribe", "topic1")
>>>>   .option("includeHeaders", "true")
>>>>   .load()df.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)", "headers")
>>>>   .as[(String, String, Map)
>>>>
>>>> I am actually wondering what this code is doing since using Map without
>>>> key and value type is rather strange. And this is actually what I get when
>>>> I am trying to do this
>>>>
>>>>
>>>> The error messages make totally sense. Now I can try to add the type
>>>> parameters to Map, assuming that the key is a String type and the value a
>>>> byte array, in this case I get
>>>>
>>>> What seems to work is this.
>>>>
>>>> I will try to continue in this direction. I'll let you know about my
>>>> findings. The documentation is in any case wrong.
>>>>
>>>>
>>>> On Thu, 3 Dec 2020 at 18:22, <eu...@gmail.com> wrote:
>>>>
>>>> Hi folks,
>>>>   I am trying to read the message headers from a Kafka structured
>>>> stream which should be stored in a column named ``headers``.
>>>> I try something like this:
>>>>
>>>> val stream = sparkSession.readStream.format("kafka")......load()
>>>>
>>>> stream.map(row => {
>>>>
>>>>  ...
>>>>
>>>>  val headers = row.getAs[HeaderT]("headers")
>>>>
>>>> ....
>>>>
>>>> })
>>>>
>>>>
>>>> My question is: what would be *HeaderT*?
>>>>
>>>> Thanks in advance
>>>>
>>>>  Eugen
>>>>
>>>>
>>>>

Re: Kafka structured straming - how to read headers

Posted by German Schiavon <gs...@gmail.com>.
I could change it , to something more generic like ?

.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)", "headers")
  .as[(String, String, Array[(String, Array[Byte])])]


On Fri, 4 Dec 2020 at 09:18, Jungtaek Lim <ka...@gmail.com>
wrote:

> Probably it's missing - that was Map from the initial proposal and changed
> to Array during review. We look to miss updating the doc in sync during
> review.
>
> I'd love to see a volunteer to fix the doc. If there's no volunteer I
> might be able to take a look.
>
> On Fri, Dec 4, 2020 at 4:09 PM German Schiavon <gs...@gmail.com>
> wrote:
>
>> oh, I didn't see that Map, it is weird :/
>>
>> I did this and it works :
>>
>> case class Header(key: String, value: Array[Byte])
>>
>> import spark.implicits._
>>   val kafkaDF = spark
>>     .readStream
>>     .format("kafka")
>>     .option("kafka.bootstrap.servers", "localhost:9092")
>>     .option("subscribe", "test")
>>     .option("includeHeaders", "true")
>>     .option("startingOffsets", "earliest")
>>     .load()
>>
>>
>> kafkaDF
>>   .select('key.cast("string"), 'value.cast("string"), 'headers.as[Header])
>>   .writeStream
>>   .format("console")
>>   .option("truncate", false)
>>   .start()
>>   .awaitTermination()
>>
>>
>> On Thu, 3 Dec 2020 at 23:11, <eu...@gmail.com> wrote:
>>
>>> Hi German,
>>>   unfortunately the documentation does not help.
>>>
>>> On Thu, 2020-12-03 at 18:41 +0100, German Schiavon wrote:
>>>
>>> Hello,
>>>
>>> see if this works, from the documentation:
>>>
>>>
>>> // Subscribe to 1 topic, with headersval df = spark
>>>   .readStream
>>>   .format("kafka")
>>>   .option("kafka.bootstrap.servers", "host1:port1,host2:port2")
>>>   .option("subscribe", "topic1")
>>>   .option("includeHeaders", "true")
>>>   .load()df.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)", "headers")
>>>   .as[(String, String, Map)
>>>
>>> I am actually wondering what this code is doing since using Map without
>>> key and value type is rather strange. And this is actually what I get when
>>> I am trying to do this
>>>
>>>
>>> The error messages make totally sense. Now I can try to add the type
>>> parameters to Map, assuming that the key is a String type and the value a
>>> byte array, in this case I get
>>>
>>> What seems to work is this.
>>>
>>> I will try to continue in this direction. I'll let you know about my
>>> findings. The documentation is in any case wrong.
>>>
>>>
>>> On Thu, 3 Dec 2020 at 18:22, <eu...@gmail.com> wrote:
>>>
>>> Hi folks,
>>>   I am trying to read the message headers from a Kafka structured stream
>>> which should be stored in a column named ``headers``.
>>> I try something like this:
>>>
>>> val stream = sparkSession.readStream.format("kafka")......load()
>>>
>>> stream.map(row => {
>>>
>>>  ...
>>>
>>>  val headers = row.getAs[HeaderT]("headers")
>>>
>>> ....
>>>
>>> })
>>>
>>>
>>> My question is: what would be *HeaderT*?
>>>
>>> Thanks in advance
>>>
>>>  Eugen
>>>
>>>
>>>

Re: Kafka structured straming - how to read headers

Posted by Jungtaek Lim <ka...@gmail.com>.
Probably it's missing - that was Map from the initial proposal and changed
to Array during review. We look to miss updating the doc in sync during
review.

I'd love to see a volunteer to fix the doc. If there's no volunteer I might
be able to take a look.

On Fri, Dec 4, 2020 at 4:09 PM German Schiavon <gs...@gmail.com>
wrote:

> oh, I didn't see that Map, it is weird :/
>
> I did this and it works :
>
> case class Header(key: String, value: Array[Byte])
>
> import spark.implicits._
>   val kafkaDF = spark
>     .readStream
>     .format("kafka")
>     .option("kafka.bootstrap.servers", "localhost:9092")
>     .option("subscribe", "test")
>     .option("includeHeaders", "true")
>     .option("startingOffsets", "earliest")
>     .load()
>
>
> kafkaDF
>   .select('key.cast("string"), 'value.cast("string"), 'headers.as[Header])
>   .writeStream
>   .format("console")
>   .option("truncate", false)
>   .start()
>   .awaitTermination()
>
>
> On Thu, 3 Dec 2020 at 23:11, <eu...@gmail.com> wrote:
>
>> Hi German,
>>   unfortunately the documentation does not help.
>>
>> On Thu, 2020-12-03 at 18:41 +0100, German Schiavon wrote:
>>
>> Hello,
>>
>> see if this works, from the documentation:
>>
>>
>> // Subscribe to 1 topic, with headersval df = spark
>>   .readStream
>>   .format("kafka")
>>   .option("kafka.bootstrap.servers", "host1:port1,host2:port2")
>>   .option("subscribe", "topic1")
>>   .option("includeHeaders", "true")
>>   .load()df.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)", "headers")
>>   .as[(String, String, Map)
>>
>> I am actually wondering what this code is doing since using Map without
>> key and value type is rather strange. And this is actually what I get when
>> I am trying to do this
>>
>>
>> The error messages make totally sense. Now I can try to add the type
>> parameters to Map, assuming that the key is a String type and the value a
>> byte array, in this case I get
>>
>> What seems to work is this.
>>
>> I will try to continue in this direction. I'll let you know about my
>> findings. The documentation is in any case wrong.
>>
>>
>> On Thu, 3 Dec 2020 at 18:22, <eu...@gmail.com> wrote:
>>
>> Hi folks,
>>   I am trying to read the message headers from a Kafka structured stream
>> which should be stored in a column named ``headers``.
>> I try something like this:
>>
>> val stream = sparkSession.readStream.format("kafka")......load()
>>
>> stream.map(row => {
>>
>>  ...
>>
>>  val headers = row.getAs[HeaderT]("headers")
>>
>> ....
>>
>> })
>>
>>
>> My question is: what would be *HeaderT*?
>>
>> Thanks in advance
>>
>>  Eugen
>>
>>
>>

Re: Kafka structured straming - how to read headers

Posted by German Schiavon <gs...@gmail.com>.
oh, I didn't see that Map, it is weird :/

I did this and it works :

case class Header(key: String, value: Array[Byte])

import spark.implicits._
  val kafkaDF = spark
    .readStream
    .format("kafka")
    .option("kafka.bootstrap.servers", "localhost:9092")
    .option("subscribe", "test")
    .option("includeHeaders", "true")
    .option("startingOffsets", "earliest")
    .load()


kafkaDF
  .select('key.cast("string"), 'value.cast("string"), 'headers.as[Header])
  .writeStream
  .format("console")
  .option("truncate", false)
  .start()
  .awaitTermination()


On Thu, 3 Dec 2020 at 23:11, <eu...@gmail.com> wrote:

> Hi German,
>   unfortunately the documentation does not help.
>
> On Thu, 2020-12-03 at 18:41 +0100, German Schiavon wrote:
>
> Hello,
>
> see if this works, from the documentation:
>
>
> // Subscribe to 1 topic, with headersval df = spark
>   .readStream
>   .format("kafka")
>   .option("kafka.bootstrap.servers", "host1:port1,host2:port2")
>   .option("subscribe", "topic1")
>   .option("includeHeaders", "true")
>   .load()df.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)", "headers")
>   .as[(String, String, Map)
>
> I am actually wondering what this code is doing since using Map without
> key and value type is rather strange. And this is actually what I get when
> I am trying to do this
>
>
> The error messages make totally sense. Now I can try to add the type
> parameters to Map, assuming that the key is a String type and the value a
> byte array, in this case I get
>
> What seems to work is this.
>
> I will try to continue in this direction. I'll let you know about my
> findings. The documentation is in any case wrong.
>
>
> On Thu, 3 Dec 2020 at 18:22, <eu...@gmail.com> wrote:
>
> Hi folks,
>   I am trying to read the message headers from a Kafka structured stream
> which should be stored in a column named ``headers``.
> I try something like this:
>
> val stream = sparkSession.readStream.format("kafka")......load()
>
> stream.map(row => {
>
>  ...
>
>  val headers = row.getAs[HeaderT]("headers")
>
> ....
>
> })
>
>
> My question is: what would be *HeaderT*?
>
> Thanks in advance
>
>  Eugen
>
>
>

Re: Kafka structured straming - how to read headers

Posted by eu...@gmail.com.
Hi German,
  unfortunately the documentation does not help. 

On Thu, 2020-12-03 at 18:41 +0100, German Schiavon wrote:
> Hello, 
> see if this works, from the documentation:
> 
> 
> // Subscribe to 1 topic, with headers
> val df = spark
> .readStream
> .format("kafka")
> .option("kafka.bootstrap.servers", "host1:port1,host2:port2")
> .option("subscribe", "topic1")
> .option("includeHeaders", "true")
> .load()
> df.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)",
> "headers")
> .as[(String, String, Map)
I am actually wondering what this code is doing since using Map without
key and value type is rather strange. And this is actually what I get
when I am trying to do this


The error messages make totally sense. Now I can try to add the type
parameters to Map, assuming that the key is a String type and the value
a byte array, in this case I get

What seems to work is this.

I will try to continue in this direction. I'll let you know about my
findings. The documentation is in any case wrong.


> On Thu, 3 Dec 2020 at 18:22, <eu...@gmail.com> wrote:
> > Hi folks,
> >   I am trying to read the message headers from a Kafka structured
> > stream which should be stored in a column named ``headers``. 
> > I try something like this:
> > 
> > val stream = sparkSession.readStream.format("kafka")......load()
> > stream.map(row => { 
> >  ...
> >  val headers = row.getAs[HeaderT]("headers")
> > ....
> > })
> > 
> > My question is: what would be HeaderT?
> > 
> > Thanks in advance
> > 
> >  Eugen


Re: Kafka structured straming - how to read headers

Posted by German Schiavon <gs...@gmail.com>.
Hello,

see if this works, from the documentation:


// Subscribe to 1 topic, with headersval df = spark
  .readStream
  .format("kafka")
  .option("kafka.bootstrap.servers", "host1:port1,host2:port2")
  .option("subscribe", "topic1")
  .option("includeHeaders", "true")
  .load()df.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)",
"headers")
  .as[(String, String, Map)]


On Thu, 3 Dec 2020 at 18:22, <eu...@gmail.com> wrote:

> Hi folks,
>   I am trying to read the message headers from a Kafka structured stream
> which should be stored in a column named ``headers``.
> I try something like this:
>
> val stream = sparkSession.readStream.format("kafka")......load()
>
> stream.map(row => {
>
>  ...
>
>  val headers = row.getAs[HeaderT]("headers")
>
> ....
>
> })
>
>
> My question is: what would be *HeaderT*?
>
> Thanks in advance
>
>  Eugen
>