Event Hubs properties kvp-value adds " to strings

[Magnus Nilsson <ma...@kth.se>]

Hello all,

I have this peculiar problem where quote " characters are added to the
beginning and end of my string values.

I get data using Structured Streaming from an Azure Event Hub using a Scala
Notebook in Azure Databricks.

The Dataframe schema received contain a property of type Map named
"properties" containing string/string key value pairs. Only one pair in my
case.

I input data from a C# program where the "properties" property in the
EventData schema is a Dictionary<string, object> where I input a
string/string property pair.

If I input a string value of _Hello_ in C# I get back a string value of
_"Hello"_ in my Dataframe.

In the body tag of the Schema I input a UTF-8 byte[] and get back a UTF-8
byte[] which I cast to string, this turns out correct, no " are added to
the string value.

If I try to pass in a UTF-8 string as a byte[] as the value of the KvP I
get a serialization exception so that's a no go.

Any idea on why or where the quote characters "" are added to the value of
the string when I read them in Spark?

Any ideas would greatly be appreciated.

//Magnus