You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by 萝卜丝炒饭 <14...@qq.com> on 2017/06/18 07:27:49 UTC
the scheme in stream reader
Hi all,
L set the scheme for DataStreamReader but when I print the scheme.It just printed:
root
|--value:string (nullable=true)
My code is
val line = ss.readStream.format("socket")
.option("ip",xxx)
.option("port",xxx)
.scheme(StructField("name",StringType）::(StructField("age", IntegerType))).load
line.printSchema
My spark version is 2.1.0.
I want the printSchema prints the schema I set in the code.How should I do please?
And my original target is the received data from socket is handled as schema directly.What should I do please?
thanks
Fei Shao
Re: the scheme in stream reader
Posted by Michael Armbrust <mi...@databricks.com>.
The socket source can't know how to parse your data. I think the right
thing would be for it to throw an exception saying that you can't set the
schema here. Would you mind opening a JIRA ticket?
If you are trying to parse data from something like JSON then you should
use from_json` on the value returned.
On Sun, Jun 18, 2017 at 12:27 AM, 萝卜丝炒饭 <14...@qq.com> wrote:
> Hi all,
>
> L set the scheme for DataStreamReader but when I print the scheme.It just
> printed:
> root
> |--value:string (nullable=true)
>
> My code is
>
> val line = ss.readStream.format("socket")
> .option("ip",xxx)
> .option("port",xxx)
> .scheme(StructField("name",StringType）::(StructField("age",
> IntegerType))).load
> line.printSchema
>
> My spark version is 2.1.0.
> I want the printSchema prints the schema I set in the code.How should I do
> please?
> And my original target is the received data from socket is handled as
> schema directly.What should I do please?
>
> thanks
> Fei Shao
>
>
>
>
>
>
>
Re: the scheme in stream reader
Posted by Michael Armbrust <mi...@databricks.com>.
The socket source can't know how to parse your data. I think the right
thing would be for it to throw an exception saying that you can't set the
schema here. Would you mind opening a JIRA ticket?
If you are trying to parse data from something like JSON then you should
use from_json` on the value returned.
On Sun, Jun 18, 2017 at 12:27 AM, 萝卜丝炒饭 <14...@qq.com> wrote:
> Hi all,
>
> L set the scheme for DataStreamReader but when I print the scheme.It just
> printed:
> root
> |--value:string (nullable=true)
>
> My code is
>
> val line = ss.readStream.format("socket")
> .option("ip",xxx)
> .option("port",xxx)
> .scheme(StructField("name",StringType）::(StructField("age",
> IntegerType))).load
> line.printSchema
>
> My spark version is 2.1.0.
> I want the printSchema prints the schema I set in the code.How should I do
> please?
> And my original target is the received data from socket is handled as
> schema directly.What should I do please?
>
> thanks
> Fei Shao
>
>
>
>
>
>
>