You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by marc nicole <mk...@gmail.com> on 2022/06/04 12:38:51 UTC
How to convert a Dataset to a Dataset?
Hi,
How to convert a Dataset<Row> to a Dataset<String>?
What i have tried is:
List<String> list = dataset.as(Encoders.STRING()).collectAsList();
Dataset<String> datasetSt = spark.createDataset(list, Encoders.STRING());
// But this line raises a org.apache.spark.sql.AnalysisException: Try to
map struct... to Tuple1, but failed as the number of fields does not line
up
Type of columns being String
How to solve this?
Re: How to convert a Dataset to a Dataset?
Posted by Stelios Philippou <st...@gmail.com>.
Hi All,
Simple in Java as well.
You can get the Dataset<Row> Directly
Dataset<Row> encodedString = df.select("Column")
.where("<Any Condition you want>")
.as(Encoders.STRING())
.toDF();
On Mon, 6 Jun 2022 at 15:26, Christophe Préaud <
christophe.preaud@kelkoogroup.com> wrote:
> Hi Marc,
>
> I'm not much familiar with Spark on Java, but according to the doc
> <https://spark.apache.org/docs/latest/sql-getting-started.html#creating-datasets>,
> it should be:
> Encoder<String> stringEncoder = Encoders.STRING();
> dataset.as(stringEncoder);
>
> For the record, it is much simpler in Scala:
> dataset.as[String]
>
> Of course, this will work if your DataFrame only contains one column of
> type String, e.g.:
> val df = spark.read.parquet("Cyrano_de_Bergerac_Acte_V.parquet")
> df.printSchema
>
> root
> |-- line: string (nullable = true)
>
> df.as[String]
>
> Otherwise, you will have to convert somehow the Row to a String, e.g. in
> Scala:
> case class Data(f1: String, f2: Int, f3: Long)
> val df = Seq(Data("a", 1, 1L), Data("b", 2, 2L), Data("c", 3, 3L),
> Data("d", 4, 4L), Data("e", 5, 5L)).toDF
> val ds = df.map(_.mkString(",")).as[String]
> ds.show
>
> +-----+
> |value|
> +-----+
> |a,1,1|
> |b,2,2|
> |c,3,3|
> |d,4,4|
> |e,5,5|
> +-----+
>
> Regards,
> Christophe.
>
> On 6/4/22 14:38, marc nicole wrote:
>
> Hi,
> How to convert a Dataset<Row> to a Dataset<String>?
> What i have tried is:
>
> List<String> list = dataset.as(Encoders.STRING()).collectAsList();
> Dataset<String> datasetSt = spark.createDataset(list, Encoders.STRING());
> // But this line raises a org.apache.spark.sql.AnalysisException: Try to
> map struct... to Tuple1, but failed as the number of fields does not line
> up
>
> Type of columns being String
> How to solve this?
>
>
>
Re: How to convert a Dataset to a Dataset?
Posted by Christophe Préaud <ch...@kelkoogroup.com>.
Hi Marc,
I'm not much familiar with Spark on Java, but according to the doc <https://spark.apache.org/docs/latest/sql-getting-started.html#creating-datasets>, it should be:
Encoder<String> stringEncoder = Encoders.STRING();
dataset.as(stringEncoder);
For the record, it is much simpler in Scala:
dataset.as[String]
Of course, this will work if your DataFrame only contains one column of type String, e.g.:
val df = spark.read.parquet("Cyrano_de_Bergerac_Acte_V.parquet")
df.printSchema
root
|-- line: string (nullable = true)
df.as[String]
Otherwise, you will have to convert somehow the Row to a String, e.g. in Scala:
case class Data(f1: String, f2: Int, f3: Long)
val df = Seq(Data("a", 1, 1L), Data("b", 2, 2L), Data("c", 3, 3L), Data("d", 4, 4L), Data("e", 5, 5L)).toDF
val ds = df.map(_.mkString(",")).as[String]
ds.show
+-----+
|value|
+-----+
|a,1,1|
|b,2,2|
|c,3,3|
|d,4,4|
|e,5,5|
+-----+
Regards,
Christophe.
On 6/4/22 14:38, marc nicole wrote:
> Hi,
> How to convert a Dataset<Row> to a Dataset<String>?
> What i have tried is:
>
> List<String> list = dataset.as <http://dataset.as>(Encoders.STRING()).collectAsList(); Dataset<String> datasetSt = spark.createDataset(list, Encoders.STRING()); // But this line raises a org.apache.spark.sql.AnalysisException: Try to map struct... to Tuple1, but failed as the number of fields does not line up
>
> Type of columns being String
> How to solve this?
Re: How to convert a Dataset to a Dataset?
Posted by marc nicole <mk...@gmail.com>.
Yes, Thanks Enrico, that was greatly helpful!
To note that i was looking at some similar option at the docs but couldn't
stumble on one.
Thanks.
Le sam. 4 juin 2022 à 19:29, Enrico Minack <in...@enrico.minack.dev> a
écrit :
> You could use .option("nullValue", "+") to tell the parser that '+' refers
> to "no value":
>
> spark.read
> .option("inferSchema", "true")
> .option("header", "true")
> .option("nullvalue", "+")
> .csv("path")
>
> Enrico
>
>
> Am 04.06.22 um 18:54 schrieb marc nicole:
>
> c1
>
> c2
>
> c3
>
> c4
>
> c5
>
> c6
>
> 1.2
>
> true
>
> A
>
> Z
>
> 120
>
> +
>
> 1.3
>
> false
>
> B
>
> X
>
> 130
>
> F
>
> +
>
> true
>
> C
>
> Y
>
> 200
>
> G
> in the above table c1 has double values except on the last row so:
>
> Dataset<Row> dataset =
> spark.read().format("csv")..option("inferSchema","true").option("header","true").load("path");
> will yield StringType as a type for column c1 similarly for c6
> I want to return the true type of each column by first discarding the "+"
> I use Dataset<String> after filtering the rows (removing "+") because i
> can re-read the new dataset using .csv() method.
> Any better idea to do that ?
>
> Le sam. 4 juin 2022 à 18:40, Enrico Minack <in...@enrico.minack.dev> a
> écrit :
>
>> Can you provide an example string (row) and the expected inferred schema?
>>
>> Enrico
>>
>>
>> Am 04.06.22 um 18:36 schrieb marc nicole:
>>
>> How to do just that? i thought we only can inferSchema when we first read
>> the dataset, or am i wrong?
>>
>> Le sam. 4 juin 2022 à 18:10, Sean Owen <sr...@gmail.com> a écrit :
>>
>>> It sounds like you want to interpret the input as strings, do some
>>> processing, then infer the schema. That has nothing to do with construing
>>> the entire row as a string like "Row[foo=bar, baz=1]"
>>>
>>> On Sat, Jun 4, 2022 at 10:32 AM marc nicole <mk...@gmail.com> wrote:
>>>
>>>> Hi Sean,
>>>>
>>>> Thanks, actually I have a dataset where I want to inferSchema after
>>>> discarding the specific String value of "+". I do this because the column
>>>> would be considered StringType while if i remove that "+" value it will be
>>>> considered DoubleType for example or something else. Basically I want to
>>>> remove "+" from all dataset rows and then inferschema.
>>>> Here my idea is to filter the rows not equal to "+" for the target
>>>> columns (potentially all of them) and then use spark.read().csv() to read
>>>> the new filtered dataset with the option inferSchema which would then yield
>>>> correct column types.
>>>> What do you think?
>>>>
>>>> Le sam. 4 juin 2022 à 15:56, Sean Owen <sr...@gmail.com> a écrit :
>>>>
>>>>> I don't think you want to do that. You get a string representation of
>>>>> structured data without the structure, at best. This is part of the reason
>>>>> it doesn't work directly this way.
>>>>> You can use a UDF to call .toString on the Row of course, but, again
>>>>> what are you really trying to do?
>>>>>
>>>>> On Sat, Jun 4, 2022 at 7:35 AM marc nicole <mk...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>> How to convert a Dataset<Row> to a Dataset<String>?
>>>>>> What i have tried is:
>>>>>>
>>>>>> List<String> list = dataset.as(Encoders.STRING()).collectAsList();
>>>>>> Dataset<String> datasetSt = spark.createDataset(list, Encoders.STRING());
>>>>>> // But this line raises a org.apache.spark.sql.AnalysisException: Try to
>>>>>> map struct... to Tuple1, but failed as the number of fields does not line
>>>>>> up
>>>>>>
>>>>>> Type of columns being String
>>>>>> How to solve this?
>>>>>>
>>>>>
>>
>
Re: How to convert a Dataset to a Dataset?
Posted by Enrico Minack <in...@enrico.minack.dev>.
You could use .option("nullValue", "+") to tell the parser that '+'
refers to "no value":
spark.read
.option("inferSchema", "true")
.option("header", "true")
.option("nullvalue", "+")
.csv("path")
Enrico
Am 04.06.22 um 18:54 schrieb marc nicole:
>
> c1
>
>
>
> c2
>
>
>
> c3
>
>
>
> c4
>
>
>
> c5
>
>
>
> c6
>
> 1.2
>
>
>
> true
>
>
>
> A
>
>
>
> Z
>
>
>
> 120
>
>
>
> +
>
> 1.3
>
>
>
> false
>
>
>
> B
>
>
>
> X
>
>
>
> 130
>
>
>
> F
>
> +
>
>
>
> true
>
>
>
> C
>
>
>
> Y
>
>
>
> 200
>
>
>
> G
>
> in the above table c1 has double values except on the last row so:
>
> Dataset<Row> dataset =
> spark.read().format("csv")..option("inferSchema","true").option("header","true").load("path");
> will yield StringType as a type for column c1 similarly for c6
> I want to return the true type of each column by first discarding the "+"
> I use Dataset<String> after filtering the rows (removing "+") because
> i can re-read the new dataset using .csv() method.
> Any better idea to do that ?
>
> Le sam. 4 juin 2022 à 18:40, Enrico Minack <in...@enrico.minack.dev> a
> écrit :
>
> Can you provide an example string (row) and the expected inferred
> schema?
>
> Enrico
>
>
> Am 04.06.22 um 18:36 schrieb marc nicole:
>> How to do just that? i thought we only can inferSchema when we
>> first read the dataset, or am i wrong?
>>
>> Le sam. 4 juin 2022 à 18:10, Sean Owen <sr...@gmail.com> a écrit :
>>
>> It sounds like you want to interpret the input as strings, do
>> some processing, then infer the schema. That has nothing to
>> do with construing the entire row as a string like
>> "Row[foo=bar, baz=1]"
>>
>> On Sat, Jun 4, 2022 at 10:32 AM marc nicole
>> <mk...@gmail.com> wrote:
>>
>> Hi Sean,
>>
>> Thanks, actually I have a dataset where I want to
>> inferSchema after discarding the specific String value of
>> "+". I do this because the column would be considered
>> StringType while if i remove that "+" value it will be
>> considered DoubleType for example or something else.
>> Basically I want to remove "+" from all dataset rows and
>> then inferschema.
>> Here my idea is to filter the rows not equal to "+" for
>> the target columns (potentially all of them) and then use
>> spark.read().csv() to read the new filtered dataset with
>> the option inferSchema which would then yield correct
>> column types.
>> What do you think?
>>
>> Le sam. 4 juin 2022 à 15:56, Sean Owen <sr...@gmail.com>
>> a écrit :
>>
>> I don't think you want to do that. You get a string
>> representation of structured data without the
>> structure, at best. This is part of the reason it
>> doesn't work directly this way.
>> You can use a UDF to call .toString on the Row of
>> course, but, again what are you really trying to do?
>>
>> On Sat, Jun 4, 2022 at 7:35 AM marc nicole
>> <mk...@gmail.com> wrote:
>>
>> Hi,
>> How to convert a Dataset<Row> to a Dataset<String>?
>> What i have tried is:
>>
>> List<String> list = dataset.as
>> <http://dataset.as>(Encoders.STRING()).collectAsList();
>> Dataset<String> datasetSt =
>> spark.createDataset(list, Encoders.STRING()); //
>> But this line raises
>> a org.apache.spark.sql.AnalysisException: Try to
>> map struct... to Tuple1, but failed as the number
>> of fields does not line up
>>
>> Type of columns being String
>> How to solve this?
>>
>
Re: How to convert a Dataset to a Dataset?
Posted by marc nicole <mk...@gmail.com>.
c1
c2
c3
c4
c5
c6
1.2
true
A
Z
120
+
1.3
false
B
X
130
F
+
true
C
Y
200
G
in the above table c1 has double values except on the last row so:
Dataset<Row> dataset =
spark.read().format("csv")..option("inferSchema","true").option("header","true").load("path");
will yield StringType as a type for column c1 similarly for c6
I want to return the true type of each column by first discarding the "+"
I use Dataset<String> after filtering the rows (removing "+") because i can
re-read the new dataset using .csv() method.
Any better idea to do that ?
Le sam. 4 juin 2022 à 18:40, Enrico Minack <in...@enrico.minack.dev> a
écrit :
> Can you provide an example string (row) and the expected inferred schema?
>
> Enrico
>
>
> Am 04.06.22 um 18:36 schrieb marc nicole:
>
> How to do just that? i thought we only can inferSchema when we first read
> the dataset, or am i wrong?
>
> Le sam. 4 juin 2022 à 18:10, Sean Owen <sr...@gmail.com> a écrit :
>
>> It sounds like you want to interpret the input as strings, do some
>> processing, then infer the schema. That has nothing to do with construing
>> the entire row as a string like "Row[foo=bar, baz=1]"
>>
>> On Sat, Jun 4, 2022 at 10:32 AM marc nicole <mk...@gmail.com> wrote:
>>
>>> Hi Sean,
>>>
>>> Thanks, actually I have a dataset where I want to inferSchema after
>>> discarding the specific String value of "+". I do this because the column
>>> would be considered StringType while if i remove that "+" value it will be
>>> considered DoubleType for example or something else. Basically I want to
>>> remove "+" from all dataset rows and then inferschema.
>>> Here my idea is to filter the rows not equal to "+" for the target
>>> columns (potentially all of them) and then use spark.read().csv() to read
>>> the new filtered dataset with the option inferSchema which would then yield
>>> correct column types.
>>> What do you think?
>>>
>>> Le sam. 4 juin 2022 à 15:56, Sean Owen <sr...@gmail.com> a écrit :
>>>
>>>> I don't think you want to do that. You get a string representation of
>>>> structured data without the structure, at best. This is part of the reason
>>>> it doesn't work directly this way.
>>>> You can use a UDF to call .toString on the Row of course, but, again
>>>> what are you really trying to do?
>>>>
>>>> On Sat, Jun 4, 2022 at 7:35 AM marc nicole <mk...@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>> How to convert a Dataset<Row> to a Dataset<String>?
>>>>> What i have tried is:
>>>>>
>>>>> List<String> list = dataset.as(Encoders.STRING()).collectAsList();
>>>>> Dataset<String> datasetSt = spark.createDataset(list, Encoders.STRING());
>>>>> // But this line raises a org.apache.spark.sql.AnalysisException: Try to
>>>>> map struct... to Tuple1, but failed as the number of fields does not line
>>>>> up
>>>>>
>>>>> Type of columns being String
>>>>> How to solve this?
>>>>>
>>>>
>
Re: How to convert a Dataset to a Dataset?
Posted by Enrico Minack <in...@enrico.minack.dev>.
Can you provide an example string (row) and the expected inferred schema?
Enrico
Am 04.06.22 um 18:36 schrieb marc nicole:
> How to do just that? i thought we only can inferSchema when we first
> read the dataset, or am i wrong?
>
> Le sam. 4 juin 2022 à 18:10, Sean Owen <sr...@gmail.com> a écrit :
>
> It sounds like you want to interpret the input as strings, do some
> processing, then infer the schema. That has nothing to do with
> construing the entire row as a string like "Row[foo=bar, baz=1]"
>
> On Sat, Jun 4, 2022 at 10:32 AM marc nicole <mk...@gmail.com>
> wrote:
>
> Hi Sean,
>
> Thanks, actually I have a dataset where I want to inferSchema
> after discarding the specific String value of "+". I do this
> because the column would be considered StringType while if i
> remove that "+" value it will be considered DoubleType for
> example or something else. Basically I want to remove "+" from
> all dataset rows and then inferschema.
> Here my idea is to filter the rows not equal to "+" for the
> target columns (potentially all of them) and then use
> spark.read().csv() to read the new filtered dataset with the
> option inferSchema which would then yield correct column types.
> What do you think?
>
> Le sam. 4 juin 2022 à 15:56, Sean Owen <sr...@gmail.com> a
> écrit :
>
> I don't think you want to do that. You get a string
> representation of structured data without the structure,
> at best. This is part of the reason it doesn't work
> directly this way.
> You can use a UDF to call .toString on the Row of course,
> but, again what are you really trying to do?
>
> On Sat, Jun 4, 2022 at 7:35 AM marc nicole
> <mk...@gmail.com> wrote:
>
> Hi,
> How to convert a Dataset<Row> to a Dataset<String>?
> What i have tried is:
>
> List<String> list = dataset.as
> <http://dataset.as>(Encoders.STRING()).collectAsList();
> Dataset<String> datasetSt = spark.createDataset(list,
> Encoders.STRING()); // But this line raises
> a org.apache.spark.sql.AnalysisException: Try to map
> struct... to Tuple1, but failed as the number of
> fields does not line up
>
> Type of columns being String
> How to solve this?
>
Re: How to convert a Dataset to a Dataset?
Posted by marc nicole <mk...@gmail.com>.
How to do just that? i thought we only can inferSchema when we first read
the dataset, or am i wrong?
Le sam. 4 juin 2022 à 18:10, Sean Owen <sr...@gmail.com> a écrit :
> It sounds like you want to interpret the input as strings, do some
> processing, then infer the schema. That has nothing to do with construing
> the entire row as a string like "Row[foo=bar, baz=1]"
>
> On Sat, Jun 4, 2022 at 10:32 AM marc nicole <mk...@gmail.com> wrote:
>
>> Hi Sean,
>>
>> Thanks, actually I have a dataset where I want to inferSchema after
>> discarding the specific String value of "+". I do this because the column
>> would be considered StringType while if i remove that "+" value it will be
>> considered DoubleType for example or something else. Basically I want to
>> remove "+" from all dataset rows and then inferschema.
>> Here my idea is to filter the rows not equal to "+" for the target
>> columns (potentially all of them) and then use spark.read().csv() to read
>> the new filtered dataset with the option inferSchema which would then yield
>> correct column types.
>> What do you think?
>>
>> Le sam. 4 juin 2022 à 15:56, Sean Owen <sr...@gmail.com> a écrit :
>>
>>> I don't think you want to do that. You get a string representation of
>>> structured data without the structure, at best. This is part of the reason
>>> it doesn't work directly this way.
>>> You can use a UDF to call .toString on the Row of course, but, again
>>> what are you really trying to do?
>>>
>>> On Sat, Jun 4, 2022 at 7:35 AM marc nicole <mk...@gmail.com> wrote:
>>>
>>>> Hi,
>>>> How to convert a Dataset<Row> to a Dataset<String>?
>>>> What i have tried is:
>>>>
>>>> List<String> list = dataset.as(Encoders.STRING()).collectAsList();
>>>> Dataset<String> datasetSt = spark.createDataset(list, Encoders.STRING());
>>>> // But this line raises a org.apache.spark.sql.AnalysisException: Try to
>>>> map struct... to Tuple1, but failed as the number of fields does not line
>>>> up
>>>>
>>>> Type of columns being String
>>>> How to solve this?
>>>>
>>>
Re: How to convert a Dataset to a Dataset?
Posted by Sean Owen <sr...@gmail.com>.
It sounds like you want to interpret the input as strings, do some
processing, then infer the schema. That has nothing to do with construing
the entire row as a string like "Row[foo=bar, baz=1]"
On Sat, Jun 4, 2022 at 10:32 AM marc nicole <mk...@gmail.com> wrote:
> Hi Sean,
>
> Thanks, actually I have a dataset where I want to inferSchema after
> discarding the specific String value of "+". I do this because the column
> would be considered StringType while if i remove that "+" value it will be
> considered DoubleType for example or something else. Basically I want to
> remove "+" from all dataset rows and then inferschema.
> Here my idea is to filter the rows not equal to "+" for the target columns
> (potentially all of them) and then use spark.read().csv() to read the new
> filtered dataset with the option inferSchema which would then yield correct
> column types.
> What do you think?
>
> Le sam. 4 juin 2022 à 15:56, Sean Owen <sr...@gmail.com> a écrit :
>
>> I don't think you want to do that. You get a string representation of
>> structured data without the structure, at best. This is part of the reason
>> it doesn't work directly this way.
>> You can use a UDF to call .toString on the Row of course, but, again
>> what are you really trying to do?
>>
>> On Sat, Jun 4, 2022 at 7:35 AM marc nicole <mk...@gmail.com> wrote:
>>
>>> Hi,
>>> How to convert a Dataset<Row> to a Dataset<String>?
>>> What i have tried is:
>>>
>>> List<String> list = dataset.as(Encoders.STRING()).collectAsList();
>>> Dataset<String> datasetSt = spark.createDataset(list, Encoders.STRING());
>>> // But this line raises a org.apache.spark.sql.AnalysisException: Try to
>>> map struct... to Tuple1, but failed as the number of fields does not line
>>> up
>>>
>>> Type of columns being String
>>> How to solve this?
>>>
>>
Re: How to convert a Dataset to a Dataset?
Posted by marc nicole <mk...@gmail.com>.
Hi Sean,
Thanks, actually I have a dataset where I want to inferSchema after
discarding the specific String value of "+". I do this because the column
would be considered StringType while if i remove that "+" value it will be
considered DoubleType for example or something else. Basically I want to
remove "+" from all dataset rows and then inferschema.
Here my idea is to filter the rows not equal to "+" for the target columns
(potentially all of them) and then use spark.read().csv() to read the new
filtered dataset with the option inferSchema which would then yield correct
column types.
What do you think?
Le sam. 4 juin 2022 à 15:56, Sean Owen <sr...@gmail.com> a écrit :
> I don't think you want to do that. You get a string representation of
> structured data without the structure, at best. This is part of the reason
> it doesn't work directly this way.
> You can use a UDF to call .toString on the Row of course, but, again
> what are you really trying to do?
>
> On Sat, Jun 4, 2022 at 7:35 AM marc nicole <mk...@gmail.com> wrote:
>
>> Hi,
>> How to convert a Dataset<Row> to a Dataset<String>?
>> What i have tried is:
>>
>> List<String> list = dataset.as(Encoders.STRING()).collectAsList();
>> Dataset<String> datasetSt = spark.createDataset(list, Encoders.STRING());
>> // But this line raises a org.apache.spark.sql.AnalysisException: Try to
>> map struct... to Tuple1, but failed as the number of fields does not line
>> up
>>
>> Type of columns being String
>> How to solve this?
>>
>
Re: How to convert a Dataset to a Dataset?
Posted by Sean Owen <sr...@gmail.com>.
I don't think you want to do that. You get a string representation of
structured data without the structure, at best. This is part of the reason
it doesn't work directly this way.
You can use a UDF to call .toString on the Row of course, but, again
what are you really trying to do?
On Sat, Jun 4, 2022 at 7:35 AM marc nicole <mk...@gmail.com> wrote:
> Hi,
> How to convert a Dataset<Row> to a Dataset<String>?
> What i have tried is:
>
> List<String> list = dataset.as(Encoders.STRING()).collectAsList();
> Dataset<String> datasetSt = spark.createDataset(list, Encoders.STRING());
> // But this line raises a org.apache.spark.sql.AnalysisException: Try to
> map struct... to Tuple1, but failed as the number of fields does not line
> up
>
> Type of columns being String
> How to solve this?
>