You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Zsolt Tóth <to...@gmail.com> on 2016/02/11 11:12:34 UTC
Inserting column to DataFrame
Hi,
I'd like to append a column of a dataframe to another DF (using Spark
1.5.2):
DataFrame outputDF = unlabelledDF.withColumn("predicted_label",
predictedDF.col("predicted"));
I get the following exception:
java.lang.IllegalArgumentException: requirement failed: DataFrame must have
the same schema as the relation to which is inserted.
DataFrame schema: StructType(StructField(predicted_label,DoubleType,true),
...<other 700 numerical (ByteType/ShortType) columns>
Relation schema: StructType(StructField(predicted_label,DoubleType,true),
...<the same 700 columns>
The interesting part is that the two schemas in the exception are exactly
the same.
The same code with other input data (with fewer, both numerical and
non-numerical column) succeeds.
Any idea why this happens?
Re: Inserting column to DataFrame
Posted by Zsolt Tóth <to...@gmail.com>.
Hi,
thanks for the answers. If joining the DataFrames is the solution, then why
does the simple withColumn() succeed for some datasets and fail for others?
2016-02-11 11:53 GMT+01:00 Michał Zieliński <zi...@gmail.com>:
> I think a good idea would be to do a join:
>
> outputDF = unlabelledDF.join(predictedDF.select(“id”,”predicted”),”id”)
>
> On 11 February 2016 at 10:12, Zsolt Tóth <to...@gmail.com> wrote:
>
>> Hi,
>>
>> I'd like to append a column of a dataframe to another DF (using Spark
>> 1.5.2):
>>
>> DataFrame outputDF = unlabelledDF.withColumn("predicted_label",
>> predictedDF.col("predicted"));
>>
>> I get the following exception:
>>
>> java.lang.IllegalArgumentException: requirement failed: DataFrame must
>> have the same schema as the relation to which is inserted.
>> DataFrame schema:
>> StructType(StructField(predicted_label,DoubleType,true), ...<other 700
>> numerical (ByteType/ShortType) columns>
>> Relation schema: StructType(StructField(predicted_label,DoubleType,true),
>> ...<the same 700 columns>
>>
>> The interesting part is that the two schemas in the exception are exactly
>> the same.
>> The same code with other input data (with fewer, both numerical and
>> non-numerical column) succeeds.
>> Any idea why this happens?
>>
>>
>
Re: Inserting column to DataFrame
Posted by Michał Zieliński <zi...@gmail.com>.
I think a good idea would be to do a join:
outputDF = unlabelledDF.join(predictedDF.select(“id”,”predicted”),”id”)
On 11 February 2016 at 10:12, Zsolt Tóth <to...@gmail.com> wrote:
> Hi,
>
> I'd like to append a column of a dataframe to another DF (using Spark
> 1.5.2):
>
> DataFrame outputDF = unlabelledDF.withColumn("predicted_label",
> predictedDF.col("predicted"));
>
> I get the following exception:
>
> java.lang.IllegalArgumentException: requirement failed: DataFrame must
> have the same schema as the relation to which is inserted.
> DataFrame schema: StructType(StructField(predicted_label,DoubleType,true),
> ...<other 700 numerical (ByteType/ShortType) columns>
> Relation schema: StructType(StructField(predicted_label,DoubleType,true),
> ...<the same 700 columns>
>
> The interesting part is that the two schemas in the exception are exactly
> the same.
> The same code with other input data (with fewer, both numerical and
> non-numerical column) succeeds.
> Any idea why this happens?
>
>
Re: Inserting column to DataFrame
Posted by Ted Yu <yu...@gmail.com>.
Seems like a bug.
Suggest filing an issue with code snippet if this can be reproduced on 1.6
branch.
Cheers
On Fri, Feb 12, 2016 at 4:25 AM, Zsolt Tóth <to...@gmail.com>
wrote:
> Sure. I ran the same job with fewer columns, the exception:
>
> java.lang.IllegalArgumentException: requirement failed: DataFrame must have the same schema as the relation to which is inserted.
> DataFrame schema: StructType(StructField(pixel0,ByteType,true), StructField(pixel1,ByteType,true), StructField(pixel10,ByteType,true), StructField(pixel100,ShortType,true), StructField(pixel101,ShortType,true), StructField(pixel102,ShortType,true), StructField(pixel103,ShortType,true), StructField(pixel105,ShortType,true), StructField(pixel106,ShortType,true), StructField(id,DoubleType,true), StructField(label,ByteType,true), StructField(predict,DoubleType,true))
> Relation schema: StructType(StructField(pixel0,ByteType,true), StructField(pixel1,ByteType,true), StructField(pixel10,ByteType,true), StructField(pixel100,ShortType,true), StructField(pixel101,ShortType,true), StructField(pixel102,ShortType,true), StructField(pixel103,ShortType,true), StructField(pixel105,ShortType,true), StructField(pixel106,ShortType,true), StructField(id,DoubleType,true), StructField(label,ByteType,true), StructField(predict,DoubleType,true))
>
> at scala.Predef$.require(Predef.scala:233)
> at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply$mcV$sp(InsertIntoHadoopFsRelation.scala:113)
> at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply(InsertIntoHadoopFsRelation.scala:108)
> at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply(InsertIntoHadoopFsRelation.scala:108)
> at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:56)
> at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation.run(InsertIntoHadoopFsRelation.scala:108)
> at org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult$lzycompute(commands.scala:57)
> at org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult(commands.scala:57)
> at org.apache.spark.sql.execution.ExecutedCommand.doExecute(commands.scala:69)
> at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:140)
> at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:138)
> at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
> at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:138)
> at org.apache.spark.sql.SQLContext$QueryExecution.toRdd$lzycompute(SQLContext.scala:933)
> at org.apache.spark.sql.SQLContext$QueryExecution.toRdd(SQLContext.scala:933)
> at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:197)
> at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:146)
> at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:137)
> at org.apache.spark.sql.DataFrameWriter.parquet(DataFrameWriter.scala:304)
>
> Regards,
>
> Zsolt
>
>
> 2016-02-12 13:11 GMT+01:00 Ted Yu <yu...@gmail.com>:
>
>> Can you pastebin the full error with all column types ?
>>
>> There should be a difference between some column(s).
>>
>> Cheers
>>
>> > On Feb 11, 2016, at 2:12 AM, Zsolt Tóth <to...@gmail.com>
>> wrote:
>> >
>> > Hi,
>> >
>> > I'd like to append a column of a dataframe to another DF (using Spark
>> 1.5.2):
>> >
>> > DataFrame outputDF = unlabelledDF.withColumn("predicted_label",
>> predictedDF.col("predicted"));
>> >
>> > I get the following exception:
>> >
>> > java.lang.IllegalArgumentException: requirement failed: DataFrame must
>> have the same schema as the relation to which is inserted.
>> > DataFrame schema:
>> StructType(StructField(predicted_label,DoubleType,true), ...<other 700
>> numerical (ByteType/ShortType) columns>
>> > Relation schema:
>> StructType(StructField(predicted_label,DoubleType,true), ...<the same 700
>> columns>
>> >
>> > The interesting part is that the two schemas in the exception are
>> exactly the same.
>> > The same code with other input data (with fewer, both numerical and
>> non-numerical column) succeeds.
>> > Any idea why this happens?
>> >
>>
>
>
Re: Inserting column to DataFrame
Posted by Zsolt Tóth <to...@gmail.com>.
Sure. I ran the same job with fewer columns, the exception:
java.lang.IllegalArgumentException: requirement failed: DataFrame must
have the same schema as the relation to which is inserted.
DataFrame schema: StructType(StructField(pixel0,ByteType,true),
StructField(pixel1,ByteType,true), StructField(pixel10,ByteType,true),
StructField(pixel100,ShortType,true),
StructField(pixel101,ShortType,true),
StructField(pixel102,ShortType,true),
StructField(pixel103,ShortType,true),
StructField(pixel105,ShortType,true),
StructField(pixel106,ShortType,true), StructField(id,DoubleType,true),
StructField(label,ByteType,true),
StructField(predict,DoubleType,true))
Relation schema: StructType(StructField(pixel0,ByteType,true),
StructField(pixel1,ByteType,true), StructField(pixel10,ByteType,true),
StructField(pixel100,ShortType,true),
StructField(pixel101,ShortType,true),
StructField(pixel102,ShortType,true),
StructField(pixel103,ShortType,true),
StructField(pixel105,ShortType,true),
StructField(pixel106,ShortType,true), StructField(id,DoubleType,true),
StructField(label,ByteType,true),
StructField(predict,DoubleType,true))
at scala.Predef$.require(Predef.scala:233)
at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply$mcV$sp(InsertIntoHadoopFsRelation.scala:113)
at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply(InsertIntoHadoopFsRelation.scala:108)
at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply(InsertIntoHadoopFsRelation.scala:108)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:56)
at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation.run(InsertIntoHadoopFsRelation.scala:108)
at org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult$lzycompute(commands.scala:57)
at org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult(commands.scala:57)
at org.apache.spark.sql.execution.ExecutedCommand.doExecute(commands.scala:69)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:140)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:138)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:138)
at org.apache.spark.sql.SQLContext$QueryExecution.toRdd$lzycompute(SQLContext.scala:933)
at org.apache.spark.sql.SQLContext$QueryExecution.toRdd(SQLContext.scala:933)
at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:197)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:146)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:137)
at org.apache.spark.sql.DataFrameWriter.parquet(DataFrameWriter.scala:304)
Regards,
Zsolt
2016-02-12 13:11 GMT+01:00 Ted Yu <yu...@gmail.com>:
> Can you pastebin the full error with all column types ?
>
> There should be a difference between some column(s).
>
> Cheers
>
> > On Feb 11, 2016, at 2:12 AM, Zsolt Tóth <to...@gmail.com>
> wrote:
> >
> > Hi,
> >
> > I'd like to append a column of a dataframe to another DF (using Spark
> 1.5.2):
> >
> > DataFrame outputDF = unlabelledDF.withColumn("predicted_label",
> predictedDF.col("predicted"));
> >
> > I get the following exception:
> >
> > java.lang.IllegalArgumentException: requirement failed: DataFrame must
> have the same schema as the relation to which is inserted.
> > DataFrame schema:
> StructType(StructField(predicted_label,DoubleType,true), ...<other 700
> numerical (ByteType/ShortType) columns>
> > Relation schema:
> StructType(StructField(predicted_label,DoubleType,true), ...<the same 700
> columns>
> >
> > The interesting part is that the two schemas in the exception are
> exactly the same.
> > The same code with other input data (with fewer, both numerical and
> non-numerical column) succeeds.
> > Any idea why this happens?
> >
>
Re: Inserting column to DataFrame
Posted by Ted Yu <yu...@gmail.com>.
Can you pastebin the full error with all column types ?
There should be a difference between some column(s).
Cheers
> On Feb 11, 2016, at 2:12 AM, Zsolt Tóth <to...@gmail.com> wrote:
>
> Hi,
>
> I'd like to append a column of a dataframe to another DF (using Spark 1.5.2):
>
> DataFrame outputDF = unlabelledDF.withColumn("predicted_label", predictedDF.col("predicted"));
>
> I get the following exception:
>
> java.lang.IllegalArgumentException: requirement failed: DataFrame must have the same schema as the relation to which is inserted.
> DataFrame schema: StructType(StructField(predicted_label,DoubleType,true), ...<other 700 numerical (ByteType/ShortType) columns>
> Relation schema: StructType(StructField(predicted_label,DoubleType,true), ...<the same 700 columns>
>
> The interesting part is that the two schemas in the exception are exactly the same.
> The same code with other input data (with fewer, both numerical and non-numerical column) succeeds.
> Any idea why this happens?
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org