You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Rico Bergmann <in...@ricobergmann.de> on 2022/02/16 16:26:23 UTC
Cast int to string not possible?
Hi!
I am reading a partitioned dataFrame into spark using automatic type inference for the partition columns. For one partition column the data contains an integer, therefor Spark uses IntegerType for this column. In general this is supposed to be a StringType column. So I tried to cast this column to StringType. But this fails with AnalysisException “cannot cast int to string”.
Is this a bug? Or is it really not allowed to cast an int to a string?
I’m using Spark 3.1.1
Best regards
Rico.
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: Cast int to string not possible?
Posted by ayan guha <gu...@gmail.com>.
Can you try to cast any other Int field which is NOT a partition column?
On Thu, 17 Feb 2022 at 7:34 pm, Gourav Sengupta <go...@gmail.com>
wrote:
> Hi,
>
> This appears interesting, casting INT to STRING has never been an issue
> for me.
>
> Can you just help us with the output of : df.printSchema() ?
>
> I prefer to use SQL, and the method I use for casting is: CAST(<<column
> name>> AS STRING) <<alias>>.
>
> Regards,
> Gourav
>
>
>
>
>
>
> On Thu, Feb 17, 2022 at 6:02 AM Rico Bergmann <in...@ricobergmann.de>
> wrote:
>
>> Here is the code snippet:
>>
>> var df = session.read().parquet(basepath);
>> for(Column partition : partitionColumnsList){
>> df = df.withColumn(partition.getName(),
>> df.col(partition.getName()).cast(partition.getType()));
>> }
>>
>> Column is a class containing Schema Information, like for example the
>> name of the column and the data type of the column.
>>
>> Best, Rico.
>>
>> > Am 17.02.2022 um 03:17 schrieb Morven Huang <mo...@gmail.com>:
>> >
>> > Hi Rico, you have any code snippet? I have no problem casting int to
>> string.
>> >
>> >> 2022年2月17日 上午12:26，Rico Bergmann <in...@ricobergmann.de> 写道：
>> >>
>> >> Hi!
>> >>
>> >> I am reading a partitioned dataFrame into spark using automatic type
>> inference for the partition columns. For one partition column the data
>> contains an integer, therefor Spark uses IntegerType for this column. In
>> general this is supposed to be a StringType column. So I tried to cast this
>> column to StringType. But this fails with AnalysisException “cannot cast
>> int to string”.
>> >>
>> >> Is this a bug? Or is it really not allowed to cast an int to a string?
>> >>
>> >> I’m using Spark 3.1.1
>> >>
>> >> Best regards
>> >>
>> >> Rico.
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>> >>
>> >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>> >
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>
>> --
Best Regards,
Ayan Guha
Re: Cast int to string not possible?
Posted by Gourav Sengupta <go...@gmail.com>.
Hi,
This appears interesting, casting INT to STRING has never been an issue for
me.
Can you just help us with the output of : df.printSchema() ?
I prefer to use SQL, and the method I use for casting is: CAST(<<column
name>> AS STRING) <<alias>>.
Regards,
Gourav
On Thu, Feb 17, 2022 at 6:02 AM Rico Bergmann <in...@ricobergmann.de> wrote:
> Here is the code snippet:
>
> var df = session.read().parquet(basepath);
> for(Column partition : partitionColumnsList){
> df = df.withColumn(partition.getName(),
> df.col(partition.getName()).cast(partition.getType()));
> }
>
> Column is a class containing Schema Information, like for example the name
> of the column and the data type of the column.
>
> Best, Rico.
>
> > Am 17.02.2022 um 03:17 schrieb Morven Huang <mo...@gmail.com>:
> >
> > Hi Rico, you have any code snippet? I have no problem casting int to
> string.
> >
> >> 2022年2月17日 上午12:26，Rico Bergmann <in...@ricobergmann.de> 写道：
> >>
> >> Hi!
> >>
> >> I am reading a partitioned dataFrame into spark using automatic type
> inference for the partition columns. For one partition column the data
> contains an integer, therefor Spark uses IntegerType for this column. In
> general this is supposed to be a StringType column. So I tried to cast this
> column to StringType. But this fails with AnalysisException “cannot cast
> int to string”.
> >>
> >> Is this a bug? Or is it really not allowed to cast an int to a string?
> >>
> >> I’m using Spark 3.1.1
> >>
> >> Best regards
> >>
> >> Rico.
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
> >>
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe e-mail: user-unsubscribe@spark.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>
Re: Cast int to string not possible?
Posted by Rico Bergmann <in...@ricobergmann.de>.
Here is the code snippet:
var df = session.read().parquet(basepath);
for(Column partition : partitionColumnsList){
df = df.withColumn(partition.getName(), df.col(partition.getName()).cast(partition.getType()));
}
Column is a class containing Schema Information, like for example the name of the column and the data type of the column.
Best, Rico.
> Am 17.02.2022 um 03:17 schrieb Morven Huang <mo...@gmail.com>:
>
> Hi Rico, you have any code snippet? I have no problem casting int to string.
>
>> 2022年2月17日 上午12:26，Rico Bergmann <in...@ricobergmann.de> 写道：
>>
>> Hi!
>>
>> I am reading a partitioned dataFrame into spark using automatic type inference for the partition columns. For one partition column the data contains an integer, therefor Spark uses IntegerType for this column. In general this is supposed to be a StringType column. So I tried to cast this column to StringType. But this fails with AnalysisException “cannot cast int to string”.
>>
>> Is this a bug? Or is it really not allowed to cast an int to a string?
>>
>> I’m using Spark 3.1.1
>>
>> Best regards
>>
>> Rico.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: Cast int to string not possible?
Posted by Morven Huang <mo...@gmail.com>.
Hi Rico, you have any code snippet? I have no problem casting int to string.
> 2022年2月17日 上午12:26，Rico Bergmann <in...@ricobergmann.de> 写道：
>
> Hi!
>
> I am reading a partitioned dataFrame into spark using automatic type inference for the partition columns. For one partition column the data contains an integer, therefor Spark uses IntegerType for this column. In general this is supposed to be a StringType column. So I tried to cast this column to StringType. But this fails with AnalysisException “cannot cast int to string”.
>
> Is this a bug? Or is it really not allowed to cast an int to a string?
>
> I’m using Spark 3.1.1
>
> Best regards
>
> Rico.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org