You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Vikas Garg <sp...@gmail.com> on 2020/11/12 14:26:08 UTC
Spark Dataset withColumn issue
In Spark Datase, if we add additional column using
withColumn
then the column is added in the last.
e.g.
val ds1 = ds.select("Col1", "Col3").withColumn("Col2", lit("sample"))
the the order of columns is >> Col1 | Col3 | Col2
I want the order to be >> Col1 | Col2 | Col3
How can I achieve this?
Re: Spark Dataset withColumn issue
Posted by Sean Owen <sr...@gmail.com>.
You can still simply select the columns by name in order, after
.withColumn()
On Thu, Nov 12, 2020 at 9:49 AM Vikas Garg <sp...@gmail.com> wrote:
> I am deriving the col2 using with colunn which is why I cant use it like
> you told me
>
> On Thu, Nov 12, 2020, 20:11 German Schiavon <gs...@gmail.com>
> wrote:
>
>> ds.select("Col1", "Col2", "Col3")
>>
>> On Thu, 12 Nov 2020 at 15:28, Vikas Garg <sp...@gmail.com> wrote:
>>
>>> In Spark Datase, if we add additional column using
>>> withColumn
>>> then the column is added in the last.
>>>
>>> e.g.
>>> val ds1 = ds.select("Col1", "Col3").withColumn("Col2", lit("sample"))
>>>
>>> the the order of columns is >> Col1 | Col3 | Col2
>>>
>>> I want the order to be >> Col1 | Col2 | Col3
>>>
>>> How can I achieve this?
>>>
>>
Re: Spark Dataset withColumn issue
Posted by "Lalwani, Jayesh" <jl...@amazon.com.INVALID>.
Note that Spark never guarantees ordering of columns. There’s nothing in Spark documentation that says that the columns will be ordered a certain way. The proposed solution relies on an implementation detail that might change in future version of Spark.
Ideally, you shouldn’t rely on Dataframe to maintain order of columns. The question is why do you care about ordering of cols? If order of data is important, then you should put it in an array
From: Vikas Garg <sp...@gmail.com>
Date: Thursday, November 12, 2020 at 12:40 PM
To: Subash Prabakar <su...@gmail.com>
Cc: German Schiavon <gs...@gmail.com>, User <us...@spark.apache.org>
Subject: RE: [EXTERNAL] Spark Dataset withColumn issue
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Ohhkkkk
Thanks a lot
On Thu, Nov 12, 2020, 21:23 Subash Prabakar <su...@gmail.com>> wrote:
Hi Vikas,
He suggested to use the select() function after your withColumn function.
val ds1 = ds.select("Col1", "Col3").withColumn("Col2", lit("sample”)).select(“Col1”, “Col2”, “Col3")
Thanks,
Subash
On Thu, Nov 12, 2020 at 9:19 PM Vikas Garg <sp...@gmail.com>> wrote:
I am deriving the col2 using with colunn which is why I cant use it like you told me
On Thu, Nov 12, 2020, 20:11 German Schiavon <gs...@gmail.com>> wrote:
ds.select("Col1", "Col2", "Col3")
On Thu, 12 Nov 2020 at 15:28, Vikas Garg <sp...@gmail.com>> wrote:
In Spark Datase, if we add additional column using
withColumn
then the column is added in the last.
e.g.
val ds1 = ds.select("Col1", "Col3").withColumn("Col2", lit("sample"))
the the order of columns is >> Col1 | Col3 | Col2
I want the order to be >> Col1 | Col2 | Col3
How can I achieve this?
Re: Spark Dataset withColumn issue
Posted by Vikas Garg <sp...@gmail.com>.
Ohhkkkk
Thanks a lot
On Thu, Nov 12, 2020, 21:23 Subash Prabakar <su...@gmail.com>
wrote:
> Hi Vikas,
>
> He suggested to use the select() function after your withColumn function.
>
> val ds1 = ds.select("Col1", "Col3").withColumn("Col2",
> lit("sample”)).select(“Col1”, “Col2”, “Col3")
>
>
> Thanks,
> Subash
>
> On Thu, Nov 12, 2020 at 9:19 PM Vikas Garg <sp...@gmail.com> wrote:
>
>> I am deriving the col2 using with colunn which is why I cant use it like
>> you told me
>>
>> On Thu, Nov 12, 2020, 20:11 German Schiavon <gs...@gmail.com>
>> wrote:
>>
>>> ds.select("Col1", "Col2", "Col3")
>>>
>>> On Thu, 12 Nov 2020 at 15:28, Vikas Garg <sp...@gmail.com> wrote:
>>>
>>>> In Spark Datase, if we add additional column using
>>>> withColumn
>>>> then the column is added in the last.
>>>>
>>>> e.g.
>>>> val ds1 = ds.select("Col1", "Col3").withColumn("Col2", lit("sample"))
>>>>
>>>> the the order of columns is >> Col1 | Col3 | Col2
>>>>
>>>> I want the order to be >> Col1 | Col2 | Col3
>>>>
>>>> How can I achieve this?
>>>>
>>>
Re: Spark Dataset withColumn issue
Posted by Subash Prabakar <su...@gmail.com>.
Hi Vikas,
He suggested to use the select() function after your withColumn function.
val ds1 = ds.select("Col1", "Col3").withColumn("Col2",
lit("sample”)).select(“Col1”, “Col2”, “Col3")
Thanks,
Subash
On Thu, Nov 12, 2020 at 9:19 PM Vikas Garg <sp...@gmail.com> wrote:
> I am deriving the col2 using with colunn which is why I cant use it like
> you told me
>
> On Thu, Nov 12, 2020, 20:11 German Schiavon <gs...@gmail.com>
> wrote:
>
>> ds.select("Col1", "Col2", "Col3")
>>
>> On Thu, 12 Nov 2020 at 15:28, Vikas Garg <sp...@gmail.com> wrote:
>>
>>> In Spark Datase, if we add additional column using
>>> withColumn
>>> then the column is added in the last.
>>>
>>> e.g.
>>> val ds1 = ds.select("Col1", "Col3").withColumn("Col2", lit("sample"))
>>>
>>> the the order of columns is >> Col1 | Col3 | Col2
>>>
>>> I want the order to be >> Col1 | Col2 | Col3
>>>
>>> How can I achieve this?
>>>
>>
Re: Spark Dataset withColumn issue
Posted by Vikas Garg <sp...@gmail.com>.
I am deriving the col2 using with colunn which is why I cant use it like
you told me
On Thu, Nov 12, 2020, 20:11 German Schiavon <gs...@gmail.com>
wrote:
> ds.select("Col1", "Col2", "Col3")
>
> On Thu, 12 Nov 2020 at 15:28, Vikas Garg <sp...@gmail.com> wrote:
>
>> In Spark Datase, if we add additional column using
>> withColumn
>> then the column is added in the last.
>>
>> e.g.
>> val ds1 = ds.select("Col1", "Col3").withColumn("Col2", lit("sample"))
>>
>> the the order of columns is >> Col1 | Col3 | Col2
>>
>> I want the order to be >> Col1 | Col2 | Col3
>>
>> How can I achieve this?
>>
>
Re: Spark Dataset withColumn issue
Posted by German Schiavon <gs...@gmail.com>.
ds.select("Col1", "Col2", "Col3")
On Thu, 12 Nov 2020 at 15:28, Vikas Garg <sp...@gmail.com> wrote:
> In Spark Datase, if we add additional column using
> withColumn
> then the column is added in the last.
>
> e.g.
> val ds1 = ds.select("Col1", "Col3").withColumn("Col2", lit("sample"))
>
> the the order of columns is >> Col1 | Col3 | Col2
>
> I want the order to be >> Col1 | Col2 | Col3
>
> How can I achieve this?
>