You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Bharathi Raja <ra...@yahoo.com.INVALID> on 2016/05/11 15:07:07 UTC

Spark 1.6.0: substring on df.select

Hi,

I have a dataframe column col1 with values something like “/client/service/version/method”. The number of “/” are not constant. 
Could you please help me to extract all methods from the column col1?

In Pig i used SUBSTRING with LAST_INDEX_OF(“/”).

Thanks in advance.
Regards,
Raja

Re: Spark 1.6.0: substring on df.select

Posted by Sun Rui <su...@163.com>.

Alternatively, you may try the built-in function:
regexp_extract

> On May 12, 2016, at 20:27, Ewan Leith <ew...@realitymine.com> wrote:
> 
> You could use a UDF pretty easily, something like this should work, the lastElement function could be changed to do pretty much any string manipulation you want.
>  
> import org.apache.spark.sql.functions.udf
>  
> def lastElement(input: String) = input.split("/").last
>  
> val lastElementUdf = udf(lastElement(_:String))
>  
> df.select(lastElementUdf ($"col1")).show()
>  
> Ewan
>  
>  
> From: Bharathi Raja [mailto:rajakbv@yahoo.com.INVALID] 
> Sent: 12 May 2016 11:40
> To: Raghavendra Pandey <ra...@gmail.com>; Bharathi Raja <ra...@yahoo.com.invalid>
> Cc: User <us...@spark.apache.org>
> Subject: RE: Spark 1.6.0: substring on df.select
>  
> Thanks Raghav. 
>  
> I have 5+ million records. I feel creating multiple come is not an optimal way.
>  
> Please suggest any other alternate solution.
> Can’t we do any string operation in DF.Select?
>  
> Regards,
> Raja
>  
> From: Raghavendra Pandey <ma...@gmail.com>
> Sent: 11 May 2016 09:04 PM
> To: Bharathi Raja <ma...@yahoo.com.invalid>
> Cc: User <ma...@spark.apache.org>
> Subject: Re: Spark 1.6.0: substring on df.select
>  
> You can create a column with count of /.  Then take max of it and create that many columns for every row with null fillers.
> 
> Raghav 
> 
> On 11 May 2016 20:37, "Bharathi Raja" <rajakbv@yahoo.com.invalid <ma...@yahoo.com.invalid>> wrote:
> Hi,
>  
> I have a dataframe column col1 with values something like “/client/service/version/method”. The number of “/” are not constant.
> Could you please help me to extract all methods from the column col1?
>  
> In Pig i used SUBSTRING with LAST_INDEX_OF(“/”).
>  
> Thanks in advance.
> Regards,
> Raja

RE: Spark 1.6.0: substring on df.select

Posted by Ewan Leith <ew...@realitymine.com>.

You could use a UDF pretty easily, something like this should work, the lastElement function could be changed to do pretty much any string manipulation you want.

import org.apache.spark.sql.functions.udf

def lastElement(input: String) = input.split("/").last

val lastElementUdf = udf(lastElement(_:String))

df.select(lastElementUdf ($"col1")).show()

Ewan


From: Bharathi Raja [mailto:rajakbv@yahoo.com.INVALID]
Sent: 12 May 2016 11:40
To: Raghavendra Pandey <ra...@gmail.com>; Bharathi Raja <ra...@yahoo.com.invalid>
Cc: User <us...@spark.apache.org>
Subject: RE: Spark 1.6.0: substring on df.select

Thanks Raghav.

I have 5+ million records. I feel creating multiple come is not an optimal way.

Please suggest any other alternate solution.
Can’t we do any string operation in DF.Select?

Regards,
Raja

From: Raghavendra Pandey<ma...@gmail.com>
Sent: 11 May 2016 09:04 PM
To: Bharathi Raja<ma...@yahoo.com.invalid>
Cc: User<ma...@spark.apache.org>
Subject: Re: Spark 1.6.0: substring on df.select


You can create a column with count of /.  Then take max of it and create that many columns for every row with null fillers.

Raghav
On 11 May 2016 20:37, "Bharathi Raja" <ra...@yahoo.com.invalid>> wrote:
Hi,

I have a dataframe column col1 with values something like “/client/service/version/method”. The number of “/” are not constant.
Could you please help me to extract all methods from the column col1?

In Pig i used SUBSTRING with LAST_INDEX_OF(“/”).

Thanks in advance.
Regards,
Raja

RE: Spark 1.6.0: substring on df.select

Posted by Bharathi Raja <ra...@yahoo.com.INVALID>.

Thanks Raghav. 

I have 5+ million records. I feel creating multiple come is not an optimal way.

Please suggest any other alternate solution.
Can’t we do any string operation in DF.Select?

Regards,
Raja

From: Raghavendra Pandey
Sent: 11 May 2016 09:04 PM
To: Bharathi Raja
Cc: User
Subject: Re: Spark 1.6.0: substring on df.select

You can create a column with count of /.  Then take max of it and create that many columns for every row with null fillers. 
Raghav 
On 11 May 2016 20:37, "Bharathi Raja" <ra...@yahoo.com.invalid> wrote:
Hi,
 
I have a dataframe column col1 with values something like “/client/service/version/method”. The number of “/” are not constant. 
Could you please help me to extract all methods from the column col1?
 
In Pig i used SUBSTRING with LAST_INDEX_OF(“/”).
 
Thanks in advance.
Regards,
Raja

Re: Spark 1.6.0: substring on df.select

Posted by Raghavendra Pandey <ra...@gmail.com>.

You can create a column with count of /.  Then take max of it and create
that many columns for every row with null fillers.

Raghav
On 11 May 2016 20:37, "Bharathi Raja" <ra...@yahoo.com.invalid> wrote:

Hi,



I have a dataframe column col1 with values something like
“/client/service/version/method”. The number of “/” are not constant.

Could you please help me to extract all methods from the column col1?



In Pig i used SUBSTRING with LAST_INDEX_OF(“/”).



Thanks in advance.

Regards,

Raja