You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Aakash Basu <aa...@gmail.com> on 2017/12/22 09:24:57 UTC
Passing an array of more than 22 elements in a UDF
Hi,
I am using Spark 2.2 using Java, can anyone please suggest me how to take
more than 22 parameters in an UDF? I mean, if I want to pass all the
parameters as an array of integers?
Thanks,
Aakash.
Re: Passing an array of more than 22 elements in a UDF
Posted by Vadim Semenov <va...@datadoghq.com>.
Functions are still limited to 22 arguments
https://github.com/scala/scala/blob/2.13.x/src/library/scala/Function22.scala
On Tue, Dec 26, 2017 at 2:19 PM, Felix Cheung <fe...@hotmail.com>
wrote:
> Generally the 22 limitation is from Scala 2.10.
>
> In Scala 2.11, the issue with case class is fixed, but with that said I’m
> not sure if with UDF in Java other limitation might apply.
>
> _____________________________
> From: Aakash Basu <aa...@gmail.com>
> Sent: Monday, December 25, 2017 9:13 PM
> Subject: Re: Passing an array of more than 22 elements in a UDF
> To: Felix Cheung <fe...@hotmail.com>
> Cc: ayan guha <gu...@gmail.com>, user <us...@spark.apache.org>
>
>
>
> What's the privilege of using that specific version for this? Please throw
> some light onto it.
>
> On Mon, Dec 25, 2017 at 6:51 AM, Felix Cheung <fe...@hotmail.com>
> wrote:
>
>> Or use it with Scala 2.11?
>>
>> ------------------------------
>> *From:* ayan guha <gu...@gmail.com>
>> *Sent:* Friday, December 22, 2017 3:15:14 AM
>> *To:* Aakash Basu
>> *Cc:* user
>> *Subject:* Re: Passing an array of more than 22 elements in a UDF
>>
>> Hi I think you are in correct track. You can stuff all your param in a
>> suitable data structure like array or dict and pass this structure as a
>> single param in your udf.
>>
>> On Fri, 22 Dec 2017 at 2:55 pm, Aakash Basu <aa...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I am using Spark 2.2 using Java, can anyone please suggest me how to
>>> take more than 22 parameters in an UDF? I mean, if I want to pass all the
>>> parameters as an array of integers?
>>>
>>> Thanks,
>>> Aakash.
>>>
>> --
>> Best Regards,
>> Ayan Guha
>>
>
>
>
>
Re: Passing an array of more than 22 elements in a UDF
Posted by Felix Cheung <fe...@hotmail.com>.
Generally the 22 limitation is from Scala 2.10.
In Scala 2.11, the issue with case class is fixed, but with that said I’m not sure if with UDF in Java other limitation might apply.
_____________________________
From: Aakash Basu <aa...@gmail.com>
Sent: Monday, December 25, 2017 9:13 PM
Subject: Re: Passing an array of more than 22 elements in a UDF
To: Felix Cheung <fe...@hotmail.com>
Cc: ayan guha <gu...@gmail.com>, user <us...@spark.apache.org>
What's the privilege of using that specific version for this? Please throw some light onto it.
On Mon, Dec 25, 2017 at 6:51 AM, Felix Cheung <fe...@hotmail.com>> wrote:
Or use it with Scala 2.11?
________________________________
From: ayan guha <gu...@gmail.com>>
Sent: Friday, December 22, 2017 3:15:14 AM
To: Aakash Basu
Cc: user
Subject: Re: Passing an array of more than 22 elements in a UDF
Hi I think you are in correct track. You can stuff all your param in a suitable data structure like array or dict and pass this structure as a single param in your udf.
On Fri, 22 Dec 2017 at 2:55 pm, Aakash Basu <aa...@gmail.com>> wrote:
Hi,
I am using Spark 2.2 using Java, can anyone please suggest me how to take more than 22 parameters in an UDF? I mean, if I want to pass all the parameters as an array of integers?
Thanks,
Aakash.
--
Best Regards,
Ayan Guha
Re: Passing an array of more than 22 elements in a UDF
Posted by Aakash Basu <aa...@gmail.com>.
What's the privilege of using that specific version for this? Please throw
some light onto it.
On Mon, Dec 25, 2017 at 6:51 AM, Felix Cheung <fe...@hotmail.com>
wrote:
> Or use it with Scala 2.11?
>
> ------------------------------
> *From:* ayan guha <gu...@gmail.com>
> *Sent:* Friday, December 22, 2017 3:15:14 AM
> *To:* Aakash Basu
> *Cc:* user
> *Subject:* Re: Passing an array of more than 22 elements in a UDF
>
> Hi I think you are in correct track. You can stuff all your param in a
> suitable data structure like array or dict and pass this structure as a
> single param in your udf.
>
> On Fri, 22 Dec 2017 at 2:55 pm, Aakash Basu <aa...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I am using Spark 2.2 using Java, can anyone please suggest me how to take
>> more than 22 parameters in an UDF? I mean, if I want to pass all the
>> parameters as an array of integers?
>>
>> Thanks,
>> Aakash.
>>
> --
> Best Regards,
> Ayan Guha
>
Re: Passing an array of more than 22 elements in a UDF
Posted by Felix Cheung <fe...@hotmail.com>.
Or use it with Scala 2.11?
________________________________
From: ayan guha <gu...@gmail.com>
Sent: Friday, December 22, 2017 3:15:14 AM
To: Aakash Basu
Cc: user
Subject: Re: Passing an array of more than 22 elements in a UDF
Hi I think you are in correct track. You can stuff all your param in a suitable data structure like array or dict and pass this structure as a single param in your udf.
On Fri, 22 Dec 2017 at 2:55 pm, Aakash Basu <aa...@gmail.com>> wrote:
Hi,
I am using Spark 2.2 using Java, can anyone please suggest me how to take more than 22 parameters in an UDF? I mean, if I want to pass all the parameters as an array of integers?
Thanks,
Aakash.
--
Best Regards,
Ayan Guha
Re: Passing an array of more than 22 elements in a UDF
Posted by ayan guha <gu...@gmail.com>.
Hi I think you are in correct track. You can stuff all your param in a
suitable data structure like array or dict and pass this structure as a
single param in your udf.
On Fri, 22 Dec 2017 at 2:55 pm, Aakash Basu <aa...@gmail.com>
wrote:
> Hi,
>
> I am using Spark 2.2 using Java, can anyone please suggest me how to take
> more than 22 parameters in an UDF? I mean, if I want to pass all the
> parameters as an array of integers?
>
> Thanks,
> Aakash.
>
--
Best Regards,
Ayan Guha