You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Mina Aslani <as...@gmail.com> on 2018/05/15 14:58:13 UTC
OneHotEncoderEstimator - java.lang.NoSuchMethodError: org.apache.spark.sql.Dataset.withColumns
Hi,
I get below error when I try to run oneHotEncoderEstimator example.
https://github.com/apache/spark/blob/b74366481cc87490adf4e69d26389ec737548c15/examples/src/main/java/org/apache/spark/examples/ml/JavaOneHotEncoderEstimatorExample.java#L67
Which is this line of the code:
https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/feature/OneHotEncoderEstimator.scala#L348
Exception in thread "streaming-job-executor-0"
java.lang.NoSuchMethodError:
org.apache.spark.sql.Dataset.withColumns(Lscala/collection/Seq;Lscala/collection/Seq;)Lorg/apache/spark/sql/Dataset;
at org.apache.spark.ml.feature.OneHotEncoderModel.transform(OneHotEncoderEstimator.scala:348)
Can you please let me know, what is the cause? Any workaround?
Seeing the example in the repo, looks like that at some point it used be
running fine. And, now it's not working. Also, oneHotEncoder is deprecated.
I really appreciate your quick response.
Regards,
Mina
Re: OneHotEncoderEstimator - java.lang.NoSuchMethodError: org.apache.spark.sql.Dataset.withColumns
Posted by Bryan Cutler <cu...@gmail.com>.
The example works for me, please check your environment and ensure you are
using Spark 2.3.0 where OneHotEncoderEstimator was introduced.
On Fri, May 18, 2018 at 12:57 AM, Matteo Cossu <el...@gmail.com> wrote:
> Hi,
>
> are you sure Dataset has a method withColumns?
>
> On 15 May 2018 at 16:58, Mina Aslani <as...@gmail.com> wrote:
>
>> Hi,
>>
>> I get below error when I try to run oneHotEncoderEstimator example.
>> https://github.com/apache/spark/blob/b74366481cc87490adf4e69
>> d26389ec737548c15/examples/src/main/java/org/apache/
>> spark/examples/ml/JavaOneHotEncoderEstimatorExample.java#L67
>>
>> Which is this line of the code:
>> https://github.com/apache/spark/blob/master/mllib/src/main/
>> scala/org/apache/spark/ml/feature/OneHotEncoderEstimator.scala#L348
>>
>> Exception in thread "streaming-job-executor-0" java.lang.NoSuchMethodError: org.apache.spark.sql.Dataset.withColumns(Lscala/collection/Seq;Lscala/collection/Seq;)Lorg/apache/spark/sql/Dataset;
>> at org.apache.spark.ml.feature.OneHotEncoderModel.transform(OneHotEncoderEstimator.scala:348)
>>
>>
>> Can you please let me know, what is the cause? Any workaround?
>>
>> Seeing the example in the repo, looks like that at some point it used be
>> running fine. And, now it's not working. Also, oneHotEncoder is deprecated.
>>
>> I really appreciate your quick response.
>>
>> Regards,
>> Mina
>>
>>
>
Re: OneHotEncoderEstimator - java.lang.NoSuchMethodError: org.apache.spark.sql.Dataset.withColumns
Posted by Matteo Cossu <el...@gmail.com>.
Hi,
are you sure Dataset has a method withColumns?
On 15 May 2018 at 16:58, Mina Aslani <as...@gmail.com> wrote:
> Hi,
>
> I get below error when I try to run oneHotEncoderEstimator example.
> https://github.com/apache/spark/blob/b74366481cc87490adf4e69d26389e
> c737548c15/examples/src/main/java/org/apache/spark/examples/ml/
> JavaOneHotEncoderEstimatorExample.java#L67
>
> Which is this line of the code:
> https://github.com/apache/spark/blob/master/mllib/src/
> main/scala/org/apache/spark/ml/feature/OneHotEncoderEstimator.scala#L348
>
> Exception in thread "streaming-job-executor-0" java.lang.NoSuchMethodError: org.apache.spark.sql.Dataset.withColumns(Lscala/collection/Seq;Lscala/collection/Seq;)Lorg/apache/spark/sql/Dataset;
> at org.apache.spark.ml.feature.OneHotEncoderModel.transform(OneHotEncoderEstimator.scala:348)
>
>
> Can you please let me know, what is the cause? Any workaround?
>
> Seeing the example in the repo, looks like that at some point it used be
> running fine. And, now it's not working. Also, oneHotEncoder is deprecated.
>
> I really appreciate your quick response.
>
> Regards,
> Mina
>
>