You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Arthur Chan <ar...@gmail.com> on 2015/06/28 14:49:07 UTC
required: org.apache.spark.streaming.dstream.DStream[org.apache.spark.mllib.linalg.Vector]
Hi,
I am trying Spark with some sample programs,
In my code, the following items are imported:
import org.apache.spark.mllib.regression.{StreamingLinearRegressionWithSGD,
LabeledPoint}
import org.apache.spark.mllib.regression.{StreamingLinearRegressionWithSGD}
import org.apache.spark.streaming.{Seconds, StreamingContext}
import scala.util.Random
I got following error:
[error] StreamingModel.scala:100: type mismatch;
[error] found :
org.apache.spark.streaming.dstream.DStream[org.apache.spark.mllib.regression.LabeledPoint]
[error] required:
org.apache.spark.streaming.dstream.DStream[org.apache.spark.mllib.linalg.Vector]
[error] model.predictOn(labeledStream).print()
[error] ^
[error] one error found
[error] (compile:compile) Compilation failed
Any idea?
Regards
Re: required: org.apache.spark.streaming.dstream.DStream[org.apache.spark.mllib.linalg.Vector]
Posted by Feynman Liang <fl...@databricks.com>.
You are trying to predict on a DStream[LabeledPoint] (data + labels) but
predictOn expects a DStream[Vector] (just the data without the labels).
Try doing:
val unlabeledStream = labeledStream.map { x => x.features }
model.predictOn(unlabeledStream).print()
On Sun, Jun 28, 2015 at 6:03 PM, Arthur Chan <ar...@gmail.com>
wrote:
> also my Spark is 1.4
>
> On Mon, Jun 29, 2015 at 9:02 AM, Arthur Chan <ar...@gmail.com>
> wrote:
>
>>
>>
>> Hi,
>>
>>
>> line 99: model.trainOn(labeledStream)
>>
>> line 100: model.predictOn(labeledStream).print()
>>
>> line 101: ssc.start()
>>
>> line 102: ssc.awaitTermination()
>>
>>
>> Regards
>>
>> On Sun, Jun 28, 2015 at 10:53 PM, Ted Yu <yu...@gmail.com> wrote:
>>
>>> Can you show us your code around line 100 ?
>>>
>>> Which Spark release are you compiling against ?
>>>
>>> Cheers
>>>
>>> On Sun, Jun 28, 2015 at 5:49 AM, Arthur Chan <ar...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am trying Spark with some sample programs,
>>>>
>>>>
>>>> In my code, the following items are imported:
>>>>
>>>> import
>>>> org.apache.spark.mllib.regression.{StreamingLinearRegressionWithSGD,
>>>> LabeledPoint}
>>>>
>>>> import
>>>> org.apache.spark.mllib.regression.{StreamingLinearRegressionWithSGD}
>>>>
>>>> import org.apache.spark.streaming.{Seconds, StreamingContext}
>>>>
>>>> import scala.util.Random
>>>>
>>>> I got following error:
>>>>
>>>> [error] StreamingModel.scala:100: type mismatch;
>>>>
>>>> [error] found :
>>>> org.apache.spark.streaming.dstream.DStream[org.apache.spark.mllib.regression.LabeledPoint]
>>>>
>>>> [error] required:
>>>> org.apache.spark.streaming.dstream.DStream[org.apache.spark.mllib.linalg.Vector]
>>>>
>>>> [error] model.predictOn(labeledStream).print()
>>>>
>>>> [error] ^
>>>>
>>>> [error] one error found
>>>>
>>>> [error] (compile:compile) Compilation failed
>>>>
>>>>
>>>> Any idea?
>>>>
>>>>
>>>> Regards
>>>>
>>>
>>>
>>
>
Re: required: org.apache.spark.streaming.dstream.DStream[org.apache.spark.mllib.linalg.Vector]
Posted by Arthur Chan <ar...@gmail.com>.
also my Spark is 1.4
On Mon, Jun 29, 2015 at 9:02 AM, Arthur Chan <ar...@gmail.com>
wrote:
>
>
> Hi,
>
>
> line 99: model.trainOn(labeledStream)
>
> line 100: model.predictOn(labeledStream).print()
>
> line 101: ssc.start()
>
> line 102: ssc.awaitTermination()
>
>
> Regards
>
> On Sun, Jun 28, 2015 at 10:53 PM, Ted Yu <yu...@gmail.com> wrote:
>
>> Can you show us your code around line 100 ?
>>
>> Which Spark release are you compiling against ?
>>
>> Cheers
>>
>> On Sun, Jun 28, 2015 at 5:49 AM, Arthur Chan <ar...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I am trying Spark with some sample programs,
>>>
>>>
>>> In my code, the following items are imported:
>>>
>>> import
>>> org.apache.spark.mllib.regression.{StreamingLinearRegressionWithSGD,
>>> LabeledPoint}
>>>
>>> import
>>> org.apache.spark.mllib.regression.{StreamingLinearRegressionWithSGD}
>>>
>>> import org.apache.spark.streaming.{Seconds, StreamingContext}
>>>
>>> import scala.util.Random
>>>
>>> I got following error:
>>>
>>> [error] StreamingModel.scala:100: type mismatch;
>>>
>>> [error] found :
>>> org.apache.spark.streaming.dstream.DStream[org.apache.spark.mllib.regression.LabeledPoint]
>>>
>>> [error] required:
>>> org.apache.spark.streaming.dstream.DStream[org.apache.spark.mllib.linalg.Vector]
>>>
>>> [error] model.predictOn(labeledStream).print()
>>>
>>> [error] ^
>>>
>>> [error] one error found
>>>
>>> [error] (compile:compile) Compilation failed
>>>
>>>
>>> Any idea?
>>>
>>>
>>> Regards
>>>
>>
>>
>
Re: required: org.apache.spark.streaming.dstream.DStream[org.apache.spark.mllib.linalg.Vector]
Posted by Arthur Chan <ar...@gmail.com>.
Hi,
line 99: model.trainOn(labeledStream)
line 100: model.predictOn(labeledStream).print()
line 101: ssc.start()
line 102: ssc.awaitTermination()
Regards
On Sun, Jun 28, 2015 at 10:53 PM, Ted Yu <yu...@gmail.com> wrote:
> Can you show us your code around line 100 ?
>
> Which Spark release are you compiling against ?
>
> Cheers
>
> On Sun, Jun 28, 2015 at 5:49 AM, Arthur Chan <ar...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I am trying Spark with some sample programs,
>>
>>
>> In my code, the following items are imported:
>>
>> import
>> org.apache.spark.mllib.regression.{StreamingLinearRegressionWithSGD,
>> LabeledPoint}
>>
>> import
>> org.apache.spark.mllib.regression.{StreamingLinearRegressionWithSGD}
>>
>> import org.apache.spark.streaming.{Seconds, StreamingContext}
>>
>> import scala.util.Random
>>
>> I got following error:
>>
>> [error] StreamingModel.scala:100: type mismatch;
>>
>> [error] found :
>> org.apache.spark.streaming.dstream.DStream[org.apache.spark.mllib.regression.LabeledPoint]
>>
>> [error] required:
>> org.apache.spark.streaming.dstream.DStream[org.apache.spark.mllib.linalg.Vector]
>>
>> [error] model.predictOn(labeledStream).print()
>>
>> [error] ^
>>
>> [error] one error found
>>
>> [error] (compile:compile) Compilation failed
>>
>>
>> Any idea?
>>
>>
>> Regards
>>
>
>
Re: required: org.apache.spark.streaming.dstream.DStream[org.apache.spark.mllib.linalg.Vector]
Posted by Ted Yu <yu...@gmail.com>.
Can you show us your code around line 100 ?
Which Spark release are you compiling against ?
Cheers
On Sun, Jun 28, 2015 at 5:49 AM, Arthur Chan <ar...@gmail.com>
wrote:
> Hi,
>
> I am trying Spark with some sample programs,
>
>
> In my code, the following items are imported:
>
> import
> org.apache.spark.mllib.regression.{StreamingLinearRegressionWithSGD,
> LabeledPoint}
>
> import org.apache.spark.mllib.regression.{StreamingLinearRegressionWithSGD}
>
> import org.apache.spark.streaming.{Seconds, StreamingContext}
>
> import scala.util.Random
>
> I got following error:
>
> [error] StreamingModel.scala:100: type mismatch;
>
> [error] found :
> org.apache.spark.streaming.dstream.DStream[org.apache.spark.mllib.regression.LabeledPoint]
>
> [error] required:
> org.apache.spark.streaming.dstream.DStream[org.apache.spark.mllib.linalg.Vector]
>
> [error] model.predictOn(labeledStream).print()
>
> [error] ^
>
> [error] one error found
>
> [error] (compile:compile) Compilation failed
>
>
> Any idea?
>
>
> Regards
>