You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Artemis User <ar...@dtechspace.com> on 2022/03/15 20:38:48 UTC
Continuous ML model training in stream mode
Has anyone done any experiments of training an ML model using stream
data? especially for unsupervised models? Any suggestions/references
are highly appreciated...
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: Continuous ML model training in stream mode
Posted by Artemis User <ar...@dtechspace.com>.
Could you please keep your response relevant to the question and the
context of the discussion thread? Thanks!
On 3/17/22 2:56 PM, Gourav Sengupta wrote:
> Dear friends,
>
> a few years ago, I was in a London meetup seeing Sean (Owen)
> demonstrate how we can try to predict the gender of individuals who
> are responding to tweets after accepting privacy agreements, in case I
> am not wrong.
>
> It was real time, it was spectacular, and it was the presentation that
> set me into data science and its applications.
>
> Thanks Sean! :)
>
> Regards,
> Gourav Sengupta
>
>
>
>
> On Tue, Mar 15, 2022 at 9:39 PM Artemis User <ar...@dtechspace.com>
> wrote:
>
> Thanks Sean! Well, it looks like we have to abandon our
> structured streaming model to use DStream for this, or do you see
> possibility to use structured streaming with ml instead of mllib?
>
> On 3/15/22 4:51 PM, Sean Owen wrote:
>> There is a streaming k-means example in Spark.
>> https://spark.apache.org/docs/latest/mllib-clustering.html#streaming-k-means
>>
>> On Tue, Mar 15, 2022, 3:46 PM Artemis User
>> <ar...@dtechspace.com> wrote:
>>
>> Has anyone done any experiments of training an ML model using
>> stream
>> data? especially for unsupervised models? Any
>> suggestions/references
>> are highly appreciated...
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>
>
unsubscribe
Posted by Basavaraj <ra...@gmail.com>.
unsubscribe
Re: Continuous ML model training in stream mode
Posted by Gourav Sengupta <go...@gmail.com>.
i think acknowledging and appreciating some of the brilliant and fantastic
people in this group is not out of context. please stop being rude and
antagonising when people are responding and starting statements by saying ,
with all respect, does not hide the rudeness.
i would have rather taken clue from the appreciations and asked for further
insights about my questions from Sean.
Regards,
Gourav
On Fri, 18 Mar 2022, 11:35 Artemis User, <ar...@dtechspace.com> wrote:
> Again, with all respect, please keep your response professional and
> relevant to the question and the context of the discussion thread! This is
> not your personal mailing list!
>
> On 3/17/22 4:16 PM, Gourav Sengupta wrote:
>
> Dearest Sean,
>
> I think in that meeting you also mentioned about your company Myrrix :)
>
> Though covered in 2011 the fundamentals on distributed computing and
> explanations of algorithms is still wonderful.
>
> [image: image.jpeg]
>
> Thanks and Regards,
> Gourav Sengupta
>
> On Thu, Mar 17, 2022 at 7:31 PM Sean Owen <sr...@gmail.com> wrote:
>
>> (Thank you, not sure that was me though)
>> I don't know of plans to expose the streaming impls in ML, as they still
>> work fine in MLlib and they also don't come up much. Continuous training is
>> relatively rare, maybe under-appreciated, but rare in practice.
>>
>> On Thu, Mar 17, 2022 at 1:57 PM Gourav Sengupta <
>> gourav.sengupta@gmail.com> wrote:
>>
>>> Dear friends,
>>>
>>> a few years ago, I was in a London meetup seeing Sean (Owen) demonstrate
>>> how we can try to predict the gender of individuals who are responding to
>>> tweets after accepting privacy agreements, in case I am not wrong.
>>>
>>> It was real time, it was spectacular, and it was the presentation that
>>> set me into data science and its applications.
>>>
>>> Thanks Sean! :)
>>>
>>> Regards,
>>> Gourav Sengupta
>>>
>>>
>>>
>>>
>>> On Tue, Mar 15, 2022 at 9:39 PM Artemis User <ar...@dtechspace.com>
>>> wrote:
>>>
>>>> Thanks Sean! Well, it looks like we have to abandon our structured
>>>> streaming model to use DStream for this, or do you see possibility to use
>>>> structured streaming with ml instead of mllib?
>>>>
>>>> On 3/15/22 4:51 PM, Sean Owen wrote:
>>>>
>>>> There is a streaming k-means example in Spark.
>>>> https://spark.apache.org/docs/latest/mllib-clustering.html#streaming-k-means
>>>>
>>>> On Tue, Mar 15, 2022, 3:46 PM Artemis User <ar...@dtechspace.com>
>>>> wrote:
>>>>
>>>>> Has anyone done any experiments of training an ML model using stream
>>>>> data? especially for unsupervised models? Any suggestions/references
>>>>> are highly appreciated...
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>>>>
>>>>>
>>>>
>
Re: Continuous ML model training in stream mode
Posted by Artemis User <ar...@dtechspace.com>.
Again, with all respect, please keep your response professional and
relevant to the question and the context of the discussion thread! This
is not your personal mailing list!
On 3/17/22 4:16 PM, Gourav Sengupta wrote:
> Dearest Sean,
>
> I think in that meeting you also mentioned about your company Myrrix :)
>
> Though covered in 2011 the fundamentals on distributed computing and
> explanations of algorithms is still wonderful.
>
> image.jpeg
>
> Thanks and Regards,
> Gourav Sengupta
>
> On Thu, Mar 17, 2022 at 7:31 PM Sean Owen <sr...@gmail.com> wrote:
>
> (Thank you, not sure that was me though)
> I don't know of plans to expose the streaming impls in ML, as they
> still work fine in MLlib and they also don't come up much.
> Continuous training is relatively rare, maybe under-appreciated,
> but rare in practice.
>
> On Thu, Mar 17, 2022 at 1:57 PM Gourav Sengupta
> <go...@gmail.com> wrote:
>
> Dear friends,
>
> a few years ago, I was in a London meetup seeing Sean (Owen)
> demonstrate how we can try to predict the gender of
> individuals who are responding to tweets after accepting
> privacy agreements, in case I am not wrong.
>
> It was real time, it was spectacular, and it was the
> presentation that set me into data science and its applications.
>
> Thanks Sean! :)
>
> Regards,
> Gourav Sengupta
>
>
>
>
> On Tue, Mar 15, 2022 at 9:39 PM Artemis User
> <ar...@dtechspace.com> wrote:
>
> Thanks Sean! Well, it looks like we have to abandon our
> structured streaming model to use DStream for this, or do
> you see possibility to use structured streaming with ml
> instead of mllib?
>
> On 3/15/22 4:51 PM, Sean Owen wrote:
>> There is a streaming k-means example in Spark.
>> https://spark.apache.org/docs/latest/mllib-clustering.html#streaming-k-means
>>
>> On Tue, Mar 15, 2022, 3:46 PM Artemis User
>> <ar...@dtechspace.com> wrote:
>>
>> Has anyone done any experiments of training an ML
>> model using stream
>> data? especially for unsupervised models? Any
>> suggestions/references
>> are highly appreciated...
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>
>
Re: Continuous ML model training in stream mode
Posted by Gourav Sengupta <go...@gmail.com>.
Dearest Sean,
I think in that meeting you also mentioned about your company Myrrix :)
Though covered in 2011 the fundamentals on distributed computing and
explanations of algorithms is still wonderful.
[image: image.jpeg]
Thanks and Regards,
Gourav Sengupta
On Thu, Mar 17, 2022 at 7:31 PM Sean Owen <sr...@gmail.com> wrote:
> (Thank you, not sure that was me though)
> I don't know of plans to expose the streaming impls in ML, as they still
> work fine in MLlib and they also don't come up much. Continuous training is
> relatively rare, maybe under-appreciated, but rare in practice.
>
> On Thu, Mar 17, 2022 at 1:57 PM Gourav Sengupta <go...@gmail.com>
> wrote:
>
>> Dear friends,
>>
>> a few years ago, I was in a London meetup seeing Sean (Owen) demonstrate
>> how we can try to predict the gender of individuals who are responding to
>> tweets after accepting privacy agreements, in case I am not wrong.
>>
>> It was real time, it was spectacular, and it was the presentation that
>> set me into data science and its applications.
>>
>> Thanks Sean! :)
>>
>> Regards,
>> Gourav Sengupta
>>
>>
>>
>>
>> On Tue, Mar 15, 2022 at 9:39 PM Artemis User <ar...@dtechspace.com>
>> wrote:
>>
>>> Thanks Sean! Well, it looks like we have to abandon our structured
>>> streaming model to use DStream for this, or do you see possibility to use
>>> structured streaming with ml instead of mllib?
>>>
>>> On 3/15/22 4:51 PM, Sean Owen wrote:
>>>
>>> There is a streaming k-means example in Spark.
>>> https://spark.apache.org/docs/latest/mllib-clustering.html#streaming-k-means
>>>
>>> On Tue, Mar 15, 2022, 3:46 PM Artemis User <ar...@dtechspace.com>
>>> wrote:
>>>
>>>> Has anyone done any experiments of training an ML model using stream
>>>> data? especially for unsupervised models? Any suggestions/references
>>>> are highly appreciated...
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>>>
>>>>
>>>
Re: Continuous ML model training in stream mode
Posted by Bitfox <bi...@bitfox.top>.
For online recommendation systems, continuous training is needed. :)
And we are a living video player, the content is changing every minute, so
a real time rec system is the must.
On Fri, Mar 18, 2022 at 3:31 AM Sean Owen <sr...@gmail.com> wrote:
> (Thank you, not sure that was me though)
> I don't know of plans to expose the streaming impls in ML, as they still
> work fine in MLlib and they also don't come up much. Continuous training is
> relatively rare, maybe under-appreciated, but rare in practice.
>
> On Thu, Mar 17, 2022 at 1:57 PM Gourav Sengupta <go...@gmail.com>
> wrote:
>
>> Dear friends,
>>
>> a few years ago, I was in a London meetup seeing Sean (Owen) demonstrate
>> how we can try to predict the gender of individuals who are responding to
>> tweets after accepting privacy agreements, in case I am not wrong.
>>
>> It was real time, it was spectacular, and it was the presentation that
>> set me into data science and its applications.
>>
>> Thanks Sean! :)
>>
>> Regards,
>> Gourav Sengupta
>>
>>
>>
>>
>> On Tue, Mar 15, 2022 at 9:39 PM Artemis User <ar...@dtechspace.com>
>> wrote:
>>
>>> Thanks Sean! Well, it looks like we have to abandon our structured
>>> streaming model to use DStream for this, or do you see possibility to use
>>> structured streaming with ml instead of mllib?
>>>
>>> On 3/15/22 4:51 PM, Sean Owen wrote:
>>>
>>> There is a streaming k-means example in Spark.
>>> https://spark.apache.org/docs/latest/mllib-clustering.html#streaming-k-means
>>>
>>> On Tue, Mar 15, 2022, 3:46 PM Artemis User <ar...@dtechspace.com>
>>> wrote:
>>>
>>>> Has anyone done any experiments of training an ML model using stream
>>>> data? especially for unsupervised models? Any suggestions/references
>>>> are highly appreciated...
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>>>
>>>>
>>>
Re: Continuous ML model training in stream mode
Posted by Sean Owen <sr...@gmail.com>.
(Thank you, not sure that was me though)
I don't know of plans to expose the streaming impls in ML, as they still
work fine in MLlib and they also don't come up much. Continuous training is
relatively rare, maybe under-appreciated, but rare in practice.
On Thu, Mar 17, 2022 at 1:57 PM Gourav Sengupta <go...@gmail.com>
wrote:
> Dear friends,
>
> a few years ago, I was in a London meetup seeing Sean (Owen) demonstrate
> how we can try to predict the gender of individuals who are responding to
> tweets after accepting privacy agreements, in case I am not wrong.
>
> It was real time, it was spectacular, and it was the presentation that set
> me into data science and its applications.
>
> Thanks Sean! :)
>
> Regards,
> Gourav Sengupta
>
>
>
>
> On Tue, Mar 15, 2022 at 9:39 PM Artemis User <ar...@dtechspace.com>
> wrote:
>
>> Thanks Sean! Well, it looks like we have to abandon our structured
>> streaming model to use DStream for this, or do you see possibility to use
>> structured streaming with ml instead of mllib?
>>
>> On 3/15/22 4:51 PM, Sean Owen wrote:
>>
>> There is a streaming k-means example in Spark.
>> https://spark.apache.org/docs/latest/mllib-clustering.html#streaming-k-means
>>
>> On Tue, Mar 15, 2022, 3:46 PM Artemis User <ar...@dtechspace.com>
>> wrote:
>>
>>> Has anyone done any experiments of training an ML model using stream
>>> data? especially for unsupervised models? Any suggestions/references
>>> are highly appreciated...
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>>
>>>
>>
Re: Continuous ML model training in stream mode
Posted by Gourav Sengupta <go...@gmail.com>.
Dear friends,
a few years ago, I was in a London meetup seeing Sean (Owen) demonstrate
how we can try to predict the gender of individuals who are responding to
tweets after accepting privacy agreements, in case I am not wrong.
It was real time, it was spectacular, and it was the presentation that set
me into data science and its applications.
Thanks Sean! :)
Regards,
Gourav Sengupta
On Tue, Mar 15, 2022 at 9:39 PM Artemis User <ar...@dtechspace.com> wrote:
> Thanks Sean! Well, it looks like we have to abandon our structured
> streaming model to use DStream for this, or do you see possibility to use
> structured streaming with ml instead of mllib?
>
> On 3/15/22 4:51 PM, Sean Owen wrote:
>
> There is a streaming k-means example in Spark.
> https://spark.apache.org/docs/latest/mllib-clustering.html#streaming-k-means
>
> On Tue, Mar 15, 2022, 3:46 PM Artemis User <ar...@dtechspace.com> wrote:
>
>> Has anyone done any experiments of training an ML model using stream
>> data? especially for unsupervised models? Any suggestions/references
>> are highly appreciated...
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>
>>
>
Re: Continuous ML model training in stream mode
Posted by Artemis User <ar...@dtechspace.com>.
Thanks Sean! Well, it looks like we have to abandon our structured
streaming model to use DStream for this, or do you see possibility to
use structured streaming with ml instead of mllib?
On 3/15/22 4:51 PM, Sean Owen wrote:
> There is a streaming k-means example in Spark.
> https://spark.apache.org/docs/latest/mllib-clustering.html#streaming-k-means
>
> On Tue, Mar 15, 2022, 3:46 PM Artemis User <ar...@dtechspace.com> wrote:
>
> Has anyone done any experiments of training an ML model using stream
> data? especially for unsupervised models? Any
> suggestions/references
> are highly appreciated...
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
Re: Continuous ML model training in stream mode
Posted by Sean Owen <sr...@gmail.com>.
There is a streaming k-means example in Spark.
https://spark.apache.org/docs/latest/mllib-clustering.html#streaming-k-means
On Tue, Mar 15, 2022, 3:46 PM Artemis User <ar...@dtechspace.com> wrote:
> Has anyone done any experiments of training an ML model using stream
> data? especially for unsupervised models? Any suggestions/references
> are highly appreciated...
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>
Re: Continuous ML model training in stream mode
Posted by Bitfox <bi...@bitfox.top>.
we are keeping the training with the input content from a streaming. But
the framework is tensorflow not spark.
On Wed, Mar 16, 2022 at 4:46 AM Artemis User <ar...@dtechspace.com> wrote:
> Has anyone done any experiments of training an ML model using stream
> data? especially for unsupervised models? Any suggestions/references
> are highly appreciated...
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>