You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Jaonary Rabarisoa <ja...@gmail.com> on 2014/02/15 08:43:20 UTC
Getting started using spark for computer vision and video analytics
Hi all,
We are new to spark and other related things like Hadoop and HDFS. Spark
documentations and specifications
seem to say that it can be an intersecting tool for real time and batch
video analytics. In our case
analytics means :
- detecting feature inside video stream (object, face, ...)
- extract feature corresponding to a given metadata (ex third parties
object bounding box)
- tracking these features across video frames
- learning from these features (clustering, classification, ...)
Nowadays there are many work that use map reduce pattern to do these
computations on a large video data and
we like to use spark for that because of its promising performance.
The difficult part is, how to get started ? It is not clear how can we
transform a video live stream (H264 encoded, rtsp protocol) and video files
into a RDD that spark can consume ? How should we store the video files
for batch processing ? Currently these files are on a streaming server.
Is it easy to extend the sliding window for spark streaming in 3D (space +
time) : a 2D sliding window inside each frame for feature detection
and the time sliding widow as usual.
Any tips and suggestion will be helpful,
Best regards,
Jaonary
Re: Getting started using spark for computer vision and video analytics
Posted by Tathagata Das <ta...@gmail.com>.
1. Yes, Spark Streaming can process many datastreams in parallel, specially
if all the datastreams are of the same type and all the streams get merged
and processed together through the same operations. You should be able to
use zeromq<http://spark.incubator.apache.org/docs/latest/api/external/zeromq/index.html#org.apache.spark.streaming.zeromq.ZeroMQUtils$>
streams
by linking to spark-streaming-zeromq<http://spark.incubator.apache.org/docs/latest/streaming-programming-guide.html#linking>
.
2. You dont have to store video data as text! Hadoop has these InputFormat
and OutputFormat API which allows you plugin in your arbitrary types. And
since Spark uses Hadoop library you can use any existing Hadoop format in
Spark. A good place to start would be learning about Hadoop InputFormat and
OutputFormat, specially the ones related to binary data like
SequenceOutputFormat<https://www.inkling.com/read/hadoop-definitive-guide-tom-white-3rd/chapter-7/output-formats>
.
TD
On Fri, Feb 21, 2014 at 2:47 AM, Jaonary Rabarisoa <ja...@gmail.com>wrote:
> Hi all,
>
> To answer Austen, yes we do computer vision + analytics. Basically, for
> every video stream we run a process that extract frames, detect and
> track features in each extracted frames. Save these features in a database
> engine ( we use mogodb) for further data analytics. Clearly, our current
> implementation doesn't scale in number of stream to process and in number
> feature to be extracted that would potentially very large. The point is
> that we have and experience on computer vision and machine learning but not
> on distributed computing, Hadoop and related things.
>
> The first idea to deal with live video stream is to use spark streaming
> with zeromq. We already have a streaming server based on zeromq that
> streams decoded frames. So no need to decode video frames inside spark. The
> question is, does spark streaming can process many data stream in parallel
> ? For the offline processing, having the ability to decode video files from
> hdfs would be nice. There some works on this but it is clearly beyond
> our experience. So, is storing video decoded frames as a text file is a
> descent alternative ? For example as (timestamp, raw pixel value). Or as a
> binary file to reduce the size but we don't find any document on reading
> binary files with spark ?
>
>
> On Sunday, February 16, 2014, Austen McRae <au...@gmail.com> wrote:
>
>> Hey Jaonary,
>>
>> Xuefeng links two good documents, but unfortunately I'm not sure if thats
>> what you're looking for. Namely, you're looking to do *computer vision*+ analytics; his links are for companies that do video + analytics based on
>> how videos are doing regarding plays, users, engagement, etc, but strictly
>> *not* computer vision related (I say this as a tech lead at Ooyala for
>> Analytics - YMMV regarding Conviva :).
>>
>> Regarding you're specific question, I'd ask if you've done this type of
>> analytics without using Spark. Spark, to the best of my knowledge, doesn't
>> have any built-in support for video decoupling or receiving of streams. If
>> you don't have this part tackled yet, I would focus on that first, as thats
>> a Hard Problem.
>>
>> If you do, then we can discuss how it might fit into using Spark, and
>> Streaming. My initial thought process would be to process a movie at a
>> single node/stream (this is due to using H264 - you cannot parallelize this
>> perfectly based on the timestamp of a frame; you'll have to know where the
>> I frames are and then read in the following P/B frames as one entity, as
>> those require the I frame to have a full image to process). Once you've
>> extracted the feature set from the video, then spark is a perfect use case
>> given its ability to hook into MLib and take advantage of quite a few java
>> libraries for doing clustering and the like (or python's equivalents).
>> Specifically, RDD's are *not* what you want to use for holding the
>> video itself; instead it should hold the output of processing a video image.
>>
>> For doing the actual Computer vision components, the best I can recommend
>> is OpenCV, with a loose combination of ffmpeg - those are all C based
>> however, and I'm not familiar with any Java equivalents or bridges between
>> them (python might be closer, but my usage of both has been via strait C).
>>
>> Finally, for storing the video streams for later use (assuming you want
>> to do this), I would recommend H264 with a strict enforcement of I frames
>> at certain time intervals. Going a step further would be to use H264
>> wrapped around HLS (Apple's format, which is a simplified version of
>> MPEG2TS), with a small chunk size. This will give you easy-to-parallelize
>> chunks of a video for re-doing feature extraction.
>>
>> Hopefully this gives you some answers and a direction on where to go
>> next; good luck!
>>
>>
>> On Sat, Feb 15, 2014 at 6:45 PM, Xuefeng Wu <be...@gmail.com> wrote:
>>
>>> Hi Jasonary,
>>>
>>> It is a real good explore,
>>>
>>> There are some cases about video analytics in spark, maybe you already
>>> know.
>>> Both of them can do analytics in Hive before move to spark.
>>>
>>>
>>> http://ampcamp.berkeley.edu/wp-content/uploads/2012/06/dilip-joseph-amp-camp-2012-conviva-using-spark.pdf
>>>
>>> http://www.computing.co.uk/ctg/analysis/2291474/a-big-data-case-study-ooyala-s-realtime-video-analytics
>>>
>>>
>>>
>>> On Sat, Feb 15, 2014 at 3:43 PM, Jaonary Rabarisoa <ja...@gmail.com>wrote:
>>>
>>>> Hi all,
>>>>
>>>> We are new to spark and other related things like Hadoop and HDFS.
>>>> Spark documentations and specifications
>>>> seem to say that it can be an intersecting tool for real time and batch
>>>> video analytics. In our case
>>>> analytics means :
>>>> - detecting feature inside video stream (object, face, ...)
>>>> - extract feature corresponding to a given metadata (ex third parties
>>>> object bounding box)
>>>> - tracking these features across video frames
>>>> - learning from these features (clustering, classification, ...)
>>>> Nowadays there are many work that use map reduce pattern to do these
>>>> computations on a large video data and
>>>> we like to use spark for that because of its promising performance.
>>>> The difficult part is, how to get started ? It is not clear how can we
>>>> transform a video live stream (H264 encoded, rtsp protocol) and video files
>>>> into a RDD that spark can consume ? How should we store the video
>>>> files for batch processing ? Currently these files are on a streaming
>>>> server.
>>>> Is it easy to extend the sliding window for spark streaming in 3D
>>>> (space + time) : a 2D sliding window inside each frame for feature detection
>>>> and the time sliding widow as usual.
>>>>
>>>> Any tips and suggestion will be helpful,
>>>>
>>>> Best regards,
>>>>
>>>>
>>>> Jaonary
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> ~Yours, Xuefeng Wu/吴雪峰 敬上
>>>
>>>
>>
Re: Getting started using spark for computer vision and video analytics
Posted by Jaonary Rabarisoa <ja...@gmail.com>.
Hi all,
To answer Austen, yes we do computer vision + analytics. Basically, for
every video stream we run a process that extract frames, detect and
track features in each extracted frames. Save these features in a database
engine ( we use mogodb) for further data analytics. Clearly, our current
implementation doesn't scale in number of stream to process and in number
feature to be extracted that would potentially very large. The point is
that we have and experience on computer vision and machine learning but not
on distributed computing, Hadoop and related things.
The first idea to deal with live video stream is to use spark streaming
with zeromq. We already have a streaming server based on zeromq that
streams decoded frames. So no need to decode video frames inside spark. The
question is, does spark streaming can process many data stream in parallel
? For the offline processing, having the ability to decode video files from
hdfs would be nice. There some works on this but it is clearly beyond
our experience. So, is storing video decoded frames as a text file is a
descent alternative ? For example as (timestamp, raw pixel value). Or as a
binary file to reduce the size but we don't find any document on reading
binary files with spark ?
On Sunday, February 16, 2014, Austen McRae <au...@gmail.com> wrote:
> Hey Jaonary,
>
> Xuefeng links two good documents, but unfortunately I'm not sure if thats
> what you're looking for. Namely, you're looking to do *computer vision*+ analytics; his links are for companies that do video + analytics based on
> how videos are doing regarding plays, users, engagement, etc, but strictly
> *not* computer vision related (I say this as a tech lead at Ooyala for
> Analytics - YMMV regarding Conviva :).
>
> Regarding you're specific question, I'd ask if you've done this type of
> analytics without using Spark. Spark, to the best of my knowledge, doesn't
> have any built-in support for video decoupling or receiving of streams. If
> you don't have this part tackled yet, I would focus on that first, as thats
> a Hard Problem.
>
> If you do, then we can discuss how it might fit into using Spark, and
> Streaming. My initial thought process would be to process a movie at a
> single node/stream (this is due to using H264 - you cannot parallelize this
> perfectly based on the timestamp of a frame; you'll have to know where the
> I frames are and then read in the following P/B frames as one entity, as
> those require the I frame to have a full image to process). Once you've
> extracted the feature set from the video, then spark is a perfect use case
> given its ability to hook into MLib and take advantage of quite a few java
> libraries for doing clustering and the like (or python's equivalents).
> Specifically, RDD's are *not* what you want to use for holding the video
> itself; instead it should hold the output of processing a video image.
>
> For doing the actual Computer vision components, the best I can recommend
> is OpenCV, with a loose combination of ffmpeg - those are all C based
> however, and I'm not familiar with any Java equivalents or bridges between
> them (python might be closer, but my usage of both has been via strait C).
>
> Finally, for storing the video streams for later use (assuming you want to
> do this), I would recommend H264 with a strict enforcement of I frames at
> certain time intervals. Going a step further would be to use H264 wrapped
> around HLS (Apple's format, which is a simplified version of MPEG2TS), with
> a small chunk size. This will give you easy-to-parallelize chunks of a
> video for re-doing feature extraction.
>
> Hopefully this gives you some answers and a direction on where to go next;
> good luck!
>
>
> On Sat, Feb 15, 2014 at 6:45 PM, Xuefeng Wu <benewu@gmail.com<javascript:_e(%7B%7D,'cvml','benewu@gmail.com');>
> > wrote:
>
>> Hi Jasonary,
>>
>> It is a real good explore,
>>
>> There are some cases about video analytics in spark, maybe you already
>> know.
>> Both of them can do analytics in Hive before move to spark.
>>
>>
>> http://ampcamp.berkeley.edu/wp-content/uploads/2012/06/dilip-joseph-amp-camp-2012-conviva-using-spark.pdf
>>
>> http://www.computing.co.uk/ctg/analysis/2291474/a-big-data-case-study-ooyala-s-realtime-video-analytics
>>
>>
>>
>> On Sat, Feb 15, 2014 at 3:43 PM, Jaonary Rabarisoa <jaonary@gmail.com<javascript:_e(%7B%7D,'cvml','jaonary@gmail.com');>
>> > wrote:
>>
>>> Hi all,
>>>
>>> We are new to spark and other related things like Hadoop and HDFS. Spark
>>> documentations and specifications
>>> seem to say that it can be an intersecting tool for real time and batch
>>> video analytics. In our case
>>> analytics means :
>>> - detecting feature inside video stream (object, face, ...)
>>> - extract feature corresponding to a given metadata (ex third parties
>>> object bounding box)
>>> - tracking these features across video frames
>>> - learning from these features (clustering, classification, ...)
>>> Nowadays there are many work that use map reduce pattern to do these
>>> computations on a large video data and
>>> we like to use spark for that because of its promising performance.
>>> The difficult part is, how to get started ? It is not clear how can we
>>> transform a video live stream (H264 encoded, rtsp protocol) and video files
>>> into a RDD that spark can consume ? How should we store the video files
>>> for batch processing ? Currently these files are on a streaming server.
>>> Is it easy to extend the sliding window for spark streaming in 3D (space
>>> + time) : a 2D sliding window inside each frame for feature detection
>>> and the time sliding widow as usual.
>>>
>>> Any tips and suggestion will be helpful,
>>>
>>> Best regards,
>>>
>>>
>>> Jaonary
>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
>>
>> --
>>
>> ~Yours, Xuefeng Wu/吴雪峰 敬上
>>
>>
>
Re: Getting started using spark for computer vision and video analytics
Posted by Austen McRae <au...@gmail.com>.
Hey Jaonary,
Xuefeng links two good documents, but unfortunately I'm not sure if thats
what you're looking for. Namely, you're looking to do *computer vision* +
analytics; his links are for companies that do video + analytics based on
how videos are doing regarding plays, users, engagement, etc, but strictly
*not* computer vision related (I say this as a tech lead at Ooyala for
Analytics - YMMV regarding Conviva :).
Regarding you're specific question, I'd ask if you've done this type of
analytics without using Spark. Spark, to the best of my knowledge, doesn't
have any built-in support for video decoupling or receiving of streams. If
you don't have this part tackled yet, I would focus on that first, as thats
a Hard Problem.
If you do, then we can discuss how it might fit into using Spark, and
Streaming. My initial thought process would be to process a movie at a
single node/stream (this is due to using H264 - you cannot parallelize this
perfectly based on the timestamp of a frame; you'll have to know where the
I frames are and then read in the following P/B frames as one entity, as
those require the I frame to have a full image to process). Once you've
extracted the feature set from the video, then spark is a perfect use case
given its ability to hook into MLib and take advantage of quite a few java
libraries for doing clustering and the like (or python's equivalents).
Specifically, RDD's are *not* what you want to use for holding the video
itself; instead it should hold the output of processing a video image.
For doing the actual Computer vision components, the best I can recommend
is OpenCV, with a loose combination of ffmpeg - those are all C based
however, and I'm not familiar with any Java equivalents or bridges between
them (python might be closer, but my usage of both has been via strait C).
Finally, for storing the video streams for later use (assuming you want to
do this), I would recommend H264 with a strict enforcement of I frames at
certain time intervals. Going a step further would be to use H264 wrapped
around HLS (Apple's format, which is a simplified version of MPEG2TS), with
a small chunk size. This will give you easy-to-parallelize chunks of a
video for re-doing feature extraction.
Hopefully this gives you some answers and a direction on where to go next;
good luck!
On Sat, Feb 15, 2014 at 6:45 PM, Xuefeng Wu <be...@gmail.com> wrote:
> Hi Jasonary,
>
> It is a real good explore,
>
> There are some cases about video analytics in spark, maybe you already
> know.
> Both of them can do analytics in Hive before move to spark.
>
>
> http://ampcamp.berkeley.edu/wp-content/uploads/2012/06/dilip-joseph-amp-camp-2012-conviva-using-spark.pdf
>
> http://www.computing.co.uk/ctg/analysis/2291474/a-big-data-case-study-ooyala-s-realtime-video-analytics
>
>
>
> On Sat, Feb 15, 2014 at 3:43 PM, Jaonary Rabarisoa <ja...@gmail.com>wrote:
>
>> Hi all,
>>
>> We are new to spark and other related things like Hadoop and HDFS. Spark
>> documentations and specifications
>> seem to say that it can be an intersecting tool for real time and batch
>> video analytics. In our case
>> analytics means :
>> - detecting feature inside video stream (object, face, ...)
>> - extract feature corresponding to a given metadata (ex third parties
>> object bounding box)
>> - tracking these features across video frames
>> - learning from these features (clustering, classification, ...)
>> Nowadays there are many work that use map reduce pattern to do these
>> computations on a large video data and
>> we like to use spark for that because of its promising performance.
>> The difficult part is, how to get started ? It is not clear how can we
>> transform a video live stream (H264 encoded, rtsp protocol) and video files
>> into a RDD that spark can consume ? How should we store the video files
>> for batch processing ? Currently these files are on a streaming server.
>> Is it easy to extend the sliding window for spark streaming in 3D (space
>> + time) : a 2D sliding window inside each frame for feature detection
>> and the time sliding widow as usual.
>>
>> Any tips and suggestion will be helpful,
>>
>> Best regards,
>>
>>
>> Jaonary
>>
>>
>>
>>
>>
>>
>
>
>
> --
>
> ~Yours, Xuefeng Wu/吴雪峰 敬上
>
>
Re: Getting started using spark for computer vision and video analytics
Posted by Xuefeng Wu <be...@gmail.com>.
Hi Jasonary,
It is a real good explore,
There are some cases about video analytics in spark, maybe you already know.
Both of them can do analytics in Hive before move to spark.
http://ampcamp.berkeley.edu/wp-content/uploads/2012/06/dilip-joseph-amp-camp-2012-conviva-using-spark.pdf
http://www.computing.co.uk/ctg/analysis/2291474/a-big-data-case-study-ooyala-s-realtime-video-analytics
On Sat, Feb 15, 2014 at 3:43 PM, Jaonary Rabarisoa <ja...@gmail.com>wrote:
> Hi all,
>
> We are new to spark and other related things like Hadoop and HDFS. Spark
> documentations and specifications
> seem to say that it can be an intersecting tool for real time and batch
> video analytics. In our case
> analytics means :
> - detecting feature inside video stream (object, face, ...)
> - extract feature corresponding to a given metadata (ex third parties
> object bounding box)
> - tracking these features across video frames
> - learning from these features (clustering, classification, ...)
> Nowadays there are many work that use map reduce pattern to do these
> computations on a large video data and
> we like to use spark for that because of its promising performance.
> The difficult part is, how to get started ? It is not clear how can we
> transform a video live stream (H264 encoded, rtsp protocol) and video files
> into a RDD that spark can consume ? How should we store the video files
> for batch processing ? Currently these files are on a streaming server.
> Is it easy to extend the sliding window for spark streaming in 3D (space +
> time) : a 2D sliding window inside each frame for feature detection
> and the time sliding widow as usual.
>
> Any tips and suggestion will be helpful,
>
> Best regards,
>
>
> Jaonary
>
>
>
>
>
>
--
~Yours, Xuefeng Wu/吴雪峰 敬上