Streaming JSON From S3?

[Paul Snively <ps...@icloud.com>]

Hi everyone,

I have a hopefully-quick question about streaming.

The StreamingContext has handy helper functions to construct several different kinds of streams, and I  understand we have access to any Hadoop store as a source for our streams. So I guess I really have two questions:

1) If I have an S3 folder full of JSON documents, is there a straightforward way to construct a stream of JSON objects, i.e. to have the stream contain parsed objects?
2) Does streaming from a folder (on any store) mean the folder is monitored somehow and new entries that appear are added to the stream?

If the answer to either of these is "no," any guidance on implementing such behavior would be greatly appreciated.

Best regards,
Paul

Re: Streaming JSON From S3?

[Evan Sparks <ev...@gmail.com>]

You can always use some non-split table file format (eg gzip) and then
a binary input format to get the "file at a time" behavior you're
looking for.

On Aug 21, 2013, at 9:57 PM, Matei Zaharia <ma...@gmail.com> wrote:

> Hi Paul,
>
> On Aug 21, 2013, at 6:11 PM, Paul Snively <ps...@icloud.com> wrote:
>
>>> Just to understand, are you trying to do a real-time application (which is what the streaming in Spark Streaming is for), or just to read an input file into a batch job?
>>
>> Well, it's an interesting case. I'm trying to take advantage of Spark Streaming's scanning of sources to automatically process new content, and possibly its sliding window support, e.g. "do something with every 5 RDDs in the stream." So it's not so much that the requirements are real time—on the contrary, the processing "in the middle" will be pretty heavyweight—but rather that streaming offers a couple of desirable ancillary features.
>
> Got it; that's fine as a use case for Spark Streaming
>
>> That's essentially what I expected. When you say "stream of Strings," is each String the entire contents of a file? If so, that would be perfectly suitable.
>
> No, unfortunately each String is one line of text. You'd have to create a Hadoop InputFormat that returns one record per file if you wanted that. Maybe we should add that as a feature in Spark by default, because it does seem like a useful way to run it.
>
> Matei
>

Re: Streaming JSON From S3?

[Matei Zaharia <ma...@gmail.com>]

Hi Paul,

On Aug 21, 2013, at 6:11 PM, Paul Snively <ps...@icloud.com> wrote:

>> Just to understand, are you trying to do a real-time application (which is what the streaming in Spark Streaming is for), or just to read an input file into a batch job?
> 
> Well, it's an interesting case. I'm trying to take advantage of Spark Streaming's scanning of sources to automatically process new content, and possibly its sliding window support, e.g. "do something with every 5 RDDs in the stream." So it's not so much that the requirements are real time—on the contrary, the processing "in the middle" will be pretty heavyweight—but rather that streaming offers a couple of desirable ancillary features.

Got it; that's fine as a use case for Spark Streaming

> That's essentially what I expected. When you say "stream of Strings," is each String the entire contents of a file? If so, that would be perfectly suitable.

No, unfortunately each String is one line of text. You'd have to create a Hadoop InputFormat that returns one record per file if you wanted that. Maybe we should add that as a feature in Spark by default, because it does seem like a useful way to run it.

Matei

Re: Streaming JSON From S3?

[Paul Snively <ps...@icloud.com>]

Hi Matei!

On Aug 20, 2013, at 11:41 AM, Matei Zaharia wrote:

> Just to understand, are you trying to do a real-time application (which is what the streaming in Spark Streaming is for), or just to read an input file into a batch job?

Well, it's an interesting case. I'm trying to take advantage of Spark Streaming's scanning of sources to automatically process new content, and possibly its sliding window support, e.g. "do something with every 5 RDDs in the stream." So it's not so much that the requirements are real time—on the contrary, the processing "in the middle" will be pretty heavyweight—but rather that streaming offers a couple of desirable ancillary features.

> For the latter, you can pass an s3n:// URL to any of Spark's file input methods (e.g. SparkContext.textFile). The easiest thing is if your JSON is configured to write one object per line -- in that case, you'd be able to use textFile to get an RDD of strings (one per line), and parse them in a map() function with your favorite JSON parser. If your objects span multiple lines, you probably need a custom Hadoop InputFormat class for JSON that you'd pass to SparkContext.hadoopFile. There might be formats like that out there. Basically look into how one would read these files in a Hadoop job.
> 
> If you are going for real-time on the other hand, you'd have to use StreamingContext.textFileStream, which watches a directory and keeps updating the stream as files are added. You can then call a map() on the stream of Strings you get to turn it into a stream of something else.

That's essentially what I expected. When you say "stream of Strings," is each String the entire contents of a file? If so, that would be perfectly suitable.

> 
> Matei

Thanks, and I look forward to meeting you at the boot camp!
Paul

Re: Streaming JSON From S3?

[Matei Zaharia <ma...@gmail.com>]

Hi Paul,

Just to understand, are you trying to do a real-time application (which is what the streaming in Spark Streaming is for), or just to read an input file into a batch job?

For the latter, you can pass an s3n:// URL to any of Spark's file input methods (e.g. SparkContext.textFile). The easiest thing is if your JSON is configured to write one object per line -- in that case, you'd be able to use textFile to get an RDD of strings (one per line), and parse them in a map() function with your favorite JSON parser. If your objects span multiple lines, you probably need a custom Hadoop InputFormat class for JSON that you'd pass to SparkContext.hadoopFile. There might be formats like that out there. Basically look into how one would read these files in a Hadoop job.

If you are going for real-time on the other hand, you'd have to use StreamingContext.textFileStream, which watches a directory and keeps updating the stream as files are added. You can then call a map() on the stream of Strings you get to turn it into a stream of something else.

Matei

On Aug 19, 2013, at 10:17 AM, Paul Snively <ps...@icloud.com> wrote:

> Hi everyone,
> 
> I have a hopefully-quick question about streaming.
> 
> The StreamingContext has handy helper functions to construct several different kinds of streams, and I  understand we have access to any Hadoop store as a source for our streams. So I guess I really have two questions:
> 
> 1) If I have an S3 folder full of JSON documents, is there a straightforward way to construct a stream of JSON objects, i.e. to have the stream contain parsed objects?
> 2) Does streaming from a folder (on any store) mean the folder is monitored somehow and new entries that appear are added to the stream?
> 
> If the answer to either of these is "no," any guidance on implementing such behavior would be greatly appreciated.
> 
> Best regards,
> Paul
>