You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Cassa L <lc...@gmail.com> on 2015/11/19 01:26:32 UTC

how to group timestamp data and filter on it

Hi,
I have a data stream (JavaDStream) in following format-
timestamp=second1,  map(key1=value1, key2=value2)
timestamp=second2,map(key1=value3, key2=value4)
timestamp=second2, map(key1=value1, key2=value5)


I want to group data by 'timestamp' first and then filter each RDD for
Key1=value1 or key1=value3 etc.

Each of above row represent POJO in RDD like:
public class Data{
    long timestamp;
    Map<String, String> map;
}

How do do this in spark? I am trying to figure out if I need to use map or
flatMap etc?

Thanks,
LCassa

RE: how to group timestamp data and filter on it

Posted by Tim Barthram <Ti...@iag.com.au>.
Hi LCassa,

Try:

Map to pair, then reduce by key.

The spark documentation is a pretty good reference for this & there are plenty of word count examples on the internet.

Warm regards,
TimB


From: Cassa L [mailto:lcassa8@gmail.com]
Sent: Thursday, 19 November 2015 11:27 AM
To: user
Subject: how to group timestamp data and filter on it

Hi,
I have a data stream (JavaDStream) in following format-
timestamp=second1,  map(key1=value1, key2=value2)
timestamp=second2,map(key1=value3, key2=value4)
timestamp=second2, map(key1=value1, key2=value5)

I want to group data by 'timestamp' first and then filter each RDD for Key1=value1 or key1=value3 etc.
Each of above row represent POJO in RDD like:
public class Data{
    long timestamp;
    Map<String, String> map;
}
How do do this in spark? I am trying to figure out if I need to use map or flatMap etc?
Thanks,
LCassa


_____________________________________________________________________

The information transmitted in this message and its attachments (if any) is intended 
only for the person or entity to which it is addressed.
The message may contain confidential and/or privileged material. Any review, 
retransmission, dissemination or other use of, or taking of any action in reliance 
upon this information, by persons or entities other than the intended recipient is 
prohibited.

If you have received this in error, please contact the sender and delete this e-mail 
and associated material from any computer.

The intended recipient of this e-mail may only use, reproduce, disclose or distribute 
the information contained in this e-mail and any attached files, with the permission 
of the sender.

This message has been scanned for viruses.
_____________________________________________________________________