Multiple Filter Effiency

[zkidkid <zk...@gmail.com>]

Hi,
Currently I am trying to count on a document with multiple filter.
Let say, here is my document:

//user field1 field2 field3
user1 0 0 1
user2 0 1 0
user3 0 0 0

I want to count on user.log for some filters like this:

Filter1: field1 == 0 & field 2 = 0
Filter2: field1 == 0 & field 3 = 1
Filter3: field1 == 0 & field 3 = 0
...
and total line.

I have tried and I found that I couldn't use "group by" or "map then reduce"
because a line could match two or more filter.

My idea now is "foreach" line and then maintain a outsite counter service.

Forexample:

        JavaRDD<String> textFile = sc.textFile(hdfs, 10);
        long start = System.currentTimeMillis();

        textFile.foreach(new VoidFunction<String>() {

            public void call(String s) {
               foreach(MyFilter filter: MyFilters){
                       if(filter.match(s)) filter.increaseOwnCounter();
               }
            }
        });


I would happy if there have another way to do it, any help is appreciate.
Thanks in advance.





--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Multiple-Filter-Effiency-tp20701.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: Multiple Filter Effiency

[Imran Rashid <im...@therashids.com>]

I think accumulators do exactly what you want.

(Scala syntax below, I'm just not familiar with the Java equivalent ...)

val f1counts = sc.accumulator (0)
val f2counts = sc.accumulator (0)
val f3counts = sc.accumulator (0)

textfile.foreach { s =>
  if(f1matches) f1counts += 1
  ...
}

Note that you could also do a normal map reduce even though a record might
match more than one filter.  In the scala api you can use flatmap to output
zero or more records:

textfile.flatmap { s =>
  Seq (
     (if (f1matches) Some ("f1" -> 1) else None),
     ...
    ).flatten
}.reduceByKey { _ + _ }
On Dec 16, 2014 2:07 AM, "zkidkid" <zk...@gmail.com> wrote:

> Hi,
> Currently I am trying to count on a document with multiple filter.
> Let say, here is my document:
>
> //user field1 field2 field3
> user1 0 0 1
> user2 0 1 0
> user3 0 0 0
>
> I want to count on user.log for some filters like this:
>
> Filter1: field1 == 0 & field 2 = 0
> Filter2: field1 == 0 & field 3 = 1
> Filter3: field1 == 0 & field 3 = 0
> ...
> and total line.
>
> I have tried and I found that I couldn't use "group by" or "map then
> reduce"
> because a line could match two or more filter.
>
> My idea now is "foreach" line and then maintain a outsite counter service.
>
> Forexample:
>
>         JavaRDD<String> textFile = sc.textFile(hdfs, 10);
>         long start = System.currentTimeMillis();
>
>         textFile.foreach(new VoidFunction<String>() {
>
>             public void call(String s) {
>                foreach(MyFilter filter: MyFilters){
>                        if(filter.match(s)) filter.increaseOwnCounter();
>                }
>             }
>         });
>
>
> I would happy if there have another way to do it, any help is appreciate.
> Thanks in advance.
>
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Multiple-Filter-Effiency-tp20701.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>