You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by nitinkak001 <ni...@gmail.com> on 2015/02/13 17:49:05 UTC
Counters in Spark
I am trying to implement counters in Spark and I guess Accumulators are the
way to do it.
My motive is to update a counter in map function and access/reset it in the
driver code. However the /println/ statement at the end still yields value
0(It should 9). Am I doing something wrong?
def main(args : Array[String]){
val conf = new SparkConf().setAppName("SortedNeighbourhoodMatching")
val sc = new SparkContext(conf)
var counter = sc.accumulable(0, "Counter")
var inputFilePath = args(0)
val inputRDD = sc.textFile(inputFilePath)
inputRDD.map { x => {
counter += 1
} }
println(counter.value)
}
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Counters-in-Spark-tp21646.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Counters in Spark
Posted by Imran Rashid <ir...@cloudera.com>.
this is more-or-less the best you can do now, but as has been pointed out,
accumulators don't quite fit the bill for counters. There is an open issue
to do something better, but no progress on that so far
https://issues.apache.org/jira/browse/SPARK-603
On Fri, Feb 13, 2015 at 11:12 AM, Mark Hamstra <ma...@clearstorydata.com>
wrote:
> Except that transformations don't have an exactly-once guarantee, so this
> way of doing counters may produce different answers across various forms of
> failures and speculative execution.
>
> On Fri, Feb 13, 2015 at 8:56 AM, Sean McNamara <
> Sean.McNamara@webtrends.com> wrote:
>
>> .map is just a transformation, so no work will actually be performed
>> until something takes action against it. Try adding a .count(), like so:
>>
>> inputRDD.map { x => {
>> counter += 1
>> } }.count()
>>
>> In case it is helpful, here are the docs on what exactly the
>> transformations and actions are:
>> http://spark.apache.org/docs/1.2.0/programming-guide.html#transformations
>> http://spark.apache.org/docs/1.2.0/programming-guide.html#actions
>>
>> Cheers,
>>
>> Sean
>>
>>
>> On Feb 13, 2015, at 9:50 AM, nitinkak001 <ni...@gmail.com> wrote:
>>
>> I am trying to implement counters in Spark and I guess Accumulators are
>> the
>> way to do it.
>>
>> My motive is to update a counter in map function and access/reset it in
>> the
>> driver code. However the /println/ statement at the end still yields value
>> 0(It should 9). Am I doing something wrong?
>>
>> def main(args : Array[String]){
>>
>> val conf = new SparkConf().setAppName("SortedNeighbourhoodMatching")
>> val sc = new SparkContext(conf)
>> var counter = sc.accumulable(0, "Counter")
>> var inputFilePath = args(0)
>> val inputRDD = sc.textFile(inputFilePath)
>>
>> inputRDD.map { x => {
>> counter += 1
>> } }
>> println(counter.value)
>> }
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Counters-in-Spark-tp21646.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>>
>
Re: Counters in Spark
Posted by Mark Hamstra <ma...@clearstorydata.com>.
Except that transformations don't have an exactly-once guarantee, so this
way of doing counters may produce different answers across various forms of
failures and speculative execution.
On Fri, Feb 13, 2015 at 8:56 AM, Sean McNamara <Se...@webtrends.com>
wrote:
> .map is just a transformation, so no work will actually be performed
> until something takes action against it. Try adding a .count(), like so:
>
> inputRDD.map { x => {
> counter += 1
> } }.count()
>
> In case it is helpful, here are the docs on what exactly the
> transformations and actions are:
> http://spark.apache.org/docs/1.2.0/programming-guide.html#transformations
> http://spark.apache.org/docs/1.2.0/programming-guide.html#actions
>
> Cheers,
>
> Sean
>
>
> On Feb 13, 2015, at 9:50 AM, nitinkak001 <ni...@gmail.com> wrote:
>
> I am trying to implement counters in Spark and I guess Accumulators are the
> way to do it.
>
> My motive is to update a counter in map function and access/reset it in the
> driver code. However the /println/ statement at the end still yields value
> 0(It should 9). Am I doing something wrong?
>
> def main(args : Array[String]){
>
> val conf = new SparkConf().setAppName("SortedNeighbourhoodMatching")
> val sc = new SparkContext(conf)
> var counter = sc.accumulable(0, "Counter")
> var inputFilePath = args(0)
> val inputRDD = sc.textFile(inputFilePath)
>
> inputRDD.map { x => {
> counter += 1
> } }
> println(counter.value)
> }
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Counters-in-Spark-tp21646.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
>
Re: Counters in Spark
Posted by Sean McNamara <Se...@Webtrends.com>.
.map is just a transformation, so no work will actually be performed until something takes action against it. Try adding a .count(), like so:
inputRDD.map { x => {
counter += 1
} }.count()
In case it is helpful, here are the docs on what exactly the transformations and actions are:
http://spark.apache.org/docs/1.2.0/programming-guide.html#transformations
http://spark.apache.org/docs/1.2.0/programming-guide.html#actions
Cheers,
Sean
On Feb 13, 2015, at 9:50 AM, nitinkak001 <ni...@gmail.com>> wrote:
I am trying to implement counters in Spark and I guess Accumulators are the
way to do it.
My motive is to update a counter in map function and access/reset it in the
driver code. However the /println/ statement at the end still yields value
0(It should 9). Am I doing something wrong?
def main(args : Array[String]){
val conf = new SparkConf().setAppName("SortedNeighbourhoodMatching")
val sc = new SparkContext(conf)
var counter = sc.accumulable(0, "Counter")
var inputFilePath = args(0)
val inputRDD = sc.textFile(inputFilePath)
inputRDD.map { x => {
counter += 1
} }
println(counter.value)
}
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Counters-in-Spark-tp21646.html
Sent from the Apache Spark User List mailing list archive at Nabble.com<http://Nabble.com>.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org<ma...@spark.apache.org>
For additional commands, e-mail: user-help@spark.apache.org<ma...@spark.apache.org>