You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Kedarnath Dixit <ke...@persistent.com> on 2017/11/13 16:58:40 UTC
Use of Accumulators
Hi,
We need some way to toggle the flag of a variable in transformation.
We are thinking to make use of spark Accumulators for this purpose.
Can we use these as below:
Variables -> Initial Value
Variable1 -> 0
Variable2 -> 0
In one of the transformations if we need to make Variable2's value to 1. Can we achieve this using Accumulators? Please confirm.
Thanks!
With Regards,
~Kedar Dixit
kedarnath_dixit@persistent.com | @kedarsdixit | M +91 90499 15588 | T +91 (20) 6703 4783
Persistent Systems | Partners In Innovation | www.persistent.com
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
Re: Use of Accumulators
Posted by Kedarnath Dixit <ke...@persistent.com>.
Yes!
Thanks!
~Kedar Dixit
Bigdata Analytics at Persistent Systems Ltd.
________________________________
From: Holden Karau <ho...@pigscanfly.ca>
Sent: 14 November 2017 20:04:50
To: Kedarnath Dixit
Cc: user@spark.apache.org
Subject: Re: Use of Accumulators
And where do you want to read the toggle back from? On the driver?
On Tue, Nov 14, 2017 at 3:52 AM Kedarnath Dixit <ke...@persistent.com>> wrote:
Hi,
Inside the transformation if there is any change for the Variable’s associated data, I want to just toggle it saying there is some change while processing the data.
Please let me know if we can runtime do this.
Thanks!
~Kedar Dixit
Bigdata Analytics at Persistent Systems Ltd.
From: Holden Karau [via Apache Spark User List] [mailto:ml+s1001560n29995h59@n3.nabble.com]
Sent: Tuesday, November 14, 2017 1:16 PM
To: Kedarnath Dixit <ke...@persistent.com>>
Subject: Re: Use of Accumulators
So you want to set an accumulator to 1 after a transformation has fully completed? Or what exactly do you want to do?
On Mon, Nov 13, 2017 at 9:47 PM vaquar khan <[hidden email]<http:///user/SendEmail.jtp?type=node&node=29995&i=0>> wrote:
Confirmed ,you can use Accumulators :)
Regards,
Vaquar khan
On Mon, Nov 13, 2017 at 10:58 AM, Kedarnath Dixit <[hidden email]<http:///user/SendEmail.jtp?type=node&node=29995&i=1>> wrote:
Hi,
We need some way to toggle the flag of a variable in transformation.
We are thinking to make use of spark Accumulators for this purpose.
Can we use these as below:
Variables -> Initial Value
Variable1 -> 0
Variable2 -> 0
In one of the transformations if we need to make Variable2's value to 1. Can we achieve this using Accumulators? Please confirm.
Thanks!
With Regards,
~Kedar Dixit
[hidden email]<http:///user/SendEmail.jtp?type=node&node=29995&i=2> | @kedarsdixit | M <a href="tel:+91%2090499%2015588" value="+919049915588" target="_blank">+91 90499 15588 | T <a href="tel:+91%2020%206703%204783" value="+912067034783" target="_blank">+91 (20) 6703 4783
Persistent Systems | Partners In Innovation | www.persistent.com<http://www.persistent.com>
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
--
Regards,
Vaquar Khan
+1 -224-436-0783
Greater Chicago
--
Twitter: https://twitter.com/holdenkarau
________________________________
If you reply to this email, your message will be added to the discussion below:
http://apache-spark-user-list.1001560.n3.nabble.com/Use-of-Accumulators-tp29975p29995.html
To start a new topic under Apache Spark User List, email ml+s1001560n1h45@n3.nabble.com<ma...@n3.nabble.com>
To unsubscribe from Apache Spark User List, click here<http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=a2VkYXJuYXRoX2RpeGl0QHBlcnNpc3RlbnQuY29tfDF8Njk0OTY1MzYz>.
NAML<http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
--
Twitter: https://twitter.com/holdenkarau
Re: Use of Accumulators
Posted by Holden Karau <ho...@pigscanfly.ca>.
And where do you want to read the toggle back from? On the driver?
On Tue, Nov 14, 2017 at 3:52 AM Kedarnath Dixit <
kedarnath_dixit@persistent.com> wrote:
> Hi,
>
>
>
> Inside the transformation if there is any change for the Variable’s
> associated data, I want to just toggle it saying there is some change while
> processing the data.
>
>
>
> Please let me know if we can runtime do this.
>
>
>
>
>
> Thanks!
>
> *~Kedar Dixit*
>
> Bigdata Analytics at Persistent Systems Ltd.
>
>
>
> *From:* Holden Karau [via Apache Spark User List] [
> mailto:ml+s1001560n29995h59@n3.nabble.com
> <ml...@n3.nabble.com>]
> *Sent:* Tuesday, November 14, 2017 1:16 PM
> *To:* Kedarnath Dixit <ke...@persistent.com>
>
>
> *Subject:* Re: Use of Accumulators
>
>
>
> So you want to set an accumulator to 1 after a transformation has fully
> completed? Or what exactly do you want to do?
>
>
>
> On Mon, Nov 13, 2017 at 9:47 PM vaquar khan <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=29995&i=0>> wrote:
>
> Confirmed ,you can use Accumulators :)
>
>
>
> Regards,
>
> Vaquar khan
>
> On Mon, Nov 13, 2017 at 10:58 AM, Kedarnath Dixit <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=29995&i=1>> wrote:
>
> Hi,
>
>
>
> We need some way to toggle the flag of a variable in transformation.
>
>
>
> We are thinking to make use of spark Accumulators for this purpose.
>
>
>
> Can we use these as below:
>
>
>
> Variables -> Initial Value
>
> Variable1 -> 0
>
> Variable2 -> 0
>
>
>
> In one of the transformations if we need to make Variable2's value to 1.
> Can we achieve this using Accumulators? Please confirm.
>
>
>
> Thanks!
>
>
>
> With Regards,
>
> *~Kedar Dixit*
>
>
>
> [hidden email] <http:///user/SendEmail.jtp?type=node&node=29995&i=2> |
> @kedarsdixit | M <a href="tel:+91%2090499%2015588 <+91%2090499%2015588>"
> value="+919049915588" target="_blank">+91 90499 15588 | T <a href="
> tel:+91%2020%206703%204783 <+91%2020%206703%204783>"
> value="+912067034783" target="_blank">+91 (20) 6703 4783
>
> *Persistent Systems | **Partners In Innovation** | www.persistent.com
> <http://www.persistent.com>*
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is
> the property of Persistent Systems Ltd. It is intended only for the use of
> the individual or entity to which it is addressed. If you are not the
> intended recipient, you are not authorized to read, retain, copy, print,
> distribute or use this message. If you have received this communication in
> error, please notify the sender and delete all copies of this message.
> Persistent Systems Ltd. does not accept any liability for virus infected
> mails.
>
> --
>
> Regards,
>
> Vaquar Khan
>
> +1 -224-436-0783
>
> Greater Chicago
>
> --
>
> Twitter: https://twitter.com/holdenkarau
>
>
> ------------------------------
>
> *If you reply to this email, your message will be added to the discussion
> below:*
>
>
> http://apache-spark-user-list.1001560.n3.nabble.com/Use-of-Accumulators-tp29975p29995.html
>
> To start a new topic under Apache Spark User List, email
> ml+s1001560n1h45@n3.nabble.com
> To unsubscribe from Apache Spark User List, click here
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=a2VkYXJuYXRoX2RpeGl0QHBlcnNpc3RlbnQuY29tfDF8Njk0OTY1MzYz>
> .
> NAML
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
Twitter: https://twitter.com/holdenkarau
RE: Use of Accumulators
Posted by Kedarnath Dixit <ke...@persistent.com>.
Hi,
Inside the transformation if there is any change for the Variable’s associated data, I want to just toggle it saying there is some change while processing the data.
Please let me know if we can runtime do this.
Thanks!
~Kedar Dixit
Bigdata Analytics at Persistent Systems Ltd.
From: Holden Karau [via Apache Spark User List] [mailto:ml+s1001560n29995h59@n3.nabble.com]
Sent: Tuesday, November 14, 2017 1:16 PM
To: Kedarnath Dixit <ke...@persistent.com>>
Subject: Re: Use of Accumulators
So you want to set an accumulator to 1 after a transformation has fully completed? Or what exactly do you want to do?
On Mon, Nov 13, 2017 at 9:47 PM vaquar khan <[hidden email]</user/SendEmail.jtp?type=node&node=29995&i=0>> wrote:
Confirmed ,you can use Accumulators :)
Regards,
Vaquar khan
On Mon, Nov 13, 2017 at 10:58 AM, Kedarnath Dixit <[hidden email]</user/SendEmail.jtp?type=node&node=29995&i=1>> wrote:
Hi,
We need some way to toggle the flag of a variable in transformation.
We are thinking to make use of spark Accumulators for this purpose.
Can we use these as below:
Variables -> Initial Value
Variable1 -> 0
Variable2 -> 0
In one of the transformations if we need to make Variable2's value to 1. Can we achieve this using Accumulators? Please confirm.
Thanks!
With Regards,
~Kedar Dixit
[hidden email]</user/SendEmail.jtp?type=node&node=29995&i=2> | @kedarsdixit | M <a href="tel:+91%2090499%2015588" value="+919049915588" target="_blank">+91 90499 15588 | T <a href="tel:+91%2020%206703%204783" value="+912067034783" target="_blank">+91 (20) 6703 4783
Persistent Systems | Partners In Innovation | www.persistent.com<http://www.persistent.com>
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
--
Regards,
Vaquar Khan
+1 -224-436-0783
Greater Chicago
--
Twitter: https://twitter.com/holdenkarau
________________________________
If you reply to this email, your message will be added to the discussion below:
http://apache-spark-user-list.1001560.n3.nabble.com/Use-of-Accumulators-tp29975p29995.html
To start a new topic under Apache Spark User List, email ml+s1001560n1h45@n3.nabble.com<ma...@n3.nabble.com>
To unsubscribe from Apache Spark User List, click here<http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=a2VkYXJuYXRoX2RpeGl0QHBlcnNpc3RlbnQuY29tfDF8Njk0OTY1MzYz>.
NAML<http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
Re: Use of Accumulators
Posted by Holden Karau <ho...@pigscanfly.ca>.
So you want to set an accumulator to 1 after a transformation has fully
completed? Or what exactly do you want to do?
On Mon, Nov 13, 2017 at 9:47 PM vaquar khan <va...@gmail.com> wrote:
> Confirmed ,you can use Accumulators :)
>
> Regards,
> Vaquar khan
>
> On Mon, Nov 13, 2017 at 10:58 AM, Kedarnath Dixit <
> kedarnath_dixit@persistent.com> wrote:
>
>> Hi,
>>
>>
>> We need some way to toggle the flag of a variable in transformation.
>>
>>
>> We are thinking to make use of spark Accumulators for this purpose.
>>
>>
>> Can we use these as below:
>>
>>
>> Variables -> Initial Value
>>
>> Variable1 -> 0
>>
>> Variable2 -> 0
>>
>>
>> In one of the transformations if we need to make Variable2's value to 1.
>> Can we achieve this using Accumulators? Please confirm.
>>
>>
>> Thanks!
>>
>>
>> With Regards,
>>
>> *~Kedar Dixit*
>>
>> kedarnath_dixit@persistent.com | @kedarsdixit | M +91 90499 15588
>> <+91%2090499%2015588> | T +91 (20) 6703 4783 <+91%2020%206703%204783>
>>
>> *Persistent Systems | **Partners In Innovation** | www.persistent.com
>> <http://www.persistent.com>*
>> DISCLAIMER
>> ==========
>> This e-mail may contain privileged and confidential information which is
>> the property of Persistent Systems Ltd. It is intended only for the use of
>> the individual or entity to which it is addressed. If you are not the
>> intended recipient, you are not authorized to read, retain, copy, print,
>> distribute or use this message. If you have received this communication in
>> error, please notify the sender and delete all copies of this message.
>> Persistent Systems Ltd. does not accept any liability for virus infected
>> mails.
>>
>
>
>
> --
> Regards,
> Vaquar Khan
> +1 -224-436-0783
> Greater Chicago
>
--
Twitter: https://twitter.com/holdenkarau
Re: Use of Accumulators
Posted by vaquar khan <va...@gmail.com>.
Confirmed ,you can use Accumulators :)
Regards,
Vaquar khan
On Mon, Nov 13, 2017 at 10:58 AM, Kedarnath Dixit <
kedarnath_dixit@persistent.com> wrote:
> Hi,
>
>
> We need some way to toggle the flag of a variable in transformation.
>
>
> We are thinking to make use of spark Accumulators for this purpose.
>
>
> Can we use these as below:
>
>
> Variables -> Initial Value
>
> Variable1 -> 0
>
> Variable2 -> 0
>
>
> In one of the transformations if we need to make Variable2's value to 1.
> Can we achieve this using Accumulators? Please confirm.
>
>
> Thanks!
>
>
> With Regards,
>
> *~Kedar Dixit*
>
> kedarnath_dixit@persistent.com | @kedarsdixit | M +91 90499 15588
> <+91%2090499%2015588> | T +91 (20) 6703 4783 <+91%2020%206703%204783>
>
> *Persistent Systems | **Partners In Innovation** | www.persistent.com
> <http://www.persistent.com>*
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is
> the property of Persistent Systems Ltd. It is intended only for the use of
> the individual or entity to which it is addressed. If you are not the
> intended recipient, you are not authorized to read, retain, copy, print,
> distribute or use this message. If you have received this communication in
> error, please notify the sender and delete all copies of this message.
> Persistent Systems Ltd. does not accept any liability for virus infected
> mails.
>
--
Regards,
Vaquar Khan
+1 -224-436-0783
Greater Chicago