You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Kevin Burton <bu...@spinn3r.com> on 2015/01/13 03:54:06 UTC
quickly counting the number of rows in a partition?
Is there a way to compute the total number of records in each RDD partition?
So say I had 4 partitions.. I’d want to have
partition 0: 100 records
partition 1: 104 records
partition 2: 90 records
partition 3: 140 records
Kevin
--
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>
Re: quickly counting the number of rows in a partition?
Posted by Sven Krasser <kr...@gmail.com>.
Yes, using mapPartitionsWithIndex, e.g. in PySpark:
>>> sc.parallelize(xrange(0,1000), 4).mapPartitionsWithIndex(lambda
idx,iter: ((idx, len(list(iter))),)).collect()
[(0, 250), (1, 250), (2, 250), (3, 250)]
(This is not the most efficient way to get the length of an iterator, but
you get the idea...)
Best,
-Sven
On Mon, Jan 12, 2015 at 6:54 PM, Kevin Burton <bu...@spinn3r.com> wrote:
> Is there a way to compute the total number of records in each RDD
> partition?
>
> So say I had 4 partitions.. I’d want to have
>
> partition 0: 100 records
> partition 1: 104 records
> partition 2: 90 records
> partition 3: 140 records
>
> Kevin
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
> <http://spinn3r.com>
>
>
--
http://sites.google.com/site/krasser/?utm_source=sig
Re: quickly counting the number of rows in a partition?
Posted by Ilya Ganelin <il...@gmail.com>.
The number of records is not stored anywhere. You either need to save it at
creation time or step through the RDD.
On Wed, Jan 14, 2015 at 1:46 PM Michael Segel <ms...@hotmail.com>
wrote:
> Sorry, but the accumulator is still going to require you to walk through
> the RDD to get an accurate count, right?
> Its not being persisted?
>
> On Jan 14, 2015, at 5:17 AM, Ganelin, Ilya <Il...@capitalone.com>
> wrote:
>
> Alternative to doing a naive toArray is to declare an accumulator per
> partition and use that. It's specifically what they were designed to do.
> See the programming guide.
>
>
>
> Sent with Good (www.good.com)
>
>
> -----Original Message-----
> *From: *Tobias Pfeiffer [tgp@preferred.jp]
> *Sent: *Tuesday, January 13, 2015 08:06 PM Eastern Standard Time
> *To: *Kevin Burton
> *Cc: *Ganelin, Ilya; user@spark.apache.org
> *Subject: *Re: quickly counting the number of rows in a partition?
>
> Hi,
>
> On Mon, Jan 12, 2015 at 8:09 PM, Ganelin, Ilya <
> Ilya.Ganelin@capitalone.com> wrote:
>
>> Use the mapPartitions function. It returns an iterator to each partition.
>> Then just get that length by converting to an array.
>>
>
> On Tue, Jan 13, 2015 at 2:50 PM, Kevin Burton <bu...@spinn3r.com> wrote:
>
>> Doesn’t that just read in all the values? The count isn’t pre-computed?
>> It’s not the end of the world if it’s not but would be faster.
>>
>
> Well, "converting to an array" may not work due to memory constraints,
> counting the items in the iterator may be better. However, there is no
> "pre-computed" value. For counting, you need to compute all values in the
> RDD, in general. If you think of
>
> items.map(x => /* throw exception */).count()
>
> then even though the count you want to get does not necessarily require
> the evaluation of the function in map() (i.e., the number is the same), you
> may not want to get the count if that code actually fails.
>
> Tobias
>
> ------------------------------
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the
> intended recipient, you are hereby notified that any review,
> retransmission, dissemination, distribution, copying or other use of, or
> taking of any action in reliance upon this information is strictly
> prohibited. If you have received this communication in error, please
> contact the sender and delete the material from your computer.
>
>
>
Re: quickly counting the number of rows in a partition?
Posted by Michael Segel <ms...@hotmail.com>.
Sorry, but the accumulator is still going to require you to walk through the RDD to get an accurate count, right?
Its not being persisted?
On Jan 14, 2015, at 5:17 AM, Ganelin, Ilya <Il...@capitalone.com> wrote:
> Alternative to doing a naive toArray is to declare an accumulator per partition and use that. It's specifically what they were designed to do. See the programming guide.
>
>
>
> Sent with Good (www.good.com)
>
>
> -----Original Message-----
> From: Tobias Pfeiffer [tgp@preferred.jp]
> Sent: Tuesday, January 13, 2015 08:06 PM Eastern Standard Time
> To: Kevin Burton
> Cc: Ganelin, Ilya; user@spark.apache.org
> Subject: Re: quickly counting the number of rows in a partition?
>
> Hi,
>
> On Mon, Jan 12, 2015 at 8:09 PM, Ganelin, Ilya <Il...@capitalone.com> wrote:
> Use the mapPartitions function. It returns an iterator to each partition. Then just get that length by converting to an array.
>
> On Tue, Jan 13, 2015 at 2:50 PM, Kevin Burton <bu...@spinn3r.com> wrote:
> Doesn’t that just read in all the values? The count isn’t pre-computed? It’s not the end of the world if it’s not but would be faster.
>
> Well, "converting to an array" may not work due to memory constraints, counting the items in the iterator may be better. However, there is no "pre-computed" value. For counting, you need to compute all values in the RDD, in general. If you think of
>
> items.map(x => /* throw exception */).count()
>
> then even though the count you want to get does not necessarily require the evaluation of the function in map() (i.e., the number is the same), you may not want to get the count if that code actually fails.
>
> Tobias
>
> The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
RE: quickly counting the number of rows in a partition?
Posted by "Ganelin, Ilya" <Il...@capitalone.com>.
Alternative to doing a naive toArray is to declare an accumulator per partition and use that. It's specifically what they were designed to do. See the programming guide.
Sent with Good (www.good.com)
-----Original Message-----
From: Tobias Pfeiffer [tgp@preferred.jp<ma...@preferred.jp>]
Sent: Tuesday, January 13, 2015 08:06 PM Eastern Standard Time
To: Kevin Burton
Cc: Ganelin, Ilya; user@spark.apache.org
Subject: Re: quickly counting the number of rows in a partition?
Hi,
On Mon, Jan 12, 2015 at 8:09 PM, Ganelin, Ilya <Il...@capitalone.com>> wrote:
Use the mapPartitions function. It returns an iterator to each partition. Then just get that length by converting to an array.
On Tue, Jan 13, 2015 at 2:50 PM, Kevin Burton <bu...@spinn3r.com>> wrote:
Doesn’t that just read in all the values? The count isn’t pre-computed? It’s not the end of the world if it’s not but would be faster.
Well, "converting to an array" may not work due to memory constraints, counting the items in the iterator may be better. However, there is no "pre-computed" value. For counting, you need to compute all values in the RDD, in general. If you think of
items.map(x => /* throw exception */).count()
then even though the count you want to get does not necessarily require the evaluation of the function in map() (i.e., the number is the same), you may not want to get the count if that code actually fails.
Tobias
________________________________________________________
The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
Re: quickly counting the number of rows in a partition?
Posted by Tobias Pfeiffer <tg...@preferred.jp>.
Hi again,
On Wed, Jan 14, 2015 at 10:06 AM, Tobias Pfeiffer <tg...@preferred.jp> wrote:
> If you think of
> items.map(x => /* throw exception */).count()
> then even though the count you want to get does not necessarily require
> the evaluation of the function in map() (i.e., the number is the same), you
> may not want to get the count if that code actually fails.
>
Sorry, I think that was a bit confusing. What I mean is: You have to
compute the whole RDD in order to give a meaningful count() result (whether
you use rdd.count() or the mapPartitions() approach).
Tobias
Re: quickly counting the number of rows in a partition?
Posted by Tobias Pfeiffer <tg...@preferred.jp>.
Hi,
On Mon, Jan 12, 2015 at 8:09 PM, Ganelin, Ilya <Il...@capitalone.com>
wrote:
> Use the mapPartitions function. It returns an iterator to each partition.
> Then just get that length by converting to an array.
>
On Tue, Jan 13, 2015 at 2:50 PM, Kevin Burton <bu...@spinn3r.com> wrote:
> Doesn’t that just read in all the values? The count isn’t pre-computed?
> It’s not the end of the world if it’s not but would be faster.
>
Well, "converting to an array" may not work due to memory constraints,
counting the items in the iterator may be better. However, there is no
"pre-computed" value. For counting, you need to compute all values in the
RDD, in general. If you think of
items.map(x => /* throw exception */).count()
then even though the count you want to get does not necessarily require the
evaluation of the function in map() (i.e., the number is the same), you may
not want to get the count if that code actually fails.
Tobias
Re: quickly counting the number of rows in a partition?
Posted by Kevin Burton <bu...@spinn3r.com>.
Doesn’t that just read in all the values? The count isn’t pre-computed?
It’s not the end of the world if it’s not but would be faster.
On Mon, Jan 12, 2015 at 8:09 PM, Ganelin, Ilya <Il...@capitalone.com>
wrote:
> Use the mapPartitions function. It returns an iterator to each
> partition. Then just get that length by converting to an array.
>
>
>
> Sent with Good (www.good.com)
>
>
>
> -----Original Message-----
> *From: *Kevin Burton [burton@spinn3r.com]
> *Sent: *Monday, January 12, 2015 09:55 PM Eastern Standard Time
> *To: *user@spark.apache.org
> *Subject: *quickly counting the number of rows in a partition?
>
> Is there a way to compute the total number of records in each RDD
> partition?
>
> So say I had 4 partitions.. I’d want to have
>
> partition 0: 100 records
> partition 1: 104 records
> partition 2: 90 records
> partition 3: 140 records
>
> Kevin
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
> <http://spinn3r.com>
>
> ------------------------------
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the
> intended recipient, you are hereby notified that any review,
> retransmission, dissemination, distribution, copying or other use of, or
> taking of any action in reliance upon this information is strictly
> prohibited. If you have received this communication in error, please
> contact the sender and delete the material from your computer.
>
--
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>