You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Sampo Niskanen <sa...@wellmo.com> on 2014/02/18 13:33:03 UTC
Testing if an RDD is empty?
Hi,
I couldn't find any documentation on how to test whether an RDD is empty.
I'm doing a reduce operation, but it throws an
UnsupportedOperationException if the RDD is empty. I'd like to check if
the RDD is empty before calling reduce.
On the Google Groups list Reynold Xin had suggested using rdd.first, but it
throws the same exception in case the RDD is empty. rdd.count() on the
other hand might do a lot of unnecessary processing just to check whether
the RDD is empty.
Is there some way to do rdd.isEmpty?
Catching the UnsupportedOperationException works, but it seems like bad
practice, as it could be an indication of some other error as well. (I'd
suggest changing it to EmptyRddException or similar, which could be a
subclass of UnsupportedOperationException. That would make the cause
explicit.)
Thanks.
* Sampo Niskanen*
*Lead developer / Wellmo*
sampo.niskanen@wellmo.com
Re: Testing if an RDD is empty?
Posted by Sean Owen <so...@cloudera.com>.
How about checking whether take(1).length == 0? If I read the code
correctly, this will only examine the first partition, at least.
On Fri, Jan 16, 2015 at 4:12 AM, Tobias Pfeiffer <tg...@preferred.jp> wrote:
> Hi,
>
> On Fri, Jan 16, 2015 at 7:31 AM, freedafeng <fr...@yahoo.com> wrote:
>>
>> I myself saw many times that my app threw out exceptions because an empty
>> RDD cannot be saved. This is not big issue, but annoying. Having a cheap
>> solution testing if an RDD is empty would be nice if there is no such
>> thing
>> available now.
>
>
> I think the cheapest you can have is computing at least one element in the
> RDD, which in the case of, say,
>
> val maybeEmptyRDD = veryExpensiveRDD.filter(false)
>
> will be just as expensive as .count().
>
> Tobias
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Testing if an RDD is empty?
Posted by Tobias Pfeiffer <tg...@preferred.jp>.
Hi,
On Fri, Jan 16, 2015 at 7:31 AM, freedafeng <fr...@yahoo.com> wrote:
>
> I myself saw many times that my app threw out exceptions because an empty
> RDD cannot be saved. This is not big issue, but annoying. Having a cheap
> solution testing if an RDD is empty would be nice if there is no such thing
> available now.
>
I think the cheapest you can have is computing at least one element in the
RDD, which in the case of, say,
val maybeEmptyRDD = veryExpensiveRDD.filter(false)
will be just as expensive as .count().
Tobias
Re: Testing if an RDD is empty?
Posted by freedafeng <fr...@yahoo.com>.
I think Sampo's thought is to get a function that only tests if a RDD is
empty. He does not want to know the size of the RDD, and getting the size of
a RDD is expensive for large data sets.
I myself saw many times that my app threw out exceptions because an empty
RDD cannot be saved. This is not big issue, but annoying. Having a cheap
solution testing if an RDD is empty would be nice if there is no such thing
available now.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Testing-if-an-RDD-is-empty-tp1678p21175.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Testing if an RDD is empty?
Posted by Al M <al...@gmail.com>.
You can also check rdd.partitions.size. This will be 0 for empty RDDs and >
0 for RDDs with data.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Testing-if-an-RDD-is-empty-tp1678p21170.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Testing if an RDD is empty?
Posted by Cheng Lian <rh...@gmail.com>.
Yea, for testing empty RDD, Guillaume's *mapPartitions* solution is
definitely better than *count* since it doesn't require traversing all
elements :-)
On Tue, Feb 18, 2014 at 10:08 PM, Guillaume Pitel <
guillaume.pitel@exensa.com> wrote:
> I think you could do something like that : (but Cheng's solution allows
> to avoid testing, so it's better) :
>
> val isEmpty = rdd.mapPartitions(iter => Iterator(!
> iter.hasNext)).reduce(_&&_)
>
> but it would be bad if the rdd is not persisted...
>
> Guillaume
>
> Unfortunately there isn't an easy way to test whether an RDD is empty
> unless you count it. But fortunately, RDD.fold can solve your problem.
> All you need to do is to provide a zero value.
>
> For an RDD r, r.reduce(f) is roughly equivalent to r.fold(r.first)(f),
> that is, the first element of r is considered as the zero value of the fold
> call. For example:
>
> val r = sc.makeRDD(1 to 4)
> r.reduce(_ + _) // can't handle empty RDD
> r.fold(0)(_ + _) // no problem with empty RDD
>
>
>
> On Tue, Feb 18, 2014 at 8:33 PM, Sampo Niskanen <sampo.niskanen@wellmo.com
> > wrote:
>
>> Hi,
>>
>> I couldn't find any documentation on how to test whether an RDD is
>> empty. I'm doing a reduce operation, but it throws an
>> UnsupportedOperationException if the RDD is empty. I'd like to check if
>> the RDD is empty before calling reduce.
>>
>> On the Google Groups list Reynold Xin had suggested using rdd.first,
>> but it throws the same exception in case the RDD is empty. rdd.count() on
>> the other hand might do a lot of unnecessary processing just to check
>> whether the RDD is empty.
>>
>> Is there some way to do rdd.isEmpty?
>>
>>
>> Catching the UnsupportedOperationException works, but it seems like bad
>> practice, as it could be an indication of some other error as well. (I'd
>> suggest changing it to EmptyRddException or similar, which could be a
>> subclass of UnsupportedOperationException. That would make the cause
>> explicit.)
>>
>>
>> Thanks.
>>
>> * Sampo Niskanen*
>> *Lead developer / Wellmo*
>> sampo.niskanen@wellmo.com
>>
>>
>
>
>
> --
> [image: eXenSa]
> *Guillaume PITEL, Président*
> +33(0)6 25 48 86 80
>
> eXenSa S.A.S. <http://www.exensa.com/>
> 41, rue Périer - 92120 Montrouge - FRANCE
> Tel +33(0)1 84 16 36 77 / Fax +33(0)9 72 28 37 05
>
Re: Testing if an RDD is empty?
Posted by Guillaume Pitel <gu...@exensa.com>.
I think you could do something like that : (but Cheng's solution allows to avoid
testing, so it's better) :
val isEmpty = rdd.mapPartitions(iter => Iterator(! iter.hasNext)).reduce(_&&_)
but it would be bad if the rdd is not persisted...
Guillaume
> Unfortunately there isn't an easy way to test whether an RDD is empty unless
> you count it. But fortunately, RDD.fold can solve your problem. All you need
> to do is to provide a zero value.
>
> For an RDD r, r.reduce(f) is roughly equivalent to r.fold(r.first)(f), that
> is, the first element of r is considered as the zero value of the fold call.
> For example:
>
> val r = sc.makeRDD(1 to 4)
> r.reduce(_ + _) // can't handle empty RDD
> r.fold(0)(_ + _) // no problem with empty RDD
>
>
>
> On Tue, Feb 18, 2014 at 8:33 PM, Sampo Niskanen <sampo.niskanen@wellmo.com
> <ma...@wellmo.com>> wrote:
>
> Hi,
>
> I couldn't find any documentation on how to test whether an RDD is empty.
> I'm doing a reduce operation, but it throws an
> UnsupportedOperationException if the RDD is empty. I'd like to check if
> the RDD is empty before calling reduce.
>
> On the Google Groups list Reynold Xin had suggested using rdd.first, but
> it throws the same exception in case the RDD is empty. rdd.count() on the
> other hand might do a lot of unnecessary processing just to check whether
> the RDD is empty.
>
> Is there some way to do rdd.isEmpty?
>
>
> Catching the UnsupportedOperationException works, but it seems like bad
> practice, as it could be an indication of some other error as well. (I'd
> suggest changing it to EmptyRddException or similar, which could be a
> subclass of UnsupportedOperationException. That would make the cause
> explicit.)
>
>
> Thanks.
>
> * Sampo Niskanen*
> ///Lead developer / Wellmo/
> sampo.niskanen@wellmo.com <ma...@wellmo.com>
>
>
--
eXenSa
*Guillaume PITEL, Président*
+33(0)6 25 48 86 80
eXenSa S.A.S. <http://www.exensa.com/>
41, rue Périer - 92120 Montrouge - FRANCE
Tel +33(0)1 84 16 36 77 / Fax +33(0)9 72 28 37 05
Re: Testing if an RDD is empty?
Posted by Cheng Lian <rh...@gmail.com>.
Unfortunately there isn't an easy way to test whether an RDD is empty
unless you count it. But fortunately, RDD.fold can solve your problem.
All you need to do is to provide a zero value.
For an RDD r, r.reduce(f) is roughly equivalent to r.fold(r.first)(f), that
is, the first element of r is considered as the zero value of the fold
call. For example:
val r = sc.makeRDD(1 to 4)
r.reduce(_ + _) // can't handle empty RDD
r.fold(0)(_ + _) // no problem with empty RDD
On Tue, Feb 18, 2014 at 8:33 PM, Sampo Niskanen
<sa...@wellmo.com>wrote:
> Hi,
>
> I couldn't find any documentation on how to test whether an RDD is empty.
> I'm doing a reduce operation, but it throws an
> UnsupportedOperationException if the RDD is empty. I'd like to check if
> the RDD is empty before calling reduce.
>
> On the Google Groups list Reynold Xin had suggested using rdd.first, but
> it throws the same exception in case the RDD is empty. rdd.count() on the
> other hand might do a lot of unnecessary processing just to check whether
> the RDD is empty.
>
> Is there some way to do rdd.isEmpty?
>
>
> Catching the UnsupportedOperationException works, but it seems like bad
> practice, as it could be an indication of some other error as well. (I'd
> suggest changing it to EmptyRddException or similar, which could be a
> subclass of UnsupportedOperationException. That would make the cause
> explicit.)
>
>
> Thanks.
>
> * Sampo Niskanen*
> *Lead developer / Wellmo*
> sampo.niskanen@wellmo.com
>
>