You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Sebastian Piu <se...@gmail.com> on 2016/02/11 22:03:29 UTC
Skip empty batches - spark streaming
I was wondering if there is there any way to skip batches with zero events
when streaming?
By skip I mean avoid the empty rdd from being created at all?
Re: Skip empty batches - spark streaming
Posted by Andy Davidson <An...@SantaCruzIntegration.com>.
You can always call rdd.isEmpty()
Andy
private static void save(JavaDStream<String> jsonRdd, String outputURI)
{
jsonTweets.foreachRDD(new VoidFunction2<JavaRDD<String>, Time>() {
private static final long serialVersionUID = 1L;
@Override
public void call(JavaRDD<String> rdd, Time time) throws
Exception {
if(!rdd.isEmpty()) {
String dirPath = outputURI + "-" + time.milliseconds();
rdd.saveAsTextFile(dirPath);
}
}
});
From: Sebastian Piu <se...@gmail.com>
Reply-To: <se...@hotmail.com>
Date: Thursday, February 11, 2016 at 1:19 PM
To: "Shixiong (Ryan) Zhu" <sh...@databricks.com>
Cc: Sebastian Piu <se...@hotmail.com>, "user @spark"
<us...@spark.apache.org>
Subject: Re: Skip empty batches - spark streaming
>
> Yes, and as far as I recall it also has partitions (empty) which screws up the
> isEmpty call if the rdd has been transformed down the line. I will have a look
> tomorrow at the office and see if I can collaborate
>
> On 11 Feb 2016 9:14 p.m., "Shixiong(Ryan) Zhu" <sh...@databricks.com>
> wrote:
>> Yeah, DirectKafkaInputDStream always returns a RDD even if it's empty. Feel
>> free to send a PR to improve it.
>>
>> On Thu, Feb 11, 2016 at 1:09 PM, Sebastian Piu <se...@gmail.com>
>> wrote:
>>>
>>> I'm using the Kafka direct stream api but I can have a look on extending it
>>> to have this behaviour
>>>
>>> Thanks!
>>>
>>> On 11 Feb 2016 9:07 p.m., "Shixiong(Ryan) Zhu" <sh...@databricks.com>
>>> wrote:
>>>> Are you using a custom input dstream? If so, you can make the `compute`
>>>> method return None to skip a batch.
>>>>
>>>> On Thu, Feb 11, 2016 at 1:03 PM, Sebastian Piu <se...@gmail.com>
>>>> wrote:
>>>>>
>>>>> I was wondering if there is there any way to skip batches with zero events
>>>>> when streaming?
>>>>> By skip I mean avoid the empty rdd from being created at all?
>>>>
>>
Re: Skip empty batches - spark streaming
Posted by Sebastian Piu <se...@gmail.com>.
Thanks for clarifying Cody. I will extend the current behaviour for my use
case. If there is anything worth sharing I'll run it through the list
Cheers
On 11 Feb 2016 9:47 p.m., "Cody Koeninger" <co...@koeninger.org> wrote:
> Please don't change the behavior of DirectKafkaInputDStream.
> Returning an empty rdd is (imho) the semantically correct thing to do, and
> some existing jobs depend on that behavior.
>
> If it's really an issue for you, you can either override
> directkafkainputdstream, or just check isEmpty as the first thing you do
> with the rdd (before any transformations)
>
> In any recent version of spark, isEmpty on a KafkaRDD is a driver-side
> only operation that is basically free.
>
>
> On Thu, Feb 11, 2016 at 3:19 PM, Sebastian Piu <se...@gmail.com>
> wrote:
>
>> Yes, and as far as I recall it also has partitions (empty) which screws
>> up the isEmpty call if the rdd has been transformed down the line. I will
>> have a look tomorrow at the office and see if I can collaborate
>> On 11 Feb 2016 9:14 p.m., "Shixiong(Ryan) Zhu" <sh...@databricks.com>
>> wrote:
>>
>>> Yeah, DirectKafkaInputDStream always returns a RDD even if it's empty.
>>> Feel free to send a PR to improve it.
>>>
>>> On Thu, Feb 11, 2016 at 1:09 PM, Sebastian Piu <se...@gmail.com>
>>> wrote:
>>>
>>>> I'm using the Kafka direct stream api but I can have a look on
>>>> extending it to have this behaviour
>>>>
>>>> Thanks!
>>>> On 11 Feb 2016 9:07 p.m., "Shixiong(Ryan) Zhu" <sh...@databricks.com>
>>>> wrote:
>>>>
>>>>> Are you using a custom input dstream? If so, you can make the
>>>>> `compute` method return None to skip a batch.
>>>>>
>>>>> On Thu, Feb 11, 2016 at 1:03 PM, Sebastian Piu <
>>>>> sebastian.piu@gmail.com> wrote:
>>>>>
>>>>>> I was wondering if there is there any way to skip batches with zero
>>>>>> events when streaming?
>>>>>> By skip I mean avoid the empty rdd from being created at all?
>>>>>>
>>>>>
>>>>>
>>>
>
Re: Skip empty batches - spark streaming
Posted by Cody Koeninger <co...@koeninger.org>.
Please don't change the behavior of DirectKafkaInputDStream.
Returning an empty rdd is (imho) the semantically correct thing to do, and
some existing jobs depend on that behavior.
If it's really an issue for you, you can either override
directkafkainputdstream, or just check isEmpty as the first thing you do
with the rdd (before any transformations)
In any recent version of spark, isEmpty on a KafkaRDD is a driver-side only
operation that is basically free.
On Thu, Feb 11, 2016 at 3:19 PM, Sebastian Piu <se...@gmail.com>
wrote:
> Yes, and as far as I recall it also has partitions (empty) which screws up
> the isEmpty call if the rdd has been transformed down the line. I will have
> a look tomorrow at the office and see if I can collaborate
> On 11 Feb 2016 9:14 p.m., "Shixiong(Ryan) Zhu" <sh...@databricks.com>
> wrote:
>
>> Yeah, DirectKafkaInputDStream always returns a RDD even if it's empty.
>> Feel free to send a PR to improve it.
>>
>> On Thu, Feb 11, 2016 at 1:09 PM, Sebastian Piu <se...@gmail.com>
>> wrote:
>>
>>> I'm using the Kafka direct stream api but I can have a look on extending
>>> it to have this behaviour
>>>
>>> Thanks!
>>> On 11 Feb 2016 9:07 p.m., "Shixiong(Ryan) Zhu" <sh...@databricks.com>
>>> wrote:
>>>
>>>> Are you using a custom input dstream? If so, you can make the `compute`
>>>> method return None to skip a batch.
>>>>
>>>> On Thu, Feb 11, 2016 at 1:03 PM, Sebastian Piu <sebastian.piu@gmail.com
>>>> > wrote:
>>>>
>>>>> I was wondering if there is there any way to skip batches with zero
>>>>> events when streaming?
>>>>> By skip I mean avoid the empty rdd from being created at all?
>>>>>
>>>>
>>>>
>>
Re: Skip empty batches - spark streaming
Posted by Sebastian Piu <se...@gmail.com>.
Yes, and as far as I recall it also has partitions (empty) which screws up
the isEmpty call if the rdd has been transformed down the line. I will have
a look tomorrow at the office and see if I can collaborate
On 11 Feb 2016 9:14 p.m., "Shixiong(Ryan) Zhu" <sh...@databricks.com>
wrote:
> Yeah, DirectKafkaInputDStream always returns a RDD even if it's empty.
> Feel free to send a PR to improve it.
>
> On Thu, Feb 11, 2016 at 1:09 PM, Sebastian Piu <se...@gmail.com>
> wrote:
>
>> I'm using the Kafka direct stream api but I can have a look on extending
>> it to have this behaviour
>>
>> Thanks!
>> On 11 Feb 2016 9:07 p.m., "Shixiong(Ryan) Zhu" <sh...@databricks.com>
>> wrote:
>>
>>> Are you using a custom input dstream? If so, you can make the `compute`
>>> method return None to skip a batch.
>>>
>>> On Thu, Feb 11, 2016 at 1:03 PM, Sebastian Piu <se...@gmail.com>
>>> wrote:
>>>
>>>> I was wondering if there is there any way to skip batches with zero
>>>> events when streaming?
>>>> By skip I mean avoid the empty rdd from being created at all?
>>>>
>>>
>>>
>
Re: Skip empty batches - spark streaming
Posted by "Shixiong(Ryan) Zhu" <sh...@databricks.com>.
Yeah, DirectKafkaInputDStream always returns a RDD even if it's empty. Feel
free to send a PR to improve it.
On Thu, Feb 11, 2016 at 1:09 PM, Sebastian Piu <se...@gmail.com>
wrote:
> I'm using the Kafka direct stream api but I can have a look on extending
> it to have this behaviour
>
> Thanks!
> On 11 Feb 2016 9:07 p.m., "Shixiong(Ryan) Zhu" <sh...@databricks.com>
> wrote:
>
>> Are you using a custom input dstream? If so, you can make the `compute`
>> method return None to skip a batch.
>>
>> On Thu, Feb 11, 2016 at 1:03 PM, Sebastian Piu <se...@gmail.com>
>> wrote:
>>
>>> I was wondering if there is there any way to skip batches with zero
>>> events when streaming?
>>> By skip I mean avoid the empty rdd from being created at all?
>>>
>>
>>
Re: Skip empty batches - spark streaming
Posted by Sebastian Piu <se...@gmail.com>.
I'm using the Kafka direct stream api but I can have a look on extending it
to have this behaviour
Thanks!
On 11 Feb 2016 9:07 p.m., "Shixiong(Ryan) Zhu" <sh...@databricks.com>
wrote:
> Are you using a custom input dstream? If so, you can make the `compute`
> method return None to skip a batch.
>
> On Thu, Feb 11, 2016 at 1:03 PM, Sebastian Piu <se...@gmail.com>
> wrote:
>
>> I was wondering if there is there any way to skip batches with zero
>> events when streaming?
>> By skip I mean avoid the empty rdd from being created at all?
>>
>
>
Re: Skip empty batches - spark streaming
Posted by "Shixiong(Ryan) Zhu" <sh...@databricks.com>.
Are you using a custom input dstream? If so, you can make the `compute`
method return None to skip a batch.
On Thu, Feb 11, 2016 at 1:03 PM, Sebastian Piu <se...@gmail.com>
wrote:
> I was wondering if there is there any way to skip batches with zero events
> when streaming?
> By skip I mean avoid the empty rdd from being created at all?
>