You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by shyla deshpande <de...@gmail.com> on 2017/07/10 00:17:10 UTC
Spark streaming giving me a bunch of WARNINGS, please help me
understand them
WARN Use an existing SparkContext, some configuration may not take effect.
I wanted to restart the spark streaming app, so stopped the
running and issued a new spark submit. Why and how it will use a existing
SparkContext?
WARN Spark is not running in local mode, therefore the checkpoint
directory must not be on the local filesystem. Directory
'file:/efs/checkpoint' appears to be on the local filesystem.
WARN overriding enable.auto.commit to false for executor
WARN overriding auto.offset.reset to none for executor
WARN overriding executor group.id to spark-executor-mygroupid
WARN overriding receive.buffer.bytes to 65536 see KAFKA-3135
WARN overriding enable.auto.commit to false for executor
WARN overriding auto.offset.reset to none for executor
Re: Spark streaming giving me a bunch of WARNINGS, please help
meunderstand them
Posted by Cody Koeninger <co...@koeninger.org>.
The warnings regarding configuration on the executor are for the
executor kafka consumer, not the driver kafka consumer.
In general, the executor kafka consumers should consume only exactly
the offsets the driver told them to, and not be involved in committing
offsets / part of the same group as the driver. That's why those
configurations are forcibly overriden.
In short those warnings are normal, the main thing you should be
concerned about is the checkpoint.
On Mon, Jul 10, 2017 at 10:26 AM, shyla deshpande
<de...@gmail.com> wrote:
> WARN Use an existing SparkContext, some configuration may not take effect.
> I wanted to restart the spark streaming app, so stopped the running
> and issued a new spark submit. Why and how it will use a existing
> SparkContext?
> => you are using checkpoint to restore the sparkcontext.
> => No, I am not using checkpoint for recovery. I need the checkpoint because
> I am doing stateful streaming.
>
> WARN Spark is not running in local mode, therefore the checkpoint directory
> must not be on the local filesystem. Directory 'file:/efs/checkpoint'
> appears to be on the local filesystem.
> =>the CP path should be HDFSand so on. If you want to use local path, the
> cluster model should be local.
> => I am using the AWS EFS mount for checkpoint because I am running in
> standalone mode.
>
> WARN overriding enable.auto.commit to false for executor
> =>stop the executor to commit the offset auto
> =>No, I don't want the autocommit. I do the commit later after my output
> operation using the commitAsync API
>
> WARN overriding auto.offset.reset to none for executor
> =>it set the index where the executor read msg
> => I set this to none, because I want it to continue from where it left off
>
> WARN overriding executor group.id to spark-executor-mygroupid
> => set the groupid of consumer. If you do not set, it will set a default and
> give a warning.
> => I am setting the groupid, mygroupid and it is adding spark-executor- as a
> prefix...
>
>
> On Mon, Jul 10, 2017 at 12:39 AM, 萝卜丝炒饭 <14...@qq.com> wrote:
>>
>> It seems you are usibg kafka 0.10.
>> See my comments below.
>>
>> ---Original---
>> From: "shyla deshpande"<de...@gmail.com>
>> Date: 2017/7/10 08:17:10
>> To: "user"<us...@spark.apache.org>;
>> Subject: Spark streaming giving me a bunch of WARNINGS, please help
>> meunderstand them
>>
>> WARN Use an existing SparkContext, some configuration may not take
>> effect.
>> I wanted to restart the spark streaming app, so stopped the
>> running and issued a new spark submit. Why and how it will use a existing
>> SparkContext?
>> => you are using checkpoint to restore the sparkcontext.
>> WARN Spark is not running in local mode, therefore the checkpoint
>> directory must not be on the local filesystem. Directory
>> 'file:/efs/checkpoint' appears to be on the local filesystem.
>> =>the CP path should be HDFSand so on.
>> If you want to use local path, the cluster model should be local.
>>
>>
>> WARN overriding enable.auto.commit to false for executor
>> =>stop the executor to commit the offset auto.
>>
>> WARN overriding auto.offset.reset to none for executor
>> =>it set the index where the executor read msg
>> WARN overriding executor group.id to spark-executor-mygroupid
>> => set the groupid of consumer. If you do not set, it will set a default
>> and give a warning.
>>
>> WARN overriding receive.buffer.bytes to 65536 see KAFKA-3135
>> WARN overriding enable.auto.commit to false for executor
>> WARN overriding auto.offset.reset to none for executor
>
>
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: Spark streaming giving me a bunch of WARNINGS, please help
meunderstand them
Posted by shyla deshpande <de...@gmail.com>.
WARN Use an existing SparkContext, some configuration may not take effect.
I wanted to restart the spark streaming app, so stopped the
running and issued a new spark submit. Why and how it will use a existing
SparkContext?
=> you are using checkpoint to restore the sparkcontext.
=> No, I am not using checkpoint for recovery. I need the checkpoint
because I am doing stateful streaming.
WARN Spark is not running in local mode, therefore the checkpoint
directory must not be on the local filesystem. Directory
'file:/efs/checkpoint' appears to be on the local filesystem.
=>the CP path should be HDFSand so on. If you want to use local path, the
cluster model should be local.
=> I am using the AWS EFS mount for checkpoint because I am running in
standalone mode.
WARN overriding enable.auto.commit to false for executor
=>stop the executor to commit the offset auto
=>No, I don't want the autocommit. I do the commit later after my output
operation using the commitAsync API
WARN overriding auto.offset.reset to none for executor
=>it set the index where the executor read msg
=> I set this to none, because I want it to continue from where it left off
WARN overriding executor group.id to spark-executor-mygroupid
=> set the groupid of consumer. If you do not set, it will set a default
and give a warning.
=> I am setting the groupid, mygroupid and it is adding spark-executor- as
a prefix...
On Mon, Jul 10, 2017 at 12:39 AM, 萝卜丝炒饭 <14...@qq.com> wrote:
> It seems you are usibg kafka 0.10.
> See my comments below.
>
> ---Original---
> *From:* "shyla deshpande"<de...@gmail.com>
> *Date:* 2017/7/10 08:17:10
> *To:* "user"<us...@spark.apache.org>;
> *Subject:* Spark streaming giving me a bunch of WARNINGS, please help
> meunderstand them
>
> WARN Use an existing SparkContext, some configuration may not take effect.
> I wanted to restart the spark streaming app, so stopped the
> running and issued a new spark submit. Why and how it will use a existing
> SparkContext?
> => you are using checkpoint to restore the sparkcontext.
> WARN Spark is not running in local mode, therefore the checkpoint
> directory must not be on the local filesystem. Directory
> 'file:/efs/checkpoint' appears to be on the local filesystem.
> =>the CP path should be HDFSand so on.
> If you want to use local path, the cluster model should be local.
>
>
> WARN overriding enable.auto.commit to false for executor
> =>stop the executor to commit the offset auto.
>
> WARN overriding auto.offset.reset to none for executor
> =>it set the index where the executor read msg
> WARN overriding executor group.id to spark-executor-mygroupid
> => set the groupid of consumer. If you do not set, it will set a default
> and give a warning.
>
> WARN overriding receive.buffer.bytes to 65536 see KAFKA-3135
> WARN overriding enable.auto.commit to false for executor
> WARN overriding auto.offset.reset to none for executor
>
Re: Spark streaming giving me a bunch of WARNINGS, please help meunderstand them
Posted by 萝卜丝炒饭 <14...@qq.com>.
It seems you are usibg kafka 0.10.
See my comments below.
---Original---
From: "shyla deshpande"<de...@gmail.com>
Date: 2017/7/10 08:17:10
To: "user"<us...@spark.apache.org>;
Subject: Spark streaming giving me a bunch of WARNINGS, please help meunderstand them
WARN Use an existing SparkContext, some configuration may not take effect.
I wanted to restart the spark streaming app, so stopped the running and issued a new spark submit. Why and how it will use a existing SparkContext?
=> you are using checkpoint to restore the sparkcontext.
WARN Spark is not running in local mode, therefore the checkpoint directory must not be on the local filesystem. Directory 'file:/efs/checkpoint' appears to be on the local filesystem.
=>the CP path should be HDFSand so on.
If you want to use local path, the cluster model should be local.
WARN overriding enable.auto.commit to false for executor
=>stop the executor to commit the offset auto.
WARN overriding auto.offset.reset to none for executor
=>it set the index where the executor read msg
WARN overriding executor group.id to spark-executor-mygroupid
=> set the groupid of consumer. If you do not set, it will set a default and give a warning.
WARN overriding receive.buffer.bytes to 65536 see KAFKA-3135
WARN overriding enable.auto.commit to false for executor
WARN overriding auto.offset.reset to none for executor