You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Vijayant Kumar <Vi...@mavenir.com.INVALID> on 2020/05/06 08:36:00 UTC
Pyspark Kafka Structured Stream not working.
Hi All,
I am getting the below error while using Pyspark Structured Streaming from Kafka Producer.
20/05/06 11:51:16 ERROR ReceiverTracker: Deregistered receiver for stream 0: Error starting receiver 0 - kafka.common.InvalidConfigException: client.id Python Kafka streamer is illegal, contains a character other than ASCII alphanumerics, '.', '_' and '-'
I am using the below code to get the messages:
broker='vm105:2181'
topic='Hello-Kafka'
print 'broker topic is ',broker,topic
kvs = KafkaUtils.createStream(ssc, \
broker, \
"Python Kafka streamer",{topic:1})
And my Submit command is like below :-
spark-submit --jars spark-streaming-kafka-0-8-assembly_2.11-2.4.4.jar test_kafka.py vm105:2181 Hello-Kafka
Can any one help me what am I missing. ?
Thanks,
Vijayant
________________________________
This e-mail message may contain confidential or proprietary information of Mavenir Systems, Inc. or its affiliates and is intended solely for the use of the intended recipient(s). If you are not the intended recipient of this message, you are hereby notified that any review, use or distribution of this information is absolutely prohibited and we request that you delete all copies in your control and contact us by e-mailing to security@mavenir.com. This message contains the views of its author and may not necessarily reflect the views of Mavenir Systems, Inc. or its affiliates, who employ systems to monitor email messages, but make no representation that such messages are authorized, secure, uncompromised, or free from computer viruses, malware, or other defects. Thank You
Re: [E] Re: Pyspark Kafka Structured Stream not working.
Posted by Jungtaek Lim <ka...@gmail.com>.
It's not either 1 or 2. Both two items are applied. I haven't played with
DStream + pyspark but given the error message is clear you'll probably want
to change the client.id "Python Kafka streamer" to accommodate the naming
convention guided in error message.
On Thu, May 7, 2020 at 3:55 PM Vijayant Kumar <Vi...@mavenir.com>
wrote:
> Hi Jungtek,
>
>
>
> Thanks for the response. It appears to be #1.
>
> I will appreciate if you can share some sample command to submit the Spark
> application.?
>
>
>
> *From:* Jungtaek Lim [mailto:kabhwan.opensource@gmail.com]
> *Sent:* Wednesday, May 06, 2020 8:24 PM
> *To:* Vijayant Kumar <Vi...@mavenir.com.invalid>
> *Cc:* user@spark.apache.org
> *Subject:* [E] Re: Pyspark Kafka Structured Stream not working.
>
>
>
> *[EXTERNAL EMAIL]* DO NOT CLICK links or attachments unless you recognize
> the sender and know the content is safe.
>
> Hi,
>
>
>
> 1. You seem to use DStream (Spark Streaming), not Structured Streaming.
>
> 2. I'm not familiar with pyspark, but looks like the error message is very
> clear - Kafka doesn't allow such name for "client.id". The error message
> guides the naming rule, so you may need to be adopted with such convention.
> (e.g. no space)
>
>
>
> Hope this helps,
>
>
>
> Thanks,
>
> Jungtaek Lim (HeartSaVioR)
>
>
>
> On Wed, May 6, 2020 at 5:36 PM Vijayant Kumar <
> Vijayant.Kumar@mavenir.com.invalid> wrote:
>
> Hi All,
>
>
>
> I am getting the below error while using Pyspark Structured Streaming from
> Kafka Producer.
>
>
>
> 20/05/06 11:51:16 ERROR ReceiverTracker: Deregistered receiver for stream
> 0: Error starting receiver 0 - kafka.common.InvalidConfigException:
> client.id Python Kafka streamer is illegal, contains a character other
> than ASCII alphanumerics, '.', '_' and '-'
>
>
>
> I am using the below code to get the messages:
>
>
>
> broker='vm105:2181'
>
> topic='Hello-Kafka'
>
> print 'broker topic is ',broker,topic
>
> kvs = KafkaUtils.createStream(ssc, \
>
> broker, \
>
> "Python Kafka streamer",{topic:1})
>
>
>
> And my Submit command is like below :-
>
> *spark-submit --jars spark-streaming-kafka-0-8-assembly_2.11-2.4.4.jar
> test_kafka.py vm105:2181 Hello-Kafka*
>
>
>
> Can any one help me what am I missing. ?
>
>
>
> Thanks,
>
> Vijayant
> ------------------------------
>
> This e-mail message may contain confidential or proprietary information of
> Mavenir Systems, Inc. or its affiliates and is intended solely for the use
> of the intended recipient(s). If you are not the intended recipient of this
> message, you are hereby notified that any review, use or distribution of
> this information is absolutely prohibited and we request that you delete
> all copies in your control and contact us by e-mailing to
> security@mavenir.com. This message contains the views of its author and
> may not necessarily reflect the views of Mavenir Systems, Inc. or its
> affiliates, who employ systems to monitor email messages, but make no
> representation that such messages are authorized, secure, uncompromised, or
> free from computer viruses, malware, or other defects. Thank You
>
> ------------------------------
>
> This e-mail message may contain confidential or proprietary information of
> Mavenir Systems, Inc. or its affiliates and is intended solely for the use
> of the intended recipient(s). If you are not the intended recipient of this
> message, you are hereby notified that any review, use or distribution of
> this information is absolutely prohibited and we request that you delete
> all copies in your control and contact us by e-mailing to
> security@mavenir.com. This message contains the views of its author and
> may not necessarily reflect the views of Mavenir Systems, Inc. or its
> affiliates, who employ systems to monitor email messages, but make no
> representation that such messages are authorized, secure, uncompromised, or
> free from computer viruses, malware, or other defects. Thank You
>
RE: [E] Re: Pyspark Kafka Structured Stream not working.
Posted by Vijayant Kumar <Vi...@mavenir.com.INVALID>.
Hi Jungtek,
Thanks for the response. It appears to be #1.
I will appreciate if you can share some sample command to submit the Spark application.?
From: Jungtaek Lim [mailto:kabhwan.opensource@gmail.com]
Sent: Wednesday, May 06, 2020 8:24 PM
To: Vijayant Kumar <Vi...@mavenir.com.invalid>
Cc: user@spark.apache.org
Subject: [E] Re: Pyspark Kafka Structured Stream not working.
[EXTERNAL EMAIL] DO NOT CLICK links or attachments unless you recognize the sender and know the content is safe.
Hi,
1. You seem to use DStream (Spark Streaming), not Structured Streaming.
2. I'm not familiar with pyspark, but looks like the error message is very clear - Kafka doesn't allow such name for "client.id<http://client.id>". The error message guides the naming rule, so you may need to be adopted with such convention. (e.g. no space)
Hope this helps,
Thanks,
Jungtaek Lim (HeartSaVioR)
On Wed, May 6, 2020 at 5:36 PM Vijayant Kumar <Vi...@mavenir.com.invalid>> wrote:
Hi All,
I am getting the below error while using Pyspark Structured Streaming from Kafka Producer.
20/05/06 11:51:16 ERROR ReceiverTracker: Deregistered receiver for stream 0: Error starting receiver 0 - kafka.common.InvalidConfigException: client.id<http://client.id> Python Kafka streamer is illegal, contains a character other than ASCII alphanumerics, '.', '_' and '-'
I am using the below code to get the messages:
broker='vm105:2181'
topic='Hello-Kafka'
print 'broker topic is ',broker,topic
kvs = KafkaUtils.createStream(ssc, \
broker, \
"Python Kafka streamer",{topic:1})
And my Submit command is like below :-
spark-submit --jars spark-streaming-kafka-0-8-assembly_2.11-2.4.4.jar test_kafka.py vm105:2181 Hello-Kafka
Can any one help me what am I missing. ?
Thanks,
Vijayant
________________________________
This e-mail message may contain confidential or proprietary information of Mavenir Systems, Inc. or its affiliates and is intended solely for the use of the intended recipient(s). If you are not the intended recipient of this message, you are hereby notified that any review, use or distribution of this information is absolutely prohibited and we request that you delete all copies in your control and contact us by e-mailing to security@mavenir.com<ma...@mavenir.com>. This message contains the views of its author and may not necessarily reflect the views of Mavenir Systems, Inc. or its affiliates, who employ systems to monitor email messages, but make no representation that such messages are authorized, secure, uncompromised, or free from computer viruses, malware, or other defects. Thank You
________________________________
This e-mail message may contain confidential or proprietary information of Mavenir Systems, Inc. or its affiliates and is intended solely for the use of the intended recipient(s). If you are not the intended recipient of this message, you are hereby notified that any review, use or distribution of this information is absolutely prohibited and we request that you delete all copies in your control and contact us by e-mailing to security@mavenir.com. This message contains the views of its author and may not necessarily reflect the views of Mavenir Systems, Inc. or its affiliates, who employ systems to monitor email messages, but make no representation that such messages are authorized, secure, uncompromised, or free from computer viruses, malware, or other defects. Thank You
Re: Pyspark Kafka Structured Stream not working.
Posted by Jungtaek Lim <ka...@gmail.com>.
Hi,
1. You seem to use DStream (Spark Streaming), not Structured Streaming.
2. I'm not familiar with pyspark, but looks like the error message is very
clear - Kafka doesn't allow such name for "client.id". The error message
guides the naming rule, so you may need to be adopted with such convention.
(e.g. no space)
Hope this helps,
Thanks,
Jungtaek Lim (HeartSaVioR)
On Wed, May 6, 2020 at 5:36 PM Vijayant Kumar
<Vi...@mavenir.com.invalid> wrote:
> Hi All,
>
>
>
> I am getting the below error while using Pyspark Structured Streaming from
> Kafka Producer.
>
>
>
> 20/05/06 11:51:16 ERROR ReceiverTracker: Deregistered receiver for stream
> 0: Error starting receiver 0 - kafka.common.InvalidConfigException:
> client.id Python Kafka streamer is illegal, contains a character other
> than ASCII alphanumerics, '.', '_' and '-'
>
>
>
> I am using the below code to get the messages:
>
>
>
> broker='vm105:2181'
>
> topic='Hello-Kafka'
>
> print 'broker topic is ',broker,topic
>
> kvs = KafkaUtils.createStream(ssc, \
>
> broker, \
>
> "Python Kafka streamer",{topic:1})
>
>
>
> And my Submit command is like below :-
>
> *spark-submit --jars spark-streaming-kafka-0-8-assembly_2.11-2.4.4.jar
> test_kafka.py vm105:2181 Hello-Kafka*
>
>
>
> Can any one help me what am I missing. ?
>
>
>
> Thanks,
>
> Vijayant
> ------------------------------
>
> This e-mail message may contain confidential or proprietary information of
> Mavenir Systems, Inc. or its affiliates and is intended solely for the use
> of the intended recipient(s). If you are not the intended recipient of this
> message, you are hereby notified that any review, use or distribution of
> this information is absolutely prohibited and we request that you delete
> all copies in your control and contact us by e-mailing to
> security@mavenir.com. This message contains the views of its author and
> may not necessarily reflect the views of Mavenir Systems, Inc. or its
> affiliates, who employ systems to monitor email messages, but make no
> representation that such messages are authorized, secure, uncompromised, or
> free from computer viruses, malware, or other defects. Thank You
>