You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Claude M <cl...@gmail.com> on 2021/02/26 20:02:10 UTC
Producer Configuration
Hello,
I created a simple Producer and when the job ran, it was getting the
following error:
Caused by: org.apache.kafka.common.errors.TimeoutException
I read about increasing the request.timeout.ms. Thus, I added the
following properties.
Properties properties = new Properties();
properties.setProperty("request.timeout.ms", "30000");
properties.setProperty("retries", "20");
DataStream<String> stream = env.addSource(new SimpleStringGenerator());
stream.addSink(new FlinkKafkaProducer<>("flink-test", new
SimpleStringSchema(), properties));
However, after the job is submitted, the User Configuration is empty,
please see attached.
Therefore, it seems these properties are taking into effect since I still
have the same problem.
Any help on these issues are appreciated, thanks.
Re: Producer Configuration
Posted by Dawid Wysakowicz <dw...@apache.org>.
Hey Claude.
Alexey is right about the page. The page from your screenshot shows only
the entries passed via
StreamExecutionEnvironment#getConfig#setGlobalJobParameters.
Configuration for individual connectors or other operators is not
displayed there.
If you need help debugging your time out issue could you post your
configuration and the stacktrace you are getting?
Best,
Dawid
On 28/02/2021 03:14, Alexey Trenikhun wrote:
> They are picked up, otherwise you would not able to write any messages
> at all. I believe the page you referring is not for displaying Kafka
> properties (in my case it is empty as well, but Kafka works). Check logs.
>
> ------------------------------------------------------------------------
> *From:* Claude M <cl...@gmail.com>
> *Sent:* Saturday, February 27, 2021 4:00:23 PM
> *To:* Alexey Trenikhun <ye...@msn.com>
> *Cc:* user <us...@flink.apache.org>
> *Subject:* Re: Producer Configuration
>
> Yes, the flink job also works in producing messages. It's just that
> after a short period of time, it fails w/ a timeout. That is why I'm
> trying to set a longer timeout period but it doesn't seem like the
> properties are being picked up.
>
> On Sat, Feb 27, 2021 at 1:17 PM Alexey Trenikhun <yender@msn.com
> <ma...@msn.com>> wrote:
>
> Can you produce messages using Kafka console producer connect
> using same properties ?
>
> ------------------------------------------------------------------------
> *From:* Claude M <claudemurad1@gmail.com
> <ma...@gmail.com>>
> *Sent:* Saturday, February 27, 2021 8:05 AM
> *To:* Alexey Trenikhun <yender@msn.com <ma...@msn.com>>
> *Cc:* user <user@flink.apache.org <ma...@flink.apache.org>>
> *Subject:* Re: Producer Configuration
>
> Thanks for your reply, yes it was specified. Sorry I forgot to
> include it:
> properties.setProperty("bootstrap.servers", "localhost:9092");
>
> On Fri, Feb 26, 2021 at 7:56 PM Alexey Trenikhun <yender@msn.com
> <ma...@msn.com>> wrote:
>
> I believe bootstrap.servers is mandatory Kafka property, but
> it looks like you didn’t set it
>
> ------------------------------------------------------------------------
> *From:* Claude M <claudemurad1@gmail.com
> <ma...@gmail.com>>
> *Sent:* Friday, February 26, 2021 12:02:10 PM
> *To:* user <user@flink.apache.org <ma...@flink.apache.org>>
> *Subject:* Producer Configuration
>
> Hello,
>
> I created a simple Producer and when the job ran, it was
> getting the following error:
> Caused by: org.apache.kafka.common.errors.TimeoutException
>
> I read about increasing the request.timeout.ms
> <http://request.timeout.ms>. Thus, I added the following
> properties.
>
> Properties properties = new Properties();
> properties.setProperty("request.timeout.ms
> <http://request.timeout.ms>", "30000");
> properties.setProperty("retries", "20");
> DataStream<String> stream = env.addSource(new
> SimpleStringGenerator());
> stream.addSink(new FlinkKafkaProducer<>("flink-test", new
> SimpleStringSchema(), properties));
>
> However, after the job is submitted, the User Configuration is
> empty, please see attached.
> Therefore, it seems these properties are taking into effect
> since I still have the same problem.
> Any help on these issues are appreciated, thanks.
>
Re: Producer Configuration
Posted by Alexey Trenikhun <ye...@msn.com>.
They are picked up, otherwise you would not able to write any messages at all. I believe the page you referring is not for displaying Kafka properties (in my case it is empty as well, but Kafka works). Check logs.
________________________________
From: Claude M <cl...@gmail.com>
Sent: Saturday, February 27, 2021 4:00:23 PM
To: Alexey Trenikhun <ye...@msn.com>
Cc: user <us...@flink.apache.org>
Subject: Re: Producer Configuration
Yes, the flink job also works in producing messages. It's just that after a short period of time, it fails w/ a timeout. That is why I'm trying to set a longer timeout period but it doesn't seem like the properties are being picked up.
On Sat, Feb 27, 2021 at 1:17 PM Alexey Trenikhun <ye...@msn.com>> wrote:
Can you produce messages using Kafka console producer connect using same properties ?
________________________________
From: Claude M <cl...@gmail.com>>
Sent: Saturday, February 27, 2021 8:05 AM
To: Alexey Trenikhun <ye...@msn.com>>
Cc: user <us...@flink.apache.org>>
Subject: Re: Producer Configuration
Thanks for your reply, yes it was specified. Sorry I forgot to include it:
properties.setProperty("bootstrap.servers", "localhost:9092");
On Fri, Feb 26, 2021 at 7:56 PM Alexey Trenikhun <ye...@msn.com>> wrote:
I believe bootstrap.servers is mandatory Kafka property, but it looks like you didn’t set it
________________________________
From: Claude M <cl...@gmail.com>>
Sent: Friday, February 26, 2021 12:02:10 PM
To: user <us...@flink.apache.org>>
Subject: Producer Configuration
Hello,
I created a simple Producer and when the job ran, it was getting the following error:
Caused by: org.apache.kafka.common.errors.TimeoutException
I read about increasing the request.timeout.ms<http://request.timeout.ms>. Thus, I added the following properties.
Properties properties = new Properties();
properties.setProperty("request.timeout.ms<http://request.timeout.ms>", "30000");
properties.setProperty("retries", "20");
DataStream<String> stream = env.addSource(new SimpleStringGenerator());
stream.addSink(new FlinkKafkaProducer<>("flink-test", new SimpleStringSchema(), properties));
However, after the job is submitted, the User Configuration is empty, please see attached.
Therefore, it seems these properties are taking into effect since I still have the same problem.
Any help on these issues are appreciated, thanks.
Re: Producer Configuration
Posted by Claude M <cl...@gmail.com>.
Yes, the flink job also works in producing messages. It's just that after
a short period of time, it fails w/ a timeout. That is why I'm trying to
set a longer timeout period but it doesn't seem like the properties are
being picked up.
On Sat, Feb 27, 2021 at 1:17 PM Alexey Trenikhun <ye...@msn.com> wrote:
> Can you produce messages using Kafka console producer connect using same
> properties ?
>
> ------------------------------
> *From:* Claude M <cl...@gmail.com>
> *Sent:* Saturday, February 27, 2021 8:05 AM
> *To:* Alexey Trenikhun <ye...@msn.com>
> *Cc:* user <us...@flink.apache.org>
> *Subject:* Re: Producer Configuration
>
> Thanks for your reply, yes it was specified. Sorry I forgot to include
> it:
> properties.setProperty("bootstrap.servers", "localhost:9092");
>
> On Fri, Feb 26, 2021 at 7:56 PM Alexey Trenikhun <ye...@msn.com> wrote:
>
> I believe bootstrap.servers is mandatory Kafka property, but it looks like
> you didn’t set it
>
> ------------------------------
> *From:* Claude M <cl...@gmail.com>
> *Sent:* Friday, February 26, 2021 12:02:10 PM
> *To:* user <us...@flink.apache.org>
> *Subject:* Producer Configuration
>
> Hello,
>
> I created a simple Producer and when the job ran, it was getting the
> following error:
> Caused by: org.apache.kafka.common.errors.TimeoutException
>
> I read about increasing the request.timeout.ms. Thus, I added the
> following properties.
>
> Properties properties = new Properties();
> properties.setProperty("request.timeout.ms", "30000");
> properties.setProperty("retries", "20");
> DataStream<String> stream = env.addSource(new SimpleStringGenerator());
> stream.addSink(new FlinkKafkaProducer<>("flink-test", new
> SimpleStringSchema(), properties));
>
> However, after the job is submitted, the User Configuration is empty,
> please see attached.
> Therefore, it seems these properties are taking into effect since I still
> have the same problem.
> Any help on these issues are appreciated, thanks.
>
>
Re: Producer Configuration
Posted by Alexey Trenikhun <ye...@msn.com>.
Can you produce messages using Kafka console producer connect using same properties ?
________________________________
From: Claude M <cl...@gmail.com>
Sent: Saturday, February 27, 2021 8:05 AM
To: Alexey Trenikhun <ye...@msn.com>
Cc: user <us...@flink.apache.org>
Subject: Re: Producer Configuration
Thanks for your reply, yes it was specified. Sorry I forgot to include it:
properties.setProperty("bootstrap.servers", "localhost:9092");
On Fri, Feb 26, 2021 at 7:56 PM Alexey Trenikhun <ye...@msn.com>> wrote:
I believe bootstrap.servers is mandatory Kafka property, but it looks like you didn’t set it
________________________________
From: Claude M <cl...@gmail.com>>
Sent: Friday, February 26, 2021 12:02:10 PM
To: user <us...@flink.apache.org>>
Subject: Producer Configuration
Hello,
I created a simple Producer and when the job ran, it was getting the following error:
Caused by: org.apache.kafka.common.errors.TimeoutException
I read about increasing the request.timeout.ms<http://request.timeout.ms>. Thus, I added the following properties.
Properties properties = new Properties();
properties.setProperty("request.timeout.ms<http://request.timeout.ms>", "30000");
properties.setProperty("retries", "20");
DataStream<String> stream = env.addSource(new SimpleStringGenerator());
stream.addSink(new FlinkKafkaProducer<>("flink-test", new SimpleStringSchema(), properties));
However, after the job is submitted, the User Configuration is empty, please see attached.
Therefore, it seems these properties are taking into effect since I still have the same problem.
Any help on these issues are appreciated, thanks.
Re: Producer Configuration
Posted by Claude M <cl...@gmail.com>.
Thanks for your reply, yes it was specified. Sorry I forgot to include it:
properties.setProperty("bootstrap.servers", "localhost:9092");
On Fri, Feb 26, 2021 at 7:56 PM Alexey Trenikhun <ye...@msn.com> wrote:
> I believe bootstrap.servers is mandatory Kafka property, but it looks like
> you didn’t set it
>
> ------------------------------
> *From:* Claude M <cl...@gmail.com>
> *Sent:* Friday, February 26, 2021 12:02:10 PM
> *To:* user <us...@flink.apache.org>
> *Subject:* Producer Configuration
>
> Hello,
>
> I created a simple Producer and when the job ran, it was getting the
> following error:
> Caused by: org.apache.kafka.common.errors.TimeoutException
>
> I read about increasing the request.timeout.ms. Thus, I added the
> following properties.
>
> Properties properties = new Properties();
> properties.setProperty("request.timeout.ms", "30000");
> properties.setProperty("retries", "20");
> DataStream<String> stream = env.addSource(new SimpleStringGenerator());
> stream.addSink(new FlinkKafkaProducer<>("flink-test", new
> SimpleStringSchema(), properties));
>
> However, after the job is submitted, the User Configuration is empty,
> please see attached.
> Therefore, it seems these properties are taking into effect since I still
> have the same problem.
> Any help on these issues are appreciated, thanks.
>
Re: Producer Configuration
Posted by Alexey Trenikhun <ye...@msn.com>.
I believe bootstrap.servers is mandatory Kafka property, but it looks like you didn’t set it
________________________________
From: Claude M <cl...@gmail.com>
Sent: Friday, February 26, 2021 12:02:10 PM
To: user <us...@flink.apache.org>
Subject: Producer Configuration
Hello,
I created a simple Producer and when the job ran, it was getting the following error:
Caused by: org.apache.kafka.common.errors.TimeoutException
I read about increasing the request.timeout.ms<http://request.timeout.ms>. Thus, I added the following properties.
Properties properties = new Properties();
properties.setProperty("request.timeout.ms<http://request.timeout.ms>", "30000");
properties.setProperty("retries", "20");
DataStream<String> stream = env.addSource(new SimpleStringGenerator());
stream.addSink(new FlinkKafkaProducer<>("flink-test", new SimpleStringSchema(), properties));
However, after the job is submitted, the User Configuration is empty, please see attached.
Therefore, it seems these properties are taking into effect since I still have the same problem.
Any help on these issues are appreciated, thanks.