You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Hemi Grs <he...@gmail.com> on 2023/05/25 00:32:26 UTC
Why I can't run more than 19 tasks?
hey everybody,
I have a problem with my apache flink, I am synchronizing from MySQL to
Elasticsearch but it seems that I can't run more than 19 tasks. it gave me
this error:
------
Caused by: org.apache.flink.util.FlinkRuntimeException:
org.apache.flink.util.FlinkRuntimeException:
java.sql.SQLTransientConnectionException: connection-pool-10.10.10.111:3306
- Connection is not available, request timed out after 30000ms. at
com.ververica.cdc.connectors.mysql.debezium.DebeziumUtils.openJdbcConnection(DebeziumUtils.java:64)
at
com.ververica.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner.discoveryCaptureTables(MySqlSnapshotSplitAssigner.java:171)
... 12 more
Caused by: org.apache.flink.util.FlinkRuntimeException:
java.sql.SQLTransientConnectionException: connection-pool-10.10.10.111:3306
- Connection is not available, request timed out after 30000ms. at
com.ververica.cdc.connectors.mysql.source.connection.JdbcConnectionFactory.connect(JdbcConnectionFactory.java:72)
at io.debezium.jdbc.JdbcConnection.connection(JdbcConnection.java:890) at
io.debezium.jdbc.JdbcConnection.connection(JdbcConnection.java:885)
at io.debezium.jdbc.JdbcConnection.connect(JdbcConnection.java:418) at
com.ververica.cdc.connectors.mysql.debezium.DebeziumUtils.openJdbcConnection(DebeziumUtils.java:61)
... 13 moreCaused by: java.sql.SQLTransientConnectionException:
connection-pool-10.10.10.111:3306 - Connection is not available, request
timed out after 30000ms.
at
com.ververica.cdc.connectors.shaded.com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696)
at
com.ververica.cdc.connectors.shaded.com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:197)
at
com.ververica.cdc.connectors.shaded.com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:162)
at
com.ververica.cdc.connectors.shaded.com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100)
at
com.ververica.cdc.connectors.mysql.source.connection.JdbcConnectionFactory.connect(JdbcConnectionFactory.java:59)
... 17 more
-----
I have try adding this 2 lines on flink-conf.yaml but doesn't do anything:
-----
env.java.opts:
"-Dcom.ververica.cdc.connectors.mysql.hikari.maximumPoolSize=100"
flink.connector.mysql-cdc.max-pool-size: 100
-----
does anybody know the solution?
Additional info, my database is doing fine, because I try creating another
apache flink server and it can run another 19 tasks, so total there 38
tasks running and it's doing fine. So how do I run many tasks on 1 server
and the server still have lots of resources.
Thanks
Re: Why I can't run more than 19 tasks?
Posted by liu ron <ro...@gmail.com>.
Hi, Hemi,
According to your context, it seems that the problem comes from MySQL
client side, maybe you can check the client configuration.
Best,
Ron
Shammon FY <zj...@gmail.com> 于2023年5月25日周四 08:40写道：
> Hi Hemi,
>
> There may be two reasons that I can think of
> 1. The number of connections exceeds the MySQL limit, you can check the
> options in my.cnf for your mysql server and increase the max connections.
> 2. Connection timeout for mysql client, you can try to add
> 'autoReconnect=true' to the connection url
>
> Best,
> Shammon FY
>
>
> On Thu, May 25, 2023 at 8:32 AM Hemi Grs <he...@gmail.com> wrote:
>
>> hey everybody,
>>
>> I have a problem with my apache flink, I am synchronizing from MySQL to
>> Elasticsearch but it seems that I can't run more than 19 tasks. it gave me
>> this error:
>>
>> ------
>> Caused by: org.apache.flink.util.FlinkRuntimeException:
>> org.apache.flink.util.FlinkRuntimeException:
>> java.sql.SQLTransientConnectionException: connection-pool-10.10.10.111:3306
>> - Connection is not available, request timed out after 30000ms. at
>> com.ververica.cdc.connectors.mysql.debezium.DebeziumUtils.openJdbcConnection(DebeziumUtils.java:64)
>> at
>> com.ververica.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner.discoveryCaptureTables(MySqlSnapshotSplitAssigner.java:171)
>> ... 12 more
>> Caused by: org.apache.flink.util.FlinkRuntimeException:
>> java.sql.SQLTransientConnectionException: connection-pool-10.10.10.111:3306
>> - Connection is not available, request timed out after 30000ms. at
>> com.ververica.cdc.connectors.mysql.source.connection.JdbcConnectionFactory.connect(JdbcConnectionFactory.java:72)
>> at io.debezium.jdbc.JdbcConnection.connection(JdbcConnection.java:890) at
>> io.debezium.jdbc.JdbcConnection.connection(JdbcConnection.java:885)
>> at io.debezium.jdbc.JdbcConnection.connect(JdbcConnection.java:418) at
>> com.ververica.cdc.connectors.mysql.debezium.DebeziumUtils.openJdbcConnection(DebeziumUtils.java:61)
>> ... 13 moreCaused by: java.sql.SQLTransientConnectionException:
>> connection-pool-10.10.10.111:3306 - Connection is not available, request
>> timed out after 30000ms.
>> at
>> com.ververica.cdc.connectors.shaded.com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696)
>> at
>> com.ververica.cdc.connectors.shaded.com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:197)
>> at
>> com.ververica.cdc.connectors.shaded.com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:162)
>> at
>> com.ververica.cdc.connectors.shaded.com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100)
>> at
>> com.ververica.cdc.connectors.mysql.source.connection.JdbcConnectionFactory.connect(JdbcConnectionFactory.java:59)
>> ... 17 more
>> -----
>>
>> I have try adding this 2 lines on flink-conf.yaml but doesn't do anything:
>> -----
>> env.java.opts:
>> "-Dcom.ververica.cdc.connectors.mysql.hikari.maximumPoolSize=100"
>> flink.connector.mysql-cdc.max-pool-size: 100
>> -----
>>
>> does anybody know the solution?
>> Additional info, my database is doing fine, because I try creating
>> another apache flink server and it can run another 19 tasks, so total there
>> 38 tasks running and it's doing fine. So how do I run many tasks on 1
>> server and the server still have lots of resources.
>>
>> Thanks
>>
>
Re: Why I can't run more than 19 tasks?
Posted by Shammon FY <zj...@gmail.com>.
Hi Hemi,
There may be two reasons that I can think of
1. The number of connections exceeds the MySQL limit, you can check the
options in my.cnf for your mysql server and increase the max connections.
2. Connection timeout for mysql client, you can try to add
'autoReconnect=true' to the connection url
Best,
Shammon FY
On Thu, May 25, 2023 at 8:32 AM Hemi Grs <he...@gmail.com> wrote:
> hey everybody,
>
> I have a problem with my apache flink, I am synchronizing from MySQL to
> Elasticsearch but it seems that I can't run more than 19 tasks. it gave me
> this error:
>
> ------
> Caused by: org.apache.flink.util.FlinkRuntimeException:
> org.apache.flink.util.FlinkRuntimeException:
> java.sql.SQLTransientConnectionException: connection-pool-10.10.10.111:3306
> - Connection is not available, request timed out after 30000ms. at
> com.ververica.cdc.connectors.mysql.debezium.DebeziumUtils.openJdbcConnection(DebeziumUtils.java:64)
> at
> com.ververica.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner.discoveryCaptureTables(MySqlSnapshotSplitAssigner.java:171)
> ... 12 more
> Caused by: org.apache.flink.util.FlinkRuntimeException:
> java.sql.SQLTransientConnectionException: connection-pool-10.10.10.111:3306
> - Connection is not available, request timed out after 30000ms. at
> com.ververica.cdc.connectors.mysql.source.connection.JdbcConnectionFactory.connect(JdbcConnectionFactory.java:72)
> at io.debezium.jdbc.JdbcConnection.connection(JdbcConnection.java:890) at
> io.debezium.jdbc.JdbcConnection.connection(JdbcConnection.java:885)
> at io.debezium.jdbc.JdbcConnection.connect(JdbcConnection.java:418) at
> com.ververica.cdc.connectors.mysql.debezium.DebeziumUtils.openJdbcConnection(DebeziumUtils.java:61)
> ... 13 moreCaused by: java.sql.SQLTransientConnectionException:
> connection-pool-10.10.10.111:3306 - Connection is not available, request
> timed out after 30000ms.
> at
> com.ververica.cdc.connectors.shaded.com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696)
> at
> com.ververica.cdc.connectors.shaded.com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:197)
> at
> com.ververica.cdc.connectors.shaded.com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:162)
> at
> com.ververica.cdc.connectors.shaded.com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100)
> at
> com.ververica.cdc.connectors.mysql.source.connection.JdbcConnectionFactory.connect(JdbcConnectionFactory.java:59)
> ... 17 more
> -----
>
> I have try adding this 2 lines on flink-conf.yaml but doesn't do anything:
> -----
> env.java.opts:
> "-Dcom.ververica.cdc.connectors.mysql.hikari.maximumPoolSize=100"
> flink.connector.mysql-cdc.max-pool-size: 100
> -----
>
> does anybody know the solution?
> Additional info, my database is doing fine, because I try creating another
> apache flink server and it can run another 19 tasks, so total there 38
> tasks running and it's doing fine. So how do I run many tasks on 1 server
> and the server still have lots of resources.
>
> Thanks
>