You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/09/06 06:33:48 UTC
[GitHub] [beam] balabarath opened a new issue, #23036: [Bug]: AWS SQSIO - SocketTimeoutException is thrown after 10 minutes of inactivity
balabarath opened a new issue, #23036:
URL: https://github.com/apache/beam/issues/23036
### What happened?
When there no messages available in the SQS queue for more than 10 minutes, the following exception has been thrown
`Error message from worker: java.io.IOException: Failed to advance source: org.apache.beam.sdk.io.aws.sqs.SqsUnboundedSource@32ea58b7
org.apache.beam.runners.dataflow.worker.WorkerCustomSources$UnboundedReaderIterator.advance(WorkerCustomSources.java:826)
org.apache.beam.runners.dataflow.worker.WorkerCustomSources$UnboundedReaderIterator.start(WorkerCustomSources.java:794)
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation$SynchronizedReaderIterator.start(ReadOperation.java:375)
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:205)
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:163)
org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:83)
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1445)
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1100(StreamingDataflowWorker.java:165)
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$7.run(StreamingDataflowWorker.java:1120)
org.apache.beam.runners.dataflow.worker.util.BoundedQueueExecutor.lambda$executeLockHeld$0(BoundedQueueExecutor.java:133)
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
java.base/java.lang.Thread.run(Thread.java:834)
Caused by: com.amazonaws.SdkClientException: Unable to execute HTTP request: Connect to sqs.eu-central-1.amazonaws.com:443 [sqs.eu-central-1.amazonaws.com/18.157.237.193] failed: connect timed out
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1216)
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1162)
com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:811)
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:779)
com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:753)
com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:713)
com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:695)
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:559)
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:539)
com.amazonaws.services.sqs.AmazonSQSClient.doInvoke(AmazonSQSClient.java:2271)
com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:2238)
com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:2227)
com.amazonaws.services.sqs.AmazonSQSClient.executeReceiveMessage(AmazonSQSClient.java:1657)
com.amazonaws.services.sqs.AmazonSQSClient.receiveMessage(AmazonSQSClient.java:1625)
org.apache.beam.sdk.io.aws.sqs.SqsUnboundedReader.pull(SqsUnboundedReader.java:660)
org.apache.beam.sdk.io.aws.sqs.SqsUnboundedReader.advance(SqsUnboundedReader.java:501)
org.apache.beam.runners.dataflow.worker.WorkerCustomSources$UnboundedReaderIterator.advance(WorkerCustomSources.java:821)
org.apache.beam.runners.dataflow.worker.WorkerCustomSources$UnboundedReaderIterator.start(WorkerCustomSources.java:794)
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation$SynchronizedReaderIterator.start(ReadOperation.java:375)
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:205)
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:163)
org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:83)
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1445)
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1100(StreamingDataflowWorker.java:165)
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$7.run(StreamingDataflowWorker.java:1120)
org.apache.beam.runners.dataflow.worker.util.BoundedQueueExecutor.lambda$executeLockHeld$0(BoundedQueueExecutor.java:133)
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to sqs.eu-central-1.amazonaws.com:443 [sqs.eu-central-1.amazonaws.com/18.157.237.193] failed: connect timed out
org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151)
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
jdk.internal.reflect.GeneratedMethodAccessor67.invoke(Unknown Source)
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.base/java.lang.reflect.Method.invoke(Method.java:566)
com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76)
com.amazonaws.http.conn.$Proxy74.connect(Unknown Source)
org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1343)
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1154)
com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:811)
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:779)
com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:753)
com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:713)
com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:695)
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:559)
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:539)
com.amazonaws.services.sqs.AmazonSQSClient.doInvoke(AmazonSQSClient.java:2271)
com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:2238)
com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:2227)
com.amazonaws.services.sqs.AmazonSQSClient.executeReceiveMessage(AmazonSQSClient.java:1657)
com.amazonaws.services.sqs.AmazonSQSClient.receiveMessage(AmazonSQSClient.java:1625)
org.apache.beam.sdk.io.aws.sqs.SqsUnboundedReader.pull(SqsUnboundedReader.java:660)
org.apache.beam.sdk.io.aws.sqs.SqsUnboundedReader.advance(SqsUnboundedReader.java:501)
org.apache.beam.runners.dataflow.worker.WorkerCustomSources$UnboundedReaderIterator.advance(WorkerCustomSources.java:821)
org.apache.beam.runners.dataflow.worker.WorkerCustomSources$UnboundedReaderIterator.start(WorkerCustomSources.java:794)
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation$SynchronizedReaderIterator.start(ReadOperation.java:375)
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:205)
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:163)
org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:83)
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1445)
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1100(StreamingDataflowWorker.java:165)
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$7.run(StreamingDataflowWorker.java:1120)
org.apache.beam.runners.dataflow.worker.util.BoundedQueueExecutor.lambda$executeLockHeld$0(BoundedQueueExecutor.java:133)
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.net.SocketTimeoutException: connect timed out
java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
java.base/java.net.Socket.connect(Socket.java:609)
org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:368)
com.amazonaws.http.conn.ssl.SdkTLSSocketFactory.connectSocket(SdkTLSSocketFactory.java:142)
org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
jdk.internal.reflect.GeneratedMethodAccessor67.invoke(Unknown Source)
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.base/java.lang.reflect.Method.invoke(Method.java:566)
com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76)
com.amazonaws.http.conn.$Proxy74.connect(Unknown Source)
org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1343)
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1154)
com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:811)
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:779)
com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:753)
com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:713)
com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:695)
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:559)
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:539)
com.amazonaws.services.sqs.AmazonSQSClient.doInvoke(AmazonSQSClient.java:2271)
com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:2238)
com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:2227)
com.amazonaws.services.sqs.AmazonSQSClient.executeReceiveMessage(AmazonSQSClient.java:1657)
com.amazonaws.services.sqs.AmazonSQSClient.receiveMessage(AmazonSQSClient.java:1625)
org.apache.beam.sdk.io.aws.sqs.SqsUnboundedReader.pull(SqsUnboundedReader.java:660)
org.apache.beam.sdk.io.aws.sqs.SqsUnboundedReader.advance(SqsUnboundedReader.java:501)
org.apache.beam.runners.dataflow.worker.WorkerCustomSources$UnboundedReaderIterator.advance(WorkerCustomSources.java:821)
org.apache.beam.runners.dataflow.worker.WorkerCustomSources$UnboundedReaderIterator.start(WorkerCustomSources.java:794)
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation$SynchronizedReaderIterator.start(ReadOperation.java:375)
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:205)
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:163)
org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:83)
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1445)
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1100(StreamingDataflowWorker.java:165)
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$7.run(StreamingDataflowWorker.java:1120)
org.apache.beam.runners.dataflow.worker.util.BoundedQueueExecutor.lambda$executeLockHeld$0(BoundedQueueExecutor.java:133)
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
java.base/java.lang.Thread.run(Thread.java:834)`
But it doesn't affect the message processing. The messages are still being processed.
### Issue Priority
Priority: 2
### Issue Component
Component: io-java-aws
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on issue #23036: [Bug]: AWS SQSIO - SocketTimeoutException is thrown after 5 minutes of inactivity
Posted by GitBox <gi...@apache.org>.
johnjcasey commented on issue #23036:
URL: https://github.com/apache/beam/issues/23036#issuecomment-1245427723
That probably means we can simply catch and log this error at a low priority.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] nsbala-tw commented on issue #23036: [Bug]: AWS SQSIO - SocketTimeoutException is thrown after 5 minutes of inactivity
Posted by GitBox <gi...@apache.org>.
nsbala-tw commented on issue #23036:
URL: https://github.com/apache/beam/issues/23036#issuecomment-1244923693
Yes. I don't see any issues in message processing flow. But after some inactivity, this error has been thrown.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] balabarath commented on issue #23036: [Bug]: AWS SQSIO - SocketTimeoutException is thrown after 5 minutes of inactivity
Posted by GitBox <gi...@apache.org>.
balabarath commented on issue #23036:
URL: https://github.com/apache/beam/issues/23036#issuecomment-1247827434
.take-issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] nsbala-tw commented on issue #23036: [Bug]: AWS SQSIO - SocketTimeoutException is thrown after 5 minutes of inactivity
Posted by GitBox <gi...@apache.org>.
nsbala-tw commented on issue #23036:
URL: https://github.com/apache/beam/issues/23036#issuecomment-1247826653
I would like to .take-issue to fix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on issue #23036: [Bug]: AWS SQSIO - SocketTimeoutException is thrown after 5 minutes of inactivity
Posted by GitBox <gi...@apache.org>.
johnjcasey commented on issue #23036:
URL: https://github.com/apache/beam/issues/23036#issuecomment-1242493247
@balabarath if more messages come onto the queue, does the IO resume properly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] balabarath commented on issue #23036: [Bug]: AWS SQSIO - SocketTimeoutException is thrown after 5 minutes of inactivity
Posted by GitBox <gi...@apache.org>.
balabarath commented on issue #23036:
URL: https://github.com/apache/beam/issues/23036#issuecomment-1244965203
@johnjcasey Yes. I don't see any issues in the message processing flow. But after some inactivity, this error has been thrown.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on issue #23036: [Bug]: AWS SQSIO - SocketTimeoutException is thrown after 5 minutes of inactivity
Posted by "mosche (via GitHub)" <gi...@apache.org>.
mosche commented on issue #23036:
URL: https://github.com/apache/beam/issues/23036#issuecomment-1639912045
@balabarath Kind ping, are you still working on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org