You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@drill.apache.org by Bob Carter <bc...@battelleecology.org> on 2017/08/03 19:35:58 UTC
Timeout error querying S3 Apache Drill 11
Error: SYSTEM ERROR: ConnectionPoolTimeoutException: Timeout waiting for connection from pool
[Error Id: c5d6ba1f-2868-48b4-8db0-63e027f15bd2 on db-migrate:31010]
(org.apache.drill.exec.work.foreman.ForemanException) Unexpected exception during fragment initialization: Unable to execute HTTP request: Timeout waiting for connection from pool
org.apache.drill.exec.work.foreman.Foreman.run():297
java.util.concurrent.ThreadPoolExecutor.runWorker():1145
java.util.concurrent.ThreadPoolExecutor$Worker.run():615
java.lang.Thread.run():748
Caused By (com.amazonaws.AmazonClientException) Unable to execute HTTP request: Timeout waiting for connection from pool
com.amazonaws.http.AmazonHttpClient.executeHelper():454
com.amazonaws.http.AmazonHttpClient.execute():232
com.amazonaws.services.s3.AmazonS3Client.invoke():3528
com.amazonaws.services.s3.AmazonS3Client.invoke():3480
com.amazonaws.services.s3.AmazonS3Client.listObjects():604
org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus():962
org.apache.hadoop.fs.s3a.S3AFileSystem.listStatus():734
org.apache.drill.exec.store.dfs.DrillFileSystem.listStatus():522
org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory.accessible():160
org.apache.drill.exec.store.dfs.FileSystemSchemaFactory$FileSystemSchema.<init>():77
org.apache.drill.exec.store.dfs.FileSystemSchemaFactory.registerSchemas():64
org.apache.drill.exec.store.dfs.FileSystemPlugin.registerSchemas():149
org.apache.drill.exec.store.StoragePluginRegistryImpl$DrillSchemaFactory.registerSchemas():396
org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema():110
org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema():99
org.apache.drill.exec.ops.QueryContext.getRootSchema():164
org.apache.drill.exec.ops.QueryContext.getRootSchema():153
org.apache.drill.exec.ops.QueryContext.getNewDefaultSchema():139
org.apache.drill.exec.planner.sql.SqlConverter.<init>():111
org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan():101
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan():79
org.apache.drill.exec.work.foreman.Foreman.runSQL():1050
org.apache.drill.exec.work.foreman.Foreman.run():280
java.util.concurrent.ThreadPoolExecutor.runWorker():1145
java.util.concurrent.ThreadPoolExecutor$Worker.run():615
java.lang.Thread.run():748
Caused By (org.apache.http.conn.ConnectionPoolTimeoutException) Timeout waiting for connection from pool
org.apache.http.impl.conn.PoolingClientConnectionManager.leaseConnection():232
org.apache.http.impl.conn.PoolingClientConnectionManager$1.getConnection():199
sun.reflect.GeneratedMethodAccessor9.invoke():-1
sun.reflect.DelegatingMethodAccessorImpl.invoke():43
java.lang.reflect.Method.invoke():606
com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke():70
com.amazonaws.http.conn.$Proxy58.getConnection():-1
org.apache.http.impl.client.DefaultRequestDirector.execute():456
org.apache.http.impl.client.AbstractHttpClient.execute():906
org.apache.http.impl.client.AbstractHttpClient.execute():805
com.amazonaws.http.AmazonHttpClient.executeHelper():384
com.amazonaws.http.AmazonHttpClient.execute():232
com.amazonaws.services.s3.AmazonS3Client.invoke():3528
com.amazonaws.services.s3.AmazonS3Client.invoke():3480
com.amazonaws.services.s3.AmazonS3Client.listObjects():604
org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus():962
org.apache.hadoop.fs.s3a.S3AFileSystem.listStatus():734
org.apache.drill.exec.store.dfs.DrillFileSystem.listStatus():522
org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory.accessible():160
org.apache.drill.exec.store.dfs.FileSystemSchemaFactory$FileSystemSchema.<init>():77
org.apache.drill.exec.store.dfs.FileSystemSchemaFactory.registerSchemas():64
org.apache.drill.exec.store.dfs.FileSystemPlugin.registerSchemas():149
org.apache.drill.exec.store.StoragePluginRegistryImpl$DrillSchemaFactory.registerSchemas():396
org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema():110
org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema():99
org.apache.drill.exec.ops.QueryContext.getRootSchema():164
org.apache.drill.exec.ops.QueryContext.getRootSchema():153
org.apache.drill.exec.ops.QueryContext.getNewDefaultSchema():139
org.apache.drill.exec.planner.sql.SqlConverter.<init>():111
org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan():101
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan():79
org.apache.drill.exec.work.foreman.Foreman.runSQL():1050
org.apache.drill.exec.work.foreman.Foreman.run():280
java.util.concurrent.ThreadPoolExecutor.runWorker():1145
java.util.concurrent.ThreadPoolExecutor$Worker.run():615
java.lang.Thread.run():748 (state=,code=0)
0: jdbc:drill:zk=local>
I add the following to the core-site.xml but it does not help.
<property>
<name>fs.s3a.connection.maximum</name>
<value>100</value>
</property>
Re: Timeout error querying S3 Apache Drill 11
Posted by Padma Penumarthy <pp...@mapr.com>.
Can you try playing around with these properties and see if that helps ?
<property>
<name>fs.s3a.connection.establish.timeout</name>
<value>5000</value>
<description>Socket connection setup timeout in milliseconds.</description>
</property>
<property>
<name>fs.s3a.connection.timeout</name>
<value>50000</value>
<description>Socket connection timeout in milliseconds.</description>
</property>
________________________________
From: Bob Carter <bc...@battelleecology.org>
Sent: Thursday, August 3, 2017 12:35 PM
To: user@drill.apache.org
Subject: Timeout error querying S3 Apache Drill 11
Error: SYSTEM ERROR: ConnectionPoolTimeoutException: Timeout waiting for connection from pool
[Error Id: c5d6ba1f-2868-48b4-8db0-63e027f15bd2 on db-migrate:31010]
(org.apache.drill.exec.work.foreman.ForemanException) Unexpected exception during fragment initialization: Unable to execute HTTP request: Timeout waiting for connection from pool
org.apache.drill.exec.work.foreman.Foreman.run():297
java.util.concurrent.ThreadPoolExecutor.runWorker():1145
java.util.concurrent.ThreadPoolExecutor$Worker.run():615
java.lang.Thread.run():748
Caused By (com.amazonaws.AmazonClientException) Unable to execute HTTP request: Timeout waiting for connection from pool
com.amazonaws.http.AmazonHttpClient.executeHelper():454
com.amazonaws.http.AmazonHttpClient.execute():232
com.amazonaws.services.s3.AmazonS3Client.invoke():3528
com.amazonaws.services.s3.AmazonS3Client.invoke():3480
com.amazonaws.services.s3.AmazonS3Client.listObjects():604
org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus():962
org.apache.hadoop.fs.s3a.S3AFileSystem.listStatus():734
org.apache.drill.exec.store.dfs.DrillFileSystem.listStatus():522
org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory.accessible():160
org.apache.drill.exec.store.dfs.FileSystemSchemaFactory$FileSystemSchema.<init>():77
org.apache.drill.exec.store.dfs.FileSystemSchemaFactory.registerSchemas():64
org.apache.drill.exec.store.dfs.FileSystemPlugin.registerSchemas():149
org.apache.drill.exec.store.StoragePluginRegistryImpl$DrillSchemaFactory.registerSchemas():396
org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema():110
org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema():99
org.apache.drill.exec.ops.QueryContext.getRootSchema():164
org.apache.drill.exec.ops.QueryContext.getRootSchema():153
org.apache.drill.exec.ops.QueryContext.getNewDefaultSchema():139
org.apache.drill.exec.planner.sql.SqlConverter.<init>():111
org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan():101
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan():79
org.apache.drill.exec.work.foreman.Foreman.runSQL():1050
org.apache.drill.exec.work.foreman.Foreman.run():280
java.util.concurrent.ThreadPoolExecutor.runWorker():1145
java.util.concurrent.ThreadPoolExecutor$Worker.run():615
java.lang.Thread.run():748
Caused By (org.apache.http.conn.ConnectionPoolTimeoutException) Timeout waiting for connection from pool
org.apache.http.impl.conn.PoolingClientConnectionManager.leaseConnection():232
org.apache.http.impl.conn.PoolingClientConnectionManager$1.getConnection():199
sun.reflect.GeneratedMethodAccessor9.invoke():-1
sun.reflect.DelegatingMethodAccessorImpl.invoke():43
java.lang.reflect.Method.invoke():606
com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke():70
com.amazonaws.http.conn.$Proxy58.getConnection():-1
org.apache.http.impl.client.DefaultRequestDirector.execute():456
org.apache.http.impl.client.AbstractHttpClient.execute():906
org.apache.http.impl.client.AbstractHttpClient.execute():805
com.amazonaws.http.AmazonHttpClient.executeHelper():384
com.amazonaws.http.AmazonHttpClient.execute():232
com.amazonaws.services.s3.AmazonS3Client.invoke():3528
com.amazonaws.services.s3.AmazonS3Client.invoke():3480
com.amazonaws.services.s3.AmazonS3Client.listObjects():604
org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus():962
org.apache.hadoop.fs.s3a.S3AFileSystem.listStatus():734
org.apache.drill.exec.store.dfs.DrillFileSystem.listStatus():522
org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory.accessible():160
org.apache.drill.exec.store.dfs.FileSystemSchemaFactory$FileSystemSchema.<init>():77
org.apache.drill.exec.store.dfs.FileSystemSchemaFactory.registerSchemas():64
org.apache.drill.exec.store.dfs.FileSystemPlugin.registerSchemas():149
org.apache.drill.exec.store.StoragePluginRegistryImpl$DrillSchemaFactory.registerSchemas():396
org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema():110
org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema():99
org.apache.drill.exec.ops.QueryContext.getRootSchema():164
org.apache.drill.exec.ops.QueryContext.getRootSchema():153
org.apache.drill.exec.ops.QueryContext.getNewDefaultSchema():139
org.apache.drill.exec.planner.sql.SqlConverter.<init>():111
org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan():101
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan():79
org.apache.drill.exec.work.foreman.Foreman.runSQL():1050
org.apache.drill.exec.work.foreman.Foreman.run():280
java.util.concurrent.ThreadPoolExecutor.runWorker():1145
java.util.concurrent.ThreadPoolExecutor$Worker.run():615
java.lang.Thread.run():748 (state=,code=0)
0: jdbc:drill:zk=local>
I add the following to the core-site.xml but it does not help.
<property>
<name>fs.s3a.connection.maximum</name>
<value>100</value>
</property>