You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by GitBox <gi...@apache.org> on 2020/02/21 23:45:56 UTC
[GitHub] [hbase] markrmiller opened a new pull request #1194: HBASE-23882:
Experimenting with low config settings.
markrmiller opened a new pull request #1194: HBASE-23882: Experimenting with low config settings.
URL: https://github.com/apache/hbase/pull/1194
Here is some initial experimentation with bringing mini cluster settings down to scale.
To start, I've been shooting for pretty minimal.
I've pulled this out of an experimental branch and so some work may still be in order to find any tests these settings are too low for.
I will continue to update this as I fine tune. I also have some other changes I'd like to dig out around thread pool sizing if I can find them again. I'll likely update again early next week.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #1194:
HBASE-23882: Experimenting with low config settings.
Posted by GitBox <gi...@apache.org>.
saintstack commented on a change in pull request #1194: HBASE-23882: Experimenting with low config settings.
URL: https://github.com/apache/hbase/pull/1194#discussion_r383674467
##########
File path: hbase-common/src/main/java/org/apache/hadoop/hbase/HBaseConfiguration.java
##########
@@ -85,16 +85,69 @@ public static Configuration addHbaseResources(Configuration conf) {
return conf;
}
+ public static Configuration create() {
+ return create(true);
+ }
+
/**
* Creates a Configuration with HBase resources
* @return a Configuration with HBase resources
*/
- public static Configuration create() {
+ public static Configuration create(boolean lowLimits) {
Configuration conf = new Configuration();
// In case HBaseConfiguration is loaded from a different classloader than
// Configuration, conf needs to be set with appropriate class loader to resolve
// HBase resources.
conf.setClassLoader(HBaseConfiguration.class.getClassLoader());
+
+
+ if (lowLimits) {
+
+ // can't set this for every test currently
+ // conf.set("hbase.regionserver.hostname", "127.0.0.1");
+
+ conf.setInt("hbase.hfilearchiver.thread.pool.max", 2);
+ conf.setInt("hbase.loadincremental.threads.max", 3);
+
+
+ conf.setInt("hbase.client.sync.wait.timeout.msec", 30000);
Review comment:
Default is 10min which is crazy for a test, yes. Test general timeout is 13mins IIRC.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #1194:
HBASE-23882: Experimenting with low config settings.
Posted by GitBox <gi...@apache.org>.
saintstack commented on a change in pull request #1194: HBASE-23882: Experimenting with low config settings.
URL: https://github.com/apache/hbase/pull/1194#discussion_r383675270
##########
File path: hbase-common/src/main/java/org/apache/hadoop/hbase/HBaseConfiguration.java
##########
@@ -85,16 +85,69 @@ public static Configuration addHbaseResources(Configuration conf) {
return conf;
}
+ public static Configuration create() {
+ return create(true);
+ }
+
/**
* Creates a Configuration with HBase resources
* @return a Configuration with HBase resources
*/
- public static Configuration create() {
+ public static Configuration create(boolean lowLimits) {
Configuration conf = new Configuration();
// In case HBaseConfiguration is loaded from a different classloader than
// Configuration, conf needs to be set with appropriate class loader to resolve
// HBase resources.
conf.setClassLoader(HBaseConfiguration.class.getClassLoader());
+
+
+ if (lowLimits) {
+
+ // can't set this for every test currently
+ // conf.set("hbase.regionserver.hostname", "127.0.0.1");
+
+ conf.setInt("hbase.hfilearchiver.thread.pool.max", 2);
+ conf.setInt("hbase.loadincremental.threads.max", 3);
+
+
+ conf.setInt("hbase.client.sync.wait.timeout.msec", 30000);
+ conf.setInt("zookeeper.recovery.retry", 5);
+ conf.setInt("hbase.client.retries.number", 10);
+ conf.setInt("hbase.hstore.open.and.close.threads.max", 1);
Review comment:
This is the default? I see it as '1'.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #1194:
HBASE-23882: Experimenting with low config settings.
Posted by GitBox <gi...@apache.org>.
saintstack commented on a change in pull request #1194: HBASE-23882: Experimenting with low config settings.
URL: https://github.com/apache/hbase/pull/1194#discussion_r383677199
##########
File path: hbase-common/src/main/java/org/apache/hadoop/hbase/HBaseConfiguration.java
##########
@@ -85,16 +85,69 @@ public static Configuration addHbaseResources(Configuration conf) {
return conf;
}
+ public static Configuration create() {
+ return create(true);
+ }
+
/**
* Creates a Configuration with HBase resources
* @return a Configuration with HBase resources
*/
- public static Configuration create() {
+ public static Configuration create(boolean lowLimits) {
Configuration conf = new Configuration();
// In case HBaseConfiguration is loaded from a different classloader than
// Configuration, conf needs to be set with appropriate class loader to resolve
// HBase resources.
conf.setClassLoader(HBaseConfiguration.class.getClassLoader());
+
+
+ if (lowLimits) {
+
+ // can't set this for every test currently
+ // conf.set("hbase.regionserver.hostname", "127.0.0.1");
+
+ conf.setInt("hbase.hfilearchiver.thread.pool.max", 2);
+ conf.setInt("hbase.loadincremental.threads.max", 3);
+
+
+ conf.setInt("hbase.client.sync.wait.timeout.msec", 30000);
+ conf.setInt("zookeeper.recovery.retry", 5);
+ conf.setInt("hbase.client.retries.number", 10);
+ conf.setInt("hbase.hstore.open.and.close.threads.max", 1);
+ conf.setInt(HConstants.REGION_SERVER_REPLICATION_HANDLER_COUNT, 2);
+
+ conf.setInt(HConstants.HBASE_CLIENT_MAX_TOTAL_TASKS, 3);
+
+ conf.setInt(HConstants.REGION_SERVER_HIGH_PRIORITY_HANDLER_COUNT, 2);
+
+
+ conf.setInt(HConstants.REPLICATION_SOURCE_MAXTHREADS_KEY, 1);
+
+ conf.setInt("hbase.netty.worker.count", 1);
+
+ conf.setInt(HConstants.MASTER_OPEN_REGION_THREADS, 3);
+
+ // conf.setInt(HConstants.MASTER_CLOSE_REGION_THREADS, 3);
+ // conf.setInt(HConstants.MASTER_SERVER_OPERATIONS_THREADS, 2);
+ // conf.setInt(HConstants.MASTER_META_SERVER_OPERATIONS_THREADS, 2);
+ // conf.setInt(HConstants.MASTER_LOG_REPLAY_OPS_THREADS, 2);
+ // conf.setInt(HConstants.ZOOKEEPER_MAX_CLIENT_CNXNS, 5);
+
+ conf.setInt("hbase.hconnection.threads.max", 30);
+ conf.setInt("hbase.hconnection.threads.keepalivetime", 30);
+ conf.setInt(HConstants.HBASE_CLIENT_MAX_TOTAL_TASKS, 10);
+
+
+
+ conf.setInt(HConstants.REGION_SERVER_HANDLER_COUNT, 2);
Review comment:
Default is 30. 2 is probably too few? Maybe not.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #1194:
HBASE-23882: Experimenting with low config settings.
Posted by GitBox <gi...@apache.org>.
saintstack commented on a change in pull request #1194: HBASE-23882: Experimenting with low config settings.
URL: https://github.com/apache/hbase/pull/1194#discussion_r383675957
##########
File path: hbase-common/src/main/java/org/apache/hadoop/hbase/HBaseConfiguration.java
##########
@@ -85,16 +85,69 @@ public static Configuration addHbaseResources(Configuration conf) {
return conf;
}
+ public static Configuration create() {
+ return create(true);
+ }
+
/**
* Creates a Configuration with HBase resources
* @return a Configuration with HBase resources
*/
- public static Configuration create() {
+ public static Configuration create(boolean lowLimits) {
Configuration conf = new Configuration();
// In case HBaseConfiguration is loaded from a different classloader than
// Configuration, conf needs to be set with appropriate class loader to resolve
// HBase resources.
conf.setClassLoader(HBaseConfiguration.class.getClassLoader());
+
+
+ if (lowLimits) {
+
+ // can't set this for every test currently
+ // conf.set("hbase.regionserver.hostname", "127.0.0.1");
+
+ conf.setInt("hbase.hfilearchiver.thread.pool.max", 2);
+ conf.setInt("hbase.loadincremental.threads.max", 3);
+
+
+ conf.setInt("hbase.client.sync.wait.timeout.msec", 30000);
+ conf.setInt("zookeeper.recovery.retry", 5);
+ conf.setInt("hbase.client.retries.number", 10);
+ conf.setInt("hbase.hstore.open.and.close.threads.max", 1);
+ conf.setInt(HConstants.REGION_SERVER_REPLICATION_HANDLER_COUNT, 2);
+
+ conf.setInt(HConstants.HBASE_CLIENT_MAX_TOTAL_TASKS, 3);
+
+ conf.setInt(HConstants.REGION_SERVER_HIGH_PRIORITY_HANDLER_COUNT, 2);
+
+
+ conf.setInt(HConstants.REPLICATION_SOURCE_MAXTHREADS_KEY, 1);
+
+ conf.setInt("hbase.netty.worker.count", 1);
Review comment:
Default is 0, am I reading that right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #1194:
HBASE-23882: Experimenting with low config settings.
Posted by GitBox <gi...@apache.org>.
saintstack commented on a change in pull request #1194: HBASE-23882: Experimenting with low config settings.
URL: https://github.com/apache/hbase/pull/1194#discussion_r383675338
##########
File path: hbase-common/src/main/java/org/apache/hadoop/hbase/HBaseConfiguration.java
##########
@@ -85,16 +85,69 @@ public static Configuration addHbaseResources(Configuration conf) {
return conf;
}
+ public static Configuration create() {
+ return create(true);
+ }
+
/**
* Creates a Configuration with HBase resources
* @return a Configuration with HBase resources
*/
- public static Configuration create() {
+ public static Configuration create(boolean lowLimits) {
Configuration conf = new Configuration();
// In case HBaseConfiguration is loaded from a different classloader than
// Configuration, conf needs to be set with appropriate class loader to resolve
// HBase resources.
conf.setClassLoader(HBaseConfiguration.class.getClassLoader());
+
+
+ if (lowLimits) {
+
+ // can't set this for every test currently
+ // conf.set("hbase.regionserver.hostname", "127.0.0.1");
+
+ conf.setInt("hbase.hfilearchiver.thread.pool.max", 2);
+ conf.setInt("hbase.loadincremental.threads.max", 3);
+
+
+ conf.setInt("hbase.client.sync.wait.timeout.msec", 30000);
+ conf.setInt("zookeeper.recovery.retry", 5);
+ conf.setInt("hbase.client.retries.number", 10);
+ conf.setInt("hbase.hstore.open.and.close.threads.max", 1);
+ conf.setInt(HConstants.REGION_SERVER_REPLICATION_HANDLER_COUNT, 2);
Review comment:
Dfault is 3
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #1194:
HBASE-23882: Experimenting with low config settings.
Posted by GitBox <gi...@apache.org>.
saintstack commented on a change in pull request #1194: HBASE-23882: Experimenting with low config settings.
URL: https://github.com/apache/hbase/pull/1194#discussion_r383673844
##########
File path: hbase-common/src/main/java/org/apache/hadoop/hbase/HBaseConfiguration.java
##########
@@ -85,16 +85,69 @@ public static Configuration addHbaseResources(Configuration conf) {
return conf;
}
+ public static Configuration create() {
+ return create(true);
+ }
+
/**
* Creates a Configuration with HBase resources
* @return a Configuration with HBase resources
*/
- public static Configuration create() {
+ public static Configuration create(boolean lowLimits) {
Configuration conf = new Configuration();
// In case HBaseConfiguration is loaded from a different classloader than
// Configuration, conf needs to be set with appropriate class loader to resolve
// HBase resources.
conf.setClassLoader(HBaseConfiguration.class.getClassLoader());
+
+
+ if (lowLimits) {
+
+ // can't set this for every test currently
+ // conf.set("hbase.regionserver.hostname", "127.0.0.1");
+
+ conf.setInt("hbase.hfilearchiver.thread.pool.max", 2);
Review comment:
Default is 8.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #1194:
HBASE-23882: Experimenting with low config settings.
Posted by GitBox <gi...@apache.org>.
saintstack commented on a change in pull request #1194: HBASE-23882: Experimenting with low config settings.
URL: https://github.com/apache/hbase/pull/1194#discussion_r383674784
##########
File path: hbase-common/src/main/java/org/apache/hadoop/hbase/HBaseConfiguration.java
##########
@@ -85,16 +85,69 @@ public static Configuration addHbaseResources(Configuration conf) {
return conf;
}
+ public static Configuration create() {
+ return create(true);
+ }
+
/**
* Creates a Configuration with HBase resources
* @return a Configuration with HBase resources
*/
- public static Configuration create() {
+ public static Configuration create(boolean lowLimits) {
Configuration conf = new Configuration();
// In case HBaseConfiguration is loaded from a different classloader than
// Configuration, conf needs to be set with appropriate class loader to resolve
// HBase resources.
conf.setClassLoader(HBaseConfiguration.class.getClassLoader());
+
+
+ if (lowLimits) {
+
+ // can't set this for every test currently
+ // conf.set("hbase.regionserver.hostname", "127.0.0.1");
+
+ conf.setInt("hbase.hfilearchiver.thread.pool.max", 2);
+ conf.setInt("hbase.loadincremental.threads.max", 3);
+
+
+ conf.setInt("hbase.client.sync.wait.timeout.msec", 30000);
+ conf.setInt("zookeeper.recovery.retry", 5);
Review comment:
Default is 30. Too much for a test. This looks good.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #1194:
HBASE-23882: Experimenting with low config settings.
Posted by GitBox <gi...@apache.org>.
saintstack commented on a change in pull request #1194: HBASE-23882: Experimenting with low config settings.
URL: https://github.com/apache/hbase/pull/1194#discussion_r383672556
##########
File path: hbase-common/src/main/java/org/apache/hadoop/hbase/HBaseConfiguration.java
##########
@@ -85,16 +85,69 @@ public static Configuration addHbaseResources(Configuration conf) {
return conf;
}
+ public static Configuration create() {
+ return create(true);
+ }
+
/**
* Creates a Configuration with HBase resources
* @return a Configuration with HBase resources
*/
- public static Configuration create() {
+ public static Configuration create(boolean lowLimits) {
Configuration conf = new Configuration();
// In case HBaseConfiguration is loaded from a different classloader than
// Configuration, conf needs to be set with appropriate class loader to resolve
// HBase resources.
conf.setClassLoader(HBaseConfiguration.class.getClassLoader());
+
+
+ if (lowLimits) {
+
+ // can't set this for every test currently
+ // conf.set("hbase.regionserver.hostname", "127.0.0.1");
+
+ conf.setInt("hbase.hfilearchiver.thread.pool.max", 2);
+ conf.setInt("hbase.loadincremental.threads.max", 3);
+
+
+ conf.setInt("hbase.client.sync.wait.timeout.msec", 30000);
+ conf.setInt("zookeeper.recovery.retry", 5);
+ conf.setInt("hbase.client.retries.number", 10);
+ conf.setInt("hbase.hstore.open.and.close.threads.max", 1);
+ conf.setInt(HConstants.REGION_SERVER_REPLICATION_HANDLER_COUNT, 2);
+
+ conf.setInt(HConstants.HBASE_CLIENT_MAX_TOTAL_TASKS, 3);
+
+ conf.setInt(HConstants.REGION_SERVER_HIGH_PRIORITY_HANDLER_COUNT, 2);
+
+
+ conf.setInt(HConstants.REPLICATION_SOURCE_MAXTHREADS_KEY, 1);
+
+ conf.setInt("hbase.netty.worker.count", 1);
+
+ conf.setInt(HConstants.MASTER_OPEN_REGION_THREADS, 3);
+
+ // conf.setInt(HConstants.MASTER_CLOSE_REGION_THREADS, 3);
+ // conf.setInt(HConstants.MASTER_SERVER_OPERATIONS_THREADS, 2);
+ // conf.setInt(HConstants.MASTER_META_SERVER_OPERATIONS_THREADS, 2);
+ // conf.setInt(HConstants.MASTER_LOG_REPLAY_OPS_THREADS, 2);
+ // conf.setInt(HConstants.ZOOKEEPER_MAX_CLIENT_CNXNS, 5);
+
+ conf.setInt("hbase.hconnection.threads.max", 30);
+ conf.setInt("hbase.hconnection.threads.keepalivetime", 30);
+ conf.setInt(HConstants.HBASE_CLIENT_MAX_TOTAL_TASKS, 10);
+
+
+
+ conf.setInt(HConstants.REGION_SERVER_HANDLER_COUNT, 2);
+ conf.setInt("hbase.master.procedure.threads", 1);
+
+ conf.setInt("dfs.namenode.handler.count", 5);
+ // conf.setInt("dfs.datanode.handler.count", 5);
+ // conf.setInt("dfs.datanode.max.transfer.threads", 5);
+ // conf.setInt("dfs.client.file-block-storage-locations.num-threads", 5);
+ }
+
Review comment:
FYI, the way we do this usually for tests is to put these settings into hbase-*/src/test/resources/hbase-site.xml
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #1194:
HBASE-23882: Experimenting with low config settings.
Posted by GitBox <gi...@apache.org>.
saintstack commented on a change in pull request #1194: HBASE-23882: Experimenting with low config settings.
URL: https://github.com/apache/hbase/pull/1194#discussion_r383677773
##########
File path: hbase-common/src/main/java/org/apache/hadoop/hbase/HBaseConfiguration.java
##########
@@ -85,16 +85,69 @@ public static Configuration addHbaseResources(Configuration conf) {
return conf;
}
+ public static Configuration create() {
+ return create(true);
+ }
+
/**
* Creates a Configuration with HBase resources
* @return a Configuration with HBase resources
*/
- public static Configuration create() {
+ public static Configuration create(boolean lowLimits) {
Configuration conf = new Configuration();
// In case HBaseConfiguration is loaded from a different classloader than
// Configuration, conf needs to be set with appropriate class loader to resolve
// HBase resources.
conf.setClassLoader(HBaseConfiguration.class.getClassLoader());
+
+
+ if (lowLimits) {
+
+ // can't set this for every test currently
+ // conf.set("hbase.regionserver.hostname", "127.0.0.1");
+
+ conf.setInt("hbase.hfilearchiver.thread.pool.max", 2);
+ conf.setInt("hbase.loadincremental.threads.max", 3);
+
+
+ conf.setInt("hbase.client.sync.wait.timeout.msec", 30000);
+ conf.setInt("zookeeper.recovery.retry", 5);
+ conf.setInt("hbase.client.retries.number", 10);
+ conf.setInt("hbase.hstore.open.and.close.threads.max", 1);
+ conf.setInt(HConstants.REGION_SERVER_REPLICATION_HANDLER_COUNT, 2);
+
+ conf.setInt(HConstants.HBASE_CLIENT_MAX_TOTAL_TASKS, 3);
+
+ conf.setInt(HConstants.REGION_SERVER_HIGH_PRIORITY_HANDLER_COUNT, 2);
+
+
+ conf.setInt(HConstants.REPLICATION_SOURCE_MAXTHREADS_KEY, 1);
+
+ conf.setInt("hbase.netty.worker.count", 1);
+
+ conf.setInt(HConstants.MASTER_OPEN_REGION_THREADS, 3);
+
+ // conf.setInt(HConstants.MASTER_CLOSE_REGION_THREADS, 3);
+ // conf.setInt(HConstants.MASTER_SERVER_OPERATIONS_THREADS, 2);
+ // conf.setInt(HConstants.MASTER_META_SERVER_OPERATIONS_THREADS, 2);
+ // conf.setInt(HConstants.MASTER_LOG_REPLAY_OPS_THREADS, 2);
+ // conf.setInt(HConstants.ZOOKEEPER_MAX_CLIENT_CNXNS, 5);
+
+ conf.setInt("hbase.hconnection.threads.max", 30);
+ conf.setInt("hbase.hconnection.threads.keepalivetime", 30);
+ conf.setInt(HConstants.HBASE_CLIENT_MAX_TOTAL_TASKS, 10);
+
+
+
+ conf.setInt(HConstants.REGION_SERVER_HANDLER_COUNT, 2);
+ conf.setInt("hbase.master.procedure.threads", 1);
+
+ conf.setInt("dfs.namenode.handler.count", 5);
Review comment:
Default is 10
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #1194:
HBASE-23882: Experimenting with low config settings.
Posted by GitBox <gi...@apache.org>.
saintstack commented on a change in pull request #1194: HBASE-23882: Experimenting with low config settings.
URL: https://github.com/apache/hbase/pull/1194#discussion_r383677004
##########
File path: hbase-common/src/main/java/org/apache/hadoop/hbase/HBaseConfiguration.java
##########
@@ -85,16 +85,69 @@ public static Configuration addHbaseResources(Configuration conf) {
return conf;
}
+ public static Configuration create() {
+ return create(true);
+ }
+
/**
* Creates a Configuration with HBase resources
* @return a Configuration with HBase resources
*/
- public static Configuration create() {
+ public static Configuration create(boolean lowLimits) {
Configuration conf = new Configuration();
// In case HBaseConfiguration is loaded from a different classloader than
// Configuration, conf needs to be set with appropriate class loader to resolve
// HBase resources.
conf.setClassLoader(HBaseConfiguration.class.getClassLoader());
+
+
+ if (lowLimits) {
+
+ // can't set this for every test currently
+ // conf.set("hbase.regionserver.hostname", "127.0.0.1");
+
+ conf.setInt("hbase.hfilearchiver.thread.pool.max", 2);
+ conf.setInt("hbase.loadincremental.threads.max", 3);
+
+
+ conf.setInt("hbase.client.sync.wait.timeout.msec", 30000);
+ conf.setInt("zookeeper.recovery.retry", 5);
+ conf.setInt("hbase.client.retries.number", 10);
+ conf.setInt("hbase.hstore.open.and.close.threads.max", 1);
+ conf.setInt(HConstants.REGION_SERVER_REPLICATION_HANDLER_COUNT, 2);
+
+ conf.setInt(HConstants.HBASE_CLIENT_MAX_TOTAL_TASKS, 3);
+
+ conf.setInt(HConstants.REGION_SERVER_HIGH_PRIORITY_HANDLER_COUNT, 2);
+
+
+ conf.setInt(HConstants.REPLICATION_SOURCE_MAXTHREADS_KEY, 1);
+
+ conf.setInt("hbase.netty.worker.count", 1);
+
+ conf.setInt(HConstants.MASTER_OPEN_REGION_THREADS, 3);
+
+ // conf.setInt(HConstants.MASTER_CLOSE_REGION_THREADS, 3);
+ // conf.setInt(HConstants.MASTER_SERVER_OPERATIONS_THREADS, 2);
+ // conf.setInt(HConstants.MASTER_META_SERVER_OPERATIONS_THREADS, 2);
+ // conf.setInt(HConstants.MASTER_LOG_REPLAY_OPS_THREADS, 2);
+ // conf.setInt(HConstants.ZOOKEEPER_MAX_CLIENT_CNXNS, 5);
+
+ conf.setInt("hbase.hconnection.threads.max", 30);
+ conf.setInt("hbase.hconnection.threads.keepalivetime", 30);
+ conf.setInt(HConstants.HBASE_CLIENT_MAX_TOTAL_TASKS, 10);
Review comment:
This a repeat? Above you have it set to 3 if I'm reading this right.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #1194:
HBASE-23882: Experimenting with low config settings.
Posted by GitBox <gi...@apache.org>.
saintstack commented on a change in pull request #1194: HBASE-23882: Experimenting with low config settings.
URL: https://github.com/apache/hbase/pull/1194#discussion_r383675495
##########
File path: hbase-common/src/main/java/org/apache/hadoop/hbase/HBaseConfiguration.java
##########
@@ -85,16 +85,69 @@ public static Configuration addHbaseResources(Configuration conf) {
return conf;
}
+ public static Configuration create() {
+ return create(true);
+ }
+
/**
* Creates a Configuration with HBase resources
* @return a Configuration with HBase resources
*/
- public static Configuration create() {
+ public static Configuration create(boolean lowLimits) {
Configuration conf = new Configuration();
// In case HBaseConfiguration is loaded from a different classloader than
// Configuration, conf needs to be set with appropriate class loader to resolve
// HBase resources.
conf.setClassLoader(HBaseConfiguration.class.getClassLoader());
+
+
+ if (lowLimits) {
+
+ // can't set this for every test currently
+ // conf.set("hbase.regionserver.hostname", "127.0.0.1");
+
+ conf.setInt("hbase.hfilearchiver.thread.pool.max", 2);
+ conf.setInt("hbase.loadincremental.threads.max", 3);
+
+
+ conf.setInt("hbase.client.sync.wait.timeout.msec", 30000);
+ conf.setInt("zookeeper.recovery.retry", 5);
+ conf.setInt("hbase.client.retries.number", 10);
+ conf.setInt("hbase.hstore.open.and.close.threads.max", 1);
+ conf.setInt(HConstants.REGION_SERVER_REPLICATION_HANDLER_COUNT, 2);
+
+ conf.setInt(HConstants.HBASE_CLIENT_MAX_TOTAL_TASKS, 3);
Review comment:
Default is 100
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #1194:
HBASE-23882: Experimenting with low config settings.
Posted by GitBox <gi...@apache.org>.
saintstack commented on a change in pull request #1194: HBASE-23882: Experimenting with low config settings.
URL: https://github.com/apache/hbase/pull/1194#discussion_r383675000
##########
File path: hbase-common/src/main/java/org/apache/hadoop/hbase/HBaseConfiguration.java
##########
@@ -85,16 +85,69 @@ public static Configuration addHbaseResources(Configuration conf) {
return conf;
}
+ public static Configuration create() {
+ return create(true);
+ }
+
/**
* Creates a Configuration with HBase resources
* @return a Configuration with HBase resources
*/
- public static Configuration create() {
+ public static Configuration create(boolean lowLimits) {
Configuration conf = new Configuration();
// In case HBaseConfiguration is loaded from a different classloader than
// Configuration, conf needs to be set with appropriate class loader to resolve
// HBase resources.
conf.setClassLoader(HBaseConfiguration.class.getClassLoader());
+
+
+ if (lowLimits) {
+
+ // can't set this for every test currently
+ // conf.set("hbase.regionserver.hostname", "127.0.0.1");
+
+ conf.setInt("hbase.hfilearchiver.thread.pool.max", 2);
+ conf.setInt("hbase.loadincremental.threads.max", 3);
+
+
+ conf.setInt("hbase.client.sync.wait.timeout.msec", 30000);
+ conf.setInt("zookeeper.recovery.retry", 5);
+ conf.setInt("hbase.client.retries.number", 10);
Review comment:
Default is 15.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #1194:
HBASE-23882: Experimenting with low config settings.
Posted by GitBox <gi...@apache.org>.
saintstack commented on a change in pull request #1194: HBASE-23882: Experimenting with low config settings.
URL: https://github.com/apache/hbase/pull/1194#discussion_r383676047
##########
File path: hbase-common/src/main/java/org/apache/hadoop/hbase/HBaseConfiguration.java
##########
@@ -85,16 +85,69 @@ public static Configuration addHbaseResources(Configuration conf) {
return conf;
}
+ public static Configuration create() {
+ return create(true);
+ }
+
/**
* Creates a Configuration with HBase resources
* @return a Configuration with HBase resources
*/
- public static Configuration create() {
+ public static Configuration create(boolean lowLimits) {
Configuration conf = new Configuration();
// In case HBaseConfiguration is loaded from a different classloader than
// Configuration, conf needs to be set with appropriate class loader to resolve
// HBase resources.
conf.setClassLoader(HBaseConfiguration.class.getClassLoader());
+
+
+ if (lowLimits) {
+
+ // can't set this for every test currently
+ // conf.set("hbase.regionserver.hostname", "127.0.0.1");
+
+ conf.setInt("hbase.hfilearchiver.thread.pool.max", 2);
+ conf.setInt("hbase.loadincremental.threads.max", 3);
+
+
+ conf.setInt("hbase.client.sync.wait.timeout.msec", 30000);
+ conf.setInt("zookeeper.recovery.retry", 5);
+ conf.setInt("hbase.client.retries.number", 10);
+ conf.setInt("hbase.hstore.open.and.close.threads.max", 1);
+ conf.setInt(HConstants.REGION_SERVER_REPLICATION_HANDLER_COUNT, 2);
+
+ conf.setInt(HConstants.HBASE_CLIENT_MAX_TOTAL_TASKS, 3);
+
+ conf.setInt(HConstants.REGION_SERVER_HIGH_PRIORITY_HANDLER_COUNT, 2);
+
+
+ conf.setInt(HConstants.REPLICATION_SOURCE_MAXTHREADS_KEY, 1);
+
+ conf.setInt("hbase.netty.worker.count", 1);
+
+ conf.setInt(HConstants.MASTER_OPEN_REGION_THREADS, 3);
Review comment:
Default is 5
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #1194:
HBASE-23882: Experimenting with low config settings.
Posted by GitBox <gi...@apache.org>.
saintstack commented on a change in pull request #1194: HBASE-23882: Experimenting with low config settings.
URL: https://github.com/apache/hbase/pull/1194#discussion_r383675669
##########
File path: hbase-common/src/main/java/org/apache/hadoop/hbase/HBaseConfiguration.java
##########
@@ -85,16 +85,69 @@ public static Configuration addHbaseResources(Configuration conf) {
return conf;
}
+ public static Configuration create() {
+ return create(true);
+ }
+
/**
* Creates a Configuration with HBase resources
* @return a Configuration with HBase resources
*/
- public static Configuration create() {
+ public static Configuration create(boolean lowLimits) {
Configuration conf = new Configuration();
// In case HBaseConfiguration is loaded from a different classloader than
// Configuration, conf needs to be set with appropriate class loader to resolve
// HBase resources.
conf.setClassLoader(HBaseConfiguration.class.getClassLoader());
+
+
+ if (lowLimits) {
+
+ // can't set this for every test currently
+ // conf.set("hbase.regionserver.hostname", "127.0.0.1");
+
+ conf.setInt("hbase.hfilearchiver.thread.pool.max", 2);
+ conf.setInt("hbase.loadincremental.threads.max", 3);
+
+
+ conf.setInt("hbase.client.sync.wait.timeout.msec", 30000);
+ conf.setInt("zookeeper.recovery.retry", 5);
+ conf.setInt("hbase.client.retries.number", 10);
+ conf.setInt("hbase.hstore.open.and.close.threads.max", 1);
+ conf.setInt(HConstants.REGION_SERVER_REPLICATION_HANDLER_COUNT, 2);
+
+ conf.setInt(HConstants.HBASE_CLIENT_MAX_TOTAL_TASKS, 3);
+
+ conf.setInt(HConstants.REGION_SERVER_HIGH_PRIORITY_HANDLER_COUNT, 2);
Review comment:
Probably keep these high because in certain circumstance test can lock up because we are starved for priority handlers.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #1194:
HBASE-23882: Experimenting with low config settings.
Posted by GitBox <gi...@apache.org>.
saintstack commented on a change in pull request #1194: HBASE-23882: Experimenting with low config settings.
URL: https://github.com/apache/hbase/pull/1194#discussion_r383676582
##########
File path: hbase-common/src/main/java/org/apache/hadoop/hbase/HBaseConfiguration.java
##########
@@ -85,16 +85,69 @@ public static Configuration addHbaseResources(Configuration conf) {
return conf;
}
+ public static Configuration create() {
+ return create(true);
+ }
+
/**
* Creates a Configuration with HBase resources
* @return a Configuration with HBase resources
*/
- public static Configuration create() {
+ public static Configuration create(boolean lowLimits) {
Configuration conf = new Configuration();
// In case HBaseConfiguration is loaded from a different classloader than
// Configuration, conf needs to be set with appropriate class loader to resolve
// HBase resources.
conf.setClassLoader(HBaseConfiguration.class.getClassLoader());
+
+
+ if (lowLimits) {
+
+ // can't set this for every test currently
+ // conf.set("hbase.regionserver.hostname", "127.0.0.1");
+
+ conf.setInt("hbase.hfilearchiver.thread.pool.max", 2);
+ conf.setInt("hbase.loadincremental.threads.max", 3);
+
+
+ conf.setInt("hbase.client.sync.wait.timeout.msec", 30000);
+ conf.setInt("zookeeper.recovery.retry", 5);
+ conf.setInt("hbase.client.retries.number", 10);
+ conf.setInt("hbase.hstore.open.and.close.threads.max", 1);
+ conf.setInt(HConstants.REGION_SERVER_REPLICATION_HANDLER_COUNT, 2);
+
+ conf.setInt(HConstants.HBASE_CLIENT_MAX_TOTAL_TASKS, 3);
+
+ conf.setInt(HConstants.REGION_SERVER_HIGH_PRIORITY_HANDLER_COUNT, 2);
+
+
+ conf.setInt(HConstants.REPLICATION_SOURCE_MAXTHREADS_KEY, 1);
+
+ conf.setInt("hbase.netty.worker.count", 1);
+
+ conf.setInt(HConstants.MASTER_OPEN_REGION_THREADS, 3);
+
+ // conf.setInt(HConstants.MASTER_CLOSE_REGION_THREADS, 3);
+ // conf.setInt(HConstants.MASTER_SERVER_OPERATIONS_THREADS, 2);
+ // conf.setInt(HConstants.MASTER_META_SERVER_OPERATIONS_THREADS, 2);
+ // conf.setInt(HConstants.MASTER_LOG_REPLAY_OPS_THREADS, 2);
+ // conf.setInt(HConstants.ZOOKEEPER_MAX_CLIENT_CNXNS, 5);
+
+ conf.setInt("hbase.hconnection.threads.max", 30);
Review comment:
Default is 256. I bet a patch w/ just this in it would make a big difference, what you think?
One though I had was running the tests with fork count of 1 watching the system for file count, thread count, i/o, memory.... usage as the test suite ran. I suppose for it to be apples to apples, would need to reuse jvm (but then tests probably wouldn't pass). Might be good for figuring the heavyweights. You probably have something that does this already (smile).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #1194:
HBASE-23882: Experimenting with low config settings.
Posted by GitBox <gi...@apache.org>.
saintstack commented on a change in pull request #1194: HBASE-23882: Experimenting with low config settings.
URL: https://github.com/apache/hbase/pull/1194#discussion_r383675758
##########
File path: hbase-common/src/main/java/org/apache/hadoop/hbase/HBaseConfiguration.java
##########
@@ -85,16 +85,69 @@ public static Configuration addHbaseResources(Configuration conf) {
return conf;
}
+ public static Configuration create() {
+ return create(true);
+ }
+
/**
* Creates a Configuration with HBase resources
* @return a Configuration with HBase resources
*/
- public static Configuration create() {
+ public static Configuration create(boolean lowLimits) {
Configuration conf = new Configuration();
// In case HBaseConfiguration is loaded from a different classloader than
// Configuration, conf needs to be set with appropriate class loader to resolve
// HBase resources.
conf.setClassLoader(HBaseConfiguration.class.getClassLoader());
+
+
+ if (lowLimits) {
+
+ // can't set this for every test currently
+ // conf.set("hbase.regionserver.hostname", "127.0.0.1");
+
+ conf.setInt("hbase.hfilearchiver.thread.pool.max", 2);
+ conf.setInt("hbase.loadincremental.threads.max", 3);
+
+
+ conf.setInt("hbase.client.sync.wait.timeout.msec", 30000);
+ conf.setInt("zookeeper.recovery.retry", 5);
+ conf.setInt("hbase.client.retries.number", 10);
+ conf.setInt("hbase.hstore.open.and.close.threads.max", 1);
+ conf.setInt(HConstants.REGION_SERVER_REPLICATION_HANDLER_COUNT, 2);
+
+ conf.setInt(HConstants.HBASE_CLIENT_MAX_TOTAL_TASKS, 3);
+
+ conf.setInt(HConstants.REGION_SERVER_HIGH_PRIORITY_HANDLER_COUNT, 2);
+
+
+ conf.setInt(HConstants.REPLICATION_SOURCE_MAXTHREADS_KEY, 1);
Review comment:
Default is 10
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1194: HBASE-23882:
Experimenting with low config settings.
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on issue #1194: HBASE-23882: Experimenting with low config settings.
URL: https://github.com/apache/hbase/pull/1194#issuecomment-589907430
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 32s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
||| _ branch-2 Compile Tests _ |
| +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 5m 42s | branch-2 passed |
| +1 :green_heart: | compile | 1m 23s | branch-2 passed |
| +1 :green_heart: | checkstyle | 1m 39s | branch-2 passed |
| +1 :green_heart: | shadedjars | 4m 37s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 57s | branch-2 passed |
| +0 :ok: | spotbugs | 3m 43s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 4m 29s | branch-2 passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 5m 17s | the patch passed |
| +1 :green_heart: | compile | 1m 21s | the patch passed |
| +1 :green_heart: | javac | 1m 21s | the patch passed |
| +1 :green_heart: | checkstyle | 1m 39s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedjars | 4m 37s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | hadoopcheck | 16m 45s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. |
| +1 :green_heart: | javadoc | 0m 54s | the patch passed |
| +1 :green_heart: | findbugs | 4m 53s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 11s | hbase-common in the patch passed. |
| -1 :x: | unit | 87m 50s | hbase-server in the patch failed. |
| +1 :green_heart: | asflicense | 0m 43s | The patch does not generate ASF License warnings. |
| | | 151m 45s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hbase.client.TestClientOperationInterrupt |
| | hadoop.hbase.security.access.TestNamespaceCommands |
| | hadoop.hbase.client.TestReplicasClient |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.4 Server=19.03.4 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1194/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1194 |
| JIRA Issue | HBASE-23882 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux b035aef41854 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1194/out/precommit/personality/provided.sh |
| git revision | branch-2 / cb838bc301 |
| Default Java | 1.8.0_181 |
| unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1194/1/artifact/out/patch-unit-hbase-server.txt |
| Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1194/1/testReport/ |
| Max. process+thread count | 5668 (vs. ulimit of 10000) |
| modules | C: hbase-common hbase-server U: . |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1194/1/console |
| versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack commented on issue #1194: HBASE-23882:
Experimenting with low config settings.
Posted by GitBox <gi...@apache.org>.
saintstack commented on issue #1194: HBASE-23882: Experimenting with low config settings.
URL: https://github.com/apache/hbase/pull/1194#issuecomment-590697838
Rerunning build to see if the failures repeat.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1194: HBASE-23882:
Experimenting with low config settings.
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on issue #1194: HBASE-23882: Experimenting with low config settings.
URL: https://github.com/apache/hbase/pull/1194#issuecomment-590748056
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 28s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
||| _ branch-2 Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 5m 39s | branch-2 passed |
| +1 :green_heart: | compile | 1m 21s | branch-2 passed |
| +1 :green_heart: | checkstyle | 1m 41s | branch-2 passed |
| +1 :green_heart: | shadedjars | 4m 36s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 55s | branch-2 passed |
| +0 :ok: | spotbugs | 3m 39s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 4m 25s | branch-2 passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 5m 21s | the patch passed |
| +1 :green_heart: | compile | 1m 21s | the patch passed |
| +1 :green_heart: | javac | 1m 21s | the patch passed |
| +1 :green_heart: | checkstyle | 1m 39s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedjars | 4m 39s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | hadoopcheck | 16m 53s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. |
| +1 :green_heart: | javadoc | 0m 56s | the patch passed |
| +1 :green_heart: | findbugs | 4m 20s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 10s | hbase-common in the patch passed. |
| -1 :x: | unit | 96m 34s | hbase-server in the patch failed. |
| +1 :green_heart: | asflicense | 0m 51s | The patch does not generate ASF License warnings. |
| | | 160m 1s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hbase.client.TestClientOperationInterrupt |
| | hadoop.hbase.master.TestSplitWALManager |
| | hadoop.hbase.security.access.TestNamespaceCommands |
| | hadoop.hbase.client.TestReplicasClient |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1194/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1194 |
| JIRA Issue | HBASE-23882 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux e4e999ac9c50 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1194/out/precommit/personality/provided.sh |
| git revision | branch-2 / 2c3621690d |
| Default Java | 1.8.0_181 |
| unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1194/2/artifact/out/patch-unit-hbase-server.txt |
| Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1194/2/testReport/ |
| Max. process+thread count | 6086 (vs. ulimit of 10000) |
| modules | C: hbase-common hbase-server U: . |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1194/2/console |
| versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #1194:
HBASE-23882: Experimenting with low config settings.
Posted by GitBox <gi...@apache.org>.
saintstack commented on a change in pull request #1194: HBASE-23882: Experimenting with low config settings.
URL: https://github.com/apache/hbase/pull/1194#discussion_r383674225
##########
File path: hbase-common/src/main/java/org/apache/hadoop/hbase/HBaseConfiguration.java
##########
@@ -85,16 +85,69 @@ public static Configuration addHbaseResources(Configuration conf) {
return conf;
}
+ public static Configuration create() {
+ return create(true);
+ }
+
/**
* Creates a Configuration with HBase resources
* @return a Configuration with HBase resources
*/
- public static Configuration create() {
+ public static Configuration create(boolean lowLimits) {
Configuration conf = new Configuration();
// In case HBaseConfiguration is loaded from a different classloader than
// Configuration, conf needs to be set with appropriate class loader to resolve
// HBase resources.
conf.setClassLoader(HBaseConfiguration.class.getClassLoader());
+
+
+ if (lowLimits) {
+
+ // can't set this for every test currently
+ // conf.set("hbase.regionserver.hostname", "127.0.0.1");
+
+ conf.setInt("hbase.hfilearchiver.thread.pool.max", 2);
+ conf.setInt("hbase.loadincremental.threads.max", 3);
Review comment:
This one is sized by processor count. Rather than hard-code it, keep it based on processor count but divide by 2 or 4 w/ minimum 3?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #1194:
HBASE-23882: Experimenting with low config settings.
Posted by GitBox <gi...@apache.org>.
saintstack commented on a change in pull request #1194: HBASE-23882: Experimenting with low config settings.
URL: https://github.com/apache/hbase/pull/1194#discussion_r383677470
##########
File path: hbase-common/src/main/java/org/apache/hadoop/hbase/HBaseConfiguration.java
##########
@@ -85,16 +85,69 @@ public static Configuration addHbaseResources(Configuration conf) {
return conf;
}
+ public static Configuration create() {
+ return create(true);
+ }
+
/**
* Creates a Configuration with HBase resources
* @return a Configuration with HBase resources
*/
- public static Configuration create() {
+ public static Configuration create(boolean lowLimits) {
Configuration conf = new Configuration();
// In case HBaseConfiguration is loaded from a different classloader than
// Configuration, conf needs to be set with appropriate class loader to resolve
// HBase resources.
conf.setClassLoader(HBaseConfiguration.class.getClassLoader());
+
+
+ if (lowLimits) {
+
+ // can't set this for every test currently
+ // conf.set("hbase.regionserver.hostname", "127.0.0.1");
+
+ conf.setInt("hbase.hfilearchiver.thread.pool.max", 2);
+ conf.setInt("hbase.loadincremental.threads.max", 3);
+
+
+ conf.setInt("hbase.client.sync.wait.timeout.msec", 30000);
+ conf.setInt("zookeeper.recovery.retry", 5);
+ conf.setInt("hbase.client.retries.number", 10);
+ conf.setInt("hbase.hstore.open.and.close.threads.max", 1);
+ conf.setInt(HConstants.REGION_SERVER_REPLICATION_HANDLER_COUNT, 2);
+
+ conf.setInt(HConstants.HBASE_CLIENT_MAX_TOTAL_TASKS, 3);
+
+ conf.setInt(HConstants.REGION_SERVER_HIGH_PRIORITY_HANDLER_COUNT, 2);
+
+
+ conf.setInt(HConstants.REPLICATION_SOURCE_MAXTHREADS_KEY, 1);
+
+ conf.setInt("hbase.netty.worker.count", 1);
+
+ conf.setInt(HConstants.MASTER_OPEN_REGION_THREADS, 3);
+
+ // conf.setInt(HConstants.MASTER_CLOSE_REGION_THREADS, 3);
+ // conf.setInt(HConstants.MASTER_SERVER_OPERATIONS_THREADS, 2);
+ // conf.setInt(HConstants.MASTER_META_SERVER_OPERATIONS_THREADS, 2);
+ // conf.setInt(HConstants.MASTER_LOG_REPLAY_OPS_THREADS, 2);
+ // conf.setInt(HConstants.ZOOKEEPER_MAX_CLIENT_CNXNS, 5);
+
+ conf.setInt("hbase.hconnection.threads.max", 30);
+ conf.setInt("hbase.hconnection.threads.keepalivetime", 30);
+ conf.setInt(HConstants.HBASE_CLIENT_MAX_TOTAL_TASKS, 10);
+
+
+
+ conf.setInt(HConstants.REGION_SERVER_HANDLER_COUNT, 2);
+ conf.setInt("hbase.master.procedure.threads", 1);
Review comment:
Default is 16. Make it based off CPU count? Cutting it down is good idea.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #1194:
HBASE-23882: Experimenting with low config settings.
Posted by GitBox <gi...@apache.org>.
saintstack commented on a change in pull request #1194: HBASE-23882: Experimenting with low config settings.
URL: https://github.com/apache/hbase/pull/1194#discussion_r383676802
##########
File path: hbase-common/src/main/java/org/apache/hadoop/hbase/HBaseConfiguration.java
##########
@@ -85,16 +85,69 @@ public static Configuration addHbaseResources(Configuration conf) {
return conf;
}
+ public static Configuration create() {
+ return create(true);
+ }
+
/**
* Creates a Configuration with HBase resources
* @return a Configuration with HBase resources
*/
- public static Configuration create() {
+ public static Configuration create(boolean lowLimits) {
Configuration conf = new Configuration();
// In case HBaseConfiguration is loaded from a different classloader than
// Configuration, conf needs to be set with appropriate class loader to resolve
// HBase resources.
conf.setClassLoader(HBaseConfiguration.class.getClassLoader());
+
+
+ if (lowLimits) {
+
+ // can't set this for every test currently
+ // conf.set("hbase.regionserver.hostname", "127.0.0.1");
+
+ conf.setInt("hbase.hfilearchiver.thread.pool.max", 2);
+ conf.setInt("hbase.loadincremental.threads.max", 3);
+
+
+ conf.setInt("hbase.client.sync.wait.timeout.msec", 30000);
+ conf.setInt("zookeeper.recovery.retry", 5);
+ conf.setInt("hbase.client.retries.number", 10);
+ conf.setInt("hbase.hstore.open.and.close.threads.max", 1);
+ conf.setInt(HConstants.REGION_SERVER_REPLICATION_HANDLER_COUNT, 2);
+
+ conf.setInt(HConstants.HBASE_CLIENT_MAX_TOTAL_TASKS, 3);
+
+ conf.setInt(HConstants.REGION_SERVER_HIGH_PRIORITY_HANDLER_COUNT, 2);
+
+
+ conf.setInt(HConstants.REPLICATION_SOURCE_MAXTHREADS_KEY, 1);
+
+ conf.setInt("hbase.netty.worker.count", 1);
+
+ conf.setInt(HConstants.MASTER_OPEN_REGION_THREADS, 3);
+
+ // conf.setInt(HConstants.MASTER_CLOSE_REGION_THREADS, 3);
+ // conf.setInt(HConstants.MASTER_SERVER_OPERATIONS_THREADS, 2);
+ // conf.setInt(HConstants.MASTER_META_SERVER_OPERATIONS_THREADS, 2);
+ // conf.setInt(HConstants.MASTER_LOG_REPLAY_OPS_THREADS, 2);
+ // conf.setInt(HConstants.ZOOKEEPER_MAX_CLIENT_CNXNS, 5);
+
+ conf.setInt("hbase.hconnection.threads.max", 30);
+ conf.setInt("hbase.hconnection.threads.keepalivetime", 30);
Review comment:
Default is 60.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services