You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-dev@james.apache.org by ro...@apache.org on 2019/10/25 07:22:23 UTC
[james-project] branch master updated (0cd066b -> 3e9a7a6)
This is an automated email from the ASF dual-hosted git repository.
rouazana pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git.
from 0cd066b Merge remote-tracking branch 'remk/JAMES-2813-enable-serialization-test'
new f96786c JAMES-2922 Add waitForActiveShards property in ElasticSearch configuration
new b66929e JAMES-2922 Set the index.write.wait_for_active_shards property while creating indices
new 3e9a7a6 JAMES-2922 Add documentation about ElasticSearch wait_for_active_shards configuration
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../backends/es/ElasticSearchConfiguration.java | 24 +++++++++--
.../james/backends/es/IndexCreationFactory.java | 19 ++++++---
.../es/ElasticSearchConfigurationTest.java | 47 +++++++++++++++++++++-
.../destination/conf/elasticsearch.properties | 1 +
.../destination/conf/elasticsearch.properties | 1 +
.../destination/conf/elasticsearch.properties | 1 +
.../destination/conf/elasticsearch.properties | 1 +
src/site/xdoc/server/config-elasticsearch.xml | 3 ++
8 files changed, 87 insertions(+), 10 deletions(-)
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[james-project] 03/03: JAMES-2922 Add documentation about
ElasticSearch wait_for_active_shards configuration
Posted by ro...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rouazana pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 3e9a7a6804f1f16c9c5eb61a882b53d4f3242b60
Author: Antoine DUPRAT <ad...@linagora.com>
AuthorDate: Thu Oct 10 13:21:18 2019 +0200
JAMES-2922 Add documentation about ElasticSearch wait_for_active_shards configuration
---
.../run/guice/cassandra-ldap/destination/conf/elasticsearch.properties | 1 +
.../cassandra-rabbitmq-ldap/destination/conf/elasticsearch.properties | 1 +
.../guice/cassandra-rabbitmq/destination/conf/elasticsearch.properties | 1 +
.../run/guice/cassandra/destination/conf/elasticsearch.properties | 1 +
src/site/xdoc/server/config-elasticsearch.xml | 3 +++
5 files changed, 7 insertions(+)
diff --git a/dockerfiles/run/guice/cassandra-ldap/destination/conf/elasticsearch.properties b/dockerfiles/run/guice/cassandra-ldap/destination/conf/elasticsearch.properties
index 08585b4..1ab3ef5 100644
--- a/dockerfiles/run/guice/cassandra-ldap/destination/conf/elasticsearch.properties
+++ b/dockerfiles/run/guice/cassandra-ldap/destination/conf/elasticsearch.properties
@@ -29,6 +29,7 @@ elasticsearch.port=9200
elasticsearch.nb.shards=5
elasticsearch.nb.replica=1
+elasticsearch.index.waitForActiveShards=1
elasticsearch.retryConnection.maxRetries=7
elasticsearch.retryConnection.minDelay=3000
# Index or not attachments (default value: true)
diff --git a/dockerfiles/run/guice/cassandra-rabbitmq-ldap/destination/conf/elasticsearch.properties b/dockerfiles/run/guice/cassandra-rabbitmq-ldap/destination/conf/elasticsearch.properties
index 08585b4..1ab3ef5 100644
--- a/dockerfiles/run/guice/cassandra-rabbitmq-ldap/destination/conf/elasticsearch.properties
+++ b/dockerfiles/run/guice/cassandra-rabbitmq-ldap/destination/conf/elasticsearch.properties
@@ -29,6 +29,7 @@ elasticsearch.port=9200
elasticsearch.nb.shards=5
elasticsearch.nb.replica=1
+elasticsearch.index.waitForActiveShards=1
elasticsearch.retryConnection.maxRetries=7
elasticsearch.retryConnection.minDelay=3000
# Index or not attachments (default value: true)
diff --git a/dockerfiles/run/guice/cassandra-rabbitmq/destination/conf/elasticsearch.properties b/dockerfiles/run/guice/cassandra-rabbitmq/destination/conf/elasticsearch.properties
index 08585b4..1ab3ef5 100644
--- a/dockerfiles/run/guice/cassandra-rabbitmq/destination/conf/elasticsearch.properties
+++ b/dockerfiles/run/guice/cassandra-rabbitmq/destination/conf/elasticsearch.properties
@@ -29,6 +29,7 @@ elasticsearch.port=9200
elasticsearch.nb.shards=5
elasticsearch.nb.replica=1
+elasticsearch.index.waitForActiveShards=1
elasticsearch.retryConnection.maxRetries=7
elasticsearch.retryConnection.minDelay=3000
# Index or not attachments (default value: true)
diff --git a/dockerfiles/run/guice/cassandra/destination/conf/elasticsearch.properties b/dockerfiles/run/guice/cassandra/destination/conf/elasticsearch.properties
index 08585b4..1ab3ef5 100644
--- a/dockerfiles/run/guice/cassandra/destination/conf/elasticsearch.properties
+++ b/dockerfiles/run/guice/cassandra/destination/conf/elasticsearch.properties
@@ -29,6 +29,7 @@ elasticsearch.port=9200
elasticsearch.nb.shards=5
elasticsearch.nb.replica=1
+elasticsearch.index.waitForActiveShards=1
elasticsearch.retryConnection.maxRetries=7
elasticsearch.retryConnection.minDelay=3000
# Index or not attachments (default value: true)
diff --git a/src/site/xdoc/server/config-elasticsearch.xml b/src/site/xdoc/server/config-elasticsearch.xml
index 4504e5b..0246aeb 100644
--- a/src/site/xdoc/server/config-elasticsearch.xml
+++ b/src/site/xdoc/server/config-elasticsearch.xml
@@ -56,6 +56,9 @@
<dd>Number of shards for index provisionned by James</dd>
<dt><strong>elasticsearch.nb.replica</strong></dt>
<dd>Number of replica for index provisionned by James (default: 0)</dd>
+ <dt><strong>elasticsearch.index.waitForActiveShards (default: 1)</strong></dt>
+ <dd>Wait for a certain number of active shard copies before proceeding with the operation.</dd>
+ <dd>You may consult the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.3/docs-index_.html#index-wait-for-active-shards">documentation</a> for more information.</dd>
<dt><strong>elasticsearch.index.mailbox.name</strong></dt>
<dd>Name of the mailbox index backed by the alias. It will be created if missing.</dd>
<dt><strong>elasticsearch.index.name</strong></dt>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[james-project] 02/03: JAMES-2922 Set the
index.write.wait_for_active_shards property while creating indices
Posted by ro...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rouazana pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit b66929e9a1cd9a7ae938d89a98c5eae0b6b1cf16
Author: Antoine DUPRAT <ad...@linagora.com>
AuthorDate: Thu Oct 10 13:20:55 2019 +0200
JAMES-2922 Set the index.write.wait_for_active_shards property while creating indices
---
.../james/backends/es/IndexCreationFactory.java | 19 +++++++++++++------
1 file changed, 13 insertions(+), 6 deletions(-)
diff --git a/backends-common/elasticsearch/src/main/java/org/apache/james/backends/es/IndexCreationFactory.java b/backends-common/elasticsearch/src/main/java/org/apache/james/backends/es/IndexCreationFactory.java
index 18bfbf8..2910c91 100644
--- a/backends-common/elasticsearch/src/main/java/org/apache/james/backends/es/IndexCreationFactory.java
+++ b/backends-common/elasticsearch/src/main/java/org/apache/james/backends/es/IndexCreationFactory.java
@@ -45,12 +45,14 @@ public class IndexCreationFactory {
public static class AliasSpecificationStep {
private final int nbShards;
private final int nbReplica;
+ private final int waitForActiveShards;
private final IndexName indexName;
private final ImmutableList.Builder<AliasName> aliases;
- AliasSpecificationStep(int nbShards, int nbReplica, IndexName indexName) {
+ AliasSpecificationStep(int nbShards, int nbReplica, int waitForActiveShards, IndexName indexName) {
this.nbShards = nbShards;
this.nbReplica = nbReplica;
+ this.waitForActiveShards = waitForActiveShards;
this.indexName = indexName;
this.aliases = ImmutableList.builder();
}
@@ -62,19 +64,21 @@ public class IndexCreationFactory {
}
public RestHighLevelClient createIndexAndAliases(RestHighLevelClient client) {
- return new IndexCreationPerformer(nbShards, nbReplica, indexName, aliases.build()).createIndexAndAliases(client);
+ return new IndexCreationPerformer(nbShards, nbReplica, waitForActiveShards, indexName, aliases.build()).createIndexAndAliases(client);
}
}
static class IndexCreationPerformer {
private final int nbShards;
private final int nbReplica;
+ private final int waitForActiveShards;
private final IndexName indexName;
private final ImmutableList<AliasName> aliases;
- public IndexCreationPerformer(int nbShards, int nbReplica, IndexName indexName, ImmutableList<AliasName> aliases) {
+ public IndexCreationPerformer(int nbShards, int nbReplica, int waitForActiveShards, IndexName indexName, ImmutableList<AliasName> aliases) {
this.nbShards = nbShards;
this.nbReplica = nbReplica;
+ this.waitForActiveShards = waitForActiveShards;
this.indexName = indexName;
this.aliases = aliases;
}
@@ -82,7 +86,7 @@ public class IndexCreationFactory {
public RestHighLevelClient createIndexAndAliases(RestHighLevelClient client) {
Preconditions.checkNotNull(indexName);
try {
- createIndexIfNeeded(client, indexName, generateSetting(nbShards, nbReplica));
+ createIndexIfNeeded(client, indexName, generateSetting(nbShards, nbReplica, waitForActiveShards));
aliases.forEach(Throwing.<AliasName>consumer(alias -> createAliasIfNeeded(client, indexName, alias))
.sneakyThrow());
} catch (IOException e) {
@@ -123,12 +127,13 @@ public class IndexCreationFactory {
}
}
- private XContentBuilder generateSetting(int nbShards, int nbReplica) throws IOException {
+ private XContentBuilder generateSetting(int nbShards, int nbReplica, int waitForActiveShards) throws IOException {
return jsonBuilder()
.startObject()
.startObject("settings")
.field("number_of_shards", nbShards)
.field("number_of_replicas", nbReplica)
+ .field("index.write.wait_for_active_shards", waitForActiveShards)
.startObject("analysis")
.startObject("normalizer")
.startObject(CASE_INSENSITIVE)
@@ -175,6 +180,7 @@ public class IndexCreationFactory {
private final int nbShards;
private final int nbReplica;
+ private final int waitForActiveShards;
public static final String CASE_INSENSITIVE = "case_insensitive";
public static final String KEEP_MAIL_AND_URL = "keep_mail_and_url";
@@ -185,10 +191,11 @@ public class IndexCreationFactory {
public IndexCreationFactory(ElasticSearchConfiguration configuration) {
this.nbShards = configuration.getNbShards();
this.nbReplica = configuration.getNbReplica();
+ this.waitForActiveShards = configuration.getWaitForActiveShards();
}
public AliasSpecificationStep useIndex(IndexName indexName) {
Preconditions.checkNotNull(indexName);
- return new AliasSpecificationStep(nbShards, nbReplica, indexName);
+ return new AliasSpecificationStep(nbShards, nbReplica, waitForActiveShards, indexName);
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[james-project] 01/03: JAMES-2922 Add waitForActiveShards property
in ElasticSearch configuration
Posted by ro...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rouazana pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit f96786c4aaa9f9fb49428c4d7475e2ea3c873279
Author: Antoine DUPRAT <ad...@linagora.com>
AuthorDate: Thu Oct 10 12:49:50 2019 +0200
JAMES-2922 Add waitForActiveShards property in ElasticSearch configuration
---
.../backends/es/ElasticSearchConfiguration.java | 24 +++++++++--
.../es/ElasticSearchConfigurationTest.java | 47 +++++++++++++++++++++-
2 files changed, 67 insertions(+), 4 deletions(-)
diff --git a/backends-common/elasticsearch/src/main/java/org/apache/james/backends/es/ElasticSearchConfiguration.java b/backends-common/elasticsearch/src/main/java/org/apache/james/backends/es/ElasticSearchConfiguration.java
index 0f21537..f50200f 100644
--- a/backends-common/elasticsearch/src/main/java/org/apache/james/backends/es/ElasticSearchConfiguration.java
+++ b/backends-common/elasticsearch/src/main/java/org/apache/james/backends/es/ElasticSearchConfiguration.java
@@ -37,12 +37,12 @@ import com.google.common.collect.ImmutableList;
public class ElasticSearchConfiguration {
-
public static class Builder {
private final ImmutableList.Builder<Host> hosts;
private Optional<Integer> nbShards;
private Optional<Integer> nbReplica;
+ private Optional<Integer> waitForActiveShards;
private Optional<Integer> minDelay;
private Optional<Integer> maxRetries;
private Optional<Duration> requestTimeout;
@@ -51,6 +51,7 @@ public class ElasticSearchConfiguration {
hosts = ImmutableList.builder();
nbShards = Optional.empty();
nbReplica = Optional.empty();
+ waitForActiveShards = Optional.empty();
minDelay = Optional.empty();
maxRetries = Optional.empty();
requestTimeout = Optional.empty();
@@ -78,6 +79,12 @@ public class ElasticSearchConfiguration {
return this;
}
+ public Builder waitForActiveShards(int waitForActiveShards) {
+ Preconditions.checkArgument(waitForActiveShards >= 0, "You need the number of waitForActiveShards to be positive");
+ this.waitForActiveShards = Optional.of(waitForActiveShards);
+ return this;
+ }
+
public Builder minDelay(Optional<Integer> minDelay) {
this.minDelay = minDelay;
return this;
@@ -100,6 +107,7 @@ public class ElasticSearchConfiguration {
hosts,
nbShards.orElse(DEFAULT_NB_SHARDS),
nbReplica.orElse(DEFAULT_NB_REPLICA),
+ waitForActiveShards.orElse(DEFAULT_WAIT_FOR_ACTIVE_SHARDS),
minDelay.orElse(DEFAULT_CONNECTION_MIN_DELAY),
maxRetries.orElse(DEFAULT_CONNECTION_MAX_RETRIES),
requestTimeout.orElse(DEFAULT_REQUEST_TIMEOUT));
@@ -114,6 +122,7 @@ public class ElasticSearchConfiguration {
public static final String ELASTICSEARCH_MASTER_HOST = "elasticsearch.masterHost";
public static final String ELASTICSEARCH_PORT = "elasticsearch.port";
public static final String ELASTICSEARCH_NB_REPLICA = "elasticsearch.nb.replica";
+ public static final String WAIT_FOR_ACTIVE_SHARDS = "elasticsearch.index.waitForActiveShards";
public static final String ELASTICSEARCH_NB_SHARDS = "elasticsearch.nb.shards";
public static final String ELASTICSEARCH_RETRY_CONNECTION_MIN_DELAY = "elasticsearch.retryConnection.minDelay";
public static final String ELASTICSEARCH_RETRY_CONNECTION_MAX_RETRIES = "elasticsearch.retryConnection.maxRetries";
@@ -123,6 +132,7 @@ public class ElasticSearchConfiguration {
public static final Duration DEFAULT_REQUEST_TIMEOUT = Duration.ofSeconds(30);
public static final int DEFAULT_NB_SHARDS = 5;
public static final int DEFAULT_NB_REPLICA = 1;
+ public static final int DEFAULT_WAIT_FOR_ACTIVE_SHARDS = 1;
public static final int DEFAULT_PORT = 9200;
public static final String LOCALHOST = "127.0.0.1";
public static final Optional<Integer> DEFAULT_PORT_AS_OPTIONAL = Optional.of(DEFAULT_PORT);
@@ -136,6 +146,7 @@ public class ElasticSearchConfiguration {
.addHosts(getHosts(configuration))
.nbShards(configuration.getInteger(ELASTICSEARCH_NB_SHARDS, DEFAULT_NB_SHARDS))
.nbReplica(configuration.getInteger(ELASTICSEARCH_NB_REPLICA, DEFAULT_NB_REPLICA))
+ .waitForActiveShards(configuration.getInteger(WAIT_FOR_ACTIVE_SHARDS, DEFAULT_WAIT_FOR_ACTIVE_SHARDS))
.minDelay(Optional.ofNullable(configuration.getInteger(ELASTICSEARCH_RETRY_CONNECTION_MIN_DELAY, null)))
.maxRetries(Optional.ofNullable(configuration.getInteger(ELASTICSEARCH_RETRY_CONNECTION_MAX_RETRIES, null)))
.build();
@@ -179,14 +190,16 @@ public class ElasticSearchConfiguration {
private final ImmutableList<Host> hosts;
private final int nbShards;
private final int nbReplica;
+ private final int waitForActiveShards;
private final int minDelay;
private final int maxRetries;
private final Duration requestTimeout;
- private ElasticSearchConfiguration(ImmutableList<Host> hosts, int nbShards, int nbReplica, int minDelay, int maxRetries, Duration requestTimeout) {
+ private ElasticSearchConfiguration(ImmutableList<Host> hosts, int nbShards, int nbReplica, int waitForActiveShards, int minDelay, int maxRetries, Duration requestTimeout) {
this.hosts = hosts;
this.nbShards = nbShards;
this.nbReplica = nbReplica;
+ this.waitForActiveShards = waitForActiveShards;
this.minDelay = minDelay;
this.maxRetries = maxRetries;
this.requestTimeout = requestTimeout;
@@ -204,6 +217,10 @@ public class ElasticSearchConfiguration {
return nbReplica;
}
+ public int getWaitForActiveShards() {
+ return waitForActiveShards;
+ }
+
public int getMinDelay() {
return minDelay;
}
@@ -223,6 +240,7 @@ public class ElasticSearchConfiguration {
return Objects.equals(this.nbShards, that.nbShards)
&& Objects.equals(this.nbReplica, that.nbReplica)
+ && Objects.equals(this.waitForActiveShards, that.waitForActiveShards)
&& Objects.equals(this.minDelay, that.minDelay)
&& Objects.equals(this.maxRetries, that.maxRetries)
&& Objects.equals(this.hosts, that.hosts)
@@ -233,6 +251,6 @@ public class ElasticSearchConfiguration {
@Override
public final int hashCode() {
- return Objects.hash(hosts, nbShards, nbReplica, minDelay, maxRetries, requestTimeout);
+ return Objects.hash(hosts, nbShards, nbReplica, waitForActiveShards, minDelay, maxRetries, requestTimeout);
}
}
diff --git a/backends-common/elasticsearch/src/test/java/org/apache/james/backends/es/ElasticSearchConfigurationTest.java b/backends-common/elasticsearch/src/test/java/org/apache/james/backends/es/ElasticSearchConfigurationTest.java
index 366c0cc..fb2f0e0 100644
--- a/backends-common/elasticsearch/src/test/java/org/apache/james/backends/es/ElasticSearchConfigurationTest.java
+++ b/backends-common/elasticsearch/src/test/java/org/apache/james/backends/es/ElasticSearchConfigurationTest.java
@@ -67,6 +67,44 @@ public class ElasticSearchConfigurationTest {
}
@Test
+ public void getWaitForActiveShardsShouldReturnConfiguredValue() throws ConfigurationException {
+ PropertiesConfiguration configuration = new PropertiesConfiguration();
+ int value = 36;
+ configuration.addProperty("elasticsearch.index.waitForActiveShards", value);
+ configuration.addProperty("elasticsearch.hosts", "127.0.0.1");
+
+ ElasticSearchConfiguration elasticSearchConfiguration = ElasticSearchConfiguration.fromProperties(configuration);
+
+ assertThat(elasticSearchConfiguration.getWaitForActiveShards())
+ .isEqualTo(value);
+ }
+
+ @Test
+ public void getWaitForActiveShardsShouldReturnConfiguredValueWhenZero() throws ConfigurationException {
+ PropertiesConfiguration configuration = new PropertiesConfiguration();
+ int value = 0;
+ configuration.addProperty("elasticsearch.index.waitForActiveShards", value);
+ configuration.addProperty("elasticsearch.hosts", "127.0.0.1");
+
+ ElasticSearchConfiguration elasticSearchConfiguration = ElasticSearchConfiguration.fromProperties(configuration);
+
+ assertThat(elasticSearchConfiguration.getWaitForActiveShards())
+ .isEqualTo(value);
+ }
+
+ @Test
+ public void getWaitForActiveShardsShouldReturnDefaultValueWhenMissing() throws ConfigurationException {
+ PropertiesConfiguration configuration = new PropertiesConfiguration();
+ configuration.addProperty("elasticsearch.hosts", "127.0.0.1");
+
+ int expectedValue = 1;
+ ElasticSearchConfiguration elasticSearchConfiguration = ElasticSearchConfiguration.fromProperties(configuration);
+
+ assertThat(elasticSearchConfiguration.getWaitForActiveShards())
+ .isEqualTo(expectedValue);
+ }
+
+ @Test
public void getNbShardsShouldReturnConfiguredValue() throws ConfigurationException {
PropertiesConfiguration configuration = new PropertiesConfiguration();
int value = 36;
@@ -257,7 +295,6 @@ public class ElasticSearchConfigurationTest {
ImmutableList.of("localhost:9200"));
}
-
@Test
public void nbReplicaShouldThrowWhenNegative() {
assertThatThrownBy(() ->
@@ -267,6 +304,14 @@ public class ElasticSearchConfigurationTest {
}
@Test
+ public void waitForActiveShardsShouldThrowWhenNegative() {
+ assertThatThrownBy(() ->
+ ElasticSearchConfiguration.builder()
+ .waitForActiveShards(-1))
+ .isInstanceOf(IllegalArgumentException.class);
+ }
+
+ @Test
public void nbShardsShouldThrowWhenNegative() {
assertThatThrownBy(() ->
ElasticSearchConfiguration.builder()
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org