You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-dev@james.apache.org by chibenwa <gi...@git.apache.org> on 2015/10/31 12:09:46 UTC
[GitHub] james-project pull request: Mailbox 257
GitHub user chibenwa opened a pull request:
https://github.com/apache/james-project/pull/13
Mailbox 257
This tool can be used if you change the technology you used as an indexer.
It an also be used to correct index inconsistency.
These inconsistencies can arrise from re ordering of non CRDT operations (example deletes), on server stops, on index information loss.
The idea with this tool is to drop all documents from the idex, read the whole mailbox, and reindex these documents.
Note that this tool is designed to be used against a running, client exposed James server. The idea is to listen on events and correct not yet indexed values (eg flags, or deletes). We do not care about additions as they are well handled by the normal indexer.
I took this to the next level : reindex the whole mailboxmanager. The idea is to get the snapshot of all mailboxes at time T. Then we reindex mailboxes one by one using the process described above. Of course the following process is event corrected (upon Mailbox deletion). We o not care about mailbox additions as they are well handled by the normal indexer.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/chibenwa/james-project MAILBOX-257
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/james-project/pull/13.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #13
----
commit 2b2388f517ce02ab784b3d17214b50c2b566ff98
Author: benwa <bt...@linagora.com>
Date: 2015-10-30T20:10:28Z
MAILBOX-257 Re-indexing documents
commit 133b93fdc729f49967aa54e4d4c33e22ab68ff54
Author: benwa <bt...@linagora.com>
Date: 2015-10-31T09:52:19Z
MAILBOX-257 Add a fake indexer
commit d978ff5f111fc1a7565cd93b96269b758d7ee91a
Author: benwa <bt...@linagora.com>
Date: 2015-10-31T10:19:54Z
MAILBOX-257 Add CLI capability to re-index documents
commit 9baa03052f8863b6e6a6e65affea69129d14a5cd
Author: benwa <bt...@linagora.com>
Date: 2015-10-31T10:35:05Z
MAILBOX-257 Add spring configuration
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[GitHub] james-project pull request: Mailbox 257
Posted by chibenwa <gi...@git.apache.org>.
Github user chibenwa commented on a diff in the pull request:
https://github.com/apache/james-project/pull/13#discussion_r43607008
--- Diff: mailbox/tool/src/main/java/org/apache/james/mailbox/indexer/ReIndexerImpl.java ---
@@ -0,0 +1,138 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.mailbox.indexer;
+
+import org.apache.james.mailbox.MailboxManager;
+import org.apache.james.mailbox.MailboxSession;
+import org.apache.james.mailbox.exception.MailboxException;
+import org.apache.james.mailbox.indexer.events.FlagsMessageEvent;
+import org.apache.james.mailbox.indexer.events.ImpactingEventType;
+import org.apache.james.mailbox.indexer.events.ImpactingMessageEvent;
+import org.apache.james.mailbox.indexer.registrations.GlobalRegistration;
+import org.apache.james.mailbox.indexer.registrations.MailboxRegistration;
+import org.apache.james.mailbox.model.MailboxPath;
+import org.apache.james.mailbox.model.MessageRange;
+import org.apache.james.mailbox.store.MailboxSessionMapperFactory;
+import org.apache.james.mailbox.store.mail.MessageMapper;
+import org.apache.james.mailbox.store.mail.model.Mailbox;
+import org.apache.james.mailbox.store.mail.model.MailboxId;
+import org.apache.james.mailbox.store.mail.model.Message;
+import org.apache.james.mailbox.store.search.ListeningMessageSearchIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+
+/**
+ * Note about live re-indexation handling :
+ *
+ * - Data races may arise... If you modify the stored value between the received event check and the index operation,
+ * you have an inconsistent behavior.
+ *
+ * This class is more about supporting changes in real time for future indexed values. If you change a flags / delete
+ * mails for instance, you will see it in the indexed value !
+ *
+ * Why only care about updates and deletions ? Additions are already handled by the indexer that behaves normaly. We
+ * should just "adapt" our indexed value to the latest value, if any. The normal indexer will take care of new stuff.
+ */
+public class ReIndexerImpl<Id extends MailboxId> implements ReIndexer {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(ReIndexerImpl.class);
+ public static final int LIMIT = 0;
+
+ private final MailboxManager mailboxManager;
+ private final ListeningMessageSearchIndex<Id> messageSearchIndex;
+ private final MailboxSessionMapperFactory<Id> mailboxSessionMapperFactory;
+
+ public ReIndexerImpl(MailboxManager mailboxManager,
+ ListeningMessageSearchIndex<Id> messageSearchIndex,
+ MailboxSessionMapperFactory<Id> mailboxSessionMapperFactory) {
+ this.mailboxManager = mailboxManager;
+ this.messageSearchIndex = messageSearchIndex;
+ this.mailboxSessionMapperFactory = mailboxSessionMapperFactory;
+ }
+
+ public void reIndex(MailboxPath path) throws MailboxException {
+ MailboxSession mailboxSession = mailboxManager.createSystemSession("re-indexing", LOGGER);
+ reIndex(path, mailboxSession);
+ }
+
+
+ public synchronized void reIndex() throws MailboxException {
+ MailboxSession mailboxSession = mailboxManager.createSystemSession("re-indexing", LOGGER);
+ List<MailboxPath> mailboxPaths = mailboxManager.list(mailboxSession);
+ GlobalRegistration globalRegistration = new GlobalRegistration();
+ mailboxManager.addGlobalListener(globalRegistration, mailboxSession);
+ for (MailboxPath mailboxPath : mailboxPaths) {
+ if (globalRegistration.indexThisPath(mailboxPath)) {
+ reIndex(mailboxPath, mailboxSession);
+ }
+ }
+ mailboxManager.removeGlobalListener(globalRegistration, mailboxSession);
+ }
+
+ private void reIndex(MailboxPath path, MailboxSession mailboxSession) throws MailboxException {
+ MailboxRegistration mailboxRegistration = new MailboxRegistration(path);
+ mailboxManager.addListener(path, mailboxRegistration, mailboxSession);
+ LOGGER.info("Intend to reindex " + path);
+ Mailbox<Id> mailbox = mailboxSessionMapperFactory.getMailboxMapper(mailboxSession).findMailboxByPath(path);
+ messageSearchIndex.delete(mailboxSession, mailbox, MessageRange.all());
+ handleIterations(mailboxSession,
+ mailboxRegistration,
+ mailbox,
+ mailboxSessionMapperFactory.getMessageMapper(mailboxSession)
+ .findInMailbox(mailbox,
+ MessageRange.all(),
+ MessageMapper.FetchType.Full,
+ LIMIT));
+ mailboxManager.removeListener(path, mailboxRegistration, mailboxSession);
+ LOGGER.info("Finish to reindex " + path);
+ }
+
+ private void handleIterations(MailboxSession mailboxSession, MailboxRegistration mailboxRegistration, Mailbox<Id> mailbox, Iterator<Message<Id>> iterator) throws MailboxException {
+ while (iterator.hasNext()) {
+ Message<Id> message = iterator.next();
+ ImpactingMessageEvent impactingMessageEvent = findMostRelevant(mailboxRegistration.getImpactingEvents(message.getUid()));
+ if (impactingMessageEvent == null) {
+ messageSearchIndex.add(mailboxSession, mailbox, message);
+ } else if (impactingMessageEvent instanceof FlagsMessageEvent) {
+ message.setFlags(((FlagsMessageEvent) impactingMessageEvent).getFlags());
+ messageSearchIndex.add(mailboxSession, mailbox, message);
+ }
+ }
+ }
+
+ private ImpactingMessageEvent findMostRelevant(Collection<ImpactingMessageEvent> messageEvents) {
+ if (messageEvents == null) {
+ return null;
+ }
+ ImpactingMessageEvent last = null;
+ for (ImpactingMessageEvent impactingMessageEvent : messageEvents) {
+ if (impactingMessageEvent.getType().equals(ImpactingEventType.Deletion)) {
+ return impactingMessageEvent;
+ }
+ last = impactingMessageEvent;
--- End diff --
Ok.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[GitHub] james-project pull request: Mailbox 257
Posted by chibenwa <gi...@git.apache.org>.
Github user chibenwa commented on a diff in the pull request:
https://github.com/apache/james-project/pull/13#discussion_r43606581
--- Diff: mailbox/tool/src/main/java/org/apache/james/mailbox/indexer/ReIndexerImpl.java ---
@@ -0,0 +1,138 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.mailbox.indexer;
+
+import org.apache.james.mailbox.MailboxManager;
+import org.apache.james.mailbox.MailboxSession;
+import org.apache.james.mailbox.exception.MailboxException;
+import org.apache.james.mailbox.indexer.events.FlagsMessageEvent;
+import org.apache.james.mailbox.indexer.events.ImpactingEventType;
+import org.apache.james.mailbox.indexer.events.ImpactingMessageEvent;
+import org.apache.james.mailbox.indexer.registrations.GlobalRegistration;
+import org.apache.james.mailbox.indexer.registrations.MailboxRegistration;
+import org.apache.james.mailbox.model.MailboxPath;
+import org.apache.james.mailbox.model.MessageRange;
+import org.apache.james.mailbox.store.MailboxSessionMapperFactory;
+import org.apache.james.mailbox.store.mail.MessageMapper;
+import org.apache.james.mailbox.store.mail.model.Mailbox;
+import org.apache.james.mailbox.store.mail.model.MailboxId;
+import org.apache.james.mailbox.store.mail.model.Message;
+import org.apache.james.mailbox.store.search.ListeningMessageSearchIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+
+/**
+ * Note about live re-indexation handling :
+ *
+ * - Data races may arise... If you modify the stored value between the received event check and the index operation,
+ * you have an inconsistent behavior.
+ *
+ * This class is more about supporting changes in real time for future indexed values. If you change a flags / delete
+ * mails for instance, you will see it in the indexed value !
+ *
+ * Why only care about updates and deletions ? Additions are already handled by the indexer that behaves normaly. We
+ * should just "adapt" our indexed value to the latest value, if any. The normal indexer will take care of new stuff.
+ */
+public class ReIndexerImpl<Id extends MailboxId> implements ReIndexer {
--- End diff --
If you say so....
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[GitHub] james-project pull request: Mailbox 257
Posted by mbaechler <gi...@git.apache.org>.
Github user mbaechler commented on a diff in the pull request:
https://github.com/apache/james-project/pull/13#discussion_r43605328
--- Diff: mailbox/tool/src/main/java/org/apache/james/mailbox/indexer/ReIndexerImpl.java ---
@@ -0,0 +1,138 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.mailbox.indexer;
+
+import org.apache.james.mailbox.MailboxManager;
+import org.apache.james.mailbox.MailboxSession;
+import org.apache.james.mailbox.exception.MailboxException;
+import org.apache.james.mailbox.indexer.events.FlagsMessageEvent;
+import org.apache.james.mailbox.indexer.events.ImpactingEventType;
+import org.apache.james.mailbox.indexer.events.ImpactingMessageEvent;
+import org.apache.james.mailbox.indexer.registrations.GlobalRegistration;
+import org.apache.james.mailbox.indexer.registrations.MailboxRegistration;
+import org.apache.james.mailbox.model.MailboxPath;
+import org.apache.james.mailbox.model.MessageRange;
+import org.apache.james.mailbox.store.MailboxSessionMapperFactory;
+import org.apache.james.mailbox.store.mail.MessageMapper;
+import org.apache.james.mailbox.store.mail.model.Mailbox;
+import org.apache.james.mailbox.store.mail.model.MailboxId;
+import org.apache.james.mailbox.store.mail.model.Message;
+import org.apache.james.mailbox.store.search.ListeningMessageSearchIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+
+/**
+ * Note about live re-indexation handling :
+ *
+ * - Data races may arise... If you modify the stored value between the received event check and the index operation,
+ * you have an inconsistent behavior.
+ *
+ * This class is more about supporting changes in real time for future indexed values. If you change a flags / delete
+ * mails for instance, you will see it in the indexed value !
+ *
+ * Why only care about updates and deletions ? Additions are already handled by the indexer that behaves normaly. We
+ * should just "adapt" our indexed value to the latest value, if any. The normal indexer will take care of new stuff.
+ */
+public class ReIndexerImpl<Id extends MailboxId> implements ReIndexer {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(ReIndexerImpl.class);
+ public static final int LIMIT = 0;
+
+ private final MailboxManager mailboxManager;
+ private final ListeningMessageSearchIndex<Id> messageSearchIndex;
+ private final MailboxSessionMapperFactory<Id> mailboxSessionMapperFactory;
+
+ public ReIndexerImpl(MailboxManager mailboxManager,
+ ListeningMessageSearchIndex<Id> messageSearchIndex,
+ MailboxSessionMapperFactory<Id> mailboxSessionMapperFactory) {
+ this.mailboxManager = mailboxManager;
+ this.messageSearchIndex = messageSearchIndex;
+ this.mailboxSessionMapperFactory = mailboxSessionMapperFactory;
+ }
+
+ public void reIndex(MailboxPath path) throws MailboxException {
+ MailboxSession mailboxSession = mailboxManager.createSystemSession("re-indexing", LOGGER);
+ reIndex(path, mailboxSession);
+ }
+
+
+ public synchronized void reIndex() throws MailboxException {
+ MailboxSession mailboxSession = mailboxManager.createSystemSession("re-indexing", LOGGER);
+ List<MailboxPath> mailboxPaths = mailboxManager.list(mailboxSession);
+ GlobalRegistration globalRegistration = new GlobalRegistration();
+ mailboxManager.addGlobalListener(globalRegistration, mailboxSession);
+ for (MailboxPath mailboxPath : mailboxPaths) {
+ if (globalRegistration.indexThisPath(mailboxPath)) {
+ reIndex(mailboxPath, mailboxSession);
+ }
+ }
+ mailboxManager.removeGlobalListener(globalRegistration, mailboxSession);
+ }
+
+ private void reIndex(MailboxPath path, MailboxSession mailboxSession) throws MailboxException {
+ MailboxRegistration mailboxRegistration = new MailboxRegistration(path);
+ mailboxManager.addListener(path, mailboxRegistration, mailboxSession);
+ LOGGER.info("Intend to reindex " + path);
--- End diff --
slf4j has placeholders with `info("message {}", argument)` syntax
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[GitHub] james-project pull request: Mailbox 257
Posted by mbaechler <gi...@git.apache.org>.
Github user mbaechler commented on a diff in the pull request:
https://github.com/apache/james-project/pull/13#discussion_r43606178
--- Diff: mailbox/tool/src/main/java/org/apache/james/mailbox/indexer/registrations/MailboxRegistration.java ---
@@ -0,0 +1,62 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.mailbox.indexer.registrations;
+
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Multimap;
+import com.google.common.collect.Multimaps;
+import org.apache.james.mailbox.MailboxListener;
+import org.apache.james.mailbox.indexer.events.FlagsMessageEvent;
+import org.apache.james.mailbox.indexer.events.ImpactingMessageEvent;
+import org.apache.james.mailbox.indexer.events.MessageDeletedEvent;
+import org.apache.james.mailbox.model.MailboxPath;
+import org.apache.james.mailbox.model.UpdatedFlags;
+
+import java.util.Collection;
+
+public class MailboxRegistration implements MailboxListener {
+
+ private final Multimap<Long, ImpactingMessageEvent> impactingMessageEvents;
+ private final MailboxPath mailboxPath;
+
+ public MailboxRegistration(MailboxPath mailboxPath) {
+ this.impactingMessageEvents = Multimaps.synchronizedMultimap(ArrayListMultimap.<Long, ImpactingMessageEvent>create());
--- End diff --
what do you expect from this synchronized ? Are event dispatched from multiple threads ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[GitHub] james-project pull request: Mailbox 257
Posted by chibenwa <gi...@git.apache.org>.
Github user chibenwa commented on a diff in the pull request:
https://github.com/apache/james-project/pull/13#discussion_r43606188
--- Diff: mailbox/tool/src/main/java/org/apache/james/mailbox/indexer/ReIndexerImpl.java ---
@@ -0,0 +1,138 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.mailbox.indexer;
+
+import org.apache.james.mailbox.MailboxManager;
+import org.apache.james.mailbox.MailboxSession;
+import org.apache.james.mailbox.exception.MailboxException;
+import org.apache.james.mailbox.indexer.events.FlagsMessageEvent;
+import org.apache.james.mailbox.indexer.events.ImpactingEventType;
+import org.apache.james.mailbox.indexer.events.ImpactingMessageEvent;
+import org.apache.james.mailbox.indexer.registrations.GlobalRegistration;
+import org.apache.james.mailbox.indexer.registrations.MailboxRegistration;
+import org.apache.james.mailbox.model.MailboxPath;
+import org.apache.james.mailbox.model.MessageRange;
+import org.apache.james.mailbox.store.MailboxSessionMapperFactory;
+import org.apache.james.mailbox.store.mail.MessageMapper;
+import org.apache.james.mailbox.store.mail.model.Mailbox;
+import org.apache.james.mailbox.store.mail.model.MailboxId;
+import org.apache.james.mailbox.store.mail.model.Message;
+import org.apache.james.mailbox.store.search.ListeningMessageSearchIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+
+/**
+ * Note about live re-indexation handling :
+ *
+ * - Data races may arise... If you modify the stored value between the received event check and the index operation,
+ * you have an inconsistent behavior.
+ *
+ * This class is more about supporting changes in real time for future indexed values. If you change a flags / delete
+ * mails for instance, you will see it in the indexed value !
+ *
+ * Why only care about updates and deletions ? Additions are already handled by the indexer that behaves normaly. We
+ * should just "adapt" our indexed value to the latest value, if any. The normal indexer will take care of new stuff.
+ */
+public class ReIndexerImpl<Id extends MailboxId> implements ReIndexer {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(ReIndexerImpl.class);
+ public static final int LIMIT = 0;
+
+ private final MailboxManager mailboxManager;
+ private final ListeningMessageSearchIndex<Id> messageSearchIndex;
+ private final MailboxSessionMapperFactory<Id> mailboxSessionMapperFactory;
+
+ public ReIndexerImpl(MailboxManager mailboxManager,
+ ListeningMessageSearchIndex<Id> messageSearchIndex,
+ MailboxSessionMapperFactory<Id> mailboxSessionMapperFactory) {
+ this.mailboxManager = mailboxManager;
+ this.messageSearchIndex = messageSearchIndex;
+ this.mailboxSessionMapperFactory = mailboxSessionMapperFactory;
+ }
+
+ public void reIndex(MailboxPath path) throws MailboxException {
+ MailboxSession mailboxSession = mailboxManager.createSystemSession("re-indexing", LOGGER);
+ reIndex(path, mailboxSession);
+ }
+
+
+ public synchronized void reIndex() throws MailboxException {
--- End diff --
I think I 'll remove it, it makes no sense in a distributed context.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[GitHub] james-project pull request: Mailbox 257
Posted by chibenwa <gi...@git.apache.org>.
Github user chibenwa closed the pull request at:
https://github.com/apache/james-project/pull/13
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[GitHub] james-project pull request: Mailbox 257
Posted by mbaechler <gi...@git.apache.org>.
Github user mbaechler commented on a diff in the pull request:
https://github.com/apache/james-project/pull/13#discussion_r43605762
--- Diff: mailbox/tool/src/main/java/org/apache/james/mailbox/indexer/registrations/MailboxRegistration.java ---
@@ -0,0 +1,62 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.mailbox.indexer.registrations;
+
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Multimap;
+import com.google.common.collect.Multimaps;
+import org.apache.james.mailbox.MailboxListener;
+import org.apache.james.mailbox.indexer.events.FlagsMessageEvent;
+import org.apache.james.mailbox.indexer.events.ImpactingMessageEvent;
+import org.apache.james.mailbox.indexer.events.MessageDeletedEvent;
+import org.apache.james.mailbox.model.MailboxPath;
+import org.apache.james.mailbox.model.UpdatedFlags;
+
+import java.util.Collection;
+
+public class MailboxRegistration implements MailboxListener {
+
+ private final Multimap<Long, ImpactingMessageEvent> impactingMessageEvents;
+ private final MailboxPath mailboxPath;
+
+ public MailboxRegistration(MailboxPath mailboxPath) {
+ this.impactingMessageEvents = Multimaps.synchronizedMultimap(ArrayListMultimap.<Long, ImpactingMessageEvent>create());
+ this.mailboxPath = mailboxPath;
+ }
+
+ public Collection<ImpactingMessageEvent> getImpactingEvents(long uid) {
--- End diff --
why do you return Collection when you can return a list ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[GitHub] james-project pull request: Mailbox 257
Posted by mbaechler <gi...@git.apache.org>.
Github user mbaechler commented on a diff in the pull request:
https://github.com/apache/james-project/pull/13#discussion_r43605382
--- Diff: mailbox/tool/src/main/java/org/apache/james/mailbox/indexer/ReIndexerImpl.java ---
@@ -0,0 +1,138 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.mailbox.indexer;
+
+import org.apache.james.mailbox.MailboxManager;
+import org.apache.james.mailbox.MailboxSession;
+import org.apache.james.mailbox.exception.MailboxException;
+import org.apache.james.mailbox.indexer.events.FlagsMessageEvent;
+import org.apache.james.mailbox.indexer.events.ImpactingEventType;
+import org.apache.james.mailbox.indexer.events.ImpactingMessageEvent;
+import org.apache.james.mailbox.indexer.registrations.GlobalRegistration;
+import org.apache.james.mailbox.indexer.registrations.MailboxRegistration;
+import org.apache.james.mailbox.model.MailboxPath;
+import org.apache.james.mailbox.model.MessageRange;
+import org.apache.james.mailbox.store.MailboxSessionMapperFactory;
+import org.apache.james.mailbox.store.mail.MessageMapper;
+import org.apache.james.mailbox.store.mail.model.Mailbox;
+import org.apache.james.mailbox.store.mail.model.MailboxId;
+import org.apache.james.mailbox.store.mail.model.Message;
+import org.apache.james.mailbox.store.search.ListeningMessageSearchIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+
+/**
+ * Note about live re-indexation handling :
+ *
+ * - Data races may arise... If you modify the stored value between the received event check and the index operation,
+ * you have an inconsistent behavior.
+ *
+ * This class is more about supporting changes in real time for future indexed values. If you change a flags / delete
+ * mails for instance, you will see it in the indexed value !
+ *
+ * Why only care about updates and deletions ? Additions are already handled by the indexer that behaves normaly. We
+ * should just "adapt" our indexed value to the latest value, if any. The normal indexer will take care of new stuff.
+ */
+public class ReIndexerImpl<Id extends MailboxId> implements ReIndexer {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(ReIndexerImpl.class);
+ public static final int LIMIT = 0;
+
+ private final MailboxManager mailboxManager;
+ private final ListeningMessageSearchIndex<Id> messageSearchIndex;
+ private final MailboxSessionMapperFactory<Id> mailboxSessionMapperFactory;
+
+ public ReIndexerImpl(MailboxManager mailboxManager,
+ ListeningMessageSearchIndex<Id> messageSearchIndex,
+ MailboxSessionMapperFactory<Id> mailboxSessionMapperFactory) {
+ this.mailboxManager = mailboxManager;
+ this.messageSearchIndex = messageSearchIndex;
+ this.mailboxSessionMapperFactory = mailboxSessionMapperFactory;
+ }
+
+ public void reIndex(MailboxPath path) throws MailboxException {
+ MailboxSession mailboxSession = mailboxManager.createSystemSession("re-indexing", LOGGER);
+ reIndex(path, mailboxSession);
+ }
+
+
+ public synchronized void reIndex() throws MailboxException {
+ MailboxSession mailboxSession = mailboxManager.createSystemSession("re-indexing", LOGGER);
+ List<MailboxPath> mailboxPaths = mailboxManager.list(mailboxSession);
+ GlobalRegistration globalRegistration = new GlobalRegistration();
+ mailboxManager.addGlobalListener(globalRegistration, mailboxSession);
+ for (MailboxPath mailboxPath : mailboxPaths) {
+ if (globalRegistration.indexThisPath(mailboxPath)) {
+ reIndex(mailboxPath, mailboxSession);
+ }
+ }
+ mailboxManager.removeGlobalListener(globalRegistration, mailboxSession);
--- End diff --
what if you throw before reaching this line ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[GitHub] james-project pull request: Mailbox 257
Posted by mbaechler <gi...@git.apache.org>.
Github user mbaechler commented on a diff in the pull request:
https://github.com/apache/james-project/pull/13#discussion_r43605165
--- Diff: mailbox/tool/src/main/java/org/apache/james/mailbox/indexer/ReIndexerImpl.java ---
@@ -0,0 +1,138 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.mailbox.indexer;
+
+import org.apache.james.mailbox.MailboxManager;
+import org.apache.james.mailbox.MailboxSession;
+import org.apache.james.mailbox.exception.MailboxException;
+import org.apache.james.mailbox.indexer.events.FlagsMessageEvent;
+import org.apache.james.mailbox.indexer.events.ImpactingEventType;
+import org.apache.james.mailbox.indexer.events.ImpactingMessageEvent;
+import org.apache.james.mailbox.indexer.registrations.GlobalRegistration;
+import org.apache.james.mailbox.indexer.registrations.MailboxRegistration;
+import org.apache.james.mailbox.model.MailboxPath;
+import org.apache.james.mailbox.model.MessageRange;
+import org.apache.james.mailbox.store.MailboxSessionMapperFactory;
+import org.apache.james.mailbox.store.mail.MessageMapper;
+import org.apache.james.mailbox.store.mail.model.Mailbox;
+import org.apache.james.mailbox.store.mail.model.MailboxId;
+import org.apache.james.mailbox.store.mail.model.Message;
+import org.apache.james.mailbox.store.search.ListeningMessageSearchIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+
+/**
+ * Note about live re-indexation handling :
+ *
+ * - Data races may arise... If you modify the stored value between the received event check and the index operation,
+ * you have an inconsistent behavior.
+ *
+ * This class is more about supporting changes in real time for future indexed values. If you change a flags / delete
+ * mails for instance, you will see it in the indexed value !
+ *
+ * Why only care about updates and deletions ? Additions are already handled by the indexer that behaves normaly. We
+ * should just "adapt" our indexed value to the latest value, if any. The normal indexer will take care of new stuff.
+ */
+public class ReIndexerImpl<Id extends MailboxId> implements ReIndexer {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(ReIndexerImpl.class);
+ public static final int LIMIT = 0;
+
+ private final MailboxManager mailboxManager;
+ private final ListeningMessageSearchIndex<Id> messageSearchIndex;
+ private final MailboxSessionMapperFactory<Id> mailboxSessionMapperFactory;
+
+ public ReIndexerImpl(MailboxManager mailboxManager,
+ ListeningMessageSearchIndex<Id> messageSearchIndex,
+ MailboxSessionMapperFactory<Id> mailboxSessionMapperFactory) {
+ this.mailboxManager = mailboxManager;
+ this.messageSearchIndex = messageSearchIndex;
+ this.mailboxSessionMapperFactory = mailboxSessionMapperFactory;
+ }
+
+ public void reIndex(MailboxPath path) throws MailboxException {
+ MailboxSession mailboxSession = mailboxManager.createSystemSession("re-indexing", LOGGER);
+ reIndex(path, mailboxSession);
+ }
+
+
+ public synchronized void reIndex() throws MailboxException {
--- End diff --
what do you think you are protecting with this synchronized ? why no synchronized when you index a single mailbox ? what do you expect in distributed context ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[GitHub] james-project pull request: Mailbox 257
Posted by mbaechler <gi...@git.apache.org>.
Github user mbaechler commented on a diff in the pull request:
https://github.com/apache/james-project/pull/13#discussion_r43605265
--- Diff: mailbox/tool/src/main/java/org/apache/james/mailbox/indexer/ReIndexerImpl.java ---
@@ -0,0 +1,138 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.mailbox.indexer;
+
+import org.apache.james.mailbox.MailboxManager;
+import org.apache.james.mailbox.MailboxSession;
+import org.apache.james.mailbox.exception.MailboxException;
+import org.apache.james.mailbox.indexer.events.FlagsMessageEvent;
+import org.apache.james.mailbox.indexer.events.ImpactingEventType;
+import org.apache.james.mailbox.indexer.events.ImpactingMessageEvent;
+import org.apache.james.mailbox.indexer.registrations.GlobalRegistration;
+import org.apache.james.mailbox.indexer.registrations.MailboxRegistration;
+import org.apache.james.mailbox.model.MailboxPath;
+import org.apache.james.mailbox.model.MessageRange;
+import org.apache.james.mailbox.store.MailboxSessionMapperFactory;
+import org.apache.james.mailbox.store.mail.MessageMapper;
+import org.apache.james.mailbox.store.mail.model.Mailbox;
+import org.apache.james.mailbox.store.mail.model.MailboxId;
+import org.apache.james.mailbox.store.mail.model.Message;
+import org.apache.james.mailbox.store.search.ListeningMessageSearchIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+
+/**
+ * Note about live re-indexation handling :
+ *
+ * - Data races may arise... If you modify the stored value between the received event check and the index operation,
+ * you have an inconsistent behavior.
+ *
+ * This class is more about supporting changes in real time for future indexed values. If you change a flags / delete
+ * mails for instance, you will see it in the indexed value !
+ *
+ * Why only care about updates and deletions ? Additions are already handled by the indexer that behaves normaly. We
+ * should just "adapt" our indexed value to the latest value, if any. The normal indexer will take care of new stuff.
+ */
+public class ReIndexerImpl<Id extends MailboxId> implements ReIndexer {
--- End diff --
I'm sure you could find a better class name
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[GitHub] james-project pull request: Mailbox 257
Posted by chibenwa <gi...@git.apache.org>.
Github user chibenwa commented on a diff in the pull request:
https://github.com/apache/james-project/pull/13#discussion_r43606900
--- Diff: mailbox/tool/src/main/java/org/apache/james/mailbox/indexer/ReIndexerImpl.java ---
@@ -0,0 +1,138 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.mailbox.indexer;
+
+import org.apache.james.mailbox.MailboxManager;
+import org.apache.james.mailbox.MailboxSession;
+import org.apache.james.mailbox.exception.MailboxException;
+import org.apache.james.mailbox.indexer.events.FlagsMessageEvent;
+import org.apache.james.mailbox.indexer.events.ImpactingEventType;
+import org.apache.james.mailbox.indexer.events.ImpactingMessageEvent;
+import org.apache.james.mailbox.indexer.registrations.GlobalRegistration;
+import org.apache.james.mailbox.indexer.registrations.MailboxRegistration;
+import org.apache.james.mailbox.model.MailboxPath;
+import org.apache.james.mailbox.model.MessageRange;
+import org.apache.james.mailbox.store.MailboxSessionMapperFactory;
+import org.apache.james.mailbox.store.mail.MessageMapper;
+import org.apache.james.mailbox.store.mail.model.Mailbox;
+import org.apache.james.mailbox.store.mail.model.MailboxId;
+import org.apache.james.mailbox.store.mail.model.Message;
+import org.apache.james.mailbox.store.search.ListeningMessageSearchIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+
+/**
+ * Note about live re-indexation handling :
+ *
+ * - Data races may arise... If you modify the stored value between the received event check and the index operation,
+ * you have an inconsistent behavior.
+ *
+ * This class is more about supporting changes in real time for future indexed values. If you change a flags / delete
+ * mails for instance, you will see it in the indexed value !
+ *
+ * Why only care about updates and deletions ? Additions are already handled by the indexer that behaves normaly. We
+ * should just "adapt" our indexed value to the latest value, if any. The normal indexer will take care of new stuff.
+ */
+public class ReIndexerImpl<Id extends MailboxId> implements ReIndexer {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(ReIndexerImpl.class);
+ public static final int LIMIT = 0;
+
+ private final MailboxManager mailboxManager;
+ private final ListeningMessageSearchIndex<Id> messageSearchIndex;
+ private final MailboxSessionMapperFactory<Id> mailboxSessionMapperFactory;
+
+ public ReIndexerImpl(MailboxManager mailboxManager,
+ ListeningMessageSearchIndex<Id> messageSearchIndex,
+ MailboxSessionMapperFactory<Id> mailboxSessionMapperFactory) {
+ this.mailboxManager = mailboxManager;
+ this.messageSearchIndex = messageSearchIndex;
+ this.mailboxSessionMapperFactory = mailboxSessionMapperFactory;
+ }
+
+ public void reIndex(MailboxPath path) throws MailboxException {
+ MailboxSession mailboxSession = mailboxManager.createSystemSession("re-indexing", LOGGER);
+ reIndex(path, mailboxSession);
+ }
+
+
+ public synchronized void reIndex() throws MailboxException {
+ MailboxSession mailboxSession = mailboxManager.createSystemSession("re-indexing", LOGGER);
+ List<MailboxPath> mailboxPaths = mailboxManager.list(mailboxSession);
+ GlobalRegistration globalRegistration = new GlobalRegistration();
+ mailboxManager.addGlobalListener(globalRegistration, mailboxSession);
+ for (MailboxPath mailboxPath : mailboxPaths) {
+ if (globalRegistration.indexThisPath(mailboxPath)) {
+ reIndex(mailboxPath, mailboxSession);
+ }
+ }
+ mailboxManager.removeGlobalListener(globalRegistration, mailboxSession);
+ }
+
+ private void reIndex(MailboxPath path, MailboxSession mailboxSession) throws MailboxException {
+ MailboxRegistration mailboxRegistration = new MailboxRegistration(path);
+ mailboxManager.addListener(path, mailboxRegistration, mailboxSession);
+ LOGGER.info("Intend to reindex " + path);
+ Mailbox<Id> mailbox = mailboxSessionMapperFactory.getMailboxMapper(mailboxSession).findMailboxByPath(path);
+ messageSearchIndex.delete(mailboxSession, mailbox, MessageRange.all());
+ handleIterations(mailboxSession,
+ mailboxRegistration,
+ mailbox,
+ mailboxSessionMapperFactory.getMessageMapper(mailboxSession)
+ .findInMailbox(mailbox,
+ MessageRange.all(),
+ MessageMapper.FetchType.Full,
+ LIMIT));
+ mailboxManager.removeListener(path, mailboxRegistration, mailboxSession);
--- End diff --
Thanks for this! If a single document can not be indexed, the process stops and we leak the listener. I guess the wanted behaviour is to not throw during a single docurement indexation, and ensure we remove the listener.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[GitHub] james-project pull request: Mailbox 257
Posted by mbaechler <gi...@git.apache.org>.
Github user mbaechler commented on a diff in the pull request:
https://github.com/apache/james-project/pull/13#discussion_r43610370
--- Diff: mailbox/tool/src/main/java/org/apache/james/mailbox/indexer/registrations/MailboxRegistration.java ---
@@ -0,0 +1,62 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.mailbox.indexer.registrations;
+
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Multimap;
+import com.google.common.collect.Multimaps;
+import org.apache.james.mailbox.MailboxListener;
+import org.apache.james.mailbox.indexer.events.FlagsMessageEvent;
+import org.apache.james.mailbox.indexer.events.ImpactingMessageEvent;
+import org.apache.james.mailbox.indexer.events.MessageDeletedEvent;
+import org.apache.james.mailbox.model.MailboxPath;
+import org.apache.james.mailbox.model.UpdatedFlags;
+
+import java.util.Collection;
+
+public class MailboxRegistration implements MailboxListener {
+
+ private final Multimap<Long, ImpactingMessageEvent> impactingMessageEvents;
+ private final MailboxPath mailboxPath;
+
+ public MailboxRegistration(MailboxPath mailboxPath) {
+ this.impactingMessageEvents = Multimaps.synchronizedMultimap(ArrayListMultimap.<Long, ImpactingMessageEvent>create());
--- End diff --
And you do you prevent deadlock from happening ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[GitHub] james-project pull request: Mailbox 257
Posted by chibenwa <gi...@git.apache.org>.
Github user chibenwa commented on a diff in the pull request:
https://github.com/apache/james-project/pull/13#discussion_r43606920
--- Diff: mailbox/tool/src/main/java/org/apache/james/mailbox/indexer/ReIndexerImpl.java ---
@@ -0,0 +1,138 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.mailbox.indexer;
+
+import org.apache.james.mailbox.MailboxManager;
+import org.apache.james.mailbox.MailboxSession;
+import org.apache.james.mailbox.exception.MailboxException;
+import org.apache.james.mailbox.indexer.events.FlagsMessageEvent;
+import org.apache.james.mailbox.indexer.events.ImpactingEventType;
+import org.apache.james.mailbox.indexer.events.ImpactingMessageEvent;
+import org.apache.james.mailbox.indexer.registrations.GlobalRegistration;
+import org.apache.james.mailbox.indexer.registrations.MailboxRegistration;
+import org.apache.james.mailbox.model.MailboxPath;
+import org.apache.james.mailbox.model.MessageRange;
+import org.apache.james.mailbox.store.MailboxSessionMapperFactory;
+import org.apache.james.mailbox.store.mail.MessageMapper;
+import org.apache.james.mailbox.store.mail.model.Mailbox;
+import org.apache.james.mailbox.store.mail.model.MailboxId;
+import org.apache.james.mailbox.store.mail.model.Message;
+import org.apache.james.mailbox.store.search.ListeningMessageSearchIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+
+/**
+ * Note about live re-indexation handling :
+ *
+ * - Data races may arise... If you modify the stored value between the received event check and the index operation,
+ * you have an inconsistent behavior.
+ *
+ * This class is more about supporting changes in real time for future indexed values. If you change a flags / delete
+ * mails for instance, you will see it in the indexed value !
+ *
+ * Why only care about updates and deletions ? Additions are already handled by the indexer that behaves normaly. We
+ * should just "adapt" our indexed value to the latest value, if any. The normal indexer will take care of new stuff.
+ */
+public class ReIndexerImpl<Id extends MailboxId> implements ReIndexer {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(ReIndexerImpl.class);
+ public static final int LIMIT = 0;
+
+ private final MailboxManager mailboxManager;
+ private final ListeningMessageSearchIndex<Id> messageSearchIndex;
+ private final MailboxSessionMapperFactory<Id> mailboxSessionMapperFactory;
+
+ public ReIndexerImpl(MailboxManager mailboxManager,
+ ListeningMessageSearchIndex<Id> messageSearchIndex,
+ MailboxSessionMapperFactory<Id> mailboxSessionMapperFactory) {
+ this.mailboxManager = mailboxManager;
+ this.messageSearchIndex = messageSearchIndex;
+ this.mailboxSessionMapperFactory = mailboxSessionMapperFactory;
+ }
+
+ public void reIndex(MailboxPath path) throws MailboxException {
+ MailboxSession mailboxSession = mailboxManager.createSystemSession("re-indexing", LOGGER);
+ reIndex(path, mailboxSession);
+ }
+
+
+ public synchronized void reIndex() throws MailboxException {
+ MailboxSession mailboxSession = mailboxManager.createSystemSession("re-indexing", LOGGER);
+ List<MailboxPath> mailboxPaths = mailboxManager.list(mailboxSession);
+ GlobalRegistration globalRegistration = new GlobalRegistration();
+ mailboxManager.addGlobalListener(globalRegistration, mailboxSession);
+ for (MailboxPath mailboxPath : mailboxPaths) {
+ if (globalRegistration.indexThisPath(mailboxPath)) {
+ reIndex(mailboxPath, mailboxSession);
+ }
+ }
+ mailboxManager.removeGlobalListener(globalRegistration, mailboxSession);
+ }
+
+ private void reIndex(MailboxPath path, MailboxSession mailboxSession) throws MailboxException {
+ MailboxRegistration mailboxRegistration = new MailboxRegistration(path);
+ mailboxManager.addListener(path, mailboxRegistration, mailboxSession);
+ LOGGER.info("Intend to reindex " + path);
+ Mailbox<Id> mailbox = mailboxSessionMapperFactory.getMailboxMapper(mailboxSession).findMailboxByPath(path);
+ messageSearchIndex.delete(mailboxSession, mailbox, MessageRange.all());
+ handleIterations(mailboxSession,
+ mailboxRegistration,
+ mailbox,
+ mailboxSessionMapperFactory.getMessageMapper(mailboxSession)
+ .findInMailbox(mailbox,
+ MessageRange.all(),
+ MessageMapper.FetchType.Full,
+ LIMIT));
+ mailboxManager.removeListener(path, mailboxRegistration, mailboxSession);
+ LOGGER.info("Finish to reindex " + path);
+ }
+
+ private void handleIterations(MailboxSession mailboxSession, MailboxRegistration mailboxRegistration, Mailbox<Id> mailbox, Iterator<Message<Id>> iterator) throws MailboxException {
+ while (iterator.hasNext()) {
+ Message<Id> message = iterator.next();
+ ImpactingMessageEvent impactingMessageEvent = findMostRelevant(mailboxRegistration.getImpactingEvents(message.getUid()));
+ if (impactingMessageEvent == null) {
+ messageSearchIndex.add(mailboxSession, mailbox, message);
+ } else if (impactingMessageEvent instanceof FlagsMessageEvent) {
+ message.setFlags(((FlagsMessageEvent) impactingMessageEvent).getFlags());
+ messageSearchIndex.add(mailboxSession, mailbox, message);
+ }
+ }
+ }
+
+ private ImpactingMessageEvent findMostRelevant(Collection<ImpactingMessageEvent> messageEvents) {
+ if (messageEvents == null) {
--- End diff --
Ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[GitHub] james-project pull request: Mailbox 257
Posted by chibenwa <gi...@git.apache.org>.
Github user chibenwa commented on a diff in the pull request:
https://github.com/apache/james-project/pull/13#discussion_r43606130
--- Diff: mailbox/tool/src/main/java/org/apache/james/mailbox/indexer/ReIndexerImpl.java ---
@@ -0,0 +1,138 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.mailbox.indexer;
+
+import org.apache.james.mailbox.MailboxManager;
+import org.apache.james.mailbox.MailboxSession;
+import org.apache.james.mailbox.exception.MailboxException;
+import org.apache.james.mailbox.indexer.events.FlagsMessageEvent;
+import org.apache.james.mailbox.indexer.events.ImpactingEventType;
+import org.apache.james.mailbox.indexer.events.ImpactingMessageEvent;
+import org.apache.james.mailbox.indexer.registrations.GlobalRegistration;
+import org.apache.james.mailbox.indexer.registrations.MailboxRegistration;
+import org.apache.james.mailbox.model.MailboxPath;
+import org.apache.james.mailbox.model.MessageRange;
+import org.apache.james.mailbox.store.MailboxSessionMapperFactory;
+import org.apache.james.mailbox.store.mail.MessageMapper;
+import org.apache.james.mailbox.store.mail.model.Mailbox;
+import org.apache.james.mailbox.store.mail.model.MailboxId;
+import org.apache.james.mailbox.store.mail.model.Message;
+import org.apache.james.mailbox.store.search.ListeningMessageSearchIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+
+/**
+ * Note about live re-indexation handling :
+ *
+ * - Data races may arise... If you modify the stored value between the received event check and the index operation,
+ * you have an inconsistent behavior.
+ *
+ * This class is more about supporting changes in real time for future indexed values. If you change a flags / delete
+ * mails for instance, you will see it in the indexed value !
+ *
+ * Why only care about updates and deletions ? Additions are already handled by the indexer that behaves normaly. We
+ * should just "adapt" our indexed value to the latest value, if any. The normal indexer will take care of new stuff.
+ */
+public class ReIndexerImpl<Id extends MailboxId> implements ReIndexer {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(ReIndexerImpl.class);
+ public static final int LIMIT = 0;
--- End diff --
Yes...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[GitHub] james-project pull request: Mailbox 257
Posted by mbaechler <gi...@git.apache.org>.
Github user mbaechler commented on a diff in the pull request:
https://github.com/apache/james-project/pull/13#discussion_r43605377
--- Diff: mailbox/tool/src/main/java/org/apache/james/mailbox/indexer/ReIndexerImpl.java ---
@@ -0,0 +1,138 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.mailbox.indexer;
+
+import org.apache.james.mailbox.MailboxManager;
+import org.apache.james.mailbox.MailboxSession;
+import org.apache.james.mailbox.exception.MailboxException;
+import org.apache.james.mailbox.indexer.events.FlagsMessageEvent;
+import org.apache.james.mailbox.indexer.events.ImpactingEventType;
+import org.apache.james.mailbox.indexer.events.ImpactingMessageEvent;
+import org.apache.james.mailbox.indexer.registrations.GlobalRegistration;
+import org.apache.james.mailbox.indexer.registrations.MailboxRegistration;
+import org.apache.james.mailbox.model.MailboxPath;
+import org.apache.james.mailbox.model.MessageRange;
+import org.apache.james.mailbox.store.MailboxSessionMapperFactory;
+import org.apache.james.mailbox.store.mail.MessageMapper;
+import org.apache.james.mailbox.store.mail.model.Mailbox;
+import org.apache.james.mailbox.store.mail.model.MailboxId;
+import org.apache.james.mailbox.store.mail.model.Message;
+import org.apache.james.mailbox.store.search.ListeningMessageSearchIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+
+/**
+ * Note about live re-indexation handling :
+ *
+ * - Data races may arise... If you modify the stored value between the received event check and the index operation,
+ * you have an inconsistent behavior.
+ *
+ * This class is more about supporting changes in real time for future indexed values. If you change a flags / delete
+ * mails for instance, you will see it in the indexed value !
+ *
+ * Why only care about updates and deletions ? Additions are already handled by the indexer that behaves normaly. We
+ * should just "adapt" our indexed value to the latest value, if any. The normal indexer will take care of new stuff.
+ */
+public class ReIndexerImpl<Id extends MailboxId> implements ReIndexer {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(ReIndexerImpl.class);
+ public static final int LIMIT = 0;
+
+ private final MailboxManager mailboxManager;
+ private final ListeningMessageSearchIndex<Id> messageSearchIndex;
+ private final MailboxSessionMapperFactory<Id> mailboxSessionMapperFactory;
+
+ public ReIndexerImpl(MailboxManager mailboxManager,
+ ListeningMessageSearchIndex<Id> messageSearchIndex,
+ MailboxSessionMapperFactory<Id> mailboxSessionMapperFactory) {
+ this.mailboxManager = mailboxManager;
+ this.messageSearchIndex = messageSearchIndex;
+ this.mailboxSessionMapperFactory = mailboxSessionMapperFactory;
+ }
+
+ public void reIndex(MailboxPath path) throws MailboxException {
+ MailboxSession mailboxSession = mailboxManager.createSystemSession("re-indexing", LOGGER);
+ reIndex(path, mailboxSession);
+ }
+
+
+ public synchronized void reIndex() throws MailboxException {
+ MailboxSession mailboxSession = mailboxManager.createSystemSession("re-indexing", LOGGER);
+ List<MailboxPath> mailboxPaths = mailboxManager.list(mailboxSession);
+ GlobalRegistration globalRegistration = new GlobalRegistration();
+ mailboxManager.addGlobalListener(globalRegistration, mailboxSession);
+ for (MailboxPath mailboxPath : mailboxPaths) {
+ if (globalRegistration.indexThisPath(mailboxPath)) {
+ reIndex(mailboxPath, mailboxSession);
+ }
+ }
+ mailboxManager.removeGlobalListener(globalRegistration, mailboxSession);
+ }
+
+ private void reIndex(MailboxPath path, MailboxSession mailboxSession) throws MailboxException {
+ MailboxRegistration mailboxRegistration = new MailboxRegistration(path);
+ mailboxManager.addListener(path, mailboxRegistration, mailboxSession);
+ LOGGER.info("Intend to reindex " + path);
+ Mailbox<Id> mailbox = mailboxSessionMapperFactory.getMailboxMapper(mailboxSession).findMailboxByPath(path);
+ messageSearchIndex.delete(mailboxSession, mailbox, MessageRange.all());
+ handleIterations(mailboxSession,
+ mailboxRegistration,
+ mailbox,
+ mailboxSessionMapperFactory.getMessageMapper(mailboxSession)
+ .findInMailbox(mailbox,
+ MessageRange.all(),
+ MessageMapper.FetchType.Full,
+ LIMIT));
+ mailboxManager.removeListener(path, mailboxRegistration, mailboxSession);
--- End diff --
what if you throw before reaching this line ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[GitHub] james-project pull request: Mailbox 257
Posted by chibenwa <gi...@git.apache.org>.
Github user chibenwa commented on a diff in the pull request:
https://github.com/apache/james-project/pull/13#discussion_r43659960
--- Diff: mailbox/tool/src/main/java/org/apache/james/mailbox/indexer/registrations/MailboxRegistration.java ---
@@ -0,0 +1,62 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.mailbox.indexer.registrations;
+
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Multimap;
+import com.google.common.collect.Multimaps;
+import org.apache.james.mailbox.MailboxListener;
+import org.apache.james.mailbox.indexer.events.FlagsMessageEvent;
+import org.apache.james.mailbox.indexer.events.ImpactingMessageEvent;
+import org.apache.james.mailbox.indexer.events.MessageDeletedEvent;
+import org.apache.james.mailbox.model.MailboxPath;
+import org.apache.james.mailbox.model.UpdatedFlags;
+
+import java.util.Collection;
+
+public class MailboxRegistration implements MailboxListener {
+
+ private final Multimap<Long, ImpactingMessageEvent> impactingMessageEvents;
+ private final MailboxPath mailboxPath;
+
+ public MailboxRegistration(MailboxPath mailboxPath) {
+ this.impactingMessageEvents = Multimaps.synchronizedMultimap(ArrayListMultimap.<Long, ImpactingMessageEvent>create());
--- End diff --
From having lock contained in a guava object, obviously. This means only one lock can be held by a bunch of code. At most one lock all the time is equal, according to me to "no situations where somthing bad can arise".
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[GitHub] james-project pull request: Mailbox 257
Posted by mbaechler <gi...@git.apache.org>.
Github user mbaechler commented on a diff in the pull request:
https://github.com/apache/james-project/pull/13#discussion_r43605054
--- Diff: mailbox/tool/src/main/java/org/apache/james/mailbox/indexer/FakeReIndexer.java ---
@@ -0,0 +1,36 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.mailbox.indexer;
+
+import org.apache.james.mailbox.exception.MailboxException;
+import org.apache.james.mailbox.model.MailboxPath;
+
+public class FakeReIndexer implements ReIndexer {
--- End diff --
It's not fake if it throws.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[GitHub] james-project pull request: Mailbox 257
Posted by chibenwa <gi...@git.apache.org>.
Github user chibenwa commented on a diff in the pull request:
https://github.com/apache/james-project/pull/13#discussion_r43606970
--- Diff: mailbox/tool/src/main/java/org/apache/james/mailbox/indexer/registrations/MailboxRegistration.java ---
@@ -0,0 +1,62 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.mailbox.indexer.registrations;
+
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Multimap;
+import com.google.common.collect.Multimaps;
+import org.apache.james.mailbox.MailboxListener;
+import org.apache.james.mailbox.indexer.events.FlagsMessageEvent;
+import org.apache.james.mailbox.indexer.events.ImpactingMessageEvent;
+import org.apache.james.mailbox.indexer.events.MessageDeletedEvent;
+import org.apache.james.mailbox.model.MailboxPath;
+import org.apache.james.mailbox.model.UpdatedFlags;
+
+import java.util.Collection;
+
+public class MailboxRegistration implements MailboxListener {
+
+ private final Multimap<Long, ImpactingMessageEvent> impactingMessageEvents;
+ private final MailboxPath mailboxPath;
+
+ public MailboxRegistration(MailboxPath mailboxPath) {
+ this.impactingMessageEvents = Multimaps.synchronizedMultimap(ArrayListMultimap.<Long, ImpactingMessageEvent>create());
+ this.mailboxPath = mailboxPath;
+ }
+
+ public Collection<ImpactingMessageEvent> getImpactingEvents(long uid) {
--- End diff --
Ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[GitHub] james-project pull request: Mailbox 257
Posted by mbaechler <gi...@git.apache.org>.
Github user mbaechler commented on a diff in the pull request:
https://github.com/apache/james-project/pull/13#discussion_r43606376
--- Diff: mailbox/tool/src/main/java/org/apache/james/mailbox/indexer/registrations/GlobalRegistration.java ---
@@ -0,0 +1,47 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.mailbox.indexer.registrations;
+
+import org.apache.james.mailbox.MailboxListener;
+import org.apache.james.mailbox.model.MailboxPath;
+
+import java.util.concurrent.ConcurrentHashMap;
+
+public class GlobalRegistration implements MailboxListener {
+
+ private final ConcurrentHashMap<MailboxPath, Boolean> impactingEvents;
+
+ public GlobalRegistration() {
+ this.impactingEvents = new ConcurrentHashMap<MailboxPath, Boolean>();
+ }
+
+ public boolean indexThisPath(MailboxPath mailboxPath) {
--- End diff --
the name suggest it does the indexing but the code is more "does this path needs indexing"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[GitHub] james-project pull request: Mailbox 257
Posted by chibenwa <gi...@git.apache.org>.
Github user chibenwa commented on a diff in the pull request:
https://github.com/apache/james-project/pull/13#discussion_r43607641
--- Diff: mailbox/tool/src/main/java/org/apache/james/mailbox/indexer/registrations/MailboxRegistration.java ---
@@ -0,0 +1,62 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.mailbox.indexer.registrations;
+
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Multimap;
+import com.google.common.collect.Multimaps;
+import org.apache.james.mailbox.MailboxListener;
+import org.apache.james.mailbox.indexer.events.FlagsMessageEvent;
+import org.apache.james.mailbox.indexer.events.ImpactingMessageEvent;
+import org.apache.james.mailbox.indexer.events.MessageDeletedEvent;
+import org.apache.james.mailbox.model.MailboxPath;
+import org.apache.james.mailbox.model.UpdatedFlags;
+
+import java.util.Collection;
+
+public class MailboxRegistration implements MailboxListener {
+
+ private final Multimap<Long, ImpactingMessageEvent> impactingMessageEvents;
+ private final MailboxPath mailboxPath;
+
+ public MailboxRegistration(MailboxPath mailboxPath) {
+ this.impactingMessageEvents = Multimaps.synchronizedMultimap(ArrayListMultimap.<Long, ImpactingMessageEvent>create());
--- End diff --
Yes it is. . . The goal is to protect from local concurency
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[GitHub] james-project pull request: Mailbox 257
Posted by mbaechler <gi...@git.apache.org>.
Github user mbaechler commented on a diff in the pull request:
https://github.com/apache/james-project/pull/13#discussion_r43605465
--- Diff: mailbox/tool/src/main/java/org/apache/james/mailbox/indexer/ReIndexerImpl.java ---
@@ -0,0 +1,138 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.mailbox.indexer;
+
+import org.apache.james.mailbox.MailboxManager;
+import org.apache.james.mailbox.MailboxSession;
+import org.apache.james.mailbox.exception.MailboxException;
+import org.apache.james.mailbox.indexer.events.FlagsMessageEvent;
+import org.apache.james.mailbox.indexer.events.ImpactingEventType;
+import org.apache.james.mailbox.indexer.events.ImpactingMessageEvent;
+import org.apache.james.mailbox.indexer.registrations.GlobalRegistration;
+import org.apache.james.mailbox.indexer.registrations.MailboxRegistration;
+import org.apache.james.mailbox.model.MailboxPath;
+import org.apache.james.mailbox.model.MessageRange;
+import org.apache.james.mailbox.store.MailboxSessionMapperFactory;
+import org.apache.james.mailbox.store.mail.MessageMapper;
+import org.apache.james.mailbox.store.mail.model.Mailbox;
+import org.apache.james.mailbox.store.mail.model.MailboxId;
+import org.apache.james.mailbox.store.mail.model.Message;
+import org.apache.james.mailbox.store.search.ListeningMessageSearchIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+
+/**
+ * Note about live re-indexation handling :
+ *
+ * - Data races may arise... If you modify the stored value between the received event check and the index operation,
+ * you have an inconsistent behavior.
+ *
+ * This class is more about supporting changes in real time for future indexed values. If you change a flags / delete
+ * mails for instance, you will see it in the indexed value !
+ *
+ * Why only care about updates and deletions ? Additions are already handled by the indexer that behaves normaly. We
+ * should just "adapt" our indexed value to the latest value, if any. The normal indexer will take care of new stuff.
+ */
+public class ReIndexerImpl<Id extends MailboxId> implements ReIndexer {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(ReIndexerImpl.class);
+ public static final int LIMIT = 0;
+
+ private final MailboxManager mailboxManager;
+ private final ListeningMessageSearchIndex<Id> messageSearchIndex;
+ private final MailboxSessionMapperFactory<Id> mailboxSessionMapperFactory;
+
+ public ReIndexerImpl(MailboxManager mailboxManager,
+ ListeningMessageSearchIndex<Id> messageSearchIndex,
+ MailboxSessionMapperFactory<Id> mailboxSessionMapperFactory) {
+ this.mailboxManager = mailboxManager;
+ this.messageSearchIndex = messageSearchIndex;
+ this.mailboxSessionMapperFactory = mailboxSessionMapperFactory;
+ }
+
+ public void reIndex(MailboxPath path) throws MailboxException {
+ MailboxSession mailboxSession = mailboxManager.createSystemSession("re-indexing", LOGGER);
+ reIndex(path, mailboxSession);
+ }
+
+
+ public synchronized void reIndex() throws MailboxException {
+ MailboxSession mailboxSession = mailboxManager.createSystemSession("re-indexing", LOGGER);
+ List<MailboxPath> mailboxPaths = mailboxManager.list(mailboxSession);
+ GlobalRegistration globalRegistration = new GlobalRegistration();
+ mailboxManager.addGlobalListener(globalRegistration, mailboxSession);
+ for (MailboxPath mailboxPath : mailboxPaths) {
+ if (globalRegistration.indexThisPath(mailboxPath)) {
+ reIndex(mailboxPath, mailboxSession);
+ }
+ }
+ mailboxManager.removeGlobalListener(globalRegistration, mailboxSession);
+ }
+
+ private void reIndex(MailboxPath path, MailboxSession mailboxSession) throws MailboxException {
+ MailboxRegistration mailboxRegistration = new MailboxRegistration(path);
+ mailboxManager.addListener(path, mailboxRegistration, mailboxSession);
+ LOGGER.info("Intend to reindex " + path);
+ Mailbox<Id> mailbox = mailboxSessionMapperFactory.getMailboxMapper(mailboxSession).findMailboxByPath(path);
+ messageSearchIndex.delete(mailboxSession, mailbox, MessageRange.all());
+ handleIterations(mailboxSession,
+ mailboxRegistration,
+ mailbox,
+ mailboxSessionMapperFactory.getMessageMapper(mailboxSession)
+ .findInMailbox(mailbox,
+ MessageRange.all(),
+ MessageMapper.FetchType.Full,
+ LIMIT));
+ mailboxManager.removeListener(path, mailboxRegistration, mailboxSession);
+ LOGGER.info("Finish to reindex " + path);
+ }
+
+ private void handleIterations(MailboxSession mailboxSession, MailboxRegistration mailboxRegistration, Mailbox<Id> mailbox, Iterator<Message<Id>> iterator) throws MailboxException {
+ while (iterator.hasNext()) {
+ Message<Id> message = iterator.next();
+ ImpactingMessageEvent impactingMessageEvent = findMostRelevant(mailboxRegistration.getImpactingEvents(message.getUid()));
+ if (impactingMessageEvent == null) {
+ messageSearchIndex.add(mailboxSession, mailbox, message);
+ } else if (impactingMessageEvent instanceof FlagsMessageEvent) {
+ message.setFlags(((FlagsMessageEvent) impactingMessageEvent).getFlags());
+ messageSearchIndex.add(mailboxSession, mailbox, message);
+ }
+ }
+ }
+
+ private ImpactingMessageEvent findMostRelevant(Collection<ImpactingMessageEvent> messageEvents) {
+ if (messageEvents == null) {
--- End diff --
why would you get null instead of empty collection ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[GitHub] james-project pull request: Mailbox 257
Posted by mbaechler <gi...@git.apache.org>.
Github user mbaechler commented on a diff in the pull request:
https://github.com/apache/james-project/pull/13#discussion_r43605079
--- Diff: mailbox/tool/src/main/java/org/apache/james/mailbox/indexer/ReIndexerImpl.java ---
@@ -0,0 +1,138 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.mailbox.indexer;
+
+import org.apache.james.mailbox.MailboxManager;
+import org.apache.james.mailbox.MailboxSession;
+import org.apache.james.mailbox.exception.MailboxException;
+import org.apache.james.mailbox.indexer.events.FlagsMessageEvent;
+import org.apache.james.mailbox.indexer.events.ImpactingEventType;
+import org.apache.james.mailbox.indexer.events.ImpactingMessageEvent;
+import org.apache.james.mailbox.indexer.registrations.GlobalRegistration;
+import org.apache.james.mailbox.indexer.registrations.MailboxRegistration;
+import org.apache.james.mailbox.model.MailboxPath;
+import org.apache.james.mailbox.model.MessageRange;
+import org.apache.james.mailbox.store.MailboxSessionMapperFactory;
+import org.apache.james.mailbox.store.mail.MessageMapper;
+import org.apache.james.mailbox.store.mail.model.Mailbox;
+import org.apache.james.mailbox.store.mail.model.MailboxId;
+import org.apache.james.mailbox.store.mail.model.Message;
+import org.apache.james.mailbox.store.search.ListeningMessageSearchIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+
+/**
+ * Note about live re-indexation handling :
+ *
+ * - Data races may arise... If you modify the stored value between the received event check and the index operation,
+ * you have an inconsistent behavior.
+ *
+ * This class is more about supporting changes in real time for future indexed values. If you change a flags / delete
+ * mails for instance, you will see it in the indexed value !
+ *
+ * Why only care about updates and deletions ? Additions are already handled by the indexer that behaves normaly. We
+ * should just "adapt" our indexed value to the latest value, if any. The normal indexer will take care of new stuff.
+ */
+public class ReIndexerImpl<Id extends MailboxId> implements ReIndexer {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(ReIndexerImpl.class);
+ public static final int LIMIT = 0;
--- End diff --
does this mean "no limit" ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[GitHub] james-project pull request: Mailbox 257
Posted by mbaechler <gi...@git.apache.org>.
Github user mbaechler commented on a diff in the pull request:
https://github.com/apache/james-project/pull/13#discussion_r43605874
--- Diff: mailbox/tool/src/main/java/org/apache/james/mailbox/indexer/ReIndexerImpl.java ---
@@ -0,0 +1,138 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.mailbox.indexer;
+
+import org.apache.james.mailbox.MailboxManager;
+import org.apache.james.mailbox.MailboxSession;
+import org.apache.james.mailbox.exception.MailboxException;
+import org.apache.james.mailbox.indexer.events.FlagsMessageEvent;
+import org.apache.james.mailbox.indexer.events.ImpactingEventType;
+import org.apache.james.mailbox.indexer.events.ImpactingMessageEvent;
+import org.apache.james.mailbox.indexer.registrations.GlobalRegistration;
+import org.apache.james.mailbox.indexer.registrations.MailboxRegistration;
+import org.apache.james.mailbox.model.MailboxPath;
+import org.apache.james.mailbox.model.MessageRange;
+import org.apache.james.mailbox.store.MailboxSessionMapperFactory;
+import org.apache.james.mailbox.store.mail.MessageMapper;
+import org.apache.james.mailbox.store.mail.model.Mailbox;
+import org.apache.james.mailbox.store.mail.model.MailboxId;
+import org.apache.james.mailbox.store.mail.model.Message;
+import org.apache.james.mailbox.store.search.ListeningMessageSearchIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+
+/**
+ * Note about live re-indexation handling :
+ *
+ * - Data races may arise... If you modify the stored value between the received event check and the index operation,
+ * you have an inconsistent behavior.
+ *
+ * This class is more about supporting changes in real time for future indexed values. If you change a flags / delete
+ * mails for instance, you will see it in the indexed value !
+ *
+ * Why only care about updates and deletions ? Additions are already handled by the indexer that behaves normaly. We
+ * should just "adapt" our indexed value to the latest value, if any. The normal indexer will take care of new stuff.
+ */
+public class ReIndexerImpl<Id extends MailboxId> implements ReIndexer {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(ReIndexerImpl.class);
+ public static final int LIMIT = 0;
+
+ private final MailboxManager mailboxManager;
+ private final ListeningMessageSearchIndex<Id> messageSearchIndex;
+ private final MailboxSessionMapperFactory<Id> mailboxSessionMapperFactory;
+
+ public ReIndexerImpl(MailboxManager mailboxManager,
+ ListeningMessageSearchIndex<Id> messageSearchIndex,
+ MailboxSessionMapperFactory<Id> mailboxSessionMapperFactory) {
+ this.mailboxManager = mailboxManager;
+ this.messageSearchIndex = messageSearchIndex;
+ this.mailboxSessionMapperFactory = mailboxSessionMapperFactory;
+ }
+
+ public void reIndex(MailboxPath path) throws MailboxException {
+ MailboxSession mailboxSession = mailboxManager.createSystemSession("re-indexing", LOGGER);
+ reIndex(path, mailboxSession);
+ }
+
+
+ public synchronized void reIndex() throws MailboxException {
+ MailboxSession mailboxSession = mailboxManager.createSystemSession("re-indexing", LOGGER);
+ List<MailboxPath> mailboxPaths = mailboxManager.list(mailboxSession);
+ GlobalRegistration globalRegistration = new GlobalRegistration();
+ mailboxManager.addGlobalListener(globalRegistration, mailboxSession);
+ for (MailboxPath mailboxPath : mailboxPaths) {
+ if (globalRegistration.indexThisPath(mailboxPath)) {
+ reIndex(mailboxPath, mailboxSession);
+ }
+ }
+ mailboxManager.removeGlobalListener(globalRegistration, mailboxSession);
+ }
+
+ private void reIndex(MailboxPath path, MailboxSession mailboxSession) throws MailboxException {
+ MailboxRegistration mailboxRegistration = new MailboxRegistration(path);
+ mailboxManager.addListener(path, mailboxRegistration, mailboxSession);
+ LOGGER.info("Intend to reindex " + path);
+ Mailbox<Id> mailbox = mailboxSessionMapperFactory.getMailboxMapper(mailboxSession).findMailboxByPath(path);
+ messageSearchIndex.delete(mailboxSession, mailbox, MessageRange.all());
+ handleIterations(mailboxSession,
+ mailboxRegistration,
+ mailbox,
+ mailboxSessionMapperFactory.getMessageMapper(mailboxSession)
+ .findInMailbox(mailbox,
+ MessageRange.all(),
+ MessageMapper.FetchType.Full,
+ LIMIT));
+ mailboxManager.removeListener(path, mailboxRegistration, mailboxSession);
+ LOGGER.info("Finish to reindex " + path);
+ }
+
+ private void handleIterations(MailboxSession mailboxSession, MailboxRegistration mailboxRegistration, Mailbox<Id> mailbox, Iterator<Message<Id>> iterator) throws MailboxException {
+ while (iterator.hasNext()) {
+ Message<Id> message = iterator.next();
+ ImpactingMessageEvent impactingMessageEvent = findMostRelevant(mailboxRegistration.getImpactingEvents(message.getUid()));
+ if (impactingMessageEvent == null) {
+ messageSearchIndex.add(mailboxSession, mailbox, message);
+ } else if (impactingMessageEvent instanceof FlagsMessageEvent) {
+ message.setFlags(((FlagsMessageEvent) impactingMessageEvent).getFlags());
+ messageSearchIndex.add(mailboxSession, mailbox, message);
+ }
+ }
+ }
+
+ private ImpactingMessageEvent findMostRelevant(Collection<ImpactingMessageEvent> messageEvents) {
+ if (messageEvents == null) {
+ return null;
+ }
+ ImpactingMessageEvent last = null;
+ for (ImpactingMessageEvent impactingMessageEvent : messageEvents) {
+ if (impactingMessageEvent.getType().equals(ImpactingEventType.Deletion)) {
+ return impactingMessageEvent;
+ }
+ last = impactingMessageEvent;
--- End diff --
if you get a list of messageEvents, you can just split your "find deletion" loop from the getLast by using Iterables.getLast()
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[GitHub] james-project pull request: Mailbox 257
Posted by chibenwa <gi...@git.apache.org>.
Github user chibenwa commented on a diff in the pull request:
https://github.com/apache/james-project/pull/13#discussion_r43606072
--- Diff: mailbox/tool/src/main/java/org/apache/james/mailbox/indexer/FakeReIndexer.java ---
@@ -0,0 +1,36 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.mailbox.indexer;
+
+import org.apache.james.mailbox.exception.MailboxException;
+import org.apache.james.mailbox.model.MailboxPath;
+
+public class FakeReIndexer implements ReIndexer {
--- End diff --
Ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[GitHub] james-project pull request: Mailbox 257
Posted by chibenwa <gi...@git.apache.org>.
Github user chibenwa commented on the pull request:
https://github.com/apache/james-project/pull/13#issuecomment-152959494
Thanks for the review.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[GitHub] james-project pull request: Mailbox 257
Posted by mbaechler <gi...@git.apache.org>.
Github user mbaechler commented on a diff in the pull request:
https://github.com/apache/james-project/pull/13#discussion_r43606223
--- Diff: mailbox/tool/src/main/java/org/apache/james/mailbox/indexer/registrations/GlobalRegistration.java ---
@@ -0,0 +1,47 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.mailbox.indexer.registrations;
+
+import org.apache.james.mailbox.MailboxListener;
+import org.apache.james.mailbox.model.MailboxPath;
+
+import java.util.concurrent.ConcurrentHashMap;
+
+public class GlobalRegistration implements MailboxListener {
+
+ private final ConcurrentHashMap<MailboxPath, Boolean> impactingEvents;
+
+ public GlobalRegistration() {
+ this.impactingEvents = new ConcurrentHashMap<MailboxPath, Boolean>();
+ }
+
+ public boolean indexThisPath(MailboxPath mailboxPath) {
+ return impactingEvents.get(mailboxPath) != null;
+ }
+
+ @Override
+ public void event(Event event) {
+ if (event instanceof MailboxDeletion) {
+ impactingEvents.putIfAbsent(event.getMailboxPath(), true);
--- End diff --
why putIfAbsent and not put ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[GitHub] james-project pull request: Mailbox 257
Posted by chibenwa <gi...@git.apache.org>.
Github user chibenwa commented on a diff in the pull request:
https://github.com/apache/james-project/pull/13#discussion_r43606590
--- Diff: mailbox/tool/src/main/java/org/apache/james/mailbox/indexer/ReIndexerImpl.java ---
@@ -0,0 +1,138 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.mailbox.indexer;
+
+import org.apache.james.mailbox.MailboxManager;
+import org.apache.james.mailbox.MailboxSession;
+import org.apache.james.mailbox.exception.MailboxException;
+import org.apache.james.mailbox.indexer.events.FlagsMessageEvent;
+import org.apache.james.mailbox.indexer.events.ImpactingEventType;
+import org.apache.james.mailbox.indexer.events.ImpactingMessageEvent;
+import org.apache.james.mailbox.indexer.registrations.GlobalRegistration;
+import org.apache.james.mailbox.indexer.registrations.MailboxRegistration;
+import org.apache.james.mailbox.model.MailboxPath;
+import org.apache.james.mailbox.model.MessageRange;
+import org.apache.james.mailbox.store.MailboxSessionMapperFactory;
+import org.apache.james.mailbox.store.mail.MessageMapper;
+import org.apache.james.mailbox.store.mail.model.Mailbox;
+import org.apache.james.mailbox.store.mail.model.MailboxId;
+import org.apache.james.mailbox.store.mail.model.Message;
+import org.apache.james.mailbox.store.search.ListeningMessageSearchIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+
+/**
+ * Note about live re-indexation handling :
+ *
+ * - Data races may arise... If you modify the stored value between the received event check and the index operation,
+ * you have an inconsistent behavior.
+ *
+ * This class is more about supporting changes in real time for future indexed values. If you change a flags / delete
+ * mails for instance, you will see it in the indexed value !
+ *
+ * Why only care about updates and deletions ? Additions are already handled by the indexer that behaves normaly. We
+ * should just "adapt" our indexed value to the latest value, if any. The normal indexer will take care of new stuff.
+ */
+public class ReIndexerImpl<Id extends MailboxId> implements ReIndexer {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(ReIndexerImpl.class);
+ public static final int LIMIT = 0;
+
+ private final MailboxManager mailboxManager;
+ private final ListeningMessageSearchIndex<Id> messageSearchIndex;
+ private final MailboxSessionMapperFactory<Id> mailboxSessionMapperFactory;
+
+ public ReIndexerImpl(MailboxManager mailboxManager,
+ ListeningMessageSearchIndex<Id> messageSearchIndex,
+ MailboxSessionMapperFactory<Id> mailboxSessionMapperFactory) {
+ this.mailboxManager = mailboxManager;
+ this.messageSearchIndex = messageSearchIndex;
+ this.mailboxSessionMapperFactory = mailboxSessionMapperFactory;
+ }
+
+ public void reIndex(MailboxPath path) throws MailboxException {
+ MailboxSession mailboxSession = mailboxManager.createSystemSession("re-indexing", LOGGER);
+ reIndex(path, mailboxSession);
+ }
+
+
+ public synchronized void reIndex() throws MailboxException {
+ MailboxSession mailboxSession = mailboxManager.createSystemSession("re-indexing", LOGGER);
+ List<MailboxPath> mailboxPaths = mailboxManager.list(mailboxSession);
+ GlobalRegistration globalRegistration = new GlobalRegistration();
+ mailboxManager.addGlobalListener(globalRegistration, mailboxSession);
+ for (MailboxPath mailboxPath : mailboxPaths) {
+ if (globalRegistration.indexThisPath(mailboxPath)) {
+ reIndex(mailboxPath, mailboxSession);
+ }
+ }
+ mailboxManager.removeGlobalListener(globalRegistration, mailboxSession);
+ }
+
+ private void reIndex(MailboxPath path, MailboxSession mailboxSession) throws MailboxException {
+ MailboxRegistration mailboxRegistration = new MailboxRegistration(path);
+ mailboxManager.addListener(path, mailboxRegistration, mailboxSession);
+ LOGGER.info("Intend to reindex " + path);
--- End diff --
Ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[GitHub] james-project pull request: Mailbox 257
Posted by chibenwa <gi...@git.apache.org>.
Github user chibenwa commented on a diff in the pull request:
https://github.com/apache/james-project/pull/13#discussion_r43606917
--- Diff: mailbox/tool/src/main/java/org/apache/james/mailbox/indexer/ReIndexerImpl.java ---
@@ -0,0 +1,138 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.mailbox.indexer;
+
+import org.apache.james.mailbox.MailboxManager;
+import org.apache.james.mailbox.MailboxSession;
+import org.apache.james.mailbox.exception.MailboxException;
+import org.apache.james.mailbox.indexer.events.FlagsMessageEvent;
+import org.apache.james.mailbox.indexer.events.ImpactingEventType;
+import org.apache.james.mailbox.indexer.events.ImpactingMessageEvent;
+import org.apache.james.mailbox.indexer.registrations.GlobalRegistration;
+import org.apache.james.mailbox.indexer.registrations.MailboxRegistration;
+import org.apache.james.mailbox.model.MailboxPath;
+import org.apache.james.mailbox.model.MessageRange;
+import org.apache.james.mailbox.store.MailboxSessionMapperFactory;
+import org.apache.james.mailbox.store.mail.MessageMapper;
+import org.apache.james.mailbox.store.mail.model.Mailbox;
+import org.apache.james.mailbox.store.mail.model.MailboxId;
+import org.apache.james.mailbox.store.mail.model.Message;
+import org.apache.james.mailbox.store.search.ListeningMessageSearchIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+
+/**
+ * Note about live re-indexation handling :
+ *
+ * - Data races may arise... If you modify the stored value between the received event check and the index operation,
+ * you have an inconsistent behavior.
+ *
+ * This class is more about supporting changes in real time for future indexed values. If you change a flags / delete
+ * mails for instance, you will see it in the indexed value !
+ *
+ * Why only care about updates and deletions ? Additions are already handled by the indexer that behaves normaly. We
+ * should just "adapt" our indexed value to the latest value, if any. The normal indexer will take care of new stuff.
+ */
+public class ReIndexerImpl<Id extends MailboxId> implements ReIndexer {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(ReIndexerImpl.class);
+ public static final int LIMIT = 0;
+
+ private final MailboxManager mailboxManager;
+ private final ListeningMessageSearchIndex<Id> messageSearchIndex;
+ private final MailboxSessionMapperFactory<Id> mailboxSessionMapperFactory;
+
+ public ReIndexerImpl(MailboxManager mailboxManager,
+ ListeningMessageSearchIndex<Id> messageSearchIndex,
+ MailboxSessionMapperFactory<Id> mailboxSessionMapperFactory) {
+ this.mailboxManager = mailboxManager;
+ this.messageSearchIndex = messageSearchIndex;
+ this.mailboxSessionMapperFactory = mailboxSessionMapperFactory;
+ }
+
+ public void reIndex(MailboxPath path) throws MailboxException {
+ MailboxSession mailboxSession = mailboxManager.createSystemSession("re-indexing", LOGGER);
+ reIndex(path, mailboxSession);
+ }
+
+
+ public synchronized void reIndex() throws MailboxException {
+ MailboxSession mailboxSession = mailboxManager.createSystemSession("re-indexing", LOGGER);
+ List<MailboxPath> mailboxPaths = mailboxManager.list(mailboxSession);
+ GlobalRegistration globalRegistration = new GlobalRegistration();
+ mailboxManager.addGlobalListener(globalRegistration, mailboxSession);
+ for (MailboxPath mailboxPath : mailboxPaths) {
+ if (globalRegistration.indexThisPath(mailboxPath)) {
+ reIndex(mailboxPath, mailboxSession);
+ }
+ }
+ mailboxManager.removeGlobalListener(globalRegistration, mailboxSession);
--- End diff --
Same thing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org