You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@james.apache.org by GitBox <gi...@apache.org> on 2021/08/09 11:02:19 UTC
[GitHub] [james-project] chibenwa opened a new pull request #584: JAMES-3544 Attachment refactoring (step 2)
chibenwa opened a new pull request #584:
URL: https://github.com/apache/james-project/pull/584
- Hold messageId within mail table
- Rework JMAP draft to reduce queries related to attachments
TODO:
- Migration to add messageId to attachment table
- Group reads on the BlobManager - this allows querying only once the message/mailbox context of attachments
Optional:
- Check if the dirty cooking is needed within ttachmentId and if we could not just rely on a UUID...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] chibenwa commented on pull request #584: JAMES-3544 Attachment refactoring (step 2)
Posted by GitBox <gi...@apache.org>.
chibenwa commented on pull request #584:
URL: https://github.com/apache/james-project/pull/584#issuecomment-900743594
Force pushed to solve conflict
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] chibenwa commented on a change in pull request #584: JAMES-3544 Attachment refactoring (step 2)
Posted by GitBox <gi...@apache.org>.
chibenwa commented on a change in pull request #584:
URL: https://github.com/apache/james-project/pull/584#discussion_r689238965
##########
File path: mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/migration/AttachmentMessageIdMigration.java
##########
@@ -0,0 +1,111 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.mailbox.cassandra.mail.migration;
+
+import static org.apache.james.util.ReactorUtils.DEFAULT_CONCURRENCY;
+
+import java.time.Clock;
+import java.time.Instant;
+import java.util.Optional;
+
+import javax.inject.Inject;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.james.backends.cassandra.migration.Migration;
+import org.apache.james.mailbox.cassandra.mail.CassandraAttachmentDAOV2;
+import org.apache.james.mailbox.cassandra.mail.CassandraAttachmentMessageIdDAO;
+import org.apache.james.mailbox.model.AttachmentId;
+import org.apache.james.mailbox.model.MessageId;
+import org.apache.james.task.Task;
+import org.apache.james.task.TaskExecutionDetails;
+import org.apache.james.task.TaskType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import reactor.core.publisher.Mono;
+
+public class AttachmentMessageIdMigration implements Migration {
+ static class AttachmentMessageIdMigrationTask implements Task {
+ private final AttachmentMessageIdMigration migration;
+
+ AttachmentMessageIdMigrationTask(AttachmentMessageIdMigration migration) {
+ this.migration = migration;
+ }
+
+ @Override
+ public Result run() throws InterruptedException {
+ return migration.runTask();
+ }
+
+ @Override
+ public TaskType type() {
+ return TYPE;
+ }
+
+ @Override
+ public Optional<TaskExecutionDetails.AdditionalInformation> details() {
+ return Optional.of(migration.getAdditionalInformation());
+ }
+ }
+
+ public static class AdditionalInformation implements TaskExecutionDetails.AdditionalInformation {
+ private final Instant timestamp;
+
+ public AdditionalInformation(Instant timestamp) {
+ this.timestamp = timestamp;
+ }
+
+ @Override
+ public Instant timestamp() {
+ return timestamp;
+ }
+ }
+
+ public static final Logger LOGGER = LoggerFactory.getLogger(AttachmentMessageIdMigration.class);
+ public static final TaskType TYPE = TaskType.of("cassandra-attachment-messageid-migration");
+ private final CassandraAttachmentMessageIdDAO attachmentMessageIdDAO;
+ private final CassandraAttachmentDAOV2 attachmentDAO;
+
+ @Inject
+ public AttachmentMessageIdMigration(CassandraAttachmentMessageIdDAO attachmentMessageIdDAO, CassandraAttachmentDAOV2 attachmentDAO) {
+ this.attachmentMessageIdDAO = attachmentMessageIdDAO;
+ this.attachmentDAO = attachmentDAO;
+ }
+
+ @Override
+ public void apply() {
+ attachmentMessageIdDAO.listAll()
+ .flatMap(this::migrate, DEFAULT_CONCURRENCY)
+ .blockLast();
+ }
+
+ private Mono<Void> migrate(Pair<AttachmentId, MessageId> entry) {
+ return attachmentDAO.insertMessageId(entry.getKey(), entry.getValue());
Review comment:
Yes it is possible that AttachmentId is already in attachmentDAO, in such cases we overwrite it.
And it is also perfectly fine performance-wise: better do extra writes that extra reads according to Cassandra phylosophy...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] chibenwa commented on pull request #584: JAMES-3544 Attachment refactoring (step 2)
Posted by GitBox <gi...@apache.org>.
chibenwa commented on pull request #584:
URL: https://github.com/apache/james-project/pull/584#issuecomment-900743594
Force pushed to solve conflict
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] chibenwa merged pull request #584: JAMES-3544 Attachment refactoring (step 2)
Posted by GitBox <gi...@apache.org>.
chibenwa merged pull request #584:
URL: https://github.com/apache/james-project/pull/584
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] vttranlina commented on a change in pull request #584: JAMES-3544 Attachment refactoring (step 2)
Posted by GitBox <gi...@apache.org>.
vttranlina commented on a change in pull request #584:
URL: https://github.com/apache/james-project/pull/584#discussion_r689232389
##########
File path: mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/migration/AttachmentMessageIdMigration.java
##########
@@ -0,0 +1,111 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.mailbox.cassandra.mail.migration;
+
+import static org.apache.james.util.ReactorUtils.DEFAULT_CONCURRENCY;
+
+import java.time.Clock;
+import java.time.Instant;
+import java.util.Optional;
+
+import javax.inject.Inject;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.james.backends.cassandra.migration.Migration;
+import org.apache.james.mailbox.cassandra.mail.CassandraAttachmentDAOV2;
+import org.apache.james.mailbox.cassandra.mail.CassandraAttachmentMessageIdDAO;
+import org.apache.james.mailbox.model.AttachmentId;
+import org.apache.james.mailbox.model.MessageId;
+import org.apache.james.task.Task;
+import org.apache.james.task.TaskExecutionDetails;
+import org.apache.james.task.TaskType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import reactor.core.publisher.Mono;
+
+public class AttachmentMessageIdMigration implements Migration {
+ static class AttachmentMessageIdMigrationTask implements Task {
+ private final AttachmentMessageIdMigration migration;
+
+ AttachmentMessageIdMigrationTask(AttachmentMessageIdMigration migration) {
+ this.migration = migration;
+ }
+
+ @Override
+ public Result run() throws InterruptedException {
+ return migration.runTask();
+ }
+
+ @Override
+ public TaskType type() {
+ return TYPE;
+ }
+
+ @Override
+ public Optional<TaskExecutionDetails.AdditionalInformation> details() {
+ return Optional.of(migration.getAdditionalInformation());
+ }
+ }
+
+ public static class AdditionalInformation implements TaskExecutionDetails.AdditionalInformation {
+ private final Instant timestamp;
+
+ public AdditionalInformation(Instant timestamp) {
+ this.timestamp = timestamp;
+ }
+
+ @Override
+ public Instant timestamp() {
+ return timestamp;
+ }
+ }
+
+ public static final Logger LOGGER = LoggerFactory.getLogger(AttachmentMessageIdMigration.class);
+ public static final TaskType TYPE = TaskType.of("cassandra-attachment-messageid-migration");
+ private final CassandraAttachmentMessageIdDAO attachmentMessageIdDAO;
+ private final CassandraAttachmentDAOV2 attachmentDAO;
+
+ @Inject
+ public AttachmentMessageIdMigration(CassandraAttachmentMessageIdDAO attachmentMessageIdDAO, CassandraAttachmentDAOV2 attachmentDAO) {
+ this.attachmentMessageIdDAO = attachmentMessageIdDAO;
+ this.attachmentDAO = attachmentDAO;
+ }
+
+ @Override
+ public void apply() {
+ attachmentMessageIdDAO.listAll()
+ .flatMap(this::migrate, DEFAULT_CONCURRENCY)
+ .blockLast();
+ }
+
+ private Mono<Void> migrate(Pair<AttachmentId, MessageId> entry) {
+ return attachmentDAO.insertMessageId(entry.getKey(), entry.getValue());
Review comment:
Is it possible that `AttachmentId` is ready in `attachmentDAO` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] chibenwa commented on a change in pull request #584: JAMES-3544 Attachment refactoring (step 2)
Posted by GitBox <gi...@apache.org>.
chibenwa commented on a change in pull request #584:
URL: https://github.com/apache/james-project/pull/584#discussion_r685670649
##########
File path: server/protocols/jmap-draft-integration-testing/memory-jmap-draft-integration-testing/src/test/java/org/apache/james/jmap/memory/MemorySpamAssassinContractTest.java
##########
@@ -1,82 +0,0 @@
-/****************************************************************
Review comment:
Oups
##########
File path: server/protocols/jmap-draft-integration-testing/memory-jmap-draft-integration-testing/src/test/java/org/apache/james/jmap/memory/MemoryVacationRelayIntegrationTest.java
##########
@@ -1,47 +0,0 @@
-/****************************************************************
Review comment:
Oups
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org