You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@james.apache.org by GitBox <gi...@apache.org> on 2022/05/04 21:28:26 UTC
[GitHub] [james-project] mbaechler opened a new pull request, #987: James 3763 BlobStore backed MailRepository
mbaechler opened a new pull request, #987:
URL: https://github.com/apache/james-project/pull/987
This branch includes commits from https://github.com/apache/james-project/pull/985 and https://github.com/apache/james-project/pull/986 and add the blobstore implementation on top.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] mbaechler commented on pull request #987: James 3763 BlobStore backed MailRepository
Posted by GitBox <gi...@apache.org>.
mbaechler commented on PR #987:
URL: https://github.com/apache/james-project/pull/987#issuecomment-1118552020
> Any chance to get this plugged into the distributed server? Documented?
We are not going to deploy this distribution for now, we'll probably only integrate it to the smtp distribution we want to deploy.
It will need some documentation efforts that we'll handle.
Feel free to reuse this impl in other contexts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] chibenwa commented on pull request #987: James 3763 BlobStore backed MailRepository
Posted by GitBox <gi...@apache.org>.
chibenwa commented on PR #987:
URL: https://github.com/apache/james-project/pull/987#issuecomment-1118641709
> we'll probably only integrate it to the smtp distribution we want to deploy.
Looks great to me ;-)
I think this should be as easy as adding a maven dependency, in theory...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] jeantil commented on pull request #987: James 3763 BlobStore backed MailRepository
Posted by GitBox <gi...@apache.org>.
jeantil commented on PR #987:
URL: https://github.com/apache/james-project/pull/987#issuecomment-1118658923
Won't it be easier to merge the other 2 PRs then rebase this one ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] chibenwa commented on pull request #987: James 3763 BlobStore backed MailRepository
Posted by GitBox <gi...@apache.org>.
chibenwa commented on PR #987:
URL: https://github.com/apache/james-project/pull/987#issuecomment-1118057977
You are missing ticket numbers on commits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] chibenwa commented on pull request #987: James 3763 BlobStore backed MailRepository
Posted by GitBox <gi...@apache.org>.
chibenwa commented on PR #987:
URL: https://github.com/apache/james-project/pull/987#issuecomment-1118655435
```
23:41:27,611 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-checkstyle-plugin:3.1.0:check (check-style) on project blob-common: You have 2 Checkstyle violations. -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-checkstyle-plugin:3.1.0:check (check-style) on project blob-common: You have 2 Checkstyle violations.
```
Cherry pick of the checkstyle fixups?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] mbaechler merged pull request #987: James 3763 BlobStore backed MailRepository
Posted by GitBox <gi...@apache.org>.
mbaechler merged PR #987:
URL: https://github.com/apache/james-project/pull/987
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] chibenwa commented on a diff in pull request #987: James 3763 BlobStore backed MailRepository
Posted by GitBox <gi...@apache.org>.
chibenwa commented on code in PR #987:
URL: https://github.com/apache/james-project/pull/987#discussion_r865485941
##########
server/mailrepository/mailrepository-blob/src/main/scala/org/apache/james/mailrepository/blob/BlobMailRepository.scala:
##########
@@ -0,0 +1,186 @@
+package org.apache.james.mailrepository.blob
+
+import java.{lang, util}
+import java.util.Date
+import java.util.stream.Stream
+
+import javax.mail.MessagingException
+import javax.mail.internet.MimeMessage
+
+import scala.jdk.CollectionConverters.IterableHasAsJava
+
+import org.apache.commons.lang3.tuple.Pair
+import org.apache.james.blob.api.BlobStore.StoragePolicy.SIZE_BASED
+import org.apache.james.blob.api.Store.Impl
+import org.apache.james.blob.api.{BlobId, BlobPartsId, BlobStore, BlobStoreDAO, BlobType, BucketName, Store}
+import org.apache.james.blob.mail.MimeMessagePartsId
+import org.apache.james.mailrepository.api.{MailKey, MailRepository}
+import org.apache.mailet.{Attribute, AttributeName, AttributeValue, Mail, PerRecipientHeaders}
+
+import com.google.common.collect.ImmutableMap
+
+import play.api.libs.json.{Format, Json}
+import reactor.core.publisher.{Flux, Mono}
+import scala.jdk.StreamConverters._
+
+import org.apache.james.core.{MailAddress, MaybeSender}
+import org.apache.james.server.blob.deduplication.BlobStoreFactory
+import org.apache.james.server.core.MailImpl
+
+
+private[blob] object serializers {
+ implicit val headerFormat: Format[Header] = Json.format[Header]
+ implicit val mailMetadataFormat: Format[MailMetadata] = Json.format[MailMetadata]
+}
+
+object BlobMailRepository {
+ private[blob] object MailPartsId {
+ private[blob] val METADATA_BLOB_TYPE = new BlobType("mailMetadata", SIZE_BASED)
+
+ class Factory extends BlobPartsId.Factory[BlobMailRepository.MailPartsId] {
+ override def generate(map: util.Map[BlobType, BlobId]) = {
+ require(map.containsKey(METADATA_BLOB_TYPE), "Expecting 'mailMetadata' blobId to be specified")
+ require(map.size == 1, "blobId other than 'mailMetadata' are not supported")
+ new BlobMailRepository.MailPartsId(map.get(METADATA_BLOB_TYPE))
+ }
+ }
+ }
+
+ private[blob] case class MailPartsId private[blob](metadataBlobId: BlobId) extends BlobPartsId {
+ override def asMap = ImmutableMap.of(MailPartsId.METADATA_BLOB_TYPE, metadataBlobId)
+
+ def toMailKey: MailKey = new MailKey(metadataBlobId.asString())
+ }
+
+ private[blob] class MailEncoder private[blob](blobStoreDAO: BlobStoreDAO, blobIdFactory: BlobId.Factory)
+ extends Impl.Encoder[(Mail, MimeMessagePartsId)] {
+
+ import serializers._
+
+ override def encode(mailAndPartsId: (Mail, MimeMessagePartsId)): Stream[Pair[BlobType, Impl.ValueToSave]] = {
+ val (mail, partsIds) = mailAndPartsId
+ val mailMetadata = MailMetadata.of(mail, partsIds)
+ val payload = Json.stringify(Json.toJson(mailMetadata))
+ val mailKey = MailKey.forMail(mail)
+ val blobId=blobIdFactory.from(mailKey.asString())
+ val save: Impl.ValueToSave = (bucketName, _) =>
+ Mono.from(blobStoreDAO.save(bucketName, blobId, payload)).`then`(Mono.just(blobId))
+
+
+ LazyList(Pair.of(MailPartsId.METADATA_BLOB_TYPE, save)).asJavaSeqStream
+ }
+ }
+
+ private[blob] class MailDecoder private[blob](blobIdFactory: BlobId.Factory)
+ extends Impl.Decoder[(Mail, MimeMessagePartsId)] {
+
+ private def readMail(mailMetadata: MailMetadata): Mail = {
+ val builder = MailImpl.builder
+ .name(mailMetadata.name)
+ .sender(mailMetadata.sender.map(MaybeSender.getMailSender).getOrElse(MaybeSender.nullSender))
+ .addRecipients(mailMetadata.recipients.map(new MailAddress(_)).asJavaCollection)
+ .remoteAddr(mailMetadata.remoteAddr)
+ .remoteHost(mailMetadata.remoteHost)
+
+ mailMetadata.state.foreach(builder.state)
+ mailMetadata.errorMessage.foreach(builder.errorMessage)
+
+ mailMetadata.lastUpdated.map(Date.from).foreach(builder.lastUpdated)
+
+ mailMetadata.attributes.foreach { case (name, value) => builder.addAttribute(new Attribute(AttributeName.of(name), AttributeValue.fromJsonString(value))) }
+
+ builder.addAllHeadersForRecipients(retrievePerRecipientHeaders(mailMetadata.perRecipientHeaders))
+
+ builder.build
+ }
+
+
+ private def retrievePerRecipientHeaders(perRecipientHeaders: Map[String, Iterable[Header]]): PerRecipientHeaders = {
+ val result = new PerRecipientHeaders()
+ perRecipientHeaders.foreach { case (key, value) =>
+ value.foreach(headers => {
+ headers.values.foreach(header => {
+ val builder = PerRecipientHeaders.Header.builder().name(headers.key).value(header)
+ result.addHeaderForRecipient(builder, new MailAddress(key))
+ })
+ })
+ }
+ result
+ }
+
+
+ import serializers._
+
+ override def decode(streams: util.Map[BlobType, Store.CloseableByteSource]): (Mail, MimeMessagePartsId) = {
+ val source = streams.get(MailPartsId.METADATA_BLOB_TYPE)
+ val value = Json.fromJson[MailMetadata](Json.parse(source.openStream())).get
+ (readMail(value), value.mimePartsId(blobIdFactory))
+ }
+ }
+}
+
+class BlobMailRepository(val blobStore: BlobStoreDAO,
+ val blobIdFactory: BlobId.Factory,
+ val mimeMessageStore: Store[MimeMessage, MimeMessagePartsId]
+ ) extends MailRepository {
+ private val bucketName: BucketName = BucketName.of("mailMetadata")
+ private val mimeMessageBucketName: BucketName = BucketName.of("mimeMessageData")
+ import BlobMailRepository._
+
+ @throws[MessagingException]
+ override def store(mc: Mail): MailKey = {
+ mimeMessageStore.save(mc.getMessage)
+ .flatMap(mimePartsId => mailMetadataStore.save((mc, mimePartsId)))
+ .map(mailPartsId => mailPartsId.toMailKey)
+ .block()
+ }
+
+ @throws[MessagingException]
+ override def size: Long = Flux.from(blobStore.listBlobs(bucketName)).count().block()
Review Comment:
There might be a higher level S3 method for getting bucket size, that might be more efficient? Would we benefit from adding it to the blobStore API ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] mbaechler commented on a diff in pull request #987: James 3763 BlobStore backed MailRepository
Posted by GitBox <gi...@apache.org>.
mbaechler commented on code in PR #987:
URL: https://github.com/apache/james-project/pull/987#discussion_r865904360
##########
server/mailrepository/mailrepository-blob/src/main/scala/org/apache/james/mailrepository/blob/BlobMailRepository.scala:
##########
@@ -0,0 +1,186 @@
+package org.apache.james.mailrepository.blob
+
+import java.{lang, util}
+import java.util.Date
+import java.util.stream.Stream
+
+import javax.mail.MessagingException
+import javax.mail.internet.MimeMessage
+
+import scala.jdk.CollectionConverters.IterableHasAsJava
+
+import org.apache.commons.lang3.tuple.Pair
+import org.apache.james.blob.api.BlobStore.StoragePolicy.SIZE_BASED
+import org.apache.james.blob.api.Store.Impl
+import org.apache.james.blob.api.{BlobId, BlobPartsId, BlobStore, BlobStoreDAO, BlobType, BucketName, Store}
+import org.apache.james.blob.mail.MimeMessagePartsId
+import org.apache.james.mailrepository.api.{MailKey, MailRepository}
+import org.apache.mailet.{Attribute, AttributeName, AttributeValue, Mail, PerRecipientHeaders}
+
+import com.google.common.collect.ImmutableMap
+
+import play.api.libs.json.{Format, Json}
+import reactor.core.publisher.{Flux, Mono}
+import scala.jdk.StreamConverters._
+
+import org.apache.james.core.{MailAddress, MaybeSender}
+import org.apache.james.server.blob.deduplication.BlobStoreFactory
+import org.apache.james.server.core.MailImpl
+
+
+private[blob] object serializers {
+ implicit val headerFormat: Format[Header] = Json.format[Header]
+ implicit val mailMetadataFormat: Format[MailMetadata] = Json.format[MailMetadata]
+}
+
+object BlobMailRepository {
+ private[blob] object MailPartsId {
+ private[blob] val METADATA_BLOB_TYPE = new BlobType("mailMetadata", SIZE_BASED)
+
+ class Factory extends BlobPartsId.Factory[BlobMailRepository.MailPartsId] {
+ override def generate(map: util.Map[BlobType, BlobId]) = {
+ require(map.containsKey(METADATA_BLOB_TYPE), "Expecting 'mailMetadata' blobId to be specified")
+ require(map.size == 1, "blobId other than 'mailMetadata' are not supported")
+ new BlobMailRepository.MailPartsId(map.get(METADATA_BLOB_TYPE))
+ }
+ }
+ }
+
+ private[blob] case class MailPartsId private[blob](metadataBlobId: BlobId) extends BlobPartsId {
+ override def asMap = ImmutableMap.of(MailPartsId.METADATA_BLOB_TYPE, metadataBlobId)
+
+ def toMailKey: MailKey = new MailKey(metadataBlobId.asString())
+ }
+
+ private[blob] class MailEncoder private[blob](blobStoreDAO: BlobStoreDAO, blobIdFactory: BlobId.Factory)
+ extends Impl.Encoder[(Mail, MimeMessagePartsId)] {
+
+ import serializers._
+
+ override def encode(mailAndPartsId: (Mail, MimeMessagePartsId)): Stream[Pair[BlobType, Impl.ValueToSave]] = {
+ val (mail, partsIds) = mailAndPartsId
+ val mailMetadata = MailMetadata.of(mail, partsIds)
+ val payload = Json.stringify(Json.toJson(mailMetadata))
+ val mailKey = MailKey.forMail(mail)
+ val blobId=blobIdFactory.from(mailKey.asString())
+ val save: Impl.ValueToSave = (bucketName, _) =>
+ Mono.from(blobStoreDAO.save(bucketName, blobId, payload)).`then`(Mono.just(blobId))
+
+
+ LazyList(Pair.of(MailPartsId.METADATA_BLOB_TYPE, save)).asJavaSeqStream
+ }
+ }
+
+ private[blob] class MailDecoder private[blob](blobIdFactory: BlobId.Factory)
+ extends Impl.Decoder[(Mail, MimeMessagePartsId)] {
+
+ private def readMail(mailMetadata: MailMetadata): Mail = {
+ val builder = MailImpl.builder
+ .name(mailMetadata.name)
+ .sender(mailMetadata.sender.map(MaybeSender.getMailSender).getOrElse(MaybeSender.nullSender))
+ .addRecipients(mailMetadata.recipients.map(new MailAddress(_)).asJavaCollection)
+ .remoteAddr(mailMetadata.remoteAddr)
+ .remoteHost(mailMetadata.remoteHost)
+
+ mailMetadata.state.foreach(builder.state)
+ mailMetadata.errorMessage.foreach(builder.errorMessage)
+
+ mailMetadata.lastUpdated.map(Date.from).foreach(builder.lastUpdated)
+
+ mailMetadata.attributes.foreach { case (name, value) => builder.addAttribute(new Attribute(AttributeName.of(name), AttributeValue.fromJsonString(value))) }
+
+ builder.addAllHeadersForRecipients(retrievePerRecipientHeaders(mailMetadata.perRecipientHeaders))
+
+ builder.build
+ }
+
+
+ private def retrievePerRecipientHeaders(perRecipientHeaders: Map[String, Iterable[Header]]): PerRecipientHeaders = {
+ val result = new PerRecipientHeaders()
+ perRecipientHeaders.foreach { case (key, value) =>
+ value.foreach(headers => {
+ headers.values.foreach(header => {
+ val builder = PerRecipientHeaders.Header.builder().name(headers.key).value(header)
+ result.addHeaderForRecipient(builder, new MailAddress(key))
+ })
+ })
+ }
+ result
+ }
+
+
+ import serializers._
+
+ override def decode(streams: util.Map[BlobType, Store.CloseableByteSource]): (Mail, MimeMessagePartsId) = {
+ val source = streams.get(MailPartsId.METADATA_BLOB_TYPE)
+ val value = Json.fromJson[MailMetadata](Json.parse(source.openStream())).get
+ (readMail(value), value.mimePartsId(blobIdFactory))
+ }
+ }
+}
+
+class BlobMailRepository(val blobStore: BlobStoreDAO,
+ val blobIdFactory: BlobId.Factory,
+ val mimeMessageStore: Store[MimeMessage, MimeMessagePartsId]
+ ) extends MailRepository {
+ private val bucketName: BucketName = BucketName.of("mailMetadata")
+ private val mimeMessageBucketName: BucketName = BucketName.of("mimeMessageData")
+ import BlobMailRepository._
+
+ @throws[MessagingException]
+ override def store(mc: Mail): MailKey = {
+ mimeMessageStore.save(mc.getMessage)
+ .flatMap(mimePartsId => mailMetadataStore.save((mc, mimePartsId)))
+ .map(mailPartsId => mailPartsId.toMailKey)
+ .block()
+ }
+
+ @throws[MessagingException]
+ override def size: Long = Flux.from(blobStore.listBlobs(bucketName)).count().block()
Review Comment:
I could not find one
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] chibenwa commented on a diff in pull request #987: James 3763 BlobStore backed MailRepository
Posted by GitBox <gi...@apache.org>.
chibenwa commented on code in PR #987:
URL: https://github.com/apache/james-project/pull/987#discussion_r868906814
##########
server/mailrepository/mailrepository-blob/src/main/scala/org/apache/james/mailrepository/blob/BlobMailRepository.scala:
##########
@@ -0,0 +1,186 @@
+package org.apache.james.mailrepository.blob
Review Comment:
License
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org