You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@james.apache.org by GitBox <gi...@apache.org> on 2022/07/26 10:17:40 UTC
[GitHub] [james-project] quantranhong1999 opened a new pull request, #1092: JAMES-3775 Write a webadmin route to feed spam to RSpamD
quantranhong1999 opened a new pull request, #1092:
URL: https://github.com/apache/james-project/pull/1092
TODO task unit tests, webadmin route , webadmin unit tests, and a bit of documentation the route.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] chibenwa commented on a diff in pull request #1092: JAMES-3775 Write a webadmin route to feed spam to RSpamD
Posted by GitBox <gi...@apache.org>.
chibenwa commented on code in PR #1092:
URL: https://github.com/apache/james-project/pull/1092#discussion_r934208167
##########
third-party/rspamd/src/main/java/org/apache/james/rspamd/task/FeedSpamToRSpamDTask.java:
##########
@@ -0,0 +1,389 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.rspamd.task;
+
+import java.time.Clock;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Date;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.concurrent.atomic.AtomicLong;
+
+import org.apache.james.mailbox.MailboxManager;
+import org.apache.james.mailbox.model.MessageResult;
+import org.apache.james.rspamd.client.RSpamDHttpClient;
+import org.apache.james.task.Task;
+import org.apache.james.task.TaskExecutionDetails;
+import org.apache.james.task.TaskType;
+import org.apache.james.user.api.UsersRepository;
+import org.apache.james.user.api.UsersRepositoryException;
+import org.apache.james.util.ReactorUtils;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.github.fge.lambdas.Throwing;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.MoreObjects;
+
+import reactor.core.publisher.Mono;
+
+public class FeedSpamToRSpamDTask implements Task {
+ public static final String SPAM_MAILBOX_NAME = "Spam";
+ public static final TaskType TASK_TYPE = TaskType.of("FeedSpamToRSpamDTask");
+
+ public static class RunningOptions {
+ public static final Optional<Integer> DEFAULT_PERIOD = Optional.empty();
+ public static final int DEFAULT_MESSAGES_PER_SECOND = 10;
+ public static final double DEFAULT_SAMPLING_PROBABILITY = 1;
+ public static final RunningOptions DEFAULT = new RunningOptions(DEFAULT_PERIOD, DEFAULT_MESSAGES_PER_SECOND,
+ DEFAULT_SAMPLING_PROBABILITY);
+
+ private final Optional<Integer> period;
+ private final int messagesPerSecond;
+ private final double samplingProbability;
+
+ public RunningOptions(@JsonProperty("period") Optional<Integer> periodInSecond,
+ @JsonProperty("messagesPerSecond") int messagesPerSecond,
+ @JsonProperty("samplingProbability") double samplingProbability) {
+ this.period = periodInSecond;
+ this.messagesPerSecond = messagesPerSecond;
+ this.samplingProbability = samplingProbability;
+ }
+
+ public Optional<Integer> getPeriod() {
+ return period;
+ }
+
+ public int getMessagesPerSecond() {
+ return messagesPerSecond;
+ }
+
+ public double getSamplingProbability() {
+ return samplingProbability;
+ }
+ }
+
+ public static class AdditionalInformation implements TaskExecutionDetails.AdditionalInformation {
+
+ private static AdditionalInformation from(Context context) {
+ Context.Snapshot snapshot = context.snapshot();
+ return new AdditionalInformation(
+ Clock.systemUTC().instant(),
+ snapshot.getSpamMessageCount(),
+ snapshot.getReportedSpamMessageCount(),
+ snapshot.getErrorCount(),
+ snapshot.getMessagesPerSecond(),
+ snapshot.getPeriod(),
+ snapshot.getSamplingProbability());
+ }
+
+ private final Instant timestamp;
+ private final long spamMessageCount;
+ private final long reportedSpamMessageCount;
+ private final long errorCount;
+ private final int messagesPerSecond;
+ private final Optional<Integer> period;
+ private final double samplingProbability;
+
+ public AdditionalInformation(Instant timestamp, long spamMessageCount, long reportedSpamMessageCount, long errorCount, int messagesPerSecond, Optional<Integer> period, double samplingProbability) {
+ this.timestamp = timestamp;
+ this.spamMessageCount = spamMessageCount;
+ this.reportedSpamMessageCount = reportedSpamMessageCount;
+ this.errorCount = errorCount;
+ this.messagesPerSecond = messagesPerSecond;
+ this.period = period;
+ this.samplingProbability = samplingProbability;
+ }
+
+ public long getSpamMessageCount() {
+ return spamMessageCount;
+ }
+
+ public long getReportedSpamMessageCount() {
+ return reportedSpamMessageCount;
+ }
+
+ public long getErrorCount() {
+ return errorCount;
+ }
+
+ public int getMessagesPerSecond() {
+ return messagesPerSecond;
+ }
+
+ public Optional<Integer> getPeriod() {
+ return period;
+ }
+
+ public double getSamplingProbability() {
+ return samplingProbability;
+ }
+
+ @Override
+ public Instant timestamp() {
+ return timestamp;
+ }
+ }
+
+ public static class Context {
+
+ public static class Snapshot {
+
+ public static Builder builder() {
+ return new Builder();
+ }
+
+ static class Builder {
+ private Optional<Long> spamMessageCount;
+ private Optional<Long> reportedSpamMessageCount;
+ private Optional<Long> errorCount;
+ private Optional<Integer> messagesPerSecond;
+ private Optional<Integer> period;
+ private Optional<Double> samplingProbability;
+
+ Builder() {
+ spamMessageCount = Optional.empty();
+ reportedSpamMessageCount = Optional.empty();
+ errorCount = Optional.empty();
+ messagesPerSecond = Optional.empty();
+ period = Optional.empty();
+ samplingProbability = Optional.empty();
+ }
+
+ public Snapshot build() {
+ return new Snapshot(
+ spamMessageCount.orElse(0L),
+ reportedSpamMessageCount.orElse(0L),
+ errorCount.orElse(0L),
+ messagesPerSecond.orElse(0),
+ period,
+ samplingProbability.orElse(1D));
+ }
+
+ public Builder spamMessageCount(long spamMessageCount) {
+ this.spamMessageCount = Optional.of(spamMessageCount);
+ return this;
+ }
+
+ public Builder reportedSpamMessageCount(long reportedSpamMessageCount) {
+ this.reportedSpamMessageCount = Optional.of(reportedSpamMessageCount);
+ return this;
+ }
+
+ public Builder errorCount(long errorCount) {
+ this.errorCount = Optional.of(errorCount);
+ return this;
+ }
+
+ public Builder messagesPerSecond(int messagesPerSecond) {
+ this.messagesPerSecond = Optional.of(messagesPerSecond);
+ return this;
+ }
+
+ public Builder period(Optional<Integer> period) {
+ this.period = period;
+ return this;
+ }
+
+ public Builder samplingProbability(double samplingProbability) {
+ this.samplingProbability = Optional.of(samplingProbability);
+ return this;
+ }
+ }
+
+ private final long spamMessageCount;
+ private final long reportedSpamMessageCount;
+ private final long errorCount;
+ private final int messagesPerSecond;
+ private final Optional<Integer> period;
+ private final double samplingProbability;
+
+ public Snapshot(long spamMessageCount, long reportedSpamMessageCount, long errorCount, int messagesPerSecond, Optional<Integer> period,
+ double samplingProbability) {
+ this.spamMessageCount = spamMessageCount;
+ this.reportedSpamMessageCount = reportedSpamMessageCount;
+ this.errorCount = errorCount;
+ this.messagesPerSecond = messagesPerSecond;
+ this.period = period;
+ this.samplingProbability = samplingProbability;
+ }
+
+ public long getSpamMessageCount() {
+ return spamMessageCount;
+ }
+
+ public long getReportedSpamMessageCount() {
+ return reportedSpamMessageCount;
+ }
+
+ public long getErrorCount() {
+ return errorCount;
+ }
+
+ public int getMessagesPerSecond() {
+ return messagesPerSecond;
+ }
+
+ public Optional<Integer> getPeriod() {
+ return period;
+ }
+
+ public double getSamplingProbability() {
+ return samplingProbability;
+ }
+
+ @Override
+ public final boolean equals(Object o) {
+ if (o instanceof Snapshot) {
+ Snapshot snapshot = (Snapshot) o;
+
+ return Objects.equals(this.spamMessageCount, snapshot.spamMessageCount)
+ && Objects.equals(this.reportedSpamMessageCount, snapshot.reportedSpamMessageCount)
+ && Objects.equals(this.errorCount, snapshot.errorCount)
+ && Objects.equals(this.messagesPerSecond, snapshot.messagesPerSecond)
+ && Objects.equals(this.samplingProbability, snapshot.samplingProbability)
+ && Objects.equals(this.period, snapshot.period);
+ }
+ return false;
+ }
+
+ @Override
+ public final int hashCode() {
+ return Objects.hash(spamMessageCount, reportedSpamMessageCount, errorCount, messagesPerSecond, period, samplingProbability);
+ }
+
+ @Override
+ public String toString() {
+ return MoreObjects.toStringHelper(this)
+ .add("spamMessageCount", spamMessageCount)
+ .add("reportedSpamMessageCount", reportedSpamMessageCount)
+ .add("errorCount", errorCount)
+ .add("messagesPerSecond", messagesPerSecond)
+ .add("period", period)
+ .add("samplingProbability", samplingProbability)
+ .toString();
+ }
+ }
+
+ private final AtomicLong spamMessageCount;
+ private final AtomicLong reportedSpamMessageCount;
+ private final AtomicLong errorCount;
+ private final Integer messagesPerSecond;
+ private final Optional<Integer> period;
+ private final Double samplingProbability;
+
+ public Context(RunningOptions runningOptions) {
+ this.spamMessageCount = new AtomicLong();
+ this.reportedSpamMessageCount = new AtomicLong();
+ this.errorCount = new AtomicLong();
+ this.messagesPerSecond = runningOptions.messagesPerSecond;
+ this.period = runningOptions.period;
+ this.samplingProbability = runningOptions.samplingProbability;
+ }
+
+ public void incrementSpamMessageCount() {
+ spamMessageCount.incrementAndGet();
+ }
+
+ public void incrementReportedSpamMessageCount(int count) {
+ reportedSpamMessageCount.addAndGet(count);
+ }
+
+ public void incrementErrorCount() {
+ errorCount.incrementAndGet();
+ }
+
+ public Snapshot snapshot() {
+ return Snapshot.builder()
+ .spamMessageCount(spamMessageCount.get())
+ .reportedSpamMessageCount(reportedSpamMessageCount.get())
+ .errorCount(errorCount.get())
+ .messagesPerSecond(messagesPerSecond)
+ .period(period)
+ .samplingProbability(samplingProbability)
+ .build();
+ }
+ }
+
+ private final GetMailboxMessagesService messagesService;
+ private final RSpamDHttpClient rSpamDHttpClient;
+ private final RunningOptions runningOptions;
+ private final Context context;
+ private final Clock clock;
+
+ public FeedSpamToRSpamDTask(MailboxManager mailboxManager, UsersRepository usersRepository, RSpamDHttpClient rSpamDHttpClient,
+ RunningOptions runningOptions, Clock clock) {
+ this.runningOptions = runningOptions;
+ this.messagesService = new GetMailboxMessagesService(mailboxManager, usersRepository);
+ this.rSpamDHttpClient = rSpamDHttpClient;
+ this.context = new Context(runningOptions);
+ this.clock = clock;
+ }
+
+ @Override
+ public Result run() {
+ Optional<Date> afterDate = runningOptions.period.map(periodInSecond -> Date.from(clock.instant().minusSeconds(periodInSecond)));
+ try {
+ return messagesService.getMailboxMessagesOfAllUser(SPAM_MAILBOX_NAME)
+ .doOnNext(messageResult -> context.incrementSpamMessageCount())
+ .filter(messageResult -> afterDate.map(date -> messageResult.getInternalDate().after(date)).orElse(true))
+ .filter(any -> randomBooleanWithProbability(runningOptions.samplingProbability))
+ .transform(ReactorUtils.<MessageResult, Task.Result>throttle()
+ .elements(runningOptions.messagesPerSecond)
+ .per(Duration.ofSeconds(1))
+ .forOperation(messageResult1 -> Mono.just(messageResult1)
+ .flatMap(Throwing.function(messageResult -> rSpamDHttpClient.reportAsSpam(messageResult.getHeaders().getInputStream(), messageResult.getBody().getInputStream())))
Review Comment:
Yes but we could wrap this using `Throwing.function` in the right place, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] chibenwa commented on a diff in pull request #1092: JAMES-3775 Write a webadmin route to feed spam to RSpamD
Posted by GitBox <gi...@apache.org>.
chibenwa commented on code in PR #1092:
URL: https://github.com/apache/james-project/pull/1092#discussion_r934105485
##########
third-party/rspamd/README.md:
##########
@@ -0,0 +1,40 @@
+## Additional webadmin endpoints
+
+### Report spam messages to RSpamD
+One can use this route to schedule a task that reports spam messages to RSpamD for its spam classify learning.
+
+```bash
+curl -XPOST 'http://ip:port/rspamd?action=reportSpam
+```
+
+This endpoint has the following param:
+- `action` (required): need to be `reportSpam`
+- `messagesPerSecond` (optional): Concurrent learns performed for RSpamD, default to 10
+- `period` (optional): duration (in seconds), only messages between `now` and `now - duration` are reported. By default,
+all messages are reported.
Review Comment:
Please allow to provide unit. EG:
`curl -XPOST 'http://ip:port/rspamd?action=reportSpam&period=1d`
This makes it way more friendly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] quantranhong1999 commented on pull request #1092: JAMES-3775 Write a webadmin route to feed spam to RSpamD
Posted by GitBox <gi...@apache.org>.
quantranhong1999 commented on PR #1092:
URL: https://github.com/apache/james-project/pull/1092#issuecomment-1201036719
You just need to read the latest commit (adapt Benoit review), the rest is the same.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] quantranhong1999 commented on a diff in pull request #1092: JAMES-3775 Write a webadmin route to feed spam to RSpamD
Posted by GitBox <gi...@apache.org>.
quantranhong1999 commented on code in PR #1092:
URL: https://github.com/apache/james-project/pull/1092#discussion_r935080092
##########
third-party/rspamd/src/main/java/org/apache/james/rspamd/route/FeedMessageRoute.java:
##########
@@ -0,0 +1,142 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.rspamd.route;
+
+import static org.apache.james.rspamd.task.FeedSpamToRSpamDTask.RunningOptions.DEFAULT_MESSAGES_PER_SECOND;
+import static org.apache.james.rspamd.task.FeedSpamToRSpamDTask.RunningOptions.DEFAULT_SAMPLING_PROBABILITY;
+
+import java.time.Clock;
+import java.time.temporal.ChronoUnit;
+import java.util.Optional;
+import java.util.function.Predicate;
+
+import javax.inject.Inject;
+
+import org.apache.james.mailbox.MailboxManager;
+import org.apache.james.mailbox.MessageIdManager;
+import org.apache.james.mailbox.store.mail.MessageMapper;
+import org.apache.james.rspamd.client.RSpamDHttpClient;
+import org.apache.james.rspamd.task.FeedSpamToRSpamDTask;
+import org.apache.james.task.Task;
+import org.apache.james.task.TaskManager;
+import org.apache.james.user.api.UsersRepository;
+import org.apache.james.util.DurationParser;
+import org.apache.james.webadmin.Routes;
+import org.apache.james.webadmin.tasks.TaskFromRequest;
+import org.apache.james.webadmin.utils.JsonTransformer;
+
+import com.google.common.base.Preconditions;
+
+import spark.Request;
+import spark.Service;
+
+public class FeedMessageRoute implements Routes {
+ public static final String BASE_PATH = "/rspamd";
+
+ private final TaskManager taskManager;
+ private final MailboxManager mailboxManager;
+ private final MessageIdManager messageIdManager;
+ private final MessageMapper messageMapper;
+ private final UsersRepository usersRepository;
+ private final RSpamDHttpClient rSpamDHttpClient;
+ private final JsonTransformer jsonTransformer;
+ private final Clock clock;
+
+ @Inject
+ public FeedMessageRoute(TaskManager taskManager, MailboxManager mailboxManager, UsersRepository usersRepository, RSpamDHttpClient rSpamDHttpClient,
+ JsonTransformer jsonTransformer, Clock clock, MessageIdManager messageIdManager, MessageMapper messageMapper) {
Review Comment:
Good remark. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] chibenwa commented on a diff in pull request #1092: JAMES-3775 Write a webadmin route to feed spam to RSpamD
Posted by GitBox <gi...@apache.org>.
chibenwa commented on code in PR #1092:
URL: https://github.com/apache/james-project/pull/1092#discussion_r934106148
##########
third-party/rspamd/README.md:
##########
@@ -0,0 +1,40 @@
+## Additional webadmin endpoints
+
+### Report spam messages to RSpamD
+One can use this route to schedule a task that reports spam messages to RSpamD for its spam classify learning.
+
+```bash
+curl -XPOST 'http://ip:port/rspamd?action=reportSpam
+```
+
+This endpoint has the following param:
+- `action` (required): need to be `reportSpam`
+- `messagesPerSecond` (optional): Concurrent learns performed for RSpamD, default to 10
+- `period` (optional): duration (in seconds), only messages between `now` and `now - duration` are reported. By default,
+all messages are reported.
+- `samplingProbability` (optional): float between 0 and 1, represent the chance to report each given message to RSpamD.
+By default, all messages are reported.
+
+Will return the task id. E.g:
+```
+{
+ "taskId": "70c12761-ab86-4321-bb6f-fde99e2f74b0"
+}
+```
+
+The scheduled task will have the following type `FeedSpamToRSpamDTask` and the following additionalInformation:
+
+```json
+{
+ "errorCount": 1,
+ "reportedSpamMessageCount": 2,
+ "runningOptions": {
+ "messagesPerSecond": 10,
+ "period": 3600,
+ "samplingProbability": 1.0
+ },
+ "spamMessageCount": 4,
+ "timestamp": "2007-12-03T10:15:30Z",
+ "type": "FeedSpamToRSpamDTask"
+}
+```
Review Comment:
What are possible response codes?
##########
third-party/rspamd/pom.xml:
##########
@@ -33,22 +33,68 @@
<description>RSpamD Java client (HTTP) and testing utilities</description>
<dependencies>
+ <dependency>
+ <groupId>${james.groupId}</groupId>
+ <artifactId>apache-james-mailbox-api</artifactId>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>${james.groupId}</groupId>
+ <artifactId>apache-james-mailbox-memory</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>${james.groupId}</groupId>
+ <artifactId>apache-james-mailbox-memory</artifactId>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
<dependency>
<groupId>${james.groupId}</groupId>
<artifactId>apache-mailet-api</artifactId>
</dependency>
+ <dependency>
+ <groupId>${james.groupId}</groupId>
+ <artifactId>event-bus-api</artifactId>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>${james.groupId}</groupId>
+ <artifactId>event-bus-in-vm</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>${james.groupId}</groupId>
+ <artifactId>james-json</artifactId>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>${james.groupId}</groupId>
+ <artifactId>james-server-data-api</artifactId>
+ </dependency>
<dependency>
<groupId>${james.groupId}</groupId>
<artifactId>james-server-guice-common</artifactId>
<type>test-jar</type>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>${james.groupId}</groupId>
+ <artifactId>james-server-guice-common</artifactId>
+ </dependency>
Review Comment:
Quick question: why is this dependency needed?
##########
third-party/rspamd/src/main/java/org/apache/james/rspamd/task/GetMailboxMessagesService.java:
##########
@@ -0,0 +1,58 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.rspamd.task;
+
+import org.apache.james.core.Username;
+import org.apache.james.mailbox.MailboxManager;
+import org.apache.james.mailbox.MailboxSession;
+import org.apache.james.mailbox.model.FetchGroup;
+import org.apache.james.mailbox.model.MailboxPath;
+import org.apache.james.mailbox.model.MessageRange;
+import org.apache.james.mailbox.model.MessageResult;
+import org.apache.james.user.api.UsersRepository;
+import org.apache.james.user.api.UsersRepositoryException;
+import org.apache.james.util.ReactorUtils;
+import org.apache.james.util.streams.Iterators;
+
+import reactor.core.publisher.Flux;
+import reactor.core.publisher.Mono;
+
+public class GetMailboxMessagesService {
+ private final MailboxManager mailboxManager;
+ private final UsersRepository userRepository;
+
+ public GetMailboxMessagesService(MailboxManager mailboxManager, UsersRepository userRepository) {
+ this.mailboxManager = mailboxManager;
+ this.userRepository = userRepository;
+ }
+
+ public Flux<MessageResult> getMailboxMessagesOfAllUser(String mailbox) throws UsersRepositoryException {
+ return Iterators.toFlux(userRepository.list())
+ .flatMap(username -> getMailboxMessagesOfAUser(username, mailbox), ReactorUtils.DEFAULT_CONCURRENCY);
+ }
+
+ private Flux<MessageResult> getMailboxMessagesOfAUser(Username username, String mailbox) {
+ MailboxSession mailboxSession = mailboxManager.createSystemSession(username);
+
+ return Mono.from(mailboxManager.getMailboxReactive(MailboxPath.forUser(username, mailbox), mailboxSession))
+ .flatMapMany(messageManager -> Flux.from(messageManager.getMessagesReactive(MessageRange.all(), FetchGroup.FULL_CONTENT, mailboxSession)));
Review Comment:
The use of FULL_CONTENT here will load all the messages content, all the time.
This means that this task will always load ALL the content of SPAM mailbox, which can be avoided with sampling/period parameters.
I suggest only loading metadata here and loading content upon reporting ham/spam.
##########
third-party/rspamd/src/main/java/org/apache/james/rspamd/route/FeedMessageRoute.java:
##########
@@ -0,0 +1,135 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.rspamd.route;
+
+import static org.apache.james.rspamd.task.FeedSpamToRSpamDTask.RunningOptions.DEFAULT_MESSAGES_PER_SECOND;
+import static org.apache.james.rspamd.task.FeedSpamToRSpamDTask.RunningOptions.DEFAULT_SAMPLING_PROBABILITY;
+
+import java.time.Clock;
+import java.util.Optional;
+
+import javax.inject.Inject;
+
+import org.apache.james.mailbox.MailboxManager;
+import org.apache.james.rspamd.client.RSpamDHttpClient;
+import org.apache.james.rspamd.task.FeedSpamToRSpamDTask;
+import org.apache.james.task.Task;
+import org.apache.james.task.TaskManager;
+import org.apache.james.user.api.UsersRepository;
+import org.apache.james.webadmin.Routes;
+import org.apache.james.webadmin.tasks.TaskFromRequest;
+import org.apache.james.webadmin.utils.JsonTransformer;
+
+import com.google.common.base.Preconditions;
+
+import spark.Request;
+import spark.Service;
+
+public class FeedMessageRoute implements Routes {
+ public static final String BASE_PATH = "/rspamd";
+
+ private final TaskManager taskManager;
+ private final MailboxManager mailboxManager;
+ private final UsersRepository usersRepository;
+ private final RSpamDHttpClient rSpamDHttpClient;
+ private final JsonTransformer jsonTransformer;
+ private final Clock clock;
+
+ @Inject
+ public FeedMessageRoute(TaskManager taskManager, MailboxManager mailboxManager, UsersRepository usersRepository, RSpamDHttpClient rSpamDHttpClient,
+ JsonTransformer jsonTransformer, Clock clock) {
+ this.taskManager = taskManager;
+ this.mailboxManager = mailboxManager;
+ this.usersRepository = usersRepository;
+ this.rSpamDHttpClient = rSpamDHttpClient;
+ this.jsonTransformer = jsonTransformer;
+ this.clock = clock;
+ }
+
+ @Override
+ public String getBasePath() {
+ return BASE_PATH;
+ }
+
+ @Override
+ public void define(Service service) {
+ TaskFromRequest feedMessageTaskRequest = this::feedMessageTaskFromRequest;
+ service.post(BASE_PATH, feedMessageTaskRequest.asRoute(taskManager), jsonTransformer);
+ }
+
+ public Task feedMessageTaskFromRequest(Request request) {
+ Preconditions.checkArgument(Optional.ofNullable(request.queryParams("action"))
+ .filter(action -> action.equals("reportSpam") || action.equals("reportHam"))
+ .isPresent(),
+ "'action' is missing or must be 'reportSpam' or 'reportHam'");
+
+ return new FeedSpamToRSpamDTask(mailboxManager, usersRepository, rSpamDHttpClient, getFeedSpamTaskRunningOptions(request), clock);
+ }
+
+ private FeedSpamToRSpamDTask.RunningOptions getFeedSpamTaskRunningOptions(Request request) {
+ Optional<Integer> periodInSecond = getPeriod(request);
+ int messagesPerSecond = getMessagesPerSecond(request).orElse(DEFAULT_MESSAGES_PER_SECOND);
+ double samplingProbability = getSamplingProbability(request).orElse(DEFAULT_SAMPLING_PROBABILITY);
+ return new FeedSpamToRSpamDTask.RunningOptions(periodInSecond, messagesPerSecond, samplingProbability);
+ }
+
+ private Optional<Integer> getPeriod(Request req) {
+ try {
+ return Optional.ofNullable(req.queryParams("period"))
+ .map(Integer::parseInt)
Review Comment:
Use DurationParser?
##########
third-party/rspamd/src/main/java/org/apache/james/rspamd/task/FeedSpamToRSpamDTask.java:
##########
@@ -0,0 +1,389 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.rspamd.task;
+
+import java.time.Clock;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Date;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.concurrent.atomic.AtomicLong;
+
+import org.apache.james.mailbox.MailboxManager;
+import org.apache.james.mailbox.model.MessageResult;
+import org.apache.james.rspamd.client.RSpamDHttpClient;
+import org.apache.james.task.Task;
+import org.apache.james.task.TaskExecutionDetails;
+import org.apache.james.task.TaskType;
+import org.apache.james.user.api.UsersRepository;
+import org.apache.james.user.api.UsersRepositoryException;
+import org.apache.james.util.ReactorUtils;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.github.fge.lambdas.Throwing;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.MoreObjects;
+
+import reactor.core.publisher.Mono;
+
+public class FeedSpamToRSpamDTask implements Task {
+ public static final String SPAM_MAILBOX_NAME = "Spam";
+ public static final TaskType TASK_TYPE = TaskType.of("FeedSpamToRSpamDTask");
+
+ public static class RunningOptions {
+ public static final Optional<Integer> DEFAULT_PERIOD = Optional.empty();
+ public static final int DEFAULT_MESSAGES_PER_SECOND = 10;
+ public static final double DEFAULT_SAMPLING_PROBABILITY = 1;
+ public static final RunningOptions DEFAULT = new RunningOptions(DEFAULT_PERIOD, DEFAULT_MESSAGES_PER_SECOND,
+ DEFAULT_SAMPLING_PROBABILITY);
+
+ private final Optional<Integer> period;
+ private final int messagesPerSecond;
+ private final double samplingProbability;
+
+ public RunningOptions(@JsonProperty("period") Optional<Integer> periodInSecond,
+ @JsonProperty("messagesPerSecond") int messagesPerSecond,
+ @JsonProperty("samplingProbability") double samplingProbability) {
+ this.period = periodInSecond;
+ this.messagesPerSecond = messagesPerSecond;
+ this.samplingProbability = samplingProbability;
+ }
+
+ public Optional<Integer> getPeriod() {
+ return period;
+ }
+
+ public int getMessagesPerSecond() {
+ return messagesPerSecond;
+ }
+
+ public double getSamplingProbability() {
+ return samplingProbability;
+ }
+ }
+
+ public static class AdditionalInformation implements TaskExecutionDetails.AdditionalInformation {
+
+ private static AdditionalInformation from(Context context) {
+ Context.Snapshot snapshot = context.snapshot();
+ return new AdditionalInformation(
+ Clock.systemUTC().instant(),
+ snapshot.getSpamMessageCount(),
+ snapshot.getReportedSpamMessageCount(),
+ snapshot.getErrorCount(),
+ snapshot.getMessagesPerSecond(),
+ snapshot.getPeriod(),
+ snapshot.getSamplingProbability());
+ }
+
+ private final Instant timestamp;
+ private final long spamMessageCount;
+ private final long reportedSpamMessageCount;
+ private final long errorCount;
+ private final int messagesPerSecond;
+ private final Optional<Integer> period;
+ private final double samplingProbability;
+
+ public AdditionalInformation(Instant timestamp, long spamMessageCount, long reportedSpamMessageCount, long errorCount, int messagesPerSecond, Optional<Integer> period, double samplingProbability) {
+ this.timestamp = timestamp;
+ this.spamMessageCount = spamMessageCount;
+ this.reportedSpamMessageCount = reportedSpamMessageCount;
+ this.errorCount = errorCount;
+ this.messagesPerSecond = messagesPerSecond;
+ this.period = period;
+ this.samplingProbability = samplingProbability;
+ }
+
+ public long getSpamMessageCount() {
+ return spamMessageCount;
+ }
+
+ public long getReportedSpamMessageCount() {
+ return reportedSpamMessageCount;
+ }
+
+ public long getErrorCount() {
+ return errorCount;
+ }
+
+ public int getMessagesPerSecond() {
+ return messagesPerSecond;
+ }
+
+ public Optional<Integer> getPeriod() {
+ return period;
+ }
+
+ public double getSamplingProbability() {
+ return samplingProbability;
+ }
+
+ @Override
+ public Instant timestamp() {
+ return timestamp;
+ }
+ }
+
+ public static class Context {
+
+ public static class Snapshot {
+
+ public static Builder builder() {
+ return new Builder();
+ }
+
+ static class Builder {
+ private Optional<Long> spamMessageCount;
+ private Optional<Long> reportedSpamMessageCount;
+ private Optional<Long> errorCount;
+ private Optional<Integer> messagesPerSecond;
+ private Optional<Integer> period;
+ private Optional<Double> samplingProbability;
+
+ Builder() {
+ spamMessageCount = Optional.empty();
+ reportedSpamMessageCount = Optional.empty();
+ errorCount = Optional.empty();
+ messagesPerSecond = Optional.empty();
+ period = Optional.empty();
+ samplingProbability = Optional.empty();
+ }
+
+ public Snapshot build() {
+ return new Snapshot(
+ spamMessageCount.orElse(0L),
+ reportedSpamMessageCount.orElse(0L),
+ errorCount.orElse(0L),
+ messagesPerSecond.orElse(0),
+ period,
+ samplingProbability.orElse(1D));
+ }
+
+ public Builder spamMessageCount(long spamMessageCount) {
+ this.spamMessageCount = Optional.of(spamMessageCount);
+ return this;
+ }
+
+ public Builder reportedSpamMessageCount(long reportedSpamMessageCount) {
+ this.reportedSpamMessageCount = Optional.of(reportedSpamMessageCount);
+ return this;
+ }
+
+ public Builder errorCount(long errorCount) {
+ this.errorCount = Optional.of(errorCount);
+ return this;
+ }
+
+ public Builder messagesPerSecond(int messagesPerSecond) {
+ this.messagesPerSecond = Optional.of(messagesPerSecond);
+ return this;
+ }
+
+ public Builder period(Optional<Integer> period) {
+ this.period = period;
+ return this;
+ }
+
+ public Builder samplingProbability(double samplingProbability) {
+ this.samplingProbability = Optional.of(samplingProbability);
+ return this;
+ }
+ }
+
+ private final long spamMessageCount;
+ private final long reportedSpamMessageCount;
+ private final long errorCount;
+ private final int messagesPerSecond;
+ private final Optional<Integer> period;
+ private final double samplingProbability;
+
+ public Snapshot(long spamMessageCount, long reportedSpamMessageCount, long errorCount, int messagesPerSecond, Optional<Integer> period,
+ double samplingProbability) {
+ this.spamMessageCount = spamMessageCount;
+ this.reportedSpamMessageCount = reportedSpamMessageCount;
+ this.errorCount = errorCount;
+ this.messagesPerSecond = messagesPerSecond;
+ this.period = period;
+ this.samplingProbability = samplingProbability;
+ }
+
+ public long getSpamMessageCount() {
+ return spamMessageCount;
+ }
+
+ public long getReportedSpamMessageCount() {
+ return reportedSpamMessageCount;
+ }
+
+ public long getErrorCount() {
+ return errorCount;
+ }
+
+ public int getMessagesPerSecond() {
+ return messagesPerSecond;
+ }
+
+ public Optional<Integer> getPeriod() {
+ return period;
+ }
+
+ public double getSamplingProbability() {
+ return samplingProbability;
+ }
+
+ @Override
+ public final boolean equals(Object o) {
+ if (o instanceof Snapshot) {
+ Snapshot snapshot = (Snapshot) o;
+
+ return Objects.equals(this.spamMessageCount, snapshot.spamMessageCount)
+ && Objects.equals(this.reportedSpamMessageCount, snapshot.reportedSpamMessageCount)
+ && Objects.equals(this.errorCount, snapshot.errorCount)
+ && Objects.equals(this.messagesPerSecond, snapshot.messagesPerSecond)
+ && Objects.equals(this.samplingProbability, snapshot.samplingProbability)
+ && Objects.equals(this.period, snapshot.period);
+ }
+ return false;
+ }
+
+ @Override
+ public final int hashCode() {
+ return Objects.hash(spamMessageCount, reportedSpamMessageCount, errorCount, messagesPerSecond, period, samplingProbability);
+ }
+
+ @Override
+ public String toString() {
+ return MoreObjects.toStringHelper(this)
+ .add("spamMessageCount", spamMessageCount)
+ .add("reportedSpamMessageCount", reportedSpamMessageCount)
+ .add("errorCount", errorCount)
+ .add("messagesPerSecond", messagesPerSecond)
+ .add("period", period)
+ .add("samplingProbability", samplingProbability)
+ .toString();
+ }
+ }
+
+ private final AtomicLong spamMessageCount;
+ private final AtomicLong reportedSpamMessageCount;
+ private final AtomicLong errorCount;
+ private final Integer messagesPerSecond;
+ private final Optional<Integer> period;
+ private final Double samplingProbability;
+
+ public Context(RunningOptions runningOptions) {
+ this.spamMessageCount = new AtomicLong();
+ this.reportedSpamMessageCount = new AtomicLong();
+ this.errorCount = new AtomicLong();
+ this.messagesPerSecond = runningOptions.messagesPerSecond;
+ this.period = runningOptions.period;
+ this.samplingProbability = runningOptions.samplingProbability;
+ }
+
+ public void incrementSpamMessageCount() {
+ spamMessageCount.incrementAndGet();
+ }
+
+ public void incrementReportedSpamMessageCount(int count) {
+ reportedSpamMessageCount.addAndGet(count);
+ }
+
+ public void incrementErrorCount() {
+ errorCount.incrementAndGet();
+ }
+
+ public Snapshot snapshot() {
+ return Snapshot.builder()
+ .spamMessageCount(spamMessageCount.get())
+ .reportedSpamMessageCount(reportedSpamMessageCount.get())
+ .errorCount(errorCount.get())
+ .messagesPerSecond(messagesPerSecond)
+ .period(period)
+ .samplingProbability(samplingProbability)
+ .build();
+ }
+ }
+
+ private final GetMailboxMessagesService messagesService;
+ private final RSpamDHttpClient rSpamDHttpClient;
+ private final RunningOptions runningOptions;
+ private final Context context;
+ private final Clock clock;
+
+ public FeedSpamToRSpamDTask(MailboxManager mailboxManager, UsersRepository usersRepository, RSpamDHttpClient rSpamDHttpClient,
+ RunningOptions runningOptions, Clock clock) {
+ this.runningOptions = runningOptions;
+ this.messagesService = new GetMailboxMessagesService(mailboxManager, usersRepository);
+ this.rSpamDHttpClient = rSpamDHttpClient;
+ this.context = new Context(runningOptions);
+ this.clock = clock;
+ }
+
+ @Override
+ public Result run() {
+ Optional<Date> afterDate = runningOptions.period.map(periodInSecond -> Date.from(clock.instant().minusSeconds(periodInSecond)));
+ try {
+ return messagesService.getMailboxMessagesOfAllUser(SPAM_MAILBOX_NAME)
+ .doOnNext(messageResult -> context.incrementSpamMessageCount())
+ .filter(messageResult -> afterDate.map(date -> messageResult.getInternalDate().after(date)).orElse(true))
+ .filter(any -> randomBooleanWithProbability(runningOptions.samplingProbability))
+ .transform(ReactorUtils.<MessageResult, Task.Result>throttle()
+ .elements(runningOptions.messagesPerSecond)
+ .per(Duration.ofSeconds(1))
+ .forOperation(messageResult1 -> Mono.just(messageResult1)
+ .flatMap(Throwing.function(messageResult -> rSpamDHttpClient.reportAsSpam(messageResult.getHeaders().getInputStream(), messageResult.getBody().getInputStream())))
+ .then(Mono.fromCallable(() -> {
+ context.incrementReportedSpamMessageCount(1);
+ return Result.COMPLETED;
+ }))
+ .onErrorResume(error -> {
+ LOGGER.error("Error when report spam message to RSpamD", error);
+ context.incrementErrorCount();
+ return Mono.just(Result.PARTIAL);
+ })))
+ .reduce(Task::combine)
+ .switchIfEmpty(Mono.just(Result.COMPLETED))
+ .block();
+ } catch (UsersRepositoryException e) {
+ LOGGER.error("Error while accessing users from repository", e);
+ return Task.Result.PARTIAL;
+ }
+ }
+
+ @Override
+ public TaskType type() {
+ return TASK_TYPE;
+ }
+
+ @Override
+ public Optional<TaskExecutionDetails.AdditionalInformation> details() {
+ return Optional.of(AdditionalInformation.from(context));
+ }
+
+ @VisibleForTesting
+ public Context.Snapshot snapshot() {
+ return context.snapshot();
+ }
+
+ public static boolean randomBooleanWithProbability(double probability) {
+ return Math.random() < probability;
+ }
Review Comment:
if we have a probability of `1.0` we probably don't need a call to random.
##########
third-party/rspamd/README.md:
##########
@@ -0,0 +1,40 @@
+## Additional webadmin endpoints
+
+### Report spam messages to RSpamD
+One can use this route to schedule a task that reports spam messages to RSpamD for its spam classify learning.
+
+```bash
+curl -XPOST 'http://ip:port/rspamd?action=reportSpam
+```
+
+This endpoint has the following param:
+- `action` (required): need to be `reportSpam`
+- `messagesPerSecond` (optional): Concurrent learns performed for RSpamD, default to 10
+- `period` (optional): duration (in seconds), only messages between `now` and `now - duration` are reported. By default,
+all messages are reported.
+- `samplingProbability` (optional): float between 0 and 1, represent the chance to report each given message to RSpamD.
+By default, all messages are reported.
+
+Will return the task id. E.g:
+```
+{
+ "taskId": "70c12761-ab86-4321-bb6f-fde99e2f74b0"
+}
+```
Review Comment:
Link to https://james.apache.org/server/manage-webadmin.html#Endpoints_returning_a_task instead (as endpoints returning tasks is something generic, you do not need to detail it here).
##########
third-party/rspamd/src/main/java/org/apache/james/rspamd/task/FeedSpamToRSpamDTask.java:
##########
@@ -0,0 +1,389 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.rspamd.task;
+
+import java.time.Clock;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Date;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.concurrent.atomic.AtomicLong;
+
+import org.apache.james.mailbox.MailboxManager;
+import org.apache.james.mailbox.model.MessageResult;
+import org.apache.james.rspamd.client.RSpamDHttpClient;
+import org.apache.james.task.Task;
+import org.apache.james.task.TaskExecutionDetails;
+import org.apache.james.task.TaskType;
+import org.apache.james.user.api.UsersRepository;
+import org.apache.james.user.api.UsersRepositoryException;
+import org.apache.james.util.ReactorUtils;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.github.fge.lambdas.Throwing;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.MoreObjects;
+
+import reactor.core.publisher.Mono;
+
+public class FeedSpamToRSpamDTask implements Task {
+ public static final String SPAM_MAILBOX_NAME = "Spam";
+ public static final TaskType TASK_TYPE = TaskType.of("FeedSpamToRSpamDTask");
+
+ public static class RunningOptions {
+ public static final Optional<Integer> DEFAULT_PERIOD = Optional.empty();
+ public static final int DEFAULT_MESSAGES_PER_SECOND = 10;
+ public static final double DEFAULT_SAMPLING_PROBABILITY = 1;
+ public static final RunningOptions DEFAULT = new RunningOptions(DEFAULT_PERIOD, DEFAULT_MESSAGES_PER_SECOND,
+ DEFAULT_SAMPLING_PROBABILITY);
+
+ private final Optional<Integer> period;
+ private final int messagesPerSecond;
+ private final double samplingProbability;
+
+ public RunningOptions(@JsonProperty("period") Optional<Integer> periodInSecond,
+ @JsonProperty("messagesPerSecond") int messagesPerSecond,
+ @JsonProperty("samplingProbability") double samplingProbability) {
+ this.period = periodInSecond;
+ this.messagesPerSecond = messagesPerSecond;
+ this.samplingProbability = samplingProbability;
+ }
+
+ public Optional<Integer> getPeriod() {
+ return period;
+ }
+
+ public int getMessagesPerSecond() {
+ return messagesPerSecond;
+ }
+
+ public double getSamplingProbability() {
+ return samplingProbability;
+ }
+ }
+
+ public static class AdditionalInformation implements TaskExecutionDetails.AdditionalInformation {
+
+ private static AdditionalInformation from(Context context) {
+ Context.Snapshot snapshot = context.snapshot();
+ return new AdditionalInformation(
+ Clock.systemUTC().instant(),
+ snapshot.getSpamMessageCount(),
+ snapshot.getReportedSpamMessageCount(),
+ snapshot.getErrorCount(),
+ snapshot.getMessagesPerSecond(),
+ snapshot.getPeriod(),
+ snapshot.getSamplingProbability());
+ }
+
+ private final Instant timestamp;
+ private final long spamMessageCount;
+ private final long reportedSpamMessageCount;
+ private final long errorCount;
+ private final int messagesPerSecond;
+ private final Optional<Integer> period;
+ private final double samplingProbability;
+
+ public AdditionalInformation(Instant timestamp, long spamMessageCount, long reportedSpamMessageCount, long errorCount, int messagesPerSecond, Optional<Integer> period, double samplingProbability) {
+ this.timestamp = timestamp;
+ this.spamMessageCount = spamMessageCount;
+ this.reportedSpamMessageCount = reportedSpamMessageCount;
+ this.errorCount = errorCount;
+ this.messagesPerSecond = messagesPerSecond;
+ this.period = period;
+ this.samplingProbability = samplingProbability;
+ }
+
+ public long getSpamMessageCount() {
+ return spamMessageCount;
+ }
+
+ public long getReportedSpamMessageCount() {
+ return reportedSpamMessageCount;
+ }
+
+ public long getErrorCount() {
+ return errorCount;
+ }
+
+ public int getMessagesPerSecond() {
+ return messagesPerSecond;
+ }
+
+ public Optional<Integer> getPeriod() {
+ return period;
+ }
+
+ public double getSamplingProbability() {
+ return samplingProbability;
+ }
+
+ @Override
+ public Instant timestamp() {
+ return timestamp;
+ }
+ }
+
+ public static class Context {
+
+ public static class Snapshot {
+
+ public static Builder builder() {
+ return new Builder();
+ }
+
+ static class Builder {
+ private Optional<Long> spamMessageCount;
+ private Optional<Long> reportedSpamMessageCount;
+ private Optional<Long> errorCount;
+ private Optional<Integer> messagesPerSecond;
+ private Optional<Integer> period;
+ private Optional<Double> samplingProbability;
+
+ Builder() {
+ spamMessageCount = Optional.empty();
+ reportedSpamMessageCount = Optional.empty();
+ errorCount = Optional.empty();
+ messagesPerSecond = Optional.empty();
+ period = Optional.empty();
+ samplingProbability = Optional.empty();
+ }
+
+ public Snapshot build() {
+ return new Snapshot(
+ spamMessageCount.orElse(0L),
+ reportedSpamMessageCount.orElse(0L),
+ errorCount.orElse(0L),
+ messagesPerSecond.orElse(0),
+ period,
+ samplingProbability.orElse(1D));
+ }
+
+ public Builder spamMessageCount(long spamMessageCount) {
+ this.spamMessageCount = Optional.of(spamMessageCount);
+ return this;
+ }
+
+ public Builder reportedSpamMessageCount(long reportedSpamMessageCount) {
+ this.reportedSpamMessageCount = Optional.of(reportedSpamMessageCount);
+ return this;
+ }
+
+ public Builder errorCount(long errorCount) {
+ this.errorCount = Optional.of(errorCount);
+ return this;
+ }
+
+ public Builder messagesPerSecond(int messagesPerSecond) {
+ this.messagesPerSecond = Optional.of(messagesPerSecond);
+ return this;
+ }
+
+ public Builder period(Optional<Integer> period) {
+ this.period = period;
+ return this;
+ }
+
+ public Builder samplingProbability(double samplingProbability) {
+ this.samplingProbability = Optional.of(samplingProbability);
+ return this;
+ }
+ }
+
+ private final long spamMessageCount;
+ private final long reportedSpamMessageCount;
+ private final long errorCount;
+ private final int messagesPerSecond;
+ private final Optional<Integer> period;
+ private final double samplingProbability;
+
+ public Snapshot(long spamMessageCount, long reportedSpamMessageCount, long errorCount, int messagesPerSecond, Optional<Integer> period,
+ double samplingProbability) {
+ this.spamMessageCount = spamMessageCount;
+ this.reportedSpamMessageCount = reportedSpamMessageCount;
+ this.errorCount = errorCount;
+ this.messagesPerSecond = messagesPerSecond;
+ this.period = period;
+ this.samplingProbability = samplingProbability;
+ }
+
+ public long getSpamMessageCount() {
+ return spamMessageCount;
+ }
+
+ public long getReportedSpamMessageCount() {
+ return reportedSpamMessageCount;
+ }
+
+ public long getErrorCount() {
+ return errorCount;
+ }
+
+ public int getMessagesPerSecond() {
+ return messagesPerSecond;
+ }
+
+ public Optional<Integer> getPeriod() {
+ return period;
+ }
+
+ public double getSamplingProbability() {
+ return samplingProbability;
+ }
+
+ @Override
+ public final boolean equals(Object o) {
+ if (o instanceof Snapshot) {
+ Snapshot snapshot = (Snapshot) o;
+
+ return Objects.equals(this.spamMessageCount, snapshot.spamMessageCount)
+ && Objects.equals(this.reportedSpamMessageCount, snapshot.reportedSpamMessageCount)
+ && Objects.equals(this.errorCount, snapshot.errorCount)
+ && Objects.equals(this.messagesPerSecond, snapshot.messagesPerSecond)
+ && Objects.equals(this.samplingProbability, snapshot.samplingProbability)
+ && Objects.equals(this.period, snapshot.period);
+ }
+ return false;
+ }
+
+ @Override
+ public final int hashCode() {
+ return Objects.hash(spamMessageCount, reportedSpamMessageCount, errorCount, messagesPerSecond, period, samplingProbability);
+ }
+
+ @Override
+ public String toString() {
+ return MoreObjects.toStringHelper(this)
+ .add("spamMessageCount", spamMessageCount)
+ .add("reportedSpamMessageCount", reportedSpamMessageCount)
+ .add("errorCount", errorCount)
+ .add("messagesPerSecond", messagesPerSecond)
+ .add("period", period)
+ .add("samplingProbability", samplingProbability)
+ .toString();
+ }
+ }
+
+ private final AtomicLong spamMessageCount;
+ private final AtomicLong reportedSpamMessageCount;
+ private final AtomicLong errorCount;
+ private final Integer messagesPerSecond;
+ private final Optional<Integer> period;
+ private final Double samplingProbability;
+
+ public Context(RunningOptions runningOptions) {
+ this.spamMessageCount = new AtomicLong();
+ this.reportedSpamMessageCount = new AtomicLong();
+ this.errorCount = new AtomicLong();
+ this.messagesPerSecond = runningOptions.messagesPerSecond;
+ this.period = runningOptions.period;
+ this.samplingProbability = runningOptions.samplingProbability;
+ }
+
+ public void incrementSpamMessageCount() {
+ spamMessageCount.incrementAndGet();
+ }
+
+ public void incrementReportedSpamMessageCount(int count) {
+ reportedSpamMessageCount.addAndGet(count);
+ }
+
+ public void incrementErrorCount() {
+ errorCount.incrementAndGet();
+ }
+
+ public Snapshot snapshot() {
+ return Snapshot.builder()
+ .spamMessageCount(spamMessageCount.get())
+ .reportedSpamMessageCount(reportedSpamMessageCount.get())
+ .errorCount(errorCount.get())
+ .messagesPerSecond(messagesPerSecond)
+ .period(period)
+ .samplingProbability(samplingProbability)
+ .build();
+ }
+ }
+
+ private final GetMailboxMessagesService messagesService;
+ private final RSpamDHttpClient rSpamDHttpClient;
+ private final RunningOptions runningOptions;
+ private final Context context;
+ private final Clock clock;
+
+ public FeedSpamToRSpamDTask(MailboxManager mailboxManager, UsersRepository usersRepository, RSpamDHttpClient rSpamDHttpClient,
+ RunningOptions runningOptions, Clock clock) {
+ this.runningOptions = runningOptions;
+ this.messagesService = new GetMailboxMessagesService(mailboxManager, usersRepository);
+ this.rSpamDHttpClient = rSpamDHttpClient;
+ this.context = new Context(runningOptions);
+ this.clock = clock;
+ }
+
+ @Override
+ public Result run() {
+ Optional<Date> afterDate = runningOptions.period.map(periodInSecond -> Date.from(clock.instant().minusSeconds(periodInSecond)));
+ try {
+ return messagesService.getMailboxMessagesOfAllUser(SPAM_MAILBOX_NAME)
+ .doOnNext(messageResult -> context.incrementSpamMessageCount())
+ .filter(messageResult -> afterDate.map(date -> messageResult.getInternalDate().after(date)).orElse(true))
+ .filter(any -> randomBooleanWithProbability(runningOptions.samplingProbability))
+ .transform(ReactorUtils.<MessageResult, Task.Result>throttle()
+ .elements(runningOptions.messagesPerSecond)
+ .per(Duration.ofSeconds(1))
+ .forOperation(messageResult1 -> Mono.just(messageResult1)
+ .flatMap(Throwing.function(messageResult -> rSpamDHttpClient.reportAsSpam(messageResult.getHeaders().getInputStream(), messageResult.getBody().getInputStream())))
Review Comment:
```suggestion
.forOperation(messageResult1 -> messageResult -> rSpamDHttpClient.reportAsSpam(messageResult.getHeaders().getInputStream(), messageResult.getBody().getInputStream())))
```
Please simpify this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] quantranhong1999 commented on a diff in pull request #1092: JAMES-3775 Write a webadmin route to feed spam to RSpamD
Posted by GitBox <gi...@apache.org>.
quantranhong1999 commented on code in PR #1092:
URL: https://github.com/apache/james-project/pull/1092#discussion_r935086939
##########
third-party/rspamd/src/main/java/org/apache/james/rspamd/route/FeedMessageRoute.java:
##########
@@ -0,0 +1,142 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.rspamd.route;
+
+import static org.apache.james.rspamd.task.FeedSpamToRSpamDTask.RunningOptions.DEFAULT_MESSAGES_PER_SECOND;
+import static org.apache.james.rspamd.task.FeedSpamToRSpamDTask.RunningOptions.DEFAULT_SAMPLING_PROBABILITY;
+
+import java.time.Clock;
+import java.time.temporal.ChronoUnit;
+import java.util.Optional;
+import java.util.function.Predicate;
+
+import javax.inject.Inject;
+
+import org.apache.james.mailbox.MailboxManager;
+import org.apache.james.mailbox.MessageIdManager;
+import org.apache.james.mailbox.store.mail.MessageMapper;
+import org.apache.james.rspamd.client.RSpamDHttpClient;
+import org.apache.james.rspamd.task.FeedSpamToRSpamDTask;
+import org.apache.james.task.Task;
+import org.apache.james.task.TaskManager;
+import org.apache.james.user.api.UsersRepository;
+import org.apache.james.util.DurationParser;
+import org.apache.james.webadmin.Routes;
+import org.apache.james.webadmin.tasks.TaskFromRequest;
+import org.apache.james.webadmin.utils.JsonTransformer;
+
+import com.google.common.base.Preconditions;
+
+import spark.Request;
+import spark.Service;
+
+public class FeedMessageRoute implements Routes {
+ public static final String BASE_PATH = "/rspamd";
+
+ private final TaskManager taskManager;
+ private final MailboxManager mailboxManager;
+ private final MessageIdManager messageIdManager;
+ private final MessageMapper messageMapper;
+ private final UsersRepository usersRepository;
+ private final RSpamDHttpClient rSpamDHttpClient;
+ private final JsonTransformer jsonTransformer;
+ private final Clock clock;
+
+ @Inject
+ public FeedMessageRoute(TaskManager taskManager, MailboxManager mailboxManager, UsersRepository usersRepository, RSpamDHttpClient rSpamDHttpClient,
+ JsonTransformer jsonTransformer, Clock clock, MessageIdManager messageIdManager, MessageMapper messageMapper) {
Review Comment:
https://github.com/apache/james-project/pull/1092/commits/05257991d344401e8aca278e8d2637a692705f4b solves this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] vttranlina commented on a diff in pull request #1092: JAMES-3775 Write a webadmin route to feed spam to RSpamD
Posted by GitBox <gi...@apache.org>.
vttranlina commented on code in PR #1092:
URL: https://github.com/apache/james-project/pull/1092#discussion_r935341039
##########
third-party/rspamd/pom.xml:
##########
@@ -74,10 +115,28 @@
<groupId>com.fasterxml.jackson.datatype</groupId>
<artifactId>jackson-datatype-jdk8</artifactId>
</dependency>
+ <dependency>
+ <groupId>com.google.inject</groupId>
+ <artifactId>guice</artifactId>
+ </dependency>
<dependency>
<groupId>io.projectreactor.netty</groupId>
<artifactId>reactor-netty</artifactId>
</dependency>
+ <dependency>
+ <groupId>net.javacrumbs.json-unit</groupId>
+ <artifactId>json-unit-assertj</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.james</groupId>
Review Comment:
-> `<groupId>${james.groupId}</groupId>`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] chibenwa commented on pull request #1092: JAMES-3775 Write a webadmin route to feed spam to RSpamD
Posted by GitBox <gi...@apache.org>.
chibenwa commented on PR #1092:
URL: https://github.com/apache/james-project/pull/1092#issuecomment-1201028856
It would have been easier for me to follow if you cherry-picked #1100 then have a fix up to adapt your PR to it.
With a force push I need to re-read everything yet-again, and I will not do it today (huge work, and it is late).
This force push takes up my time and creates (unecessary) delays on your work, and a rebase was needed anyway to get rid of the no-review commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] quantranhong1999 commented on pull request #1092: JAMES-3775 Write a webadmin route to feed spam to RSpamD
Posted by GitBox <gi...@apache.org>.
quantranhong1999 commented on PR #1092:
URL: https://github.com/apache/james-project/pull/1092#issuecomment-1201982989
> I don't think that Adapt Benoit review and Adapt Tung review are proper commit names though... :P
Otherwise use fixup it is the same anyway. Yeh I just wait for the approval and rebase.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] quantranhong1999 commented on a diff in pull request #1092: JAMES-3775 Write a webadmin route to feed spam to RSpamD
Posted by GitBox <gi...@apache.org>.
quantranhong1999 commented on code in PR #1092:
URL: https://github.com/apache/james-project/pull/1092#discussion_r934129346
##########
third-party/rspamd/pom.xml:
##########
@@ -33,22 +33,68 @@
<description>RSpamD Java client (HTTP) and testing utilities</description>
<dependencies>
+ <dependency>
+ <groupId>${james.groupId}</groupId>
+ <artifactId>apache-james-mailbox-api</artifactId>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>${james.groupId}</groupId>
+ <artifactId>apache-james-mailbox-memory</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>${james.groupId}</groupId>
+ <artifactId>apache-james-mailbox-memory</artifactId>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
<dependency>
<groupId>${james.groupId}</groupId>
<artifactId>apache-mailet-api</artifactId>
</dependency>
+ <dependency>
+ <groupId>${james.groupId}</groupId>
+ <artifactId>event-bus-api</artifactId>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>${james.groupId}</groupId>
+ <artifactId>event-bus-in-vm</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>${james.groupId}</groupId>
+ <artifactId>james-json</artifactId>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>${james.groupId}</groupId>
+ <artifactId>james-server-data-api</artifactId>
+ </dependency>
<dependency>
<groupId>${james.groupId}</groupId>
<artifactId>james-server-guice-common</artifactId>
<type>test-jar</type>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>${james.groupId}</groupId>
+ <artifactId>james-server-guice-common</artifactId>
+ </dependency>
Review Comment:
`RSpamDModule` to specify webadmin route bean and DTO modules (just not want one forget which to add).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] vttranlina commented on a diff in pull request #1092: JAMES-3775 Write a webadmin route to feed spam to RSpamD
Posted by GitBox <gi...@apache.org>.
vttranlina commented on code in PR #1092:
URL: https://github.com/apache/james-project/pull/1092#discussion_r935074706
##########
third-party/rspamd/src/main/java/org/apache/james/rspamd/route/FeedMessageRoute.java:
##########
@@ -0,0 +1,142 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.rspamd.route;
+
+import static org.apache.james.rspamd.task.FeedSpamToRSpamDTask.RunningOptions.DEFAULT_MESSAGES_PER_SECOND;
+import static org.apache.james.rspamd.task.FeedSpamToRSpamDTask.RunningOptions.DEFAULT_SAMPLING_PROBABILITY;
+
+import java.time.Clock;
+import java.time.temporal.ChronoUnit;
+import java.util.Optional;
+import java.util.function.Predicate;
+
+import javax.inject.Inject;
+
+import org.apache.james.mailbox.MailboxManager;
+import org.apache.james.mailbox.MessageIdManager;
+import org.apache.james.mailbox.store.mail.MessageMapper;
+import org.apache.james.rspamd.client.RSpamDHttpClient;
+import org.apache.james.rspamd.task.FeedSpamToRSpamDTask;
+import org.apache.james.task.Task;
+import org.apache.james.task.TaskManager;
+import org.apache.james.user.api.UsersRepository;
+import org.apache.james.util.DurationParser;
+import org.apache.james.webadmin.Routes;
+import org.apache.james.webadmin.tasks.TaskFromRequest;
+import org.apache.james.webadmin.utils.JsonTransformer;
+
+import com.google.common.base.Preconditions;
+
+import spark.Request;
+import spark.Service;
+
+public class FeedMessageRoute implements Routes {
+ public static final String BASE_PATH = "/rspamd";
+
+ private final TaskManager taskManager;
+ private final MailboxManager mailboxManager;
+ private final MessageIdManager messageIdManager;
+ private final MessageMapper messageMapper;
+ private final UsersRepository usersRepository;
+ private final RSpamDHttpClient rSpamDHttpClient;
+ private final JsonTransformer jsonTransformer;
+ private final Clock clock;
+
+ @Inject
+ public FeedMessageRoute(TaskManager taskManager, MailboxManager mailboxManager, UsersRepository usersRepository, RSpamDHttpClient rSpamDHttpClient,
+ JsonTransformer jsonTransformer, Clock clock, MessageIdManager messageIdManager, MessageMapper messageMapper) {
Review Comment:
I don't think we already have a singleton for `MessageMapper`,
MessageMapper is created for each MailboxSession.
->
```
MailboxSessionMapperFactory.getMessageMapper(MailboxSession)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] quantranhong1999 commented on a diff in pull request #1092: JAMES-3775 Write a webadmin route to feed spam to RSpamD
Posted by GitBox <gi...@apache.org>.
quantranhong1999 commented on code in PR #1092:
URL: https://github.com/apache/james-project/pull/1092#discussion_r934146803
##########
third-party/rspamd/src/main/java/org/apache/james/rspamd/task/FeedSpamToRSpamDTask.java:
##########
@@ -0,0 +1,389 @@
+/****************************************************************
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information *
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the *
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the *
+ * specific language governing permissions and limitations *
+ * under the License. *
+ ****************************************************************/
+
+package org.apache.james.rspamd.task;
+
+import java.time.Clock;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Date;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.concurrent.atomic.AtomicLong;
+
+import org.apache.james.mailbox.MailboxManager;
+import org.apache.james.mailbox.model.MessageResult;
+import org.apache.james.rspamd.client.RSpamDHttpClient;
+import org.apache.james.task.Task;
+import org.apache.james.task.TaskExecutionDetails;
+import org.apache.james.task.TaskType;
+import org.apache.james.user.api.UsersRepository;
+import org.apache.james.user.api.UsersRepositoryException;
+import org.apache.james.util.ReactorUtils;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.github.fge.lambdas.Throwing;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.MoreObjects;
+
+import reactor.core.publisher.Mono;
+
+public class FeedSpamToRSpamDTask implements Task {
+ public static final String SPAM_MAILBOX_NAME = "Spam";
+ public static final TaskType TASK_TYPE = TaskType.of("FeedSpamToRSpamDTask");
+
+ public static class RunningOptions {
+ public static final Optional<Integer> DEFAULT_PERIOD = Optional.empty();
+ public static final int DEFAULT_MESSAGES_PER_SECOND = 10;
+ public static final double DEFAULT_SAMPLING_PROBABILITY = 1;
+ public static final RunningOptions DEFAULT = new RunningOptions(DEFAULT_PERIOD, DEFAULT_MESSAGES_PER_SECOND,
+ DEFAULT_SAMPLING_PROBABILITY);
+
+ private final Optional<Integer> period;
+ private final int messagesPerSecond;
+ private final double samplingProbability;
+
+ public RunningOptions(@JsonProperty("period") Optional<Integer> periodInSecond,
+ @JsonProperty("messagesPerSecond") int messagesPerSecond,
+ @JsonProperty("samplingProbability") double samplingProbability) {
+ this.period = periodInSecond;
+ this.messagesPerSecond = messagesPerSecond;
+ this.samplingProbability = samplingProbability;
+ }
+
+ public Optional<Integer> getPeriod() {
+ return period;
+ }
+
+ public int getMessagesPerSecond() {
+ return messagesPerSecond;
+ }
+
+ public double getSamplingProbability() {
+ return samplingProbability;
+ }
+ }
+
+ public static class AdditionalInformation implements TaskExecutionDetails.AdditionalInformation {
+
+ private static AdditionalInformation from(Context context) {
+ Context.Snapshot snapshot = context.snapshot();
+ return new AdditionalInformation(
+ Clock.systemUTC().instant(),
+ snapshot.getSpamMessageCount(),
+ snapshot.getReportedSpamMessageCount(),
+ snapshot.getErrorCount(),
+ snapshot.getMessagesPerSecond(),
+ snapshot.getPeriod(),
+ snapshot.getSamplingProbability());
+ }
+
+ private final Instant timestamp;
+ private final long spamMessageCount;
+ private final long reportedSpamMessageCount;
+ private final long errorCount;
+ private final int messagesPerSecond;
+ private final Optional<Integer> period;
+ private final double samplingProbability;
+
+ public AdditionalInformation(Instant timestamp, long spamMessageCount, long reportedSpamMessageCount, long errorCount, int messagesPerSecond, Optional<Integer> period, double samplingProbability) {
+ this.timestamp = timestamp;
+ this.spamMessageCount = spamMessageCount;
+ this.reportedSpamMessageCount = reportedSpamMessageCount;
+ this.errorCount = errorCount;
+ this.messagesPerSecond = messagesPerSecond;
+ this.period = period;
+ this.samplingProbability = samplingProbability;
+ }
+
+ public long getSpamMessageCount() {
+ return spamMessageCount;
+ }
+
+ public long getReportedSpamMessageCount() {
+ return reportedSpamMessageCount;
+ }
+
+ public long getErrorCount() {
+ return errorCount;
+ }
+
+ public int getMessagesPerSecond() {
+ return messagesPerSecond;
+ }
+
+ public Optional<Integer> getPeriod() {
+ return period;
+ }
+
+ public double getSamplingProbability() {
+ return samplingProbability;
+ }
+
+ @Override
+ public Instant timestamp() {
+ return timestamp;
+ }
+ }
+
+ public static class Context {
+
+ public static class Snapshot {
+
+ public static Builder builder() {
+ return new Builder();
+ }
+
+ static class Builder {
+ private Optional<Long> spamMessageCount;
+ private Optional<Long> reportedSpamMessageCount;
+ private Optional<Long> errorCount;
+ private Optional<Integer> messagesPerSecond;
+ private Optional<Integer> period;
+ private Optional<Double> samplingProbability;
+
+ Builder() {
+ spamMessageCount = Optional.empty();
+ reportedSpamMessageCount = Optional.empty();
+ errorCount = Optional.empty();
+ messagesPerSecond = Optional.empty();
+ period = Optional.empty();
+ samplingProbability = Optional.empty();
+ }
+
+ public Snapshot build() {
+ return new Snapshot(
+ spamMessageCount.orElse(0L),
+ reportedSpamMessageCount.orElse(0L),
+ errorCount.orElse(0L),
+ messagesPerSecond.orElse(0),
+ period,
+ samplingProbability.orElse(1D));
+ }
+
+ public Builder spamMessageCount(long spamMessageCount) {
+ this.spamMessageCount = Optional.of(spamMessageCount);
+ return this;
+ }
+
+ public Builder reportedSpamMessageCount(long reportedSpamMessageCount) {
+ this.reportedSpamMessageCount = Optional.of(reportedSpamMessageCount);
+ return this;
+ }
+
+ public Builder errorCount(long errorCount) {
+ this.errorCount = Optional.of(errorCount);
+ return this;
+ }
+
+ public Builder messagesPerSecond(int messagesPerSecond) {
+ this.messagesPerSecond = Optional.of(messagesPerSecond);
+ return this;
+ }
+
+ public Builder period(Optional<Integer> period) {
+ this.period = period;
+ return this;
+ }
+
+ public Builder samplingProbability(double samplingProbability) {
+ this.samplingProbability = Optional.of(samplingProbability);
+ return this;
+ }
+ }
+
+ private final long spamMessageCount;
+ private final long reportedSpamMessageCount;
+ private final long errorCount;
+ private final int messagesPerSecond;
+ private final Optional<Integer> period;
+ private final double samplingProbability;
+
+ public Snapshot(long spamMessageCount, long reportedSpamMessageCount, long errorCount, int messagesPerSecond, Optional<Integer> period,
+ double samplingProbability) {
+ this.spamMessageCount = spamMessageCount;
+ this.reportedSpamMessageCount = reportedSpamMessageCount;
+ this.errorCount = errorCount;
+ this.messagesPerSecond = messagesPerSecond;
+ this.period = period;
+ this.samplingProbability = samplingProbability;
+ }
+
+ public long getSpamMessageCount() {
+ return spamMessageCount;
+ }
+
+ public long getReportedSpamMessageCount() {
+ return reportedSpamMessageCount;
+ }
+
+ public long getErrorCount() {
+ return errorCount;
+ }
+
+ public int getMessagesPerSecond() {
+ return messagesPerSecond;
+ }
+
+ public Optional<Integer> getPeriod() {
+ return period;
+ }
+
+ public double getSamplingProbability() {
+ return samplingProbability;
+ }
+
+ @Override
+ public final boolean equals(Object o) {
+ if (o instanceof Snapshot) {
+ Snapshot snapshot = (Snapshot) o;
+
+ return Objects.equals(this.spamMessageCount, snapshot.spamMessageCount)
+ && Objects.equals(this.reportedSpamMessageCount, snapshot.reportedSpamMessageCount)
+ && Objects.equals(this.errorCount, snapshot.errorCount)
+ && Objects.equals(this.messagesPerSecond, snapshot.messagesPerSecond)
+ && Objects.equals(this.samplingProbability, snapshot.samplingProbability)
+ && Objects.equals(this.period, snapshot.period);
+ }
+ return false;
+ }
+
+ @Override
+ public final int hashCode() {
+ return Objects.hash(spamMessageCount, reportedSpamMessageCount, errorCount, messagesPerSecond, period, samplingProbability);
+ }
+
+ @Override
+ public String toString() {
+ return MoreObjects.toStringHelper(this)
+ .add("spamMessageCount", spamMessageCount)
+ .add("reportedSpamMessageCount", reportedSpamMessageCount)
+ .add("errorCount", errorCount)
+ .add("messagesPerSecond", messagesPerSecond)
+ .add("period", period)
+ .add("samplingProbability", samplingProbability)
+ .toString();
+ }
+ }
+
+ private final AtomicLong spamMessageCount;
+ private final AtomicLong reportedSpamMessageCount;
+ private final AtomicLong errorCount;
+ private final Integer messagesPerSecond;
+ private final Optional<Integer> period;
+ private final Double samplingProbability;
+
+ public Context(RunningOptions runningOptions) {
+ this.spamMessageCount = new AtomicLong();
+ this.reportedSpamMessageCount = new AtomicLong();
+ this.errorCount = new AtomicLong();
+ this.messagesPerSecond = runningOptions.messagesPerSecond;
+ this.period = runningOptions.period;
+ this.samplingProbability = runningOptions.samplingProbability;
+ }
+
+ public void incrementSpamMessageCount() {
+ spamMessageCount.incrementAndGet();
+ }
+
+ public void incrementReportedSpamMessageCount(int count) {
+ reportedSpamMessageCount.addAndGet(count);
+ }
+
+ public void incrementErrorCount() {
+ errorCount.incrementAndGet();
+ }
+
+ public Snapshot snapshot() {
+ return Snapshot.builder()
+ .spamMessageCount(spamMessageCount.get())
+ .reportedSpamMessageCount(reportedSpamMessageCount.get())
+ .errorCount(errorCount.get())
+ .messagesPerSecond(messagesPerSecond)
+ .period(period)
+ .samplingProbability(samplingProbability)
+ .build();
+ }
+ }
+
+ private final GetMailboxMessagesService messagesService;
+ private final RSpamDHttpClient rSpamDHttpClient;
+ private final RunningOptions runningOptions;
+ private final Context context;
+ private final Clock clock;
+
+ public FeedSpamToRSpamDTask(MailboxManager mailboxManager, UsersRepository usersRepository, RSpamDHttpClient rSpamDHttpClient,
+ RunningOptions runningOptions, Clock clock) {
+ this.runningOptions = runningOptions;
+ this.messagesService = new GetMailboxMessagesService(mailboxManager, usersRepository);
+ this.rSpamDHttpClient = rSpamDHttpClient;
+ this.context = new Context(runningOptions);
+ this.clock = clock;
+ }
+
+ @Override
+ public Result run() {
+ Optional<Date> afterDate = runningOptions.period.map(periodInSecond -> Date.from(clock.instant().minusSeconds(periodInSecond)));
+ try {
+ return messagesService.getMailboxMessagesOfAllUser(SPAM_MAILBOX_NAME)
+ .doOnNext(messageResult -> context.incrementSpamMessageCount())
+ .filter(messageResult -> afterDate.map(date -> messageResult.getInternalDate().after(date)).orElse(true))
+ .filter(any -> randomBooleanWithProbability(runningOptions.samplingProbability))
+ .transform(ReactorUtils.<MessageResult, Task.Result>throttle()
+ .elements(runningOptions.messagesPerSecond)
+ .per(Duration.ofSeconds(1))
+ .forOperation(messageResult1 -> Mono.just(messageResult1)
+ .flatMap(Throwing.function(messageResult -> rSpamDHttpClient.reportAsSpam(messageResult.getHeaders().getInputStream(), messageResult.getBody().getInputStream())))
Review Comment:
Switching to this, we would have to do a verbose try catch to use `.fullContent`. I put in some effort but did not find a better solution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] chibenwa commented on a diff in pull request #1092: JAMES-3775 Write a webadmin route to feed spam to RSpamD
Posted by GitBox <gi...@apache.org>.
chibenwa commented on code in PR #1092:
URL: https://github.com/apache/james-project/pull/1092#discussion_r934132620
##########
third-party/rspamd/pom.xml:
##########
@@ -33,22 +33,68 @@
<description>RSpamD Java client (HTTP) and testing utilities</description>
<dependencies>
+ <dependency>
+ <groupId>${james.groupId}</groupId>
+ <artifactId>apache-james-mailbox-api</artifactId>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>${james.groupId}</groupId>
+ <artifactId>apache-james-mailbox-memory</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>${james.groupId}</groupId>
+ <artifactId>apache-james-mailbox-memory</artifactId>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
<dependency>
<groupId>${james.groupId}</groupId>
<artifactId>apache-mailet-api</artifactId>
</dependency>
+ <dependency>
+ <groupId>${james.groupId}</groupId>
+ <artifactId>event-bus-api</artifactId>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>${james.groupId}</groupId>
+ <artifactId>event-bus-in-vm</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>${james.groupId}</groupId>
+ <artifactId>james-json</artifactId>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>${james.groupId}</groupId>
+ <artifactId>james-server-data-api</artifactId>
+ </dependency>
<dependency>
<groupId>${james.groupId}</groupId>
<artifactId>james-server-guice-common</artifactId>
<type>test-jar</type>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>${james.groupId}</groupId>
+ <artifactId>james-server-guice-common</artifactId>
+ </dependency>
Review Comment:
Guice only is not enough?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] Arsnael commented on pull request #1092: JAMES-3775 Write a webadmin route to feed spam to RSpamD
Posted by GitBox <gi...@apache.org>.
Arsnael commented on PR #1092:
URL: https://github.com/apache/james-project/pull/1092#issuecomment-1201982251
I don't think that Adapt Benoit review and Adapt Tung review are proper commit names though... :P
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] chibenwa commented on pull request #1092: JAMES-3775 Write a webadmin route to feed spam to RSpamD
Posted by GitBox <gi...@apache.org>.
chibenwa commented on PR #1092:
URL: https://github.com/apache/james-project/pull/1092#issuecomment-1201047703
Yes I missed the "adapt benoit review". Sorry I may still be a bit jet-lagged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] Arsnael commented on pull request #1092: JAMES-3775 Write a webadmin route to feed spam to RSpamD
Posted by GitBox <gi...@apache.org>.
Arsnael commented on PR #1092:
URL: https://github.com/apache/james-project/pull/1092#issuecomment-1201985496
I disagree... fixups are a standard and easy to spot when you squash up. This... So ok it's a small PR but on a bigger scale can easily be missed :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
[GitHub] [james-project] chibenwa merged pull request #1092: JAMES-3775 Write a webadmin route to feed spam to RSpamD
Posted by GitBox <gi...@apache.org>.
chibenwa merged PR #1092:
URL: https://github.com/apache/james-project/pull/1092
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org