You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by trixpan <gi...@git.apache.org> on 2017/03/02 13:58:04 UTC
[GitHub] nifi pull request #1553: NIFI-1449 - Migrate PutEmail tests from Mock class ...
GitHub user trixpan opened a pull request:
https://github.com/apache/nifi/pull/1553
NIFI-1449 - Migrate PutEmail tests from Mock class to ListenSMTP
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [X] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [X] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [X] Has your PR been rebased against the latest commit within the target branch (typically master)?
- [X] Is your initial contribution a single, squashed commit?
### For code changes:
- [X] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
- [X] Have you written or updated unit tests to verify your changes?
- [X] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [X] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
- [X] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
- [X] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/trixpan/nifi NIFI-1449
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/1553.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1553
----
commit 707b7aa7d8285f2e199ab5b0a03e4bbcc3340855
Author: Andre F de Miranda <tr...@users.noreply.github.com>
Date: 2017-03-02T13:23:26Z
NIFI-1449 - Migrate PutEmail tests from Mock class to ListenSMTP
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1553: NIFI-1449 - Migrate PutEmail tests from Mock class to List...
Posted by JPercivall <gi...@git.apache.org>.
Github user JPercivall commented on the issue:
https://github.com/apache/nifi/pull/1553
Also the travis failures appear to be unrelated but there are merge conflicts.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1553: NIFI-1449 - Migrate PutEmail tests from Mock class to List...
Posted by trixpan <gi...@git.apache.org>.
Github user trixpan commented on the issue:
https://github.com/apache/nifi/pull/1553
@olegz - any luck reviewing this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1553: NIFI-1449 - Migrate PutEmail tests from Mock class ...
Posted by olegz <gi...@git.apache.org>.
Github user olegz commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1553#discussion_r103939988
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestPutEmail.java ---
@@ -102,13 +91,16 @@ public void testExceptionWhenSending() {
runner.assertQueueEmpty();
runner.assertAllFlowFilesTransferred(PutEmail.REL_FAILURE);
- assertEquals("Expected an attempt to send a single message", 1, processor.getMessages().size());
+ List<MockFlowFile> results = runner.getFlowFilesForRelationship(PutEmail.REL_FAILURE);
+ assertEquals("Expected an attempt to send a single message", 1, results.size());
}
@Test
public void testOutgoingMessage() throws Exception {
// verifies that are set on the outgoing Message correctly
- runner.setProperty(PutEmail.SMTP_HOSTNAME, "smtp-host");
+ runner.setProperty(PutEmail.SMTP_AUTH, "false");
+ runner.setProperty(PutEmail.SMTP_HOSTNAME, "localhost");
+ runner.setProperty(PutEmail.SMTP_PORT, String.valueOf(port));
runner.setProperty(PutEmail.HEADER_XMAILER, "TestingNiFi");
runner.setProperty(PutEmail.FROM, "test@apache.org");
--- End diff --
Not sure if that's ok to use legitimate email address as test data especially as TO/FROM email address. NOt saying it's wrong, simply raising the question.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1553: NIFI-1449 - Migrate PutEmail tests from Mock class ...
Posted by trixpan <gi...@git.apache.org>.
Github user trixpan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1553#discussion_r104021856
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestPutEmail.java ---
@@ -230,27 +234,42 @@ public void testOutgoingMessageAttachment() throws Exception {
runner.assertAllFlowFilesTransferred(PutEmail.REL_SUCCESS);
// Verify that the Message was populated correctly
- assertEquals("Expected a single message to be sent", 1, processor.getMessages().size());
- Message message = processor.getMessages().get(0);
- assertEquals("test@apache.org", message.getFrom()[0].toString());
- assertEquals("X-Mailer Header", "TestingNiFi", message.getHeader("X-Mailer")[0]);
- assertEquals("recipient@apache.org", message.getRecipients(RecipientType.TO)[0].toString());
-
- assertTrue(message.getContent() instanceof MimeMultipart);
-
- final MimeMultipart multipart = (MimeMultipart) message.getContent();
- final BodyPart part = multipart.getBodyPart(0);
- final InputStream is = part.getDataHandler().getInputStream();
- final String decodedText = StringUtils.newStringUtf8(Base64.decodeBase64(IOUtils.toString(is, "UTF-8")));
- assertEquals("Message Body", decodedText);
-
- final BodyPart attachPart = multipart.getBodyPart(1);
- final InputStream attachIs = attachPart.getDataHandler().getInputStream();
- final String text = IOUtils.toString(attachIs, "UTF-8");
- assertEquals("Some text", text);
-
- assertNull(message.getRecipients(RecipientType.BCC));
- assertNull(message.getRecipients(RecipientType.CC));
+ assertEquals("Expected a single message to be sent", 1, smtpRunner.getFlowFilesForRelationship(PutEmail.REL_SUCCESS).size());
+
+ MockFlowFile message;
+
+ message = smtpRunner.getFlowFilesForRelationship(PutEmail.REL_SUCCESS).get(0);
+
+ extractHeadersRunner.enqueue(message);
+ extractHeadersRunner.run();
+ extractHeadersRunner.shutdown();
+
+ message = extractHeadersRunner.getFlowFilesForRelationship(ExtractEmailHeaders.REL_SUCCESS).get(0);
+
+ assertEquals("test@apache.org", message.getAttribute("email.headers.from.0"));
+ assertEquals("X-Mailer Header", "TestingNiFi", message.getAttribute("email.headers.x-mailer"));
+
+ // Needed in order to avoid errors like
+ // "already in use for an active callback or InputStream created by ProcessSession.read(FlowFile) has not been closed"
+
+
+ message = extractHeadersRunner.getFlowFilesForRelationship(ExtractEmailHeaders.REL_SUCCESS).get(0);
+
+// extractAttachmentsRunner.enqueue(message);
+// extractAttachmentsRunner.run();
+//
+// final List<MockFlowFile> splits = extractAttachmentsRunner.getFlowFilesForRelationship(ExtractEmailAttachments.REL_ATTACHMENTS);
+//
+// final MimeMultipart multipart = (MimeMultipart) message.getContent();
+// final BodyPart part = multipart.getBodyPart(0);
+// final InputStream is = part.getDataHandler().getInputStream();
+// final String decodedText = StringUtils.newStringUtf8(Base64.decodeBase64(IOUtils.toString(is, "UTF-8")));
+// assertEquals("Message Body", decodedText);
+//
+// final BodyPart attachPart = multipart.getBodyPart(1);
+// final InputStream attachIs = attachPart.getDataHandler().getInputStream();
+// final String text = IOUtils.toString(attachIs, "UTF-8");
+// assertEquals("Some text", text);
--- End diff --
@olegz, yes. They are a remaining part of the test that have not yet been addressed because of the bug I mentioned on my first comment
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1553: NIFI-1449 - Migrate PutEmail tests from Mock class ...
Posted by trixpan <gi...@git.apache.org>.
Github user trixpan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1553#discussion_r104113002
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestPutEmail.java ---
@@ -230,27 +234,42 @@ public void testOutgoingMessageAttachment() throws Exception {
runner.assertAllFlowFilesTransferred(PutEmail.REL_SUCCESS);
// Verify that the Message was populated correctly
- assertEquals("Expected a single message to be sent", 1, processor.getMessages().size());
- Message message = processor.getMessages().get(0);
- assertEquals("test@apache.org", message.getFrom()[0].toString());
- assertEquals("X-Mailer Header", "TestingNiFi", message.getHeader("X-Mailer")[0]);
- assertEquals("recipient@apache.org", message.getRecipients(RecipientType.TO)[0].toString());
-
- assertTrue(message.getContent() instanceof MimeMultipart);
-
- final MimeMultipart multipart = (MimeMultipart) message.getContent();
- final BodyPart part = multipart.getBodyPart(0);
- final InputStream is = part.getDataHandler().getInputStream();
- final String decodedText = StringUtils.newStringUtf8(Base64.decodeBase64(IOUtils.toString(is, "UTF-8")));
- assertEquals("Message Body", decodedText);
-
- final BodyPart attachPart = multipart.getBodyPart(1);
- final InputStream attachIs = attachPart.getDataHandler().getInputStream();
- final String text = IOUtils.toString(attachIs, "UTF-8");
- assertEquals("Some text", text);
-
- assertNull(message.getRecipients(RecipientType.BCC));
- assertNull(message.getRecipients(RecipientType.CC));
+ assertEquals("Expected a single message to be sent", 1, smtpRunner.getFlowFilesForRelationship(PutEmail.REL_SUCCESS).size());
+
+ MockFlowFile message;
+
+ message = smtpRunner.getFlowFilesForRelationship(PutEmail.REL_SUCCESS).get(0);
+
+ extractHeadersRunner.enqueue(message);
+ extractHeadersRunner.run();
+ extractHeadersRunner.shutdown();
+
+ message = extractHeadersRunner.getFlowFilesForRelationship(ExtractEmailHeaders.REL_SUCCESS).get(0);
+
+ assertEquals("test@apache.org", message.getAttribute("email.headers.from.0"));
+ assertEquals("X-Mailer Header", "TestingNiFi", message.getAttribute("email.headers.x-mailer"));
+
+ // Needed in order to avoid errors like
+ // "already in use for an active callback or InputStream created by ProcessSession.read(FlowFile) has not been closed"
+
+
+ message = extractHeadersRunner.getFlowFilesForRelationship(ExtractEmailHeaders.REL_SUCCESS).get(0);
+
+// extractAttachmentsRunner.enqueue(message);
+// extractAttachmentsRunner.run();
+//
+// final List<MockFlowFile> splits = extractAttachmentsRunner.getFlowFilesForRelationship(ExtractEmailAttachments.REL_ATTACHMENTS);
+//
+// final MimeMultipart multipart = (MimeMultipart) message.getContent();
+// final BodyPart part = multipart.getBodyPart(0);
+// final InputStream is = part.getDataHandler().getInputStream();
+// final String decodedText = StringUtils.newStringUtf8(Base64.decodeBase64(IOUtils.toString(is, "UTF-8")));
+// assertEquals("Message Body", decodedText);
+//
+// final BodyPart attachPart = multipart.getBodyPart(1);
+// final InputStream attachIs = attachPart.getDataHandler().getInputStream();
+// final String text = IOUtils.toString(attachIs, "UTF-8");
+// assertEquals("Some text", text);
--- End diff --
@olegz - I did some further investigation and noticed the issue is triggered when we reach to:
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ExtractEmailAttachments.java#L145
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1553: NIFI-1449 - Migrate PutEmail tests from Mock class ...
Posted by trixpan <gi...@git.apache.org>.
Github user trixpan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1553#discussion_r104170950
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestPutEmail.java ---
@@ -230,27 +234,42 @@ public void testOutgoingMessageAttachment() throws Exception {
runner.assertAllFlowFilesTransferred(PutEmail.REL_SUCCESS);
// Verify that the Message was populated correctly
- assertEquals("Expected a single message to be sent", 1, processor.getMessages().size());
- Message message = processor.getMessages().get(0);
- assertEquals("test@apache.org", message.getFrom()[0].toString());
- assertEquals("X-Mailer Header", "TestingNiFi", message.getHeader("X-Mailer")[0]);
- assertEquals("recipient@apache.org", message.getRecipients(RecipientType.TO)[0].toString());
-
- assertTrue(message.getContent() instanceof MimeMultipart);
-
- final MimeMultipart multipart = (MimeMultipart) message.getContent();
- final BodyPart part = multipart.getBodyPart(0);
- final InputStream is = part.getDataHandler().getInputStream();
- final String decodedText = StringUtils.newStringUtf8(Base64.decodeBase64(IOUtils.toString(is, "UTF-8")));
- assertEquals("Message Body", decodedText);
-
- final BodyPart attachPart = multipart.getBodyPart(1);
- final InputStream attachIs = attachPart.getDataHandler().getInputStream();
- final String text = IOUtils.toString(attachIs, "UTF-8");
- assertEquals("Some text", text);
-
- assertNull(message.getRecipients(RecipientType.BCC));
- assertNull(message.getRecipients(RecipientType.CC));
+ assertEquals("Expected a single message to be sent", 1, smtpRunner.getFlowFilesForRelationship(PutEmail.REL_SUCCESS).size());
+
+ MockFlowFile message;
+
+ message = smtpRunner.getFlowFilesForRelationship(PutEmail.REL_SUCCESS).get(0);
+
+ extractHeadersRunner.enqueue(message);
+ extractHeadersRunner.run();
+ extractHeadersRunner.shutdown();
+
+ message = extractHeadersRunner.getFlowFilesForRelationship(ExtractEmailHeaders.REL_SUCCESS).get(0);
+
+ assertEquals("test@apache.org", message.getAttribute("email.headers.from.0"));
+ assertEquals("X-Mailer Header", "TestingNiFi", message.getAttribute("email.headers.x-mailer"));
+
+ // Needed in order to avoid errors like
+ // "already in use for an active callback or InputStream created by ProcessSession.read(FlowFile) has not been closed"
+
+
+ message = extractHeadersRunner.getFlowFilesForRelationship(ExtractEmailHeaders.REL_SUCCESS).get(0);
+
+// extractAttachmentsRunner.enqueue(message);
+// extractAttachmentsRunner.run();
+//
+// final List<MockFlowFile> splits = extractAttachmentsRunner.getFlowFilesForRelationship(ExtractEmailAttachments.REL_ATTACHMENTS);
+//
+// final MimeMultipart multipart = (MimeMultipart) message.getContent();
+// final BodyPart part = multipart.getBodyPart(0);
+// final InputStream is = part.getDataHandler().getInputStream();
+// final String decodedText = StringUtils.newStringUtf8(Base64.decodeBase64(IOUtils.toString(is, "UTF-8")));
+// assertEquals("Message Body", decodedText);
+//
+// final BodyPart attachPart = multipart.getBodyPart(1);
+// final InputStream attachIs = attachPart.getDataHandler().getInputStream();
+// final String text = IOUtils.toString(attachIs, "UTF-8");
+// assertEquals("Some text", text);
--- End diff --
@olegz
@bbende has shed some light and the issue seems to be caused by the way TestRunner/MockFlowFile works: Feeding a runner with flowfiles from a distinct processor may cause some issues like the one observed.
Tested the theory using SplitText instead of ExtractEmailAttributes and a similar error message arises:
```
[pool-3-thread-1] INFO org.apache.nifi.processors.standard.GetFile - GetFile[id=21d110fa-75a0-4cc8-b425-af6c55f352df] added FlowFile[0,sample.msg,893B] to flow
[pool-4-thread-1] ERROR org.apache.nifi.processors.standard.SplitText - SplitText[id=e064a6bb-5640-4008-b490-4e6b127244af] SplitText[id=e064a6bb-5640-4008-b490-4e6b127244af] failed to process due to org.apache.nifi.processor.exception.FlowFileHandlingException: FlowFile[0,sample.msg,893B] is not the most recent version of this flow file within this session; rolling back session: org.apache.nifi.processor.exception.FlowFileHandlingException: FlowFile[0,sample.msg,893B] is not the most recent version of this flow file within this session
java.lang.AssertionError: org.apache.nifi.processor.exception.FlowFileHandlingException: FlowFile[0,sample.msg,893B] is not the most recent version of this flow file within this session
```
Will continue the test migration and let you know when ready for review.
Cheers
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1553: NIFI-1449 - Migrate PutEmail tests from Mock class ...
Posted by JPercivall <gi...@git.apache.org>.
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1553#discussion_r114464910
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestPutEmail.java ---
@@ -17,83 +17,64 @@
package org.apache.nifi.processors.standard;
import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertNull;
+
import static org.junit.Assert.assertTrue;
-import java.io.InputStream;
-import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
-import javax.mail.BodyPart;
-import javax.mail.Message;
-import javax.mail.MessagingException;
-import javax.mail.internet.MimeMessage.RecipientType;
-import javax.mail.internet.MimeMultipart;
-
import org.apache.commons.codec.binary.Base64;
import org.apache.commons.codec.binary.StringUtils;
-import org.apache.commons.io.IOUtils;
+import org.apache.nifi.processors.email.ExtractEmailAttachments;
+import org.apache.nifi.processors.email.ExtractEmailHeaders;
+import org.apache.nifi.processors.email.ListenSMTP;
+import org.apache.nifi.remote.io.socket.NetworkUtils;
import org.apache.nifi.util.LogMessage;
+import org.apache.nifi.util.MockFlowFile;
import org.apache.nifi.util.TestRunner;
import org.apache.nifi.util.TestRunners;
+import org.hamcrest.CoreMatchers;
+import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
-public class TestPutEmail {
- /**
- * Extension to PutEmail that stubs out the calls to
- * Transport.sendMessage().
- *
- * <p>
- * All sent messages are records in a list available via the
- * {@link #getMessages()} method.</p>
- * <p> Calling
- * {@link #setException(MessagingException)} will cause the supplied exception to be
- * thrown when sendMessage is invoked.
- * </p>
- */
- private static final class PutEmailExtension extends PutEmail {
- private MessagingException e;
- private ArrayList<Message> messages = new ArrayList<>();
-
- @Override
- protected void send(Message msg) throws MessagingException {
- messages.add(msg);
- if (this.e != null) {
- throw e;
- }
- }
-
- void setException(final MessagingException e) {
- this.e = e;
- }
-
- List<Message> getMessages() {
- return messages;
- }
- }
- PutEmailExtension processor;
+public class TestPutEmail {
+
TestRunner runner;
+ TestRunner smtpRunner;
+ TestRunner extractHeadersRunner;
+ TestRunner extractAttachmentsRunner;
+ int port;
@Before
public void setup() {
- processor = new PutEmailExtension();
- runner = TestRunners.newTestRunner(processor);
+ runner = TestRunners.newTestRunner(PutEmail.class);
+ smtpRunner = TestRunners.newTestRunner(ListenSMTP.class);
+ extractHeadersRunner = TestRunners.newTestRunner(ExtractEmailHeaders.class);
+ extractAttachmentsRunner = TestRunners.newTestRunner(ExtractEmailAttachments.class);
+
+ port = NetworkUtils.availablePort();
+ smtpRunner.setProperty("SMTP_PORT", String.valueOf(port));
+ smtpRunner.setProperty("SMTP_MAXIMUM_CONNECTIONS", "3");
+ smtpRunner.setProperty("SMTP_TIMEOUT", "10 seconds");
+ smtpRunner.run(1,false);
--- End diff --
I believe this line is the only one that needs to be run before each method and the rest can be run before the class, specifically the network port.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1553: NIFI-1449 - Migrate PutEmail tests from Mock class ...
Posted by trixpan <gi...@git.apache.org>.
Github user trixpan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1553#discussion_r104114824
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestPutEmail.java ---
@@ -230,27 +234,42 @@ public void testOutgoingMessageAttachment() throws Exception {
runner.assertAllFlowFilesTransferred(PutEmail.REL_SUCCESS);
// Verify that the Message was populated correctly
- assertEquals("Expected a single message to be sent", 1, processor.getMessages().size());
- Message message = processor.getMessages().get(0);
- assertEquals("test@apache.org", message.getFrom()[0].toString());
- assertEquals("X-Mailer Header", "TestingNiFi", message.getHeader("X-Mailer")[0]);
- assertEquals("recipient@apache.org", message.getRecipients(RecipientType.TO)[0].toString());
-
- assertTrue(message.getContent() instanceof MimeMultipart);
-
- final MimeMultipart multipart = (MimeMultipart) message.getContent();
- final BodyPart part = multipart.getBodyPart(0);
- final InputStream is = part.getDataHandler().getInputStream();
- final String decodedText = StringUtils.newStringUtf8(Base64.decodeBase64(IOUtils.toString(is, "UTF-8")));
- assertEquals("Message Body", decodedText);
-
- final BodyPart attachPart = multipart.getBodyPart(1);
- final InputStream attachIs = attachPart.getDataHandler().getInputStream();
- final String text = IOUtils.toString(attachIs, "UTF-8");
- assertEquals("Some text", text);
-
- assertNull(message.getRecipients(RecipientType.BCC));
- assertNull(message.getRecipients(RecipientType.CC));
+ assertEquals("Expected a single message to be sent", 1, smtpRunner.getFlowFilesForRelationship(PutEmail.REL_SUCCESS).size());
+
+ MockFlowFile message;
+
+ message = smtpRunner.getFlowFilesForRelationship(PutEmail.REL_SUCCESS).get(0);
+
+ extractHeadersRunner.enqueue(message);
+ extractHeadersRunner.run();
+ extractHeadersRunner.shutdown();
+
+ message = extractHeadersRunner.getFlowFilesForRelationship(ExtractEmailHeaders.REL_SUCCESS).get(0);
+
+ assertEquals("test@apache.org", message.getAttribute("email.headers.from.0"));
+ assertEquals("X-Mailer Header", "TestingNiFi", message.getAttribute("email.headers.x-mailer"));
+
+ // Needed in order to avoid errors like
+ // "already in use for an active callback or InputStream created by ProcessSession.read(FlowFile) has not been closed"
+
+
+ message = extractHeadersRunner.getFlowFilesForRelationship(ExtractEmailHeaders.REL_SUCCESS).get(0);
+
+// extractAttachmentsRunner.enqueue(message);
+// extractAttachmentsRunner.run();
+//
+// final List<MockFlowFile> splits = extractAttachmentsRunner.getFlowFilesForRelationship(ExtractEmailAttachments.REL_ATTACHMENTS);
+//
+// final MimeMultipart multipart = (MimeMultipart) message.getContent();
+// final BodyPart part = multipart.getBodyPart(0);
+// final InputStream is = part.getDataHandler().getInputStream();
+// final String decodedText = StringUtils.newStringUtf8(Base64.decodeBase64(IOUtils.toString(is, "UTF-8")));
+// assertEquals("Message Body", decodedText);
+//
+// final BodyPart attachPart = multipart.getBodyPart(1);
+// final InputStream attachIs = attachPart.getDataHandler().getInputStream();
+// final String text = IOUtils.toString(attachIs, "UTF-8");
+// assertEquals("Some text", text);
--- End diff --
I think I understand the issue triggering the but (although I wonder how the code seems to have worked in the first place...)
Trying to fix
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1553: NIFI-1449 - Migrate PutEmail tests from Mock class ...
Posted by olegz <gi...@git.apache.org>.
Github user olegz commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1553#discussion_r103938291
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestPutEmail.java ---
@@ -230,27 +234,42 @@ public void testOutgoingMessageAttachment() throws Exception {
runner.assertAllFlowFilesTransferred(PutEmail.REL_SUCCESS);
// Verify that the Message was populated correctly
- assertEquals("Expected a single message to be sent", 1, processor.getMessages().size());
- Message message = processor.getMessages().get(0);
- assertEquals("test@apache.org", message.getFrom()[0].toString());
- assertEquals("X-Mailer Header", "TestingNiFi", message.getHeader("X-Mailer")[0]);
- assertEquals("recipient@apache.org", message.getRecipients(RecipientType.TO)[0].toString());
-
- assertTrue(message.getContent() instanceof MimeMultipart);
-
- final MimeMultipart multipart = (MimeMultipart) message.getContent();
- final BodyPart part = multipart.getBodyPart(0);
- final InputStream is = part.getDataHandler().getInputStream();
- final String decodedText = StringUtils.newStringUtf8(Base64.decodeBase64(IOUtils.toString(is, "UTF-8")));
- assertEquals("Message Body", decodedText);
-
- final BodyPart attachPart = multipart.getBodyPart(1);
- final InputStream attachIs = attachPart.getDataHandler().getInputStream();
- final String text = IOUtils.toString(attachIs, "UTF-8");
- assertEquals("Some text", text);
-
- assertNull(message.getRecipients(RecipientType.BCC));
- assertNull(message.getRecipients(RecipientType.CC));
+ assertEquals("Expected a single message to be sent", 1, smtpRunner.getFlowFilesForRelationship(PutEmail.REL_SUCCESS).size());
+
+ MockFlowFile message;
+
+ message = smtpRunner.getFlowFilesForRelationship(PutEmail.REL_SUCCESS).get(0);
+
+ extractHeadersRunner.enqueue(message);
+ extractHeadersRunner.run();
+ extractHeadersRunner.shutdown();
+
+ message = extractHeadersRunner.getFlowFilesForRelationship(ExtractEmailHeaders.REL_SUCCESS).get(0);
+
+ assertEquals("test@apache.org", message.getAttribute("email.headers.from.0"));
+ assertEquals("X-Mailer Header", "TestingNiFi", message.getAttribute("email.headers.x-mailer"));
+
+ // Needed in order to avoid errors like
+ // "already in use for an active callback or InputStream created by ProcessSession.read(FlowFile) has not been closed"
+
+
+ message = extractHeadersRunner.getFlowFilesForRelationship(ExtractEmailHeaders.REL_SUCCESS).get(0);
+
+// extractAttachmentsRunner.enqueue(message);
+// extractAttachmentsRunner.run();
+//
+// final List<MockFlowFile> splits = extractAttachmentsRunner.getFlowFilesForRelationship(ExtractEmailAttachments.REL_ATTACHMENTS);
+//
+// final MimeMultipart multipart = (MimeMultipart) message.getContent();
+// final BodyPart part = multipart.getBodyPart(0);
+// final InputStream is = part.getDataHandler().getInputStream();
+// final String decodedText = StringUtils.newStringUtf8(Base64.decodeBase64(IOUtils.toString(is, "UTF-8")));
+// assertEquals("Message Body", decodedText);
+//
+// final BodyPart attachPart = multipart.getBodyPart(1);
+// final InputStream attachIs = attachPart.getDataHandler().getInputStream();
+// final String text = IOUtils.toString(attachIs, "UTF-8");
+// assertEquals("Some text", text);
--- End diff --
Are these comments necessary?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1553: NIFI-1449 - Migrate PutEmail tests from Mock class to List...
Posted by trixpan <gi...@git.apache.org>.
Github user trixpan commented on the issue:
https://github.com/apache/nifi/pull/1553
Thanks for the assistance previously, it should be ready for review.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1553: NIFI-1449 - Migrate PutEmail tests from Mock class ...
Posted by olegz <gi...@git.apache.org>.
Github user olegz commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1553#discussion_r104022318
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestPutEmail.java ---
@@ -230,27 +234,42 @@ public void testOutgoingMessageAttachment() throws Exception {
runner.assertAllFlowFilesTransferred(PutEmail.REL_SUCCESS);
// Verify that the Message was populated correctly
- assertEquals("Expected a single message to be sent", 1, processor.getMessages().size());
- Message message = processor.getMessages().get(0);
- assertEquals("test@apache.org", message.getFrom()[0].toString());
- assertEquals("X-Mailer Header", "TestingNiFi", message.getHeader("X-Mailer")[0]);
- assertEquals("recipient@apache.org", message.getRecipients(RecipientType.TO)[0].toString());
-
- assertTrue(message.getContent() instanceof MimeMultipart);
-
- final MimeMultipart multipart = (MimeMultipart) message.getContent();
- final BodyPart part = multipart.getBodyPart(0);
- final InputStream is = part.getDataHandler().getInputStream();
- final String decodedText = StringUtils.newStringUtf8(Base64.decodeBase64(IOUtils.toString(is, "UTF-8")));
- assertEquals("Message Body", decodedText);
-
- final BodyPart attachPart = multipart.getBodyPart(1);
- final InputStream attachIs = attachPart.getDataHandler().getInputStream();
- final String text = IOUtils.toString(attachIs, "UTF-8");
- assertEquals("Some text", text);
-
- assertNull(message.getRecipients(RecipientType.BCC));
- assertNull(message.getRecipients(RecipientType.CC));
+ assertEquals("Expected a single message to be sent", 1, smtpRunner.getFlowFilesForRelationship(PutEmail.REL_SUCCESS).size());
+
+ MockFlowFile message;
+
+ message = smtpRunner.getFlowFilesForRelationship(PutEmail.REL_SUCCESS).get(0);
+
+ extractHeadersRunner.enqueue(message);
+ extractHeadersRunner.run();
+ extractHeadersRunner.shutdown();
+
+ message = extractHeadersRunner.getFlowFilesForRelationship(ExtractEmailHeaders.REL_SUCCESS).get(0);
+
+ assertEquals("test@apache.org", message.getAttribute("email.headers.from.0"));
+ assertEquals("X-Mailer Header", "TestingNiFi", message.getAttribute("email.headers.x-mailer"));
+
+ // Needed in order to avoid errors like
+ // "already in use for an active callback or InputStream created by ProcessSession.read(FlowFile) has not been closed"
+
+
+ message = extractHeadersRunner.getFlowFilesForRelationship(ExtractEmailHeaders.REL_SUCCESS).get(0);
+
+// extractAttachmentsRunner.enqueue(message);
+// extractAttachmentsRunner.run();
+//
+// final List<MockFlowFile> splits = extractAttachmentsRunner.getFlowFilesForRelationship(ExtractEmailAttachments.REL_ATTACHMENTS);
+//
+// final MimeMultipart multipart = (MimeMultipart) message.getContent();
+// final BodyPart part = multipart.getBodyPart(0);
+// final InputStream is = part.getDataHandler().getInputStream();
+// final String decodedText = StringUtils.newStringUtf8(Base64.decodeBase64(IOUtils.toString(is, "UTF-8")));
+// assertEquals("Message Body", decodedText);
+//
+// final BodyPart attachPart = multipart.getBodyPart(1);
+// final InputStream attachIs = attachPart.getDataHandler().getInputStream();
+// final String text = IOUtils.toString(attachIs, "UTF-8");
+// assertEquals("Some text", text);
--- End diff --
I see. Then we need to look at it as it may be a nasty bug in the processor itself with the way FlowFile(s) interacts with session. The test helped to uncover the issue that IMHO needs to be addressed before the merge. If you want I can take a look at it. LMK.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1553: NIFI-1449 - Migrate PutEmail tests from Mock class to List...
Posted by trixpan <gi...@git.apache.org>.
Github user trixpan commented on the issue:
https://github.com/apache/nifi/pull/1553
@joewitt - Perhaps you would you be able to give me some insight here.
For some reason when I run the following lines of testOutgoingMessageAttachment
```
extractAttachmentsRunner.enqueue(message);
extractAttachmentsRunner.run();
```
I hit the following error message:
```
[pool-5-thread-1] ERROR org.apache.nifi.processors.email.ExtractEmailAttachments - ExtractEmailAttachments[id=a209646b-88f5-4a59-bafb-7aba063ed481] Could not parse the flowfile FlowFile[0,632793727136184.mockFlowFile,917B] as an email, treating as failure: java.lang.IllegalStateException: FlowFile[0,632793806568613.mockFlowFile,0B] already in use for an active callback or InputStream created by ProcessSession.read(FlowFile) has not been closed
[pool-5-thread-1] ERROR org.apache.nifi.processors.email.ExtractEmailAttachments - ExtractEmailAttachments[id=a209646b-88f5-4a59-bafb-7aba063ed481] ExtractEmailAttachments[id=a209646b-88f5-4a59-bafb-7aba063ed481] failed to process due to org.apache.nifi.processor.exception.FlowFileHandlingException: FlowFile[0,632793727136184.mockFlowFile,917B] is not the most recent version of this flow file within this session; rolling back session: org.apache.nifi.processor.exception.FlowFileHandlingException: FlowFile[0,632793727136184.mockFlowFile,917B] is not the most recent version of this flow file within this session
java.lang.AssertionError: org.apache.nifi.processor.exception.FlowFileHandlingException: FlowFile[0,632793727136184.mockFlowFile,917B] is not the most recent version of this flow file within this session
```
Not sure if this is a bug on ExtractEmailHeaders / ExtractEmailAttachments or if it is a mistake within the test script itself?
Cheers
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1553: NIFI-1449 - Migrate PutEmail tests from Mock class ...
Posted by trixpan <gi...@git.apache.org>.
Github user trixpan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1553#discussion_r104143931
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestPutEmail.java ---
@@ -230,27 +234,42 @@ public void testOutgoingMessageAttachment() throws Exception {
runner.assertAllFlowFilesTransferred(PutEmail.REL_SUCCESS);
// Verify that the Message was populated correctly
- assertEquals("Expected a single message to be sent", 1, processor.getMessages().size());
- Message message = processor.getMessages().get(0);
- assertEquals("test@apache.org", message.getFrom()[0].toString());
- assertEquals("X-Mailer Header", "TestingNiFi", message.getHeader("X-Mailer")[0]);
- assertEquals("recipient@apache.org", message.getRecipients(RecipientType.TO)[0].toString());
-
- assertTrue(message.getContent() instanceof MimeMultipart);
-
- final MimeMultipart multipart = (MimeMultipart) message.getContent();
- final BodyPart part = multipart.getBodyPart(0);
- final InputStream is = part.getDataHandler().getInputStream();
- final String decodedText = StringUtils.newStringUtf8(Base64.decodeBase64(IOUtils.toString(is, "UTF-8")));
- assertEquals("Message Body", decodedText);
-
- final BodyPart attachPart = multipart.getBodyPart(1);
- final InputStream attachIs = attachPart.getDataHandler().getInputStream();
- final String text = IOUtils.toString(attachIs, "UTF-8");
- assertEquals("Some text", text);
-
- assertNull(message.getRecipients(RecipientType.BCC));
- assertNull(message.getRecipients(RecipientType.CC));
+ assertEquals("Expected a single message to be sent", 1, smtpRunner.getFlowFilesForRelationship(PutEmail.REL_SUCCESS).size());
+
+ MockFlowFile message;
+
+ message = smtpRunner.getFlowFilesForRelationship(PutEmail.REL_SUCCESS).get(0);
+
+ extractHeadersRunner.enqueue(message);
+ extractHeadersRunner.run();
+ extractHeadersRunner.shutdown();
+
+ message = extractHeadersRunner.getFlowFilesForRelationship(ExtractEmailHeaders.REL_SUCCESS).get(0);
+
+ assertEquals("test@apache.org", message.getAttribute("email.headers.from.0"));
+ assertEquals("X-Mailer Header", "TestingNiFi", message.getAttribute("email.headers.x-mailer"));
+
+ // Needed in order to avoid errors like
+ // "already in use for an active callback or InputStream created by ProcessSession.read(FlowFile) has not been closed"
+
+
+ message = extractHeadersRunner.getFlowFilesForRelationship(ExtractEmailHeaders.REL_SUCCESS).get(0);
+
+// extractAttachmentsRunner.enqueue(message);
+// extractAttachmentsRunner.run();
+//
+// final List<MockFlowFile> splits = extractAttachmentsRunner.getFlowFilesForRelationship(ExtractEmailAttachments.REL_ATTACHMENTS);
+//
+// final MimeMultipart multipart = (MimeMultipart) message.getContent();
+// final BodyPart part = multipart.getBodyPart(0);
+// final InputStream is = part.getDataHandler().getInputStream();
+// final String decodedText = StringUtils.newStringUtf8(Base64.decodeBase64(IOUtils.toString(is, "UTF-8")));
+// assertEquals("Message Body", decodedText);
+//
+// final BodyPart attachPart = multipart.getBodyPart(1);
+// final InputStream attachIs = attachPart.getDataHandler().getInputStream();
+// final String text = IOUtils.toString(attachIs, "UTF-8");
+// assertEquals("Some text", text);
--- End diff --
@olegz no luck solving the issue. Also, based on some testing where I bypassed ExtractEmailHeaders, this does not seem to be directly related to the interaction between ExtractEmailHeaders and ExtractEmailAttachments
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1553: NIFI-1449 - Migrate PutEmail tests from Mock class ...
Posted by trixpan <gi...@git.apache.org>.
Github user trixpan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1553#discussion_r104027988
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestPutEmail.java ---
@@ -102,13 +91,16 @@ public void testExceptionWhenSending() {
runner.assertQueueEmpty();
runner.assertAllFlowFilesTransferred(PutEmail.REL_FAILURE);
- assertEquals("Expected an attempt to send a single message", 1, processor.getMessages().size());
+ List<MockFlowFile> results = runner.getFlowFilesForRelationship(PutEmail.REL_FAILURE);
+ assertEquals("Expected an attempt to send a single message", 1, results.size());
}
@Test
public void testOutgoingMessage() throws Exception {
// verifies that are set on the outgoing Message correctly
- runner.setProperty(PutEmail.SMTP_HOSTNAME, "smtp-host");
+ runner.setProperty(PutEmail.SMTP_AUTH, "false");
+ runner.setProperty(PutEmail.SMTP_HOSTNAME, "localhost");
+ runner.setProperty(PutEmail.SMTP_PORT, String.valueOf(port));
runner.setProperty(PutEmail.HEADER_XMAILER, "TestingNiFi");
runner.setProperty(PutEmail.FROM, "test@apache.org");
--- End diff --
I just re-used the previously used email. Happy to change to something under the nifi subdomain
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1553: NIFI-1449 - Migrate PutEmail tests from Mock class ...
Posted by trixpan <gi...@git.apache.org>.
Github user trixpan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1553#discussion_r104027407
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestPutEmail.java ---
@@ -230,27 +234,42 @@ public void testOutgoingMessageAttachment() throws Exception {
runner.assertAllFlowFilesTransferred(PutEmail.REL_SUCCESS);
// Verify that the Message was populated correctly
- assertEquals("Expected a single message to be sent", 1, processor.getMessages().size());
- Message message = processor.getMessages().get(0);
- assertEquals("test@apache.org", message.getFrom()[0].toString());
- assertEquals("X-Mailer Header", "TestingNiFi", message.getHeader("X-Mailer")[0]);
- assertEquals("recipient@apache.org", message.getRecipients(RecipientType.TO)[0].toString());
-
- assertTrue(message.getContent() instanceof MimeMultipart);
-
- final MimeMultipart multipart = (MimeMultipart) message.getContent();
- final BodyPart part = multipart.getBodyPart(0);
- final InputStream is = part.getDataHandler().getInputStream();
- final String decodedText = StringUtils.newStringUtf8(Base64.decodeBase64(IOUtils.toString(is, "UTF-8")));
- assertEquals("Message Body", decodedText);
-
- final BodyPart attachPart = multipart.getBodyPart(1);
- final InputStream attachIs = attachPart.getDataHandler().getInputStream();
- final String text = IOUtils.toString(attachIs, "UTF-8");
- assertEquals("Some text", text);
-
- assertNull(message.getRecipients(RecipientType.BCC));
- assertNull(message.getRecipients(RecipientType.CC));
+ assertEquals("Expected a single message to be sent", 1, smtpRunner.getFlowFilesForRelationship(PutEmail.REL_SUCCESS).size());
+
+ MockFlowFile message;
+
+ message = smtpRunner.getFlowFilesForRelationship(PutEmail.REL_SUCCESS).get(0);
+
+ extractHeadersRunner.enqueue(message);
+ extractHeadersRunner.run();
+ extractHeadersRunner.shutdown();
+
+ message = extractHeadersRunner.getFlowFilesForRelationship(ExtractEmailHeaders.REL_SUCCESS).get(0);
+
+ assertEquals("test@apache.org", message.getAttribute("email.headers.from.0"));
+ assertEquals("X-Mailer Header", "TestingNiFi", message.getAttribute("email.headers.x-mailer"));
+
+ // Needed in order to avoid errors like
+ // "already in use for an active callback or InputStream created by ProcessSession.read(FlowFile) has not been closed"
+
+
+ message = extractHeadersRunner.getFlowFilesForRelationship(ExtractEmailHeaders.REL_SUCCESS).get(0);
+
+// extractAttachmentsRunner.enqueue(message);
+// extractAttachmentsRunner.run();
+//
+// final List<MockFlowFile> splits = extractAttachmentsRunner.getFlowFilesForRelationship(ExtractEmailAttachments.REL_ATTACHMENTS);
+//
+// final MimeMultipart multipart = (MimeMultipart) message.getContent();
+// final BodyPart part = multipart.getBodyPart(0);
+// final InputStream is = part.getDataHandler().getInputStream();
+// final String decodedText = StringUtils.newStringUtf8(Base64.decodeBase64(IOUtils.toString(is, "UTF-8")));
+// assertEquals("Message Body", decodedText);
+//
+// final BodyPart attachPart = multipart.getBodyPart(1);
+// final InputStream attachIs = attachPart.getDataHandler().getInputStream();
+// final String text = IOUtils.toString(attachIs, "UTF-8");
+// assertEquals("Some text", text);
--- End diff --
@olegz,woyld appreciate if you could have a look. I had it a go yesterday evening but the reason wasn't obvious to me (tbh it was 1am when I noted it so you get the picture...)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---