You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by pvillard31 <gi...@git.apache.org> on 2016/04/13 11:31:59 UTC
[GitHub] nifi pull request: NIFI-1755 Fixed remote process group status cou...
GitHub user pvillard31 opened a pull request:
https://github.com/apache/nifi/pull/347
NIFI-1755 Fixed remote process group status counts by only considering connected remote ports
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/pvillard31/nifi NIFI-1755
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/347.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #347
----
commit 147f528eb8e5e7a4da5f6e49c5f16657227d5e58
Author: Pierre Villard <pi...@gmail.com>
Date: 2016-04-13T09:31:23Z
NIFI-1755 Fixed remote process group status counts by only considering connected remote ports
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1755 Fixed remote process group status cou...
Posted by olegz <gi...@git.apache.org>.
Github user olegz commented on the pull request:
https://github.com/apache/nifi/pull/347#issuecomment-217000289
@pvillard31 below is the test you can use to validate this issue. Basically if you execute it on current master it will fail the assertions and then with your changes it will succeed.
Also keep in mind that you would need a second PR (one as an update to this one and another for 0.x branch that is essentially 0.7.0-SNAPSHOT). The reason being is that _UserService_ was migrated to _KeyService_ so the test below won't compile on 0.x where you still have to use UserService. I am sure you'll figure it out. Also, feel free to change the test class/method names to be more reflective what's being tested.
Cheers
```java
package org.apache.nifi.controller;
import static org.junit.Assert.assertEquals;
import static org.mockito.Mockito.mock;
import java.net.URL;
import java.util.Collections;
import org.apache.nifi.admin.service.AuditService;
import org.apache.nifi.admin.service.KeyService;
import org.apache.nifi.controller.repository.RepositoryStatusReport;
import org.apache.nifi.controller.repository.RingBufferEventRepository;
import org.apache.nifi.controller.repository.StandardFlowFileEvent;
import org.apache.nifi.controller.repository.StandardRepositoryStatusReport;
import org.apache.nifi.controller.status.ProcessGroupStatus;
import org.apache.nifi.groups.ProcessGroup;
import org.apache.nifi.groups.RemoteProcessGroup;
import org.apache.nifi.groups.RemoteProcessGroupPortDescriptor;
import org.apache.nifi.provenance.MockProvenanceEventRepository;
import org.apache.nifi.remote.RemoteGroupPort;
import org.apache.nifi.remote.StandardRemoteProcessGroupPortDescriptor;
import org.apache.nifi.util.NiFiProperties;
import org.junit.After;
import org.junit.Before;
import org.junit.BeforeClass;
import org.junit.Test;
public class RPGStatusTest {
private final KeyService userService = mock(KeyService.class);
private final AuditService auditService = mock(AuditService.class);
private volatile FlowController controller;
@BeforeClass
public static void beforeClass() {
try {
URL url = ClassLoader.getSystemClassLoader().getResource("nifi.properties");
System.setProperty("nifi.properties.file.path", url.getFile());
} catch (Exception e) {
throw new IllegalStateException("Failed to discover nifi.properties at the root of the classpath", e);
}
}
@Before
public void before() {
NiFiProperties properties = NiFiProperties.getInstance();
properties.setProperty(NiFiProperties.PROVENANCE_REPO_IMPLEMENTATION_CLASS, MockProvenanceEventRepository.class.getName());
properties.setProperty(NiFiProperties.STATE_MANAGEMENT_CONFIG_FILE, "src/test/resources/state-management.xml");
properties.setProperty(NiFiProperties.STATE_MANAGEMENT_LOCAL_PROVIDER_ID, "local-provider");
properties.setProperty(NiFiProperties.REMOTE_INPUT_HOST, "localhost");
properties.setProperty("nifi.remote.input.secure", "false");
RingBufferEventRepository repository = new RingBufferEventRepository(1);
this.controller = FlowController.createStandaloneInstance(repository, NiFiProperties.getInstance(),
this.userService, this.auditService, null);
}
@After
public void after() {
this.controller.shutdown(false);
}
@Test
public void testPr347() throws Exception {
ProcessGroup receiverGroup = controller.createProcessGroup("SITE");
NiFiTestUtils.setControllerRootGroup(this.controller, receiverGroup);
RemoteProcessGroup remoteProcessGroup = controller.createRemoteProcessGroup("SENDER_REMOTE","http://foo:1234/nifi");
receiverGroup.addRemoteProcessGroup(remoteProcessGroup);
String inputPortId = "inputId";
StandardRemoteProcessGroupPortDescriptor inputPortDescriptor = new StandardRemoteProcessGroupPortDescriptor();
inputPortDescriptor.setId(inputPortId);
inputPortDescriptor.setName("inputPort");
remoteProcessGroup.setInputPorts(Collections.<RemoteProcessGroupPortDescriptor> singleton(inputPortDescriptor));
RemoteGroupPort inputPort = remoteProcessGroup.getInputPort(inputPortId);
inputPort.setProcessGroup(receiverGroup);
String outputPortId = "outputId";
StandardRemoteProcessGroupPortDescriptor outputPortDescriptor = new StandardRemoteProcessGroupPortDescriptor();
outputPortDescriptor.setId(outputPortId);
outputPortDescriptor.setName("outputPport");
remoteProcessGroup.setOutputPorts(Collections.<RemoteProcessGroupPortDescriptor> singleton(outputPortDescriptor));
RemoteGroupPort outputPort = remoteProcessGroup.getOutputPort(outputPortId);
outputPort.setProcessGroup(receiverGroup);
RepositoryStatusReport rp = new StandardRepositoryStatusReport();
StandardFlowFileEvent inputEvent = new StandardFlowFileEvent(inputPortId);
inputEvent.setBytesSent(1234);
rp.addReportEntry(inputEvent);
StandardFlowFileEvent outputEvent = new StandardFlowFileEvent(outputPortId);
outputEvent.setFlowFilesReceived(45);
rp.addReportEntry(outputEvent);
ProcessGroupStatus status = controller.getGroupStatus(receiverGroup, rp);
assertEquals(0, status.getFlowFilesReceived());
assertEquals(0, status.getBytesSent());
}
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1755 Fixed remote process group status cou...
Posted by pvillard31 <gi...@git.apache.org>.
Github user pvillard31 commented on the pull request:
https://github.com/apache/nifi/pull/347#issuecomment-216631894
@olegz I must admit I tried to use your code as mentioned in your link and got it working but I didn't find a way to validate the change since it seems to simulate S2S without all the code around about statistics and so on. Everything is quite "nested" and I didn't find a way to get the expected result. (Speaking of that, I saw a new JIRA NIFI-1820 (in addition to NIFI-1331) to address this part). If you want to have a look, that would be great (and I am sure I'll learn a lot!).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1755 Fixed remote process group status cou...
Posted by olegz <gi...@git.apache.org>.
Github user olegz commented on the pull request:
https://github.com/apache/nifi/pull/347#issuecomment-219489958
merging
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1755 Fixed remote process group status cou...
Posted by pvillard31 <gi...@git.apache.org>.
Github user pvillard31 commented on the pull request:
https://github.com/apache/nifi/pull/347#issuecomment-218867090
@olegz Thanks Oleg! Not exactly what I had in mind but it does the trick :) (I was trying to set up an environment with two simple processors sending flow files to RPG with different frequencies and ensure the correct values of counters in RPG status). This PR is updated for master branch and I just sent a PR for 0.x branch.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1755 Fixed remote process group status cou...
Posted by pvillard31 <gi...@git.apache.org>.
Github user pvillard31 commented on the pull request:
https://github.com/apache/nifi/pull/347#issuecomment-211283811
@olegz I agree a unit test would be nice, but I must admit that I am not sure to see how to do it. It involves RPG, remote ports, remote processors and I don't see how I can mock the whole remote flow to get the expected testing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1755 Fixed remote process group status cou...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/nifi/pull/347
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1755 Fixed remote process group status cou...
Posted by olegz <gi...@git.apache.org>.
Github user olegz commented on the pull request:
https://github.com/apache/nifi/pull/347#issuecomment-213020240
@pvillard31 Actually I totally forgot about the effort I started a while back. There is actually a branch for it https://github.com/olegz/nifi/tree/int-test/nifi-integration-tests. Not much there but S2S test is (totally headless).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1755 Fixed remote process group status cou...
Posted by olegz <gi...@git.apache.org>.
Github user olegz commented on the pull request:
https://github.com/apache/nifi/pull/347#issuecomment-212995061
@pvillard31 let me dig some more. Yes you have to have the "other side" running but I had some other code bits to simulate that. I'll post here once I find
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1755 Fixed remote process group status cou...
Posted by olegz <gi...@git.apache.org>.
Github user olegz commented on the pull request:
https://github.com/apache/nifi/pull/347#issuecomment-211417082
@pvillard31 I was gonna play with it a bit more and possibly create the test, but getting tied up with other things. Yet I was able to dig up one of my experiments that I used to play with RPGs in unit test environment, and since you already familiar with FlowController, you may make sense of it. Anyway, let me know.
```java
System.setProperty("nifi.properties.file.path",
"/Users/ozhurakousky/dev/nifi-integration/src/main/resources/nifi.properties");
NiFiProperties properties = NiFiProperties.getInstance();
RingBufferEventRepository repository = new RingBufferEventRepository(1);
UserService us = mock(UserService.class);
AuditService as = mock(AuditService.class);
FlowController controller = FlowController.createStandaloneInstance(repository, properties, us, as, null);
ProcessGroup senderGroup = controller.createProcessGroup("SENDER");
senderGroup.setName("SENDER");
RemoteProcessGroup remoteProcessGroup = controller.createRemoteProcessGroup("SENDER_REMOTE", "http://localhost:8080/nifi");
senderGroup.addRemoteProcessGroup(remoteProcessGroup);
// INPUT PORT (DESTINATIOIN)
StandardRemoteProcessGroupPortDescriptor inputPortDescriptor = new StandardRemoteProcessGroupPortDescriptor();
inputPortDescriptor.setId("67b1950c-554e-4cb6-92fb-7e313b015e8c");
inputPortDescriptor.setName("in");
remoteProcessGroup.setInputPorts(Collections.<RemoteProcessGroupPortDescriptor> singleton(inputPortDescriptor));
RemoteGroupPort inputPort = remoteProcessGroup.getInputPort("67b1950c-554e-4cb6-92fb-7e313b015e8c");
inputPort.setProcessGroup(senderGroup);
System.out.println(inputPort.isRunning());
System.out.println(inputPort.isTargetRunning());
System.out.println(inputPort.isValid());
// SOURCE
ProcessorNode source = controller.createProcessor(GenerateFlowFile.class.getName(), "A");
source.setProperty("File Size", "10 b");
source.setScheduldingPeriod("2 sec");
source.setProcessGroup(senderGroup);
senderGroup.addProcessor(source);
// CONNECTION
Connection connection = controller.createConnection("MyConnectionId", "MyConnectionName", source,
remoteProcessGroup.getInputPort("67b1950c-554e-4cb6-92fb-7e313b015e8c"),
Collections.singletonList("success"));
connection.setProcessGroup(senderGroup);
senderGroup.addConnection(connection);
inputPort.addConnection(connection);
remoteProcessGroup.startTransmitting();
// inputPort.verifyCanStart();
// senderGroup.addInputPort(inputPort);
// senderGroup.startInputPort(inputPort);
//controller.startTransmitting(remoteProcessGroup.getInputPort("in"));
senderGroup.startProcessor(source);
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1755 Fixed remote process group status cou...
Posted by olegz <gi...@git.apache.org>.
Github user olegz commented on the pull request:
https://github.com/apache/nifi/pull/347#issuecomment-216635676
Sure, I'll give it a shot
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1755 Fixed remote process group status cou...
Posted by olegz <gi...@git.apache.org>.
Github user olegz commented on the pull request:
https://github.com/apache/nifi/pull/347#issuecomment-216612204
@pvillard31 Are you planning to add tests for this? If not I can take a crack at it. LMK.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1755 Fixed remote process group status cou...
Posted by olegz <gi...@git.apache.org>.
Github user olegz commented on the pull request:
https://github.com/apache/nifi/pull/347#issuecomment-211126901
@pvillard31 Changes look good, but was wondering if you could add some tests
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1755 Fixed remote process group status cou...
Posted by olegz <gi...@git.apache.org>.
Github user olegz commented on the pull request:
https://github.com/apache/nifi/pull/347#issuecomment-211324090
@pvillard31 right, not the simplest thing to do, but let me dig as I believe I have something. Will let you know
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1755 Fixed remote process group status cou...
Posted by pvillard31 <gi...@git.apache.org>.
Github user pvillard31 commented on the pull request:
https://github.com/apache/nifi/pull/347#issuecomment-212876061
@olegz Thanks for the code. However if I try to run it, it will try to connect to localhost:8080/nifi and will fail:
594 [NiFi Site-to-Site Connection Pool Maintenance] WARN org.apache.nifi.remote.client.socket.EndpointConnectionPool - EndpointConnectionPool[Cluster URL=http://localhost:8080/nifi] Unable to refresh Remote Group's peers due to {}
java.net.ConnectException: Connection refused
It makes sense since there is no NiFi instance running. Did you have in mind to really have an instance running or to simulate the remote end in some ways?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---