You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by markap14 <gi...@git.apache.org> on 2016/08/02 13:57:07 UTC

[GitHub] nifi pull request #771: NIFI-2452

GitHub user markap14 opened a pull request:

    https://github.com/apache/nifi/pull/771

    NIFI-2452

    Ensure that we keep track of how many references we have to each lucene searcher and only close the underlying index reader if there are no references to the searcher.  Also updated to prefer newer provenance events over older provenance events, and calculate FlowFile lineage based on an event id instead of a FlowFile UUID, as it's much more efficient

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/markap14/nifi NIFI-2452

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/nifi/pull/771.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #771
    
----
commit 3595b8bbc590d98ff1ad939ca2bd022b257307b6
Author: Mark Payne <ma...@hotmail.com>
Date:   2016-08-01T18:51:02Z

    NIFI-2452: Ensure that we do not close Index Readers that are still in use

commit 87100428902db677a014a3936f2dda4aceee5dd9
Author: Mark Payne <ma...@hotmail.com>
Date:   2016-08-02T13:56:05Z

    NIFI-2452: Ensure that we keep track of how many references we have to each lucene searcher and only close the underlying index reader if there are no references to the searcher.  Also updated to prefer newer provenance events over older provenance events, and calculate FlowFile lineage based on an event id instead of a FlowFile UUID, as it's much more efficient

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #771: NIFI-2452

Posted by joewitt <gi...@git.apache.org>.
Github user joewitt commented on a diff in the pull request:

    https://github.com/apache/nifi/pull/771#discussion_r73170934
  
    --- Diff: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/heartbeat/AbstractHeartbeatMonitor.java ---
    @@ -203,16 +203,24 @@ private void processHeartbeat(final NodeHeartbeat heartbeat) {
                 final DisconnectionCode disconnectionCode = connectionStatus.getDisconnectCode();
     
                 // Determine whether or not the node should be allowed to be in the cluster still, depending on its reason for disconnection.
    -            if (disconnectionCode == DisconnectionCode.LACK_OF_HEARTBEAT || disconnectionCode == DisconnectionCode.UNABLE_TO_COMMUNICATE) {
    --- End diff --
    
    was this heartbeat stuff supposed to be in this PR?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #771: NIFI-2452

Posted by joewitt <gi...@git.apache.org>.
Github user joewitt commented on a diff in the pull request:

    https://github.com/apache/nifi/pull/771#discussion_r73171377
  
    --- Diff: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/FlowController.java ---
    @@ -16,7 +16,38 @@
      */
     package org.apache.nifi.controller;
     
    -import com.sun.jersey.api.client.ClientHandlerException;
    --- End diff --
    
    this set of changes also appears to not belong in this PR?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #771: NIFI-2452

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/nifi/pull/771


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #771: NIFI-2452

Posted by joewitt <gi...@git.apache.org>.
Github user joewitt commented on a diff in the pull request:

    https://github.com/apache/nifi/pull/771#discussion_r73173007
  
    --- Diff: nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java ---
    @@ -2166,6 +2171,28 @@ private Lineage computeLineage(final Collection<String> flowFileUuids, final NiF
         }
     
         @Override
    +    public ComputeLineageSubmission submitLineageComputation(final long eventId, final NiFiUser user) {
    +        final ProvenanceEventRecord event;
    +        try {
    +            event = getEvent(eventId);
    +        } catch (final Exception e) {
    +            logger.error("Failed to retrieve Provenance Event with ID " + eventId + " to calculate data lineage due to: " + e, e);
    +            final AsyncLineageSubmission result = new AsyncLineageSubmission(LineageComputationType.FLOWFILE_LINEAGE, eventId, Collections.<String> emptySet(), 1, user.getIdentity());
    +            result.getResult().setError("Failed to retrieve Provenance Event with ID " + eventId + ". See logs for more information.");
    +            return result;
    +        }
    +
    +        if (event == null) {
    +            final AsyncLineageSubmission result = new AsyncLineageSubmission(LineageComputationType.FLOWFILE_LINEAGE, eventId, Collections.<String> emptySet(), 1, user.getIdentity());
    +            result.getResult().setError("Could not find Provenance Event with ID " + eventId);
    +            lineageSubmissionMap.put(result.getLineageIdentifier(), result);
    +            return result;
    +        }
    +
    +        return submitLineageComputation(Collections.singleton(event.getFlowFileUuid()), user, LineageComputationType.FLOWFILE_LINEAGE, eventId, event.getLineageStartDate(), Long.MAX_VALUE);
    --- End diff --
    
    so first we search on the eventId then we search on the flowfile itself.  Can you describe why/what is happening there?  Why isn't the first query sufficient?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #771: NIFI-2452

Posted by markap14 <gi...@git.apache.org>.
Github user markap14 commented on a diff in the pull request:

    https://github.com/apache/nifi/pull/771#discussion_r73220717
  
    --- Diff: nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java ---
    @@ -2166,6 +2171,28 @@ private Lineage computeLineage(final Collection<String> flowFileUuids, final NiF
         }
     
         @Override
    +    public ComputeLineageSubmission submitLineageComputation(final long eventId, final NiFiUser user) {
    +        final ProvenanceEventRecord event;
    +        try {
    +            event = getEvent(eventId);
    +        } catch (final Exception e) {
    +            logger.error("Failed to retrieve Provenance Event with ID " + eventId + " to calculate data lineage due to: " + e, e);
    +            final AsyncLineageSubmission result = new AsyncLineageSubmission(LineageComputationType.FLOWFILE_LINEAGE, eventId, Collections.<String> emptySet(), 1, user.getIdentity());
    +            result.getResult().setError("Failed to retrieve Provenance Event with ID " + eventId + ". See logs for more information.");
    +            return result;
    +        }
    +
    +        if (event == null) {
    +            final AsyncLineageSubmission result = new AsyncLineageSubmission(LineageComputationType.FLOWFILE_LINEAGE, eventId, Collections.<String> emptySet(), 1, user.getIdentity());
    +            result.getResult().setError("Could not find Provenance Event with ID " + eventId);
    +            lineageSubmissionMap.put(result.getLineageIdentifier(), result);
    +            return result;
    +        }
    +
    +        return submitLineageComputation(Collections.singleton(event.getFlowFileUuid()), user, LineageComputationType.FLOWFILE_LINEAGE, eventId, event.getLineageStartDate(), Long.MAX_VALUE);
    --- End diff --
    
    When we obtain the event by ID above, we are not querying anything - we are simply looking up the event directly by ID. We then formulate a query based on the UUID of the FlowFile. This is different than passing in the UUID of the FlowFile directly, though, because when we look up the Event directly, we also have the Lineage Start Date. Knowing that Lineage Start Date can dramatically reduce the amount of searching required if the start date is fairly recent.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #771: NIFI-2452

Posted by markap14 <gi...@git.apache.org>.
Github user markap14 commented on a diff in the pull request:

    https://github.com/apache/nifi/pull/771#discussion_r73220870
  
    --- Diff: nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/test/java/org/apache/nifi/provenance/TestPersistentProvenanceRepository.java ---
    @@ -481,6 +485,165 @@ public void testIndexAndCompressOnRolloverAndSubsequentSearch() throws IOExcepti
             assertTrue(newRecordSet.getMatchingEvents().isEmpty());
         }
     
    +    // TODO: Switch to 10,000.
    --- End diff --
    
    Whoops! Overlooked that. Was just a comment that I added in so that if i forgot to change the timeout back to 10,000 millis, the reviewer would catch it - glad it worked! :) Made it much longer so that i could set a breakpoint. Will change back to 10,000 millis and update the PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #771: NIFI-2452

Posted by joewitt <gi...@git.apache.org>.
Github user joewitt commented on a diff in the pull request:

    https://github.com/apache/nifi/pull/771#discussion_r73174984
  
    --- Diff: nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/test/java/org/apache/nifi/provenance/TestPersistentProvenanceRepository.java ---
    @@ -481,6 +485,165 @@ public void testIndexAndCompressOnRolloverAndSubsequentSearch() throws IOExcepti
             assertTrue(newRecordSet.getMatchingEvents().isEmpty());
         }
     
    +    // TODO: Switch to 10,000.
    --- End diff --
    
    TODO Reminder?  Do you still want to do?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #771: NIFI-2452

Posted by markap14 <gi...@git.apache.org>.
Github user markap14 commented on a diff in the pull request:

    https://github.com/apache/nifi/pull/771#discussion_r73220319
  
    --- Diff: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/FlowController.java ---
    @@ -16,7 +16,38 @@
      */
     package org.apache.nifi.controller;
     
    -import com.sun.jersey.api.client.ClientHandlerException;
    --- End diff --
    
    @joewitt this change was due to a bug that I ran into when testing this PR. The bug was preventing me from moving forward, and was a rather trivial fix, so I just included it in the PR. I could have created a separate JIRA and PR but since it would have blocked this PR it seemed more prudent to just roll it in.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---