You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "Jeroen Dries (JIRA)" <ji...@apache.org> on 2018/05/02 09:19:00 UTC
[jira] [Commented] (NIFI-3511) Searching High Volume Provenance
Seemingly Applies Arbitrary Range Restriction
[ https://issues.apache.org/jira/browse/NIFI-3511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460762#comment-16460762 ]
Jeroen Dries commented on NIFI-3511:
------------------------------------
I seem to have a similar situation, where I do not see any provenance data, when viewing it through the context menu of a processor. If I however click the 'search' icon in that window, and search again, it does find results.
In the second case, a time range is added to the query.
First query, no results:
{code:java}
{"provenance":{"id":"20164253-0163-1000-ede7-4d45d1b1555e","uri":"https://dataflow-prod.vgt.vito.be:8443/nifi-api/provenance/20164253-0163-1000-ede7-4d45d1b1555e","submissionTime":"05/02/2018 11:01:59.764 CEST","expiration":"05/02/2018 11:31:59.836 CEST","percentCompleted":100,"finished":true,"request":{"searchTerms":{"Component ID":"8d7b3f7a-917b-32a8-fddd-2b0508a47e7d"},"maxResults":1000},"results":{"provenanceEvents":[],"total":"0","totalCount":0,"generated":"11:01:59 CEST","oldestEvent":"04/18/2018 11:01:09 CEST","timeOffset":7200000}}}{code}
Second query, the search dialog added a time range filter:
{code:java}
{"provenance":{"id":"2017060f-0163-1000-9b31-223c733b5d55","uri":"https://dataflow-prod.vgt.vito.be:8443/nifi-api/provenance/2017060f-0163-1000-9b31-223c733b5d55","submissionTime":"05/02/2018 11:02:49.871 CEST","expiration":"05/02/2018 11:32:49.973 CEST","percentCompleted":100,"finished":true,"request":{"searchTerms":{"Component ID":"8d7b3f7a-917b-32a8-fddd-2b0508a47e7d"},"startDate":"05/02/2018 00:00:00 CEST","endDate":"05/02/2018 23:59:59 CEST","maxResults":1000},"results":{"provenanceEvents":[{"id":"2376288","eventId":2376288,"eventTime":"05/02/2018 09:21:56.686 CEST","eventType":"CONTENT_MODIFIED","flowFileUuid":"3dc06b4a-7179-49ee-9909-488c99b4f38e","fileSize":"63 bytes","fileSizeBytes":63,"groupId":"a45fb614-a399-3b4a-e55e-e9c936700ebe","componentId":"8d7b3f7a-917b-32a8-fddd-2b0508a47e7d","componentType":"ReplaceText","componentName":"SetStatus"},{"id":"2375759","eventId":2375759,"eventTime":"05/02/2018 09:13:44.175 CEST","eventType":"CONTENT_MODIFIED","flowFileUuid":"79d0ed74-5471-49b7-8b78-e02898845582","fileSize":"63 bytes","fileSizeBytes":63,"groupId":"a45fb614-a399-3b4a-e55e-e9c936700ebe","componentId":"8d7b3f7a-917b-32a8-fddd-2b0508a47e7d","componentType":"ReplaceText","componentName":"SetStatus"},{"id":"2374698","eventId":2374698,"eventTime":"05/02/2018 08:58:22.156 CEST","eventType":"CONTENT_MODIFIED","flowFileUuid":"49762410-9adf-41b7-b0b5-f06623986ebc","fileSize":"63 bytes","fileSizeBytes":63,"groupId":"a45fb614-a399-3b4a-e55e-e9c936700ebe","componentId":"8d7b3f7a-917b-32a8-fddd-2b0508a47e7d","componentType":"ReplaceText","componentName":"SetStatus"},{"id":"2363246","eventId":2363246,"eventTime":"05/02/2018 05:20:13.162 CEST","eventType":"CONTENT_MODIFIED","flowFileUuid":"4469fe89-aaaf-4d31-8178-da4757e7779b","fileSize":"63 bytes","fileSizeBytes":63,"groupId":"a45fb614-a399-3b4a-e55e-e9c936700ebe","componentId":"8d7b3f7a-917b-32a8-fddd-2b0508a47e7d","componentType":"ReplaceText","componentName":"SetStatus"},{"id":"2363245","eventId":2363245,"eventTime":"05/02/2018 05:20:13.161 CEST","eventType":"CONTENT_MODIFIED","flowFileUuid":"7783a038-46a5-442c-bf01-2c6630c7a3dd","fileSize":"63 bytes","fileSizeBytes":63,"groupId":"a45fb614-a399-3b4a-e55e-e9c936700ebe","componentId":"8d7b3f7a-917b-32a8-fddd-2b0508a47e7d","componentType":"ReplaceText","componentName":"SetStatus"},{"id":"2362110","eventId":2362110,"eventTime":"05/02/2018 05:01:22.591 CEST","eventType":"CONTENT_MODIFIED","flowFileUuid":"49d89d69-2586-4c7a-84a7-ffc0a3ba3cae","fileSize":"63 bytes","fileSizeBytes":63,"groupId":"a45fb614-a399-3b4a-e55e-e9c936700ebe","componentId":"8d7b3f7a-917b-32a8-fddd-2b0508a47e7d","componentType":"ReplaceText","componentName":"SetStatus"},
...
,{"id":"2353289","eventId":2353289,"eventTime":"05/02/2018 02:10:54.191 CEST","eventType":"CONTENT_MODIFIED","flowFileUuid":"2223e1e3-555c-41d9-be46-2e02e44cac5a","fileSize":"63 bytes","fileSizeBytes":63,"groupId":"a45fb614-a399-3b4a-e55e-e9c936700ebe","componentId":"8d7b3f7a-917b-32a8-fddd-2b0508a47e7d","componentType":"ReplaceText","componentName":"SetStatus"}],"total":"23","totalCount":23,"generated":"11:02:49 CEST","oldestEvent":"04/18/2018 11:01:09 CEST","timeOffset":7200000}}}{code}
> Searching High Volume Provenance Seemingly Applies Arbitrary Range Restriction
> -------------------------------------------------------------------------------
>
> Key: NIFI-3511
> URL: https://issues.apache.org/jira/browse/NIFI-3511
> Project: Apache NiFi
> Issue Type: Bug
> Reporter: Ricky Saltzer
> Assignee: Ricky Saltzer
> Priority: Major
>
> This issue seems to happen in environments which have a large volume of FlowFiles, or in our case, a single FlowFile that is expanded into several hundred thousand FlowFiles (several hundred a day).
> There are times when we need to validate a certain FlowFile passed through our data flow. It appears that even though we search on a specific attribute that did in fact pass through the data flow within the time range specified, it some how misses being searched unless we make the time range smaller.
> Does query runner stops searching after hitting some sort of limit?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)