You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by GitBox <gi...@apache.org> on 2020/02/22 01:01:00 UTC
[GitHub] [nifi] MikeThomsen opened a new pull request #4072: Nifi 7182
MikeThomsen opened a new pull request #4072: Nifi 7182
URL: https://github.com/apache/nifi/pull/4072
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
#### Description of PR
_Enables X functionality; fixes bug NIFI-YYYY._
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with **NIFI-XXXX** where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [ ] Has your PR been rebased against the latest commit within the target branch (typically `master`)?
- [ ] Is your initial contribution a single, squashed commit? _Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not `squash` or use `--force` when pushing to allow for clean monitoring of changes._
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via `mvn -Pcontrib-check clean install` at the root `nifi` folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] Have you verified that the full build is successful on both JDK 8 and JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main `LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main `NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to .name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] ijokarumawak commented on issue #4072: NIFI-7182
Posted by GitBox <gi...@apache.org>.
ijokarumawak commented on issue #4072: NIFI-7182
URL: https://github.com/apache/nifi/pull/4072#issuecomment-614473511
@MikeThomsen @joewitt
The test is using Solr range facetting which depends on system date. Specifically the test uses `facet.range.start=NOW/MINUTE` and `facet.range.end=NOW/MINUTE+1MINUTE`.
That range condition can be fail if the minutes when the processor executes the query, and the date field value are different. That can be happen by a long GC pause or in an extremely bad timing.
For example, if `createSolrClient()` method creates the doc at `10:23:59.999`, and the query is executed at `10:24:001`, then the facet range would be `10:24 to 10:25` which does not query documents created at `10:23`.
I was able to reproduce the assertion error by addint `Thread.sleep(60_000)` at the `testAllFacet Categories` method between it creates `solorClient` and `runner.run`.
Since the test does not have to filter documents by date range, we can make the range wider (cannot remove the range as Solr requires it). I suggest changing the facet range as follows:
```
runner.setProperty("facet.range.start", "NOW/MINUTE-3MINUTE");
runner.setProperty("facet.range.end", "NOW/MINUTE+3MINUTE");
```
This makes the test passing even with sleeping 1 minute between doc creation and query execution.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] MikeThomsen commented on issue #4072: Nifi 7182
Posted by GitBox <gi...@apache.org>.
MikeThomsen commented on issue #4072: Nifi 7182
URL: https://github.com/apache/nifi/pull/4072#issuecomment-589908571
* Looked over the solrconf, looks good. It's storing everything in memory so there are no FS artifacts to worry about between runs.
* Query is super simple: `*:*` which is a wildcard to return everything.
* Nothing obvious that is related to file system-related issues.
If you have any stacktraces, that would be great. Otherwise, we might have to mark this with the ignore annotation and wait until there's a stacktrace from one of your builds.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] joewitt commented on issue #4072: Nifi 7182
Posted by GitBox <gi...@apache.org>.
joewitt commented on issue #4072: Nifi 7182
URL: https://github.com/apache/nifi/pull/4072#issuecomment-589903017
a machine I do builds on all the time. i've only seen failures in the solr tests rarely. But that it can fail in the way it did suggests there is some timing thing we need to sort out.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] MikeThomsen commented on issue #4072: Nifi 7182
Posted by GitBox <gi...@apache.org>.
MikeThomsen commented on issue #4072: Nifi 7182
URL: https://github.com/apache/nifi/pull/4072#issuecomment-589907087
Ok. I'll look at the query. I think you said it mysteriously craps the bed in those cases and goes from finding at least some documents to finding none. If that be the case, it could be something wonky with the `commit()` call fully refreshing the searchers in embedded mode.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] joewitt commented on issue #4072: Nifi 7182
Posted by GitBox <gi...@apache.org>.
joewitt commented on issue #4072: Nifi 7182
URL: https://github.com/apache/nifi/pull/4072#issuecomment-589898290
@MikeThomsen the original cause of the jira was that the test actually failed on line 191 assertion (not just because they were swapped).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] MikeThomsen commented on issue #4072: Nifi 7182
Posted by GitBox <gi...@apache.org>.
MikeThomsen commented on issue #4072: Nifi 7182
URL: https://github.com/apache/nifi/pull/4072#issuecomment-589897876
Made the changes requested. Also noticed some windows line encoding weirdness when I saved the file. Weird because I'm doing this on a Mac. So two separate commits. Feel free to drop the dos2unix one.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] MikeThomsen commented on issue #4072: Nifi 7182
Posted by GitBox <gi...@apache.org>.
MikeThomsen commented on issue #4072: Nifi 7182
URL: https://github.com/apache/nifi/pull/4072#issuecomment-589902741
@joewitt I tried running it a few times, and couldn't replicate the failure. Do you have any idea what configuration it was running on when it failed?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] MikeThomsen commented on issue #4072: NIFI-7182
Posted by GitBox <gi...@apache.org>.
MikeThomsen commented on issue #4072: NIFI-7182
URL: https://github.com/apache/nifi/pull/4072#issuecomment-590123514
It's been a few years since I last really used Solr, but the `commit()` there is a blocking call that forces a flush to the Lucene indexes and rewarms the "searchers" which are the Java objects that actually execute the queries. So this makes no sense to me because of that and the fact that queries are not asynchronous. So there should be no concept of "waiting for the results" in this use case.
@bbende @ijokarumawak thoughts?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] joewitt commented on issue #4072: NIFI-7182
Posted by GitBox <gi...@apache.org>.
joewitt commented on issue #4072: NIFI-7182
URL: https://github.com/apache/nifi/pull/4072#issuecomment-590108277
There is no stack trace of importance here because what failed was the assertion. It got nothing but expected 10 things.
We have provenance tests which require waiting a bit to let indexing occur before attempting the query. It 'seems like' this could be a similar problem here. Do you agree? If so rather than ignoring could we try checking if results are there yet and if not try again in a few seconds and if not done within some time cause the test to timeout instead?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services