You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by GitBox <gi...@apache.org> on 2022/09/27 15:02:33 UTC
[GitHub] [lucene] iverase opened a new issue, #11827: Release manager should review lucene benchmarks before building release candidates
iverase opened a new issue, #11827:
URL: https://github.com/apache/lucene/issues/11827
### Description
We were about to release Lucene 9.4.0 with an important performance regression. This regression was showing up in the benchmarks but obviously we missed reviewing them. The regression was actually caught during the daily monitoring of Elasticsearch development instead.
Speaking to other Lucene committers we thought that it may be a good idea to add a new task to the release manager handbook to review the Lucene benchmarks before building the release candidates. Obviously we are not expecting the release manager to find all regressions but something as obvious as the one in Lucene 9.4.0 would have been possible.
wdyt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] mikemccand commented on issue #11827: Release manager should review lucene benchmarks before building release candidates
Posted by GitBox <gi...@apache.org>.
mikemccand commented on issue #11827:
URL: https://github.com/apache/lucene/issues/11827#issuecomment-1260788095
This was a spinoff from #11824.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] rmuir commented on issue #11827: Release manager should review lucene benchmarks before building release candidates
Posted by GitBox <gi...@apache.org>.
rmuir commented on issue #11827:
URL: https://github.com/apache/lucene/issues/11827#issuecomment-1259797860
i think a good step is to just make sure that we review the benchmark results before releasing and send an email out about anything that looks suspicious.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] uschindler commented on issue #11827: Release manager should review lucene benchmarks before building release candidates
Posted by GitBox <gi...@apache.org>.
uschindler commented on issue #11827:
URL: https://github.com/apache/lucene/issues/11827#issuecomment-1260501918
I fully agree, some checks should be done.
But here are a few bits that came into my mind:
- The @mikemccand benchmarks are running against main branch only. So the first check should be *before* creating the release branch, so we know that the start of branch is "clean".
- Anything else that gets included into release branch should be monitored by the RM if it negatively affects perf on main.
- Is there a benchmark available for the 9.x branch? If it would be configurable / modular we could have a base branch (against main and in parallel against 9.x). When a release branch is created we could compare the changes on the release branch with the baselines!?
These are just ideas, but basically the important thing would be: Before creating the release branch do a quick check through the benchmark suite on Mike's web page and find potential issues. During the release process when stuff is merged into release branch monitor all merged changes manually on main's benchmark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] navneet1v commented on issue #11827: Release manager should review lucene benchmarks before building release candidates
Posted by GitBox <gi...@apache.org>.
navneet1v commented on issue #11827:
URL: https://github.com/apache/lucene/issues/11827#issuecomment-1332603551
@iverase Where are we seeing the degradation? We are seeing the degradation in K-NN Queries between 9.3 and 9.4 release. I am trying to figure out if there are other clients of lucene who are also facing the same issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] msokolov commented on issue #11827: Release manager should review lucene benchmarks before building release candidates
Posted by GitBox <gi...@apache.org>.
msokolov commented on issue #11827:
URL: https://github.com/apache/lucene/issues/11827#issuecomment-1259830880
yup. Possibly too if Mike M is bored he could implement an alarming system :) or export the data somehow so we could bolt one on the side?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] mikemccand commented on issue #11827: Release manager should review lucene benchmarks before building release candidates
Posted by GitBox <gi...@apache.org>.
mikemccand commented on issue #11827:
URL: https://github.com/apache/lucene/issues/11827#issuecomment-1260778045
> yup. Possibly too if Mike M is bored he could implement an alarming system :) or export the data somehow so we could bolt one on the side?
Actually I rather like the alarm idea. Maybe we could start with something simple -- more than X% change over Y data points? Maybe for starters the nightly bench just writes these alarms in (blinking red font, heh) at the [top of the index page](https://home.apache.org/~mikemccand/lucenebench/)? If it seems like that is not too noisy, maybe we find some stronger way to "notify" us.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] iverase commented on issue #11827: Release manager should review lucene benchmarks before building release candidates
Posted by GitBox <gi...@apache.org>.
iverase commented on issue #11827:
URL: https://github.com/apache/lucene/issues/11827#issuecomment-1333324626
@navneet1v this issue was triggered by a regression on a geo query and was addressed before the release. I have no knowledge of any regression in vector search.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org