You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@ambari.apache.org by Jonathan Hurley <jh...@hortonworks.com> on 2017/03/01 20:51:39 UTC
Review Request 57215: Startup Annotation Scanning Takes Too Long
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57215/
-----------------------------------------------------------
Review request for Ambari, Nate Cole and Robert Levas.
Bugs: AMBARI-20262
https://issues.apache.org/jira/browse/AMBARI-20262
Repository: ambari
Description
-------
Currently, Ambari uses Spring's {{ClassPathScanningCandidateComponentProvider}} along with an {{AnnotationTypeFilter}} to find classes annotated in our classpath:
```
ClassPathScanningCandidateComponentProvider scanner =
new ClassPathScanningCandidateComponentProvider(false);
// match only singletons that are eager listeners
for (Class<? extends Annotation> cls : classes) {
scanner.addIncludeFilter(new AnnotationTypeFilter(cls));
}
beanDefinitions = scanner.findCandidateComponents(AMBARI_PACKAGE);
```
This takes roughly 19 seconds on normal deployments. We can reduce this time to roughly 8 seconds by switching over to Google's {{ClassPath}} via our internal {{ClasspathScannerUtils}}
Diffs
-----
ambari-server/src/main/java/org/apache/ambari/server/audit/AuditLoggerModule.java d8b8466
ambari-server/src/main/java/org/apache/ambari/server/cleanup/ClasspathScannerUtils.java 01487f0
ambari-server/src/main/java/org/apache/ambari/server/cleanup/CleanupModule.java d2a7583
ambari-server/src/main/java/org/apache/ambari/server/controller/ControllerModule.java 8646e51
ambari-server/src/test/java/org/apache/ambari/server/orm/InMemoryDefaultTestModule.java 3090108
Diff: https://reviews.apache.org/r/57215/diff/1/
Testing
-------
PENDING
Thanks,
Jonathan Hurley
Re: Review Request 57215: Startup Annotation Scanning Takes Too Long
Posted by Jonathan Hurley <jh...@hortonworks.com>.
> On March 1, 2017, 4:52 p.m., Robert Nettleton wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/controller/ControllerModule.java
> > Line 695 (original), 694 (patched)
> > <https://reviews.apache.org/r/57215/diff/1/?file=1652902#file1652902line711>
> >
> > Minor question: Should these log messages really be at the INFO level?
> >
> > It seems like we already have too much noise in the ambari-server startup log, so it's not clear to me why these would be moved from DEBUG.
I debated this as well. I think it's nice to know that they were picked up, but then again, we probably don't need them at info ... I'll move these to `debug`
- Jonathan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57215/#review167580
-----------------------------------------------------------
On March 1, 2017, 3:51 p.m., Jonathan Hurley wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57215/
> -----------------------------------------------------------
>
> (Updated March 1, 2017, 3:51 p.m.)
>
>
> Review request for Ambari, Nate Cole and Robert Levas.
>
>
> Bugs: AMBARI-20262
> https://issues.apache.org/jira/browse/AMBARI-20262
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Currently, Ambari uses Spring's {{ClassPathScanningCandidateComponentProvider}} along with an {{AnnotationTypeFilter}} to find classes annotated in our classpath:
>
> ```
> ClassPathScanningCandidateComponentProvider scanner =
> new ClassPathScanningCandidateComponentProvider(false);
>
> // match only singletons that are eager listeners
> for (Class<? extends Annotation> cls : classes) {
> scanner.addIncludeFilter(new AnnotationTypeFilter(cls));
> }
>
> beanDefinitions = scanner.findCandidateComponents(AMBARI_PACKAGE);
> ```
>
> This takes roughly 19 seconds on normal deployments. We can reduce this time to roughly 8 seconds by switching over to Google's {{ClassPath}} via our internal {{ClasspathScannerUtils}}
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/audit/AuditLoggerModule.java d8b8466
> ambari-server/src/main/java/org/apache/ambari/server/cleanup/ClasspathScannerUtils.java 01487f0
> ambari-server/src/main/java/org/apache/ambari/server/cleanup/CleanupModule.java d2a7583
> ambari-server/src/main/java/org/apache/ambari/server/controller/ControllerModule.java 8646e51
> ambari-server/src/test/java/org/apache/ambari/server/orm/InMemoryDefaultTestModule.java 3090108
>
>
> Diff: https://reviews.apache.org/r/57215/diff/1/
>
>
> Testing
> -------
>
> PENDING
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 57215: Startup Annotation Scanning Takes Too Long
Posted by Robert Nettleton <rn...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57215/#review167580
-----------------------------------------------------------
Ship it!
ambari-server/src/main/java/org/apache/ambari/server/controller/ControllerModule.java
Line 695 (original), 694 (patched)
<https://reviews.apache.org/r/57215/#comment239480>
Minor question: Should these log messages really be at the INFO level?
It seems like we already have too much noise in the ambari-server startup log, so it's not clear to me why these would be moved from DEBUG.
- Robert Nettleton
On March 1, 2017, 8:51 p.m., Jonathan Hurley wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57215/
> -----------------------------------------------------------
>
> (Updated March 1, 2017, 8:51 p.m.)
>
>
> Review request for Ambari, Nate Cole and Robert Levas.
>
>
> Bugs: AMBARI-20262
> https://issues.apache.org/jira/browse/AMBARI-20262
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Currently, Ambari uses Spring's {{ClassPathScanningCandidateComponentProvider}} along with an {{AnnotationTypeFilter}} to find classes annotated in our classpath:
>
> ```
> ClassPathScanningCandidateComponentProvider scanner =
> new ClassPathScanningCandidateComponentProvider(false);
>
> // match only singletons that are eager listeners
> for (Class<? extends Annotation> cls : classes) {
> scanner.addIncludeFilter(new AnnotationTypeFilter(cls));
> }
>
> beanDefinitions = scanner.findCandidateComponents(AMBARI_PACKAGE);
> ```
>
> This takes roughly 19 seconds on normal deployments. We can reduce this time to roughly 8 seconds by switching over to Google's {{ClassPath}} via our internal {{ClasspathScannerUtils}}
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/audit/AuditLoggerModule.java d8b8466
> ambari-server/src/main/java/org/apache/ambari/server/cleanup/ClasspathScannerUtils.java 01487f0
> ambari-server/src/main/java/org/apache/ambari/server/cleanup/CleanupModule.java d2a7583
> ambari-server/src/main/java/org/apache/ambari/server/controller/ControllerModule.java 8646e51
> ambari-server/src/test/java/org/apache/ambari/server/orm/InMemoryDefaultTestModule.java 3090108
>
>
> Diff: https://reviews.apache.org/r/57215/diff/1/
>
>
> Testing
> -------
>
> PENDING
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 57215: Startup Annotation Scanning Takes Too Long
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57215/#review167568
-----------------------------------------------------------
ambari-server/src/main/java/org/apache/ambari/server/cleanup/ClasspathScannerUtils.java
Line 99 (original), 100 (patched)
<https://reviews.apache.org/r/57215/#comment239462>
This was previously comparing `Annotation` to `Class` which didn't match.
ambari-server/src/main/java/org/apache/ambari/server/controller/ControllerModule.java
Lines 526-527 (original), 526-527 (patched)
<https://reviews.apache.org/r/57215/#comment239461>
This is really the only important change here - essentially Spring's annotation scanning was taking like 19 seconds. We have a classpath scanner that uses Google's Reflections library which is much faster.
- Jonathan Hurley
On March 1, 2017, 3:51 p.m., Jonathan Hurley wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57215/
> -----------------------------------------------------------
>
> (Updated March 1, 2017, 3:51 p.m.)
>
>
> Review request for Ambari, Nate Cole and Robert Levas.
>
>
> Bugs: AMBARI-20262
> https://issues.apache.org/jira/browse/AMBARI-20262
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Currently, Ambari uses Spring's {{ClassPathScanningCandidateComponentProvider}} along with an {{AnnotationTypeFilter}} to find classes annotated in our classpath:
>
> ```
> ClassPathScanningCandidateComponentProvider scanner =
> new ClassPathScanningCandidateComponentProvider(false);
>
> // match only singletons that are eager listeners
> for (Class<? extends Annotation> cls : classes) {
> scanner.addIncludeFilter(new AnnotationTypeFilter(cls));
> }
>
> beanDefinitions = scanner.findCandidateComponents(AMBARI_PACKAGE);
> ```
>
> This takes roughly 19 seconds on normal deployments. We can reduce this time to roughly 8 seconds by switching over to Google's {{ClassPath}} via our internal {{ClasspathScannerUtils}}
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/audit/AuditLoggerModule.java d8b8466
> ambari-server/src/main/java/org/apache/ambari/server/cleanup/ClasspathScannerUtils.java 01487f0
> ambari-server/src/main/java/org/apache/ambari/server/cleanup/CleanupModule.java d2a7583
> ambari-server/src/main/java/org/apache/ambari/server/controller/ControllerModule.java 8646e51
> ambari-server/src/test/java/org/apache/ambari/server/orm/InMemoryDefaultTestModule.java 3090108
>
>
> Diff: https://reviews.apache.org/r/57215/diff/1/
>
>
> Testing
> -------
>
> PENDING
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 57215: Startup Annotation Scanning Takes Too Long
Posted by Nate Cole <nc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57215/#review167677
-----------------------------------------------------------
Ship it!
Ship It!
- Nate Cole
On March 1, 2017, 3:51 p.m., Jonathan Hurley wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57215/
> -----------------------------------------------------------
>
> (Updated March 1, 2017, 3:51 p.m.)
>
>
> Review request for Ambari, Nate Cole and Robert Levas.
>
>
> Bugs: AMBARI-20262
> https://issues.apache.org/jira/browse/AMBARI-20262
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Currently, Ambari uses Spring's {{ClassPathScanningCandidateComponentProvider}} along with an {{AnnotationTypeFilter}} to find classes annotated in our classpath:
>
> ```
> ClassPathScanningCandidateComponentProvider scanner =
> new ClassPathScanningCandidateComponentProvider(false);
>
> // match only singletons that are eager listeners
> for (Class<? extends Annotation> cls : classes) {
> scanner.addIncludeFilter(new AnnotationTypeFilter(cls));
> }
>
> beanDefinitions = scanner.findCandidateComponents(AMBARI_PACKAGE);
> ```
>
> This takes roughly 19 seconds on normal deployments. We can reduce this time to roughly 8 seconds by switching over to Google's {{ClassPath}} via our internal {{ClasspathScannerUtils}}
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/audit/AuditLoggerModule.java d8b8466
> ambari-server/src/main/java/org/apache/ambari/server/cleanup/ClasspathScannerUtils.java 01487f0
> ambari-server/src/main/java/org/apache/ambari/server/cleanup/CleanupModule.java d2a7583
> ambari-server/src/main/java/org/apache/ambari/server/controller/ControllerModule.java 8646e51
> ambari-server/src/test/java/org/apache/ambari/server/orm/InMemoryDefaultTestModule.java 3090108
>
>
> Diff: https://reviews.apache.org/r/57215/diff/1/
>
>
> Testing
> -------
>
> PENDING
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 57215: Startup Annotation Scanning Takes Too Long
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57215/
-----------------------------------------------------------
(Updated March 2, 2017, 10:48 a.m.)
Review request for Ambari, Nate Cole and Robert Levas.
Bugs: AMBARI-20262
https://issues.apache.org/jira/browse/AMBARI-20262
Repository: ambari
Description
-------
Currently, Ambari uses Spring's {{ClassPathScanningCandidateComponentProvider}} along with an {{AnnotationTypeFilter}} to find classes annotated in our classpath:
```
ClassPathScanningCandidateComponentProvider scanner =
new ClassPathScanningCandidateComponentProvider(false);
// match only singletons that are eager listeners
for (Class<? extends Annotation> cls : classes) {
scanner.addIncludeFilter(new AnnotationTypeFilter(cls));
}
beanDefinitions = scanner.findCandidateComponents(AMBARI_PACKAGE);
```
This takes roughly 19 seconds on normal deployments. We can reduce this time to roughly 8 seconds by switching over to Google's {{ClassPath}} via our internal {{ClasspathScannerUtils}}
Diffs (updated)
-----
ambari-server/src/main/java/org/apache/ambari/server/audit/AuditLoggerModule.java d8b8466
ambari-server/src/main/java/org/apache/ambari/server/cleanup/ClasspathScannerUtils.java 01487f0
ambari-server/src/main/java/org/apache/ambari/server/cleanup/CleanupModule.java d2a7583
ambari-server/src/main/java/org/apache/ambari/server/controller/ControllerModule.java 8646e51
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/UpgradeResourceProviderHDP22Test.java 40c7e50
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/UpgradeResourceProviderTest.java 4d609b4
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/WidgetResourceProviderTest.java a4f6469
ambari-server/src/test/java/org/apache/ambari/server/orm/InMemoryDefaultTestModule.java 3090108
Diff: https://reviews.apache.org/r/57215/diff/2/
Changes: https://reviews.apache.org/r/57215/diff/1-2/
Testing (updated)
-------
mvn clean test
Thanks,
Jonathan Hurley