You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Rohit Yadav <bh...@apache.org> on 2013/02/21 14:52:25 UTC
[DISCUSS] Fix Bloat: Getting rid of autoscanning and using xml for
spring DI
I'm experimenting with removing all @Components and moving beans into
an xml that could be read by spring to do DI.
This decreases the load time and
Errors:
- Abstract classes annotated as spring components
- Annotations for project which are not included in either build
profiles: usage, simulator, oss (default) and nonoss
- Some project don't even build or are not part of any build
artefacts, looks like the @Component annotation was done in a hurry :)
- Interfaces being annotated as spring components
- Prasanna, Abhi and I spoke with Spring gurus Vijay V and Vaibhav on
how to fix this bloat, so in applicationContext.xml we do some aop
stuff to have transaction stuff we get from @DB annotation and this
needs fixing as it's applying the following for all classes and
methods (dude bloat, run! :)
<aop:config proxy-target-class="true">
<aop:aspect id="dbContextBuilder" ref="transactionContextBuilder">
<aop:pointcut id="captureAnyMethod"
expression="execution(* *(..))"
/>
Methodology and results:
I wrote a python program that would go through the whole source code
and fix all java files (except test, awsapi) who have @Component and
create bean rules in an xml file. Next I had to manually fix few
things (see error above) and I got the mgmt server running, the load
time had improved a lot (little faster now) after disabling
autoscanning and just using the xml generated. I see the benefit of
just using the xml and not having spring as a build time dependency
(we're using @Component in our classes so :).
This was an experiment mostly because a lot of folks are having spring
memory issues, especially Prasanna's devcloud-ci cannot run and check
our commits as memory kills dom0. So, if no one objects may I go ahead
and commit it? For me the jvm took about 650MB real memory on osx.
Right now I made it for all build profiles, nonoss being the superset,
my experiment can be checked out from here:
https://github.com/bhaisaab/cloudstack/tree/spring-annotation-to-xml
Regards.
The python program, in case anyone wants to replicate or is curious:
(defaulter is just a file containing result of: file . | grep java$ >
defaulter)
f = open('defaulter', 'r')
files = f.read().split('\n')[:-1]
files = filter(lambda x: not x.startswith('./awsapi') and
x.find('/test/') == -1, files)
xmldata = ""
bean = """ <bean id="%s" class="%s"/>\n"""
xml = """<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:context="http://www.springframework.org/schema/context"
xmlns:tx="http://www.springframework.org/schema/tx"
xmlns:aop="http://www.springframework.org/schema/aop"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
http://www.springframework.org/schema/tx
http://www.springframework.org/schema/tx/spring-tx-3.0.xsd
http://www.springframework.org/schema/aop
http://www.springframework.org/schema/aop/spring-aop-3.0.xsd
http://www.springframework.org/schema/context
http://www.springframework.org/schema/context/spring-context-3.0.xsd">
%s
</beans>
"""
misses = []
for file in files:
name = file.split('/')[-1].split('.')[0]
name = name[0].lower() + name[1:]
pkgname = ""
idx = 0
paths = file.split('/')[1:]
paths[-1] = paths[-1].split('.')[0]
for str in paths:
if str == 'org' or str == 'com':
pkgname = '.'.join(paths[idx:])
idx += 1
f = open(file, 'r')
data = f.read()
f.close()
idx = data.find('@Component')
if idx == -1:
misses.append(file)
continue
string = '@Component'
if data[len(string)+idx] == '(':
endidx = data.find(')', len(string)+idx)
name = data[len(string)+idx+1:endidx].split('=')[-1].replace('"', '')
name = name.replace("'", '')
data = data[:idx] + data[endidx+2:]
else:
data = data.replace(string, '')
data = data.replace('import
org.springframework.stereotype.Component;\n', '')
f = open(file, 'w')
f.write(data)
f.close()
print name, pkgname
xmldata += bean % (name, pkgname)
xmldata += "\n"
xml = xml % xmldata
f = open('./client/tomcatconf/defaultComponentContext.xml.in', 'w')
f.write(xml)
f.close()
f = open('misses', 'w')
f.write('\n'.join(misses))
f.close()
Re: [DISCUSS] Fix Bloat: Getting rid of autoscanning and using xml
for spring DI
Posted by Kelven Yang <ke...@citrix.com>.
I did heap analysis today. It is quite certain that the problem is not
related with component auto-scanning, it is related with the memory leak
introduced by Spring AOP. I'm now working on a solution to it.
Kelven
On 2/21/13 3:02 PM, "Kelven Yang" <ke...@citrix.com> wrote:
>Rohit,
>
>I'll checkout out your work about using XML instead of @Component.
>However, I'm not sure if the memory leak issue is related with auto-scan
>at all. I'm now doing a heap analysis and will update with the result
>
>Kelven
>
>
>
>On 2/21/13 7:02 AM, "Rohit Yadav" <bh...@apache.org> wrote:
>
>>Okay Chip, posted a comment on the issue.
>>
>>Hi Kelven, you want to check my fix. Thanks.
>>
>>Regards.
>>
>>On Thu, Feb 21, 2013 at 8:17 PM, Chip Childers
>><ch...@sungard.com> wrote:
>>>
>>> Can you coordinate with Kelvin on this? He's got CLOUDSTACK-1276, and
>>> had previously mentioned this exact same thing (using the xml file vs.
>>> annotations).
>>>
>>> On Thu, Feb 21, 2013 at 07:26:42PM +0530, Rohit Yadav wrote:
>>>> Just to add, we will have separate xmls for every kind of profile, I
>>>> can make every plugin have its own spring xml which is copied and
>>>> injected to the web.xml comments, suggestions, flames?
>>>>
>>>> Possible future experiments:
>>>> - Spring xmls artefacts based
>>>> - Have a way to fix AOP, don't add pointcuts on all classes, methods
>>>> - Make DI framework replaceable in CloudStack, I would go for Guice ;)
>>>>
>>>> Regards.
>>>>
>>>> On Thu, Feb 21, 2013 at 7:22 PM, Rohit Yadav <bh...@apache.org>
>>>>wrote:
>>>> > I'm experimenting with removing all @Components and moving beans
>>>>into
>>>> > an xml that could be read by spring to do DI.
>>>> > This decreases the load time and
>>>> >
>>>> > Errors:
>>>> > - Abstract classes annotated as spring components
>>>> > - Annotations for project which are not included in either build
>>>> > profiles: usage, simulator, oss (default) and nonoss
>>>> > - Some project don't even build or are not part of any build
>>>> > artefacts, looks like the @Component annotation was done in a hurry
>>>>:)
>>>> > - Interfaces being annotated as spring components
>>>> > - Prasanna, Abhi and I spoke with Spring gurus Vijay V and Vaibhav
>>>>on
>>>> > how to fix this bloat, so in applicationContext.xml we do some aop
>>>> > stuff to have transaction stuff we get from @DB annotation and this
>>>> > needs fixing as it's applying the following for all classes and
>>>> > methods (dude bloat, run! :)
>>>> >
>>>> > <aop:config proxy-target-class="true">
>>>> > <aop:aspect id="dbContextBuilder"
>>>>ref="transactionContextBuilder">
>>>> > <aop:pointcut id="captureAnyMethod"
>>>> > expression="execution(* *(..))"
>>>> > />
>>>> >
>>>> > Methodology and results:
>>>> > I wrote a python program that would go through the whole source code
>>>> > and fix all java files (except test, awsapi) who have @Component and
>>>> > create bean rules in an xml file. Next I had to manually fix few
>>>> > things (see error above) and I got the mgmt server running, the load
>>>> > time had improved a lot (little faster now) after disabling
>>>> > autoscanning and just using the xml generated. I see the benefit of
>>>> > just using the xml and not having spring as a build time dependency
>>>> > (we're using @Component in our classes so :).
>>>> >
>>>> > This was an experiment mostly because a lot of folks are having
>>>>spring
>>>> > memory issues, especially Prasanna's devcloud-ci cannot run and
>>>>check
>>>> > our commits as memory kills dom0. So, if no one objects may I go
>>>>ahead
>>>> > and commit it? For me the jvm took about 650MB real memory on osx.
>>>> >
>>>> > Right now I made it for all build profiles, nonoss being the
>>>>superset,
>>>> > my experiment can be checked out from here:
>>>> > https://github.com/bhaisaab/cloudstack/tree/spring-annotation-to-xml
>>>> >
>>>> > Regards.
>>>> >
>>>> > The python program, in case anyone wants to replicate or is curious:
>>>> > (defaulter is just a file containing result of: file . | grep java$
>>>>>
>>>> > defaulter)
>>>> > f = open('defaulter', 'r')
>>>> > files = f.read().split('\n')[:-1]
>>>> > files = filter(lambda x: not x.startswith('./awsapi') and
>>>> > x.find('/test/') == -1, files)
>>>> > xmldata = ""
>>>> > bean = """ <bean id="%s" class="%s"/>\n"""
>>>> > xml = """<!--
>>>> > Licensed to the Apache Software Foundation (ASF) under one
>>>> > or more contributor license agreements. See the NOTICE file
>>>> > distributed with this work for additional information
>>>> > regarding copyright ownership. The ASF licenses this file
>>>> > to you under the Apache License, Version 2.0 (the
>>>> > "License"); you may not use this file except in compliance
>>>> > with the License. You may obtain a copy of the License at
>>>> >
>>>> > http://www.apache.org/licenses/LICENSE-2.0
>>>> >
>>>> > Unless required by applicable law or agreed to in writing,
>>>> > software distributed under the License is distributed on an
>>>> > "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
>>>> > KIND, either express or implied. See the License for the
>>>> > specific language governing permissions and limitations
>>>> > under the License.
>>>> > -->
>>>> > <beans xmlns="http://www.springframework.org/schema/beans"
>>>> > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>>> > xmlns:context="http://www.springframework.org/schema/context"
>>>> > xmlns:tx="http://www.springframework.org/schema/tx"
>>>> > xmlns:aop="http://www.springframework.org/schema/aop"
>>>> > xsi:schemaLocation="http://www.springframework.org/schema/beans
>>>> >
>>>> > http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
>>>> > http://www.springframework.org/schema/tx
>>>> >
>>>>http://www.springframework.org/schema/tx/spring-tx-3.0.xsd
>>>> > http://www.springframework.org/schema/aop
>>>> >
>>>> > http://www.springframework.org/schema/aop/spring-aop-3.0.xsd
>>>> > http://www.springframework.org/schema/context
>>>> >
>>>> >
>>>>http://www.springframework.org/schema/context/spring-context-3.0.xsd">
>>>> >
>>>> > %s
>>>> > </beans>
>>>> > """
>>>> > misses = []
>>>> > for file in files:
>>>> > name = file.split('/')[-1].split('.')[0]
>>>> > name = name[0].lower() + name[1:]
>>>> > pkgname = ""
>>>> > idx = 0
>>>> > paths = file.split('/')[1:]
>>>> > paths[-1] = paths[-1].split('.')[0]
>>>> > for str in paths:
>>>> > if str == 'org' or str == 'com':
>>>> > pkgname = '.'.join(paths[idx:])
>>>> > idx += 1
>>>> > f = open(file, 'r')
>>>> > data = f.read()
>>>> > f.close()
>>>> > idx = data.find('@Component')
>>>> > if idx == -1:
>>>> > misses.append(file)
>>>> > continue
>>>> > string = '@Component'
>>>> > if data[len(string)+idx] == '(':
>>>> > endidx = data.find(')', len(string)+idx)
>>>> > name =
>>>>data[len(string)+idx+1:endidx].split('=')[-1].replace('"', '')
>>>> > name = name.replace("'", '')
>>>> > data = data[:idx] + data[endidx+2:]
>>>> > else:
>>>> > data = data.replace(string, '')
>>>> > data = data.replace('import
>>>> > org.springframework.stereotype.Component;\n', '')
>>>> > f = open(file, 'w')
>>>> > f.write(data)
>>>> > f.close()
>>>> > print name, pkgname
>>>> > xmldata += bean % (name, pkgname)
>>>> > xmldata += "\n"
>>>> >
>>>> > xml = xml % xmldata
>>>> > f = open('./client/tomcatconf/defaultComponentContext.xml.in', 'w')
>>>> > f.write(xml)
>>>> > f.close()
>>>> >
>>>> > f = open('misses', 'w')
>>>> > f.write('\n'.join(misses))
>>>> > f.close()
>>>>
>
Re: [DISCUSS] Fix Bloat: Getting rid of autoscanning and using xml
for spring DI
Posted by Kelven Yang <ke...@citrix.com>.
Rohit,
I'll checkout out your work about using XML instead of @Component.
However, I'm not sure if the memory leak issue is related with auto-scan
at all. I'm now doing a heap analysis and will update with the result
Kelven
On 2/21/13 7:02 AM, "Rohit Yadav" <bh...@apache.org> wrote:
>Okay Chip, posted a comment on the issue.
>
>Hi Kelven, you want to check my fix. Thanks.
>
>Regards.
>
>On Thu, Feb 21, 2013 at 8:17 PM, Chip Childers
><ch...@sungard.com> wrote:
>>
>> Can you coordinate with Kelvin on this? He's got CLOUDSTACK-1276, and
>> had previously mentioned this exact same thing (using the xml file vs.
>> annotations).
>>
>> On Thu, Feb 21, 2013 at 07:26:42PM +0530, Rohit Yadav wrote:
>>> Just to add, we will have separate xmls for every kind of profile, I
>>> can make every plugin have its own spring xml which is copied and
>>> injected to the web.xml comments, suggestions, flames?
>>>
>>> Possible future experiments:
>>> - Spring xmls artefacts based
>>> - Have a way to fix AOP, don't add pointcuts on all classes, methods
>>> - Make DI framework replaceable in CloudStack, I would go for Guice ;)
>>>
>>> Regards.
>>>
>>> On Thu, Feb 21, 2013 at 7:22 PM, Rohit Yadav <bh...@apache.org>
>>>wrote:
>>> > I'm experimenting with removing all @Components and moving beans into
>>> > an xml that could be read by spring to do DI.
>>> > This decreases the load time and
>>> >
>>> > Errors:
>>> > - Abstract classes annotated as spring components
>>> > - Annotations for project which are not included in either build
>>> > profiles: usage, simulator, oss (default) and nonoss
>>> > - Some project don't even build or are not part of any build
>>> > artefacts, looks like the @Component annotation was done in a hurry
>>>:)
>>> > - Interfaces being annotated as spring components
>>> > - Prasanna, Abhi and I spoke with Spring gurus Vijay V and Vaibhav on
>>> > how to fix this bloat, so in applicationContext.xml we do some aop
>>> > stuff to have transaction stuff we get from @DB annotation and this
>>> > needs fixing as it's applying the following for all classes and
>>> > methods (dude bloat, run! :)
>>> >
>>> > <aop:config proxy-target-class="true">
>>> > <aop:aspect id="dbContextBuilder"
>>>ref="transactionContextBuilder">
>>> > <aop:pointcut id="captureAnyMethod"
>>> > expression="execution(* *(..))"
>>> > />
>>> >
>>> > Methodology and results:
>>> > I wrote a python program that would go through the whole source code
>>> > and fix all java files (except test, awsapi) who have @Component and
>>> > create bean rules in an xml file. Next I had to manually fix few
>>> > things (see error above) and I got the mgmt server running, the load
>>> > time had improved a lot (little faster now) after disabling
>>> > autoscanning and just using the xml generated. I see the benefit of
>>> > just using the xml and not having spring as a build time dependency
>>> > (we're using @Component in our classes so :).
>>> >
>>> > This was an experiment mostly because a lot of folks are having
>>>spring
>>> > memory issues, especially Prasanna's devcloud-ci cannot run and check
>>> > our commits as memory kills dom0. So, if no one objects may I go
>>>ahead
>>> > and commit it? For me the jvm took about 650MB real memory on osx.
>>> >
>>> > Right now I made it for all build profiles, nonoss being the
>>>superset,
>>> > my experiment can be checked out from here:
>>> > https://github.com/bhaisaab/cloudstack/tree/spring-annotation-to-xml
>>> >
>>> > Regards.
>>> >
>>> > The python program, in case anyone wants to replicate or is curious:
>>> > (defaulter is just a file containing result of: file . | grep java$ >
>>> > defaulter)
>>> > f = open('defaulter', 'r')
>>> > files = f.read().split('\n')[:-1]
>>> > files = filter(lambda x: not x.startswith('./awsapi') and
>>> > x.find('/test/') == -1, files)
>>> > xmldata = ""
>>> > bean = """ <bean id="%s" class="%s"/>\n"""
>>> > xml = """<!--
>>> > Licensed to the Apache Software Foundation (ASF) under one
>>> > or more contributor license agreements. See the NOTICE file
>>> > distributed with this work for additional information
>>> > regarding copyright ownership. The ASF licenses this file
>>> > to you under the Apache License, Version 2.0 (the
>>> > "License"); you may not use this file except in compliance
>>> > with the License. You may obtain a copy of the License at
>>> >
>>> > http://www.apache.org/licenses/LICENSE-2.0
>>> >
>>> > Unless required by applicable law or agreed to in writing,
>>> > software distributed under the License is distributed on an
>>> > "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
>>> > KIND, either express or implied. See the License for the
>>> > specific language governing permissions and limitations
>>> > under the License.
>>> > -->
>>> > <beans xmlns="http://www.springframework.org/schema/beans"
>>> > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>> > xmlns:context="http://www.springframework.org/schema/context"
>>> > xmlns:tx="http://www.springframework.org/schema/tx"
>>> > xmlns:aop="http://www.springframework.org/schema/aop"
>>> > xsi:schemaLocation="http://www.springframework.org/schema/beans
>>> >
>>> > http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
>>> > http://www.springframework.org/schema/tx
>>> >
>>>http://www.springframework.org/schema/tx/spring-tx-3.0.xsd
>>> > http://www.springframework.org/schema/aop
>>> >
>>> > http://www.springframework.org/schema/aop/spring-aop-3.0.xsd
>>> > http://www.springframework.org/schema/context
>>> >
>>> >
>>>http://www.springframework.org/schema/context/spring-context-3.0.xsd">
>>> >
>>> > %s
>>> > </beans>
>>> > """
>>> > misses = []
>>> > for file in files:
>>> > name = file.split('/')[-1].split('.')[0]
>>> > name = name[0].lower() + name[1:]
>>> > pkgname = ""
>>> > idx = 0
>>> > paths = file.split('/')[1:]
>>> > paths[-1] = paths[-1].split('.')[0]
>>> > for str in paths:
>>> > if str == 'org' or str == 'com':
>>> > pkgname = '.'.join(paths[idx:])
>>> > idx += 1
>>> > f = open(file, 'r')
>>> > data = f.read()
>>> > f.close()
>>> > idx = data.find('@Component')
>>> > if idx == -1:
>>> > misses.append(file)
>>> > continue
>>> > string = '@Component'
>>> > if data[len(string)+idx] == '(':
>>> > endidx = data.find(')', len(string)+idx)
>>> > name =
>>>data[len(string)+idx+1:endidx].split('=')[-1].replace('"', '')
>>> > name = name.replace("'", '')
>>> > data = data[:idx] + data[endidx+2:]
>>> > else:
>>> > data = data.replace(string, '')
>>> > data = data.replace('import
>>> > org.springframework.stereotype.Component;\n', '')
>>> > f = open(file, 'w')
>>> > f.write(data)
>>> > f.close()
>>> > print name, pkgname
>>> > xmldata += bean % (name, pkgname)
>>> > xmldata += "\n"
>>> >
>>> > xml = xml % xmldata
>>> > f = open('./client/tomcatconf/defaultComponentContext.xml.in', 'w')
>>> > f.write(xml)
>>> > f.close()
>>> >
>>> > f = open('misses', 'w')
>>> > f.write('\n'.join(misses))
>>> > f.close()
>>>
Re: [DISCUSS] Fix Bloat: Getting rid of autoscanning and using xml
for spring DI
Posted by Rohit Yadav <bh...@apache.org>.
Okay Chip, posted a comment on the issue.
Hi Kelven, you want to check my fix. Thanks.
Regards.
On Thu, Feb 21, 2013 at 8:17 PM, Chip Childers
<ch...@sungard.com> wrote:
>
> Can you coordinate with Kelvin on this? He's got CLOUDSTACK-1276, and
> had previously mentioned this exact same thing (using the xml file vs.
> annotations).
>
> On Thu, Feb 21, 2013 at 07:26:42PM +0530, Rohit Yadav wrote:
>> Just to add, we will have separate xmls for every kind of profile, I
>> can make every plugin have its own spring xml which is copied and
>> injected to the web.xml comments, suggestions, flames?
>>
>> Possible future experiments:
>> - Spring xmls artefacts based
>> - Have a way to fix AOP, don't add pointcuts on all classes, methods
>> - Make DI framework replaceable in CloudStack, I would go for Guice ;)
>>
>> Regards.
>>
>> On Thu, Feb 21, 2013 at 7:22 PM, Rohit Yadav <bh...@apache.org> wrote:
>> > I'm experimenting with removing all @Components and moving beans into
>> > an xml that could be read by spring to do DI.
>> > This decreases the load time and
>> >
>> > Errors:
>> > - Abstract classes annotated as spring components
>> > - Annotations for project which are not included in either build
>> > profiles: usage, simulator, oss (default) and nonoss
>> > - Some project don't even build or are not part of any build
>> > artefacts, looks like the @Component annotation was done in a hurry :)
>> > - Interfaces being annotated as spring components
>> > - Prasanna, Abhi and I spoke with Spring gurus Vijay V and Vaibhav on
>> > how to fix this bloat, so in applicationContext.xml we do some aop
>> > stuff to have transaction stuff we get from @DB annotation and this
>> > needs fixing as it's applying the following for all classes and
>> > methods (dude bloat, run! :)
>> >
>> > <aop:config proxy-target-class="true">
>> > <aop:aspect id="dbContextBuilder" ref="transactionContextBuilder">
>> > <aop:pointcut id="captureAnyMethod"
>> > expression="execution(* *(..))"
>> > />
>> >
>> > Methodology and results:
>> > I wrote a python program that would go through the whole source code
>> > and fix all java files (except test, awsapi) who have @Component and
>> > create bean rules in an xml file. Next I had to manually fix few
>> > things (see error above) and I got the mgmt server running, the load
>> > time had improved a lot (little faster now) after disabling
>> > autoscanning and just using the xml generated. I see the benefit of
>> > just using the xml and not having spring as a build time dependency
>> > (we're using @Component in our classes so :).
>> >
>> > This was an experiment mostly because a lot of folks are having spring
>> > memory issues, especially Prasanna's devcloud-ci cannot run and check
>> > our commits as memory kills dom0. So, if no one objects may I go ahead
>> > and commit it? For me the jvm took about 650MB real memory on osx.
>> >
>> > Right now I made it for all build profiles, nonoss being the superset,
>> > my experiment can be checked out from here:
>> > https://github.com/bhaisaab/cloudstack/tree/spring-annotation-to-xml
>> >
>> > Regards.
>> >
>> > The python program, in case anyone wants to replicate or is curious:
>> > (defaulter is just a file containing result of: file . | grep java$ >
>> > defaulter)
>> > f = open('defaulter', 'r')
>> > files = f.read().split('\n')[:-1]
>> > files = filter(lambda x: not x.startswith('./awsapi') and
>> > x.find('/test/') == -1, files)
>> > xmldata = ""
>> > bean = """ <bean id="%s" class="%s"/>\n"""
>> > xml = """<!--
>> > Licensed to the Apache Software Foundation (ASF) under one
>> > or more contributor license agreements. See the NOTICE file
>> > distributed with this work for additional information
>> > regarding copyright ownership. The ASF licenses this file
>> > to you under the Apache License, Version 2.0 (the
>> > "License"); you may not use this file except in compliance
>> > with the License. You may obtain a copy of the License at
>> >
>> > http://www.apache.org/licenses/LICENSE-2.0
>> >
>> > Unless required by applicable law or agreed to in writing,
>> > software distributed under the License is distributed on an
>> > "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
>> > KIND, either express or implied. See the License for the
>> > specific language governing permissions and limitations
>> > under the License.
>> > -->
>> > <beans xmlns="http://www.springframework.org/schema/beans"
>> > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>> > xmlns:context="http://www.springframework.org/schema/context"
>> > xmlns:tx="http://www.springframework.org/schema/tx"
>> > xmlns:aop="http://www.springframework.org/schema/aop"
>> > xsi:schemaLocation="http://www.springframework.org/schema/beans
>> >
>> > http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
>> > http://www.springframework.org/schema/tx
>> > http://www.springframework.org/schema/tx/spring-tx-3.0.xsd
>> > http://www.springframework.org/schema/aop
>> >
>> > http://www.springframework.org/schema/aop/spring-aop-3.0.xsd
>> > http://www.springframework.org/schema/context
>> >
>> > http://www.springframework.org/schema/context/spring-context-3.0.xsd">
>> >
>> > %s
>> > </beans>
>> > """
>> > misses = []
>> > for file in files:
>> > name = file.split('/')[-1].split('.')[0]
>> > name = name[0].lower() + name[1:]
>> > pkgname = ""
>> > idx = 0
>> > paths = file.split('/')[1:]
>> > paths[-1] = paths[-1].split('.')[0]
>> > for str in paths:
>> > if str == 'org' or str == 'com':
>> > pkgname = '.'.join(paths[idx:])
>> > idx += 1
>> > f = open(file, 'r')
>> > data = f.read()
>> > f.close()
>> > idx = data.find('@Component')
>> > if idx == -1:
>> > misses.append(file)
>> > continue
>> > string = '@Component'
>> > if data[len(string)+idx] == '(':
>> > endidx = data.find(')', len(string)+idx)
>> > name = data[len(string)+idx+1:endidx].split('=')[-1].replace('"', '')
>> > name = name.replace("'", '')
>> > data = data[:idx] + data[endidx+2:]
>> > else:
>> > data = data.replace(string, '')
>> > data = data.replace('import
>> > org.springframework.stereotype.Component;\n', '')
>> > f = open(file, 'w')
>> > f.write(data)
>> > f.close()
>> > print name, pkgname
>> > xmldata += bean % (name, pkgname)
>> > xmldata += "\n"
>> >
>> > xml = xml % xmldata
>> > f = open('./client/tomcatconf/defaultComponentContext.xml.in', 'w')
>> > f.write(xml)
>> > f.close()
>> >
>> > f = open('misses', 'w')
>> > f.write('\n'.join(misses))
>> > f.close()
>>
Re: [DISCUSS] Fix Bloat: Getting rid of autoscanning and using xml
for spring DI
Posted by Chip Childers <ch...@sungard.com>.
Can you coordinate with Kelvin on this? He's got CLOUDSTACK-1276, and
had previously mentioned this exact same thing (using the xml file vs.
annotations).
On Thu, Feb 21, 2013 at 07:26:42PM +0530, Rohit Yadav wrote:
> Just to add, we will have separate xmls for every kind of profile, I
> can make every plugin have its own spring xml which is copied and
> injected to the web.xml comments, suggestions, flames?
>
> Possible future experiments:
> - Spring xmls artefacts based
> - Have a way to fix AOP, don't add pointcuts on all classes, methods
> - Make DI framework replaceable in CloudStack, I would go for Guice ;)
>
> Regards.
>
> On Thu, Feb 21, 2013 at 7:22 PM, Rohit Yadav <bh...@apache.org> wrote:
> > I'm experimenting with removing all @Components and moving beans into
> > an xml that could be read by spring to do DI.
> > This decreases the load time and
> >
> > Errors:
> > - Abstract classes annotated as spring components
> > - Annotations for project which are not included in either build
> > profiles: usage, simulator, oss (default) and nonoss
> > - Some project don't even build or are not part of any build
> > artefacts, looks like the @Component annotation was done in a hurry :)
> > - Interfaces being annotated as spring components
> > - Prasanna, Abhi and I spoke with Spring gurus Vijay V and Vaibhav on
> > how to fix this bloat, so in applicationContext.xml we do some aop
> > stuff to have transaction stuff we get from @DB annotation and this
> > needs fixing as it's applying the following for all classes and
> > methods (dude bloat, run! :)
> >
> > <aop:config proxy-target-class="true">
> > <aop:aspect id="dbContextBuilder" ref="transactionContextBuilder">
> > <aop:pointcut id="captureAnyMethod"
> > expression="execution(* *(..))"
> > />
> >
> > Methodology and results:
> > I wrote a python program that would go through the whole source code
> > and fix all java files (except test, awsapi) who have @Component and
> > create bean rules in an xml file. Next I had to manually fix few
> > things (see error above) and I got the mgmt server running, the load
> > time had improved a lot (little faster now) after disabling
> > autoscanning and just using the xml generated. I see the benefit of
> > just using the xml and not having spring as a build time dependency
> > (we're using @Component in our classes so :).
> >
> > This was an experiment mostly because a lot of folks are having spring
> > memory issues, especially Prasanna's devcloud-ci cannot run and check
> > our commits as memory kills dom0. So, if no one objects may I go ahead
> > and commit it? For me the jvm took about 650MB real memory on osx.
> >
> > Right now I made it for all build profiles, nonoss being the superset,
> > my experiment can be checked out from here:
> > https://github.com/bhaisaab/cloudstack/tree/spring-annotation-to-xml
> >
> > Regards.
> >
> > The python program, in case anyone wants to replicate or is curious:
> > (defaulter is just a file containing result of: file . | grep java$ >
> > defaulter)
> > f = open('defaulter', 'r')
> > files = f.read().split('\n')[:-1]
> > files = filter(lambda x: not x.startswith('./awsapi') and
> > x.find('/test/') == -1, files)
> > xmldata = ""
> > bean = """ <bean id="%s" class="%s"/>\n"""
> > xml = """<!--
> > Licensed to the Apache Software Foundation (ASF) under one
> > or more contributor license agreements. See the NOTICE file
> > distributed with this work for additional information
> > regarding copyright ownership. The ASF licenses this file
> > to you under the Apache License, Version 2.0 (the
> > "License"); you may not use this file except in compliance
> > with the License. You may obtain a copy of the License at
> >
> > http://www.apache.org/licenses/LICENSE-2.0
> >
> > Unless required by applicable law or agreed to in writing,
> > software distributed under the License is distributed on an
> > "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
> > KIND, either express or implied. See the License for the
> > specific language governing permissions and limitations
> > under the License.
> > -->
> > <beans xmlns="http://www.springframework.org/schema/beans"
> > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> > xmlns:context="http://www.springframework.org/schema/context"
> > xmlns:tx="http://www.springframework.org/schema/tx"
> > xmlns:aop="http://www.springframework.org/schema/aop"
> > xsi:schemaLocation="http://www.springframework.org/schema/beans
> >
> > http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
> > http://www.springframework.org/schema/tx
> > http://www.springframework.org/schema/tx/spring-tx-3.0.xsd
> > http://www.springframework.org/schema/aop
> >
> > http://www.springframework.org/schema/aop/spring-aop-3.0.xsd
> > http://www.springframework.org/schema/context
> >
> > http://www.springframework.org/schema/context/spring-context-3.0.xsd">
> >
> > %s
> > </beans>
> > """
> > misses = []
> > for file in files:
> > name = file.split('/')[-1].split('.')[0]
> > name = name[0].lower() + name[1:]
> > pkgname = ""
> > idx = 0
> > paths = file.split('/')[1:]
> > paths[-1] = paths[-1].split('.')[0]
> > for str in paths:
> > if str == 'org' or str == 'com':
> > pkgname = '.'.join(paths[idx:])
> > idx += 1
> > f = open(file, 'r')
> > data = f.read()
> > f.close()
> > idx = data.find('@Component')
> > if idx == -1:
> > misses.append(file)
> > continue
> > string = '@Component'
> > if data[len(string)+idx] == '(':
> > endidx = data.find(')', len(string)+idx)
> > name = data[len(string)+idx+1:endidx].split('=')[-1].replace('"', '')
> > name = name.replace("'", '')
> > data = data[:idx] + data[endidx+2:]
> > else:
> > data = data.replace(string, '')
> > data = data.replace('import
> > org.springframework.stereotype.Component;\n', '')
> > f = open(file, 'w')
> > f.write(data)
> > f.close()
> > print name, pkgname
> > xmldata += bean % (name, pkgname)
> > xmldata += "\n"
> >
> > xml = xml % xmldata
> > f = open('./client/tomcatconf/defaultComponentContext.xml.in', 'w')
> > f.write(xml)
> > f.close()
> >
> > f = open('misses', 'w')
> > f.write('\n'.join(misses))
> > f.close()
>
Re: [DISCUSS] Fix Bloat: Getting rid of autoscanning and using xml
for spring DI
Posted by Rohit Yadav <bh...@apache.org>.
Just to add, we will have separate xmls for every kind of profile, I
can make every plugin have its own spring xml which is copied and
injected to the web.xml comments, suggestions, flames?
Possible future experiments:
- Spring xmls artefacts based
- Have a way to fix AOP, don't add pointcuts on all classes, methods
- Make DI framework replaceable in CloudStack, I would go for Guice ;)
Regards.
On Thu, Feb 21, 2013 at 7:22 PM, Rohit Yadav <bh...@apache.org> wrote:
> I'm experimenting with removing all @Components and moving beans into
> an xml that could be read by spring to do DI.
> This decreases the load time and
>
> Errors:
> - Abstract classes annotated as spring components
> - Annotations for project which are not included in either build
> profiles: usage, simulator, oss (default) and nonoss
> - Some project don't even build or are not part of any build
> artefacts, looks like the @Component annotation was done in a hurry :)
> - Interfaces being annotated as spring components
> - Prasanna, Abhi and I spoke with Spring gurus Vijay V and Vaibhav on
> how to fix this bloat, so in applicationContext.xml we do some aop
> stuff to have transaction stuff we get from @DB annotation and this
> needs fixing as it's applying the following for all classes and
> methods (dude bloat, run! :)
>
> <aop:config proxy-target-class="true">
> <aop:aspect id="dbContextBuilder" ref="transactionContextBuilder">
> <aop:pointcut id="captureAnyMethod"
> expression="execution(* *(..))"
> />
>
> Methodology and results:
> I wrote a python program that would go through the whole source code
> and fix all java files (except test, awsapi) who have @Component and
> create bean rules in an xml file. Next I had to manually fix few
> things (see error above) and I got the mgmt server running, the load
> time had improved a lot (little faster now) after disabling
> autoscanning and just using the xml generated. I see the benefit of
> just using the xml and not having spring as a build time dependency
> (we're using @Component in our classes so :).
>
> This was an experiment mostly because a lot of folks are having spring
> memory issues, especially Prasanna's devcloud-ci cannot run and check
> our commits as memory kills dom0. So, if no one objects may I go ahead
> and commit it? For me the jvm took about 650MB real memory on osx.
>
> Right now I made it for all build profiles, nonoss being the superset,
> my experiment can be checked out from here:
> https://github.com/bhaisaab/cloudstack/tree/spring-annotation-to-xml
>
> Regards.
>
> The python program, in case anyone wants to replicate or is curious:
> (defaulter is just a file containing result of: file . | grep java$ >
> defaulter)
> f = open('defaulter', 'r')
> files = f.read().split('\n')[:-1]
> files = filter(lambda x: not x.startswith('./awsapi') and
> x.find('/test/') == -1, files)
> xmldata = ""
> bean = """ <bean id="%s" class="%s"/>\n"""
> xml = """<!--
> Licensed to the Apache Software Foundation (ASF) under one
> or more contributor license agreements. See the NOTICE file
> distributed with this work for additional information
> regarding copyright ownership. The ASF licenses this file
> to you under the Apache License, Version 2.0 (the
> "License"); you may not use this file except in compliance
> with the License. You may obtain a copy of the License at
>
> http://www.apache.org/licenses/LICENSE-2.0
>
> Unless required by applicable law or agreed to in writing,
> software distributed under the License is distributed on an
> "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
> KIND, either express or implied. See the License for the
> specific language governing permissions and limitations
> under the License.
> -->
> <beans xmlns="http://www.springframework.org/schema/beans"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xmlns:context="http://www.springframework.org/schema/context"
> xmlns:tx="http://www.springframework.org/schema/tx"
> xmlns:aop="http://www.springframework.org/schema/aop"
> xsi:schemaLocation="http://www.springframework.org/schema/beans
>
> http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
> http://www.springframework.org/schema/tx
> http://www.springframework.org/schema/tx/spring-tx-3.0.xsd
> http://www.springframework.org/schema/aop
>
> http://www.springframework.org/schema/aop/spring-aop-3.0.xsd
> http://www.springframework.org/schema/context
>
> http://www.springframework.org/schema/context/spring-context-3.0.xsd">
>
> %s
> </beans>
> """
> misses = []
> for file in files:
> name = file.split('/')[-1].split('.')[0]
> name = name[0].lower() + name[1:]
> pkgname = ""
> idx = 0
> paths = file.split('/')[1:]
> paths[-1] = paths[-1].split('.')[0]
> for str in paths:
> if str == 'org' or str == 'com':
> pkgname = '.'.join(paths[idx:])
> idx += 1
> f = open(file, 'r')
> data = f.read()
> f.close()
> idx = data.find('@Component')
> if idx == -1:
> misses.append(file)
> continue
> string = '@Component'
> if data[len(string)+idx] == '(':
> endidx = data.find(')', len(string)+idx)
> name = data[len(string)+idx+1:endidx].split('=')[-1].replace('"', '')
> name = name.replace("'", '')
> data = data[:idx] + data[endidx+2:]
> else:
> data = data.replace(string, '')
> data = data.replace('import
> org.springframework.stereotype.Component;\n', '')
> f = open(file, 'w')
> f.write(data)
> f.close()
> print name, pkgname
> xmldata += bean % (name, pkgname)
> xmldata += "\n"
>
> xml = xml % xmldata
> f = open('./client/tomcatconf/defaultComponentContext.xml.in', 'w')
> f.write(xml)
> f.close()
>
> f = open('misses', 'w')
> f.write('\n'.join(misses))
> f.close()