You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@harmony.apache.org by Wenlong Li <we...@gmail.com> on 2008/12/19 08:45:35 UTC
[VM] On-demand class library parsing is ready to commit
Hello, all,
I did the pre-commit test on the patch of on-demand class library
parsing (https://issues.apache.org/jira/browse/HARMONY-6039), and it
works well now.
Can Harmony incorporate this feature?
Via on-demand class parsing, we can reduce startup time from 20+
seconds to 3 seconds for cold runing, and 170 ms to 140 ms for warm-up
running on Core 2 Duo with Windows.
After applying the patch, please note there is some change to add new modules.
(1) If you want to add new modules/libraries, please don't put them in
the bootclasspath.properties file. This file now only saves modules
needed during startup (the VM startup only accesses class libraries in
eight modules)
(2) For new modules/libraries, please put them in the
modulelibrarymapping.properties file. You should specify the module
name and its exported class library. Here is one example:
math.jar=java.math, where "math.jar" means the module name, and
"java.math" means the class libraries this module exports.
Thx,
Wenlong
Re: [VM] On-demand class library parsing is ready to commit
Posted by Wenlong Li <we...@gmail.com>.
Ok, I see what you mean.
In my windows test, I start the windows OS, and then let it run for a
few minutes. Then I measure the data. Later, I restart the machine,
and re-measure data. Another test configuration is let the computer
run for a few hours, I collect the performance data.
On Thu, Dec 25, 2008 at 4:27 PM, Aleksey Shipilev
<al...@gmail.com> wrote:
> Hi,
>
> On Thu, Dec 25, 2008 at 11:22 AM, Wenlong Li <we...@gmail.com> wrote:
>> Do you suggest conducting experiments in Linux always?
>> You may restart your windows machine, and then teerminating all
>> running processes and run the Harmony or SUN to see the response time
>> before Hello World appears.
>
> I doubt that's the problem with Windows at all, I bet it's much more
> about exact Windows configuration you working on. 20 sec startup seem
> to be really bad if there no obstacles for JVM to run (like antivirus
> which blocks JARs while loading). I wouldn't trust this machine :)
>
> Thanks,
> Aleksey.
>
Re: [VM] On-demand class library parsing is ready to commit
Posted by Wenlong Li <we...@gmail.com>.
Don't understand. Can you pls detail your comment?
On Thu, Dec 25, 2008 at 10:00 PM, Pavel Pervov <pm...@gmail.com> wrote:
> If this would be VM-side automatically produced configuration file...
>
> WBR,
> Pavel.
>
> On Thu, Dec 25, 2008 at 4:58 PM, Wenlong Li <we...@gmail.com> wrote:
>> btw, because adding new module is rare case, manually modifying the
>> bootclasspath.properties is not an issue?
>>
>> If so, can I conclude adding another property file with same update
>> frequency as bootclasspath would be fine as well?
>>
>> Pls kindly correct me if my understanding is wrong.
>>
>> On Thu, Dec 25, 2008 at 9:05 PM, Pavel Pervov <pm...@gmail.com> wrote:
>>> Wenlong,
>>>
>>> Note, that bootclasspath.properties is only changed on adding new
>>> module. This is pretty rare occasion, I believe.
>>>
>>> WBR,
>>> Pavel.
>>>
>>> On Thu, Dec 25, 2008 at 3:48 PM, Wenlong Li <we...@gmail.com> wrote:
>>>> Thx for your advice. Alexey.
>>>>
>>>> Here I have one question: do you know how the bootclasspath.properties
>>>> is maintained, manually or automatically?
>>>>
>>>> Another comment is I would like to treat the patch as DRLVM specific
>>>> optimization, e.g., it targets for improving VM creation time. So that
>>>> is possible to move all updates to DRLVM part to eliminate potential
>>>> modularity and compatibility problem.
>>>>
>>>> thx,
>>>> Wenlong
>>>>
>>>> On Thu, Dec 25, 2008 at 5:32 PM, Aleksey Shipilev
>>>> <al...@gmail.com> wrote:
>>>>> Hi, Wenlong.
>>>>>
>>>>> On Thu, Dec 25, 2008 at 11:49 AM, Wenlong Li <we...@gmail.com> wrote:
>>>>>> btw, Alexey, Let's go back to discuss whether there is a need to
>>>>>> include this feature in Harmony, given 17% performance gain in Linux
>>>>>> when using your methodology. For windows test, I will double check the
>>>>>> backgroud process as you pointed out.
>>>>>
>>>>> My opinion was already expressed after I had finished the tests from
>>>>> my side: the boost can be achieved in specific conditions, so whether
>>>>> it's worth including into Harmony really depends on how much mess the
>>>>> patch would introduce besides the "performance boost". From what I
>>>>> see, the patch obliges the maintainer to maintain the correct mapping
>>>>> between jars and Java packages. This new feature is also spread
>>>>> between Classlib and VM, but it seems like DRLVM specific. In this
>>>>> case I would rather stay without the patch.
>>>>>
>>>>> Personally (if I'll be committer) I would accept the patch with two
>>>>> serious modifications:
>>>>> 1. Stay within DRLVM, do not introduce this feature into Classlib,
>>>>> get the thing tested and evolved on DRLVM side. Otherwise it might
>>>>> break the compatibility with other VMs.
>>>>> 2. Make the mapping generated automatically (during build process?)
>>>>> to free the burden for maintainers.
>>>>>
>>>>> Thanks,
>>>>> Aleksey.
>>>>>
>>>>
>>>
>>
>
Re: [VM] On-demand class library parsing is ready to commit
Posted by Aleksey Shipilev <al...@gmail.com>.
Hi Wenlong!
I assumed that you're talking about H6038_2.patch.
1. On the performance side: the result of performance assessment on
HelloWorld is below. The test system is:
- Pentium D 820 2.8 Ghz / 2 Gb DDR2-667
- WD 3200KS, 320 Gb, 16 Mb cache
- Gentoo Linux x86, 2.6.23
- Harmony r734360, release build
"cold-start" (invalidate caches):
clean: (2.49 +- 0.13) secs
ondemand: (2.31 +- 0.10) secs
"warm-start" (don't invalidate caches);
clean: (0.42 +- 0.01) secs
ondemand: (0.40 +- 0.01) secs
Ok, we might presume the patch finally gives a small-small boost :)
2. On the conformance side: we need to work out the patch a bit:
a. externalize magic numbers
b. externalize inlined string literals
("modulelibrarymapping.properties", etc.)
All other things look ok for me.
Thanks,
Aleksey.
On Wed, Jan 14, 2009 at 12:33 PM, Wenlong Li <we...@gmail.com> wrote:
> All,
>
> I submit a new patch for on-demand class loading and parsing. All
> codes are put in VM side, and the mapping info is automatically
> produced.
>
> Pls see https://issues.apache.org/jira/browse/HARMONY-6039
>
> Comments are welcome.
>
> Thx, Wenlong
> Managed Runtime Technology Center, Intel
>
> On Wed, Jan 7, 2009 at 12:08 PM, Wenlong Li <we...@gmail.com> wrote:
>> All,
>> At this moment, I move all updates in classlib to VM side such that
>> there is no modularity issue. Next step is to produce the mapping
>> between module and library efficiently and accurately.
>>
>> Comments are welcome.
>>
>> Thx, Wenlong
>> Managed Runtime Technology Center, Intel
>>
>> On Tue, Jan 6, 2009 at 11:08 PM, Wenlong Li <we...@gmail.com> wrote:
>>> Thx :)
>>>
>>> On Tue, Jan 6, 2009 at 10:35 PM, Alexei Fedotov
>>> <al...@gmail.com> wrote:
>>>> Sure.
>>>>
>>>> 1. If you dig into SetClasspathFromString, you will see that it starts from
>>>> splitting the given classpath into pieces. You already know the new piece
>>>> you add and may skip splitting step.
>>>>
>>>> 2. If I understand you code correctly, the case "pdest >
>>>> (*it).second->bytes" might be a subject of a negative assertion. Adding this
>>>> assrtion would speed up bug discovery.
>>>>
>>>> Thanks.
>>>>
>>>>
>>>> On Tue, Jan 6, 2009 at 5:09 AM, Wenlong Li <we...@gmail.com> wrote:
>>>>
>>>>> Yes, Xiao-Feng's understanding is correct. The patch loads and parses
>>>>> modules on demand. If no class in swing.jar is not requested, then
>>>>> this module will not be loaded.
>>>>>
>>>>> btw, Alexei, you said "SetClasspathFromString" and "pdest >
>>>>> (*it).second->bytes" are not efficient. Can you share more comments on
>>>>> them? I just reused some code in Harmony, and didn't optimize them
>>>>> further.
>>>>>
>>>>> Thx, Wenlong
>>>>> Managed Runtime Technology Center, Intel
>>>>>
>>>>> On Fri, Dec 26, 2008 at 5:16 PM, Xiao-Feng Li <xi...@gmail.com>
>>>>> wrote:
>>>>> > On Fri, Dec 26, 2008 at 5:08 PM, Alexei Fedotov
>>>>> > <al...@gmail.com> wrote:
>>>>> >> Xiao Feng,
>>>>> >> Thank you for explaining.
>>>>> >>
>>>>> >> I get more minor comments on more commented code, ineffective
>>>>> >> SetClasspathFromString usage, non-covered unexpected case when pdest >
>>>>> >> (*it).second->bytes. One major comment on crossing vm module boundary
>>>>> >> still remains. But now I'm happy with the design.
>>>>> >
>>>>> > Alexei, yes, I agree with your comments. These parts should be
>>>>> > improved. (Still, this is my personal opinion. :) Let's wait Wenlong
>>>>> > speaking.)
>>>>> >
>>>>> > Thanks,
>>>>> > xiaofeng
>>>>> >
>>>>> >> Sorry for being slow.
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> On Fri, Dec 26, 2008 at 11:40 AM, Xiao-Feng Li <xi...@gmail.com>
>>>>> wrote:
>>>>> >>> On Fri, Dec 26, 2008 at 4:03 PM, Alexei Fedotov
>>>>> >>> <al...@gmail.com> wrote:
>>>>> >>>> Xiao-Feng,
>>>>> >>>> Continuing with the server example could you please give me a hint
>>>>> where
>>>>> >>>> decision to load swing.jar or not is taken in the patch? My initial
>>>>> >>>> perception was that the list of what to load was hardcoded and was not
>>>>> >>>> constructed dynamically depending on application.
>>>>> >>>
>>>>> >>> Alexei, here is the patch code I found:
>>>>> >>>
>>>>> >>> line 245:
>>>>> >>> + // Find which jar exports this package
>>>>> >>> + if (pkgName != NULL) {
>>>>> >>> + char *boot_class_path =
>>>>> >>> env->JavaProperties()->get(VM_BOOT_CLASS_DIR);
>>>>> >>> + char *pendingClassPath = NULL;
>>>>> >>> + apr_pool_t *tmp_pool;
>>>>> >>> + apr_pool_create(&tmp_pool, NULL);
>>>>> >>> + while (it != env->pending_jar_set.end()) {
>>>>> >>> + pdest = strstr( (*it).second->bytes, pkgName );
>>>>> >>> + if (pdest != NULL) {
>>>>> >>> + pendingClassPath =
>>>>> >>> (char*)STD_MALLOC(strlen(boot_class_path)
>>>>> >>> + +
>>>>> strlen((*it).first->bytes) + 1);
>>>>> >>> + strcpy(pendingClassPath, boot_class_path);
>>>>> >>> + strcat(pendingClassPath, (*it).first->bytes);
>>>>> >>> + // Open this found jar, and read all classes
>>>>> >>> contained in this jar
>>>>> >>> + SetClasspathFromString(pendingClassPath,
>>>>> tmp_pool);
>>>>> >>> + // Erase the found jar from pending jar list
>>>>> >>> as it has been parsed
>>>>> >>> + env->pending_jar_set.erase(it++);
>>>>> >>> + STD_FREE(pendingClassPath);
>>>>> >>> + } else {
>>>>> >>>
>>>>> >>> It checks if a JAR has the requested package, then loads it if yes. I
>>>>> >>> am not sure if this is what you were asking. (Btw, this is only my
>>>>> >>> understanding of his patch. I am not speaking for Wenlong.)
>>>>> >>>
>>>>> >>> Thanks,
>>>>> >>> xiaofeng
>>>>> >>>
>>>>> >>>> Thanks.
>>>>> >>>>
>>>>> >>>>
>>>>> >>>> On Fri, Dec 26, 2008 at 4:14 AM, Xiao-Feng Li <xi...@gmail.com>
>>>>> wrote:
>>>>> >>>>
>>>>> >>>>> On Fri, Dec 26, 2008 at 12:49 AM, Alexei Fedotov
>>>>> >>>>> <al...@gmail.com> wrote:
>>>>> >>>>> > Aleksey,
>>>>> >>>>> > I like your conclusion.
>>>>> >>>>> >
>>>>> >>>>> > Wenlong,
>>>>> >>>>> > I'm trying to understand the real life value of the "abstract"
>>>>> startup
>>>>> >>>>> > time metric you've suggested. Does Harmony with your patch load
>>>>> >>>>> > swing.jar for a server application? Do I understand that loading
>>>>> >>>>> > happens, though it happens later compared to VM without your patch?
>>>>> I
>>>>> >>>>> > believe that the proper design of delayed loading should answer
>>>>> "no"
>>>>> >>>>> > to this question.
>>>>> >>>>>
>>>>> >>>>> I checked the patch, and I found the answer is indeed "No" as you
>>>>> expected.
>>>>> >>>>>
>>>>> >>>>> Thanks,
>>>>> >>>>> xiaofeng
>>>>> >>>>>
>>>>> >>>>> > In other words, I appreciate if you describe which real use cases
>>>>> are
>>>>> >>>>> > improved by this patch.
>>>>> >>>>> > Thanks!
>>>>> >>>>>
>>>>> >>>>> > On Thu, Dec 25, 2008 at 7:29 PM, Aleksey Shipilev
>>>>> >>>>> > <al...@gmail.com> wrote:
>>>>> >>>>> >> Ok, here's the catch.
>>>>> >>>>> >>
>>>>> >>>>> >> bootclasspath.properties is the SortedSet<JARfile>, which
>>>>> enumerates
>>>>> >>>>> >> the JARs available for bootclassloader. The set of such the JARs
>>>>> is
>>>>> >>>>> >> really stable because modular decomposition of classlib is stable.
>>>>> >>>>> >> That's why nobody bothers with automatic generation of it: it only
>>>>> >>>>> >> should be updated when new JAR file arrives.
>>>>> >>>>> >> Modulelibrarymapping.properties is different on this point, it's
>>>>> the
>>>>> >>>>> >> Map<PackageName,JARfile>, which should be updated each time the
>>>>> new
>>>>> >>>>> >> *package* is introduced. I'm not talking about java.* packages
>>>>> >>>>> >> (they're standardized), rather about org.apache.harmony.*.
>>>>> >>>>> >>
>>>>> >>>>> >> Automatic generation of this property file gives two advantages:
>>>>> >>>>> >> 1. Error-prone. Prevent yourself from hand-messing with mapping
>>>>> and
>>>>> >>>>> >> getting spurious ClassNotFoundException. BTW, what's the behaviour
>>>>> in
>>>>> >>>>> >> case the mapping is wrong?
>>>>> >>>>> >> 2. "Researchful". There're lot of guys around who enjoys the
>>>>> >>>>> >> modularity of Harmony classlib and eventually they might want to
>>>>> split
>>>>> >>>>> >> the packages even deeper, into smaller pieces. Then automatic
>>>>> >>>>> >> generation would enable them to quickly roll-in and experiment
>>>>> with
>>>>> >>>>> >> different package layouts and their impact on performance. They
>>>>> could
>>>>> >>>>> >> use ordinary bootclasspath.properties, but your feature wouldn't
>>>>> be
>>>>> >>>>> >> used by them then ;)
>>>>> >>>>> >>
>>>>> >>>>> >> That's merely a housekeeping procedure. I believe that anything
>>>>> which
>>>>> >>>>> >> could be done more than once, eventually would be done more than
>>>>> once.
>>>>> >>>>> >> Hence it should be automated. You say that the file was generated
>>>>> from
>>>>> >>>>> >> manifests of JARs, so is it hard to just tie the same tool into
>>>>> DRLVM
>>>>> >>>>> >> build process?
>>>>> >>>>> >>
>>>>> >>>>> >> As for DRLVM-specific, my opinion that this is because the patch:
>>>>> >>>>> >> a. breaks the compatibility of classlib (you change
>>>>> >>>>> >> bootclasspath.properties, right?) with other VMs.
>>>>> >>>>> >> b. treated in DRLVM classloader only.
>>>>> >>>>> >>
>>>>> >>>>> >> Of course eventually this feature might be used by others, but IMO
>>>>> we
>>>>> >>>>> >> should be careful about other guys who use the same classlib. I'd
>>>>> >>>>> >> rather wait for some incubation on DRLVM side first.
>>>>> >>>>> >>
>>>>> >>>>> >> Thanks,
>>>>> >>>>> >> Aleksey.
>>>>> >>>>> >>
>>>>> >>>>> >> On Thu, Dec 25, 2008 at 6:18 PM, Wenlong Li <we...@gmail.com>
>>>>> wrote:
>>>>> >>>>> >>> I see. In fact, my file doesn't need track change at the class
>>>>> >>>>> >>> granularity. Instead, it only needs know package info provided in
>>>>> the
>>>>> >>>>> >>> manifest file. When class is added to a library, do we need
>>>>> change
>>>>> >>>>> >>> the manifest as well?
>>>>> >>>>> >>>
>>>>> >>>>> >>> btw, I guess there is a mis-understanding: my
>>>>> modulelibrarymapping
>>>>> >>>>> >>> file only records package info provided by manfiest in each
>>>>> module. It
>>>>> >>>>> >>> doesn't relate to each class.
>>>>> >>>>> >>>
>>>>> >>>>> >>> thx,
>>>>> >>>>> >>> Wenlong
>>>>> >>>>> >>>
>>>>> >>>>> >>> On Thu, Dec 25, 2008 at 10:55 PM, Pavel Pervov <
>>>>> pmcfirst@gmail.com>
>>>>> >>>>> wrote:
>>>>> >>>>> >>>> Classes are added to class library from time to time. I'm not
>>>>> sure how
>>>>> >>>>> >>>> it can be possible to track these changes manually.
>>>>> >>>>> >>>>
>>>>> >>>>> >>>> WBR,
>>>>> >>>>> >>>> Pavel.
>>>>> >>>>> >>>>
>>>>> >>>>> >>>>
>>>>> >>>>> >>>> On Thu, Dec 25, 2008 at 5:09 PM, Wenlong Li <we...@gmail.com>
>>>>> >>>>> wrote:
>>>>> >>>>> >>>>> Sorry, one more question: bootclasspath.properties is classlib
>>>>> >>>>> >>>>> specific file, why we could not make a vm specific file
>>>>> manually?
>>>>> >>>>> Just
>>>>> >>>>> >>>>> curious to know the possible reason. :)
>>>>> >>>>> >>>>>
>>>>> >>>>> >>>>> thx,
>>>>> >>>>> >>>>> Wenlong
>>>>> >>>>> >>>>>
>>>>> >>>>> >>>>> On Thu, Dec 25, 2008 at 10:00 PM, Pavel Pervov <
>>>>> pmcfirst@gmail.com>
>>>>> >>>>> wrote:
>>>>> >>>>> >>>>>> If this would be VM-side automatically produced configuration
>>>>> >>>>> file...
>>>>> >>>>> >>>>>>
>>>>> >>>>> >>>>>> WBR,
>>>>> >>>>> >>>>>> Pavel.
>>>>> >>>>> >>>>>>
>>>>> >>>>> >>>>>> On Thu, Dec 25, 2008 at 4:58 PM, Wenlong Li <
>>>>> wenlong@gmail.com>
>>>>> >>>>> wrote:
>>>>> >>>>> >>>>>>> btw, because adding new module is rare case, manually
>>>>> modifying the
>>>>> >>>>> >>>>>>> bootclasspath.properties is not an issue?
>>>>> >>>>> >>>>>>>
>>>>> >>>>> >>>>>>> If so, can I conclude adding another property file with same
>>>>> update
>>>>> >>>>> >>>>>>> frequency as bootclasspath would be fine as well?
>>>>> >>>>> >>>>>>>
>>>>> >>>>> >>>>>>> Pls kindly correct me if my understanding is wrong.
>>>>> >>>>> >>>>>>>
>>>>> >>>>> >>>>>>> On Thu, Dec 25, 2008 at 9:05 PM, Pavel Pervov <
>>>>> pmcfirst@gmail.com>
>>>>> >>>>> wrote:
>>>>> >>>>> >>>>>>>> Wenlong,
>>>>> >>>>> >>>>>>>>
>>>>> >>>>> >>>>>>>> Note, that bootclasspath.properties is only changed on
>>>>> adding new
>>>>> >>>>> >>>>>>>> module. This is pretty rare occasion, I believe.
>>>>> >>>>> >>>>>>>>
>>>>> >>>>> >>>>>>>> WBR,
>>>>> >>>>> >>>>>>>> Pavel.
>>>>> >>>>> >>>>>>>>
>>>>> >>>>> >>>>>>>> On Thu, Dec 25, 2008 at 3:48 PM, Wenlong Li <
>>>>> wenlong@gmail.com>
>>>>> >>>>> wrote:
>>>>> >>>>> >>>>>>>>> Thx for your advice. Alexey.
>>>>> >>>>> >>>>>>>>>
>>>>> >>>>> >>>>>>>>> Here I have one question: do you know how the
>>>>> >>>>> bootclasspath.properties
>>>>> >>>>> >>>>>>>>> is maintained, manually or automatically?
>>>>> >>>>> >>>>>>>>>
>>>>> >>>>> >>>>>>>>> Another comment is I would like to treat the patch as DRLVM
>>>>> >>>>> specific
>>>>> >>>>> >>>>>>>>> optimization, e.g., it targets for improving VM creation
>>>>> time. So
>>>>> >>>>> that
>>>>> >>>>> >>>>>>>>> is possible to move all updates to DRLVM part to eliminate
>>>>> >>>>> potential
>>>>> >>>>> >>>>>>>>> modularity and compatibility problem.
>>>>> >>>>> >>>>>>>>>
>>>>> >>>>> >>>>>>>>> thx,
>>>>> >>>>> >>>>>>>>> Wenlong
>>>>> >>>>> >>>>>>>>>
>>>>> >>>>> >>>>>>>>> On Thu, Dec 25, 2008 at 5:32 PM, Aleksey Shipilev
>>>>> >>>>> >>>>>>>>> <al...@gmail.com> wrote:
>>>>> >>>>> >>>>>>>>>> Hi, Wenlong.
>>>>> >>>>> >>>>>>>>>>
>>>>> >>>>> >>>>>>>>>> On Thu, Dec 25, 2008 at 11:49 AM, Wenlong Li <
>>>>> wenlong@gmail.com>
>>>>> >>>>> wrote:
>>>>> >>>>> >>>>>>>>>>> btw, Alexey, Let's go back to discuss whether there is a
>>>>> need
>>>>> >>>>> to
>>>>> >>>>> >>>>>>>>>>> include this feature in Harmony, given 17% performance
>>>>> gain in
>>>>> >>>>> Linux
>>>>> >>>>> >>>>>>>>>>> when using your methodology. For windows test, I will
>>>>> double
>>>>> >>>>> check the
>>>>> >>>>> >>>>>>>>>>> backgroud process as you pointed out.
>>>>> >>>>> >>>>>>>>>>
>>>>> >>>>> >>>>>>>>>> My opinion was already expressed after I had finished the
>>>>> tests
>>>>> >>>>> from
>>>>> >>>>> >>>>>>>>>> my side: the boost can be achieved in specific conditions,
>>>>> so
>>>>> >>>>> whether
>>>>> >>>>> >>>>>>>>>> it's worth including into Harmony really depends on how
>>>>> much
>>>>> >>>>> mess the
>>>>> >>>>> >>>>>>>>>> patch would introduce besides the "performance boost".
>>>>> From what
>>>>> >>>>> I
>>>>> >>>>> >>>>>>>>>> see, the patch obliges the maintainer to maintain the
>>>>> correct
>>>>> >>>>> mapping
>>>>> >>>>> >>>>>>>>>> between jars and Java packages. This new feature is also
>>>>> spread
>>>>> >>>>> >>>>>>>>>> between Classlib and VM, but it seems like DRLVM specific.
>>>>> In
>>>>> >>>>> this
>>>>> >>>>> >>>>>>>>>> case I would rather stay without the patch.
>>>>> >>>>> >>>>>>>>>>
>>>>> >>>>> >>>>>>>>>> Personally (if I'll be committer) I would accept the patch
>>>>> with
>>>>> >>>>> two
>>>>> >>>>> >>>>>>>>>> serious modifications:
>>>>> >>>>> >>>>>>>>>> 1. Stay within DRLVM, do not introduce this feature into
>>>>> >>>>> Classlib,
>>>>> >>>>> >>>>>>>>>> get the thing tested and evolved on DRLVM side. Otherwise
>>>>> it
>>>>> >>>>> might
>>>>> >>>>> >>>>>>>>>> break the compatibility with other VMs.
>>>>> >>>>> >>>>>>>>>> 2. Make the mapping generated automatically (during build
>>>>> >>>>> process?)
>>>>> >>>>> >>>>>>>>>> to free the burden for maintainers.
>>>>> >>>>> >>>>>>>>>>
>>>>> >>>>> >>>>>>>>>> Thanks,
>>>>> >>>>> >>>>>>>>>> Aleksey.
>>>>> >>>>> >>>>>>>>>>
>>>>> >>>>> >>>>>>>>>
>>>>> >>>>> >>>>>>>>
>>>>> >>>>> >>>>>>>
>>>>> >>>>> >>>>>>
>>>>> >>>>> >>>>>
>>>>> >>>>> >>>>
>>>>> >>>>> >>>
>>>>> >>>>> >>
>>>>> >>>>> >
>>>>> >>>>> >
>>>>> >>>>> >
>>>>> >>>>> > --
>>>>> >>>>> > С уважением,
>>>>> >>>>> > Алексей Федотов,
>>>>> >>>>> > ЗАО «Телеком Экспресс»
>>>>> >>>>> >
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>> --
>>>>> >>>>> http://xiao-feng.blogspot.com
>>>>> >>>>>
>>>>> >>>>
>>>>> >>>>
>>>>> >>>>
>>>>> >>>> --
>>>>> >>>> С уважением,
>>>>> >>>> Алексей Федотов,
>>>>> >>>> ЗАО «Телеком Экспресс»
>>>>> >>>>
>>>>> >>>
>>>>> >>>
>>>>> >>>
>>>>> >>> --
>>>>> >>> http://xiao-feng.blogspot.com
>>>>> >>>
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> --
>>>>> >> С уважением,
>>>>> >> Алексей Федотов,
>>>>> >> ЗАО «Телеком Экспресс»
>>>>> >>
>>>>> >
>>>>> >
>>>>> >
>>>>> > --
>>>>> > http://xiao-feng.blogspot.com
>>>>> >
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> С уважением,
>>>> Алексей Федотов,
>>>> ЗАО «Телеком Экспресс»
>>>>
>>>
>>
>
Re: [VM] On-demand class library parsing is ready to commit
Posted by Alexei Fedotov <al...@gmail.com>.
Wenlong,
It would be very nice of you to explicitely outline in the last JIRA
comment and/or the cover letter which patches are intended to
reviewed (and committed). The only patch which is intended for commit
is H6039.patch_2, isn't it?
[1] http://issues.apache.org/jira/secure/attachment/12397873/H6039.patch_2
On Wed, Jan 14, 2009 at 12:33 PM, Wenlong Li <we...@gmail.com> wrote:
> All,
>
> I submit a new patch for on-demand class loading and parsing. All
> codes are put in VM side, and the mapping info is automatically
> produced.
>
> Pls see https://issues.apache.org/jira/browse/HARMONY-6039
>
> Comments are welcome.
>
> Thx, Wenlong
> Managed Runtime Technology Center, Intel
>
> On Wed, Jan 7, 2009 at 12:08 PM, Wenlong Li <we...@gmail.com> wrote:
>> All,
>> At this moment, I move all updates in classlib to VM side such that
>> there is no modularity issue. Next step is to produce the mapping
>> between module and library efficiently and accurately.
>>
>> Comments are welcome.
>>
>> Thx, Wenlong
>> Managed Runtime Technology Center, Intel
>>
>> On Tue, Jan 6, 2009 at 11:08 PM, Wenlong Li <we...@gmail.com> wrote:
>>> Thx :)
>>>
>>> On Tue, Jan 6, 2009 at 10:35 PM, Alexei Fedotov
>>> <al...@gmail.com> wrote:
>>>> Sure.
>>>>
>>>> 1. If you dig into SetClasspathFromString, you will see that it starts from
>>>> splitting the given classpath into pieces. You already know the new piece
>>>> you add and may skip splitting step.
>>>>
>>>> 2. If I understand you code correctly, the case "pdest >
>>>> (*it).second->bytes" might be a subject of a negative assertion. Adding this
>>>> assrtion would speed up bug discovery.
>>>>
>>>> Thanks.
>>>>
>>>>
>>>> On Tue, Jan 6, 2009 at 5:09 AM, Wenlong Li <we...@gmail.com> wrote:
>>>>
>>>>> Yes, Xiao-Feng's understanding is correct. The patch loads and parses
>>>>> modules on demand. If no class in swing.jar is not requested, then
>>>>> this module will not be loaded.
>>>>>
>>>>> btw, Alexei, you said "SetClasspathFromString" and "pdest >
>>>>> (*it).second->bytes" are not efficient. Can you share more comments on
>>>>> them? I just reused some code in Harmony, and didn't optimize them
>>>>> further.
>>>>>
>>>>> Thx, Wenlong
>>>>> Managed Runtime Technology Center, Intel
>>>>>
>>>>> On Fri, Dec 26, 2008 at 5:16 PM, Xiao-Feng Li <xi...@gmail.com>
>>>>> wrote:
>>>>> > On Fri, Dec 26, 2008 at 5:08 PM, Alexei Fedotov
>>>>> > <al...@gmail.com> wrote:
>>>>> >> Xiao Feng,
>>>>> >> Thank you for explaining.
>>>>> >>
>>>>> >> I get more minor comments on more commented code, ineffective
>>>>> >> SetClasspathFromString usage, non-covered unexpected case when pdest >
>>>>> >> (*it).second->bytes. One major comment on crossing vm module boundary
>>>>> >> still remains. But now I'm happy with the design.
>>>>> >
>>>>> > Alexei, yes, I agree with your comments. These parts should be
>>>>> > improved. (Still, this is my personal opinion. :) Let's wait Wenlong
>>>>> > speaking.)
>>>>> >
>>>>> > Thanks,
>>>>> > xiaofeng
>>>>> >
>>>>> >> Sorry for being slow.
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> On Fri, Dec 26, 2008 at 11:40 AM, Xiao-Feng Li <xi...@gmail.com>
>>>>> wrote:
>>>>> >>> On Fri, Dec 26, 2008 at 4:03 PM, Alexei Fedotov
>>>>> >>> <al...@gmail.com> wrote:
>>>>> >>>> Xiao-Feng,
>>>>> >>>> Continuing with the server example could you please give me a hint
>>>>> where
>>>>> >>>> decision to load swing.jar or not is taken in the patch? My initial
>>>>> >>>> perception was that the list of what to load was hardcoded and was not
>>>>> >>>> constructed dynamically depending on application.
>>>>> >>>
>>>>> >>> Alexei, here is the patch code I found:
>>>>> >>>
>>>>> >>> line 245:
>>>>> >>> + // Find which jar exports this package
>>>>> >>> + if (pkgName != NULL) {
>>>>> >>> + char *boot_class_path =
>>>>> >>> env->JavaProperties()->get(VM_BOOT_CLASS_DIR);
>>>>> >>> + char *pendingClassPath = NULL;
>>>>> >>> + apr_pool_t *tmp_pool;
>>>>> >>> + apr_pool_create(&tmp_pool, NULL);
>>>>> >>> + while (it != env->pending_jar_set.end()) {
>>>>> >>> + pdest = strstr( (*it).second->bytes, pkgName );
>>>>> >>> + if (pdest != NULL) {
>>>>> >>> + pendingClassPath =
>>>>> >>> (char*)STD_MALLOC(strlen(boot_class_path)
>>>>> >>> + +
>>>>> strlen((*it).first->bytes) + 1);
>>>>> >>> + strcpy(pendingClassPath, boot_class_path);
>>>>> >>> + strcat(pendingClassPath, (*it).first->bytes);
>>>>> >>> + // Open this found jar, and read all classes
>>>>> >>> contained in this jar
>>>>> >>> + SetClasspathFromString(pendingClassPath,
>>>>> tmp_pool);
>>>>> >>> + // Erase the found jar from pending jar list
>>>>> >>> as it has been parsed
>>>>> >>> + env->pending_jar_set.erase(it++);
>>>>> >>> + STD_FREE(pendingClassPath);
>>>>> >>> + } else {
>>>>> >>>
>>>>> >>> It checks if a JAR has the requested package, then loads it if yes. I
>>>>> >>> am not sure if this is what you were asking. (Btw, this is only my
>>>>> >>> understanding of his patch. I am not speaking for Wenlong.)
>>>>> >>>
>>>>> >>> Thanks,
>>>>> >>> xiaofeng
>>>>> >>>
>>>>> >>>> Thanks.
>>>>> >>>>
>>>>> >>>>
>>>>> >>>> On Fri, Dec 26, 2008 at 4:14 AM, Xiao-Feng Li <xi...@gmail.com>
>>>>> wrote:
>>>>> >>>>
>>>>> >>>>> On Fri, Dec 26, 2008 at 12:49 AM, Alexei Fedotov
>>>>> >>>>> <al...@gmail.com> wrote:
>>>>> >>>>> > Aleksey,
>>>>> >>>>> > I like your conclusion.
>>>>> >>>>> >
>>>>> >>>>> > Wenlong,
>>>>> >>>>> > I'm trying to understand the real life value of the "abstract"
>>>>> startup
>>>>> >>>>> > time metric you've suggested. Does Harmony with your patch load
>>>>> >>>>> > swing.jar for a server application? Do I understand that loading
>>>>> >>>>> > happens, though it happens later compared to VM without your patch?
>>>>> I
>>>>> >>>>> > believe that the proper design of delayed loading should answer
>>>>> "no"
>>>>> >>>>> > to this question.
>>>>> >>>>>
>>>>> >>>>> I checked the patch, and I found the answer is indeed "No" as you
>>>>> expected.
>>>>> >>>>>
>>>>> >>>>> Thanks,
>>>>> >>>>> xiaofeng
>>>>> >>>>>
>>>>> >>>>> > In other words, I appreciate if you describe which real use cases
>>>>> are
>>>>> >>>>> > improved by this patch.
>>>>> >>>>> > Thanks!
>>>>> >>>>>
>>>>> >>>>> > On Thu, Dec 25, 2008 at 7:29 PM, Aleksey Shipilev
>>>>> >>>>> > <al...@gmail.com> wrote:
>>>>> >>>>> >> Ok, here's the catch.
>>>>> >>>>> >>
>>>>> >>>>> >> bootclasspath.properties is the SortedSet<JARfile>, which
>>>>> enumerates
>>>>> >>>>> >> the JARs available for bootclassloader. The set of such the JARs
>>>>> is
>>>>> >>>>> >> really stable because modular decomposition of classlib is stable.
>>>>> >>>>> >> That's why nobody bothers with automatic generation of it: it only
>>>>> >>>>> >> should be updated when new JAR file arrives.
>>>>> >>>>> >> Modulelibrarymapping.properties is different on this point, it's
>>>>> the
>>>>> >>>>> >> Map<PackageName,JARfile>, which should be updated each time the
>>>>> new
>>>>> >>>>> >> *package* is introduced. I'm not talking about java.* packages
>>>>> >>>>> >> (they're standardized), rather about org.apache.harmony.*.
>>>>> >>>>> >>
>>>>> >>>>> >> Automatic generation of this property file gives two advantages:
>>>>> >>>>> >> 1. Error-prone. Prevent yourself from hand-messing with mapping
>>>>> and
>>>>> >>>>> >> getting spurious ClassNotFoundException. BTW, what's the behaviour
>>>>> in
>>>>> >>>>> >> case the mapping is wrong?
>>>>> >>>>> >> 2. "Researchful". There're lot of guys around who enjoys the
>>>>> >>>>> >> modularity of Harmony classlib and eventually they might want to
>>>>> split
>>>>> >>>>> >> the packages even deeper, into smaller pieces. Then automatic
>>>>> >>>>> >> generation would enable them to quickly roll-in and experiment
>>>>> with
>>>>> >>>>> >> different package layouts and their impact on performance. They
>>>>> could
>>>>> >>>>> >> use ordinary bootclasspath.properties, but your feature wouldn't
>>>>> be
>>>>> >>>>> >> used by them then ;)
>>>>> >>>>> >>
>>>>> >>>>> >> That's merely a housekeeping procedure. I believe that anything
>>>>> which
>>>>> >>>>> >> could be done more than once, eventually would be done more than
>>>>> once.
>>>>> >>>>> >> Hence it should be automated. You say that the file was generated
>>>>> from
>>>>> >>>>> >> manifests of JARs, so is it hard to just tie the same tool into
>>>>> DRLVM
>>>>> >>>>> >> build process?
>>>>> >>>>> >>
>>>>> >>>>> >> As for DRLVM-specific, my opinion that this is because the patch:
>>>>> >>>>> >> a. breaks the compatibility of classlib (you change
>>>>> >>>>> >> bootclasspath.properties, right?) with other VMs.
>>>>> >>>>> >> b. treated in DRLVM classloader only.
>>>>> >>>>> >>
>>>>> >>>>> >> Of course eventually this feature might be used by others, but IMO
>>>>> we
>>>>> >>>>> >> should be careful about other guys who use the same classlib. I'd
>>>>> >>>>> >> rather wait for some incubation on DRLVM side first.
>>>>> >>>>> >>
>>>>> >>>>> >> Thanks,
>>>>> >>>>> >> Aleksey.
>>>>> >>>>> >>
>>>>> >>>>> >> On Thu, Dec 25, 2008 at 6:18 PM, Wenlong Li <we...@gmail.com>
>>>>> wrote:
>>>>> >>>>> >>> I see. In fact, my file doesn't need track change at the class
>>>>> >>>>> >>> granularity. Instead, it only needs know package info provided in
>>>>> the
>>>>> >>>>> >>> manifest file. When class is added to a library, do we need
>>>>> change
>>>>> >>>>> >>> the manifest as well?
>>>>> >>>>> >>>
>>>>> >>>>> >>> btw, I guess there is a mis-understanding: my
>>>>> modulelibrarymapping
>>>>> >>>>> >>> file only records package info provided by manfiest in each
>>>>> module. It
>>>>> >>>>> >>> doesn't relate to each class.
>>>>> >>>>> >>>
>>>>> >>>>> >>> thx,
>>>>> >>>>> >>> Wenlong
>>>>> >>>>> >>>
>>>>> >>>>> >>> On Thu, Dec 25, 2008 at 10:55 PM, Pavel Pervov <
>>>>> pmcfirst@gmail.com>
>>>>> >>>>> wrote:
>>>>> >>>>> >>>> Classes are added to class library from time to time. I'm not
>>>>> sure how
>>>>> >>>>> >>>> it can be possible to track these changes manually.
>>>>> >>>>> >>>>
>>>>> >>>>> >>>> WBR,
>>>>> >>>>> >>>> Pavel.
>>>>> >>>>> >>>>
>>>>> >>>>> >>>>
>>>>> >>>>> >>>> On Thu, Dec 25, 2008 at 5:09 PM, Wenlong Li <we...@gmail.com>
>>>>> >>>>> wrote:
>>>>> >>>>> >>>>> Sorry, one more question: bootclasspath.properties is classlib
>>>>> >>>>> >>>>> specific file, why we could not make a vm specific file
>>>>> manually?
>>>>> >>>>> Just
>>>>> >>>>> >>>>> curious to know the possible reason. :)
>>>>> >>>>> >>>>>
>>>>> >>>>> >>>>> thx,
>>>>> >>>>> >>>>> Wenlong
>>>>> >>>>> >>>>>
>>>>> >>>>> >>>>> On Thu, Dec 25, 2008 at 10:00 PM, Pavel Pervov <
>>>>> pmcfirst@gmail.com>
>>>>> >>>>> wrote:
>>>>> >>>>> >>>>>> If this would be VM-side automatically produced configuration
>>>>> >>>>> file...
>>>>> >>>>> >>>>>>
>>>>> >>>>> >>>>>> WBR,
>>>>> >>>>> >>>>>> Pavel.
>>>>> >>>>> >>>>>>
>>>>> >>>>> >>>>>> On Thu, Dec 25, 2008 at 4:58 PM, Wenlong Li <
>>>>> wenlong@gmail.com>
>>>>> >>>>> wrote:
>>>>> >>>>> >>>>>>> btw, because adding new module is rare case, manually
>>>>> modifying the
>>>>> >>>>> >>>>>>> bootclasspath.properties is not an issue?
>>>>> >>>>> >>>>>>>
>>>>> >>>>> >>>>>>> If so, can I conclude adding another property file with same
>>>>> update
>>>>> >>>>> >>>>>>> frequency as bootclasspath would be fine as well?
>>>>> >>>>> >>>>>>>
>>>>> >>>>> >>>>>>> Pls kindly correct me if my understanding is wrong.
>>>>> >>>>> >>>>>>>
>>>>> >>>>> >>>>>>> On Thu, Dec 25, 2008 at 9:05 PM, Pavel Pervov <
>>>>> pmcfirst@gmail.com>
>>>>> >>>>> wrote:
>>>>> >>>>> >>>>>>>> Wenlong,
>>>>> >>>>> >>>>>>>>
>>>>> >>>>> >>>>>>>> Note, that bootclasspath.properties is only changed on
>>>>> adding new
>>>>> >>>>> >>>>>>>> module. This is pretty rare occasion, I believe.
>>>>> >>>>> >>>>>>>>
>>>>> >>>>> >>>>>>>> WBR,
>>>>> >>>>> >>>>>>>> Pavel.
>>>>> >>>>> >>>>>>>>
>>>>> >>>>> >>>>>>>> On Thu, Dec 25, 2008 at 3:48 PM, Wenlong Li <
>>>>> wenlong@gmail.com>
>>>>> >>>>> wrote:
>>>>> >>>>> >>>>>>>>> Thx for your advice. Alexey.
>>>>> >>>>> >>>>>>>>>
>>>>> >>>>> >>>>>>>>> Here I have one question: do you know how the
>>>>> >>>>> bootclasspath.properties
>>>>> >>>>> >>>>>>>>> is maintained, manually or automatically?
>>>>> >>>>> >>>>>>>>>
>>>>> >>>>> >>>>>>>>> Another comment is I would like to treat the patch as DRLVM
>>>>> >>>>> specific
>>>>> >>>>> >>>>>>>>> optimization, e.g., it targets for improving VM creation
>>>>> time. So
>>>>> >>>>> that
>>>>> >>>>> >>>>>>>>> is possible to move all updates to DRLVM part to eliminate
>>>>> >>>>> potential
>>>>> >>>>> >>>>>>>>> modularity and compatibility problem.
>>>>> >>>>> >>>>>>>>>
>>>>> >>>>> >>>>>>>>> thx,
>>>>> >>>>> >>>>>>>>> Wenlong
>>>>> >>>>> >>>>>>>>>
>>>>> >>>>> >>>>>>>>> On Thu, Dec 25, 2008 at 5:32 PM, Aleksey Shipilev
>>>>> >>>>> >>>>>>>>> <al...@gmail.com> wrote:
>>>>> >>>>> >>>>>>>>>> Hi, Wenlong.
>>>>> >>>>> >>>>>>>>>>
>>>>> >>>>> >>>>>>>>>> On Thu, Dec 25, 2008 at 11:49 AM, Wenlong Li <
>>>>> wenlong@gmail.com>
>>>>> >>>>> wrote:
>>>>> >>>>> >>>>>>>>>>> btw, Alexey, Let's go back to discuss whether there is a
>>>>> need
>>>>> >>>>> to
>>>>> >>>>> >>>>>>>>>>> include this feature in Harmony, given 17% performance
>>>>> gain in
>>>>> >>>>> Linux
>>>>> >>>>> >>>>>>>>>>> when using your methodology. For windows test, I will
>>>>> double
>>>>> >>>>> check the
>>>>> >>>>> >>>>>>>>>>> backgroud process as you pointed out.
>>>>> >>>>> >>>>>>>>>>
>>>>> >>>>> >>>>>>>>>> My opinion was already expressed after I had finished the
>>>>> tests
>>>>> >>>>> from
>>>>> >>>>> >>>>>>>>>> my side: the boost can be achieved in specific conditions,
>>>>> so
>>>>> >>>>> whether
>>>>> >>>>> >>>>>>>>>> it's worth including into Harmony really depends on how
>>>>> much
>>>>> >>>>> mess the
>>>>> >>>>> >>>>>>>>>> patch would introduce besides the "performance boost".
>>>>> From what
>>>>> >>>>> I
>>>>> >>>>> >>>>>>>>>> see, the patch obliges the maintainer to maintain the
>>>>> correct
>>>>> >>>>> mapping
>>>>> >>>>> >>>>>>>>>> between jars and Java packages. This new feature is also
>>>>> spread
>>>>> >>>>> >>>>>>>>>> between Classlib and VM, but it seems like DRLVM specific.
>>>>> In
>>>>> >>>>> this
>>>>> >>>>> >>>>>>>>>> case I would rather stay without the patch.
>>>>> >>>>> >>>>>>>>>>
>>>>> >>>>> >>>>>>>>>> Personally (if I'll be committer) I would accept the patch
>>>>> with
>>>>> >>>>> two
>>>>> >>>>> >>>>>>>>>> serious modifications:
>>>>> >>>>> >>>>>>>>>> 1. Stay within DRLVM, do not introduce this feature into
>>>>> >>>>> Classlib,
>>>>> >>>>> >>>>>>>>>> get the thing tested and evolved on DRLVM side. Otherwise
>>>>> it
>>>>> >>>>> might
>>>>> >>>>> >>>>>>>>>> break the compatibility with other VMs.
>>>>> >>>>> >>>>>>>>>> 2. Make the mapping generated automatically (during build
>>>>> >>>>> process?)
>>>>> >>>>> >>>>>>>>>> to free the burden for maintainers.
>>>>> >>>>> >>>>>>>>>>
>>>>> >>>>> >>>>>>>>>> Thanks,
>>>>> >>>>> >>>>>>>>>> Aleksey.
>>>>> >>>>> >>>>>>>>>>
>>>>> >>>>> >>>>>>>>>
>>>>> >>>>> >>>>>>>>
>>>>> >>>>> >>>>>>>
>>>>> >>>>> >>>>>>
>>>>> >>>>> >>>>>
>>>>> >>>>> >>>>
>>>>> >>>>> >>>
>>>>> >>>>> >>
>>>>> >>>>> >
>>>>> >>>>> >
>>>>> >>>>> >
>>>>> >>>>> > --
>>>>> >>>>> > С уважением,
>>>>> >>>>> > Алексей Федотов,
>>>>> >>>>> > ЗАО «Телеком Экспресс»
>>>>> >>>>> >
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>> --
>>>>> >>>>> http://xiao-feng.blogspot.com
>>>>> >>>>>
>>>>> >>>>
>>>>> >>>>
>>>>> >>>>
>>>>> >>>> --
>>>>> >>>> С уважением,
>>>>> >>>> Алексей Федотов,
>>>>> >>>> ЗАО «Телеком Экспресс»
>>>>> >>>>
>>>>> >>>
>>>>> >>>
>>>>> >>>
>>>>> >>> --
>>>>> >>> http://xiao-feng.blogspot.com
>>>>> >>>
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> --
>>>>> >> С уважением,
>>>>> >> Алексей Федотов,
>>>>> >> ЗАО «Телеком Экспресс»
>>>>> >>
>>>>> >
>>>>> >
>>>>> >
>>>>> > --
>>>>> > http://xiao-feng.blogspot.com
>>>>> >
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> С уважением,
>>>> Алексей Федотов,
>>>> ЗАО «Телеком Экспресс»
>>>>
>>>
>>
>
--
С уважением,
Алексей Федотов,
ЗАО «Телеком Экспресс»
Re: [VM] On-demand class library parsing is ready to commit
Posted by Wenlong Li <we...@gmail.com>.
I once used VTune, but it doesn't help too much for this case in
identifying the bottleneck. later I manually insert instrumentation
code to collect execution time, and found class loading in VM_init1
and VM_init2 is time consuming, where jar loading & parsing one of the
hot spots. Just FYI
On Wed, Jan 14, 2009 at 10:25 PM, Pavel Pervov <pm...@gmail.com> wrote:
> BTW, had anyone a chance to profile JVM_CreateJavaVM with a tool like
> VTune or gprof? It'd be interesting to see where most of time is being
> spent during VM creation - it may be something completely different
> than jar files processing.
>
> Pavel.
>
> On Wed, Jan 14, 2009 at 4:41 PM, Wenlong Li <we...@gmail.com> wrote:
>> Btw, Aleksey, to avoid confusion, I expect this patch help VM creation
>> time or at least help the memory footprint since some modules are
>> never used after their loading and parsing. :)
>>
>> On Wed, Jan 14, 2009 at 9:38 PM, Wenlong Li <we...@gmail.com> wrote:
>>> Alexei,
>>>
>>> Sorry for confusing. The patch for review is H6039.patch_2. Please
>>> kindly provide your comment.
>>>
>>> Aleksey,
>>>
>>> I have not measured the performance before completing the code review.
>>> I will do that later.
>>>
>>> thx, Wenlong
>>>
>>> On Wed, Jan 14, 2009 at 9:14 PM, Wenlong Li <we...@gmail.com> wrote:
>>>> Pavel,
>>>>
>>>> Pls see my comments in the JIRA.
>>>>
>>>> thx, Wenlong
>>>>
>>>> On Wed, Jan 14, 2009 at 8:44 PM, Pavel Pervov <pm...@gmail.com> wrote:
>>>>> Please, also, check that jar entry caches still work correctly after your patch.
>>>>>
>>>>> Pavel.
>>>>>
>>>>> On Wed, Jan 14, 2009 at 12:33 PM, Wenlong Li <we...@gmail.com> wrote:
>>>>>> All,
>>>>>>
>>>>>> I submit a new patch for on-demand class loading and parsing. All
>>>>>> codes are put in VM side, and the mapping info is automatically
>>>>>> produced.
>>>>>>
>>>>>> Pls see https://issues.apache.org/jira/browse/HARMONY-6039
>>>>>>
>>>>>> Comments are welcome.
>>>>>>
>>>>>> Thx, Wenlong
>>>>>> Managed Runtime Technology Center, Intel
>>>>>>
>>>>>> On Wed, Jan 7, 2009 at 12:08 PM, Wenlong Li <we...@gmail.com> wrote:
>>>>>>> All,
>>>>>>> At this moment, I move all updates in classlib to VM side such that
>>>>>>> there is no modularity issue. Next step is to produce the mapping
>>>>>>> between module and library efficiently and accurately.
>>>>>>>
>>>>>>> Comments are welcome.
>>>>>>>
>>>>>>> Thx, Wenlong
>>>>>>> Managed Runtime Technology Center, Intel
>>>>>>>
>>>>>>> On Tue, Jan 6, 2009 at 11:08 PM, Wenlong Li <we...@gmail.com> wrote:
>>>>>>>> Thx :)
>>>>>>>>
>>>>>>>> On Tue, Jan 6, 2009 at 10:35 PM, Alexei Fedotov
>>>>>>>> <al...@gmail.com> wrote:
>>>>>>>>> Sure.
>>>>>>>>>
>>>>>>>>> 1. If you dig into SetClasspathFromString, you will see that it starts from
>>>>>>>>> splitting the given classpath into pieces. You already know the new piece
>>>>>>>>> you add and may skip splitting step.
>>>>>>>>>
>>>>>>>>> 2. If I understand you code correctly, the case "pdest >
>>>>>>>>> (*it).second->bytes" might be a subject of a negative assertion. Adding this
>>>>>>>>> assrtion would speed up bug discovery.
>>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Jan 6, 2009 at 5:09 AM, Wenlong Li <we...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Yes, Xiao-Feng's understanding is correct. The patch loads and parses
>>>>>>>>>> modules on demand. If no class in swing.jar is not requested, then
>>>>>>>>>> this module will not be loaded.
>>>>>>>>>>
>>>>>>>>>> btw, Alexei, you said "SetClasspathFromString" and "pdest >
>>>>>>>>>> (*it).second->bytes" are not efficient. Can you share more comments on
>>>>>>>>>> them? I just reused some code in Harmony, and didn't optimize them
>>>>>>>>>> further.
>>>>>>>>>>
>>>>>>>>>> Thx, Wenlong
>>>>>>>>>> Managed Runtime Technology Center, Intel
>>>>>>>>>>
>>>>>>>>>> On Fri, Dec 26, 2008 at 5:16 PM, Xiao-Feng Li <xi...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>> > On Fri, Dec 26, 2008 at 5:08 PM, Alexei Fedotov
>>>>>>>>>> > <al...@gmail.com> wrote:
>>>>>>>>>> >> Xiao Feng,
>>>>>>>>>> >> Thank you for explaining.
>>>>>>>>>> >>
>>>>>>>>>> >> I get more minor comments on more commented code, ineffective
>>>>>>>>>> >> SetClasspathFromString usage, non-covered unexpected case when pdest >
>>>>>>>>>> >> (*it).second->bytes. One major comment on crossing vm module boundary
>>>>>>>>>> >> still remains. But now I'm happy with the design.
>>>>>>>>>> >
>>>>>>>>>> > Alexei, yes, I agree with your comments. These parts should be
>>>>>>>>>> > improved. (Still, this is my personal opinion. :) Let's wait Wenlong
>>>>>>>>>> > speaking.)
>>>>>>>>>> >
>>>>>>>>>> > Thanks,
>>>>>>>>>> > xiaofeng
>>>>>>>>>> >
>>>>>>>>>> >> Sorry for being slow.
>>>>>>>>>> >>
>>>>>>>>>> >>
>>>>>>>>>> >>
>>>>>>>>>> >> On Fri, Dec 26, 2008 at 11:40 AM, Xiao-Feng Li <xi...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>> >>> On Fri, Dec 26, 2008 at 4:03 PM, Alexei Fedotov
>>>>>>>>>> >>> <al...@gmail.com> wrote:
>>>>>>>>>> >>>> Xiao-Feng,
>>>>>>>>>> >>>> Continuing with the server example could you please give me a hint
>>>>>>>>>> where
>>>>>>>>>> >>>> decision to load swing.jar or not is taken in the patch? My initial
>>>>>>>>>> >>>> perception was that the list of what to load was hardcoded and was not
>>>>>>>>>> >>>> constructed dynamically depending on application.
>>>>>>>>>> >>>
>>>>>>>>>> >>> Alexei, here is the patch code I found:
>>>>>>>>>> >>>
>>>>>>>>>> >>> line 245:
>>>>>>>>>> >>> + // Find which jar exports this package
>>>>>>>>>> >>> + if (pkgName != NULL) {
>>>>>>>>>> >>> + char *boot_class_path =
>>>>>>>>>> >>> env->JavaProperties()->get(VM_BOOT_CLASS_DIR);
>>>>>>>>>> >>> + char *pendingClassPath = NULL;
>>>>>>>>>> >>> + apr_pool_t *tmp_pool;
>>>>>>>>>> >>> + apr_pool_create(&tmp_pool, NULL);
>>>>>>>>>> >>> + while (it != env->pending_jar_set.end()) {
>>>>>>>>>> >>> + pdest = strstr( (*it).second->bytes, pkgName );
>>>>>>>>>> >>> + if (pdest != NULL) {
>>>>>>>>>> >>> + pendingClassPath =
>>>>>>>>>> >>> (char*)STD_MALLOC(strlen(boot_class_path)
>>>>>>>>>> >>> + +
>>>>>>>>>> strlen((*it).first->bytes) + 1);
>>>>>>>>>> >>> + strcpy(pendingClassPath, boot_class_path);
>>>>>>>>>> >>> + strcat(pendingClassPath, (*it).first->bytes);
>>>>>>>>>> >>> + // Open this found jar, and read all classes
>>>>>>>>>> >>> contained in this jar
>>>>>>>>>> >>> + SetClasspathFromString(pendingClassPath,
>>>>>>>>>> tmp_pool);
>>>>>>>>>> >>> + // Erase the found jar from pending jar list
>>>>>>>>>> >>> as it has been parsed
>>>>>>>>>> >>> + env->pending_jar_set.erase(it++);
>>>>>>>>>> >>> + STD_FREE(pendingClassPath);
>>>>>>>>>> >>> + } else {
>>>>>>>>>> >>>
>>>>>>>>>> >>> It checks if a JAR has the requested package, then loads it if yes. I
>>>>>>>>>> >>> am not sure if this is what you were asking. (Btw, this is only my
>>>>>>>>>> >>> understanding of his patch. I am not speaking for Wenlong.)
>>>>>>>>>> >>>
>>>>>>>>>> >>> Thanks,
>>>>>>>>>> >>> xiaofeng
>>>>>>>>>> >>>
>>>>>>>>>> >>>> Thanks.
>>>>>>>>>> >>>>
>>>>>>>>>> >>>>
>>>>>>>>>> >>>> On Fri, Dec 26, 2008 at 4:14 AM, Xiao-Feng Li <xi...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>> >>>>
>>>>>>>>>> >>>>> On Fri, Dec 26, 2008 at 12:49 AM, Alexei Fedotov
>>>>>>>>>> >>>>> <al...@gmail.com> wrote:
>>>>>>>>>> >>>>> > Aleksey,
>>>>>>>>>> >>>>> > I like your conclusion.
>>>>>>>>>> >>>>> >
>>>>>>>>>> >>>>> > Wenlong,
>>>>>>>>>> >>>>> > I'm trying to understand the real life value of the "abstract"
>>>>>>>>>> startup
>>>>>>>>>> >>>>> > time metric you've suggested. Does Harmony with your patch load
>>>>>>>>>> >>>>> > swing.jar for a server application? Do I understand that loading
>>>>>>>>>> >>>>> > happens, though it happens later compared to VM without your patch?
>>>>>>>>>> I
>>>>>>>>>> >>>>> > believe that the proper design of delayed loading should answer
>>>>>>>>>> "no"
>>>>>>>>>> >>>>> > to this question.
>>>>>>>>>> >>>>>
>>>>>>>>>> >>>>> I checked the patch, and I found the answer is indeed "No" as you
>>>>>>>>>> expected.
>>>>>>>>>> >>>>>
>>>>>>>>>> >>>>> Thanks,
>>>>>>>>>> >>>>> xiaofeng
>>>>>>>>>> >>>>>
>>>>>>>>>> >>>>> > In other words, I appreciate if you describe which real use cases
>>>>>>>>>> are
>>>>>>>>>> >>>>> > improved by this patch.
>>>>>>>>>> >>>>> > Thanks!
>>>>>>>>>> >>>>>
>>>>>>>>>> >>>>> > On Thu, Dec 25, 2008 at 7:29 PM, Aleksey Shipilev
>>>>>>>>>> >>>>> > <al...@gmail.com> wrote:
>>>>>>>>>> >>>>> >> Ok, here's the catch.
>>>>>>>>>> >>>>> >>
>>>>>>>>>> >>>>> >> bootclasspath.properties is the SortedSet<JARfile>, which
>>>>>>>>>> enumerates
>>>>>>>>>> >>>>> >> the JARs available for bootclassloader. The set of such the JARs
>>>>>>>>>> is
>>>>>>>>>> >>>>> >> really stable because modular decomposition of classlib is stable.
>>>>>>>>>> >>>>> >> That's why nobody bothers with automatic generation of it: it only
>>>>>>>>>> >>>>> >> should be updated when new JAR file arrives.
>>>>>>>>>> >>>>> >> Modulelibrarymapping.properties is different on this point, it's
>>>>>>>>>> the
>>>>>>>>>> >>>>> >> Map<PackageName,JARfile>, which should be updated each time the
>>>>>>>>>> new
>>>>>>>>>> >>>>> >> *package* is introduced. I'm not talking about java.* packages
>>>>>>>>>> >>>>> >> (they're standardized), rather about org.apache.harmony.*.
>>>>>>>>>> >>>>> >>
>>>>>>>>>> >>>>> >> Automatic generation of this property file gives two advantages:
>>>>>>>>>> >>>>> >> 1. Error-prone. Prevent yourself from hand-messing with mapping
>>>>>>>>>> and
>>>>>>>>>> >>>>> >> getting spurious ClassNotFoundException. BTW, what's the behaviour
>>>>>>>>>> in
>>>>>>>>>> >>>>> >> case the mapping is wrong?
>>>>>>>>>> >>>>> >> 2. "Researchful". There're lot of guys around who enjoys the
>>>>>>>>>> >>>>> >> modularity of Harmony classlib and eventually they might want to
>>>>>>>>>> split
>>>>>>>>>> >>>>> >> the packages even deeper, into smaller pieces. Then automatic
>>>>>>>>>> >>>>> >> generation would enable them to quickly roll-in and experiment
>>>>>>>>>> with
>>>>>>>>>> >>>>> >> different package layouts and their impact on performance. They
>>>>>>>>>> could
>>>>>>>>>> >>>>> >> use ordinary bootclasspath.properties, but your feature wouldn't
>>>>>>>>>> be
>>>>>>>>>> >>>>> >> used by them then ;)
>>>>>>>>>> >>>>> >>
>>>>>>>>>> >>>>> >> That's merely a housekeeping procedure. I believe that anything
>>>>>>>>>> which
>>>>>>>>>> >>>>> >> could be done more than once, eventually would be done more than
>>>>>>>>>> once.
>>>>>>>>>> >>>>> >> Hence it should be automated. You say that the file was generated
>>>>>>>>>> from
>>>>>>>>>> >>>>> >> manifests of JARs, so is it hard to just tie the same tool into
>>>>>>>>>> DRLVM
>>>>>>>>>> >>>>> >> build process?
>>>>>>>>>> >>>>> >>
>>>>>>>>>> >>>>> >> As for DRLVM-specific, my opinion that this is because the patch:
>>>>>>>>>> >>>>> >> a. breaks the compatibility of classlib (you change
>>>>>>>>>> >>>>> >> bootclasspath.properties, right?) with other VMs.
>>>>>>>>>> >>>>> >> b. treated in DRLVM classloader only.
>>>>>>>>>> >>>>> >>
>>>>>>>>>> >>>>> >> Of course eventually this feature might be used by others, but IMO
>>>>>>>>>> we
>>>>>>>>>> >>>>> >> should be careful about other guys who use the same classlib. I'd
>>>>>>>>>> >>>>> >> rather wait for some incubation on DRLVM side first.
>>>>>>>>>> >>>>> >>
>>>>>>>>>> >>>>> >> Thanks,
>>>>>>>>>> >>>>> >> Aleksey.
>>>>>>>>>> >>>>> >>
>>>>>>>>>> >>>>> >> On Thu, Dec 25, 2008 at 6:18 PM, Wenlong Li <we...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>> >>>>> >>> I see. In fact, my file doesn't need track change at the class
>>>>>>>>>> >>>>> >>> granularity. Instead, it only needs know package info provided in
>>>>>>>>>> the
>>>>>>>>>> >>>>> >>> manifest file. When class is added to a library, do we need
>>>>>>>>>> change
>>>>>>>>>> >>>>> >>> the manifest as well?
>>>>>>>>>> >>>>> >>>
>>>>>>>>>> >>>>> >>> btw, I guess there is a mis-understanding: my
>>>>>>>>>> modulelibrarymapping
>>>>>>>>>> >>>>> >>> file only records package info provided by manfiest in each
>>>>>>>>>> module. It
>>>>>>>>>> >>>>> >>> doesn't relate to each class.
>>>>>>>>>> >>>>> >>>
>>>>>>>>>> >>>>> >>> thx,
>>>>>>>>>> >>>>> >>> Wenlong
>>>>>>>>>> >>>>> >>>
>>>>>>>>>> >>>>> >>> On Thu, Dec 25, 2008 at 10:55 PM, Pavel Pervov <
>>>>>>>>>> pmcfirst@gmail.com>
>>>>>>>>>> >>>>> wrote:
>>>>>>>>>> >>>>> >>>> Classes are added to class library from time to time. I'm not
>>>>>>>>>> sure how
>>>>>>>>>> >>>>> >>>> it can be possible to track these changes manually.
>>>>>>>>>> >>>>> >>>>
>>>>>>>>>> >>>>> >>>> WBR,
>>>>>>>>>> >>>>> >>>> Pavel.
>>>>>>>>>> >>>>> >>>>
>>>>>>>>>> >>>>> >>>>
>>>>>>>>>> >>>>> >>>> On Thu, Dec 25, 2008 at 5:09 PM, Wenlong Li <we...@gmail.com>
>>>>>>>>>> >>>>> wrote:
>>>>>>>>>> >>>>> >>>>> Sorry, one more question: bootclasspath.properties is classlib
>>>>>>>>>> >>>>> >>>>> specific file, why we could not make a vm specific file
>>>>>>>>>> manually?
>>>>>>>>>> >>>>> Just
>>>>>>>>>> >>>>> >>>>> curious to know the possible reason. :)
>>>>>>>>>> >>>>> >>>>>
>>>>>>>>>> >>>>> >>>>> thx,
>>>>>>>>>> >>>>> >>>>> Wenlong
>>>>>>>>>> >>>>> >>>>>
>>>>>>>>>> >>>>> >>>>> On Thu, Dec 25, 2008 at 10:00 PM, Pavel Pervov <
>>>>>>>>>> pmcfirst@gmail.com>
>>>>>>>>>> >>>>> wrote:
>>>>>>>>>> >>>>> >>>>>> If this would be VM-side automatically produced configuration
>>>>>>>>>> >>>>> file...
>>>>>>>>>> >>>>> >>>>>>
>>>>>>>>>> >>>>> >>>>>> WBR,
>>>>>>>>>> >>>>> >>>>>> Pavel.
>>>>>>>>>> >>>>> >>>>>>
>>>>>>>>>> >>>>> >>>>>> On Thu, Dec 25, 2008 at 4:58 PM, Wenlong Li <
>>>>>>>>>> wenlong@gmail.com>
>>>>>>>>>> >>>>> wrote:
>>>>>>>>>> >>>>> >>>>>>> btw, because adding new module is rare case, manually
>>>>>>>>>> modifying the
>>>>>>>>>> >>>>> >>>>>>> bootclasspath.properties is not an issue?
>>>>>>>>>> >>>>> >>>>>>>
>>>>>>>>>> >>>>> >>>>>>> If so, can I conclude adding another property file with same
>>>>>>>>>> update
>>>>>>>>>> >>>>> >>>>>>> frequency as bootclasspath would be fine as well?
>>>>>>>>>> >>>>> >>>>>>>
>>>>>>>>>> >>>>> >>>>>>> Pls kindly correct me if my understanding is wrong.
>>>>>>>>>> >>>>> >>>>>>>
>>>>>>>>>> >>>>> >>>>>>> On Thu, Dec 25, 2008 at 9:05 PM, Pavel Pervov <
>>>>>>>>>> pmcfirst@gmail.com>
>>>>>>>>>> >>>>> wrote:
>>>>>>>>>> >>>>> >>>>>>>> Wenlong,
>>>>>>>>>> >>>>> >>>>>>>>
>>>>>>>>>> >>>>> >>>>>>>> Note, that bootclasspath.properties is only changed on
>>>>>>>>>> adding new
>>>>>>>>>> >>>>> >>>>>>>> module. This is pretty rare occasion, I believe.
>>>>>>>>>> >>>>> >>>>>>>>
>>>>>>>>>> >>>>> >>>>>>>> WBR,
>>>>>>>>>> >>>>> >>>>>>>> Pavel.
>>>>>>>>>> >>>>> >>>>>>>>
>>>>>>>>>> >>>>> >>>>>>>> On Thu, Dec 25, 2008 at 3:48 PM, Wenlong Li <
>>>>>>>>>> wenlong@gmail.com>
>>>>>>>>>> >>>>> wrote:
>>>>>>>>>> >>>>> >>>>>>>>> Thx for your advice. Alexey.
>>>>>>>>>> >>>>> >>>>>>>>>
>>>>>>>>>> >>>>> >>>>>>>>> Here I have one question: do you know how the
>>>>>>>>>> >>>>> bootclasspath.properties
>>>>>>>>>> >>>>> >>>>>>>>> is maintained, manually or automatically?
>>>>>>>>>> >>>>> >>>>>>>>>
>>>>>>>>>> >>>>> >>>>>>>>> Another comment is I would like to treat the patch as DRLVM
>>>>>>>>>> >>>>> specific
>>>>>>>>>> >>>>> >>>>>>>>> optimization, e.g., it targets for improving VM creation
>>>>>>>>>> time. So
>>>>>>>>>> >>>>> that
>>>>>>>>>> >>>>> >>>>>>>>> is possible to move all updates to DRLVM part to eliminate
>>>>>>>>>> >>>>> potential
>>>>>>>>>> >>>>> >>>>>>>>> modularity and compatibility problem.
>>>>>>>>>> >>>>> >>>>>>>>>
>>>>>>>>>> >>>>> >>>>>>>>> thx,
>>>>>>>>>> >>>>> >>>>>>>>> Wenlong
>>>>>>>>>> >>>>> >>>>>>>>>
>>>>>>>>>> >>>>> >>>>>>>>> On Thu, Dec 25, 2008 at 5:32 PM, Aleksey Shipilev
>>>>>>>>>> >>>>> >>>>>>>>> <al...@gmail.com> wrote:
>>>>>>>>>> >>>>> >>>>>>>>>> Hi, Wenlong.
>>>>>>>>>> >>>>> >>>>>>>>>>
>>>>>>>>>> >>>>> >>>>>>>>>> On Thu, Dec 25, 2008 at 11:49 AM, Wenlong Li <
>>>>>>>>>> wenlong@gmail.com>
>>>>>>>>>> >>>>> wrote:
>>>>>>>>>> >>>>> >>>>>>>>>>> btw, Alexey, Let's go back to discuss whether there is a
>>>>>>>>>> need
>>>>>>>>>> >>>>> to
>>>>>>>>>> >>>>> >>>>>>>>>>> include this feature in Harmony, given 17% performance
>>>>>>>>>> gain in
>>>>>>>>>> >>>>> Linux
>>>>>>>>>> >>>>> >>>>>>>>>>> when using your methodology. For windows test, I will
>>>>>>>>>> double
>>>>>>>>>> >>>>> check the
>>>>>>>>>> >>>>> >>>>>>>>>>> backgroud process as you pointed out.
>>>>>>>>>> >>>>> >>>>>>>>>>
>>>>>>>>>> >>>>> >>>>>>>>>> My opinion was already expressed after I had finished the
>>>>>>>>>> tests
>>>>>>>>>> >>>>> from
>>>>>>>>>> >>>>> >>>>>>>>>> my side: the boost can be achieved in specific conditions,
>>>>>>>>>> so
>>>>>>>>>> >>>>> whether
>>>>>>>>>> >>>>> >>>>>>>>>> it's worth including into Harmony really depends on how
>>>>>>>>>> much
>>>>>>>>>> >>>>> mess the
>>>>>>>>>> >>>>> >>>>>>>>>> patch would introduce besides the "performance boost".
>>>>>>>>>> From what
>>>>>>>>>> >>>>> I
>>>>>>>>>> >>>>> >>>>>>>>>> see, the patch obliges the maintainer to maintain the
>>>>>>>>>> correct
>>>>>>>>>> >>>>> mapping
>>>>>>>>>> >>>>> >>>>>>>>>> between jars and Java packages. This new feature is also
>>>>>>>>>> spread
>>>>>>>>>> >>>>> >>>>>>>>>> between Classlib and VM, but it seems like DRLVM specific.
>>>>>>>>>> In
>>>>>>>>>> >>>>> this
>>>>>>>>>> >>>>> >>>>>>>>>> case I would rather stay without the patch.
>>>>>>>>>> >>>>> >>>>>>>>>>
>>>>>>>>>> >>>>> >>>>>>>>>> Personally (if I'll be committer) I would accept the patch
>>>>>>>>>> with
>>>>>>>>>> >>>>> two
>>>>>>>>>> >>>>> >>>>>>>>>> serious modifications:
>>>>>>>>>> >>>>> >>>>>>>>>> 1. Stay within DRLVM, do not introduce this feature into
>>>>>>>>>> >>>>> Classlib,
>>>>>>>>>> >>>>> >>>>>>>>>> get the thing tested and evolved on DRLVM side. Otherwise
>>>>>>>>>> it
>>>>>>>>>> >>>>> might
>>>>>>>>>> >>>>> >>>>>>>>>> break the compatibility with other VMs.
>>>>>>>>>> >>>>> >>>>>>>>>> 2. Make the mapping generated automatically (during build
>>>>>>>>>> >>>>> process?)
>>>>>>>>>> >>>>> >>>>>>>>>> to free the burden for maintainers.
>>>>>>>>>> >>>>> >>>>>>>>>>
>>>>>>>>>> >>>>> >>>>>>>>>> Thanks,
>>>>>>>>>> >>>>> >>>>>>>>>> Aleksey.
>>>>>>>>>> >>>>> >>>>>>>>>>
>>>>>>>>>> >>>>> >>>>>>>>>
>>>>>>>>>> >>>>> >>>>>>>>
>>>>>>>>>> >>>>> >>>>>>>
>>>>>>>>>> >>>>> >>>>>>
>>>>>>>>>> >>>>> >>>>>
>>>>>>>>>> >>>>> >>>>
>>>>>>>>>> >>>>> >>>
>>>>>>>>>> >>>>> >>
>>>>>>>>>> >>>>> >
>>>>>>>>>> >>>>> >
>>>>>>>>>> >>>>> >
>>>>>>>>>> >>>>> > --
>>>>>>>>>> >>>>> > С уважением,
>>>>>>>>>> >>>>> > Алексей Федотов,
>>>>>>>>>> >>>>> > ЗАО «Телеком Экспресс»
>>>>>>>>>> >>>>> >
>>>>>>>>>> >>>>>
>>>>>>>>>> >>>>>
>>>>>>>>>> >>>>>
>>>>>>>>>> >>>>> --
>>>>>>>>>> >>>>> http://xiao-feng.blogspot.com
>>>>>>>>>> >>>>>
>>>>>>>>>> >>>>
>>>>>>>>>> >>>>
>>>>>>>>>> >>>>
>>>>>>>>>> >>>> --
>>>>>>>>>> >>>> С уважением,
>>>>>>>>>> >>>> Алексей Федотов,
>>>>>>>>>> >>>> ЗАО «Телеком Экспресс»
>>>>>>>>>> >>>>
>>>>>>>>>> >>>
>>>>>>>>>> >>>
>>>>>>>>>> >>>
>>>>>>>>>> >>> --
>>>>>>>>>> >>> http://xiao-feng.blogspot.com
>>>>>>>>>> >>>
>>>>>>>>>> >>
>>>>>>>>>> >>
>>>>>>>>>> >>
>>>>>>>>>> >> --
>>>>>>>>>> >> С уважением,
>>>>>>>>>> >> Алексей Федотов,
>>>>>>>>>> >> ЗАО «Телеком Экспресс»
>>>>>>>>>> >>
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> > --
>>>>>>>>>> > http://xiao-feng.blogspot.com
>>>>>>>>>> >
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> С уважением,
>>>>>>>>> Алексей Федотов,
>>>>>>>>> ЗАО «Телеком Экспресс»
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: [VM] On-demand class library parsing is ready to commit
Posted by Pavel Pervov <pm...@gmail.com>.
BTW, had anyone a chance to profile JVM_CreateJavaVM with a tool like
VTune or gprof? It'd be interesting to see where most of time is being
spent during VM creation - it may be something completely different
than jar files processing.
Pavel.
On Wed, Jan 14, 2009 at 4:41 PM, Wenlong Li <we...@gmail.com> wrote:
> Btw, Aleksey, to avoid confusion, I expect this patch help VM creation
> time or at least help the memory footprint since some modules are
> never used after their loading and parsing. :)
>
> On Wed, Jan 14, 2009 at 9:38 PM, Wenlong Li <we...@gmail.com> wrote:
>> Alexei,
>>
>> Sorry for confusing. The patch for review is H6039.patch_2. Please
>> kindly provide your comment.
>>
>> Aleksey,
>>
>> I have not measured the performance before completing the code review.
>> I will do that later.
>>
>> thx, Wenlong
>>
>> On Wed, Jan 14, 2009 at 9:14 PM, Wenlong Li <we...@gmail.com> wrote:
>>> Pavel,
>>>
>>> Pls see my comments in the JIRA.
>>>
>>> thx, Wenlong
>>>
>>> On Wed, Jan 14, 2009 at 8:44 PM, Pavel Pervov <pm...@gmail.com> wrote:
>>>> Please, also, check that jar entry caches still work correctly after your patch.
>>>>
>>>> Pavel.
>>>>
>>>> On Wed, Jan 14, 2009 at 12:33 PM, Wenlong Li <we...@gmail.com> wrote:
>>>>> All,
>>>>>
>>>>> I submit a new patch for on-demand class loading and parsing. All
>>>>> codes are put in VM side, and the mapping info is automatically
>>>>> produced.
>>>>>
>>>>> Pls see https://issues.apache.org/jira/browse/HARMONY-6039
>>>>>
>>>>> Comments are welcome.
>>>>>
>>>>> Thx, Wenlong
>>>>> Managed Runtime Technology Center, Intel
>>>>>
>>>>> On Wed, Jan 7, 2009 at 12:08 PM, Wenlong Li <we...@gmail.com> wrote:
>>>>>> All,
>>>>>> At this moment, I move all updates in classlib to VM side such that
>>>>>> there is no modularity issue. Next step is to produce the mapping
>>>>>> between module and library efficiently and accurately.
>>>>>>
>>>>>> Comments are welcome.
>>>>>>
>>>>>> Thx, Wenlong
>>>>>> Managed Runtime Technology Center, Intel
>>>>>>
>>>>>> On Tue, Jan 6, 2009 at 11:08 PM, Wenlong Li <we...@gmail.com> wrote:
>>>>>>> Thx :)
>>>>>>>
>>>>>>> On Tue, Jan 6, 2009 at 10:35 PM, Alexei Fedotov
>>>>>>> <al...@gmail.com> wrote:
>>>>>>>> Sure.
>>>>>>>>
>>>>>>>> 1. If you dig into SetClasspathFromString, you will see that it starts from
>>>>>>>> splitting the given classpath into pieces. You already know the new piece
>>>>>>>> you add and may skip splitting step.
>>>>>>>>
>>>>>>>> 2. If I understand you code correctly, the case "pdest >
>>>>>>>> (*it).second->bytes" might be a subject of a negative assertion. Adding this
>>>>>>>> assrtion would speed up bug discovery.
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Jan 6, 2009 at 5:09 AM, Wenlong Li <we...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Yes, Xiao-Feng's understanding is correct. The patch loads and parses
>>>>>>>>> modules on demand. If no class in swing.jar is not requested, then
>>>>>>>>> this module will not be loaded.
>>>>>>>>>
>>>>>>>>> btw, Alexei, you said "SetClasspathFromString" and "pdest >
>>>>>>>>> (*it).second->bytes" are not efficient. Can you share more comments on
>>>>>>>>> them? I just reused some code in Harmony, and didn't optimize them
>>>>>>>>> further.
>>>>>>>>>
>>>>>>>>> Thx, Wenlong
>>>>>>>>> Managed Runtime Technology Center, Intel
>>>>>>>>>
>>>>>>>>> On Fri, Dec 26, 2008 at 5:16 PM, Xiao-Feng Li <xi...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>> > On Fri, Dec 26, 2008 at 5:08 PM, Alexei Fedotov
>>>>>>>>> > <al...@gmail.com> wrote:
>>>>>>>>> >> Xiao Feng,
>>>>>>>>> >> Thank you for explaining.
>>>>>>>>> >>
>>>>>>>>> >> I get more minor comments on more commented code, ineffective
>>>>>>>>> >> SetClasspathFromString usage, non-covered unexpected case when pdest >
>>>>>>>>> >> (*it).second->bytes. One major comment on crossing vm module boundary
>>>>>>>>> >> still remains. But now I'm happy with the design.
>>>>>>>>> >
>>>>>>>>> > Alexei, yes, I agree with your comments. These parts should be
>>>>>>>>> > improved. (Still, this is my personal opinion. :) Let's wait Wenlong
>>>>>>>>> > speaking.)
>>>>>>>>> >
>>>>>>>>> > Thanks,
>>>>>>>>> > xiaofeng
>>>>>>>>> >
>>>>>>>>> >> Sorry for being slow.
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >> On Fri, Dec 26, 2008 at 11:40 AM, Xiao-Feng Li <xi...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>> >>> On Fri, Dec 26, 2008 at 4:03 PM, Alexei Fedotov
>>>>>>>>> >>> <al...@gmail.com> wrote:
>>>>>>>>> >>>> Xiao-Feng,
>>>>>>>>> >>>> Continuing with the server example could you please give me a hint
>>>>>>>>> where
>>>>>>>>> >>>> decision to load swing.jar or not is taken in the patch? My initial
>>>>>>>>> >>>> perception was that the list of what to load was hardcoded and was not
>>>>>>>>> >>>> constructed dynamically depending on application.
>>>>>>>>> >>>
>>>>>>>>> >>> Alexei, here is the patch code I found:
>>>>>>>>> >>>
>>>>>>>>> >>> line 245:
>>>>>>>>> >>> + // Find which jar exports this package
>>>>>>>>> >>> + if (pkgName != NULL) {
>>>>>>>>> >>> + char *boot_class_path =
>>>>>>>>> >>> env->JavaProperties()->get(VM_BOOT_CLASS_DIR);
>>>>>>>>> >>> + char *pendingClassPath = NULL;
>>>>>>>>> >>> + apr_pool_t *tmp_pool;
>>>>>>>>> >>> + apr_pool_create(&tmp_pool, NULL);
>>>>>>>>> >>> + while (it != env->pending_jar_set.end()) {
>>>>>>>>> >>> + pdest = strstr( (*it).second->bytes, pkgName );
>>>>>>>>> >>> + if (pdest != NULL) {
>>>>>>>>> >>> + pendingClassPath =
>>>>>>>>> >>> (char*)STD_MALLOC(strlen(boot_class_path)
>>>>>>>>> >>> + +
>>>>>>>>> strlen((*it).first->bytes) + 1);
>>>>>>>>> >>> + strcpy(pendingClassPath, boot_class_path);
>>>>>>>>> >>> + strcat(pendingClassPath, (*it).first->bytes);
>>>>>>>>> >>> + // Open this found jar, and read all classes
>>>>>>>>> >>> contained in this jar
>>>>>>>>> >>> + SetClasspathFromString(pendingClassPath,
>>>>>>>>> tmp_pool);
>>>>>>>>> >>> + // Erase the found jar from pending jar list
>>>>>>>>> >>> as it has been parsed
>>>>>>>>> >>> + env->pending_jar_set.erase(it++);
>>>>>>>>> >>> + STD_FREE(pendingClassPath);
>>>>>>>>> >>> + } else {
>>>>>>>>> >>>
>>>>>>>>> >>> It checks if a JAR has the requested package, then loads it if yes. I
>>>>>>>>> >>> am not sure if this is what you were asking. (Btw, this is only my
>>>>>>>>> >>> understanding of his patch. I am not speaking for Wenlong.)
>>>>>>>>> >>>
>>>>>>>>> >>> Thanks,
>>>>>>>>> >>> xiaofeng
>>>>>>>>> >>>
>>>>>>>>> >>>> Thanks.
>>>>>>>>> >>>>
>>>>>>>>> >>>>
>>>>>>>>> >>>> On Fri, Dec 26, 2008 at 4:14 AM, Xiao-Feng Li <xi...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>> >>>>
>>>>>>>>> >>>>> On Fri, Dec 26, 2008 at 12:49 AM, Alexei Fedotov
>>>>>>>>> >>>>> <al...@gmail.com> wrote:
>>>>>>>>> >>>>> > Aleksey,
>>>>>>>>> >>>>> > I like your conclusion.
>>>>>>>>> >>>>> >
>>>>>>>>> >>>>> > Wenlong,
>>>>>>>>> >>>>> > I'm trying to understand the real life value of the "abstract"
>>>>>>>>> startup
>>>>>>>>> >>>>> > time metric you've suggested. Does Harmony with your patch load
>>>>>>>>> >>>>> > swing.jar for a server application? Do I understand that loading
>>>>>>>>> >>>>> > happens, though it happens later compared to VM without your patch?
>>>>>>>>> I
>>>>>>>>> >>>>> > believe that the proper design of delayed loading should answer
>>>>>>>>> "no"
>>>>>>>>> >>>>> > to this question.
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> I checked the patch, and I found the answer is indeed "No" as you
>>>>>>>>> expected.
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> Thanks,
>>>>>>>>> >>>>> xiaofeng
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> > In other words, I appreciate if you describe which real use cases
>>>>>>>>> are
>>>>>>>>> >>>>> > improved by this patch.
>>>>>>>>> >>>>> > Thanks!
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> > On Thu, Dec 25, 2008 at 7:29 PM, Aleksey Shipilev
>>>>>>>>> >>>>> > <al...@gmail.com> wrote:
>>>>>>>>> >>>>> >> Ok, here's the catch.
>>>>>>>>> >>>>> >>
>>>>>>>>> >>>>> >> bootclasspath.properties is the SortedSet<JARfile>, which
>>>>>>>>> enumerates
>>>>>>>>> >>>>> >> the JARs available for bootclassloader. The set of such the JARs
>>>>>>>>> is
>>>>>>>>> >>>>> >> really stable because modular decomposition of classlib is stable.
>>>>>>>>> >>>>> >> That's why nobody bothers with automatic generation of it: it only
>>>>>>>>> >>>>> >> should be updated when new JAR file arrives.
>>>>>>>>> >>>>> >> Modulelibrarymapping.properties is different on this point, it's
>>>>>>>>> the
>>>>>>>>> >>>>> >> Map<PackageName,JARfile>, which should be updated each time the
>>>>>>>>> new
>>>>>>>>> >>>>> >> *package* is introduced. I'm not talking about java.* packages
>>>>>>>>> >>>>> >> (they're standardized), rather about org.apache.harmony.*.
>>>>>>>>> >>>>> >>
>>>>>>>>> >>>>> >> Automatic generation of this property file gives two advantages:
>>>>>>>>> >>>>> >> 1. Error-prone. Prevent yourself from hand-messing with mapping
>>>>>>>>> and
>>>>>>>>> >>>>> >> getting spurious ClassNotFoundException. BTW, what's the behaviour
>>>>>>>>> in
>>>>>>>>> >>>>> >> case the mapping is wrong?
>>>>>>>>> >>>>> >> 2. "Researchful". There're lot of guys around who enjoys the
>>>>>>>>> >>>>> >> modularity of Harmony classlib and eventually they might want to
>>>>>>>>> split
>>>>>>>>> >>>>> >> the packages even deeper, into smaller pieces. Then automatic
>>>>>>>>> >>>>> >> generation would enable them to quickly roll-in and experiment
>>>>>>>>> with
>>>>>>>>> >>>>> >> different package layouts and their impact on performance. They
>>>>>>>>> could
>>>>>>>>> >>>>> >> use ordinary bootclasspath.properties, but your feature wouldn't
>>>>>>>>> be
>>>>>>>>> >>>>> >> used by them then ;)
>>>>>>>>> >>>>> >>
>>>>>>>>> >>>>> >> That's merely a housekeeping procedure. I believe that anything
>>>>>>>>> which
>>>>>>>>> >>>>> >> could be done more than once, eventually would be done more than
>>>>>>>>> once.
>>>>>>>>> >>>>> >> Hence it should be automated. You say that the file was generated
>>>>>>>>> from
>>>>>>>>> >>>>> >> manifests of JARs, so is it hard to just tie the same tool into
>>>>>>>>> DRLVM
>>>>>>>>> >>>>> >> build process?
>>>>>>>>> >>>>> >>
>>>>>>>>> >>>>> >> As for DRLVM-specific, my opinion that this is because the patch:
>>>>>>>>> >>>>> >> a. breaks the compatibility of classlib (you change
>>>>>>>>> >>>>> >> bootclasspath.properties, right?) with other VMs.
>>>>>>>>> >>>>> >> b. treated in DRLVM classloader only.
>>>>>>>>> >>>>> >>
>>>>>>>>> >>>>> >> Of course eventually this feature might be used by others, but IMO
>>>>>>>>> we
>>>>>>>>> >>>>> >> should be careful about other guys who use the same classlib. I'd
>>>>>>>>> >>>>> >> rather wait for some incubation on DRLVM side first.
>>>>>>>>> >>>>> >>
>>>>>>>>> >>>>> >> Thanks,
>>>>>>>>> >>>>> >> Aleksey.
>>>>>>>>> >>>>> >>
>>>>>>>>> >>>>> >> On Thu, Dec 25, 2008 at 6:18 PM, Wenlong Li <we...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>> >>>>> >>> I see. In fact, my file doesn't need track change at the class
>>>>>>>>> >>>>> >>> granularity. Instead, it only needs know package info provided in
>>>>>>>>> the
>>>>>>>>> >>>>> >>> manifest file. When class is added to a library, do we need
>>>>>>>>> change
>>>>>>>>> >>>>> >>> the manifest as well?
>>>>>>>>> >>>>> >>>
>>>>>>>>> >>>>> >>> btw, I guess there is a mis-understanding: my
>>>>>>>>> modulelibrarymapping
>>>>>>>>> >>>>> >>> file only records package info provided by manfiest in each
>>>>>>>>> module. It
>>>>>>>>> >>>>> >>> doesn't relate to each class.
>>>>>>>>> >>>>> >>>
>>>>>>>>> >>>>> >>> thx,
>>>>>>>>> >>>>> >>> Wenlong
>>>>>>>>> >>>>> >>>
>>>>>>>>> >>>>> >>> On Thu, Dec 25, 2008 at 10:55 PM, Pavel Pervov <
>>>>>>>>> pmcfirst@gmail.com>
>>>>>>>>> >>>>> wrote:
>>>>>>>>> >>>>> >>>> Classes are added to class library from time to time. I'm not
>>>>>>>>> sure how
>>>>>>>>> >>>>> >>>> it can be possible to track these changes manually.
>>>>>>>>> >>>>> >>>>
>>>>>>>>> >>>>> >>>> WBR,
>>>>>>>>> >>>>> >>>> Pavel.
>>>>>>>>> >>>>> >>>>
>>>>>>>>> >>>>> >>>>
>>>>>>>>> >>>>> >>>> On Thu, Dec 25, 2008 at 5:09 PM, Wenlong Li <we...@gmail.com>
>>>>>>>>> >>>>> wrote:
>>>>>>>>> >>>>> >>>>> Sorry, one more question: bootclasspath.properties is classlib
>>>>>>>>> >>>>> >>>>> specific file, why we could not make a vm specific file
>>>>>>>>> manually?
>>>>>>>>> >>>>> Just
>>>>>>>>> >>>>> >>>>> curious to know the possible reason. :)
>>>>>>>>> >>>>> >>>>>
>>>>>>>>> >>>>> >>>>> thx,
>>>>>>>>> >>>>> >>>>> Wenlong
>>>>>>>>> >>>>> >>>>>
>>>>>>>>> >>>>> >>>>> On Thu, Dec 25, 2008 at 10:00 PM, Pavel Pervov <
>>>>>>>>> pmcfirst@gmail.com>
>>>>>>>>> >>>>> wrote:
>>>>>>>>> >>>>> >>>>>> If this would be VM-side automatically produced configuration
>>>>>>>>> >>>>> file...
>>>>>>>>> >>>>> >>>>>>
>>>>>>>>> >>>>> >>>>>> WBR,
>>>>>>>>> >>>>> >>>>>> Pavel.
>>>>>>>>> >>>>> >>>>>>
>>>>>>>>> >>>>> >>>>>> On Thu, Dec 25, 2008 at 4:58 PM, Wenlong Li <
>>>>>>>>> wenlong@gmail.com>
>>>>>>>>> >>>>> wrote:
>>>>>>>>> >>>>> >>>>>>> btw, because adding new module is rare case, manually
>>>>>>>>> modifying the
>>>>>>>>> >>>>> >>>>>>> bootclasspath.properties is not an issue?
>>>>>>>>> >>>>> >>>>>>>
>>>>>>>>> >>>>> >>>>>>> If so, can I conclude adding another property file with same
>>>>>>>>> update
>>>>>>>>> >>>>> >>>>>>> frequency as bootclasspath would be fine as well?
>>>>>>>>> >>>>> >>>>>>>
>>>>>>>>> >>>>> >>>>>>> Pls kindly correct me if my understanding is wrong.
>>>>>>>>> >>>>> >>>>>>>
>>>>>>>>> >>>>> >>>>>>> On Thu, Dec 25, 2008 at 9:05 PM, Pavel Pervov <
>>>>>>>>> pmcfirst@gmail.com>
>>>>>>>>> >>>>> wrote:
>>>>>>>>> >>>>> >>>>>>>> Wenlong,
>>>>>>>>> >>>>> >>>>>>>>
>>>>>>>>> >>>>> >>>>>>>> Note, that bootclasspath.properties is only changed on
>>>>>>>>> adding new
>>>>>>>>> >>>>> >>>>>>>> module. This is pretty rare occasion, I believe.
>>>>>>>>> >>>>> >>>>>>>>
>>>>>>>>> >>>>> >>>>>>>> WBR,
>>>>>>>>> >>>>> >>>>>>>> Pavel.
>>>>>>>>> >>>>> >>>>>>>>
>>>>>>>>> >>>>> >>>>>>>> On Thu, Dec 25, 2008 at 3:48 PM, Wenlong Li <
>>>>>>>>> wenlong@gmail.com>
>>>>>>>>> >>>>> wrote:
>>>>>>>>> >>>>> >>>>>>>>> Thx for your advice. Alexey.
>>>>>>>>> >>>>> >>>>>>>>>
>>>>>>>>> >>>>> >>>>>>>>> Here I have one question: do you know how the
>>>>>>>>> >>>>> bootclasspath.properties
>>>>>>>>> >>>>> >>>>>>>>> is maintained, manually or automatically?
>>>>>>>>> >>>>> >>>>>>>>>
>>>>>>>>> >>>>> >>>>>>>>> Another comment is I would like to treat the patch as DRLVM
>>>>>>>>> >>>>> specific
>>>>>>>>> >>>>> >>>>>>>>> optimization, e.g., it targets for improving VM creation
>>>>>>>>> time. So
>>>>>>>>> >>>>> that
>>>>>>>>> >>>>> >>>>>>>>> is possible to move all updates to DRLVM part to eliminate
>>>>>>>>> >>>>> potential
>>>>>>>>> >>>>> >>>>>>>>> modularity and compatibility problem.
>>>>>>>>> >>>>> >>>>>>>>>
>>>>>>>>> >>>>> >>>>>>>>> thx,
>>>>>>>>> >>>>> >>>>>>>>> Wenlong
>>>>>>>>> >>>>> >>>>>>>>>
>>>>>>>>> >>>>> >>>>>>>>> On Thu, Dec 25, 2008 at 5:32 PM, Aleksey Shipilev
>>>>>>>>> >>>>> >>>>>>>>> <al...@gmail.com> wrote:
>>>>>>>>> >>>>> >>>>>>>>>> Hi, Wenlong.
>>>>>>>>> >>>>> >>>>>>>>>>
>>>>>>>>> >>>>> >>>>>>>>>> On Thu, Dec 25, 2008 at 11:49 AM, Wenlong Li <
>>>>>>>>> wenlong@gmail.com>
>>>>>>>>> >>>>> wrote:
>>>>>>>>> >>>>> >>>>>>>>>>> btw, Alexey, Let's go back to discuss whether there is a
>>>>>>>>> need
>>>>>>>>> >>>>> to
>>>>>>>>> >>>>> >>>>>>>>>>> include this feature in Harmony, given 17% performance
>>>>>>>>> gain in
>>>>>>>>> >>>>> Linux
>>>>>>>>> >>>>> >>>>>>>>>>> when using your methodology. For windows test, I will
>>>>>>>>> double
>>>>>>>>> >>>>> check the
>>>>>>>>> >>>>> >>>>>>>>>>> backgroud process as you pointed out.
>>>>>>>>> >>>>> >>>>>>>>>>
>>>>>>>>> >>>>> >>>>>>>>>> My opinion was already expressed after I had finished the
>>>>>>>>> tests
>>>>>>>>> >>>>> from
>>>>>>>>> >>>>> >>>>>>>>>> my side: the boost can be achieved in specific conditions,
>>>>>>>>> so
>>>>>>>>> >>>>> whether
>>>>>>>>> >>>>> >>>>>>>>>> it's worth including into Harmony really depends on how
>>>>>>>>> much
>>>>>>>>> >>>>> mess the
>>>>>>>>> >>>>> >>>>>>>>>> patch would introduce besides the "performance boost".
>>>>>>>>> From what
>>>>>>>>> >>>>> I
>>>>>>>>> >>>>> >>>>>>>>>> see, the patch obliges the maintainer to maintain the
>>>>>>>>> correct
>>>>>>>>> >>>>> mapping
>>>>>>>>> >>>>> >>>>>>>>>> between jars and Java packages. This new feature is also
>>>>>>>>> spread
>>>>>>>>> >>>>> >>>>>>>>>> between Classlib and VM, but it seems like DRLVM specific.
>>>>>>>>> In
>>>>>>>>> >>>>> this
>>>>>>>>> >>>>> >>>>>>>>>> case I would rather stay without the patch.
>>>>>>>>> >>>>> >>>>>>>>>>
>>>>>>>>> >>>>> >>>>>>>>>> Personally (if I'll be committer) I would accept the patch
>>>>>>>>> with
>>>>>>>>> >>>>> two
>>>>>>>>> >>>>> >>>>>>>>>> serious modifications:
>>>>>>>>> >>>>> >>>>>>>>>> 1. Stay within DRLVM, do not introduce this feature into
>>>>>>>>> >>>>> Classlib,
>>>>>>>>> >>>>> >>>>>>>>>> get the thing tested and evolved on DRLVM side. Otherwise
>>>>>>>>> it
>>>>>>>>> >>>>> might
>>>>>>>>> >>>>> >>>>>>>>>> break the compatibility with other VMs.
>>>>>>>>> >>>>> >>>>>>>>>> 2. Make the mapping generated automatically (during build
>>>>>>>>> >>>>> process?)
>>>>>>>>> >>>>> >>>>>>>>>> to free the burden for maintainers.
>>>>>>>>> >>>>> >>>>>>>>>>
>>>>>>>>> >>>>> >>>>>>>>>> Thanks,
>>>>>>>>> >>>>> >>>>>>>>>> Aleksey.
>>>>>>>>> >>>>> >>>>>>>>>>
>>>>>>>>> >>>>> >>>>>>>>>
>>>>>>>>> >>>>> >>>>>>>>
>>>>>>>>> >>>>> >>>>>>>
>>>>>>>>> >>>>> >>>>>>
>>>>>>>>> >>>>> >>>>>
>>>>>>>>> >>>>> >>>>
>>>>>>>>> >>>>> >>>
>>>>>>>>> >>>>> >>
>>>>>>>>> >>>>> >
>>>>>>>>> >>>>> >
>>>>>>>>> >>>>> >
>>>>>>>>> >>>>> > --
>>>>>>>>> >>>>> > С уважением,
>>>>>>>>> >>>>> > Алексей Федотов,
>>>>>>>>> >>>>> > ЗАО «Телеком Экспресс»
>>>>>>>>> >>>>> >
>>>>>>>>> >>>>>
>>>>>>>>> >>>>>
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> --
>>>>>>>>> >>>>> http://xiao-feng.blogspot.com
>>>>>>>>> >>>>>
>>>>>>>>> >>>>
>>>>>>>>> >>>>
>>>>>>>>> >>>>
>>>>>>>>> >>>> --
>>>>>>>>> >>>> С уважением,
>>>>>>>>> >>>> Алексей Федотов,
>>>>>>>>> >>>> ЗАО «Телеком Экспресс»
>>>>>>>>> >>>>
>>>>>>>>> >>>
>>>>>>>>> >>>
>>>>>>>>> >>>
>>>>>>>>> >>> --
>>>>>>>>> >>> http://xiao-feng.blogspot.com
>>>>>>>>> >>>
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >> --
>>>>>>>>> >> С уважением,
>>>>>>>>> >> Алексей Федотов,
>>>>>>>>> >> ЗАО «Телеком Экспресс»
>>>>>>>>> >>
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > --
>>>>>>>>> > http://xiao-feng.blogspot.com
>>>>>>>>> >
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> С уважением,
>>>>>>>> Алексей Федотов,
>>>>>>>> ЗАО «Телеком Экспресс»
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: [VM] On-demand class library parsing is ready to commit
Posted by Wenlong Li <we...@gmail.com>.
Btw, Aleksey, to avoid confusion, I expect this patch help VM creation
time or at least help the memory footprint since some modules are
never used after their loading and parsing. :)
On Wed, Jan 14, 2009 at 9:38 PM, Wenlong Li <we...@gmail.com> wrote:
> Alexei,
>
> Sorry for confusing. The patch for review is H6039.patch_2. Please
> kindly provide your comment.
>
> Aleksey,
>
> I have not measured the performance before completing the code review.
> I will do that later.
>
> thx, Wenlong
>
> On Wed, Jan 14, 2009 at 9:14 PM, Wenlong Li <we...@gmail.com> wrote:
>> Pavel,
>>
>> Pls see my comments in the JIRA.
>>
>> thx, Wenlong
>>
>> On Wed, Jan 14, 2009 at 8:44 PM, Pavel Pervov <pm...@gmail.com> wrote:
>>> Please, also, check that jar entry caches still work correctly after your patch.
>>>
>>> Pavel.
>>>
>>> On Wed, Jan 14, 2009 at 12:33 PM, Wenlong Li <we...@gmail.com> wrote:
>>>> All,
>>>>
>>>> I submit a new patch for on-demand class loading and parsing. All
>>>> codes are put in VM side, and the mapping info is automatically
>>>> produced.
>>>>
>>>> Pls see https://issues.apache.org/jira/browse/HARMONY-6039
>>>>
>>>> Comments are welcome.
>>>>
>>>> Thx, Wenlong
>>>> Managed Runtime Technology Center, Intel
>>>>
>>>> On Wed, Jan 7, 2009 at 12:08 PM, Wenlong Li <we...@gmail.com> wrote:
>>>>> All,
>>>>> At this moment, I move all updates in classlib to VM side such that
>>>>> there is no modularity issue. Next step is to produce the mapping
>>>>> between module and library efficiently and accurately.
>>>>>
>>>>> Comments are welcome.
>>>>>
>>>>> Thx, Wenlong
>>>>> Managed Runtime Technology Center, Intel
>>>>>
>>>>> On Tue, Jan 6, 2009 at 11:08 PM, Wenlong Li <we...@gmail.com> wrote:
>>>>>> Thx :)
>>>>>>
>>>>>> On Tue, Jan 6, 2009 at 10:35 PM, Alexei Fedotov
>>>>>> <al...@gmail.com> wrote:
>>>>>>> Sure.
>>>>>>>
>>>>>>> 1. If you dig into SetClasspathFromString, you will see that it starts from
>>>>>>> splitting the given classpath into pieces. You already know the new piece
>>>>>>> you add and may skip splitting step.
>>>>>>>
>>>>>>> 2. If I understand you code correctly, the case "pdest >
>>>>>>> (*it).second->bytes" might be a subject of a negative assertion. Adding this
>>>>>>> assrtion would speed up bug discovery.
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Jan 6, 2009 at 5:09 AM, Wenlong Li <we...@gmail.com> wrote:
>>>>>>>
>>>>>>>> Yes, Xiao-Feng's understanding is correct. The patch loads and parses
>>>>>>>> modules on demand. If no class in swing.jar is not requested, then
>>>>>>>> this module will not be loaded.
>>>>>>>>
>>>>>>>> btw, Alexei, you said "SetClasspathFromString" and "pdest >
>>>>>>>> (*it).second->bytes" are not efficient. Can you share more comments on
>>>>>>>> them? I just reused some code in Harmony, and didn't optimize them
>>>>>>>> further.
>>>>>>>>
>>>>>>>> Thx, Wenlong
>>>>>>>> Managed Runtime Technology Center, Intel
>>>>>>>>
>>>>>>>> On Fri, Dec 26, 2008 at 5:16 PM, Xiao-Feng Li <xi...@gmail.com>
>>>>>>>> wrote:
>>>>>>>> > On Fri, Dec 26, 2008 at 5:08 PM, Alexei Fedotov
>>>>>>>> > <al...@gmail.com> wrote:
>>>>>>>> >> Xiao Feng,
>>>>>>>> >> Thank you for explaining.
>>>>>>>> >>
>>>>>>>> >> I get more minor comments on more commented code, ineffective
>>>>>>>> >> SetClasspathFromString usage, non-covered unexpected case when pdest >
>>>>>>>> >> (*it).second->bytes. One major comment on crossing vm module boundary
>>>>>>>> >> still remains. But now I'm happy with the design.
>>>>>>>> >
>>>>>>>> > Alexei, yes, I agree with your comments. These parts should be
>>>>>>>> > improved. (Still, this is my personal opinion. :) Let's wait Wenlong
>>>>>>>> > speaking.)
>>>>>>>> >
>>>>>>>> > Thanks,
>>>>>>>> > xiaofeng
>>>>>>>> >
>>>>>>>> >> Sorry for being slow.
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >> On Fri, Dec 26, 2008 at 11:40 AM, Xiao-Feng Li <xi...@gmail.com>
>>>>>>>> wrote:
>>>>>>>> >>> On Fri, Dec 26, 2008 at 4:03 PM, Alexei Fedotov
>>>>>>>> >>> <al...@gmail.com> wrote:
>>>>>>>> >>>> Xiao-Feng,
>>>>>>>> >>>> Continuing with the server example could you please give me a hint
>>>>>>>> where
>>>>>>>> >>>> decision to load swing.jar or not is taken in the patch? My initial
>>>>>>>> >>>> perception was that the list of what to load was hardcoded and was not
>>>>>>>> >>>> constructed dynamically depending on application.
>>>>>>>> >>>
>>>>>>>> >>> Alexei, here is the patch code I found:
>>>>>>>> >>>
>>>>>>>> >>> line 245:
>>>>>>>> >>> + // Find which jar exports this package
>>>>>>>> >>> + if (pkgName != NULL) {
>>>>>>>> >>> + char *boot_class_path =
>>>>>>>> >>> env->JavaProperties()->get(VM_BOOT_CLASS_DIR);
>>>>>>>> >>> + char *pendingClassPath = NULL;
>>>>>>>> >>> + apr_pool_t *tmp_pool;
>>>>>>>> >>> + apr_pool_create(&tmp_pool, NULL);
>>>>>>>> >>> + while (it != env->pending_jar_set.end()) {
>>>>>>>> >>> + pdest = strstr( (*it).second->bytes, pkgName );
>>>>>>>> >>> + if (pdest != NULL) {
>>>>>>>> >>> + pendingClassPath =
>>>>>>>> >>> (char*)STD_MALLOC(strlen(boot_class_path)
>>>>>>>> >>> + +
>>>>>>>> strlen((*it).first->bytes) + 1);
>>>>>>>> >>> + strcpy(pendingClassPath, boot_class_path);
>>>>>>>> >>> + strcat(pendingClassPath, (*it).first->bytes);
>>>>>>>> >>> + // Open this found jar, and read all classes
>>>>>>>> >>> contained in this jar
>>>>>>>> >>> + SetClasspathFromString(pendingClassPath,
>>>>>>>> tmp_pool);
>>>>>>>> >>> + // Erase the found jar from pending jar list
>>>>>>>> >>> as it has been parsed
>>>>>>>> >>> + env->pending_jar_set.erase(it++);
>>>>>>>> >>> + STD_FREE(pendingClassPath);
>>>>>>>> >>> + } else {
>>>>>>>> >>>
>>>>>>>> >>> It checks if a JAR has the requested package, then loads it if yes. I
>>>>>>>> >>> am not sure if this is what you were asking. (Btw, this is only my
>>>>>>>> >>> understanding of his patch. I am not speaking for Wenlong.)
>>>>>>>> >>>
>>>>>>>> >>> Thanks,
>>>>>>>> >>> xiaofeng
>>>>>>>> >>>
>>>>>>>> >>>> Thanks.
>>>>>>>> >>>>
>>>>>>>> >>>>
>>>>>>>> >>>> On Fri, Dec 26, 2008 at 4:14 AM, Xiao-Feng Li <xi...@gmail.com>
>>>>>>>> wrote:
>>>>>>>> >>>>
>>>>>>>> >>>>> On Fri, Dec 26, 2008 at 12:49 AM, Alexei Fedotov
>>>>>>>> >>>>> <al...@gmail.com> wrote:
>>>>>>>> >>>>> > Aleksey,
>>>>>>>> >>>>> > I like your conclusion.
>>>>>>>> >>>>> >
>>>>>>>> >>>>> > Wenlong,
>>>>>>>> >>>>> > I'm trying to understand the real life value of the "abstract"
>>>>>>>> startup
>>>>>>>> >>>>> > time metric you've suggested. Does Harmony with your patch load
>>>>>>>> >>>>> > swing.jar for a server application? Do I understand that loading
>>>>>>>> >>>>> > happens, though it happens later compared to VM without your patch?
>>>>>>>> I
>>>>>>>> >>>>> > believe that the proper design of delayed loading should answer
>>>>>>>> "no"
>>>>>>>> >>>>> > to this question.
>>>>>>>> >>>>>
>>>>>>>> >>>>> I checked the patch, and I found the answer is indeed "No" as you
>>>>>>>> expected.
>>>>>>>> >>>>>
>>>>>>>> >>>>> Thanks,
>>>>>>>> >>>>> xiaofeng
>>>>>>>> >>>>>
>>>>>>>> >>>>> > In other words, I appreciate if you describe which real use cases
>>>>>>>> are
>>>>>>>> >>>>> > improved by this patch.
>>>>>>>> >>>>> > Thanks!
>>>>>>>> >>>>>
>>>>>>>> >>>>> > On Thu, Dec 25, 2008 at 7:29 PM, Aleksey Shipilev
>>>>>>>> >>>>> > <al...@gmail.com> wrote:
>>>>>>>> >>>>> >> Ok, here's the catch.
>>>>>>>> >>>>> >>
>>>>>>>> >>>>> >> bootclasspath.properties is the SortedSet<JARfile>, which
>>>>>>>> enumerates
>>>>>>>> >>>>> >> the JARs available for bootclassloader. The set of such the JARs
>>>>>>>> is
>>>>>>>> >>>>> >> really stable because modular decomposition of classlib is stable.
>>>>>>>> >>>>> >> That's why nobody bothers with automatic generation of it: it only
>>>>>>>> >>>>> >> should be updated when new JAR file arrives.
>>>>>>>> >>>>> >> Modulelibrarymapping.properties is different on this point, it's
>>>>>>>> the
>>>>>>>> >>>>> >> Map<PackageName,JARfile>, which should be updated each time the
>>>>>>>> new
>>>>>>>> >>>>> >> *package* is introduced. I'm not talking about java.* packages
>>>>>>>> >>>>> >> (they're standardized), rather about org.apache.harmony.*.
>>>>>>>> >>>>> >>
>>>>>>>> >>>>> >> Automatic generation of this property file gives two advantages:
>>>>>>>> >>>>> >> 1. Error-prone. Prevent yourself from hand-messing with mapping
>>>>>>>> and
>>>>>>>> >>>>> >> getting spurious ClassNotFoundException. BTW, what's the behaviour
>>>>>>>> in
>>>>>>>> >>>>> >> case the mapping is wrong?
>>>>>>>> >>>>> >> 2. "Researchful". There're lot of guys around who enjoys the
>>>>>>>> >>>>> >> modularity of Harmony classlib and eventually they might want to
>>>>>>>> split
>>>>>>>> >>>>> >> the packages even deeper, into smaller pieces. Then automatic
>>>>>>>> >>>>> >> generation would enable them to quickly roll-in and experiment
>>>>>>>> with
>>>>>>>> >>>>> >> different package layouts and their impact on performance. They
>>>>>>>> could
>>>>>>>> >>>>> >> use ordinary bootclasspath.properties, but your feature wouldn't
>>>>>>>> be
>>>>>>>> >>>>> >> used by them then ;)
>>>>>>>> >>>>> >>
>>>>>>>> >>>>> >> That's merely a housekeeping procedure. I believe that anything
>>>>>>>> which
>>>>>>>> >>>>> >> could be done more than once, eventually would be done more than
>>>>>>>> once.
>>>>>>>> >>>>> >> Hence it should be automated. You say that the file was generated
>>>>>>>> from
>>>>>>>> >>>>> >> manifests of JARs, so is it hard to just tie the same tool into
>>>>>>>> DRLVM
>>>>>>>> >>>>> >> build process?
>>>>>>>> >>>>> >>
>>>>>>>> >>>>> >> As for DRLVM-specific, my opinion that this is because the patch:
>>>>>>>> >>>>> >> a. breaks the compatibility of classlib (you change
>>>>>>>> >>>>> >> bootclasspath.properties, right?) with other VMs.
>>>>>>>> >>>>> >> b. treated in DRLVM classloader only.
>>>>>>>> >>>>> >>
>>>>>>>> >>>>> >> Of course eventually this feature might be used by others, but IMO
>>>>>>>> we
>>>>>>>> >>>>> >> should be careful about other guys who use the same classlib. I'd
>>>>>>>> >>>>> >> rather wait for some incubation on DRLVM side first.
>>>>>>>> >>>>> >>
>>>>>>>> >>>>> >> Thanks,
>>>>>>>> >>>>> >> Aleksey.
>>>>>>>> >>>>> >>
>>>>>>>> >>>>> >> On Thu, Dec 25, 2008 at 6:18 PM, Wenlong Li <we...@gmail.com>
>>>>>>>> wrote:
>>>>>>>> >>>>> >>> I see. In fact, my file doesn't need track change at the class
>>>>>>>> >>>>> >>> granularity. Instead, it only needs know package info provided in
>>>>>>>> the
>>>>>>>> >>>>> >>> manifest file. When class is added to a library, do we need
>>>>>>>> change
>>>>>>>> >>>>> >>> the manifest as well?
>>>>>>>> >>>>> >>>
>>>>>>>> >>>>> >>> btw, I guess there is a mis-understanding: my
>>>>>>>> modulelibrarymapping
>>>>>>>> >>>>> >>> file only records package info provided by manfiest in each
>>>>>>>> module. It
>>>>>>>> >>>>> >>> doesn't relate to each class.
>>>>>>>> >>>>> >>>
>>>>>>>> >>>>> >>> thx,
>>>>>>>> >>>>> >>> Wenlong
>>>>>>>> >>>>> >>>
>>>>>>>> >>>>> >>> On Thu, Dec 25, 2008 at 10:55 PM, Pavel Pervov <
>>>>>>>> pmcfirst@gmail.com>
>>>>>>>> >>>>> wrote:
>>>>>>>> >>>>> >>>> Classes are added to class library from time to time. I'm not
>>>>>>>> sure how
>>>>>>>> >>>>> >>>> it can be possible to track these changes manually.
>>>>>>>> >>>>> >>>>
>>>>>>>> >>>>> >>>> WBR,
>>>>>>>> >>>>> >>>> Pavel.
>>>>>>>> >>>>> >>>>
>>>>>>>> >>>>> >>>>
>>>>>>>> >>>>> >>>> On Thu, Dec 25, 2008 at 5:09 PM, Wenlong Li <we...@gmail.com>
>>>>>>>> >>>>> wrote:
>>>>>>>> >>>>> >>>>> Sorry, one more question: bootclasspath.properties is classlib
>>>>>>>> >>>>> >>>>> specific file, why we could not make a vm specific file
>>>>>>>> manually?
>>>>>>>> >>>>> Just
>>>>>>>> >>>>> >>>>> curious to know the possible reason. :)
>>>>>>>> >>>>> >>>>>
>>>>>>>> >>>>> >>>>> thx,
>>>>>>>> >>>>> >>>>> Wenlong
>>>>>>>> >>>>> >>>>>
>>>>>>>> >>>>> >>>>> On Thu, Dec 25, 2008 at 10:00 PM, Pavel Pervov <
>>>>>>>> pmcfirst@gmail.com>
>>>>>>>> >>>>> wrote:
>>>>>>>> >>>>> >>>>>> If this would be VM-side automatically produced configuration
>>>>>>>> >>>>> file...
>>>>>>>> >>>>> >>>>>>
>>>>>>>> >>>>> >>>>>> WBR,
>>>>>>>> >>>>> >>>>>> Pavel.
>>>>>>>> >>>>> >>>>>>
>>>>>>>> >>>>> >>>>>> On Thu, Dec 25, 2008 at 4:58 PM, Wenlong Li <
>>>>>>>> wenlong@gmail.com>
>>>>>>>> >>>>> wrote:
>>>>>>>> >>>>> >>>>>>> btw, because adding new module is rare case, manually
>>>>>>>> modifying the
>>>>>>>> >>>>> >>>>>>> bootclasspath.properties is not an issue?
>>>>>>>> >>>>> >>>>>>>
>>>>>>>> >>>>> >>>>>>> If so, can I conclude adding another property file with same
>>>>>>>> update
>>>>>>>> >>>>> >>>>>>> frequency as bootclasspath would be fine as well?
>>>>>>>> >>>>> >>>>>>>
>>>>>>>> >>>>> >>>>>>> Pls kindly correct me if my understanding is wrong.
>>>>>>>> >>>>> >>>>>>>
>>>>>>>> >>>>> >>>>>>> On Thu, Dec 25, 2008 at 9:05 PM, Pavel Pervov <
>>>>>>>> pmcfirst@gmail.com>
>>>>>>>> >>>>> wrote:
>>>>>>>> >>>>> >>>>>>>> Wenlong,
>>>>>>>> >>>>> >>>>>>>>
>>>>>>>> >>>>> >>>>>>>> Note, that bootclasspath.properties is only changed on
>>>>>>>> adding new
>>>>>>>> >>>>> >>>>>>>> module. This is pretty rare occasion, I believe.
>>>>>>>> >>>>> >>>>>>>>
>>>>>>>> >>>>> >>>>>>>> WBR,
>>>>>>>> >>>>> >>>>>>>> Pavel.
>>>>>>>> >>>>> >>>>>>>>
>>>>>>>> >>>>> >>>>>>>> On Thu, Dec 25, 2008 at 3:48 PM, Wenlong Li <
>>>>>>>> wenlong@gmail.com>
>>>>>>>> >>>>> wrote:
>>>>>>>> >>>>> >>>>>>>>> Thx for your advice. Alexey.
>>>>>>>> >>>>> >>>>>>>>>
>>>>>>>> >>>>> >>>>>>>>> Here I have one question: do you know how the
>>>>>>>> >>>>> bootclasspath.properties
>>>>>>>> >>>>> >>>>>>>>> is maintained, manually or automatically?
>>>>>>>> >>>>> >>>>>>>>>
>>>>>>>> >>>>> >>>>>>>>> Another comment is I would like to treat the patch as DRLVM
>>>>>>>> >>>>> specific
>>>>>>>> >>>>> >>>>>>>>> optimization, e.g., it targets for improving VM creation
>>>>>>>> time. So
>>>>>>>> >>>>> that
>>>>>>>> >>>>> >>>>>>>>> is possible to move all updates to DRLVM part to eliminate
>>>>>>>> >>>>> potential
>>>>>>>> >>>>> >>>>>>>>> modularity and compatibility problem.
>>>>>>>> >>>>> >>>>>>>>>
>>>>>>>> >>>>> >>>>>>>>> thx,
>>>>>>>> >>>>> >>>>>>>>> Wenlong
>>>>>>>> >>>>> >>>>>>>>>
>>>>>>>> >>>>> >>>>>>>>> On Thu, Dec 25, 2008 at 5:32 PM, Aleksey Shipilev
>>>>>>>> >>>>> >>>>>>>>> <al...@gmail.com> wrote:
>>>>>>>> >>>>> >>>>>>>>>> Hi, Wenlong.
>>>>>>>> >>>>> >>>>>>>>>>
>>>>>>>> >>>>> >>>>>>>>>> On Thu, Dec 25, 2008 at 11:49 AM, Wenlong Li <
>>>>>>>> wenlong@gmail.com>
>>>>>>>> >>>>> wrote:
>>>>>>>> >>>>> >>>>>>>>>>> btw, Alexey, Let's go back to discuss whether there is a
>>>>>>>> need
>>>>>>>> >>>>> to
>>>>>>>> >>>>> >>>>>>>>>>> include this feature in Harmony, given 17% performance
>>>>>>>> gain in
>>>>>>>> >>>>> Linux
>>>>>>>> >>>>> >>>>>>>>>>> when using your methodology. For windows test, I will
>>>>>>>> double
>>>>>>>> >>>>> check the
>>>>>>>> >>>>> >>>>>>>>>>> backgroud process as you pointed out.
>>>>>>>> >>>>> >>>>>>>>>>
>>>>>>>> >>>>> >>>>>>>>>> My opinion was already expressed after I had finished the
>>>>>>>> tests
>>>>>>>> >>>>> from
>>>>>>>> >>>>> >>>>>>>>>> my side: the boost can be achieved in specific conditions,
>>>>>>>> so
>>>>>>>> >>>>> whether
>>>>>>>> >>>>> >>>>>>>>>> it's worth including into Harmony really depends on how
>>>>>>>> much
>>>>>>>> >>>>> mess the
>>>>>>>> >>>>> >>>>>>>>>> patch would introduce besides the "performance boost".
>>>>>>>> From what
>>>>>>>> >>>>> I
>>>>>>>> >>>>> >>>>>>>>>> see, the patch obliges the maintainer to maintain the
>>>>>>>> correct
>>>>>>>> >>>>> mapping
>>>>>>>> >>>>> >>>>>>>>>> between jars and Java packages. This new feature is also
>>>>>>>> spread
>>>>>>>> >>>>> >>>>>>>>>> between Classlib and VM, but it seems like DRLVM specific.
>>>>>>>> In
>>>>>>>> >>>>> this
>>>>>>>> >>>>> >>>>>>>>>> case I would rather stay without the patch.
>>>>>>>> >>>>> >>>>>>>>>>
>>>>>>>> >>>>> >>>>>>>>>> Personally (if I'll be committer) I would accept the patch
>>>>>>>> with
>>>>>>>> >>>>> two
>>>>>>>> >>>>> >>>>>>>>>> serious modifications:
>>>>>>>> >>>>> >>>>>>>>>> 1. Stay within DRLVM, do not introduce this feature into
>>>>>>>> >>>>> Classlib,
>>>>>>>> >>>>> >>>>>>>>>> get the thing tested and evolved on DRLVM side. Otherwise
>>>>>>>> it
>>>>>>>> >>>>> might
>>>>>>>> >>>>> >>>>>>>>>> break the compatibility with other VMs.
>>>>>>>> >>>>> >>>>>>>>>> 2. Make the mapping generated automatically (during build
>>>>>>>> >>>>> process?)
>>>>>>>> >>>>> >>>>>>>>>> to free the burden for maintainers.
>>>>>>>> >>>>> >>>>>>>>>>
>>>>>>>> >>>>> >>>>>>>>>> Thanks,
>>>>>>>> >>>>> >>>>>>>>>> Aleksey.
>>>>>>>> >>>>> >>>>>>>>>>
>>>>>>>> >>>>> >>>>>>>>>
>>>>>>>> >>>>> >>>>>>>>
>>>>>>>> >>>>> >>>>>>>
>>>>>>>> >>>>> >>>>>>
>>>>>>>> >>>>> >>>>>
>>>>>>>> >>>>> >>>>
>>>>>>>> >>>>> >>>
>>>>>>>> >>>>> >>
>>>>>>>> >>>>> >
>>>>>>>> >>>>> >
>>>>>>>> >>>>> >
>>>>>>>> >>>>> > --
>>>>>>>> >>>>> > С уважением,
>>>>>>>> >>>>> > Алексей Федотов,
>>>>>>>> >>>>> > ЗАО «Телеком Экспресс»
>>>>>>>> >>>>> >
>>>>>>>> >>>>>
>>>>>>>> >>>>>
>>>>>>>> >>>>>
>>>>>>>> >>>>> --
>>>>>>>> >>>>> http://xiao-feng.blogspot.com
>>>>>>>> >>>>>
>>>>>>>> >>>>
>>>>>>>> >>>>
>>>>>>>> >>>>
>>>>>>>> >>>> --
>>>>>>>> >>>> С уважением,
>>>>>>>> >>>> Алексей Федотов,
>>>>>>>> >>>> ЗАО «Телеком Экспресс»
>>>>>>>> >>>>
>>>>>>>> >>>
>>>>>>>> >>>
>>>>>>>> >>>
>>>>>>>> >>> --
>>>>>>>> >>> http://xiao-feng.blogspot.com
>>>>>>>> >>>
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >> --
>>>>>>>> >> С уважением,
>>>>>>>> >> Алексей Федотов,
>>>>>>>> >> ЗАО «Телеком Экспресс»
>>>>>>>> >>
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > --
>>>>>>>> > http://xiao-feng.blogspot.com
>>>>>>>> >
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> С уважением,
>>>>>>> Алексей Федотов,
>>>>>>> ЗАО «Телеком Экспресс»
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: [VM] On-demand class library parsing is ready to commit
Posted by Alexei Fedotov <al...@gmail.com>.
Here review continues and completes,
https://issues.apache.org/jira/browse/HARMONY-6039?focusedCommentId=12664090#action_12664090
Overall, review comments about restructuring the code, avoiding
cut&paste and separating new changes are also pleas of making the
review of these parts more productive.
Wenlong, thanks for the patch!
On Thu, Jan 15, 2009 at 11:49 AM, Alexei Fedotov
<al...@gmail.com> wrote:
> Hello Wenlong,
>
> The intention of the following review is to improve the readability of
> the code. Please find my comments preceded with patch line numbers and
> fix anything you find worthy to fix.
>
> 9
> excessive comment length
>
> 9-
> missed description of parameters (e.g. @param mapFile <description>)
> and return value
> do we need to pass mapFile through the parameter chain? may it be an element of
>
> 22, 24
> we don't need both versions of each function, do we?
> using one version (esp. of SetBCPElement) would make the whole code size smaller
>
> it would be easier for me to review your deltas of functions if you
> don't make the full copies of them
>
> 37
> seems that environment.h has c/apr style set of includes
> can we hide <map> and related typedef in sources to maintain C/apr
> style of interfaces
> is it possible to use more specific header (e.g. related to jar
> parsing) than environment.h for JarFilePackageMapping definition?
>
> 93
> *the* bootstrap classpath
>
> 96-
> the proper bracket style is specified here
> https://issues.apache.org/jira/secure/attachment/12353745/format.sh
> [well, the whole file is formatted strangely - Pavel, could you comment?]
>
> 97
> such -> the
>
> 150
> *a* temp pool
>
> 154-
> putting map file operations into separate .cpp file with a clear and
> readable interface function names in the corresponding .h interface
> would not make existing code less readable
>
> that .h file would be a proper place for new types you introduce, not
> environment.h
>
> you may also use the proper Apache formatting in the new file
>
> 190-
> cannot understand where the signature file comes from - I cannot find
> apr_file_write for it
> the explanatory comment is welcome
>
> if both mapping and signature files are things introduced by this
> patch why don't we use one file instead of two
>
> 200
> can this be replaced with assert(luni_path)?
>
> 213
> +1 to Aleksey's comment on literals
>
> [have to go, will continue later]
>
> 434
> commented code
>
> Thanks.
>
>
> On Wed, Jan 14, 2009 at 4:38 PM, Wenlong Li <we...@gmail.com> wrote:
>>
>> Alexei,
>>
>> Sorry for confusing. The patch for review is H6039.patch_2. Please
>> kindly provide your comment.
>>
>> Aleksey,
>>
>> I have not measured the performance before completing the code review.
>> I will do that later.
>>
>> thx, Wenlong
>>
>> On Wed, Jan 14, 2009 at 9:14 PM, Wenlong Li <we...@gmail.com> wrote:
>> > Pavel,
>> >
>> > Pls see my comments in the JIRA.
>> >
>> > thx, Wenlong
>> >
>> > On Wed, Jan 14, 2009 at 8:44 PM, Pavel Pervov <pm...@gmail.com> wrote:
>> >> Please, also, check that jar entry caches still work correctly after your patch.
>> >>
>> >> Pavel.
>> >>
>> >> On Wed, Jan 14, 2009 at 12:33 PM, Wenlong Li <we...@gmail.com> wrote:
>> >>> All,
>> >>>
>> >>> I submit a new patch for on-demand class loading and parsing. All
>> >>> codes are put in VM side, and the mapping info is automatically
>> >>> produced.
>> >>>
>> >>> Pls see https://issues.apache.org/jira/browse/HARMONY-6039
>> >>>
>> >>> Comments are welcome.
>> >>>
>> >>> Thx, Wenlong
>> >>> Managed Runtime Technology Center, Intel
>> >>>
>> >>> On Wed, Jan 7, 2009 at 12:08 PM, Wenlong Li <we...@gmail.com> wrote:
>> >>>> All,
>> >>>> At this moment, I move all updates in classlib to VM side such that
>> >>>> there is no modularity issue. Next step is to produce the mapping
>> >>>> between module and library efficiently and accurately.
>> >>>>
>> >>>> Comments are welcome.
>> >>>>
>> >>>> Thx, Wenlong
>> >>>> Managed Runtime Technology Center, Intel
>> >>>>
>> >>>> On Tue, Jan 6, 2009 at 11:08 PM, Wenlong Li <we...@gmail.com> wrote:
>> >>>>> Thx :)
>> >>>>>
>> >>>>> On Tue, Jan 6, 2009 at 10:35 PM, Alexei Fedotov
>> >>>>> <al...@gmail.com> wrote:
>> >>>>>> Sure.
>> >>>>>>
>> >>>>>> 1. If you dig into SetClasspathFromString, you will see that it starts from
>> >>>>>> splitting the given classpath into pieces. You already know the new piece
>> >>>>>> you add and may skip splitting step.
>> >>>>>>
>> >>>>>> 2. If I understand you code correctly, the case "pdest >
>> >>>>>> (*it).second->bytes" might be a subject of a negative assertion. Adding this
>> >>>>>> assrtion would speed up bug discovery.
>> >>>>>>
>> >>>>>> Thanks.
>> >>>>>>
>> >>>>>>
>> >>>>>> On Tue, Jan 6, 2009 at 5:09 AM, Wenlong Li <we...@gmail.com> wrote:
>> >>>>>>
>> >>>>>>> Yes, Xiao-Feng's understanding is correct. The patch loads and parses
>> >>>>>>> modules on demand. If no class in swing.jar is not requested, then
>> >>>>>>> this module will not be loaded.
>> >>>>>>>
>> >>>>>>> btw, Alexei, you said "SetClasspathFromString" and "pdest >
>> >>>>>>> (*it).second->bytes" are not efficient. Can you share more comments on
>> >>>>>>> them? I just reused some code in Harmony, and didn't optimize them
>> >>>>>>> further.
>> >>>>>>>
>> >>>>>>> Thx, Wenlong
>> >>>>>>> Managed Runtime Technology Center, Intel
>> >>>>>>>
>> >>>>>>> On Fri, Dec 26, 2008 at 5:16 PM, Xiao-Feng Li <xi...@gmail.com>
>> >>>>>>> wrote:
>> >>>>>>> > On Fri, Dec 26, 2008 at 5:08 PM, Alexei Fedotov
>> >>>>>>> > <al...@gmail.com> wrote:
>> >>>>>>> >> Xiao Feng,
>> >>>>>>> >> Thank you for explaining.
>> >>>>>>> >>
>> >>>>>>> >> I get more minor comments on more commented code, ineffective
>> >>>>>>> >> SetClasspathFromString usage, non-covered unexpected case when pdest >
>> >>>>>>> >> (*it).second->bytes. One major comment on crossing vm module boundary
>> >>>>>>> >> still remains. But now I'm happy with the design.
>> >>>>>>> >
>> >>>>>>> > Alexei, yes, I agree with your comments. These parts should be
>> >>>>>>> > improved. (Still, this is my personal opinion. :) Let's wait Wenlong
>> >>>>>>> > speaking.)
>> >>>>>>> >
>> >>>>>>> > Thanks,
>> >>>>>>> > xiaofeng
>> >>>>>>> >
>> >>>>>>> >> Sorry for being slow.
>> >>>>>>> >>
>> >>>>>>> >>
>> >>>>>>> >>
>> >>>>>>> >> On Fri, Dec 26, 2008 at 11:40 AM, Xiao-Feng Li <xi...@gmail.com>
>> >>>>>>> wrote:
>> >>>>>>> >>> On Fri, Dec 26, 2008 at 4:03 PM, Alexei Fedotov
>> >>>>>>> >>> <al...@gmail.com> wrote:
>> >>>>>>> >>>> Xiao-Feng,
>> >>>>>>> >>>> Continuing with the server example could you please give me a hint
>> >>>>>>> where
>> >>>>>>> >>>> decision to load swing.jar or not is taken in the patch? My initial
>> >>>>>>> >>>> perception was that the list of what to load was hardcoded and was not
>> >>>>>>> >>>> constructed dynamically depending on application.
>> >>>>>>> >>>
>> >>>>>>> >>> Alexei, here is the patch code I found:
>> >>>>>>> >>>
>> >>>>>>> >>> line 245:
>> >>>>>>> >>> + // Find which jar exports this package
>> >>>>>>> >>> + if (pkgName != NULL) {
>> >>>>>>> >>> + char *boot_class_path =
>> >>>>>>> >>> env->JavaProperties()->get(VM_BOOT_CLASS_DIR);
>> >>>>>>> >>> + char *pendingClassPath = NULL;
>> >>>>>>> >>> + apr_pool_t *tmp_pool;
>> >>>>>>> >>> + apr_pool_create(&tmp_pool, NULL);
>> >>>>>>> >>> + while (it != env->pending_jar_set.end()) {
>> >>>>>>> >>> + pdest = strstr( (*it).second->bytes, pkgName );
>> >>>>>>> >>> + if (pdest != NULL) {
>> >>>>>>> >>> + pendingClassPath =
>> >>>>>>> >>> (char*)STD_MALLOC(strlen(boot_class_path)
>> >>>>>>> >>> + +
>> >>>>>>> strlen((*it).first->bytes) + 1);
>> >>>>>>> >>> + strcpy(pendingClassPath, boot_class_path);
>> >>>>>>> >>> + strcat(pendingClassPath, (*it).first->bytes);
>> >>>>>>> >>> + // Open this found jar, and read all classes
>> >>>>>>> >>> contained in this jar
>> >>>>>>> >>> + SetClasspathFromString(pendingClassPath,
>> >>>>>>> tmp_pool);
>> >>>>>>> >>> + // Erase the found jar from pending jar list
>> >>>>>>> >>> as it has been parsed
>> >>>>>>> >>> + env->pending_jar_set.erase(it++);
>> >>>>>>> >>> + STD_FREE(pendingClassPath);
>> >>>>>>> >>> + } else {
>> >>>>>>> >>>
>> >>>>>>> >>> It checks if a JAR has the requested package, then loads it if yes. I
>> >>>>>>> >>> am not sure if this is what you were asking. (Btw, this is only my
>> >>>>>>> >>> understanding of his patch. I am not speaking for Wenlong.)
>> >>>>>>> >>>
>> >>>>>>> >>> Thanks,
>> >>>>>>> >>> xiaofeng
>> >>>>>>> >>>
>> >>>>>>> >>>> Thanks.
>> >>>>>>> >>>>
>> >>>>>>> >>>>
>> >>>>>>> >>>> On Fri, Dec 26, 2008 at 4:14 AM, Xiao-Feng Li <xi...@gmail.com>
>> >>>>>>> wrote:
>> >>>>>>> >>>>
>> >>>>>>> >>>>> On Fri, Dec 26, 2008 at 12:49 AM, Alexei Fedotov
>> >>>>>>> >>>>> <al...@gmail.com> wrote:
>> >>>>>>> >>>>> > Aleksey,
>> >>>>>>> >>>>> > I like your conclusion.
>> >>>>>>> >>>>> >
>> >>>>>>> >>>>> > Wenlong,
>> >>>>>>> >>>>> > I'm trying to understand the real life value of the "abstract"
>> >>>>>>> startup
>> >>>>>>> >>>>> > time metric you've suggested. Does Harmony with your patch load
>> >>>>>>> >>>>> > swing.jar for a server application? Do I understand that loading
>> >>>>>>> >>>>> > happens, though it happens later compared to VM without your patch?
>> >>>>>>> I
>> >>>>>>> >>>>> > believe that the proper design of delayed loading should answer
>> >>>>>>> "no"
>> >>>>>>> >>>>> > to this question.
>> >>>>>>> >>>>>
>> >>>>>>> >>>>> I checked the patch, and I found the answer is indeed "No" as you
>> >>>>>>> expected.
>> >>>>>>> >>>>>
>> >>>>>>> >>>>> Thanks,
>> >>>>>>> >>>>> xiaofeng
>> >>>>>>> >>>>>
>> >>>>>>> >>>>> > In other words, I appreciate if you describe which real use cases
>> >>>>>>> are
>> >>>>>>> >>>>> > improved by this patch.
>> >>>>>>> >>>>> > Thanks!
>> >>>>>>> >>>>>
>> >>>>>>> >>>>> > On Thu, Dec 25, 2008 at 7:29 PM, Aleksey Shipilev
>> >>>>>>> >>>>> > <al...@gmail.com> wrote:
>> >>>>>>> >>>>> >> Ok, here's the catch.
>> >>>>>>> >>>>> >>
>> >>>>>>> >>>>> >> bootclasspath.properties is the SortedSet<JARfile>, which
>> >>>>>>> enumerates
>> >>>>>>> >>>>> >> the JARs available for bootclassloader. The set of such the JARs
>> >>>>>>> is
>> >>>>>>> >>>>> >> really stable because modular decomposition of classlib is stable.
>> >>>>>>> >>>>> >> That's why nobody bothers with automatic generation of it: it only
>> >>>>>>> >>>>> >> should be updated when new JAR file arrives.
>> >>>>>>> >>>>> >> Modulelibrarymapping.properties is different on this point, it's
>> >>>>>>> the
>> >>>>>>> >>>>> >> Map<PackageName,JARfile>, which should be updated each time the
>> >>>>>>> new
>> >>>>>>> >>>>> >> *package* is introduced. I'm not talking about java.* packages
>> >>>>>>> >>>>> >> (they're standardized), rather about org.apache.harmony.*.
>> >>>>>>> >>>>> >>
>> >>>>>>> >>>>> >> Automatic generation of this property file gives two advantages:
>> >>>>>>> >>>>> >> 1. Error-prone. Prevent yourself from hand-messing with mapping
>> >>>>>>> and
>> >>>>>>> >>>>> >> getting spurious ClassNotFoundException. BTW, what's the behaviour
>> >>>>>>> in
>> >>>>>>> >>>>> >> case the mapping is wrong?
>> >>>>>>> >>>>> >> 2. "Researchful". There're lot of guys around who enjoys the
>> >>>>>>> >>>>> >> modularity of Harmony classlib and eventually they might want to
>> >>>>>>> split
>> >>>>>>> >>>>> >> the packages even deeper, into smaller pieces. Then automatic
>> >>>>>>> >>>>> >> generation would enable them to quickly roll-in and experiment
>> >>>>>>> with
>> >>>>>>> >>>>> >> different package layouts and their impact on performance. They
>> >>>>>>> could
>> >>>>>>> >>>>> >> use ordinary bootclasspath.properties, but your feature wouldn't
>> >>>>>>> be
>> >>>>>>> >>>>> >> used by them then ;)
>> >>>>>>> >>>>> >>
>> >>>>>>> >>>>> >> That's merely a housekeeping procedure. I believe that anything
>> >>>>>>> which
>> >>>>>>> >>>>> >> could be done more than once, eventually would be done more than
>> >>>>>>> once.
>> >>>>>>> >>>>> >> Hence it should be automated. You say that the file was generated
>> >>>>>>> from
>> >>>>>>> >>>>> >> manifests of JARs, so is it hard to just tie the same tool into
>> >>>>>>> DRLVM
>> >>>>>>> >>>>> >> build process?
>> >>>>>>> >>>>> >>
>> >>>>>>> >>>>> >> As for DRLVM-specific, my opinion that this is because the patch:
>> >>>>>>> >>>>> >> a. breaks the compatibility of classlib (you change
>> >>>>>>> >>>>> >> bootclasspath.properties, right?) with other VMs.
>> >>>>>>> >>>>> >> b. treated in DRLVM classloader only.
>> >>>>>>> >>>>> >>
>> >>>>>>> >>>>> >> Of course eventually this feature might be used by others, but IMO
>> >>>>>>> we
>> >>>>>>> >>>>> >> should be careful about other guys who use the same classlib. I'd
>> >>>>>>> >>>>> >> rather wait for some incubation on DRLVM side first.
>> >>>>>>> >>>>> >>
>> >>>>>>> >>>>> >> Thanks,
>> >>>>>>> >>>>> >> Aleksey.
>> >>>>>>> >>>>> >>
>> >>>>>>> >>>>> >> On Thu, Dec 25, 2008 at 6:18 PM, Wenlong Li <we...@gmail.com>
>> >>>>>>> wrote:
>> >>>>>>> >>>>> >>> I see. In fact, my file doesn't need track change at the class
>> >>>>>>> >>>>> >>> granularity. Instead, it only needs know package info provided in
>> >>>>>>> the
>> >>>>>>> >>>>> >>> manifest file. When class is added to a library, do we need
>> >>>>>>> change
>> >>>>>>> >>>>> >>> the manifest as well?
>> >>>>>>> >>>>> >>>
>> >>>>>>> >>>>> >>> btw, I guess there is a mis-understanding: my
>> >>>>>>> modulelibrarymapping
>> >>>>>>> >>>>> >>> file only records package info provided by manfiest in each
>> >>>>>>> module. It
>> >>>>>>> >>>>> >>> doesn't relate to each class.
>> >>>>>>> >>>>> >>>
>> >>>>>>> >>>>> >>> thx,
>> >>>>>>> >>>>> >>> Wenlong
>> >>>>>>> >>>>> >>>
>> >>>>>>> >>>>> >>> On Thu, Dec 25, 2008 at 10:55 PM, Pavel Pervov <
>> >>>>>>> pmcfirst@gmail.com>
>> >>>>>>> >>>>> wrote:
>> >>>>>>> >>>>> >>>> Classes are added to class library from time to time. I'm not
>> >>>>>>> sure how
>> >>>>>>> >>>>> >>>> it can be possible to track these changes manually.
>> >>>>>>> >>>>> >>>>
>> >>>>>>> >>>>> >>>> WBR,
>> >>>>>>> >>>>> >>>> Pavel.
>> >>>>>>> >>>>> >>>>
>> >>>>>>> >>>>> >>>>
>> >>>>>>> >>>>> >>>> On Thu, Dec 25, 2008 at 5:09 PM, Wenlong Li <we...@gmail.com>
>> >>>>>>> >>>>> wrote:
>> >>>>>>> >>>>> >>>>> Sorry, one more question: bootclasspath.properties is classlib
>> >>>>>>> >>>>> >>>>> specific file, why we could not make a vm specific file
>> >>>>>>> manually?
>> >>>>>>> >>>>> Just
>> >>>>>>> >>>>> >>>>> curious to know the possible reason. :)
>> >>>>>>> >>>>> >>>>>
>> >>>>>>> >>>>> >>>>> thx,
>> >>>>>>> >>>>> >>>>> Wenlong
>> >>>>>>> >>>>> >>>>>
>> >>>>>>> >>>>> >>>>> On Thu, Dec 25, 2008 at 10:00 PM, Pavel Pervov <
>> >>>>>>> pmcfirst@gmail.com>
>> >>>>>>> >>>>> wrote:
>> >>>>>>> >>>>> >>>>>> If this would be VM-side automatically produced configuration
>> >>>>>>> >>>>> file...
>> >>>>>>> >>>>> >>>>>>
>> >>>>>>> >>>>> >>>>>> WBR,
>> >>>>>>> >>>>> >>>>>> Pavel.
>> >>>>>>> >>>>> >>>>>>
>> >>>>>>> >>>>> >>>>>> On Thu, Dec 25, 2008 at 4:58 PM, Wenlong Li <
>> >>>>>>> wenlong@gmail.com>
>> >>>>>>> >>>>> wrote:
>> >>>>>>> >>>>> >>>>>>> btw, because adding new module is rare case, manually
>> >>>>>>> modifying the
>> >>>>>>> >>>>> >>>>>>> bootclasspath.properties is not an issue?
>> >>>>>>> >>>>> >>>>>>>
>> >>>>>>> >>>>> >>>>>>> If so, can I conclude adding another property file with same
>> >>>>>>> update
>> >>>>>>> >>>>> >>>>>>> frequency as bootclasspath would be fine as well?
>> >>>>>>> >>>>> >>>>>>>
>> >>>>>>> >>>>> >>>>>>> Pls kindly correct me if my understanding is wrong.
>> >>>>>>> >>>>> >>>>>>>
>> >>>>>>> >>>>> >>>>>>> On Thu, Dec 25, 2008 at 9:05 PM, Pavel Pervov <
>> >>>>>>> pmcfirst@gmail.com>
>> >>>>>>> >>>>> wrote:
>> >>>>>>> >>>>> >>>>>>>> Wenlong,
>> >>>>>>> >>>>> >>>>>>>>
>> >>>>>>> >>>>> >>>>>>>> Note, that bootclasspath.properties is only changed on
>> >>>>>>> adding new
>> >>>>>>> >>>>> >>>>>>>> module. This is pretty rare occasion, I believe.
>> >>>>>>> >>>>> >>>>>>>>
>> >>>>>>> >>>>> >>>>>>>> WBR,
>> >>>>>>> >>>>> >>>>>>>> Pavel.
>> >>>>>>> >>>>> >>>>>>>>
>> >>>>>>> >>>>> >>>>>>>> On Thu, Dec 25, 2008 at 3:48 PM, Wenlong Li <
>> >>>>>>> wenlong@gmail.com>
>> >>>>>>> >>>>> wrote:
>> >>>>>>> >>>>> >>>>>>>>> Thx for your advice. Alexey.
>> >>>>>>> >>>>> >>>>>>>>>
>> >>>>>>> >>>>> >>>>>>>>> Here I have one question: do you know how the
>> >>>>>>> >>>>> bootclasspath.properties
>> >>>>>>> >>>>> >>>>>>>>> is maintained, manually or automatically?
>> >>>>>>> >>>>> >>>>>>>>>
>> >>>>>>> >>>>> >>>>>>>>> Another comment is I would like to treat the patch as DRLVM
>> >>>>>>> >>>>> specific
>> >>>>>>> >>>>> >>>>>>>>> optimization, e.g., it targets for improving VM creation
>> >>>>>>> time. So
>> >>>>>>> >>>>> that
>> >>>>>>> >>>>> >>>>>>>>> is possible to move all updates to DRLVM part to eliminate
>> >>>>>>> >>>>> potential
>> >>>>>>> >>>>> >>>>>>>>> modularity and compatibility problem.
>> >>>>>>> >>>>> >>>>>>>>>
>> >>>>>>> >>>>> >>>>>>>>> thx,
>> >>>>>>> >>>>> >>>>>>>>> Wenlong
>> >>>>>>> >>>>> >>>>>>>>>
>> >>>>>>> >>>>> >>>>>>>>> On Thu, Dec 25, 2008 at 5:32 PM, Aleksey Shipilev
>> >>>>>>> >>>>> >>>>>>>>> <al...@gmail.com> wrote:
>> >>>>>>> >>>>> >>>>>>>>>> Hi, Wenlong.
>> >>>>>>> >>>>> >>>>>>>>>>
>> >>>>>>> >>>>> >>>>>>>>>> On Thu, Dec 25, 2008 at 11:49 AM, Wenlong Li <
>> >>>>>>> wenlong@gmail.com>
>> >>>>>>> >>>>> wrote:
>> >>>>>>> >>>>> >>>>>>>>>>> btw, Alexey, Let's go back to discuss whether there is a
>> >>>>>>> need
>> >>>>>>> >>>>> to
>> >>>>>>> >>>>> >>>>>>>>>>> include this feature in Harmony, given 17% performance
>> >>>>>>> gain in
>> >>>>>>> >>>>> Linux
>> >>>>>>> >>>>> >>>>>>>>>>> when using your methodology. For windows test, I will
>> >>>>>>> double
>> >>>>>>> >>>>> check the
>> >>>>>>> >>>>> >>>>>>>>>>> backgroud process as you pointed out.
>> >>>>>>> >>>>> >>>>>>>>>>
>> >>>>>>> >>>>> >>>>>>>>>> My opinion was already expressed after I had finished the
>> >>>>>>> tests
>> >>>>>>> >>>>> from
>> >>>>>>> >>>>> >>>>>>>>>> my side: the boost can be achieved in specific conditions,
>> >>>>>>> so
>> >>>>>>> >>>>> whether
>> >>>>>>> >>>>> >>>>>>>>>> it's worth including into Harmony really depends on how
>> >>>>>>> much
>> >>>>>>> >>>>> mess the
>> >>>>>>> >>>>> >>>>>>>>>> patch would introduce besides the "performance boost".
>> >>>>>>> From what
>> >>>>>>> >>>>> I
>> >>>>>>> >>>>> >>>>>>>>>> see, the patch obliges the maintainer to maintain the
>> >>>>>>> correct
>> >>>>>>> >>>>> mapping
>> >>>>>>> >>>>> >>>>>>>>>> between jars and Java packages. This new feature is also
>> >>>>>>> spread
>> >>>>>>> >>>>> >>>>>>>>>> between Classlib and VM, but it seems like DRLVM specific.
>> >>>>>>> In
>> >>>>>>> >>>>> this
>> >>>>>>> >>>>> >>>>>>>>>> case I would rather stay without the patch.
>> >>>>>>> >>>>> >>>>>>>>>>
>> >>>>>>> >>>>> >>>>>>>>>> Personally (if I'll be committer) I would accept the patch
>> >>>>>>> with
>> >>>>>>> >>>>> two
>> >>>>>>> >>>>> >>>>>>>>>> serious modifications:
>> >>>>>>> >>>>> >>>>>>>>>> 1. Stay within DRLVM, do not introduce this feature into
>> >>>>>>> >>>>> Classlib,
>> >>>>>>> >>>>> >>>>>>>>>> get the thing tested and evolved on DRLVM side. Otherwise
>> >>>>>>> it
>> >>>>>>> >>>>> might
>> >>>>>>> >>>>> >>>>>>>>>> break the compatibility with other VMs.
>> >>>>>>> >>>>> >>>>>>>>>> 2. Make the mapping generated automatically (during build
>> >>>>>>> >>>>> process?)
>> >>>>>>> >>>>> >>>>>>>>>> to free the burden for maintainers.
>> >>>>>>> >>>>> >>>>>>>>>>
>> >>>>>>> >>>>> >>>>>>>>>> Thanks,
>> >>>>>>> >>>>> >>>>>>>>>> Aleksey.
>> >>>>>>> >>>>> >>>>>>>>>>
>> >>>>>>> >>>>> >>>>>>>>>
>> >>>>>>> >>>>> >>>>>>>>
>> >>>>>>> >>>>> >>>>>>>
>> >>>>>>> >>>>> >>>>>>
>> >>>>>>> >>>>> >>>>>
>> >>>>>>> >>>>> >>>>
>> >>>>>>> >>>>> >>>
>> >>>>>>> >>>>> >>
>> >>>>>>> >>>>> >
>> >>>>>>> >>>>> >
>> >>>>>>> >>>>> >
>> >>>>>>> >>>>> > --
>> >>>>>>> >>>>> > С уважением,
>> >>>>>>> >>>>> > Алексей Федотов,
>> >>>>>>> >>>>> > ЗАО «Телеком Экспресс»
>> >>>>>>> >>>>> >
>> >>>>>>> >>>>>
>> >>>>>>> >>>>>
>> >>>>>>> >>>>>
>> >>>>>>> >>>>> --
>> >>>>>>> >>>>> http://xiao-feng.blogspot.com
>> >>>>>>> >>>>>
>> >>>>>>> >>>>
>> >>>>>>> >>>>
>> >>>>>>> >>>>
>> >>>>>>> >>>> --
>> >>>>>>> >>>> С уважением,
>> >>>>>>> >>>> Алексей Федотов,
>> >>>>>>> >>>> ЗАО «Телеком Экспресс»
>> >>>>>>> >>>>
>> >>>>>>> >>>
>> >>>>>>> >>>
>> >>>>>>> >>>
>> >>>>>>> >>> --
>> >>>>>>> >>> http://xiao-feng.blogspot.com
>> >>>>>>> >>>
>> >>>>>>> >>
>> >>>>>>> >>
>> >>>>>>> >>
>> >>>>>>> >> --
>> >>>>>>> >> С уважением,
>> >>>>>>> >> Алексей Федотов,
>> >>>>>>> >> ЗАО «Телеком Экспресс»
>> >>>>>>> >>
>> >>>>>>> >
>> >>>>>>> >
>> >>>>>>> >
>> >>>>>>> > --
>> >>>>>>> > http://xiao-feng.blogspot.com
>> >>>>>>> >
>> >>>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> --
>> >>>>>> С уважением,
>> >>>>>> Алексей Федотов,
>> >>>>>> ЗАО «Телеком Экспресс»
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>> >
>
>
>
> --
> С уважением,
> Алексей Федотов,
> ЗАО «Телеком Экспресс»
>
--
С уважением,
Алексей Федотов,
ЗАО «Телеком Экспресс»
Re: [VM] On-demand class library parsing is ready to commit
Posted by Alexei Fedotov <al...@gmail.com>.
Hello Wenlong,
The intention of the following review is to improve the readability of
the code. Please find my comments preceded with patch line numbers and
fix anything you find worthy to fix.
9
excessive comment length
9-
missed description of parameters (e.g. @param mapFile <description>)
and return value
do we need to pass mapFile through the parameter chain? may it be an element of
22, 24
we don't need both versions of each function, do we?
using one version (esp. of SetBCPElement) would make the whole code size smaller
it would be easier for me to review your deltas of functions if you
don't make the full copies of them
37
seems that environment.h has c/apr style set of includes
can we hide <map> and related typedef in sources to maintain C/apr
style of interfaces
is it possible to use more specific header (e.g. related to jar
parsing) than environment.h for JarFilePackageMapping definition?
93
*the* bootstrap classpath
96-
the proper bracket style is specified here
https://issues.apache.org/jira/secure/attachment/12353745/format.sh
[well, the whole file is formatted strangely - Pavel, could you comment?]
97
such -> the
150
*a* temp pool
154-
putting map file operations into separate .cpp file with a clear and
readable interface function names in the corresponding .h interface
would not make existing code less readable
that .h file would be a proper place for new types you introduce, not
environment.h
you may also use the proper Apache formatting in the new file
190-
cannot understand where the signature file comes from - I cannot find
apr_file_write for it
the explanatory comment is welcome
if both mapping and signature files are things introduced by this
patch why don't we use one file instead of two
200
can this be replaced with assert(luni_path)?
213
+1 to Aleksey's comment on literals
[have to go, will continue later]
434
commented code
Thanks.
On Wed, Jan 14, 2009 at 4:38 PM, Wenlong Li <we...@gmail.com> wrote:
>
> Alexei,
>
> Sorry for confusing. The patch for review is H6039.patch_2. Please
> kindly provide your comment.
>
> Aleksey,
>
> I have not measured the performance before completing the code review.
> I will do that later.
>
> thx, Wenlong
>
> On Wed, Jan 14, 2009 at 9:14 PM, Wenlong Li <we...@gmail.com> wrote:
> > Pavel,
> >
> > Pls see my comments in the JIRA.
> >
> > thx, Wenlong
> >
> > On Wed, Jan 14, 2009 at 8:44 PM, Pavel Pervov <pm...@gmail.com> wrote:
> >> Please, also, check that jar entry caches still work correctly after your patch.
> >>
> >> Pavel.
> >>
> >> On Wed, Jan 14, 2009 at 12:33 PM, Wenlong Li <we...@gmail.com> wrote:
> >>> All,
> >>>
> >>> I submit a new patch for on-demand class loading and parsing. All
> >>> codes are put in VM side, and the mapping info is automatically
> >>> produced.
> >>>
> >>> Pls see https://issues.apache.org/jira/browse/HARMONY-6039
> >>>
> >>> Comments are welcome.
> >>>
> >>> Thx, Wenlong
> >>> Managed Runtime Technology Center, Intel
> >>>
> >>> On Wed, Jan 7, 2009 at 12:08 PM, Wenlong Li <we...@gmail.com> wrote:
> >>>> All,
> >>>> At this moment, I move all updates in classlib to VM side such that
> >>>> there is no modularity issue. Next step is to produce the mapping
> >>>> between module and library efficiently and accurately.
> >>>>
> >>>> Comments are welcome.
> >>>>
> >>>> Thx, Wenlong
> >>>> Managed Runtime Technology Center, Intel
> >>>>
> >>>> On Tue, Jan 6, 2009 at 11:08 PM, Wenlong Li <we...@gmail.com> wrote:
> >>>>> Thx :)
> >>>>>
> >>>>> On Tue, Jan 6, 2009 at 10:35 PM, Alexei Fedotov
> >>>>> <al...@gmail.com> wrote:
> >>>>>> Sure.
> >>>>>>
> >>>>>> 1. If you dig into SetClasspathFromString, you will see that it starts from
> >>>>>> splitting the given classpath into pieces. You already know the new piece
> >>>>>> you add and may skip splitting step.
> >>>>>>
> >>>>>> 2. If I understand you code correctly, the case "pdest >
> >>>>>> (*it).second->bytes" might be a subject of a negative assertion. Adding this
> >>>>>> assrtion would speed up bug discovery.
> >>>>>>
> >>>>>> Thanks.
> >>>>>>
> >>>>>>
> >>>>>> On Tue, Jan 6, 2009 at 5:09 AM, Wenlong Li <we...@gmail.com> wrote:
> >>>>>>
> >>>>>>> Yes, Xiao-Feng's understanding is correct. The patch loads and parses
> >>>>>>> modules on demand. If no class in swing.jar is not requested, then
> >>>>>>> this module will not be loaded.
> >>>>>>>
> >>>>>>> btw, Alexei, you said "SetClasspathFromString" and "pdest >
> >>>>>>> (*it).second->bytes" are not efficient. Can you share more comments on
> >>>>>>> them? I just reused some code in Harmony, and didn't optimize them
> >>>>>>> further.
> >>>>>>>
> >>>>>>> Thx, Wenlong
> >>>>>>> Managed Runtime Technology Center, Intel
> >>>>>>>
> >>>>>>> On Fri, Dec 26, 2008 at 5:16 PM, Xiao-Feng Li <xi...@gmail.com>
> >>>>>>> wrote:
> >>>>>>> > On Fri, Dec 26, 2008 at 5:08 PM, Alexei Fedotov
> >>>>>>> > <al...@gmail.com> wrote:
> >>>>>>> >> Xiao Feng,
> >>>>>>> >> Thank you for explaining.
> >>>>>>> >>
> >>>>>>> >> I get more minor comments on more commented code, ineffective
> >>>>>>> >> SetClasspathFromString usage, non-covered unexpected case when pdest >
> >>>>>>> >> (*it).second->bytes. One major comment on crossing vm module boundary
> >>>>>>> >> still remains. But now I'm happy with the design.
> >>>>>>> >
> >>>>>>> > Alexei, yes, I agree with your comments. These parts should be
> >>>>>>> > improved. (Still, this is my personal opinion. :) Let's wait Wenlong
> >>>>>>> > speaking.)
> >>>>>>> >
> >>>>>>> > Thanks,
> >>>>>>> > xiaofeng
> >>>>>>> >
> >>>>>>> >> Sorry for being slow.
> >>>>>>> >>
> >>>>>>> >>
> >>>>>>> >>
> >>>>>>> >> On Fri, Dec 26, 2008 at 11:40 AM, Xiao-Feng Li <xi...@gmail.com>
> >>>>>>> wrote:
> >>>>>>> >>> On Fri, Dec 26, 2008 at 4:03 PM, Alexei Fedotov
> >>>>>>> >>> <al...@gmail.com> wrote:
> >>>>>>> >>>> Xiao-Feng,
> >>>>>>> >>>> Continuing with the server example could you please give me a hint
> >>>>>>> where
> >>>>>>> >>>> decision to load swing.jar or not is taken in the patch? My initial
> >>>>>>> >>>> perception was that the list of what to load was hardcoded and was not
> >>>>>>> >>>> constructed dynamically depending on application.
> >>>>>>> >>>
> >>>>>>> >>> Alexei, here is the patch code I found:
> >>>>>>> >>>
> >>>>>>> >>> line 245:
> >>>>>>> >>> + // Find which jar exports this package
> >>>>>>> >>> + if (pkgName != NULL) {
> >>>>>>> >>> + char *boot_class_path =
> >>>>>>> >>> env->JavaProperties()->get(VM_BOOT_CLASS_DIR);
> >>>>>>> >>> + char *pendingClassPath = NULL;
> >>>>>>> >>> + apr_pool_t *tmp_pool;
> >>>>>>> >>> + apr_pool_create(&tmp_pool, NULL);
> >>>>>>> >>> + while (it != env->pending_jar_set.end()) {
> >>>>>>> >>> + pdest = strstr( (*it).second->bytes, pkgName );
> >>>>>>> >>> + if (pdest != NULL) {
> >>>>>>> >>> + pendingClassPath =
> >>>>>>> >>> (char*)STD_MALLOC(strlen(boot_class_path)
> >>>>>>> >>> + +
> >>>>>>> strlen((*it).first->bytes) + 1);
> >>>>>>> >>> + strcpy(pendingClassPath, boot_class_path);
> >>>>>>> >>> + strcat(pendingClassPath, (*it).first->bytes);
> >>>>>>> >>> + // Open this found jar, and read all classes
> >>>>>>> >>> contained in this jar
> >>>>>>> >>> + SetClasspathFromString(pendingClassPath,
> >>>>>>> tmp_pool);
> >>>>>>> >>> + // Erase the found jar from pending jar list
> >>>>>>> >>> as it has been parsed
> >>>>>>> >>> + env->pending_jar_set.erase(it++);
> >>>>>>> >>> + STD_FREE(pendingClassPath);
> >>>>>>> >>> + } else {
> >>>>>>> >>>
> >>>>>>> >>> It checks if a JAR has the requested package, then loads it if yes. I
> >>>>>>> >>> am not sure if this is what you were asking. (Btw, this is only my
> >>>>>>> >>> understanding of his patch. I am not speaking for Wenlong.)
> >>>>>>> >>>
> >>>>>>> >>> Thanks,
> >>>>>>> >>> xiaofeng
> >>>>>>> >>>
> >>>>>>> >>>> Thanks.
> >>>>>>> >>>>
> >>>>>>> >>>>
> >>>>>>> >>>> On Fri, Dec 26, 2008 at 4:14 AM, Xiao-Feng Li <xi...@gmail.com>
> >>>>>>> wrote:
> >>>>>>> >>>>
> >>>>>>> >>>>> On Fri, Dec 26, 2008 at 12:49 AM, Alexei Fedotov
> >>>>>>> >>>>> <al...@gmail.com> wrote:
> >>>>>>> >>>>> > Aleksey,
> >>>>>>> >>>>> > I like your conclusion.
> >>>>>>> >>>>> >
> >>>>>>> >>>>> > Wenlong,
> >>>>>>> >>>>> > I'm trying to understand the real life value of the "abstract"
> >>>>>>> startup
> >>>>>>> >>>>> > time metric you've suggested. Does Harmony with your patch load
> >>>>>>> >>>>> > swing.jar for a server application? Do I understand that loading
> >>>>>>> >>>>> > happens, though it happens later compared to VM without your patch?
> >>>>>>> I
> >>>>>>> >>>>> > believe that the proper design of delayed loading should answer
> >>>>>>> "no"
> >>>>>>> >>>>> > to this question.
> >>>>>>> >>>>>
> >>>>>>> >>>>> I checked the patch, and I found the answer is indeed "No" as you
> >>>>>>> expected.
> >>>>>>> >>>>>
> >>>>>>> >>>>> Thanks,
> >>>>>>> >>>>> xiaofeng
> >>>>>>> >>>>>
> >>>>>>> >>>>> > In other words, I appreciate if you describe which real use cases
> >>>>>>> are
> >>>>>>> >>>>> > improved by this patch.
> >>>>>>> >>>>> > Thanks!
> >>>>>>> >>>>>
> >>>>>>> >>>>> > On Thu, Dec 25, 2008 at 7:29 PM, Aleksey Shipilev
> >>>>>>> >>>>> > <al...@gmail.com> wrote:
> >>>>>>> >>>>> >> Ok, here's the catch.
> >>>>>>> >>>>> >>
> >>>>>>> >>>>> >> bootclasspath.properties is the SortedSet<JARfile>, which
> >>>>>>> enumerates
> >>>>>>> >>>>> >> the JARs available for bootclassloader. The set of such the JARs
> >>>>>>> is
> >>>>>>> >>>>> >> really stable because modular decomposition of classlib is stable.
> >>>>>>> >>>>> >> That's why nobody bothers with automatic generation of it: it only
> >>>>>>> >>>>> >> should be updated when new JAR file arrives.
> >>>>>>> >>>>> >> Modulelibrarymapping.properties is different on this point, it's
> >>>>>>> the
> >>>>>>> >>>>> >> Map<PackageName,JARfile>, which should be updated each time the
> >>>>>>> new
> >>>>>>> >>>>> >> *package* is introduced. I'm not talking about java.* packages
> >>>>>>> >>>>> >> (they're standardized), rather about org.apache.harmony.*.
> >>>>>>> >>>>> >>
> >>>>>>> >>>>> >> Automatic generation of this property file gives two advantages:
> >>>>>>> >>>>> >> 1. Error-prone. Prevent yourself from hand-messing with mapping
> >>>>>>> and
> >>>>>>> >>>>> >> getting spurious ClassNotFoundException. BTW, what's the behaviour
> >>>>>>> in
> >>>>>>> >>>>> >> case the mapping is wrong?
> >>>>>>> >>>>> >> 2. "Researchful". There're lot of guys around who enjoys the
> >>>>>>> >>>>> >> modularity of Harmony classlib and eventually they might want to
> >>>>>>> split
> >>>>>>> >>>>> >> the packages even deeper, into smaller pieces. Then automatic
> >>>>>>> >>>>> >> generation would enable them to quickly roll-in and experiment
> >>>>>>> with
> >>>>>>> >>>>> >> different package layouts and their impact on performance. They
> >>>>>>> could
> >>>>>>> >>>>> >> use ordinary bootclasspath.properties, but your feature wouldn't
> >>>>>>> be
> >>>>>>> >>>>> >> used by them then ;)
> >>>>>>> >>>>> >>
> >>>>>>> >>>>> >> That's merely a housekeeping procedure. I believe that anything
> >>>>>>> which
> >>>>>>> >>>>> >> could be done more than once, eventually would be done more than
> >>>>>>> once.
> >>>>>>> >>>>> >> Hence it should be automated. You say that the file was generated
> >>>>>>> from
> >>>>>>> >>>>> >> manifests of JARs, so is it hard to just tie the same tool into
> >>>>>>> DRLVM
> >>>>>>> >>>>> >> build process?
> >>>>>>> >>>>> >>
> >>>>>>> >>>>> >> As for DRLVM-specific, my opinion that this is because the patch:
> >>>>>>> >>>>> >> a. breaks the compatibility of classlib (you change
> >>>>>>> >>>>> >> bootclasspath.properties, right?) with other VMs.
> >>>>>>> >>>>> >> b. treated in DRLVM classloader only.
> >>>>>>> >>>>> >>
> >>>>>>> >>>>> >> Of course eventually this feature might be used by others, but IMO
> >>>>>>> we
> >>>>>>> >>>>> >> should be careful about other guys who use the same classlib. I'd
> >>>>>>> >>>>> >> rather wait for some incubation on DRLVM side first.
> >>>>>>> >>>>> >>
> >>>>>>> >>>>> >> Thanks,
> >>>>>>> >>>>> >> Aleksey.
> >>>>>>> >>>>> >>
> >>>>>>> >>>>> >> On Thu, Dec 25, 2008 at 6:18 PM, Wenlong Li <we...@gmail.com>
> >>>>>>> wrote:
> >>>>>>> >>>>> >>> I see. In fact, my file doesn't need track change at the class
> >>>>>>> >>>>> >>> granularity. Instead, it only needs know package info provided in
> >>>>>>> the
> >>>>>>> >>>>> >>> manifest file. When class is added to a library, do we need
> >>>>>>> change
> >>>>>>> >>>>> >>> the manifest as well?
> >>>>>>> >>>>> >>>
> >>>>>>> >>>>> >>> btw, I guess there is a mis-understanding: my
> >>>>>>> modulelibrarymapping
> >>>>>>> >>>>> >>> file only records package info provided by manfiest in each
> >>>>>>> module. It
> >>>>>>> >>>>> >>> doesn't relate to each class.
> >>>>>>> >>>>> >>>
> >>>>>>> >>>>> >>> thx,
> >>>>>>> >>>>> >>> Wenlong
> >>>>>>> >>>>> >>>
> >>>>>>> >>>>> >>> On Thu, Dec 25, 2008 at 10:55 PM, Pavel Pervov <
> >>>>>>> pmcfirst@gmail.com>
> >>>>>>> >>>>> wrote:
> >>>>>>> >>>>> >>>> Classes are added to class library from time to time. I'm not
> >>>>>>> sure how
> >>>>>>> >>>>> >>>> it can be possible to track these changes manually.
> >>>>>>> >>>>> >>>>
> >>>>>>> >>>>> >>>> WBR,
> >>>>>>> >>>>> >>>> Pavel.
> >>>>>>> >>>>> >>>>
> >>>>>>> >>>>> >>>>
> >>>>>>> >>>>> >>>> On Thu, Dec 25, 2008 at 5:09 PM, Wenlong Li <we...@gmail.com>
> >>>>>>> >>>>> wrote:
> >>>>>>> >>>>> >>>>> Sorry, one more question: bootclasspath.properties is classlib
> >>>>>>> >>>>> >>>>> specific file, why we could not make a vm specific file
> >>>>>>> manually?
> >>>>>>> >>>>> Just
> >>>>>>> >>>>> >>>>> curious to know the possible reason. :)
> >>>>>>> >>>>> >>>>>
> >>>>>>> >>>>> >>>>> thx,
> >>>>>>> >>>>> >>>>> Wenlong
> >>>>>>> >>>>> >>>>>
> >>>>>>> >>>>> >>>>> On Thu, Dec 25, 2008 at 10:00 PM, Pavel Pervov <
> >>>>>>> pmcfirst@gmail.com>
> >>>>>>> >>>>> wrote:
> >>>>>>> >>>>> >>>>>> If this would be VM-side automatically produced configuration
> >>>>>>> >>>>> file...
> >>>>>>> >>>>> >>>>>>
> >>>>>>> >>>>> >>>>>> WBR,
> >>>>>>> >>>>> >>>>>> Pavel.
> >>>>>>> >>>>> >>>>>>
> >>>>>>> >>>>> >>>>>> On Thu, Dec 25, 2008 at 4:58 PM, Wenlong Li <
> >>>>>>> wenlong@gmail.com>
> >>>>>>> >>>>> wrote:
> >>>>>>> >>>>> >>>>>>> btw, because adding new module is rare case, manually
> >>>>>>> modifying the
> >>>>>>> >>>>> >>>>>>> bootclasspath.properties is not an issue?
> >>>>>>> >>>>> >>>>>>>
> >>>>>>> >>>>> >>>>>>> If so, can I conclude adding another property file with same
> >>>>>>> update
> >>>>>>> >>>>> >>>>>>> frequency as bootclasspath would be fine as well?
> >>>>>>> >>>>> >>>>>>>
> >>>>>>> >>>>> >>>>>>> Pls kindly correct me if my understanding is wrong.
> >>>>>>> >>>>> >>>>>>>
> >>>>>>> >>>>> >>>>>>> On Thu, Dec 25, 2008 at 9:05 PM, Pavel Pervov <
> >>>>>>> pmcfirst@gmail.com>
> >>>>>>> >>>>> wrote:
> >>>>>>> >>>>> >>>>>>>> Wenlong,
> >>>>>>> >>>>> >>>>>>>>
> >>>>>>> >>>>> >>>>>>>> Note, that bootclasspath.properties is only changed on
> >>>>>>> adding new
> >>>>>>> >>>>> >>>>>>>> module. This is pretty rare occasion, I believe.
> >>>>>>> >>>>> >>>>>>>>
> >>>>>>> >>>>> >>>>>>>> WBR,
> >>>>>>> >>>>> >>>>>>>> Pavel.
> >>>>>>> >>>>> >>>>>>>>
> >>>>>>> >>>>> >>>>>>>> On Thu, Dec 25, 2008 at 3:48 PM, Wenlong Li <
> >>>>>>> wenlong@gmail.com>
> >>>>>>> >>>>> wrote:
> >>>>>>> >>>>> >>>>>>>>> Thx for your advice. Alexey.
> >>>>>>> >>>>> >>>>>>>>>
> >>>>>>> >>>>> >>>>>>>>> Here I have one question: do you know how the
> >>>>>>> >>>>> bootclasspath.properties
> >>>>>>> >>>>> >>>>>>>>> is maintained, manually or automatically?
> >>>>>>> >>>>> >>>>>>>>>
> >>>>>>> >>>>> >>>>>>>>> Another comment is I would like to treat the patch as DRLVM
> >>>>>>> >>>>> specific
> >>>>>>> >>>>> >>>>>>>>> optimization, e.g., it targets for improving VM creation
> >>>>>>> time. So
> >>>>>>> >>>>> that
> >>>>>>> >>>>> >>>>>>>>> is possible to move all updates to DRLVM part to eliminate
> >>>>>>> >>>>> potential
> >>>>>>> >>>>> >>>>>>>>> modularity and compatibility problem.
> >>>>>>> >>>>> >>>>>>>>>
> >>>>>>> >>>>> >>>>>>>>> thx,
> >>>>>>> >>>>> >>>>>>>>> Wenlong
> >>>>>>> >>>>> >>>>>>>>>
> >>>>>>> >>>>> >>>>>>>>> On Thu, Dec 25, 2008 at 5:32 PM, Aleksey Shipilev
> >>>>>>> >>>>> >>>>>>>>> <al...@gmail.com> wrote:
> >>>>>>> >>>>> >>>>>>>>>> Hi, Wenlong.
> >>>>>>> >>>>> >>>>>>>>>>
> >>>>>>> >>>>> >>>>>>>>>> On Thu, Dec 25, 2008 at 11:49 AM, Wenlong Li <
> >>>>>>> wenlong@gmail.com>
> >>>>>>> >>>>> wrote:
> >>>>>>> >>>>> >>>>>>>>>>> btw, Alexey, Let's go back to discuss whether there is a
> >>>>>>> need
> >>>>>>> >>>>> to
> >>>>>>> >>>>> >>>>>>>>>>> include this feature in Harmony, given 17% performance
> >>>>>>> gain in
> >>>>>>> >>>>> Linux
> >>>>>>> >>>>> >>>>>>>>>>> when using your methodology. For windows test, I will
> >>>>>>> double
> >>>>>>> >>>>> check the
> >>>>>>> >>>>> >>>>>>>>>>> backgroud process as you pointed out.
> >>>>>>> >>>>> >>>>>>>>>>
> >>>>>>> >>>>> >>>>>>>>>> My opinion was already expressed after I had finished the
> >>>>>>> tests
> >>>>>>> >>>>> from
> >>>>>>> >>>>> >>>>>>>>>> my side: the boost can be achieved in specific conditions,
> >>>>>>> so
> >>>>>>> >>>>> whether
> >>>>>>> >>>>> >>>>>>>>>> it's worth including into Harmony really depends on how
> >>>>>>> much
> >>>>>>> >>>>> mess the
> >>>>>>> >>>>> >>>>>>>>>> patch would introduce besides the "performance boost".
> >>>>>>> From what
> >>>>>>> >>>>> I
> >>>>>>> >>>>> >>>>>>>>>> see, the patch obliges the maintainer to maintain the
> >>>>>>> correct
> >>>>>>> >>>>> mapping
> >>>>>>> >>>>> >>>>>>>>>> between jars and Java packages. This new feature is also
> >>>>>>> spread
> >>>>>>> >>>>> >>>>>>>>>> between Classlib and VM, but it seems like DRLVM specific.
> >>>>>>> In
> >>>>>>> >>>>> this
> >>>>>>> >>>>> >>>>>>>>>> case I would rather stay without the patch.
> >>>>>>> >>>>> >>>>>>>>>>
> >>>>>>> >>>>> >>>>>>>>>> Personally (if I'll be committer) I would accept the patch
> >>>>>>> with
> >>>>>>> >>>>> two
> >>>>>>> >>>>> >>>>>>>>>> serious modifications:
> >>>>>>> >>>>> >>>>>>>>>> 1. Stay within DRLVM, do not introduce this feature into
> >>>>>>> >>>>> Classlib,
> >>>>>>> >>>>> >>>>>>>>>> get the thing tested and evolved on DRLVM side. Otherwise
> >>>>>>> it
> >>>>>>> >>>>> might
> >>>>>>> >>>>> >>>>>>>>>> break the compatibility with other VMs.
> >>>>>>> >>>>> >>>>>>>>>> 2. Make the mapping generated automatically (during build
> >>>>>>> >>>>> process?)
> >>>>>>> >>>>> >>>>>>>>>> to free the burden for maintainers.
> >>>>>>> >>>>> >>>>>>>>>>
> >>>>>>> >>>>> >>>>>>>>>> Thanks,
> >>>>>>> >>>>> >>>>>>>>>> Aleksey.
> >>>>>>> >>>>> >>>>>>>>>>
> >>>>>>> >>>>> >>>>>>>>>
> >>>>>>> >>>>> >>>>>>>>
> >>>>>>> >>>>> >>>>>>>
> >>>>>>> >>>>> >>>>>>
> >>>>>>> >>>>> >>>>>
> >>>>>>> >>>>> >>>>
> >>>>>>> >>>>> >>>
> >>>>>>> >>>>> >>
> >>>>>>> >>>>> >
> >>>>>>> >>>>> >
> >>>>>>> >>>>> >
> >>>>>>> >>>>> > --
> >>>>>>> >>>>> > С уважением,
> >>>>>>> >>>>> > Алексей Федотов,
> >>>>>>> >>>>> > ЗАО «Телеком Экспресс»
> >>>>>>> >>>>> >
> >>>>>>> >>>>>
> >>>>>>> >>>>>
> >>>>>>> >>>>>
> >>>>>>> >>>>> --
> >>>>>>> >>>>> http://xiao-feng.blogspot.com
> >>>>>>> >>>>>
> >>>>>>> >>>>
> >>>>>>> >>>>
> >>>>>>> >>>>
> >>>>>>> >>>> --
> >>>>>>> >>>> С уважением,
> >>>>>>> >>>> Алексей Федотов,
> >>>>>>> >>>> ЗАО «Телеком Экспресс»
> >>>>>>> >>>>
> >>>>>>> >>>
> >>>>>>> >>>
> >>>>>>> >>>
> >>>>>>> >>> --
> >>>>>>> >>> http://xiao-feng.blogspot.com
> >>>>>>> >>>
> >>>>>>> >>
> >>>>>>> >>
> >>>>>>> >>
> >>>>>>> >> --
> >>>>>>> >> С уважением,
> >>>>>>> >> Алексей Федотов,
> >>>>>>> >> ЗАО «Телеком Экспресс»
> >>>>>>> >>
> >>>>>>> >
> >>>>>>> >
> >>>>>>> >
> >>>>>>> > --
> >>>>>>> > http://xiao-feng.blogspot.com
> >>>>>>> >
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> С уважением,
> >>>>>> Алексей Федотов,
> >>>>>> ЗАО «Телеком Экспресс»
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> >
--
С уважением,
Алексей Федотов,
ЗАО «Телеком Экспресс»
Re: [VM] On-demand class library parsing is ready to commit
Posted by Wenlong Li <we...@gmail.com>.
Alexei,
Sorry for confusing. The patch for review is H6039.patch_2. Please
kindly provide your comment.
Aleksey,
I have not measured the performance before completing the code review.
I will do that later.
thx, Wenlong
On Wed, Jan 14, 2009 at 9:14 PM, Wenlong Li <we...@gmail.com> wrote:
> Pavel,
>
> Pls see my comments in the JIRA.
>
> thx, Wenlong
>
> On Wed, Jan 14, 2009 at 8:44 PM, Pavel Pervov <pm...@gmail.com> wrote:
>> Please, also, check that jar entry caches still work correctly after your patch.
>>
>> Pavel.
>>
>> On Wed, Jan 14, 2009 at 12:33 PM, Wenlong Li <we...@gmail.com> wrote:
>>> All,
>>>
>>> I submit a new patch for on-demand class loading and parsing. All
>>> codes are put in VM side, and the mapping info is automatically
>>> produced.
>>>
>>> Pls see https://issues.apache.org/jira/browse/HARMONY-6039
>>>
>>> Comments are welcome.
>>>
>>> Thx, Wenlong
>>> Managed Runtime Technology Center, Intel
>>>
>>> On Wed, Jan 7, 2009 at 12:08 PM, Wenlong Li <we...@gmail.com> wrote:
>>>> All,
>>>> At this moment, I move all updates in classlib to VM side such that
>>>> there is no modularity issue. Next step is to produce the mapping
>>>> between module and library efficiently and accurately.
>>>>
>>>> Comments are welcome.
>>>>
>>>> Thx, Wenlong
>>>> Managed Runtime Technology Center, Intel
>>>>
>>>> On Tue, Jan 6, 2009 at 11:08 PM, Wenlong Li <we...@gmail.com> wrote:
>>>>> Thx :)
>>>>>
>>>>> On Tue, Jan 6, 2009 at 10:35 PM, Alexei Fedotov
>>>>> <al...@gmail.com> wrote:
>>>>>> Sure.
>>>>>>
>>>>>> 1. If you dig into SetClasspathFromString, you will see that it starts from
>>>>>> splitting the given classpath into pieces. You already know the new piece
>>>>>> you add and may skip splitting step.
>>>>>>
>>>>>> 2. If I understand you code correctly, the case "pdest >
>>>>>> (*it).second->bytes" might be a subject of a negative assertion. Adding this
>>>>>> assrtion would speed up bug discovery.
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>>
>>>>>> On Tue, Jan 6, 2009 at 5:09 AM, Wenlong Li <we...@gmail.com> wrote:
>>>>>>
>>>>>>> Yes, Xiao-Feng's understanding is correct. The patch loads and parses
>>>>>>> modules on demand. If no class in swing.jar is not requested, then
>>>>>>> this module will not be loaded.
>>>>>>>
>>>>>>> btw, Alexei, you said "SetClasspathFromString" and "pdest >
>>>>>>> (*it).second->bytes" are not efficient. Can you share more comments on
>>>>>>> them? I just reused some code in Harmony, and didn't optimize them
>>>>>>> further.
>>>>>>>
>>>>>>> Thx, Wenlong
>>>>>>> Managed Runtime Technology Center, Intel
>>>>>>>
>>>>>>> On Fri, Dec 26, 2008 at 5:16 PM, Xiao-Feng Li <xi...@gmail.com>
>>>>>>> wrote:
>>>>>>> > On Fri, Dec 26, 2008 at 5:08 PM, Alexei Fedotov
>>>>>>> > <al...@gmail.com> wrote:
>>>>>>> >> Xiao Feng,
>>>>>>> >> Thank you for explaining.
>>>>>>> >>
>>>>>>> >> I get more minor comments on more commented code, ineffective
>>>>>>> >> SetClasspathFromString usage, non-covered unexpected case when pdest >
>>>>>>> >> (*it).second->bytes. One major comment on crossing vm module boundary
>>>>>>> >> still remains. But now I'm happy with the design.
>>>>>>> >
>>>>>>> > Alexei, yes, I agree with your comments. These parts should be
>>>>>>> > improved. (Still, this is my personal opinion. :) Let's wait Wenlong
>>>>>>> > speaking.)
>>>>>>> >
>>>>>>> > Thanks,
>>>>>>> > xiaofeng
>>>>>>> >
>>>>>>> >> Sorry for being slow.
>>>>>>> >>
>>>>>>> >>
>>>>>>> >>
>>>>>>> >> On Fri, Dec 26, 2008 at 11:40 AM, Xiao-Feng Li <xi...@gmail.com>
>>>>>>> wrote:
>>>>>>> >>> On Fri, Dec 26, 2008 at 4:03 PM, Alexei Fedotov
>>>>>>> >>> <al...@gmail.com> wrote:
>>>>>>> >>>> Xiao-Feng,
>>>>>>> >>>> Continuing with the server example could you please give me a hint
>>>>>>> where
>>>>>>> >>>> decision to load swing.jar or not is taken in the patch? My initial
>>>>>>> >>>> perception was that the list of what to load was hardcoded and was not
>>>>>>> >>>> constructed dynamically depending on application.
>>>>>>> >>>
>>>>>>> >>> Alexei, here is the patch code I found:
>>>>>>> >>>
>>>>>>> >>> line 245:
>>>>>>> >>> + // Find which jar exports this package
>>>>>>> >>> + if (pkgName != NULL) {
>>>>>>> >>> + char *boot_class_path =
>>>>>>> >>> env->JavaProperties()->get(VM_BOOT_CLASS_DIR);
>>>>>>> >>> + char *pendingClassPath = NULL;
>>>>>>> >>> + apr_pool_t *tmp_pool;
>>>>>>> >>> + apr_pool_create(&tmp_pool, NULL);
>>>>>>> >>> + while (it != env->pending_jar_set.end()) {
>>>>>>> >>> + pdest = strstr( (*it).second->bytes, pkgName );
>>>>>>> >>> + if (pdest != NULL) {
>>>>>>> >>> + pendingClassPath =
>>>>>>> >>> (char*)STD_MALLOC(strlen(boot_class_path)
>>>>>>> >>> + +
>>>>>>> strlen((*it).first->bytes) + 1);
>>>>>>> >>> + strcpy(pendingClassPath, boot_class_path);
>>>>>>> >>> + strcat(pendingClassPath, (*it).first->bytes);
>>>>>>> >>> + // Open this found jar, and read all classes
>>>>>>> >>> contained in this jar
>>>>>>> >>> + SetClasspathFromString(pendingClassPath,
>>>>>>> tmp_pool);
>>>>>>> >>> + // Erase the found jar from pending jar list
>>>>>>> >>> as it has been parsed
>>>>>>> >>> + env->pending_jar_set.erase(it++);
>>>>>>> >>> + STD_FREE(pendingClassPath);
>>>>>>> >>> + } else {
>>>>>>> >>>
>>>>>>> >>> It checks if a JAR has the requested package, then loads it if yes. I
>>>>>>> >>> am not sure if this is what you were asking. (Btw, this is only my
>>>>>>> >>> understanding of his patch. I am not speaking for Wenlong.)
>>>>>>> >>>
>>>>>>> >>> Thanks,
>>>>>>> >>> xiaofeng
>>>>>>> >>>
>>>>>>> >>>> Thanks.
>>>>>>> >>>>
>>>>>>> >>>>
>>>>>>> >>>> On Fri, Dec 26, 2008 at 4:14 AM, Xiao-Feng Li <xi...@gmail.com>
>>>>>>> wrote:
>>>>>>> >>>>
>>>>>>> >>>>> On Fri, Dec 26, 2008 at 12:49 AM, Alexei Fedotov
>>>>>>> >>>>> <al...@gmail.com> wrote:
>>>>>>> >>>>> > Aleksey,
>>>>>>> >>>>> > I like your conclusion.
>>>>>>> >>>>> >
>>>>>>> >>>>> > Wenlong,
>>>>>>> >>>>> > I'm trying to understand the real life value of the "abstract"
>>>>>>> startup
>>>>>>> >>>>> > time metric you've suggested. Does Harmony with your patch load
>>>>>>> >>>>> > swing.jar for a server application? Do I understand that loading
>>>>>>> >>>>> > happens, though it happens later compared to VM without your patch?
>>>>>>> I
>>>>>>> >>>>> > believe that the proper design of delayed loading should answer
>>>>>>> "no"
>>>>>>> >>>>> > to this question.
>>>>>>> >>>>>
>>>>>>> >>>>> I checked the patch, and I found the answer is indeed "No" as you
>>>>>>> expected.
>>>>>>> >>>>>
>>>>>>> >>>>> Thanks,
>>>>>>> >>>>> xiaofeng
>>>>>>> >>>>>
>>>>>>> >>>>> > In other words, I appreciate if you describe which real use cases
>>>>>>> are
>>>>>>> >>>>> > improved by this patch.
>>>>>>> >>>>> > Thanks!
>>>>>>> >>>>>
>>>>>>> >>>>> > On Thu, Dec 25, 2008 at 7:29 PM, Aleksey Shipilev
>>>>>>> >>>>> > <al...@gmail.com> wrote:
>>>>>>> >>>>> >> Ok, here's the catch.
>>>>>>> >>>>> >>
>>>>>>> >>>>> >> bootclasspath.properties is the SortedSet<JARfile>, which
>>>>>>> enumerates
>>>>>>> >>>>> >> the JARs available for bootclassloader. The set of such the JARs
>>>>>>> is
>>>>>>> >>>>> >> really stable because modular decomposition of classlib is stable.
>>>>>>> >>>>> >> That's why nobody bothers with automatic generation of it: it only
>>>>>>> >>>>> >> should be updated when new JAR file arrives.
>>>>>>> >>>>> >> Modulelibrarymapping.properties is different on this point, it's
>>>>>>> the
>>>>>>> >>>>> >> Map<PackageName,JARfile>, which should be updated each time the
>>>>>>> new
>>>>>>> >>>>> >> *package* is introduced. I'm not talking about java.* packages
>>>>>>> >>>>> >> (they're standardized), rather about org.apache.harmony.*.
>>>>>>> >>>>> >>
>>>>>>> >>>>> >> Automatic generation of this property file gives two advantages:
>>>>>>> >>>>> >> 1. Error-prone. Prevent yourself from hand-messing with mapping
>>>>>>> and
>>>>>>> >>>>> >> getting spurious ClassNotFoundException. BTW, what's the behaviour
>>>>>>> in
>>>>>>> >>>>> >> case the mapping is wrong?
>>>>>>> >>>>> >> 2. "Researchful". There're lot of guys around who enjoys the
>>>>>>> >>>>> >> modularity of Harmony classlib and eventually they might want to
>>>>>>> split
>>>>>>> >>>>> >> the packages even deeper, into smaller pieces. Then automatic
>>>>>>> >>>>> >> generation would enable them to quickly roll-in and experiment
>>>>>>> with
>>>>>>> >>>>> >> different package layouts and their impact on performance. They
>>>>>>> could
>>>>>>> >>>>> >> use ordinary bootclasspath.properties, but your feature wouldn't
>>>>>>> be
>>>>>>> >>>>> >> used by them then ;)
>>>>>>> >>>>> >>
>>>>>>> >>>>> >> That's merely a housekeeping procedure. I believe that anything
>>>>>>> which
>>>>>>> >>>>> >> could be done more than once, eventually would be done more than
>>>>>>> once.
>>>>>>> >>>>> >> Hence it should be automated. You say that the file was generated
>>>>>>> from
>>>>>>> >>>>> >> manifests of JARs, so is it hard to just tie the same tool into
>>>>>>> DRLVM
>>>>>>> >>>>> >> build process?
>>>>>>> >>>>> >>
>>>>>>> >>>>> >> As for DRLVM-specific, my opinion that this is because the patch:
>>>>>>> >>>>> >> a. breaks the compatibility of classlib (you change
>>>>>>> >>>>> >> bootclasspath.properties, right?) with other VMs.
>>>>>>> >>>>> >> b. treated in DRLVM classloader only.
>>>>>>> >>>>> >>
>>>>>>> >>>>> >> Of course eventually this feature might be used by others, but IMO
>>>>>>> we
>>>>>>> >>>>> >> should be careful about other guys who use the same classlib. I'd
>>>>>>> >>>>> >> rather wait for some incubation on DRLVM side first.
>>>>>>> >>>>> >>
>>>>>>> >>>>> >> Thanks,
>>>>>>> >>>>> >> Aleksey.
>>>>>>> >>>>> >>
>>>>>>> >>>>> >> On Thu, Dec 25, 2008 at 6:18 PM, Wenlong Li <we...@gmail.com>
>>>>>>> wrote:
>>>>>>> >>>>> >>> I see. In fact, my file doesn't need track change at the class
>>>>>>> >>>>> >>> granularity. Instead, it only needs know package info provided in
>>>>>>> the
>>>>>>> >>>>> >>> manifest file. When class is added to a library, do we need
>>>>>>> change
>>>>>>> >>>>> >>> the manifest as well?
>>>>>>> >>>>> >>>
>>>>>>> >>>>> >>> btw, I guess there is a mis-understanding: my
>>>>>>> modulelibrarymapping
>>>>>>> >>>>> >>> file only records package info provided by manfiest in each
>>>>>>> module. It
>>>>>>> >>>>> >>> doesn't relate to each class.
>>>>>>> >>>>> >>>
>>>>>>> >>>>> >>> thx,
>>>>>>> >>>>> >>> Wenlong
>>>>>>> >>>>> >>>
>>>>>>> >>>>> >>> On Thu, Dec 25, 2008 at 10:55 PM, Pavel Pervov <
>>>>>>> pmcfirst@gmail.com>
>>>>>>> >>>>> wrote:
>>>>>>> >>>>> >>>> Classes are added to class library from time to time. I'm not
>>>>>>> sure how
>>>>>>> >>>>> >>>> it can be possible to track these changes manually.
>>>>>>> >>>>> >>>>
>>>>>>> >>>>> >>>> WBR,
>>>>>>> >>>>> >>>> Pavel.
>>>>>>> >>>>> >>>>
>>>>>>> >>>>> >>>>
>>>>>>> >>>>> >>>> On Thu, Dec 25, 2008 at 5:09 PM, Wenlong Li <we...@gmail.com>
>>>>>>> >>>>> wrote:
>>>>>>> >>>>> >>>>> Sorry, one more question: bootclasspath.properties is classlib
>>>>>>> >>>>> >>>>> specific file, why we could not make a vm specific file
>>>>>>> manually?
>>>>>>> >>>>> Just
>>>>>>> >>>>> >>>>> curious to know the possible reason. :)
>>>>>>> >>>>> >>>>>
>>>>>>> >>>>> >>>>> thx,
>>>>>>> >>>>> >>>>> Wenlong
>>>>>>> >>>>> >>>>>
>>>>>>> >>>>> >>>>> On Thu, Dec 25, 2008 at 10:00 PM, Pavel Pervov <
>>>>>>> pmcfirst@gmail.com>
>>>>>>> >>>>> wrote:
>>>>>>> >>>>> >>>>>> If this would be VM-side automatically produced configuration
>>>>>>> >>>>> file...
>>>>>>> >>>>> >>>>>>
>>>>>>> >>>>> >>>>>> WBR,
>>>>>>> >>>>> >>>>>> Pavel.
>>>>>>> >>>>> >>>>>>
>>>>>>> >>>>> >>>>>> On Thu, Dec 25, 2008 at 4:58 PM, Wenlong Li <
>>>>>>> wenlong@gmail.com>
>>>>>>> >>>>> wrote:
>>>>>>> >>>>> >>>>>>> btw, because adding new module is rare case, manually
>>>>>>> modifying the
>>>>>>> >>>>> >>>>>>> bootclasspath.properties is not an issue?
>>>>>>> >>>>> >>>>>>>
>>>>>>> >>>>> >>>>>>> If so, can I conclude adding another property file with same
>>>>>>> update
>>>>>>> >>>>> >>>>>>> frequency as bootclasspath would be fine as well?
>>>>>>> >>>>> >>>>>>>
>>>>>>> >>>>> >>>>>>> Pls kindly correct me if my understanding is wrong.
>>>>>>> >>>>> >>>>>>>
>>>>>>> >>>>> >>>>>>> On Thu, Dec 25, 2008 at 9:05 PM, Pavel Pervov <
>>>>>>> pmcfirst@gmail.com>
>>>>>>> >>>>> wrote:
>>>>>>> >>>>> >>>>>>>> Wenlong,
>>>>>>> >>>>> >>>>>>>>
>>>>>>> >>>>> >>>>>>>> Note, that bootclasspath.properties is only changed on
>>>>>>> adding new
>>>>>>> >>>>> >>>>>>>> module. This is pretty rare occasion, I believe.
>>>>>>> >>>>> >>>>>>>>
>>>>>>> >>>>> >>>>>>>> WBR,
>>>>>>> >>>>> >>>>>>>> Pavel.
>>>>>>> >>>>> >>>>>>>>
>>>>>>> >>>>> >>>>>>>> On Thu, Dec 25, 2008 at 3:48 PM, Wenlong Li <
>>>>>>> wenlong@gmail.com>
>>>>>>> >>>>> wrote:
>>>>>>> >>>>> >>>>>>>>> Thx for your advice. Alexey.
>>>>>>> >>>>> >>>>>>>>>
>>>>>>> >>>>> >>>>>>>>> Here I have one question: do you know how the
>>>>>>> >>>>> bootclasspath.properties
>>>>>>> >>>>> >>>>>>>>> is maintained, manually or automatically?
>>>>>>> >>>>> >>>>>>>>>
>>>>>>> >>>>> >>>>>>>>> Another comment is I would like to treat the patch as DRLVM
>>>>>>> >>>>> specific
>>>>>>> >>>>> >>>>>>>>> optimization, e.g., it targets for improving VM creation
>>>>>>> time. So
>>>>>>> >>>>> that
>>>>>>> >>>>> >>>>>>>>> is possible to move all updates to DRLVM part to eliminate
>>>>>>> >>>>> potential
>>>>>>> >>>>> >>>>>>>>> modularity and compatibility problem.
>>>>>>> >>>>> >>>>>>>>>
>>>>>>> >>>>> >>>>>>>>> thx,
>>>>>>> >>>>> >>>>>>>>> Wenlong
>>>>>>> >>>>> >>>>>>>>>
>>>>>>> >>>>> >>>>>>>>> On Thu, Dec 25, 2008 at 5:32 PM, Aleksey Shipilev
>>>>>>> >>>>> >>>>>>>>> <al...@gmail.com> wrote:
>>>>>>> >>>>> >>>>>>>>>> Hi, Wenlong.
>>>>>>> >>>>> >>>>>>>>>>
>>>>>>> >>>>> >>>>>>>>>> On Thu, Dec 25, 2008 at 11:49 AM, Wenlong Li <
>>>>>>> wenlong@gmail.com>
>>>>>>> >>>>> wrote:
>>>>>>> >>>>> >>>>>>>>>>> btw, Alexey, Let's go back to discuss whether there is a
>>>>>>> need
>>>>>>> >>>>> to
>>>>>>> >>>>> >>>>>>>>>>> include this feature in Harmony, given 17% performance
>>>>>>> gain in
>>>>>>> >>>>> Linux
>>>>>>> >>>>> >>>>>>>>>>> when using your methodology. For windows test, I will
>>>>>>> double
>>>>>>> >>>>> check the
>>>>>>> >>>>> >>>>>>>>>>> backgroud process as you pointed out.
>>>>>>> >>>>> >>>>>>>>>>
>>>>>>> >>>>> >>>>>>>>>> My opinion was already expressed after I had finished the
>>>>>>> tests
>>>>>>> >>>>> from
>>>>>>> >>>>> >>>>>>>>>> my side: the boost can be achieved in specific conditions,
>>>>>>> so
>>>>>>> >>>>> whether
>>>>>>> >>>>> >>>>>>>>>> it's worth including into Harmony really depends on how
>>>>>>> much
>>>>>>> >>>>> mess the
>>>>>>> >>>>> >>>>>>>>>> patch would introduce besides the "performance boost".
>>>>>>> From what
>>>>>>> >>>>> I
>>>>>>> >>>>> >>>>>>>>>> see, the patch obliges the maintainer to maintain the
>>>>>>> correct
>>>>>>> >>>>> mapping
>>>>>>> >>>>> >>>>>>>>>> between jars and Java packages. This new feature is also
>>>>>>> spread
>>>>>>> >>>>> >>>>>>>>>> between Classlib and VM, but it seems like DRLVM specific.
>>>>>>> In
>>>>>>> >>>>> this
>>>>>>> >>>>> >>>>>>>>>> case I would rather stay without the patch.
>>>>>>> >>>>> >>>>>>>>>>
>>>>>>> >>>>> >>>>>>>>>> Personally (if I'll be committer) I would accept the patch
>>>>>>> with
>>>>>>> >>>>> two
>>>>>>> >>>>> >>>>>>>>>> serious modifications:
>>>>>>> >>>>> >>>>>>>>>> 1. Stay within DRLVM, do not introduce this feature into
>>>>>>> >>>>> Classlib,
>>>>>>> >>>>> >>>>>>>>>> get the thing tested and evolved on DRLVM side. Otherwise
>>>>>>> it
>>>>>>> >>>>> might
>>>>>>> >>>>> >>>>>>>>>> break the compatibility with other VMs.
>>>>>>> >>>>> >>>>>>>>>> 2. Make the mapping generated automatically (during build
>>>>>>> >>>>> process?)
>>>>>>> >>>>> >>>>>>>>>> to free the burden for maintainers.
>>>>>>> >>>>> >>>>>>>>>>
>>>>>>> >>>>> >>>>>>>>>> Thanks,
>>>>>>> >>>>> >>>>>>>>>> Aleksey.
>>>>>>> >>>>> >>>>>>>>>>
>>>>>>> >>>>> >>>>>>>>>
>>>>>>> >>>>> >>>>>>>>
>>>>>>> >>>>> >>>>>>>
>>>>>>> >>>>> >>>>>>
>>>>>>> >>>>> >>>>>
>>>>>>> >>>>> >>>>
>>>>>>> >>>>> >>>
>>>>>>> >>>>> >>
>>>>>>> >>>>> >
>>>>>>> >>>>> >
>>>>>>> >>>>> >
>>>>>>> >>>>> > --
>>>>>>> >>>>> > С уважением,
>>>>>>> >>>>> > Алексей Федотов,
>>>>>>> >>>>> > ЗАО «Телеком Экспресс»
>>>>>>> >>>>> >
>>>>>>> >>>>>
>>>>>>> >>>>>
>>>>>>> >>>>>
>>>>>>> >>>>> --
>>>>>>> >>>>> http://xiao-feng.blogspot.com
>>>>>>> >>>>>
>>>>>>> >>>>
>>>>>>> >>>>
>>>>>>> >>>>
>>>>>>> >>>> --
>>>>>>> >>>> С уважением,
>>>>>>> >>>> Алексей Федотов,
>>>>>>> >>>> ЗАО «Телеком Экспресс»
>>>>>>> >>>>
>>>>>>> >>>
>>>>>>> >>>
>>>>>>> >>>
>>>>>>> >>> --
>>>>>>> >>> http://xiao-feng.blogspot.com
>>>>>>> >>>
>>>>>>> >>
>>>>>>> >>
>>>>>>> >>
>>>>>>> >> --
>>>>>>> >> С уважением,
>>>>>>> >> Алексей Федотов,
>>>>>>> >> ЗАО «Телеком Экспресс»
>>>>>>> >>
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> > --
>>>>>>> > http://xiao-feng.blogspot.com
>>>>>>> >
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> С уважением,
>>>>>> Алексей Федотов,
>>>>>> ЗАО «Телеком Экспресс»
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: [VM] On-demand class library parsing is ready to commit
Posted by Wenlong Li <we...@gmail.com>.
Pavel,
Pls see my comments in the JIRA.
thx, Wenlong
On Wed, Jan 14, 2009 at 8:44 PM, Pavel Pervov <pm...@gmail.com> wrote:
> Please, also, check that jar entry caches still work correctly after your patch.
>
> Pavel.
>
> On Wed, Jan 14, 2009 at 12:33 PM, Wenlong Li <we...@gmail.com> wrote:
>> All,
>>
>> I submit a new patch for on-demand class loading and parsing. All
>> codes are put in VM side, and the mapping info is automatically
>> produced.
>>
>> Pls see https://issues.apache.org/jira/browse/HARMONY-6039
>>
>> Comments are welcome.
>>
>> Thx, Wenlong
>> Managed Runtime Technology Center, Intel
>>
>> On Wed, Jan 7, 2009 at 12:08 PM, Wenlong Li <we...@gmail.com> wrote:
>>> All,
>>> At this moment, I move all updates in classlib to VM side such that
>>> there is no modularity issue. Next step is to produce the mapping
>>> between module and library efficiently and accurately.
>>>
>>> Comments are welcome.
>>>
>>> Thx, Wenlong
>>> Managed Runtime Technology Center, Intel
>>>
>>> On Tue, Jan 6, 2009 at 11:08 PM, Wenlong Li <we...@gmail.com> wrote:
>>>> Thx :)
>>>>
>>>> On Tue, Jan 6, 2009 at 10:35 PM, Alexei Fedotov
>>>> <al...@gmail.com> wrote:
>>>>> Sure.
>>>>>
>>>>> 1. If you dig into SetClasspathFromString, you will see that it starts from
>>>>> splitting the given classpath into pieces. You already know the new piece
>>>>> you add and may skip splitting step.
>>>>>
>>>>> 2. If I understand you code correctly, the case "pdest >
>>>>> (*it).second->bytes" might be a subject of a negative assertion. Adding this
>>>>> assrtion would speed up bug discovery.
>>>>>
>>>>> Thanks.
>>>>>
>>>>>
>>>>> On Tue, Jan 6, 2009 at 5:09 AM, Wenlong Li <we...@gmail.com> wrote:
>>>>>
>>>>>> Yes, Xiao-Feng's understanding is correct. The patch loads and parses
>>>>>> modules on demand. If no class in swing.jar is not requested, then
>>>>>> this module will not be loaded.
>>>>>>
>>>>>> btw, Alexei, you said "SetClasspathFromString" and "pdest >
>>>>>> (*it).second->bytes" are not efficient. Can you share more comments on
>>>>>> them? I just reused some code in Harmony, and didn't optimize them
>>>>>> further.
>>>>>>
>>>>>> Thx, Wenlong
>>>>>> Managed Runtime Technology Center, Intel
>>>>>>
>>>>>> On Fri, Dec 26, 2008 at 5:16 PM, Xiao-Feng Li <xi...@gmail.com>
>>>>>> wrote:
>>>>>> > On Fri, Dec 26, 2008 at 5:08 PM, Alexei Fedotov
>>>>>> > <al...@gmail.com> wrote:
>>>>>> >> Xiao Feng,
>>>>>> >> Thank you for explaining.
>>>>>> >>
>>>>>> >> I get more minor comments on more commented code, ineffective
>>>>>> >> SetClasspathFromString usage, non-covered unexpected case when pdest >
>>>>>> >> (*it).second->bytes. One major comment on crossing vm module boundary
>>>>>> >> still remains. But now I'm happy with the design.
>>>>>> >
>>>>>> > Alexei, yes, I agree with your comments. These parts should be
>>>>>> > improved. (Still, this is my personal opinion. :) Let's wait Wenlong
>>>>>> > speaking.)
>>>>>> >
>>>>>> > Thanks,
>>>>>> > xiaofeng
>>>>>> >
>>>>>> >> Sorry for being slow.
>>>>>> >>
>>>>>> >>
>>>>>> >>
>>>>>> >> On Fri, Dec 26, 2008 at 11:40 AM, Xiao-Feng Li <xi...@gmail.com>
>>>>>> wrote:
>>>>>> >>> On Fri, Dec 26, 2008 at 4:03 PM, Alexei Fedotov
>>>>>> >>> <al...@gmail.com> wrote:
>>>>>> >>>> Xiao-Feng,
>>>>>> >>>> Continuing with the server example could you please give me a hint
>>>>>> where
>>>>>> >>>> decision to load swing.jar or not is taken in the patch? My initial
>>>>>> >>>> perception was that the list of what to load was hardcoded and was not
>>>>>> >>>> constructed dynamically depending on application.
>>>>>> >>>
>>>>>> >>> Alexei, here is the patch code I found:
>>>>>> >>>
>>>>>> >>> line 245:
>>>>>> >>> + // Find which jar exports this package
>>>>>> >>> + if (pkgName != NULL) {
>>>>>> >>> + char *boot_class_path =
>>>>>> >>> env->JavaProperties()->get(VM_BOOT_CLASS_DIR);
>>>>>> >>> + char *pendingClassPath = NULL;
>>>>>> >>> + apr_pool_t *tmp_pool;
>>>>>> >>> + apr_pool_create(&tmp_pool, NULL);
>>>>>> >>> + while (it != env->pending_jar_set.end()) {
>>>>>> >>> + pdest = strstr( (*it).second->bytes, pkgName );
>>>>>> >>> + if (pdest != NULL) {
>>>>>> >>> + pendingClassPath =
>>>>>> >>> (char*)STD_MALLOC(strlen(boot_class_path)
>>>>>> >>> + +
>>>>>> strlen((*it).first->bytes) + 1);
>>>>>> >>> + strcpy(pendingClassPath, boot_class_path);
>>>>>> >>> + strcat(pendingClassPath, (*it).first->bytes);
>>>>>> >>> + // Open this found jar, and read all classes
>>>>>> >>> contained in this jar
>>>>>> >>> + SetClasspathFromString(pendingClassPath,
>>>>>> tmp_pool);
>>>>>> >>> + // Erase the found jar from pending jar list
>>>>>> >>> as it has been parsed
>>>>>> >>> + env->pending_jar_set.erase(it++);
>>>>>> >>> + STD_FREE(pendingClassPath);
>>>>>> >>> + } else {
>>>>>> >>>
>>>>>> >>> It checks if a JAR has the requested package, then loads it if yes. I
>>>>>> >>> am not sure if this is what you were asking. (Btw, this is only my
>>>>>> >>> understanding of his patch. I am not speaking for Wenlong.)
>>>>>> >>>
>>>>>> >>> Thanks,
>>>>>> >>> xiaofeng
>>>>>> >>>
>>>>>> >>>> Thanks.
>>>>>> >>>>
>>>>>> >>>>
>>>>>> >>>> On Fri, Dec 26, 2008 at 4:14 AM, Xiao-Feng Li <xi...@gmail.com>
>>>>>> wrote:
>>>>>> >>>>
>>>>>> >>>>> On Fri, Dec 26, 2008 at 12:49 AM, Alexei Fedotov
>>>>>> >>>>> <al...@gmail.com> wrote:
>>>>>> >>>>> > Aleksey,
>>>>>> >>>>> > I like your conclusion.
>>>>>> >>>>> >
>>>>>> >>>>> > Wenlong,
>>>>>> >>>>> > I'm trying to understand the real life value of the "abstract"
>>>>>> startup
>>>>>> >>>>> > time metric you've suggested. Does Harmony with your patch load
>>>>>> >>>>> > swing.jar for a server application? Do I understand that loading
>>>>>> >>>>> > happens, though it happens later compared to VM without your patch?
>>>>>> I
>>>>>> >>>>> > believe that the proper design of delayed loading should answer
>>>>>> "no"
>>>>>> >>>>> > to this question.
>>>>>> >>>>>
>>>>>> >>>>> I checked the patch, and I found the answer is indeed "No" as you
>>>>>> expected.
>>>>>> >>>>>
>>>>>> >>>>> Thanks,
>>>>>> >>>>> xiaofeng
>>>>>> >>>>>
>>>>>> >>>>> > In other words, I appreciate if you describe which real use cases
>>>>>> are
>>>>>> >>>>> > improved by this patch.
>>>>>> >>>>> > Thanks!
>>>>>> >>>>>
>>>>>> >>>>> > On Thu, Dec 25, 2008 at 7:29 PM, Aleksey Shipilev
>>>>>> >>>>> > <al...@gmail.com> wrote:
>>>>>> >>>>> >> Ok, here's the catch.
>>>>>> >>>>> >>
>>>>>> >>>>> >> bootclasspath.properties is the SortedSet<JARfile>, which
>>>>>> enumerates
>>>>>> >>>>> >> the JARs available for bootclassloader. The set of such the JARs
>>>>>> is
>>>>>> >>>>> >> really stable because modular decomposition of classlib is stable.
>>>>>> >>>>> >> That's why nobody bothers with automatic generation of it: it only
>>>>>> >>>>> >> should be updated when new JAR file arrives.
>>>>>> >>>>> >> Modulelibrarymapping.properties is different on this point, it's
>>>>>> the
>>>>>> >>>>> >> Map<PackageName,JARfile>, which should be updated each time the
>>>>>> new
>>>>>> >>>>> >> *package* is introduced. I'm not talking about java.* packages
>>>>>> >>>>> >> (they're standardized), rather about org.apache.harmony.*.
>>>>>> >>>>> >>
>>>>>> >>>>> >> Automatic generation of this property file gives two advantages:
>>>>>> >>>>> >> 1. Error-prone. Prevent yourself from hand-messing with mapping
>>>>>> and
>>>>>> >>>>> >> getting spurious ClassNotFoundException. BTW, what's the behaviour
>>>>>> in
>>>>>> >>>>> >> case the mapping is wrong?
>>>>>> >>>>> >> 2. "Researchful". There're lot of guys around who enjoys the
>>>>>> >>>>> >> modularity of Harmony classlib and eventually they might want to
>>>>>> split
>>>>>> >>>>> >> the packages even deeper, into smaller pieces. Then automatic
>>>>>> >>>>> >> generation would enable them to quickly roll-in and experiment
>>>>>> with
>>>>>> >>>>> >> different package layouts and their impact on performance. They
>>>>>> could
>>>>>> >>>>> >> use ordinary bootclasspath.properties, but your feature wouldn't
>>>>>> be
>>>>>> >>>>> >> used by them then ;)
>>>>>> >>>>> >>
>>>>>> >>>>> >> That's merely a housekeeping procedure. I believe that anything
>>>>>> which
>>>>>> >>>>> >> could be done more than once, eventually would be done more than
>>>>>> once.
>>>>>> >>>>> >> Hence it should be automated. You say that the file was generated
>>>>>> from
>>>>>> >>>>> >> manifests of JARs, so is it hard to just tie the same tool into
>>>>>> DRLVM
>>>>>> >>>>> >> build process?
>>>>>> >>>>> >>
>>>>>> >>>>> >> As for DRLVM-specific, my opinion that this is because the patch:
>>>>>> >>>>> >> a. breaks the compatibility of classlib (you change
>>>>>> >>>>> >> bootclasspath.properties, right?) with other VMs.
>>>>>> >>>>> >> b. treated in DRLVM classloader only.
>>>>>> >>>>> >>
>>>>>> >>>>> >> Of course eventually this feature might be used by others, but IMO
>>>>>> we
>>>>>> >>>>> >> should be careful about other guys who use the same classlib. I'd
>>>>>> >>>>> >> rather wait for some incubation on DRLVM side first.
>>>>>> >>>>> >>
>>>>>> >>>>> >> Thanks,
>>>>>> >>>>> >> Aleksey.
>>>>>> >>>>> >>
>>>>>> >>>>> >> On Thu, Dec 25, 2008 at 6:18 PM, Wenlong Li <we...@gmail.com>
>>>>>> wrote:
>>>>>> >>>>> >>> I see. In fact, my file doesn't need track change at the class
>>>>>> >>>>> >>> granularity. Instead, it only needs know package info provided in
>>>>>> the
>>>>>> >>>>> >>> manifest file. When class is added to a library, do we need
>>>>>> change
>>>>>> >>>>> >>> the manifest as well?
>>>>>> >>>>> >>>
>>>>>> >>>>> >>> btw, I guess there is a mis-understanding: my
>>>>>> modulelibrarymapping
>>>>>> >>>>> >>> file only records package info provided by manfiest in each
>>>>>> module. It
>>>>>> >>>>> >>> doesn't relate to each class.
>>>>>> >>>>> >>>
>>>>>> >>>>> >>> thx,
>>>>>> >>>>> >>> Wenlong
>>>>>> >>>>> >>>
>>>>>> >>>>> >>> On Thu, Dec 25, 2008 at 10:55 PM, Pavel Pervov <
>>>>>> pmcfirst@gmail.com>
>>>>>> >>>>> wrote:
>>>>>> >>>>> >>>> Classes are added to class library from time to time. I'm not
>>>>>> sure how
>>>>>> >>>>> >>>> it can be possible to track these changes manually.
>>>>>> >>>>> >>>>
>>>>>> >>>>> >>>> WBR,
>>>>>> >>>>> >>>> Pavel.
>>>>>> >>>>> >>>>
>>>>>> >>>>> >>>>
>>>>>> >>>>> >>>> On Thu, Dec 25, 2008 at 5:09 PM, Wenlong Li <we...@gmail.com>
>>>>>> >>>>> wrote:
>>>>>> >>>>> >>>>> Sorry, one more question: bootclasspath.properties is classlib
>>>>>> >>>>> >>>>> specific file, why we could not make a vm specific file
>>>>>> manually?
>>>>>> >>>>> Just
>>>>>> >>>>> >>>>> curious to know the possible reason. :)
>>>>>> >>>>> >>>>>
>>>>>> >>>>> >>>>> thx,
>>>>>> >>>>> >>>>> Wenlong
>>>>>> >>>>> >>>>>
>>>>>> >>>>> >>>>> On Thu, Dec 25, 2008 at 10:00 PM, Pavel Pervov <
>>>>>> pmcfirst@gmail.com>
>>>>>> >>>>> wrote:
>>>>>> >>>>> >>>>>> If this would be VM-side automatically produced configuration
>>>>>> >>>>> file...
>>>>>> >>>>> >>>>>>
>>>>>> >>>>> >>>>>> WBR,
>>>>>> >>>>> >>>>>> Pavel.
>>>>>> >>>>> >>>>>>
>>>>>> >>>>> >>>>>> On Thu, Dec 25, 2008 at 4:58 PM, Wenlong Li <
>>>>>> wenlong@gmail.com>
>>>>>> >>>>> wrote:
>>>>>> >>>>> >>>>>>> btw, because adding new module is rare case, manually
>>>>>> modifying the
>>>>>> >>>>> >>>>>>> bootclasspath.properties is not an issue?
>>>>>> >>>>> >>>>>>>
>>>>>> >>>>> >>>>>>> If so, can I conclude adding another property file with same
>>>>>> update
>>>>>> >>>>> >>>>>>> frequency as bootclasspath would be fine as well?
>>>>>> >>>>> >>>>>>>
>>>>>> >>>>> >>>>>>> Pls kindly correct me if my understanding is wrong.
>>>>>> >>>>> >>>>>>>
>>>>>> >>>>> >>>>>>> On Thu, Dec 25, 2008 at 9:05 PM, Pavel Pervov <
>>>>>> pmcfirst@gmail.com>
>>>>>> >>>>> wrote:
>>>>>> >>>>> >>>>>>>> Wenlong,
>>>>>> >>>>> >>>>>>>>
>>>>>> >>>>> >>>>>>>> Note, that bootclasspath.properties is only changed on
>>>>>> adding new
>>>>>> >>>>> >>>>>>>> module. This is pretty rare occasion, I believe.
>>>>>> >>>>> >>>>>>>>
>>>>>> >>>>> >>>>>>>> WBR,
>>>>>> >>>>> >>>>>>>> Pavel.
>>>>>> >>>>> >>>>>>>>
>>>>>> >>>>> >>>>>>>> On Thu, Dec 25, 2008 at 3:48 PM, Wenlong Li <
>>>>>> wenlong@gmail.com>
>>>>>> >>>>> wrote:
>>>>>> >>>>> >>>>>>>>> Thx for your advice. Alexey.
>>>>>> >>>>> >>>>>>>>>
>>>>>> >>>>> >>>>>>>>> Here I have one question: do you know how the
>>>>>> >>>>> bootclasspath.properties
>>>>>> >>>>> >>>>>>>>> is maintained, manually or automatically?
>>>>>> >>>>> >>>>>>>>>
>>>>>> >>>>> >>>>>>>>> Another comment is I would like to treat the patch as DRLVM
>>>>>> >>>>> specific
>>>>>> >>>>> >>>>>>>>> optimization, e.g., it targets for improving VM creation
>>>>>> time. So
>>>>>> >>>>> that
>>>>>> >>>>> >>>>>>>>> is possible to move all updates to DRLVM part to eliminate
>>>>>> >>>>> potential
>>>>>> >>>>> >>>>>>>>> modularity and compatibility problem.
>>>>>> >>>>> >>>>>>>>>
>>>>>> >>>>> >>>>>>>>> thx,
>>>>>> >>>>> >>>>>>>>> Wenlong
>>>>>> >>>>> >>>>>>>>>
>>>>>> >>>>> >>>>>>>>> On Thu, Dec 25, 2008 at 5:32 PM, Aleksey Shipilev
>>>>>> >>>>> >>>>>>>>> <al...@gmail.com> wrote:
>>>>>> >>>>> >>>>>>>>>> Hi, Wenlong.
>>>>>> >>>>> >>>>>>>>>>
>>>>>> >>>>> >>>>>>>>>> On Thu, Dec 25, 2008 at 11:49 AM, Wenlong Li <
>>>>>> wenlong@gmail.com>
>>>>>> >>>>> wrote:
>>>>>> >>>>> >>>>>>>>>>> btw, Alexey, Let's go back to discuss whether there is a
>>>>>> need
>>>>>> >>>>> to
>>>>>> >>>>> >>>>>>>>>>> include this feature in Harmony, given 17% performance
>>>>>> gain in
>>>>>> >>>>> Linux
>>>>>> >>>>> >>>>>>>>>>> when using your methodology. For windows test, I will
>>>>>> double
>>>>>> >>>>> check the
>>>>>> >>>>> >>>>>>>>>>> backgroud process as you pointed out.
>>>>>> >>>>> >>>>>>>>>>
>>>>>> >>>>> >>>>>>>>>> My opinion was already expressed after I had finished the
>>>>>> tests
>>>>>> >>>>> from
>>>>>> >>>>> >>>>>>>>>> my side: the boost can be achieved in specific conditions,
>>>>>> so
>>>>>> >>>>> whether
>>>>>> >>>>> >>>>>>>>>> it's worth including into Harmony really depends on how
>>>>>> much
>>>>>> >>>>> mess the
>>>>>> >>>>> >>>>>>>>>> patch would introduce besides the "performance boost".
>>>>>> From what
>>>>>> >>>>> I
>>>>>> >>>>> >>>>>>>>>> see, the patch obliges the maintainer to maintain the
>>>>>> correct
>>>>>> >>>>> mapping
>>>>>> >>>>> >>>>>>>>>> between jars and Java packages. This new feature is also
>>>>>> spread
>>>>>> >>>>> >>>>>>>>>> between Classlib and VM, but it seems like DRLVM specific.
>>>>>> In
>>>>>> >>>>> this
>>>>>> >>>>> >>>>>>>>>> case I would rather stay without the patch.
>>>>>> >>>>> >>>>>>>>>>
>>>>>> >>>>> >>>>>>>>>> Personally (if I'll be committer) I would accept the patch
>>>>>> with
>>>>>> >>>>> two
>>>>>> >>>>> >>>>>>>>>> serious modifications:
>>>>>> >>>>> >>>>>>>>>> 1. Stay within DRLVM, do not introduce this feature into
>>>>>> >>>>> Classlib,
>>>>>> >>>>> >>>>>>>>>> get the thing tested and evolved on DRLVM side. Otherwise
>>>>>> it
>>>>>> >>>>> might
>>>>>> >>>>> >>>>>>>>>> break the compatibility with other VMs.
>>>>>> >>>>> >>>>>>>>>> 2. Make the mapping generated automatically (during build
>>>>>> >>>>> process?)
>>>>>> >>>>> >>>>>>>>>> to free the burden for maintainers.
>>>>>> >>>>> >>>>>>>>>>
>>>>>> >>>>> >>>>>>>>>> Thanks,
>>>>>> >>>>> >>>>>>>>>> Aleksey.
>>>>>> >>>>> >>>>>>>>>>
>>>>>> >>>>> >>>>>>>>>
>>>>>> >>>>> >>>>>>>>
>>>>>> >>>>> >>>>>>>
>>>>>> >>>>> >>>>>>
>>>>>> >>>>> >>>>>
>>>>>> >>>>> >>>>
>>>>>> >>>>> >>>
>>>>>> >>>>> >>
>>>>>> >>>>> >
>>>>>> >>>>> >
>>>>>> >>>>> >
>>>>>> >>>>> > --
>>>>>> >>>>> > С уважением,
>>>>>> >>>>> > Алексей Федотов,
>>>>>> >>>>> > ЗАО «Телеком Экспресс»
>>>>>> >>>>> >
>>>>>> >>>>>
>>>>>> >>>>>
>>>>>> >>>>>
>>>>>> >>>>> --
>>>>>> >>>>> http://xiao-feng.blogspot.com
>>>>>> >>>>>
>>>>>> >>>>
>>>>>> >>>>
>>>>>> >>>>
>>>>>> >>>> --
>>>>>> >>>> С уважением,
>>>>>> >>>> Алексей Федотов,
>>>>>> >>>> ЗАО «Телеком Экспресс»
>>>>>> >>>>
>>>>>> >>>
>>>>>> >>>
>>>>>> >>>
>>>>>> >>> --
>>>>>> >>> http://xiao-feng.blogspot.com
>>>>>> >>>
>>>>>> >>
>>>>>> >>
>>>>>> >>
>>>>>> >> --
>>>>>> >> С уважением,
>>>>>> >> Алексей Федотов,
>>>>>> >> ЗАО «Телеком Экспресс»
>>>>>> >>
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > --
>>>>>> > http://xiao-feng.blogspot.com
>>>>>> >
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> С уважением,
>>>>> Алексей Федотов,
>>>>> ЗАО «Телеком Экспресс»
>>>>>
>>>>
>>>
>>
>
Re: [VM] On-demand class library parsing is ready to commit
Posted by Pavel Pervov <pm...@gmail.com>.
Please, also, check that jar entry caches still work correctly after your patch.
Pavel.
On Wed, Jan 14, 2009 at 12:33 PM, Wenlong Li <we...@gmail.com> wrote:
> All,
>
> I submit a new patch for on-demand class loading and parsing. All
> codes are put in VM side, and the mapping info is automatically
> produced.
>
> Pls see https://issues.apache.org/jira/browse/HARMONY-6039
>
> Comments are welcome.
>
> Thx, Wenlong
> Managed Runtime Technology Center, Intel
>
> On Wed, Jan 7, 2009 at 12:08 PM, Wenlong Li <we...@gmail.com> wrote:
>> All,
>> At this moment, I move all updates in classlib to VM side such that
>> there is no modularity issue. Next step is to produce the mapping
>> between module and library efficiently and accurately.
>>
>> Comments are welcome.
>>
>> Thx, Wenlong
>> Managed Runtime Technology Center, Intel
>>
>> On Tue, Jan 6, 2009 at 11:08 PM, Wenlong Li <we...@gmail.com> wrote:
>>> Thx :)
>>>
>>> On Tue, Jan 6, 2009 at 10:35 PM, Alexei Fedotov
>>> <al...@gmail.com> wrote:
>>>> Sure.
>>>>
>>>> 1. If you dig into SetClasspathFromString, you will see that it starts from
>>>> splitting the given classpath into pieces. You already know the new piece
>>>> you add and may skip splitting step.
>>>>
>>>> 2. If I understand you code correctly, the case "pdest >
>>>> (*it).second->bytes" might be a subject of a negative assertion. Adding this
>>>> assrtion would speed up bug discovery.
>>>>
>>>> Thanks.
>>>>
>>>>
>>>> On Tue, Jan 6, 2009 at 5:09 AM, Wenlong Li <we...@gmail.com> wrote:
>>>>
>>>>> Yes, Xiao-Feng's understanding is correct. The patch loads and parses
>>>>> modules on demand. If no class in swing.jar is not requested, then
>>>>> this module will not be loaded.
>>>>>
>>>>> btw, Alexei, you said "SetClasspathFromString" and "pdest >
>>>>> (*it).second->bytes" are not efficient. Can you share more comments on
>>>>> them? I just reused some code in Harmony, and didn't optimize them
>>>>> further.
>>>>>
>>>>> Thx, Wenlong
>>>>> Managed Runtime Technology Center, Intel
>>>>>
>>>>> On Fri, Dec 26, 2008 at 5:16 PM, Xiao-Feng Li <xi...@gmail.com>
>>>>> wrote:
>>>>> > On Fri, Dec 26, 2008 at 5:08 PM, Alexei Fedotov
>>>>> > <al...@gmail.com> wrote:
>>>>> >> Xiao Feng,
>>>>> >> Thank you for explaining.
>>>>> >>
>>>>> >> I get more minor comments on more commented code, ineffective
>>>>> >> SetClasspathFromString usage, non-covered unexpected case when pdest >
>>>>> >> (*it).second->bytes. One major comment on crossing vm module boundary
>>>>> >> still remains. But now I'm happy with the design.
>>>>> >
>>>>> > Alexei, yes, I agree with your comments. These parts should be
>>>>> > improved. (Still, this is my personal opinion. :) Let's wait Wenlong
>>>>> > speaking.)
>>>>> >
>>>>> > Thanks,
>>>>> > xiaofeng
>>>>> >
>>>>> >> Sorry for being slow.
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> On Fri, Dec 26, 2008 at 11:40 AM, Xiao-Feng Li <xi...@gmail.com>
>>>>> wrote:
>>>>> >>> On Fri, Dec 26, 2008 at 4:03 PM, Alexei Fedotov
>>>>> >>> <al...@gmail.com> wrote:
>>>>> >>>> Xiao-Feng,
>>>>> >>>> Continuing with the server example could you please give me a hint
>>>>> where
>>>>> >>>> decision to load swing.jar or not is taken in the patch? My initial
>>>>> >>>> perception was that the list of what to load was hardcoded and was not
>>>>> >>>> constructed dynamically depending on application.
>>>>> >>>
>>>>> >>> Alexei, here is the patch code I found:
>>>>> >>>
>>>>> >>> line 245:
>>>>> >>> + // Find which jar exports this package
>>>>> >>> + if (pkgName != NULL) {
>>>>> >>> + char *boot_class_path =
>>>>> >>> env->JavaProperties()->get(VM_BOOT_CLASS_DIR);
>>>>> >>> + char *pendingClassPath = NULL;
>>>>> >>> + apr_pool_t *tmp_pool;
>>>>> >>> + apr_pool_create(&tmp_pool, NULL);
>>>>> >>> + while (it != env->pending_jar_set.end()) {
>>>>> >>> + pdest = strstr( (*it).second->bytes, pkgName );
>>>>> >>> + if (pdest != NULL) {
>>>>> >>> + pendingClassPath =
>>>>> >>> (char*)STD_MALLOC(strlen(boot_class_path)
>>>>> >>> + +
>>>>> strlen((*it).first->bytes) + 1);
>>>>> >>> + strcpy(pendingClassPath, boot_class_path);
>>>>> >>> + strcat(pendingClassPath, (*it).first->bytes);
>>>>> >>> + // Open this found jar, and read all classes
>>>>> >>> contained in this jar
>>>>> >>> + SetClasspathFromString(pendingClassPath,
>>>>> tmp_pool);
>>>>> >>> + // Erase the found jar from pending jar list
>>>>> >>> as it has been parsed
>>>>> >>> + env->pending_jar_set.erase(it++);
>>>>> >>> + STD_FREE(pendingClassPath);
>>>>> >>> + } else {
>>>>> >>>
>>>>> >>> It checks if a JAR has the requested package, then loads it if yes. I
>>>>> >>> am not sure if this is what you were asking. (Btw, this is only my
>>>>> >>> understanding of his patch. I am not speaking for Wenlong.)
>>>>> >>>
>>>>> >>> Thanks,
>>>>> >>> xiaofeng
>>>>> >>>
>>>>> >>>> Thanks.
>>>>> >>>>
>>>>> >>>>
>>>>> >>>> On Fri, Dec 26, 2008 at 4:14 AM, Xiao-Feng Li <xi...@gmail.com>
>>>>> wrote:
>>>>> >>>>
>>>>> >>>>> On Fri, Dec 26, 2008 at 12:49 AM, Alexei Fedotov
>>>>> >>>>> <al...@gmail.com> wrote:
>>>>> >>>>> > Aleksey,
>>>>> >>>>> > I like your conclusion.
>>>>> >>>>> >
>>>>> >>>>> > Wenlong,
>>>>> >>>>> > I'm trying to understand the real life value of the "abstract"
>>>>> startup
>>>>> >>>>> > time metric you've suggested. Does Harmony with your patch load
>>>>> >>>>> > swing.jar for a server application? Do I understand that loading
>>>>> >>>>> > happens, though it happens later compared to VM without your patch?
>>>>> I
>>>>> >>>>> > believe that the proper design of delayed loading should answer
>>>>> "no"
>>>>> >>>>> > to this question.
>>>>> >>>>>
>>>>> >>>>> I checked the patch, and I found the answer is indeed "No" as you
>>>>> expected.
>>>>> >>>>>
>>>>> >>>>> Thanks,
>>>>> >>>>> xiaofeng
>>>>> >>>>>
>>>>> >>>>> > In other words, I appreciate if you describe which real use cases
>>>>> are
>>>>> >>>>> > improved by this patch.
>>>>> >>>>> > Thanks!
>>>>> >>>>>
>>>>> >>>>> > On Thu, Dec 25, 2008 at 7:29 PM, Aleksey Shipilev
>>>>> >>>>> > <al...@gmail.com> wrote:
>>>>> >>>>> >> Ok, here's the catch.
>>>>> >>>>> >>
>>>>> >>>>> >> bootclasspath.properties is the SortedSet<JARfile>, which
>>>>> enumerates
>>>>> >>>>> >> the JARs available for bootclassloader. The set of such the JARs
>>>>> is
>>>>> >>>>> >> really stable because modular decomposition of classlib is stable.
>>>>> >>>>> >> That's why nobody bothers with automatic generation of it: it only
>>>>> >>>>> >> should be updated when new JAR file arrives.
>>>>> >>>>> >> Modulelibrarymapping.properties is different on this point, it's
>>>>> the
>>>>> >>>>> >> Map<PackageName,JARfile>, which should be updated each time the
>>>>> new
>>>>> >>>>> >> *package* is introduced. I'm not talking about java.* packages
>>>>> >>>>> >> (they're standardized), rather about org.apache.harmony.*.
>>>>> >>>>> >>
>>>>> >>>>> >> Automatic generation of this property file gives two advantages:
>>>>> >>>>> >> 1. Error-prone. Prevent yourself from hand-messing with mapping
>>>>> and
>>>>> >>>>> >> getting spurious ClassNotFoundException. BTW, what's the behaviour
>>>>> in
>>>>> >>>>> >> case the mapping is wrong?
>>>>> >>>>> >> 2. "Researchful". There're lot of guys around who enjoys the
>>>>> >>>>> >> modularity of Harmony classlib and eventually they might want to
>>>>> split
>>>>> >>>>> >> the packages even deeper, into smaller pieces. Then automatic
>>>>> >>>>> >> generation would enable them to quickly roll-in and experiment
>>>>> with
>>>>> >>>>> >> different package layouts and their impact on performance. They
>>>>> could
>>>>> >>>>> >> use ordinary bootclasspath.properties, but your feature wouldn't
>>>>> be
>>>>> >>>>> >> used by them then ;)
>>>>> >>>>> >>
>>>>> >>>>> >> That's merely a housekeeping procedure. I believe that anything
>>>>> which
>>>>> >>>>> >> could be done more than once, eventually would be done more than
>>>>> once.
>>>>> >>>>> >> Hence it should be automated. You say that the file was generated
>>>>> from
>>>>> >>>>> >> manifests of JARs, so is it hard to just tie the same tool into
>>>>> DRLVM
>>>>> >>>>> >> build process?
>>>>> >>>>> >>
>>>>> >>>>> >> As for DRLVM-specific, my opinion that this is because the patch:
>>>>> >>>>> >> a. breaks the compatibility of classlib (you change
>>>>> >>>>> >> bootclasspath.properties, right?) with other VMs.
>>>>> >>>>> >> b. treated in DRLVM classloader only.
>>>>> >>>>> >>
>>>>> >>>>> >> Of course eventually this feature might be used by others, but IMO
>>>>> we
>>>>> >>>>> >> should be careful about other guys who use the same classlib. I'd
>>>>> >>>>> >> rather wait for some incubation on DRLVM side first.
>>>>> >>>>> >>
>>>>> >>>>> >> Thanks,
>>>>> >>>>> >> Aleksey.
>>>>> >>>>> >>
>>>>> >>>>> >> On Thu, Dec 25, 2008 at 6:18 PM, Wenlong Li <we...@gmail.com>
>>>>> wrote:
>>>>> >>>>> >>> I see. In fact, my file doesn't need track change at the class
>>>>> >>>>> >>> granularity. Instead, it only needs know package info provided in
>>>>> the
>>>>> >>>>> >>> manifest file. When class is added to a library, do we need
>>>>> change
>>>>> >>>>> >>> the manifest as well?
>>>>> >>>>> >>>
>>>>> >>>>> >>> btw, I guess there is a mis-understanding: my
>>>>> modulelibrarymapping
>>>>> >>>>> >>> file only records package info provided by manfiest in each
>>>>> module. It
>>>>> >>>>> >>> doesn't relate to each class.
>>>>> >>>>> >>>
>>>>> >>>>> >>> thx,
>>>>> >>>>> >>> Wenlong
>>>>> >>>>> >>>
>>>>> >>>>> >>> On Thu, Dec 25, 2008 at 10:55 PM, Pavel Pervov <
>>>>> pmcfirst@gmail.com>
>>>>> >>>>> wrote:
>>>>> >>>>> >>>> Classes are added to class library from time to time. I'm not
>>>>> sure how
>>>>> >>>>> >>>> it can be possible to track these changes manually.
>>>>> >>>>> >>>>
>>>>> >>>>> >>>> WBR,
>>>>> >>>>> >>>> Pavel.
>>>>> >>>>> >>>>
>>>>> >>>>> >>>>
>>>>> >>>>> >>>> On Thu, Dec 25, 2008 at 5:09 PM, Wenlong Li <we...@gmail.com>
>>>>> >>>>> wrote:
>>>>> >>>>> >>>>> Sorry, one more question: bootclasspath.properties is classlib
>>>>> >>>>> >>>>> specific file, why we could not make a vm specific file
>>>>> manually?
>>>>> >>>>> Just
>>>>> >>>>> >>>>> curious to know the possible reason. :)
>>>>> >>>>> >>>>>
>>>>> >>>>> >>>>> thx,
>>>>> >>>>> >>>>> Wenlong
>>>>> >>>>> >>>>>
>>>>> >>>>> >>>>> On Thu, Dec 25, 2008 at 10:00 PM, Pavel Pervov <
>>>>> pmcfirst@gmail.com>
>>>>> >>>>> wrote:
>>>>> >>>>> >>>>>> If this would be VM-side automatically produced configuration
>>>>> >>>>> file...
>>>>> >>>>> >>>>>>
>>>>> >>>>> >>>>>> WBR,
>>>>> >>>>> >>>>>> Pavel.
>>>>> >>>>> >>>>>>
>>>>> >>>>> >>>>>> On Thu, Dec 25, 2008 at 4:58 PM, Wenlong Li <
>>>>> wenlong@gmail.com>
>>>>> >>>>> wrote:
>>>>> >>>>> >>>>>>> btw, because adding new module is rare case, manually
>>>>> modifying the
>>>>> >>>>> >>>>>>> bootclasspath.properties is not an issue?
>>>>> >>>>> >>>>>>>
>>>>> >>>>> >>>>>>> If so, can I conclude adding another property file with same
>>>>> update
>>>>> >>>>> >>>>>>> frequency as bootclasspath would be fine as well?
>>>>> >>>>> >>>>>>>
>>>>> >>>>> >>>>>>> Pls kindly correct me if my understanding is wrong.
>>>>> >>>>> >>>>>>>
>>>>> >>>>> >>>>>>> On Thu, Dec 25, 2008 at 9:05 PM, Pavel Pervov <
>>>>> pmcfirst@gmail.com>
>>>>> >>>>> wrote:
>>>>> >>>>> >>>>>>>> Wenlong,
>>>>> >>>>> >>>>>>>>
>>>>> >>>>> >>>>>>>> Note, that bootclasspath.properties is only changed on
>>>>> adding new
>>>>> >>>>> >>>>>>>> module. This is pretty rare occasion, I believe.
>>>>> >>>>> >>>>>>>>
>>>>> >>>>> >>>>>>>> WBR,
>>>>> >>>>> >>>>>>>> Pavel.
>>>>> >>>>> >>>>>>>>
>>>>> >>>>> >>>>>>>> On Thu, Dec 25, 2008 at 3:48 PM, Wenlong Li <
>>>>> wenlong@gmail.com>
>>>>> >>>>> wrote:
>>>>> >>>>> >>>>>>>>> Thx for your advice. Alexey.
>>>>> >>>>> >>>>>>>>>
>>>>> >>>>> >>>>>>>>> Here I have one question: do you know how the
>>>>> >>>>> bootclasspath.properties
>>>>> >>>>> >>>>>>>>> is maintained, manually or automatically?
>>>>> >>>>> >>>>>>>>>
>>>>> >>>>> >>>>>>>>> Another comment is I would like to treat the patch as DRLVM
>>>>> >>>>> specific
>>>>> >>>>> >>>>>>>>> optimization, e.g., it targets for improving VM creation
>>>>> time. So
>>>>> >>>>> that
>>>>> >>>>> >>>>>>>>> is possible to move all updates to DRLVM part to eliminate
>>>>> >>>>> potential
>>>>> >>>>> >>>>>>>>> modularity and compatibility problem.
>>>>> >>>>> >>>>>>>>>
>>>>> >>>>> >>>>>>>>> thx,
>>>>> >>>>> >>>>>>>>> Wenlong
>>>>> >>>>> >>>>>>>>>
>>>>> >>>>> >>>>>>>>> On Thu, Dec 25, 2008 at 5:32 PM, Aleksey Shipilev
>>>>> >>>>> >>>>>>>>> <al...@gmail.com> wrote:
>>>>> >>>>> >>>>>>>>>> Hi, Wenlong.
>>>>> >>>>> >>>>>>>>>>
>>>>> >>>>> >>>>>>>>>> On Thu, Dec 25, 2008 at 11:49 AM, Wenlong Li <
>>>>> wenlong@gmail.com>
>>>>> >>>>> wrote:
>>>>> >>>>> >>>>>>>>>>> btw, Alexey, Let's go back to discuss whether there is a
>>>>> need
>>>>> >>>>> to
>>>>> >>>>> >>>>>>>>>>> include this feature in Harmony, given 17% performance
>>>>> gain in
>>>>> >>>>> Linux
>>>>> >>>>> >>>>>>>>>>> when using your methodology. For windows test, I will
>>>>> double
>>>>> >>>>> check the
>>>>> >>>>> >>>>>>>>>>> backgroud process as you pointed out.
>>>>> >>>>> >>>>>>>>>>
>>>>> >>>>> >>>>>>>>>> My opinion was already expressed after I had finished the
>>>>> tests
>>>>> >>>>> from
>>>>> >>>>> >>>>>>>>>> my side: the boost can be achieved in specific conditions,
>>>>> so
>>>>> >>>>> whether
>>>>> >>>>> >>>>>>>>>> it's worth including into Harmony really depends on how
>>>>> much
>>>>> >>>>> mess the
>>>>> >>>>> >>>>>>>>>> patch would introduce besides the "performance boost".
>>>>> From what
>>>>> >>>>> I
>>>>> >>>>> >>>>>>>>>> see, the patch obliges the maintainer to maintain the
>>>>> correct
>>>>> >>>>> mapping
>>>>> >>>>> >>>>>>>>>> between jars and Java packages. This new feature is also
>>>>> spread
>>>>> >>>>> >>>>>>>>>> between Classlib and VM, but it seems like DRLVM specific.
>>>>> In
>>>>> >>>>> this
>>>>> >>>>> >>>>>>>>>> case I would rather stay without the patch.
>>>>> >>>>> >>>>>>>>>>
>>>>> >>>>> >>>>>>>>>> Personally (if I'll be committer) I would accept the patch
>>>>> with
>>>>> >>>>> two
>>>>> >>>>> >>>>>>>>>> serious modifications:
>>>>> >>>>> >>>>>>>>>> 1. Stay within DRLVM, do not introduce this feature into
>>>>> >>>>> Classlib,
>>>>> >>>>> >>>>>>>>>> get the thing tested and evolved on DRLVM side. Otherwise
>>>>> it
>>>>> >>>>> might
>>>>> >>>>> >>>>>>>>>> break the compatibility with other VMs.
>>>>> >>>>> >>>>>>>>>> 2. Make the mapping generated automatically (during build
>>>>> >>>>> process?)
>>>>> >>>>> >>>>>>>>>> to free the burden for maintainers.
>>>>> >>>>> >>>>>>>>>>
>>>>> >>>>> >>>>>>>>>> Thanks,
>>>>> >>>>> >>>>>>>>>> Aleksey.
>>>>> >>>>> >>>>>>>>>>
>>>>> >>>>> >>>>>>>>>
>>>>> >>>>> >>>>>>>>
>>>>> >>>>> >>>>>>>
>>>>> >>>>> >>>>>>
>>>>> >>>>> >>>>>
>>>>> >>>>> >>>>
>>>>> >>>>> >>>
>>>>> >>>>> >>
>>>>> >>>>> >
>>>>> >>>>> >
>>>>> >>>>> >
>>>>> >>>>> > --
>>>>> >>>>> > С уважением,
>>>>> >>>>> > Алексей Федотов,
>>>>> >>>>> > ЗАО «Телеком Экспресс»
>>>>> >>>>> >
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>> --
>>>>> >>>>> http://xiao-feng.blogspot.com
>>>>> >>>>>
>>>>> >>>>
>>>>> >>>>
>>>>> >>>>
>>>>> >>>> --
>>>>> >>>> С уважением,
>>>>> >>>> Алексей Федотов,
>>>>> >>>> ЗАО «Телеком Экспресс»
>>>>> >>>>
>>>>> >>>
>>>>> >>>
>>>>> >>>
>>>>> >>> --
>>>>> >>> http://xiao-feng.blogspot.com
>>>>> >>>
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> --
>>>>> >> С уважением,
>>>>> >> Алексей Федотов,
>>>>> >> ЗАО «Телеком Экспресс»
>>>>> >>
>>>>> >
>>>>> >
>>>>> >
>>>>> > --
>>>>> > http://xiao-feng.blogspot.com
>>>>> >
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> С уважением,
>>>> Алексей Федотов,
>>>> ЗАО «Телеком Экспресс»
>>>>
>>>
>>
>
Re: [VM] On-demand class library parsing is ready to commit
Posted by Pavel Pervov <pm...@gmail.com>.
Wenlong,
Please, see my comments in the JIRA.
WBR,
Pavel.
On Wed, Jan 14, 2009 at 12:33 PM, Wenlong Li <we...@gmail.com> wrote:
> All,
>
> I submit a new patch for on-demand class loading and parsing. All
> codes are put in VM side, and the mapping info is automatically
> produced.
>
> Pls see https://issues.apache.org/jira/browse/HARMONY-6039
>
> Comments are welcome.
>
> Thx, Wenlong
> Managed Runtime Technology Center, Intel
>
> On Wed, Jan 7, 2009 at 12:08 PM, Wenlong Li <we...@gmail.com> wrote:
>> All,
>> At this moment, I move all updates in classlib to VM side such that
>> there is no modularity issue. Next step is to produce the mapping
>> between module and library efficiently and accurately.
>>
>> Comments are welcome.
>>
>> Thx, Wenlong
>> Managed Runtime Technology Center, Intel
>>
>> On Tue, Jan 6, 2009 at 11:08 PM, Wenlong Li <we...@gmail.com> wrote:
>>> Thx :)
>>>
>>> On Tue, Jan 6, 2009 at 10:35 PM, Alexei Fedotov
>>> <al...@gmail.com> wrote:
>>>> Sure.
>>>>
>>>> 1. If you dig into SetClasspathFromString, you will see that it starts from
>>>> splitting the given classpath into pieces. You already know the new piece
>>>> you add and may skip splitting step.
>>>>
>>>> 2. If I understand you code correctly, the case "pdest >
>>>> (*it).second->bytes" might be a subject of a negative assertion. Adding this
>>>> assrtion would speed up bug discovery.
>>>>
>>>> Thanks.
>>>>
>>>>
>>>> On Tue, Jan 6, 2009 at 5:09 AM, Wenlong Li <we...@gmail.com> wrote:
>>>>
>>>>> Yes, Xiao-Feng's understanding is correct. The patch loads and parses
>>>>> modules on demand. If no class in swing.jar is not requested, then
>>>>> this module will not be loaded.
>>>>>
>>>>> btw, Alexei, you said "SetClasspathFromString" and "pdest >
>>>>> (*it).second->bytes" are not efficient. Can you share more comments on
>>>>> them? I just reused some code in Harmony, and didn't optimize them
>>>>> further.
>>>>>
>>>>> Thx, Wenlong
>>>>> Managed Runtime Technology Center, Intel
>>>>>
>>>>> On Fri, Dec 26, 2008 at 5:16 PM, Xiao-Feng Li <xi...@gmail.com>
>>>>> wrote:
>>>>> > On Fri, Dec 26, 2008 at 5:08 PM, Alexei Fedotov
>>>>> > <al...@gmail.com> wrote:
>>>>> >> Xiao Feng,
>>>>> >> Thank you for explaining.
>>>>> >>
>>>>> >> I get more minor comments on more commented code, ineffective
>>>>> >> SetClasspathFromString usage, non-covered unexpected case when pdest >
>>>>> >> (*it).second->bytes. One major comment on crossing vm module boundary
>>>>> >> still remains. But now I'm happy with the design.
>>>>> >
>>>>> > Alexei, yes, I agree with your comments. These parts should be
>>>>> > improved. (Still, this is my personal opinion. :) Let's wait Wenlong
>>>>> > speaking.)
>>>>> >
>>>>> > Thanks,
>>>>> > xiaofeng
>>>>> >
>>>>> >> Sorry for being slow.
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> On Fri, Dec 26, 2008 at 11:40 AM, Xiao-Feng Li <xi...@gmail.com>
>>>>> wrote:
>>>>> >>> On Fri, Dec 26, 2008 at 4:03 PM, Alexei Fedotov
>>>>> >>> <al...@gmail.com> wrote:
>>>>> >>>> Xiao-Feng,
>>>>> >>>> Continuing with the server example could you please give me a hint
>>>>> where
>>>>> >>>> decision to load swing.jar or not is taken in the patch? My initial
>>>>> >>>> perception was that the list of what to load was hardcoded and was not
>>>>> >>>> constructed dynamically depending on application.
>>>>> >>>
>>>>> >>> Alexei, here is the patch code I found:
>>>>> >>>
>>>>> >>> line 245:
>>>>> >>> + // Find which jar exports this package
>>>>> >>> + if (pkgName != NULL) {
>>>>> >>> + char *boot_class_path =
>>>>> >>> env->JavaProperties()->get(VM_BOOT_CLASS_DIR);
>>>>> >>> + char *pendingClassPath = NULL;
>>>>> >>> + apr_pool_t *tmp_pool;
>>>>> >>> + apr_pool_create(&tmp_pool, NULL);
>>>>> >>> + while (it != env->pending_jar_set.end()) {
>>>>> >>> + pdest = strstr( (*it).second->bytes, pkgName );
>>>>> >>> + if (pdest != NULL) {
>>>>> >>> + pendingClassPath =
>>>>> >>> (char*)STD_MALLOC(strlen(boot_class_path)
>>>>> >>> + +
>>>>> strlen((*it).first->bytes) + 1);
>>>>> >>> + strcpy(pendingClassPath, boot_class_path);
>>>>> >>> + strcat(pendingClassPath, (*it).first->bytes);
>>>>> >>> + // Open this found jar, and read all classes
>>>>> >>> contained in this jar
>>>>> >>> + SetClasspathFromString(pendingClassPath,
>>>>> tmp_pool);
>>>>> >>> + // Erase the found jar from pending jar list
>>>>> >>> as it has been parsed
>>>>> >>> + env->pending_jar_set.erase(it++);
>>>>> >>> + STD_FREE(pendingClassPath);
>>>>> >>> + } else {
>>>>> >>>
>>>>> >>> It checks if a JAR has the requested package, then loads it if yes. I
>>>>> >>> am not sure if this is what you were asking. (Btw, this is only my
>>>>> >>> understanding of his patch. I am not speaking for Wenlong.)
>>>>> >>>
>>>>> >>> Thanks,
>>>>> >>> xiaofeng
>>>>> >>>
>>>>> >>>> Thanks.
>>>>> >>>>
>>>>> >>>>
>>>>> >>>> On Fri, Dec 26, 2008 at 4:14 AM, Xiao-Feng Li <xi...@gmail.com>
>>>>> wrote:
>>>>> >>>>
>>>>> >>>>> On Fri, Dec 26, 2008 at 12:49 AM, Alexei Fedotov
>>>>> >>>>> <al...@gmail.com> wrote:
>>>>> >>>>> > Aleksey,
>>>>> >>>>> > I like your conclusion.
>>>>> >>>>> >
>>>>> >>>>> > Wenlong,
>>>>> >>>>> > I'm trying to understand the real life value of the "abstract"
>>>>> startup
>>>>> >>>>> > time metric you've suggested. Does Harmony with your patch load
>>>>> >>>>> > swing.jar for a server application? Do I understand that loading
>>>>> >>>>> > happens, though it happens later compared to VM without your patch?
>>>>> I
>>>>> >>>>> > believe that the proper design of delayed loading should answer
>>>>> "no"
>>>>> >>>>> > to this question.
>>>>> >>>>>
>>>>> >>>>> I checked the patch, and I found the answer is indeed "No" as you
>>>>> expected.
>>>>> >>>>>
>>>>> >>>>> Thanks,
>>>>> >>>>> xiaofeng
>>>>> >>>>>
>>>>> >>>>> > In other words, I appreciate if you describe which real use cases
>>>>> are
>>>>> >>>>> > improved by this patch.
>>>>> >>>>> > Thanks!
>>>>> >>>>>
>>>>> >>>>> > On Thu, Dec 25, 2008 at 7:29 PM, Aleksey Shipilev
>>>>> >>>>> > <al...@gmail.com> wrote:
>>>>> >>>>> >> Ok, here's the catch.
>>>>> >>>>> >>
>>>>> >>>>> >> bootclasspath.properties is the SortedSet<JARfile>, which
>>>>> enumerates
>>>>> >>>>> >> the JARs available for bootclassloader. The set of such the JARs
>>>>> is
>>>>> >>>>> >> really stable because modular decomposition of classlib is stable.
>>>>> >>>>> >> That's why nobody bothers with automatic generation of it: it only
>>>>> >>>>> >> should be updated when new JAR file arrives.
>>>>> >>>>> >> Modulelibrarymapping.properties is different on this point, it's
>>>>> the
>>>>> >>>>> >> Map<PackageName,JARfile>, which should be updated each time the
>>>>> new
>>>>> >>>>> >> *package* is introduced. I'm not talking about java.* packages
>>>>> >>>>> >> (they're standardized), rather about org.apache.harmony.*.
>>>>> >>>>> >>
>>>>> >>>>> >> Automatic generation of this property file gives two advantages:
>>>>> >>>>> >> 1. Error-prone. Prevent yourself from hand-messing with mapping
>>>>> and
>>>>> >>>>> >> getting spurious ClassNotFoundException. BTW, what's the behaviour
>>>>> in
>>>>> >>>>> >> case the mapping is wrong?
>>>>> >>>>> >> 2. "Researchful". There're lot of guys around who enjoys the
>>>>> >>>>> >> modularity of Harmony classlib and eventually they might want to
>>>>> split
>>>>> >>>>> >> the packages even deeper, into smaller pieces. Then automatic
>>>>> >>>>> >> generation would enable them to quickly roll-in and experiment
>>>>> with
>>>>> >>>>> >> different package layouts and their impact on performance. They
>>>>> could
>>>>> >>>>> >> use ordinary bootclasspath.properties, but your feature wouldn't
>>>>> be
>>>>> >>>>> >> used by them then ;)
>>>>> >>>>> >>
>>>>> >>>>> >> That's merely a housekeeping procedure. I believe that anything
>>>>> which
>>>>> >>>>> >> could be done more than once, eventually would be done more than
>>>>> once.
>>>>> >>>>> >> Hence it should be automated. You say that the file was generated
>>>>> from
>>>>> >>>>> >> manifests of JARs, so is it hard to just tie the same tool into
>>>>> DRLVM
>>>>> >>>>> >> build process?
>>>>> >>>>> >>
>>>>> >>>>> >> As for DRLVM-specific, my opinion that this is because the patch:
>>>>> >>>>> >> a. breaks the compatibility of classlib (you change
>>>>> >>>>> >> bootclasspath.properties, right?) with other VMs.
>>>>> >>>>> >> b. treated in DRLVM classloader only.
>>>>> >>>>> >>
>>>>> >>>>> >> Of course eventually this feature might be used by others, but IMO
>>>>> we
>>>>> >>>>> >> should be careful about other guys who use the same classlib. I'd
>>>>> >>>>> >> rather wait for some incubation on DRLVM side first.
>>>>> >>>>> >>
>>>>> >>>>> >> Thanks,
>>>>> >>>>> >> Aleksey.
>>>>> >>>>> >>
>>>>> >>>>> >> On Thu, Dec 25, 2008 at 6:18 PM, Wenlong Li <we...@gmail.com>
>>>>> wrote:
>>>>> >>>>> >>> I see. In fact, my file doesn't need track change at the class
>>>>> >>>>> >>> granularity. Instead, it only needs know package info provided in
>>>>> the
>>>>> >>>>> >>> manifest file. When class is added to a library, do we need
>>>>> change
>>>>> >>>>> >>> the manifest as well?
>>>>> >>>>> >>>
>>>>> >>>>> >>> btw, I guess there is a mis-understanding: my
>>>>> modulelibrarymapping
>>>>> >>>>> >>> file only records package info provided by manfiest in each
>>>>> module. It
>>>>> >>>>> >>> doesn't relate to each class.
>>>>> >>>>> >>>
>>>>> >>>>> >>> thx,
>>>>> >>>>> >>> Wenlong
>>>>> >>>>> >>>
>>>>> >>>>> >>> On Thu, Dec 25, 2008 at 10:55 PM, Pavel Pervov <
>>>>> pmcfirst@gmail.com>
>>>>> >>>>> wrote:
>>>>> >>>>> >>>> Classes are added to class library from time to time. I'm not
>>>>> sure how
>>>>> >>>>> >>>> it can be possible to track these changes manually.
>>>>> >>>>> >>>>
>>>>> >>>>> >>>> WBR,
>>>>> >>>>> >>>> Pavel.
>>>>> >>>>> >>>>
>>>>> >>>>> >>>>
>>>>> >>>>> >>>> On Thu, Dec 25, 2008 at 5:09 PM, Wenlong Li <we...@gmail.com>
>>>>> >>>>> wrote:
>>>>> >>>>> >>>>> Sorry, one more question: bootclasspath.properties is classlib
>>>>> >>>>> >>>>> specific file, why we could not make a vm specific file
>>>>> manually?
>>>>> >>>>> Just
>>>>> >>>>> >>>>> curious to know the possible reason. :)
>>>>> >>>>> >>>>>
>>>>> >>>>> >>>>> thx,
>>>>> >>>>> >>>>> Wenlong
>>>>> >>>>> >>>>>
>>>>> >>>>> >>>>> On Thu, Dec 25, 2008 at 10:00 PM, Pavel Pervov <
>>>>> pmcfirst@gmail.com>
>>>>> >>>>> wrote:
>>>>> >>>>> >>>>>> If this would be VM-side automatically produced configuration
>>>>> >>>>> file...
>>>>> >>>>> >>>>>>
>>>>> >>>>> >>>>>> WBR,
>>>>> >>>>> >>>>>> Pavel.
>>>>> >>>>> >>>>>>
>>>>> >>>>> >>>>>> On Thu, Dec 25, 2008 at 4:58 PM, Wenlong Li <
>>>>> wenlong@gmail.com>
>>>>> >>>>> wrote:
>>>>> >>>>> >>>>>>> btw, because adding new module is rare case, manually
>>>>> modifying the
>>>>> >>>>> >>>>>>> bootclasspath.properties is not an issue?
>>>>> >>>>> >>>>>>>
>>>>> >>>>> >>>>>>> If so, can I conclude adding another property file with same
>>>>> update
>>>>> >>>>> >>>>>>> frequency as bootclasspath would be fine as well?
>>>>> >>>>> >>>>>>>
>>>>> >>>>> >>>>>>> Pls kindly correct me if my understanding is wrong.
>>>>> >>>>> >>>>>>>
>>>>> >>>>> >>>>>>> On Thu, Dec 25, 2008 at 9:05 PM, Pavel Pervov <
>>>>> pmcfirst@gmail.com>
>>>>> >>>>> wrote:
>>>>> >>>>> >>>>>>>> Wenlong,
>>>>> >>>>> >>>>>>>>
>>>>> >>>>> >>>>>>>> Note, that bootclasspath.properties is only changed on
>>>>> adding new
>>>>> >>>>> >>>>>>>> module. This is pretty rare occasion, I believe.
>>>>> >>>>> >>>>>>>>
>>>>> >>>>> >>>>>>>> WBR,
>>>>> >>>>> >>>>>>>> Pavel.
>>>>> >>>>> >>>>>>>>
>>>>> >>>>> >>>>>>>> On Thu, Dec 25, 2008 at 3:48 PM, Wenlong Li <
>>>>> wenlong@gmail.com>
>>>>> >>>>> wrote:
>>>>> >>>>> >>>>>>>>> Thx for your advice. Alexey.
>>>>> >>>>> >>>>>>>>>
>>>>> >>>>> >>>>>>>>> Here I have one question: do you know how the
>>>>> >>>>> bootclasspath.properties
>>>>> >>>>> >>>>>>>>> is maintained, manually or automatically?
>>>>> >>>>> >>>>>>>>>
>>>>> >>>>> >>>>>>>>> Another comment is I would like to treat the patch as DRLVM
>>>>> >>>>> specific
>>>>> >>>>> >>>>>>>>> optimization, e.g., it targets for improving VM creation
>>>>> time. So
>>>>> >>>>> that
>>>>> >>>>> >>>>>>>>> is possible to move all updates to DRLVM part to eliminate
>>>>> >>>>> potential
>>>>> >>>>> >>>>>>>>> modularity and compatibility problem.
>>>>> >>>>> >>>>>>>>>
>>>>> >>>>> >>>>>>>>> thx,
>>>>> >>>>> >>>>>>>>> Wenlong
>>>>> >>>>> >>>>>>>>>
>>>>> >>>>> >>>>>>>>> On Thu, Dec 25, 2008 at 5:32 PM, Aleksey Shipilev
>>>>> >>>>> >>>>>>>>> <al...@gmail.com> wrote:
>>>>> >>>>> >>>>>>>>>> Hi, Wenlong.
>>>>> >>>>> >>>>>>>>>>
>>>>> >>>>> >>>>>>>>>> On Thu, Dec 25, 2008 at 11:49 AM, Wenlong Li <
>>>>> wenlong@gmail.com>
>>>>> >>>>> wrote:
>>>>> >>>>> >>>>>>>>>>> btw, Alexey, Let's go back to discuss whether there is a
>>>>> need
>>>>> >>>>> to
>>>>> >>>>> >>>>>>>>>>> include this feature in Harmony, given 17% performance
>>>>> gain in
>>>>> >>>>> Linux
>>>>> >>>>> >>>>>>>>>>> when using your methodology. For windows test, I will
>>>>> double
>>>>> >>>>> check the
>>>>> >>>>> >>>>>>>>>>> backgroud process as you pointed out.
>>>>> >>>>> >>>>>>>>>>
>>>>> >>>>> >>>>>>>>>> My opinion was already expressed after I had finished the
>>>>> tests
>>>>> >>>>> from
>>>>> >>>>> >>>>>>>>>> my side: the boost can be achieved in specific conditions,
>>>>> so
>>>>> >>>>> whether
>>>>> >>>>> >>>>>>>>>> it's worth including into Harmony really depends on how
>>>>> much
>>>>> >>>>> mess the
>>>>> >>>>> >>>>>>>>>> patch would introduce besides the "performance boost".
>>>>> From what
>>>>> >>>>> I
>>>>> >>>>> >>>>>>>>>> see, the patch obliges the maintainer to maintain the
>>>>> correct
>>>>> >>>>> mapping
>>>>> >>>>> >>>>>>>>>> between jars and Java packages. This new feature is also
>>>>> spread
>>>>> >>>>> >>>>>>>>>> between Classlib and VM, but it seems like DRLVM specific.
>>>>> In
>>>>> >>>>> this
>>>>> >>>>> >>>>>>>>>> case I would rather stay without the patch.
>>>>> >>>>> >>>>>>>>>>
>>>>> >>>>> >>>>>>>>>> Personally (if I'll be committer) I would accept the patch
>>>>> with
>>>>> >>>>> two
>>>>> >>>>> >>>>>>>>>> serious modifications:
>>>>> >>>>> >>>>>>>>>> 1. Stay within DRLVM, do not introduce this feature into
>>>>> >>>>> Classlib,
>>>>> >>>>> >>>>>>>>>> get the thing tested and evolved on DRLVM side. Otherwise
>>>>> it
>>>>> >>>>> might
>>>>> >>>>> >>>>>>>>>> break the compatibility with other VMs.
>>>>> >>>>> >>>>>>>>>> 2. Make the mapping generated automatically (during build
>>>>> >>>>> process?)
>>>>> >>>>> >>>>>>>>>> to free the burden for maintainers.
>>>>> >>>>> >>>>>>>>>>
>>>>> >>>>> >>>>>>>>>> Thanks,
>>>>> >>>>> >>>>>>>>>> Aleksey.
>>>>> >>>>> >>>>>>>>>>
>>>>> >>>>> >>>>>>>>>
>>>>> >>>>> >>>>>>>>
>>>>> >>>>> >>>>>>>
>>>>> >>>>> >>>>>>
>>>>> >>>>> >>>>>
>>>>> >>>>> >>>>
>>>>> >>>>> >>>
>>>>> >>>>> >>
>>>>> >>>>> >
>>>>> >>>>> >
>>>>> >>>>> >
>>>>> >>>>> > --
>>>>> >>>>> > С уважением,
>>>>> >>>>> > Алексей Федотов,
>>>>> >>>>> > ЗАО «Телеком Экспресс»
>>>>> >>>>> >
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>> --
>>>>> >>>>> http://xiao-feng.blogspot.com
>>>>> >>>>>
>>>>> >>>>
>>>>> >>>>
>>>>> >>>>
>>>>> >>>> --
>>>>> >>>> С уважением,
>>>>> >>>> Алексей Федотов,
>>>>> >>>> ЗАО «Телеком Экспресс»
>>>>> >>>>
>>>>> >>>
>>>>> >>>
>>>>> >>>
>>>>> >>> --
>>>>> >>> http://xiao-feng.blogspot.com
>>>>> >>>
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> --
>>>>> >> С уважением,
>>>>> >> Алексей Федотов,
>>>>> >> ЗАО «Телеком Экспресс»
>>>>> >>
>>>>> >
>>>>> >
>>>>> >
>>>>> > --
>>>>> > http://xiao-feng.blogspot.com
>>>>> >
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> С уважением,
>>>> Алексей Федотов,
>>>> ЗАО «Телеком Экспресс»
>>>>
>>>
>>
>
Re: [VM] On-demand class library parsing is ready to commit
Posted by Wenlong Li <we...@gmail.com>.
All,
I submit a new patch for on-demand class loading and parsing. All
codes are put in VM side, and the mapping info is automatically
produced.
Pls see https://issues.apache.org/jira/browse/HARMONY-6039
Comments are welcome.
Thx, Wenlong
Managed Runtime Technology Center, Intel
On Wed, Jan 7, 2009 at 12:08 PM, Wenlong Li <we...@gmail.com> wrote:
> All,
> At this moment, I move all updates in classlib to VM side such that
> there is no modularity issue. Next step is to produce the mapping
> between module and library efficiently and accurately.
>
> Comments are welcome.
>
> Thx, Wenlong
> Managed Runtime Technology Center, Intel
>
> On Tue, Jan 6, 2009 at 11:08 PM, Wenlong Li <we...@gmail.com> wrote:
>> Thx :)
>>
>> On Tue, Jan 6, 2009 at 10:35 PM, Alexei Fedotov
>> <al...@gmail.com> wrote:
>>> Sure.
>>>
>>> 1. If you dig into SetClasspathFromString, you will see that it starts from
>>> splitting the given classpath into pieces. You already know the new piece
>>> you add and may skip splitting step.
>>>
>>> 2. If I understand you code correctly, the case "pdest >
>>> (*it).second->bytes" might be a subject of a negative assertion. Adding this
>>> assrtion would speed up bug discovery.
>>>
>>> Thanks.
>>>
>>>
>>> On Tue, Jan 6, 2009 at 5:09 AM, Wenlong Li <we...@gmail.com> wrote:
>>>
>>>> Yes, Xiao-Feng's understanding is correct. The patch loads and parses
>>>> modules on demand. If no class in swing.jar is not requested, then
>>>> this module will not be loaded.
>>>>
>>>> btw, Alexei, you said "SetClasspathFromString" and "pdest >
>>>> (*it).second->bytes" are not efficient. Can you share more comments on
>>>> them? I just reused some code in Harmony, and didn't optimize them
>>>> further.
>>>>
>>>> Thx, Wenlong
>>>> Managed Runtime Technology Center, Intel
>>>>
>>>> On Fri, Dec 26, 2008 at 5:16 PM, Xiao-Feng Li <xi...@gmail.com>
>>>> wrote:
>>>> > On Fri, Dec 26, 2008 at 5:08 PM, Alexei Fedotov
>>>> > <al...@gmail.com> wrote:
>>>> >> Xiao Feng,
>>>> >> Thank you for explaining.
>>>> >>
>>>> >> I get more minor comments on more commented code, ineffective
>>>> >> SetClasspathFromString usage, non-covered unexpected case when pdest >
>>>> >> (*it).second->bytes. One major comment on crossing vm module boundary
>>>> >> still remains. But now I'm happy with the design.
>>>> >
>>>> > Alexei, yes, I agree with your comments. These parts should be
>>>> > improved. (Still, this is my personal opinion. :) Let's wait Wenlong
>>>> > speaking.)
>>>> >
>>>> > Thanks,
>>>> > xiaofeng
>>>> >
>>>> >> Sorry for being slow.
>>>> >>
>>>> >>
>>>> >>
>>>> >> On Fri, Dec 26, 2008 at 11:40 AM, Xiao-Feng Li <xi...@gmail.com>
>>>> wrote:
>>>> >>> On Fri, Dec 26, 2008 at 4:03 PM, Alexei Fedotov
>>>> >>> <al...@gmail.com> wrote:
>>>> >>>> Xiao-Feng,
>>>> >>>> Continuing with the server example could you please give me a hint
>>>> where
>>>> >>>> decision to load swing.jar or not is taken in the patch? My initial
>>>> >>>> perception was that the list of what to load was hardcoded and was not
>>>> >>>> constructed dynamically depending on application.
>>>> >>>
>>>> >>> Alexei, here is the patch code I found:
>>>> >>>
>>>> >>> line 245:
>>>> >>> + // Find which jar exports this package
>>>> >>> + if (pkgName != NULL) {
>>>> >>> + char *boot_class_path =
>>>> >>> env->JavaProperties()->get(VM_BOOT_CLASS_DIR);
>>>> >>> + char *pendingClassPath = NULL;
>>>> >>> + apr_pool_t *tmp_pool;
>>>> >>> + apr_pool_create(&tmp_pool, NULL);
>>>> >>> + while (it != env->pending_jar_set.end()) {
>>>> >>> + pdest = strstr( (*it).second->bytes, pkgName );
>>>> >>> + if (pdest != NULL) {
>>>> >>> + pendingClassPath =
>>>> >>> (char*)STD_MALLOC(strlen(boot_class_path)
>>>> >>> + +
>>>> strlen((*it).first->bytes) + 1);
>>>> >>> + strcpy(pendingClassPath, boot_class_path);
>>>> >>> + strcat(pendingClassPath, (*it).first->bytes);
>>>> >>> + // Open this found jar, and read all classes
>>>> >>> contained in this jar
>>>> >>> + SetClasspathFromString(pendingClassPath,
>>>> tmp_pool);
>>>> >>> + // Erase the found jar from pending jar list
>>>> >>> as it has been parsed
>>>> >>> + env->pending_jar_set.erase(it++);
>>>> >>> + STD_FREE(pendingClassPath);
>>>> >>> + } else {
>>>> >>>
>>>> >>> It checks if a JAR has the requested package, then loads it if yes. I
>>>> >>> am not sure if this is what you were asking. (Btw, this is only my
>>>> >>> understanding of his patch. I am not speaking for Wenlong.)
>>>> >>>
>>>> >>> Thanks,
>>>> >>> xiaofeng
>>>> >>>
>>>> >>>> Thanks.
>>>> >>>>
>>>> >>>>
>>>> >>>> On Fri, Dec 26, 2008 at 4:14 AM, Xiao-Feng Li <xi...@gmail.com>
>>>> wrote:
>>>> >>>>
>>>> >>>>> On Fri, Dec 26, 2008 at 12:49 AM, Alexei Fedotov
>>>> >>>>> <al...@gmail.com> wrote:
>>>> >>>>> > Aleksey,
>>>> >>>>> > I like your conclusion.
>>>> >>>>> >
>>>> >>>>> > Wenlong,
>>>> >>>>> > I'm trying to understand the real life value of the "abstract"
>>>> startup
>>>> >>>>> > time metric you've suggested. Does Harmony with your patch load
>>>> >>>>> > swing.jar for a server application? Do I understand that loading
>>>> >>>>> > happens, though it happens later compared to VM without your patch?
>>>> I
>>>> >>>>> > believe that the proper design of delayed loading should answer
>>>> "no"
>>>> >>>>> > to this question.
>>>> >>>>>
>>>> >>>>> I checked the patch, and I found the answer is indeed "No" as you
>>>> expected.
>>>> >>>>>
>>>> >>>>> Thanks,
>>>> >>>>> xiaofeng
>>>> >>>>>
>>>> >>>>> > In other words, I appreciate if you describe which real use cases
>>>> are
>>>> >>>>> > improved by this patch.
>>>> >>>>> > Thanks!
>>>> >>>>>
>>>> >>>>> > On Thu, Dec 25, 2008 at 7:29 PM, Aleksey Shipilev
>>>> >>>>> > <al...@gmail.com> wrote:
>>>> >>>>> >> Ok, here's the catch.
>>>> >>>>> >>
>>>> >>>>> >> bootclasspath.properties is the SortedSet<JARfile>, which
>>>> enumerates
>>>> >>>>> >> the JARs available for bootclassloader. The set of such the JARs
>>>> is
>>>> >>>>> >> really stable because modular decomposition of classlib is stable.
>>>> >>>>> >> That's why nobody bothers with automatic generation of it: it only
>>>> >>>>> >> should be updated when new JAR file arrives.
>>>> >>>>> >> Modulelibrarymapping.properties is different on this point, it's
>>>> the
>>>> >>>>> >> Map<PackageName,JARfile>, which should be updated each time the
>>>> new
>>>> >>>>> >> *package* is introduced. I'm not talking about java.* packages
>>>> >>>>> >> (they're standardized), rather about org.apache.harmony.*.
>>>> >>>>> >>
>>>> >>>>> >> Automatic generation of this property file gives two advantages:
>>>> >>>>> >> 1. Error-prone. Prevent yourself from hand-messing with mapping
>>>> and
>>>> >>>>> >> getting spurious ClassNotFoundException. BTW, what's the behaviour
>>>> in
>>>> >>>>> >> case the mapping is wrong?
>>>> >>>>> >> 2. "Researchful". There're lot of guys around who enjoys the
>>>> >>>>> >> modularity of Harmony classlib and eventually they might want to
>>>> split
>>>> >>>>> >> the packages even deeper, into smaller pieces. Then automatic
>>>> >>>>> >> generation would enable them to quickly roll-in and experiment
>>>> with
>>>> >>>>> >> different package layouts and their impact on performance. They
>>>> could
>>>> >>>>> >> use ordinary bootclasspath.properties, but your feature wouldn't
>>>> be
>>>> >>>>> >> used by them then ;)
>>>> >>>>> >>
>>>> >>>>> >> That's merely a housekeeping procedure. I believe that anything
>>>> which
>>>> >>>>> >> could be done more than once, eventually would be done more than
>>>> once.
>>>> >>>>> >> Hence it should be automated. You say that the file was generated
>>>> from
>>>> >>>>> >> manifests of JARs, so is it hard to just tie the same tool into
>>>> DRLVM
>>>> >>>>> >> build process?
>>>> >>>>> >>
>>>> >>>>> >> As for DRLVM-specific, my opinion that this is because the patch:
>>>> >>>>> >> a. breaks the compatibility of classlib (you change
>>>> >>>>> >> bootclasspath.properties, right?) with other VMs.
>>>> >>>>> >> b. treated in DRLVM classloader only.
>>>> >>>>> >>
>>>> >>>>> >> Of course eventually this feature might be used by others, but IMO
>>>> we
>>>> >>>>> >> should be careful about other guys who use the same classlib. I'd
>>>> >>>>> >> rather wait for some incubation on DRLVM side first.
>>>> >>>>> >>
>>>> >>>>> >> Thanks,
>>>> >>>>> >> Aleksey.
>>>> >>>>> >>
>>>> >>>>> >> On Thu, Dec 25, 2008 at 6:18 PM, Wenlong Li <we...@gmail.com>
>>>> wrote:
>>>> >>>>> >>> I see. In fact, my file doesn't need track change at the class
>>>> >>>>> >>> granularity. Instead, it only needs know package info provided in
>>>> the
>>>> >>>>> >>> manifest file. When class is added to a library, do we need
>>>> change
>>>> >>>>> >>> the manifest as well?
>>>> >>>>> >>>
>>>> >>>>> >>> btw, I guess there is a mis-understanding: my
>>>> modulelibrarymapping
>>>> >>>>> >>> file only records package info provided by manfiest in each
>>>> module. It
>>>> >>>>> >>> doesn't relate to each class.
>>>> >>>>> >>>
>>>> >>>>> >>> thx,
>>>> >>>>> >>> Wenlong
>>>> >>>>> >>>
>>>> >>>>> >>> On Thu, Dec 25, 2008 at 10:55 PM, Pavel Pervov <
>>>> pmcfirst@gmail.com>
>>>> >>>>> wrote:
>>>> >>>>> >>>> Classes are added to class library from time to time. I'm not
>>>> sure how
>>>> >>>>> >>>> it can be possible to track these changes manually.
>>>> >>>>> >>>>
>>>> >>>>> >>>> WBR,
>>>> >>>>> >>>> Pavel.
>>>> >>>>> >>>>
>>>> >>>>> >>>>
>>>> >>>>> >>>> On Thu, Dec 25, 2008 at 5:09 PM, Wenlong Li <we...@gmail.com>
>>>> >>>>> wrote:
>>>> >>>>> >>>>> Sorry, one more question: bootclasspath.properties is classlib
>>>> >>>>> >>>>> specific file, why we could not make a vm specific file
>>>> manually?
>>>> >>>>> Just
>>>> >>>>> >>>>> curious to know the possible reason. :)
>>>> >>>>> >>>>>
>>>> >>>>> >>>>> thx,
>>>> >>>>> >>>>> Wenlong
>>>> >>>>> >>>>>
>>>> >>>>> >>>>> On Thu, Dec 25, 2008 at 10:00 PM, Pavel Pervov <
>>>> pmcfirst@gmail.com>
>>>> >>>>> wrote:
>>>> >>>>> >>>>>> If this would be VM-side automatically produced configuration
>>>> >>>>> file...
>>>> >>>>> >>>>>>
>>>> >>>>> >>>>>> WBR,
>>>> >>>>> >>>>>> Pavel.
>>>> >>>>> >>>>>>
>>>> >>>>> >>>>>> On Thu, Dec 25, 2008 at 4:58 PM, Wenlong Li <
>>>> wenlong@gmail.com>
>>>> >>>>> wrote:
>>>> >>>>> >>>>>>> btw, because adding new module is rare case, manually
>>>> modifying the
>>>> >>>>> >>>>>>> bootclasspath.properties is not an issue?
>>>> >>>>> >>>>>>>
>>>> >>>>> >>>>>>> If so, can I conclude adding another property file with same
>>>> update
>>>> >>>>> >>>>>>> frequency as bootclasspath would be fine as well?
>>>> >>>>> >>>>>>>
>>>> >>>>> >>>>>>> Pls kindly correct me if my understanding is wrong.
>>>> >>>>> >>>>>>>
>>>> >>>>> >>>>>>> On Thu, Dec 25, 2008 at 9:05 PM, Pavel Pervov <
>>>> pmcfirst@gmail.com>
>>>> >>>>> wrote:
>>>> >>>>> >>>>>>>> Wenlong,
>>>> >>>>> >>>>>>>>
>>>> >>>>> >>>>>>>> Note, that bootclasspath.properties is only changed on
>>>> adding new
>>>> >>>>> >>>>>>>> module. This is pretty rare occasion, I believe.
>>>> >>>>> >>>>>>>>
>>>> >>>>> >>>>>>>> WBR,
>>>> >>>>> >>>>>>>> Pavel.
>>>> >>>>> >>>>>>>>
>>>> >>>>> >>>>>>>> On Thu, Dec 25, 2008 at 3:48 PM, Wenlong Li <
>>>> wenlong@gmail.com>
>>>> >>>>> wrote:
>>>> >>>>> >>>>>>>>> Thx for your advice. Alexey.
>>>> >>>>> >>>>>>>>>
>>>> >>>>> >>>>>>>>> Here I have one question: do you know how the
>>>> >>>>> bootclasspath.properties
>>>> >>>>> >>>>>>>>> is maintained, manually or automatically?
>>>> >>>>> >>>>>>>>>
>>>> >>>>> >>>>>>>>> Another comment is I would like to treat the patch as DRLVM
>>>> >>>>> specific
>>>> >>>>> >>>>>>>>> optimization, e.g., it targets for improving VM creation
>>>> time. So
>>>> >>>>> that
>>>> >>>>> >>>>>>>>> is possible to move all updates to DRLVM part to eliminate
>>>> >>>>> potential
>>>> >>>>> >>>>>>>>> modularity and compatibility problem.
>>>> >>>>> >>>>>>>>>
>>>> >>>>> >>>>>>>>> thx,
>>>> >>>>> >>>>>>>>> Wenlong
>>>> >>>>> >>>>>>>>>
>>>> >>>>> >>>>>>>>> On Thu, Dec 25, 2008 at 5:32 PM, Aleksey Shipilev
>>>> >>>>> >>>>>>>>> <al...@gmail.com> wrote:
>>>> >>>>> >>>>>>>>>> Hi, Wenlong.
>>>> >>>>> >>>>>>>>>>
>>>> >>>>> >>>>>>>>>> On Thu, Dec 25, 2008 at 11:49 AM, Wenlong Li <
>>>> wenlong@gmail.com>
>>>> >>>>> wrote:
>>>> >>>>> >>>>>>>>>>> btw, Alexey, Let's go back to discuss whether there is a
>>>> need
>>>> >>>>> to
>>>> >>>>> >>>>>>>>>>> include this feature in Harmony, given 17% performance
>>>> gain in
>>>> >>>>> Linux
>>>> >>>>> >>>>>>>>>>> when using your methodology. For windows test, I will
>>>> double
>>>> >>>>> check the
>>>> >>>>> >>>>>>>>>>> backgroud process as you pointed out.
>>>> >>>>> >>>>>>>>>>
>>>> >>>>> >>>>>>>>>> My opinion was already expressed after I had finished the
>>>> tests
>>>> >>>>> from
>>>> >>>>> >>>>>>>>>> my side: the boost can be achieved in specific conditions,
>>>> so
>>>> >>>>> whether
>>>> >>>>> >>>>>>>>>> it's worth including into Harmony really depends on how
>>>> much
>>>> >>>>> mess the
>>>> >>>>> >>>>>>>>>> patch would introduce besides the "performance boost".
>>>> From what
>>>> >>>>> I
>>>> >>>>> >>>>>>>>>> see, the patch obliges the maintainer to maintain the
>>>> correct
>>>> >>>>> mapping
>>>> >>>>> >>>>>>>>>> between jars and Java packages. This new feature is also
>>>> spread
>>>> >>>>> >>>>>>>>>> between Classlib and VM, but it seems like DRLVM specific.
>>>> In
>>>> >>>>> this
>>>> >>>>> >>>>>>>>>> case I would rather stay without the patch.
>>>> >>>>> >>>>>>>>>>
>>>> >>>>> >>>>>>>>>> Personally (if I'll be committer) I would accept the patch
>>>> with
>>>> >>>>> two
>>>> >>>>> >>>>>>>>>> serious modifications:
>>>> >>>>> >>>>>>>>>> 1. Stay within DRLVM, do not introduce this feature into
>>>> >>>>> Classlib,
>>>> >>>>> >>>>>>>>>> get the thing tested and evolved on DRLVM side. Otherwise
>>>> it
>>>> >>>>> might
>>>> >>>>> >>>>>>>>>> break the compatibility with other VMs.
>>>> >>>>> >>>>>>>>>> 2. Make the mapping generated automatically (during build
>>>> >>>>> process?)
>>>> >>>>> >>>>>>>>>> to free the burden for maintainers.
>>>> >>>>> >>>>>>>>>>
>>>> >>>>> >>>>>>>>>> Thanks,
>>>> >>>>> >>>>>>>>>> Aleksey.
>>>> >>>>> >>>>>>>>>>
>>>> >>>>> >>>>>>>>>
>>>> >>>>> >>>>>>>>
>>>> >>>>> >>>>>>>
>>>> >>>>> >>>>>>
>>>> >>>>> >>>>>
>>>> >>>>> >>>>
>>>> >>>>> >>>
>>>> >>>>> >>
>>>> >>>>> >
>>>> >>>>> >
>>>> >>>>> >
>>>> >>>>> > --
>>>> >>>>> > С уважением,
>>>> >>>>> > Алексей Федотов,
>>>> >>>>> > ЗАО «Телеком Экспресс»
>>>> >>>>> >
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> --
>>>> >>>>> http://xiao-feng.blogspot.com
>>>> >>>>>
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>> --
>>>> >>>> С уважением,
>>>> >>>> Алексей Федотов,
>>>> >>>> ЗАО «Телеком Экспресс»
>>>> >>>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> --
>>>> >>> http://xiao-feng.blogspot.com
>>>> >>>
>>>> >>
>>>> >>
>>>> >>
>>>> >> --
>>>> >> С уважением,
>>>> >> Алексей Федотов,
>>>> >> ЗАО «Телеком Экспресс»
>>>> >>
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > http://xiao-feng.blogspot.com
>>>> >
>>>>
>>>
>>>
>>>
>>> --
>>> С уважением,
>>> Алексей Федотов,
>>> ЗАО «Телеком Экспресс»
>>>
>>
>
Re: [VM] On-demand class library parsing is ready to commit
Posted by Wenlong Li <we...@gmail.com>.
All,
At this moment, I move all updates in classlib to VM side such that
there is no modularity issue. Next step is to produce the mapping
between module and library efficiently and accurately.
Comments are welcome.
Thx, Wenlong
Managed Runtime Technology Center, Intel
On Tue, Jan 6, 2009 at 11:08 PM, Wenlong Li <we...@gmail.com> wrote:
> Thx :)
>
> On Tue, Jan 6, 2009 at 10:35 PM, Alexei Fedotov
> <al...@gmail.com> wrote:
>> Sure.
>>
>> 1. If you dig into SetClasspathFromString, you will see that it starts from
>> splitting the given classpath into pieces. You already know the new piece
>> you add and may skip splitting step.
>>
>> 2. If I understand you code correctly, the case "pdest >
>> (*it).second->bytes" might be a subject of a negative assertion. Adding this
>> assrtion would speed up bug discovery.
>>
>> Thanks.
>>
>>
>> On Tue, Jan 6, 2009 at 5:09 AM, Wenlong Li <we...@gmail.com> wrote:
>>
>>> Yes, Xiao-Feng's understanding is correct. The patch loads and parses
>>> modules on demand. If no class in swing.jar is not requested, then
>>> this module will not be loaded.
>>>
>>> btw, Alexei, you said "SetClasspathFromString" and "pdest >
>>> (*it).second->bytes" are not efficient. Can you share more comments on
>>> them? I just reused some code in Harmony, and didn't optimize them
>>> further.
>>>
>>> Thx, Wenlong
>>> Managed Runtime Technology Center, Intel
>>>
>>> On Fri, Dec 26, 2008 at 5:16 PM, Xiao-Feng Li <xi...@gmail.com>
>>> wrote:
>>> > On Fri, Dec 26, 2008 at 5:08 PM, Alexei Fedotov
>>> > <al...@gmail.com> wrote:
>>> >> Xiao Feng,
>>> >> Thank you for explaining.
>>> >>
>>> >> I get more minor comments on more commented code, ineffective
>>> >> SetClasspathFromString usage, non-covered unexpected case when pdest >
>>> >> (*it).second->bytes. One major comment on crossing vm module boundary
>>> >> still remains. But now I'm happy with the design.
>>> >
>>> > Alexei, yes, I agree with your comments. These parts should be
>>> > improved. (Still, this is my personal opinion. :) Let's wait Wenlong
>>> > speaking.)
>>> >
>>> > Thanks,
>>> > xiaofeng
>>> >
>>> >> Sorry for being slow.
>>> >>
>>> >>
>>> >>
>>> >> On Fri, Dec 26, 2008 at 11:40 AM, Xiao-Feng Li <xi...@gmail.com>
>>> wrote:
>>> >>> On Fri, Dec 26, 2008 at 4:03 PM, Alexei Fedotov
>>> >>> <al...@gmail.com> wrote:
>>> >>>> Xiao-Feng,
>>> >>>> Continuing with the server example could you please give me a hint
>>> where
>>> >>>> decision to load swing.jar or not is taken in the patch? My initial
>>> >>>> perception was that the list of what to load was hardcoded and was not
>>> >>>> constructed dynamically depending on application.
>>> >>>
>>> >>> Alexei, here is the patch code I found:
>>> >>>
>>> >>> line 245:
>>> >>> + // Find which jar exports this package
>>> >>> + if (pkgName != NULL) {
>>> >>> + char *boot_class_path =
>>> >>> env->JavaProperties()->get(VM_BOOT_CLASS_DIR);
>>> >>> + char *pendingClassPath = NULL;
>>> >>> + apr_pool_t *tmp_pool;
>>> >>> + apr_pool_create(&tmp_pool, NULL);
>>> >>> + while (it != env->pending_jar_set.end()) {
>>> >>> + pdest = strstr( (*it).second->bytes, pkgName );
>>> >>> + if (pdest != NULL) {
>>> >>> + pendingClassPath =
>>> >>> (char*)STD_MALLOC(strlen(boot_class_path)
>>> >>> + +
>>> strlen((*it).first->bytes) + 1);
>>> >>> + strcpy(pendingClassPath, boot_class_path);
>>> >>> + strcat(pendingClassPath, (*it).first->bytes);
>>> >>> + // Open this found jar, and read all classes
>>> >>> contained in this jar
>>> >>> + SetClasspathFromString(pendingClassPath,
>>> tmp_pool);
>>> >>> + // Erase the found jar from pending jar list
>>> >>> as it has been parsed
>>> >>> + env->pending_jar_set.erase(it++);
>>> >>> + STD_FREE(pendingClassPath);
>>> >>> + } else {
>>> >>>
>>> >>> It checks if a JAR has the requested package, then loads it if yes. I
>>> >>> am not sure if this is what you were asking. (Btw, this is only my
>>> >>> understanding of his patch. I am not speaking for Wenlong.)
>>> >>>
>>> >>> Thanks,
>>> >>> xiaofeng
>>> >>>
>>> >>>> Thanks.
>>> >>>>
>>> >>>>
>>> >>>> On Fri, Dec 26, 2008 at 4:14 AM, Xiao-Feng Li <xi...@gmail.com>
>>> wrote:
>>> >>>>
>>> >>>>> On Fri, Dec 26, 2008 at 12:49 AM, Alexei Fedotov
>>> >>>>> <al...@gmail.com> wrote:
>>> >>>>> > Aleksey,
>>> >>>>> > I like your conclusion.
>>> >>>>> >
>>> >>>>> > Wenlong,
>>> >>>>> > I'm trying to understand the real life value of the "abstract"
>>> startup
>>> >>>>> > time metric you've suggested. Does Harmony with your patch load
>>> >>>>> > swing.jar for a server application? Do I understand that loading
>>> >>>>> > happens, though it happens later compared to VM without your patch?
>>> I
>>> >>>>> > believe that the proper design of delayed loading should answer
>>> "no"
>>> >>>>> > to this question.
>>> >>>>>
>>> >>>>> I checked the patch, and I found the answer is indeed "No" as you
>>> expected.
>>> >>>>>
>>> >>>>> Thanks,
>>> >>>>> xiaofeng
>>> >>>>>
>>> >>>>> > In other words, I appreciate if you describe which real use cases
>>> are
>>> >>>>> > improved by this patch.
>>> >>>>> > Thanks!
>>> >>>>>
>>> >>>>> > On Thu, Dec 25, 2008 at 7:29 PM, Aleksey Shipilev
>>> >>>>> > <al...@gmail.com> wrote:
>>> >>>>> >> Ok, here's the catch.
>>> >>>>> >>
>>> >>>>> >> bootclasspath.properties is the SortedSet<JARfile>, which
>>> enumerates
>>> >>>>> >> the JARs available for bootclassloader. The set of such the JARs
>>> is
>>> >>>>> >> really stable because modular decomposition of classlib is stable.
>>> >>>>> >> That's why nobody bothers with automatic generation of it: it only
>>> >>>>> >> should be updated when new JAR file arrives.
>>> >>>>> >> Modulelibrarymapping.properties is different on this point, it's
>>> the
>>> >>>>> >> Map<PackageName,JARfile>, which should be updated each time the
>>> new
>>> >>>>> >> *package* is introduced. I'm not talking about java.* packages
>>> >>>>> >> (they're standardized), rather about org.apache.harmony.*.
>>> >>>>> >>
>>> >>>>> >> Automatic generation of this property file gives two advantages:
>>> >>>>> >> 1. Error-prone. Prevent yourself from hand-messing with mapping
>>> and
>>> >>>>> >> getting spurious ClassNotFoundException. BTW, what's the behaviour
>>> in
>>> >>>>> >> case the mapping is wrong?
>>> >>>>> >> 2. "Researchful". There're lot of guys around who enjoys the
>>> >>>>> >> modularity of Harmony classlib and eventually they might want to
>>> split
>>> >>>>> >> the packages even deeper, into smaller pieces. Then automatic
>>> >>>>> >> generation would enable them to quickly roll-in and experiment
>>> with
>>> >>>>> >> different package layouts and their impact on performance. They
>>> could
>>> >>>>> >> use ordinary bootclasspath.properties, but your feature wouldn't
>>> be
>>> >>>>> >> used by them then ;)
>>> >>>>> >>
>>> >>>>> >> That's merely a housekeeping procedure. I believe that anything
>>> which
>>> >>>>> >> could be done more than once, eventually would be done more than
>>> once.
>>> >>>>> >> Hence it should be automated. You say that the file was generated
>>> from
>>> >>>>> >> manifests of JARs, so is it hard to just tie the same tool into
>>> DRLVM
>>> >>>>> >> build process?
>>> >>>>> >>
>>> >>>>> >> As for DRLVM-specific, my opinion that this is because the patch:
>>> >>>>> >> a. breaks the compatibility of classlib (you change
>>> >>>>> >> bootclasspath.properties, right?) with other VMs.
>>> >>>>> >> b. treated in DRLVM classloader only.
>>> >>>>> >>
>>> >>>>> >> Of course eventually this feature might be used by others, but IMO
>>> we
>>> >>>>> >> should be careful about other guys who use the same classlib. I'd
>>> >>>>> >> rather wait for some incubation on DRLVM side first.
>>> >>>>> >>
>>> >>>>> >> Thanks,
>>> >>>>> >> Aleksey.
>>> >>>>> >>
>>> >>>>> >> On Thu, Dec 25, 2008 at 6:18 PM, Wenlong Li <we...@gmail.com>
>>> wrote:
>>> >>>>> >>> I see. In fact, my file doesn't need track change at the class
>>> >>>>> >>> granularity. Instead, it only needs know package info provided in
>>> the
>>> >>>>> >>> manifest file. When class is added to a library, do we need
>>> change
>>> >>>>> >>> the manifest as well?
>>> >>>>> >>>
>>> >>>>> >>> btw, I guess there is a mis-understanding: my
>>> modulelibrarymapping
>>> >>>>> >>> file only records package info provided by manfiest in each
>>> module. It
>>> >>>>> >>> doesn't relate to each class.
>>> >>>>> >>>
>>> >>>>> >>> thx,
>>> >>>>> >>> Wenlong
>>> >>>>> >>>
>>> >>>>> >>> On Thu, Dec 25, 2008 at 10:55 PM, Pavel Pervov <
>>> pmcfirst@gmail.com>
>>> >>>>> wrote:
>>> >>>>> >>>> Classes are added to class library from time to time. I'm not
>>> sure how
>>> >>>>> >>>> it can be possible to track these changes manually.
>>> >>>>> >>>>
>>> >>>>> >>>> WBR,
>>> >>>>> >>>> Pavel.
>>> >>>>> >>>>
>>> >>>>> >>>>
>>> >>>>> >>>> On Thu, Dec 25, 2008 at 5:09 PM, Wenlong Li <we...@gmail.com>
>>> >>>>> wrote:
>>> >>>>> >>>>> Sorry, one more question: bootclasspath.properties is classlib
>>> >>>>> >>>>> specific file, why we could not make a vm specific file
>>> manually?
>>> >>>>> Just
>>> >>>>> >>>>> curious to know the possible reason. :)
>>> >>>>> >>>>>
>>> >>>>> >>>>> thx,
>>> >>>>> >>>>> Wenlong
>>> >>>>> >>>>>
>>> >>>>> >>>>> On Thu, Dec 25, 2008 at 10:00 PM, Pavel Pervov <
>>> pmcfirst@gmail.com>
>>> >>>>> wrote:
>>> >>>>> >>>>>> If this would be VM-side automatically produced configuration
>>> >>>>> file...
>>> >>>>> >>>>>>
>>> >>>>> >>>>>> WBR,
>>> >>>>> >>>>>> Pavel.
>>> >>>>> >>>>>>
>>> >>>>> >>>>>> On Thu, Dec 25, 2008 at 4:58 PM, Wenlong Li <
>>> wenlong@gmail.com>
>>> >>>>> wrote:
>>> >>>>> >>>>>>> btw, because adding new module is rare case, manually
>>> modifying the
>>> >>>>> >>>>>>> bootclasspath.properties is not an issue?
>>> >>>>> >>>>>>>
>>> >>>>> >>>>>>> If so, can I conclude adding another property file with same
>>> update
>>> >>>>> >>>>>>> frequency as bootclasspath would be fine as well?
>>> >>>>> >>>>>>>
>>> >>>>> >>>>>>> Pls kindly correct me if my understanding is wrong.
>>> >>>>> >>>>>>>
>>> >>>>> >>>>>>> On Thu, Dec 25, 2008 at 9:05 PM, Pavel Pervov <
>>> pmcfirst@gmail.com>
>>> >>>>> wrote:
>>> >>>>> >>>>>>>> Wenlong,
>>> >>>>> >>>>>>>>
>>> >>>>> >>>>>>>> Note, that bootclasspath.properties is only changed on
>>> adding new
>>> >>>>> >>>>>>>> module. This is pretty rare occasion, I believe.
>>> >>>>> >>>>>>>>
>>> >>>>> >>>>>>>> WBR,
>>> >>>>> >>>>>>>> Pavel.
>>> >>>>> >>>>>>>>
>>> >>>>> >>>>>>>> On Thu, Dec 25, 2008 at 3:48 PM, Wenlong Li <
>>> wenlong@gmail.com>
>>> >>>>> wrote:
>>> >>>>> >>>>>>>>> Thx for your advice. Alexey.
>>> >>>>> >>>>>>>>>
>>> >>>>> >>>>>>>>> Here I have one question: do you know how the
>>> >>>>> bootclasspath.properties
>>> >>>>> >>>>>>>>> is maintained, manually or automatically?
>>> >>>>> >>>>>>>>>
>>> >>>>> >>>>>>>>> Another comment is I would like to treat the patch as DRLVM
>>> >>>>> specific
>>> >>>>> >>>>>>>>> optimization, e.g., it targets for improving VM creation
>>> time. So
>>> >>>>> that
>>> >>>>> >>>>>>>>> is possible to move all updates to DRLVM part to eliminate
>>> >>>>> potential
>>> >>>>> >>>>>>>>> modularity and compatibility problem.
>>> >>>>> >>>>>>>>>
>>> >>>>> >>>>>>>>> thx,
>>> >>>>> >>>>>>>>> Wenlong
>>> >>>>> >>>>>>>>>
>>> >>>>> >>>>>>>>> On Thu, Dec 25, 2008 at 5:32 PM, Aleksey Shipilev
>>> >>>>> >>>>>>>>> <al...@gmail.com> wrote:
>>> >>>>> >>>>>>>>>> Hi, Wenlong.
>>> >>>>> >>>>>>>>>>
>>> >>>>> >>>>>>>>>> On Thu, Dec 25, 2008 at 11:49 AM, Wenlong Li <
>>> wenlong@gmail.com>
>>> >>>>> wrote:
>>> >>>>> >>>>>>>>>>> btw, Alexey, Let's go back to discuss whether there is a
>>> need
>>> >>>>> to
>>> >>>>> >>>>>>>>>>> include this feature in Harmony, given 17% performance
>>> gain in
>>> >>>>> Linux
>>> >>>>> >>>>>>>>>>> when using your methodology. For windows test, I will
>>> double
>>> >>>>> check the
>>> >>>>> >>>>>>>>>>> backgroud process as you pointed out.
>>> >>>>> >>>>>>>>>>
>>> >>>>> >>>>>>>>>> My opinion was already expressed after I had finished the
>>> tests
>>> >>>>> from
>>> >>>>> >>>>>>>>>> my side: the boost can be achieved in specific conditions,
>>> so
>>> >>>>> whether
>>> >>>>> >>>>>>>>>> it's worth including into Harmony really depends on how
>>> much
>>> >>>>> mess the
>>> >>>>> >>>>>>>>>> patch would introduce besides the "performance boost".
>>> From what
>>> >>>>> I
>>> >>>>> >>>>>>>>>> see, the patch obliges the maintainer to maintain the
>>> correct
>>> >>>>> mapping
>>> >>>>> >>>>>>>>>> between jars and Java packages. This new feature is also
>>> spread
>>> >>>>> >>>>>>>>>> between Classlib and VM, but it seems like DRLVM specific.
>>> In
>>> >>>>> this
>>> >>>>> >>>>>>>>>> case I would rather stay without the patch.
>>> >>>>> >>>>>>>>>>
>>> >>>>> >>>>>>>>>> Personally (if I'll be committer) I would accept the patch
>>> with
>>> >>>>> two
>>> >>>>> >>>>>>>>>> serious modifications:
>>> >>>>> >>>>>>>>>> 1. Stay within DRLVM, do not introduce this feature into
>>> >>>>> Classlib,
>>> >>>>> >>>>>>>>>> get the thing tested and evolved on DRLVM side. Otherwise
>>> it
>>> >>>>> might
>>> >>>>> >>>>>>>>>> break the compatibility with other VMs.
>>> >>>>> >>>>>>>>>> 2. Make the mapping generated automatically (during build
>>> >>>>> process?)
>>> >>>>> >>>>>>>>>> to free the burden for maintainers.
>>> >>>>> >>>>>>>>>>
>>> >>>>> >>>>>>>>>> Thanks,
>>> >>>>> >>>>>>>>>> Aleksey.
>>> >>>>> >>>>>>>>>>
>>> >>>>> >>>>>>>>>
>>> >>>>> >>>>>>>>
>>> >>>>> >>>>>>>
>>> >>>>> >>>>>>
>>> >>>>> >>>>>
>>> >>>>> >>>>
>>> >>>>> >>>
>>> >>>>> >>
>>> >>>>> >
>>> >>>>> >
>>> >>>>> >
>>> >>>>> > --
>>> >>>>> > С уважением,
>>> >>>>> > Алексей Федотов,
>>> >>>>> > ЗАО «Телеком Экспресс»
>>> >>>>> >
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> --
>>> >>>>> http://xiao-feng.blogspot.com
>>> >>>>>
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> --
>>> >>>> С уважением,
>>> >>>> Алексей Федотов,
>>> >>>> ЗАО «Телеком Экспресс»
>>> >>>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> --
>>> >>> http://xiao-feng.blogspot.com
>>> >>>
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> С уважением,
>>> >> Алексей Федотов,
>>> >> ЗАО «Телеком Экспресс»
>>> >>
>>> >
>>> >
>>> >
>>> > --
>>> > http://xiao-feng.blogspot.com
>>> >
>>>
>>
>>
>>
>> --
>> С уважением,
>> Алексей Федотов,
>> ЗАО «Телеком Экспресс»
>>
>
Re: [VM] On-demand class library parsing is ready to commit
Posted by Wenlong Li <we...@gmail.com>.
Thx :)
On Tue, Jan 6, 2009 at 10:35 PM, Alexei Fedotov
<al...@gmail.com> wrote:
> Sure.
>
> 1. If you dig into SetClasspathFromString, you will see that it starts from
> splitting the given classpath into pieces. You already know the new piece
> you add and may skip splitting step.
>
> 2. If I understand you code correctly, the case "pdest >
> (*it).second->bytes" might be a subject of a negative assertion. Adding this
> assrtion would speed up bug discovery.
>
> Thanks.
>
>
> On Tue, Jan 6, 2009 at 5:09 AM, Wenlong Li <we...@gmail.com> wrote:
>
>> Yes, Xiao-Feng's understanding is correct. The patch loads and parses
>> modules on demand. If no class in swing.jar is not requested, then
>> this module will not be loaded.
>>
>> btw, Alexei, you said "SetClasspathFromString" and "pdest >
>> (*it).second->bytes" are not efficient. Can you share more comments on
>> them? I just reused some code in Harmony, and didn't optimize them
>> further.
>>
>> Thx, Wenlong
>> Managed Runtime Technology Center, Intel
>>
>> On Fri, Dec 26, 2008 at 5:16 PM, Xiao-Feng Li <xi...@gmail.com>
>> wrote:
>> > On Fri, Dec 26, 2008 at 5:08 PM, Alexei Fedotov
>> > <al...@gmail.com> wrote:
>> >> Xiao Feng,
>> >> Thank you for explaining.
>> >>
>> >> I get more minor comments on more commented code, ineffective
>> >> SetClasspathFromString usage, non-covered unexpected case when pdest >
>> >> (*it).second->bytes. One major comment on crossing vm module boundary
>> >> still remains. But now I'm happy with the design.
>> >
>> > Alexei, yes, I agree with your comments. These parts should be
>> > improved. (Still, this is my personal opinion. :) Let's wait Wenlong
>> > speaking.)
>> >
>> > Thanks,
>> > xiaofeng
>> >
>> >> Sorry for being slow.
>> >>
>> >>
>> >>
>> >> On Fri, Dec 26, 2008 at 11:40 AM, Xiao-Feng Li <xi...@gmail.com>
>> wrote:
>> >>> On Fri, Dec 26, 2008 at 4:03 PM, Alexei Fedotov
>> >>> <al...@gmail.com> wrote:
>> >>>> Xiao-Feng,
>> >>>> Continuing with the server example could you please give me a hint
>> where
>> >>>> decision to load swing.jar or not is taken in the patch? My initial
>> >>>> perception was that the list of what to load was hardcoded and was not
>> >>>> constructed dynamically depending on application.
>> >>>
>> >>> Alexei, here is the patch code I found:
>> >>>
>> >>> line 245:
>> >>> + // Find which jar exports this package
>> >>> + if (pkgName != NULL) {
>> >>> + char *boot_class_path =
>> >>> env->JavaProperties()->get(VM_BOOT_CLASS_DIR);
>> >>> + char *pendingClassPath = NULL;
>> >>> + apr_pool_t *tmp_pool;
>> >>> + apr_pool_create(&tmp_pool, NULL);
>> >>> + while (it != env->pending_jar_set.end()) {
>> >>> + pdest = strstr( (*it).second->bytes, pkgName );
>> >>> + if (pdest != NULL) {
>> >>> + pendingClassPath =
>> >>> (char*)STD_MALLOC(strlen(boot_class_path)
>> >>> + +
>> strlen((*it).first->bytes) + 1);
>> >>> + strcpy(pendingClassPath, boot_class_path);
>> >>> + strcat(pendingClassPath, (*it).first->bytes);
>> >>> + // Open this found jar, and read all classes
>> >>> contained in this jar
>> >>> + SetClasspathFromString(pendingClassPath,
>> tmp_pool);
>> >>> + // Erase the found jar from pending jar list
>> >>> as it has been parsed
>> >>> + env->pending_jar_set.erase(it++);
>> >>> + STD_FREE(pendingClassPath);
>> >>> + } else {
>> >>>
>> >>> It checks if a JAR has the requested package, then loads it if yes. I
>> >>> am not sure if this is what you were asking. (Btw, this is only my
>> >>> understanding of his patch. I am not speaking for Wenlong.)
>> >>>
>> >>> Thanks,
>> >>> xiaofeng
>> >>>
>> >>>> Thanks.
>> >>>>
>> >>>>
>> >>>> On Fri, Dec 26, 2008 at 4:14 AM, Xiao-Feng Li <xi...@gmail.com>
>> wrote:
>> >>>>
>> >>>>> On Fri, Dec 26, 2008 at 12:49 AM, Alexei Fedotov
>> >>>>> <al...@gmail.com> wrote:
>> >>>>> > Aleksey,
>> >>>>> > I like your conclusion.
>> >>>>> >
>> >>>>> > Wenlong,
>> >>>>> > I'm trying to understand the real life value of the "abstract"
>> startup
>> >>>>> > time metric you've suggested. Does Harmony with your patch load
>> >>>>> > swing.jar for a server application? Do I understand that loading
>> >>>>> > happens, though it happens later compared to VM without your patch?
>> I
>> >>>>> > believe that the proper design of delayed loading should answer
>> "no"
>> >>>>> > to this question.
>> >>>>>
>> >>>>> I checked the patch, and I found the answer is indeed "No" as you
>> expected.
>> >>>>>
>> >>>>> Thanks,
>> >>>>> xiaofeng
>> >>>>>
>> >>>>> > In other words, I appreciate if you describe which real use cases
>> are
>> >>>>> > improved by this patch.
>> >>>>> > Thanks!
>> >>>>>
>> >>>>> > On Thu, Dec 25, 2008 at 7:29 PM, Aleksey Shipilev
>> >>>>> > <al...@gmail.com> wrote:
>> >>>>> >> Ok, here's the catch.
>> >>>>> >>
>> >>>>> >> bootclasspath.properties is the SortedSet<JARfile>, which
>> enumerates
>> >>>>> >> the JARs available for bootclassloader. The set of such the JARs
>> is
>> >>>>> >> really stable because modular decomposition of classlib is stable.
>> >>>>> >> That's why nobody bothers with automatic generation of it: it only
>> >>>>> >> should be updated when new JAR file arrives.
>> >>>>> >> Modulelibrarymapping.properties is different on this point, it's
>> the
>> >>>>> >> Map<PackageName,JARfile>, which should be updated each time the
>> new
>> >>>>> >> *package* is introduced. I'm not talking about java.* packages
>> >>>>> >> (they're standardized), rather about org.apache.harmony.*.
>> >>>>> >>
>> >>>>> >> Automatic generation of this property file gives two advantages:
>> >>>>> >> 1. Error-prone. Prevent yourself from hand-messing with mapping
>> and
>> >>>>> >> getting spurious ClassNotFoundException. BTW, what's the behaviour
>> in
>> >>>>> >> case the mapping is wrong?
>> >>>>> >> 2. "Researchful". There're lot of guys around who enjoys the
>> >>>>> >> modularity of Harmony classlib and eventually they might want to
>> split
>> >>>>> >> the packages even deeper, into smaller pieces. Then automatic
>> >>>>> >> generation would enable them to quickly roll-in and experiment
>> with
>> >>>>> >> different package layouts and their impact on performance. They
>> could
>> >>>>> >> use ordinary bootclasspath.properties, but your feature wouldn't
>> be
>> >>>>> >> used by them then ;)
>> >>>>> >>
>> >>>>> >> That's merely a housekeeping procedure. I believe that anything
>> which
>> >>>>> >> could be done more than once, eventually would be done more than
>> once.
>> >>>>> >> Hence it should be automated. You say that the file was generated
>> from
>> >>>>> >> manifests of JARs, so is it hard to just tie the same tool into
>> DRLVM
>> >>>>> >> build process?
>> >>>>> >>
>> >>>>> >> As for DRLVM-specific, my opinion that this is because the patch:
>> >>>>> >> a. breaks the compatibility of classlib (you change
>> >>>>> >> bootclasspath.properties, right?) with other VMs.
>> >>>>> >> b. treated in DRLVM classloader only.
>> >>>>> >>
>> >>>>> >> Of course eventually this feature might be used by others, but IMO
>> we
>> >>>>> >> should be careful about other guys who use the same classlib. I'd
>> >>>>> >> rather wait for some incubation on DRLVM side first.
>> >>>>> >>
>> >>>>> >> Thanks,
>> >>>>> >> Aleksey.
>> >>>>> >>
>> >>>>> >> On Thu, Dec 25, 2008 at 6:18 PM, Wenlong Li <we...@gmail.com>
>> wrote:
>> >>>>> >>> I see. In fact, my file doesn't need track change at the class
>> >>>>> >>> granularity. Instead, it only needs know package info provided in
>> the
>> >>>>> >>> manifest file. When class is added to a library, do we need
>> change
>> >>>>> >>> the manifest as well?
>> >>>>> >>>
>> >>>>> >>> btw, I guess there is a mis-understanding: my
>> modulelibrarymapping
>> >>>>> >>> file only records package info provided by manfiest in each
>> module. It
>> >>>>> >>> doesn't relate to each class.
>> >>>>> >>>
>> >>>>> >>> thx,
>> >>>>> >>> Wenlong
>> >>>>> >>>
>> >>>>> >>> On Thu, Dec 25, 2008 at 10:55 PM, Pavel Pervov <
>> pmcfirst@gmail.com>
>> >>>>> wrote:
>> >>>>> >>>> Classes are added to class library from time to time. I'm not
>> sure how
>> >>>>> >>>> it can be possible to track these changes manually.
>> >>>>> >>>>
>> >>>>> >>>> WBR,
>> >>>>> >>>> Pavel.
>> >>>>> >>>>
>> >>>>> >>>>
>> >>>>> >>>> On Thu, Dec 25, 2008 at 5:09 PM, Wenlong Li <we...@gmail.com>
>> >>>>> wrote:
>> >>>>> >>>>> Sorry, one more question: bootclasspath.properties is classlib
>> >>>>> >>>>> specific file, why we could not make a vm specific file
>> manually?
>> >>>>> Just
>> >>>>> >>>>> curious to know the possible reason. :)
>> >>>>> >>>>>
>> >>>>> >>>>> thx,
>> >>>>> >>>>> Wenlong
>> >>>>> >>>>>
>> >>>>> >>>>> On Thu, Dec 25, 2008 at 10:00 PM, Pavel Pervov <
>> pmcfirst@gmail.com>
>> >>>>> wrote:
>> >>>>> >>>>>> If this would be VM-side automatically produced configuration
>> >>>>> file...
>> >>>>> >>>>>>
>> >>>>> >>>>>> WBR,
>> >>>>> >>>>>> Pavel.
>> >>>>> >>>>>>
>> >>>>> >>>>>> On Thu, Dec 25, 2008 at 4:58 PM, Wenlong Li <
>> wenlong@gmail.com>
>> >>>>> wrote:
>> >>>>> >>>>>>> btw, because adding new module is rare case, manually
>> modifying the
>> >>>>> >>>>>>> bootclasspath.properties is not an issue?
>> >>>>> >>>>>>>
>> >>>>> >>>>>>> If so, can I conclude adding another property file with same
>> update
>> >>>>> >>>>>>> frequency as bootclasspath would be fine as well?
>> >>>>> >>>>>>>
>> >>>>> >>>>>>> Pls kindly correct me if my understanding is wrong.
>> >>>>> >>>>>>>
>> >>>>> >>>>>>> On Thu, Dec 25, 2008 at 9:05 PM, Pavel Pervov <
>> pmcfirst@gmail.com>
>> >>>>> wrote:
>> >>>>> >>>>>>>> Wenlong,
>> >>>>> >>>>>>>>
>> >>>>> >>>>>>>> Note, that bootclasspath.properties is only changed on
>> adding new
>> >>>>> >>>>>>>> module. This is pretty rare occasion, I believe.
>> >>>>> >>>>>>>>
>> >>>>> >>>>>>>> WBR,
>> >>>>> >>>>>>>> Pavel.
>> >>>>> >>>>>>>>
>> >>>>> >>>>>>>> On Thu, Dec 25, 2008 at 3:48 PM, Wenlong Li <
>> wenlong@gmail.com>
>> >>>>> wrote:
>> >>>>> >>>>>>>>> Thx for your advice. Alexey.
>> >>>>> >>>>>>>>>
>> >>>>> >>>>>>>>> Here I have one question: do you know how the
>> >>>>> bootclasspath.properties
>> >>>>> >>>>>>>>> is maintained, manually or automatically?
>> >>>>> >>>>>>>>>
>> >>>>> >>>>>>>>> Another comment is I would like to treat the patch as DRLVM
>> >>>>> specific
>> >>>>> >>>>>>>>> optimization, e.g., it targets for improving VM creation
>> time. So
>> >>>>> that
>> >>>>> >>>>>>>>> is possible to move all updates to DRLVM part to eliminate
>> >>>>> potential
>> >>>>> >>>>>>>>> modularity and compatibility problem.
>> >>>>> >>>>>>>>>
>> >>>>> >>>>>>>>> thx,
>> >>>>> >>>>>>>>> Wenlong
>> >>>>> >>>>>>>>>
>> >>>>> >>>>>>>>> On Thu, Dec 25, 2008 at 5:32 PM, Aleksey Shipilev
>> >>>>> >>>>>>>>> <al...@gmail.com> wrote:
>> >>>>> >>>>>>>>>> Hi, Wenlong.
>> >>>>> >>>>>>>>>>
>> >>>>> >>>>>>>>>> On Thu, Dec 25, 2008 at 11:49 AM, Wenlong Li <
>> wenlong@gmail.com>
>> >>>>> wrote:
>> >>>>> >>>>>>>>>>> btw, Alexey, Let's go back to discuss whether there is a
>> need
>> >>>>> to
>> >>>>> >>>>>>>>>>> include this feature in Harmony, given 17% performance
>> gain in
>> >>>>> Linux
>> >>>>> >>>>>>>>>>> when using your methodology. For windows test, I will
>> double
>> >>>>> check the
>> >>>>> >>>>>>>>>>> backgroud process as you pointed out.
>> >>>>> >>>>>>>>>>
>> >>>>> >>>>>>>>>> My opinion was already expressed after I had finished the
>> tests
>> >>>>> from
>> >>>>> >>>>>>>>>> my side: the boost can be achieved in specific conditions,
>> so
>> >>>>> whether
>> >>>>> >>>>>>>>>> it's worth including into Harmony really depends on how
>> much
>> >>>>> mess the
>> >>>>> >>>>>>>>>> patch would introduce besides the "performance boost".
>> From what
>> >>>>> I
>> >>>>> >>>>>>>>>> see, the patch obliges the maintainer to maintain the
>> correct
>> >>>>> mapping
>> >>>>> >>>>>>>>>> between jars and Java packages. This new feature is also
>> spread
>> >>>>> >>>>>>>>>> between Classlib and VM, but it seems like DRLVM specific.
>> In
>> >>>>> this
>> >>>>> >>>>>>>>>> case I would rather stay without the patch.
>> >>>>> >>>>>>>>>>
>> >>>>> >>>>>>>>>> Personally (if I'll be committer) I would accept the patch
>> with
>> >>>>> two
>> >>>>> >>>>>>>>>> serious modifications:
>> >>>>> >>>>>>>>>> 1. Stay within DRLVM, do not introduce this feature into
>> >>>>> Classlib,
>> >>>>> >>>>>>>>>> get the thing tested and evolved on DRLVM side. Otherwise
>> it
>> >>>>> might
>> >>>>> >>>>>>>>>> break the compatibility with other VMs.
>> >>>>> >>>>>>>>>> 2. Make the mapping generated automatically (during build
>> >>>>> process?)
>> >>>>> >>>>>>>>>> to free the burden for maintainers.
>> >>>>> >>>>>>>>>>
>> >>>>> >>>>>>>>>> Thanks,
>> >>>>> >>>>>>>>>> Aleksey.
>> >>>>> >>>>>>>>>>
>> >>>>> >>>>>>>>>
>> >>>>> >>>>>>>>
>> >>>>> >>>>>>>
>> >>>>> >>>>>>
>> >>>>> >>>>>
>> >>>>> >>>>
>> >>>>> >>>
>> >>>>> >>
>> >>>>> >
>> >>>>> >
>> >>>>> >
>> >>>>> > --
>> >>>>> > С уважением,
>> >>>>> > Алексей Федотов,
>> >>>>> > ЗАО «Телеком Экспресс»
>> >>>>> >
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> --
>> >>>>> http://xiao-feng.blogspot.com
>> >>>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> --
>> >>>> С уважением,
>> >>>> Алексей Федотов,
>> >>>> ЗАО «Телеком Экспресс»
>> >>>>
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> http://xiao-feng.blogspot.com
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >> С уважением,
>> >> Алексей Федотов,
>> >> ЗАО «Телеком Экспресс»
>> >>
>> >
>> >
>> >
>> > --
>> > http://xiao-feng.blogspot.com
>> >
>>
>
>
>
> --
> С уважением,
> Алексей Федотов,
> ЗАО «Телеком Экспресс»
>
Re: [VM] On-demand class library parsing is ready to commit
Posted by Alexei Fedotov <al...@gmail.com>.
Sure.
1. If you dig into SetClasspathFromString, you will see that it starts from
splitting the given classpath into pieces. You already know the new piece
you add and may skip splitting step.
2. If I understand you code correctly, the case "pdest >
(*it).second->bytes" might be a subject of a negative assertion. Adding this
assrtion would speed up bug discovery.
Thanks.
On Tue, Jan 6, 2009 at 5:09 AM, Wenlong Li <we...@gmail.com> wrote:
> Yes, Xiao-Feng's understanding is correct. The patch loads and parses
> modules on demand. If no class in swing.jar is not requested, then
> this module will not be loaded.
>
> btw, Alexei, you said "SetClasspathFromString" and "pdest >
> (*it).second->bytes" are not efficient. Can you share more comments on
> them? I just reused some code in Harmony, and didn't optimize them
> further.
>
> Thx, Wenlong
> Managed Runtime Technology Center, Intel
>
> On Fri, Dec 26, 2008 at 5:16 PM, Xiao-Feng Li <xi...@gmail.com>
> wrote:
> > On Fri, Dec 26, 2008 at 5:08 PM, Alexei Fedotov
> > <al...@gmail.com> wrote:
> >> Xiao Feng,
> >> Thank you for explaining.
> >>
> >> I get more minor comments on more commented code, ineffective
> >> SetClasspathFromString usage, non-covered unexpected case when pdest >
> >> (*it).second->bytes. One major comment on crossing vm module boundary
> >> still remains. But now I'm happy with the design.
> >
> > Alexei, yes, I agree with your comments. These parts should be
> > improved. (Still, this is my personal opinion. :) Let's wait Wenlong
> > speaking.)
> >
> > Thanks,
> > xiaofeng
> >
> >> Sorry for being slow.
> >>
> >>
> >>
> >> On Fri, Dec 26, 2008 at 11:40 AM, Xiao-Feng Li <xi...@gmail.com>
> wrote:
> >>> On Fri, Dec 26, 2008 at 4:03 PM, Alexei Fedotov
> >>> <al...@gmail.com> wrote:
> >>>> Xiao-Feng,
> >>>> Continuing with the server example could you please give me a hint
> where
> >>>> decision to load swing.jar or not is taken in the patch? My initial
> >>>> perception was that the list of what to load was hardcoded and was not
> >>>> constructed dynamically depending on application.
> >>>
> >>> Alexei, here is the patch code I found:
> >>>
> >>> line 245:
> >>> + // Find which jar exports this package
> >>> + if (pkgName != NULL) {
> >>> + char *boot_class_path =
> >>> env->JavaProperties()->get(VM_BOOT_CLASS_DIR);
> >>> + char *pendingClassPath = NULL;
> >>> + apr_pool_t *tmp_pool;
> >>> + apr_pool_create(&tmp_pool, NULL);
> >>> + while (it != env->pending_jar_set.end()) {
> >>> + pdest = strstr( (*it).second->bytes, pkgName );
> >>> + if (pdest != NULL) {
> >>> + pendingClassPath =
> >>> (char*)STD_MALLOC(strlen(boot_class_path)
> >>> + +
> strlen((*it).first->bytes) + 1);
> >>> + strcpy(pendingClassPath, boot_class_path);
> >>> + strcat(pendingClassPath, (*it).first->bytes);
> >>> + // Open this found jar, and read all classes
> >>> contained in this jar
> >>> + SetClasspathFromString(pendingClassPath,
> tmp_pool);
> >>> + // Erase the found jar from pending jar list
> >>> as it has been parsed
> >>> + env->pending_jar_set.erase(it++);
> >>> + STD_FREE(pendingClassPath);
> >>> + } else {
> >>>
> >>> It checks if a JAR has the requested package, then loads it if yes. I
> >>> am not sure if this is what you were asking. (Btw, this is only my
> >>> understanding of his patch. I am not speaking for Wenlong.)
> >>>
> >>> Thanks,
> >>> xiaofeng
> >>>
> >>>> Thanks.
> >>>>
> >>>>
> >>>> On Fri, Dec 26, 2008 at 4:14 AM, Xiao-Feng Li <xi...@gmail.com>
> wrote:
> >>>>
> >>>>> On Fri, Dec 26, 2008 at 12:49 AM, Alexei Fedotov
> >>>>> <al...@gmail.com> wrote:
> >>>>> > Aleksey,
> >>>>> > I like your conclusion.
> >>>>> >
> >>>>> > Wenlong,
> >>>>> > I'm trying to understand the real life value of the "abstract"
> startup
> >>>>> > time metric you've suggested. Does Harmony with your patch load
> >>>>> > swing.jar for a server application? Do I understand that loading
> >>>>> > happens, though it happens later compared to VM without your patch?
> I
> >>>>> > believe that the proper design of delayed loading should answer
> "no"
> >>>>> > to this question.
> >>>>>
> >>>>> I checked the patch, and I found the answer is indeed "No" as you
> expected.
> >>>>>
> >>>>> Thanks,
> >>>>> xiaofeng
> >>>>>
> >>>>> > In other words, I appreciate if you describe which real use cases
> are
> >>>>> > improved by this patch.
> >>>>> > Thanks!
> >>>>>
> >>>>> > On Thu, Dec 25, 2008 at 7:29 PM, Aleksey Shipilev
> >>>>> > <al...@gmail.com> wrote:
> >>>>> >> Ok, here's the catch.
> >>>>> >>
> >>>>> >> bootclasspath.properties is the SortedSet<JARfile>, which
> enumerates
> >>>>> >> the JARs available for bootclassloader. The set of such the JARs
> is
> >>>>> >> really stable because modular decomposition of classlib is stable.
> >>>>> >> That's why nobody bothers with automatic generation of it: it only
> >>>>> >> should be updated when new JAR file arrives.
> >>>>> >> Modulelibrarymapping.properties is different on this point, it's
> the
> >>>>> >> Map<PackageName,JARfile>, which should be updated each time the
> new
> >>>>> >> *package* is introduced. I'm not talking about java.* packages
> >>>>> >> (they're standardized), rather about org.apache.harmony.*.
> >>>>> >>
> >>>>> >> Automatic generation of this property file gives two advantages:
> >>>>> >> 1. Error-prone. Prevent yourself from hand-messing with mapping
> and
> >>>>> >> getting spurious ClassNotFoundException. BTW, what's the behaviour
> in
> >>>>> >> case the mapping is wrong?
> >>>>> >> 2. "Researchful". There're lot of guys around who enjoys the
> >>>>> >> modularity of Harmony classlib and eventually they might want to
> split
> >>>>> >> the packages even deeper, into smaller pieces. Then automatic
> >>>>> >> generation would enable them to quickly roll-in and experiment
> with
> >>>>> >> different package layouts and their impact on performance. They
> could
> >>>>> >> use ordinary bootclasspath.properties, but your feature wouldn't
> be
> >>>>> >> used by them then ;)
> >>>>> >>
> >>>>> >> That's merely a housekeeping procedure. I believe that anything
> which
> >>>>> >> could be done more than once, eventually would be done more than
> once.
> >>>>> >> Hence it should be automated. You say that the file was generated
> from
> >>>>> >> manifests of JARs, so is it hard to just tie the same tool into
> DRLVM
> >>>>> >> build process?
> >>>>> >>
> >>>>> >> As for DRLVM-specific, my opinion that this is because the patch:
> >>>>> >> a. breaks the compatibility of classlib (you change
> >>>>> >> bootclasspath.properties, right?) with other VMs.
> >>>>> >> b. treated in DRLVM classloader only.
> >>>>> >>
> >>>>> >> Of course eventually this feature might be used by others, but IMO
> we
> >>>>> >> should be careful about other guys who use the same classlib. I'd
> >>>>> >> rather wait for some incubation on DRLVM side first.
> >>>>> >>
> >>>>> >> Thanks,
> >>>>> >> Aleksey.
> >>>>> >>
> >>>>> >> On Thu, Dec 25, 2008 at 6:18 PM, Wenlong Li <we...@gmail.com>
> wrote:
> >>>>> >>> I see. In fact, my file doesn't need track change at the class
> >>>>> >>> granularity. Instead, it only needs know package info provided in
> the
> >>>>> >>> manifest file. When class is added to a library, do we need
> change
> >>>>> >>> the manifest as well?
> >>>>> >>>
> >>>>> >>> btw, I guess there is a mis-understanding: my
> modulelibrarymapping
> >>>>> >>> file only records package info provided by manfiest in each
> module. It
> >>>>> >>> doesn't relate to each class.
> >>>>> >>>
> >>>>> >>> thx,
> >>>>> >>> Wenlong
> >>>>> >>>
> >>>>> >>> On Thu, Dec 25, 2008 at 10:55 PM, Pavel Pervov <
> pmcfirst@gmail.com>
> >>>>> wrote:
> >>>>> >>>> Classes are added to class library from time to time. I'm not
> sure how
> >>>>> >>>> it can be possible to track these changes manually.
> >>>>> >>>>
> >>>>> >>>> WBR,
> >>>>> >>>> Pavel.
> >>>>> >>>>
> >>>>> >>>>
> >>>>> >>>> On Thu, Dec 25, 2008 at 5:09 PM, Wenlong Li <we...@gmail.com>
> >>>>> wrote:
> >>>>> >>>>> Sorry, one more question: bootclasspath.properties is classlib
> >>>>> >>>>> specific file, why we could not make a vm specific file
> manually?
> >>>>> Just
> >>>>> >>>>> curious to know the possible reason. :)
> >>>>> >>>>>
> >>>>> >>>>> thx,
> >>>>> >>>>> Wenlong
> >>>>> >>>>>
> >>>>> >>>>> On Thu, Dec 25, 2008 at 10:00 PM, Pavel Pervov <
> pmcfirst@gmail.com>
> >>>>> wrote:
> >>>>> >>>>>> If this would be VM-side automatically produced configuration
> >>>>> file...
> >>>>> >>>>>>
> >>>>> >>>>>> WBR,
> >>>>> >>>>>> Pavel.
> >>>>> >>>>>>
> >>>>> >>>>>> On Thu, Dec 25, 2008 at 4:58 PM, Wenlong Li <
> wenlong@gmail.com>
> >>>>> wrote:
> >>>>> >>>>>>> btw, because adding new module is rare case, manually
> modifying the
> >>>>> >>>>>>> bootclasspath.properties is not an issue?
> >>>>> >>>>>>>
> >>>>> >>>>>>> If so, can I conclude adding another property file with same
> update
> >>>>> >>>>>>> frequency as bootclasspath would be fine as well?
> >>>>> >>>>>>>
> >>>>> >>>>>>> Pls kindly correct me if my understanding is wrong.
> >>>>> >>>>>>>
> >>>>> >>>>>>> On Thu, Dec 25, 2008 at 9:05 PM, Pavel Pervov <
> pmcfirst@gmail.com>
> >>>>> wrote:
> >>>>> >>>>>>>> Wenlong,
> >>>>> >>>>>>>>
> >>>>> >>>>>>>> Note, that bootclasspath.properties is only changed on
> adding new
> >>>>> >>>>>>>> module. This is pretty rare occasion, I believe.
> >>>>> >>>>>>>>
> >>>>> >>>>>>>> WBR,
> >>>>> >>>>>>>> Pavel.
> >>>>> >>>>>>>>
> >>>>> >>>>>>>> On Thu, Dec 25, 2008 at 3:48 PM, Wenlong Li <
> wenlong@gmail.com>
> >>>>> wrote:
> >>>>> >>>>>>>>> Thx for your advice. Alexey.
> >>>>> >>>>>>>>>
> >>>>> >>>>>>>>> Here I have one question: do you know how the
> >>>>> bootclasspath.properties
> >>>>> >>>>>>>>> is maintained, manually or automatically?
> >>>>> >>>>>>>>>
> >>>>> >>>>>>>>> Another comment is I would like to treat the patch as DRLVM
> >>>>> specific
> >>>>> >>>>>>>>> optimization, e.g., it targets for improving VM creation
> time. So
> >>>>> that
> >>>>> >>>>>>>>> is possible to move all updates to DRLVM part to eliminate
> >>>>> potential
> >>>>> >>>>>>>>> modularity and compatibility problem.
> >>>>> >>>>>>>>>
> >>>>> >>>>>>>>> thx,
> >>>>> >>>>>>>>> Wenlong
> >>>>> >>>>>>>>>
> >>>>> >>>>>>>>> On Thu, Dec 25, 2008 at 5:32 PM, Aleksey Shipilev
> >>>>> >>>>>>>>> <al...@gmail.com> wrote:
> >>>>> >>>>>>>>>> Hi, Wenlong.
> >>>>> >>>>>>>>>>
> >>>>> >>>>>>>>>> On Thu, Dec 25, 2008 at 11:49 AM, Wenlong Li <
> wenlong@gmail.com>
> >>>>> wrote:
> >>>>> >>>>>>>>>>> btw, Alexey, Let's go back to discuss whether there is a
> need
> >>>>> to
> >>>>> >>>>>>>>>>> include this feature in Harmony, given 17% performance
> gain in
> >>>>> Linux
> >>>>> >>>>>>>>>>> when using your methodology. For windows test, I will
> double
> >>>>> check the
> >>>>> >>>>>>>>>>> backgroud process as you pointed out.
> >>>>> >>>>>>>>>>
> >>>>> >>>>>>>>>> My opinion was already expressed after I had finished the
> tests
> >>>>> from
> >>>>> >>>>>>>>>> my side: the boost can be achieved in specific conditions,
> so
> >>>>> whether
> >>>>> >>>>>>>>>> it's worth including into Harmony really depends on how
> much
> >>>>> mess the
> >>>>> >>>>>>>>>> patch would introduce besides the "performance boost".
> From what
> >>>>> I
> >>>>> >>>>>>>>>> see, the patch obliges the maintainer to maintain the
> correct
> >>>>> mapping
> >>>>> >>>>>>>>>> between jars and Java packages. This new feature is also
> spread
> >>>>> >>>>>>>>>> between Classlib and VM, but it seems like DRLVM specific.
> In
> >>>>> this
> >>>>> >>>>>>>>>> case I would rather stay without the patch.
> >>>>> >>>>>>>>>>
> >>>>> >>>>>>>>>> Personally (if I'll be committer) I would accept the patch
> with
> >>>>> two
> >>>>> >>>>>>>>>> serious modifications:
> >>>>> >>>>>>>>>> 1. Stay within DRLVM, do not introduce this feature into
> >>>>> Classlib,
> >>>>> >>>>>>>>>> get the thing tested and evolved on DRLVM side. Otherwise
> it
> >>>>> might
> >>>>> >>>>>>>>>> break the compatibility with other VMs.
> >>>>> >>>>>>>>>> 2. Make the mapping generated automatically (during build
> >>>>> process?)
> >>>>> >>>>>>>>>> to free the burden for maintainers.
> >>>>> >>>>>>>>>>
> >>>>> >>>>>>>>>> Thanks,
> >>>>> >>>>>>>>>> Aleksey.
> >>>>> >>>>>>>>>>
> >>>>> >>>>>>>>>
> >>>>> >>>>>>>>
> >>>>> >>>>>>>
> >>>>> >>>>>>
> >>>>> >>>>>
> >>>>> >>>>
> >>>>> >>>
> >>>>> >>
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> > --
> >>>>> > С уважением,
> >>>>> > Алексей Федотов,
> >>>>> > ЗАО «Телеком Экспресс»
> >>>>> >
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> http://xiao-feng.blogspot.com
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> С уважением,
> >>>> Алексей Федотов,
> >>>> ЗАО «Телеком Экспресс»
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> http://xiao-feng.blogspot.com
> >>>
> >>
> >>
> >>
> >> --
> >> С уважением,
> >> Алексей Федотов,
> >> ЗАО «Телеком Экспресс»
> >>
> >
> >
> >
> > --
> > http://xiao-feng.blogspot.com
> >
>
--
С уважением,
Алексей Федотов,
ЗАО «Телеком Экспресс»
Re: [VM] On-demand class library parsing is ready to commit
Posted by Wenlong Li <we...@gmail.com>.
Yes, Xiao-Feng's understanding is correct. The patch loads and parses
modules on demand. If no class in swing.jar is not requested, then
this module will not be loaded.
btw, Alexei, you said "SetClasspathFromString" and "pdest >
(*it).second->bytes" are not efficient. Can you share more comments on
them? I just reused some code in Harmony, and didn't optimize them
further.
Thx, Wenlong
Managed Runtime Technology Center, Intel
On Fri, Dec 26, 2008 at 5:16 PM, Xiao-Feng Li <xi...@gmail.com> wrote:
> On Fri, Dec 26, 2008 at 5:08 PM, Alexei Fedotov
> <al...@gmail.com> wrote:
>> Xiao Feng,
>> Thank you for explaining.
>>
>> I get more minor comments on more commented code, ineffective
>> SetClasspathFromString usage, non-covered unexpected case when pdest >
>> (*it).second->bytes. One major comment on crossing vm module boundary
>> still remains. But now I'm happy with the design.
>
> Alexei, yes, I agree with your comments. These parts should be
> improved. (Still, this is my personal opinion. :) Let's wait Wenlong
> speaking.)
>
> Thanks,
> xiaofeng
>
>> Sorry for being slow.
>>
>>
>>
>> On Fri, Dec 26, 2008 at 11:40 AM, Xiao-Feng Li <xi...@gmail.com> wrote:
>>> On Fri, Dec 26, 2008 at 4:03 PM, Alexei Fedotov
>>> <al...@gmail.com> wrote:
>>>> Xiao-Feng,
>>>> Continuing with the server example could you please give me a hint where
>>>> decision to load swing.jar or not is taken in the patch? My initial
>>>> perception was that the list of what to load was hardcoded and was not
>>>> constructed dynamically depending on application.
>>>
>>> Alexei, here is the patch code I found:
>>>
>>> line 245:
>>> + // Find which jar exports this package
>>> + if (pkgName != NULL) {
>>> + char *boot_class_path =
>>> env->JavaProperties()->get(VM_BOOT_CLASS_DIR);
>>> + char *pendingClassPath = NULL;
>>> + apr_pool_t *tmp_pool;
>>> + apr_pool_create(&tmp_pool, NULL);
>>> + while (it != env->pending_jar_set.end()) {
>>> + pdest = strstr( (*it).second->bytes, pkgName );
>>> + if (pdest != NULL) {
>>> + pendingClassPath =
>>> (char*)STD_MALLOC(strlen(boot_class_path)
>>> + + strlen((*it).first->bytes) + 1);
>>> + strcpy(pendingClassPath, boot_class_path);
>>> + strcat(pendingClassPath, (*it).first->bytes);
>>> + // Open this found jar, and read all classes
>>> contained in this jar
>>> + SetClasspathFromString(pendingClassPath, tmp_pool);
>>> + // Erase the found jar from pending jar list
>>> as it has been parsed
>>> + env->pending_jar_set.erase(it++);
>>> + STD_FREE(pendingClassPath);
>>> + } else {
>>>
>>> It checks if a JAR has the requested package, then loads it if yes. I
>>> am not sure if this is what you were asking. (Btw, this is only my
>>> understanding of his patch. I am not speaking for Wenlong.)
>>>
>>> Thanks,
>>> xiaofeng
>>>
>>>> Thanks.
>>>>
>>>>
>>>> On Fri, Dec 26, 2008 at 4:14 AM, Xiao-Feng Li <xi...@gmail.com> wrote:
>>>>
>>>>> On Fri, Dec 26, 2008 at 12:49 AM, Alexei Fedotov
>>>>> <al...@gmail.com> wrote:
>>>>> > Aleksey,
>>>>> > I like your conclusion.
>>>>> >
>>>>> > Wenlong,
>>>>> > I'm trying to understand the real life value of the "abstract" startup
>>>>> > time metric you've suggested. Does Harmony with your patch load
>>>>> > swing.jar for a server application? Do I understand that loading
>>>>> > happens, though it happens later compared to VM without your patch? I
>>>>> > believe that the proper design of delayed loading should answer "no"
>>>>> > to this question.
>>>>>
>>>>> I checked the patch, and I found the answer is indeed "No" as you expected.
>>>>>
>>>>> Thanks,
>>>>> xiaofeng
>>>>>
>>>>> > In other words, I appreciate if you describe which real use cases are
>>>>> > improved by this patch.
>>>>> > Thanks!
>>>>>
>>>>> > On Thu, Dec 25, 2008 at 7:29 PM, Aleksey Shipilev
>>>>> > <al...@gmail.com> wrote:
>>>>> >> Ok, here's the catch.
>>>>> >>
>>>>> >> bootclasspath.properties is the SortedSet<JARfile>, which enumerates
>>>>> >> the JARs available for bootclassloader. The set of such the JARs is
>>>>> >> really stable because modular decomposition of classlib is stable.
>>>>> >> That's why nobody bothers with automatic generation of it: it only
>>>>> >> should be updated when new JAR file arrives.
>>>>> >> Modulelibrarymapping.properties is different on this point, it's the
>>>>> >> Map<PackageName,JARfile>, which should be updated each time the new
>>>>> >> *package* is introduced. I'm not talking about java.* packages
>>>>> >> (they're standardized), rather about org.apache.harmony.*.
>>>>> >>
>>>>> >> Automatic generation of this property file gives two advantages:
>>>>> >> 1. Error-prone. Prevent yourself from hand-messing with mapping and
>>>>> >> getting spurious ClassNotFoundException. BTW, what's the behaviour in
>>>>> >> case the mapping is wrong?
>>>>> >> 2. "Researchful". There're lot of guys around who enjoys the
>>>>> >> modularity of Harmony classlib and eventually they might want to split
>>>>> >> the packages even deeper, into smaller pieces. Then automatic
>>>>> >> generation would enable them to quickly roll-in and experiment with
>>>>> >> different package layouts and their impact on performance. They could
>>>>> >> use ordinary bootclasspath.properties, but your feature wouldn't be
>>>>> >> used by them then ;)
>>>>> >>
>>>>> >> That's merely a housekeeping procedure. I believe that anything which
>>>>> >> could be done more than once, eventually would be done more than once.
>>>>> >> Hence it should be automated. You say that the file was generated from
>>>>> >> manifests of JARs, so is it hard to just tie the same tool into DRLVM
>>>>> >> build process?
>>>>> >>
>>>>> >> As for DRLVM-specific, my opinion that this is because the patch:
>>>>> >> a. breaks the compatibility of classlib (you change
>>>>> >> bootclasspath.properties, right?) with other VMs.
>>>>> >> b. treated in DRLVM classloader only.
>>>>> >>
>>>>> >> Of course eventually this feature might be used by others, but IMO we
>>>>> >> should be careful about other guys who use the same classlib. I'd
>>>>> >> rather wait for some incubation on DRLVM side first.
>>>>> >>
>>>>> >> Thanks,
>>>>> >> Aleksey.
>>>>> >>
>>>>> >> On Thu, Dec 25, 2008 at 6:18 PM, Wenlong Li <we...@gmail.com> wrote:
>>>>> >>> I see. In fact, my file doesn't need track change at the class
>>>>> >>> granularity. Instead, it only needs know package info provided in the
>>>>> >>> manifest file. When class is added to a library, do we need change
>>>>> >>> the manifest as well?
>>>>> >>>
>>>>> >>> btw, I guess there is a mis-understanding: my modulelibrarymapping
>>>>> >>> file only records package info provided by manfiest in each module. It
>>>>> >>> doesn't relate to each class.
>>>>> >>>
>>>>> >>> thx,
>>>>> >>> Wenlong
>>>>> >>>
>>>>> >>> On Thu, Dec 25, 2008 at 10:55 PM, Pavel Pervov <pm...@gmail.com>
>>>>> wrote:
>>>>> >>>> Classes are added to class library from time to time. I'm not sure how
>>>>> >>>> it can be possible to track these changes manually.
>>>>> >>>>
>>>>> >>>> WBR,
>>>>> >>>> Pavel.
>>>>> >>>>
>>>>> >>>>
>>>>> >>>> On Thu, Dec 25, 2008 at 5:09 PM, Wenlong Li <we...@gmail.com>
>>>>> wrote:
>>>>> >>>>> Sorry, one more question: bootclasspath.properties is classlib
>>>>> >>>>> specific file, why we could not make a vm specific file manually?
>>>>> Just
>>>>> >>>>> curious to know the possible reason. :)
>>>>> >>>>>
>>>>> >>>>> thx,
>>>>> >>>>> Wenlong
>>>>> >>>>>
>>>>> >>>>> On Thu, Dec 25, 2008 at 10:00 PM, Pavel Pervov <pm...@gmail.com>
>>>>> wrote:
>>>>> >>>>>> If this would be VM-side automatically produced configuration
>>>>> file...
>>>>> >>>>>>
>>>>> >>>>>> WBR,
>>>>> >>>>>> Pavel.
>>>>> >>>>>>
>>>>> >>>>>> On Thu, Dec 25, 2008 at 4:58 PM, Wenlong Li <we...@gmail.com>
>>>>> wrote:
>>>>> >>>>>>> btw, because adding new module is rare case, manually modifying the
>>>>> >>>>>>> bootclasspath.properties is not an issue?
>>>>> >>>>>>>
>>>>> >>>>>>> If so, can I conclude adding another property file with same update
>>>>> >>>>>>> frequency as bootclasspath would be fine as well?
>>>>> >>>>>>>
>>>>> >>>>>>> Pls kindly correct me if my understanding is wrong.
>>>>> >>>>>>>
>>>>> >>>>>>> On Thu, Dec 25, 2008 at 9:05 PM, Pavel Pervov <pm...@gmail.com>
>>>>> wrote:
>>>>> >>>>>>>> Wenlong,
>>>>> >>>>>>>>
>>>>> >>>>>>>> Note, that bootclasspath.properties is only changed on adding new
>>>>> >>>>>>>> module. This is pretty rare occasion, I believe.
>>>>> >>>>>>>>
>>>>> >>>>>>>> WBR,
>>>>> >>>>>>>> Pavel.
>>>>> >>>>>>>>
>>>>> >>>>>>>> On Thu, Dec 25, 2008 at 3:48 PM, Wenlong Li <we...@gmail.com>
>>>>> wrote:
>>>>> >>>>>>>>> Thx for your advice. Alexey.
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> Here I have one question: do you know how the
>>>>> bootclasspath.properties
>>>>> >>>>>>>>> is maintained, manually or automatically?
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> Another comment is I would like to treat the patch as DRLVM
>>>>> specific
>>>>> >>>>>>>>> optimization, e.g., it targets for improving VM creation time. So
>>>>> that
>>>>> >>>>>>>>> is possible to move all updates to DRLVM part to eliminate
>>>>> potential
>>>>> >>>>>>>>> modularity and compatibility problem.
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> thx,
>>>>> >>>>>>>>> Wenlong
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> On Thu, Dec 25, 2008 at 5:32 PM, Aleksey Shipilev
>>>>> >>>>>>>>> <al...@gmail.com> wrote:
>>>>> >>>>>>>>>> Hi, Wenlong.
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>> On Thu, Dec 25, 2008 at 11:49 AM, Wenlong Li <we...@gmail.com>
>>>>> wrote:
>>>>> >>>>>>>>>>> btw, Alexey, Let's go back to discuss whether there is a need
>>>>> to
>>>>> >>>>>>>>>>> include this feature in Harmony, given 17% performance gain in
>>>>> Linux
>>>>> >>>>>>>>>>> when using your methodology. For windows test, I will double
>>>>> check the
>>>>> >>>>>>>>>>> backgroud process as you pointed out.
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>> My opinion was already expressed after I had finished the tests
>>>>> from
>>>>> >>>>>>>>>> my side: the boost can be achieved in specific conditions, so
>>>>> whether
>>>>> >>>>>>>>>> it's worth including into Harmony really depends on how much
>>>>> mess the
>>>>> >>>>>>>>>> patch would introduce besides the "performance boost". From what
>>>>> I
>>>>> >>>>>>>>>> see, the patch obliges the maintainer to maintain the correct
>>>>> mapping
>>>>> >>>>>>>>>> between jars and Java packages. This new feature is also spread
>>>>> >>>>>>>>>> between Classlib and VM, but it seems like DRLVM specific. In
>>>>> this
>>>>> >>>>>>>>>> case I would rather stay without the patch.
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>> Personally (if I'll be committer) I would accept the patch with
>>>>> two
>>>>> >>>>>>>>>> serious modifications:
>>>>> >>>>>>>>>> 1. Stay within DRLVM, do not introduce this feature into
>>>>> Classlib,
>>>>> >>>>>>>>>> get the thing tested and evolved on DRLVM side. Otherwise it
>>>>> might
>>>>> >>>>>>>>>> break the compatibility with other VMs.
>>>>> >>>>>>>>>> 2. Make the mapping generated automatically (during build
>>>>> process?)
>>>>> >>>>>>>>>> to free the burden for maintainers.
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>> Thanks,
>>>>> >>>>>>>>>> Aleksey.
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>
>>>>> >>>>>>>>
>>>>> >>>>>>>
>>>>> >>>>>>
>>>>> >>>>>
>>>>> >>>>
>>>>> >>>
>>>>> >>
>>>>> >
>>>>> >
>>>>> >
>>>>> > --
>>>>> > С уважением,
>>>>> > Алексей Федотов,
>>>>> > ЗАО «Телеком Экспресс»
>>>>> >
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> http://xiao-feng.blogspot.com
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> С уважением,
>>>> Алексей Федотов,
>>>> ЗАО «Телеком Экспресс»
>>>>
>>>
>>>
>>>
>>> --
>>> http://xiao-feng.blogspot.com
>>>
>>
>>
>>
>> --
>> С уважением,
>> Алексей Федотов,
>> ЗАО «Телеком Экспресс»
>>
>
>
>
> --
> http://xiao-feng.blogspot.com
>
Re: [VM] On-demand class library parsing is ready to commit
Posted by Xiao-Feng Li <xi...@gmail.com>.
On Fri, Dec 26, 2008 at 5:08 PM, Alexei Fedotov
<al...@gmail.com> wrote:
> Xiao Feng,
> Thank you for explaining.
>
> I get more minor comments on more commented code, ineffective
> SetClasspathFromString usage, non-covered unexpected case when pdest >
> (*it).second->bytes. One major comment on crossing vm module boundary
> still remains. But now I'm happy with the design.
Alexei, yes, I agree with your comments. These parts should be
improved. (Still, this is my personal opinion. :) Let's wait Wenlong
speaking.)
Thanks,
xiaofeng
> Sorry for being slow.
>
>
>
> On Fri, Dec 26, 2008 at 11:40 AM, Xiao-Feng Li <xi...@gmail.com> wrote:
>> On Fri, Dec 26, 2008 at 4:03 PM, Alexei Fedotov
>> <al...@gmail.com> wrote:
>>> Xiao-Feng,
>>> Continuing with the server example could you please give me a hint where
>>> decision to load swing.jar or not is taken in the patch? My initial
>>> perception was that the list of what to load was hardcoded and was not
>>> constructed dynamically depending on application.
>>
>> Alexei, here is the patch code I found:
>>
>> line 245:
>> + // Find which jar exports this package
>> + if (pkgName != NULL) {
>> + char *boot_class_path =
>> env->JavaProperties()->get(VM_BOOT_CLASS_DIR);
>> + char *pendingClassPath = NULL;
>> + apr_pool_t *tmp_pool;
>> + apr_pool_create(&tmp_pool, NULL);
>> + while (it != env->pending_jar_set.end()) {
>> + pdest = strstr( (*it).second->bytes, pkgName );
>> + if (pdest != NULL) {
>> + pendingClassPath =
>> (char*)STD_MALLOC(strlen(boot_class_path)
>> + + strlen((*it).first->bytes) + 1);
>> + strcpy(pendingClassPath, boot_class_path);
>> + strcat(pendingClassPath, (*it).first->bytes);
>> + // Open this found jar, and read all classes
>> contained in this jar
>> + SetClasspathFromString(pendingClassPath, tmp_pool);
>> + // Erase the found jar from pending jar list
>> as it has been parsed
>> + env->pending_jar_set.erase(it++);
>> + STD_FREE(pendingClassPath);
>> + } else {
>>
>> It checks if a JAR has the requested package, then loads it if yes. I
>> am not sure if this is what you were asking. (Btw, this is only my
>> understanding of his patch. I am not speaking for Wenlong.)
>>
>> Thanks,
>> xiaofeng
>>
>>> Thanks.
>>>
>>>
>>> On Fri, Dec 26, 2008 at 4:14 AM, Xiao-Feng Li <xi...@gmail.com> wrote:
>>>
>>>> On Fri, Dec 26, 2008 at 12:49 AM, Alexei Fedotov
>>>> <al...@gmail.com> wrote:
>>>> > Aleksey,
>>>> > I like your conclusion.
>>>> >
>>>> > Wenlong,
>>>> > I'm trying to understand the real life value of the "abstract" startup
>>>> > time metric you've suggested. Does Harmony with your patch load
>>>> > swing.jar for a server application? Do I understand that loading
>>>> > happens, though it happens later compared to VM without your patch? I
>>>> > believe that the proper design of delayed loading should answer "no"
>>>> > to this question.
>>>>
>>>> I checked the patch, and I found the answer is indeed "No" as you expected.
>>>>
>>>> Thanks,
>>>> xiaofeng
>>>>
>>>> > In other words, I appreciate if you describe which real use cases are
>>>> > improved by this patch.
>>>> > Thanks!
>>>>
>>>> > On Thu, Dec 25, 2008 at 7:29 PM, Aleksey Shipilev
>>>> > <al...@gmail.com> wrote:
>>>> >> Ok, here's the catch.
>>>> >>
>>>> >> bootclasspath.properties is the SortedSet<JARfile>, which enumerates
>>>> >> the JARs available for bootclassloader. The set of such the JARs is
>>>> >> really stable because modular decomposition of classlib is stable.
>>>> >> That's why nobody bothers with automatic generation of it: it only
>>>> >> should be updated when new JAR file arrives.
>>>> >> Modulelibrarymapping.properties is different on this point, it's the
>>>> >> Map<PackageName,JARfile>, which should be updated each time the new
>>>> >> *package* is introduced. I'm not talking about java.* packages
>>>> >> (they're standardized), rather about org.apache.harmony.*.
>>>> >>
>>>> >> Automatic generation of this property file gives two advantages:
>>>> >> 1. Error-prone. Prevent yourself from hand-messing with mapping and
>>>> >> getting spurious ClassNotFoundException. BTW, what's the behaviour in
>>>> >> case the mapping is wrong?
>>>> >> 2. "Researchful". There're lot of guys around who enjoys the
>>>> >> modularity of Harmony classlib and eventually they might want to split
>>>> >> the packages even deeper, into smaller pieces. Then automatic
>>>> >> generation would enable them to quickly roll-in and experiment with
>>>> >> different package layouts and their impact on performance. They could
>>>> >> use ordinary bootclasspath.properties, but your feature wouldn't be
>>>> >> used by them then ;)
>>>> >>
>>>> >> That's merely a housekeeping procedure. I believe that anything which
>>>> >> could be done more than once, eventually would be done more than once.
>>>> >> Hence it should be automated. You say that the file was generated from
>>>> >> manifests of JARs, so is it hard to just tie the same tool into DRLVM
>>>> >> build process?
>>>> >>
>>>> >> As for DRLVM-specific, my opinion that this is because the patch:
>>>> >> a. breaks the compatibility of classlib (you change
>>>> >> bootclasspath.properties, right?) with other VMs.
>>>> >> b. treated in DRLVM classloader only.
>>>> >>
>>>> >> Of course eventually this feature might be used by others, but IMO we
>>>> >> should be careful about other guys who use the same classlib. I'd
>>>> >> rather wait for some incubation on DRLVM side first.
>>>> >>
>>>> >> Thanks,
>>>> >> Aleksey.
>>>> >>
>>>> >> On Thu, Dec 25, 2008 at 6:18 PM, Wenlong Li <we...@gmail.com> wrote:
>>>> >>> I see. In fact, my file doesn't need track change at the class
>>>> >>> granularity. Instead, it only needs know package info provided in the
>>>> >>> manifest file. When class is added to a library, do we need change
>>>> >>> the manifest as well?
>>>> >>>
>>>> >>> btw, I guess there is a mis-understanding: my modulelibrarymapping
>>>> >>> file only records package info provided by manfiest in each module. It
>>>> >>> doesn't relate to each class.
>>>> >>>
>>>> >>> thx,
>>>> >>> Wenlong
>>>> >>>
>>>> >>> On Thu, Dec 25, 2008 at 10:55 PM, Pavel Pervov <pm...@gmail.com>
>>>> wrote:
>>>> >>>> Classes are added to class library from time to time. I'm not sure how
>>>> >>>> it can be possible to track these changes manually.
>>>> >>>>
>>>> >>>> WBR,
>>>> >>>> Pavel.
>>>> >>>>
>>>> >>>>
>>>> >>>> On Thu, Dec 25, 2008 at 5:09 PM, Wenlong Li <we...@gmail.com>
>>>> wrote:
>>>> >>>>> Sorry, one more question: bootclasspath.properties is classlib
>>>> >>>>> specific file, why we could not make a vm specific file manually?
>>>> Just
>>>> >>>>> curious to know the possible reason. :)
>>>> >>>>>
>>>> >>>>> thx,
>>>> >>>>> Wenlong
>>>> >>>>>
>>>> >>>>> On Thu, Dec 25, 2008 at 10:00 PM, Pavel Pervov <pm...@gmail.com>
>>>> wrote:
>>>> >>>>>> If this would be VM-side automatically produced configuration
>>>> file...
>>>> >>>>>>
>>>> >>>>>> WBR,
>>>> >>>>>> Pavel.
>>>> >>>>>>
>>>> >>>>>> On Thu, Dec 25, 2008 at 4:58 PM, Wenlong Li <we...@gmail.com>
>>>> wrote:
>>>> >>>>>>> btw, because adding new module is rare case, manually modifying the
>>>> >>>>>>> bootclasspath.properties is not an issue?
>>>> >>>>>>>
>>>> >>>>>>> If so, can I conclude adding another property file with same update
>>>> >>>>>>> frequency as bootclasspath would be fine as well?
>>>> >>>>>>>
>>>> >>>>>>> Pls kindly correct me if my understanding is wrong.
>>>> >>>>>>>
>>>> >>>>>>> On Thu, Dec 25, 2008 at 9:05 PM, Pavel Pervov <pm...@gmail.com>
>>>> wrote:
>>>> >>>>>>>> Wenlong,
>>>> >>>>>>>>
>>>> >>>>>>>> Note, that bootclasspath.properties is only changed on adding new
>>>> >>>>>>>> module. This is pretty rare occasion, I believe.
>>>> >>>>>>>>
>>>> >>>>>>>> WBR,
>>>> >>>>>>>> Pavel.
>>>> >>>>>>>>
>>>> >>>>>>>> On Thu, Dec 25, 2008 at 3:48 PM, Wenlong Li <we...@gmail.com>
>>>> wrote:
>>>> >>>>>>>>> Thx for your advice. Alexey.
>>>> >>>>>>>>>
>>>> >>>>>>>>> Here I have one question: do you know how the
>>>> bootclasspath.properties
>>>> >>>>>>>>> is maintained, manually or automatically?
>>>> >>>>>>>>>
>>>> >>>>>>>>> Another comment is I would like to treat the patch as DRLVM
>>>> specific
>>>> >>>>>>>>> optimization, e.g., it targets for improving VM creation time. So
>>>> that
>>>> >>>>>>>>> is possible to move all updates to DRLVM part to eliminate
>>>> potential
>>>> >>>>>>>>> modularity and compatibility problem.
>>>> >>>>>>>>>
>>>> >>>>>>>>> thx,
>>>> >>>>>>>>> Wenlong
>>>> >>>>>>>>>
>>>> >>>>>>>>> On Thu, Dec 25, 2008 at 5:32 PM, Aleksey Shipilev
>>>> >>>>>>>>> <al...@gmail.com> wrote:
>>>> >>>>>>>>>> Hi, Wenlong.
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> On Thu, Dec 25, 2008 at 11:49 AM, Wenlong Li <we...@gmail.com>
>>>> wrote:
>>>> >>>>>>>>>>> btw, Alexey, Let's go back to discuss whether there is a need
>>>> to
>>>> >>>>>>>>>>> include this feature in Harmony, given 17% performance gain in
>>>> Linux
>>>> >>>>>>>>>>> when using your methodology. For windows test, I will double
>>>> check the
>>>> >>>>>>>>>>> backgroud process as you pointed out.
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> My opinion was already expressed after I had finished the tests
>>>> from
>>>> >>>>>>>>>> my side: the boost can be achieved in specific conditions, so
>>>> whether
>>>> >>>>>>>>>> it's worth including into Harmony really depends on how much
>>>> mess the
>>>> >>>>>>>>>> patch would introduce besides the "performance boost". From what
>>>> I
>>>> >>>>>>>>>> see, the patch obliges the maintainer to maintain the correct
>>>> mapping
>>>> >>>>>>>>>> between jars and Java packages. This new feature is also spread
>>>> >>>>>>>>>> between Classlib and VM, but it seems like DRLVM specific. In
>>>> this
>>>> >>>>>>>>>> case I would rather stay without the patch.
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> Personally (if I'll be committer) I would accept the patch with
>>>> two
>>>> >>>>>>>>>> serious modifications:
>>>> >>>>>>>>>> 1. Stay within DRLVM, do not introduce this feature into
>>>> Classlib,
>>>> >>>>>>>>>> get the thing tested and evolved on DRLVM side. Otherwise it
>>>> might
>>>> >>>>>>>>>> break the compatibility with other VMs.
>>>> >>>>>>>>>> 2. Make the mapping generated automatically (during build
>>>> process?)
>>>> >>>>>>>>>> to free the burden for maintainers.
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> Thanks,
>>>> >>>>>>>>>> Aleksey.
>>>> >>>>>>>>>>
>>>> >>>>>>>>>
>>>> >>>>>>>>
>>>> >>>>>>>
>>>> >>>>>>
>>>> >>>>>
>>>> >>>>
>>>> >>>
>>>> >>
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > С уважением,
>>>> > Алексей Федотов,
>>>> > ЗАО «Телеком Экспресс»
>>>> >
>>>>
>>>>
>>>>
>>>> --
>>>> http://xiao-feng.blogspot.com
>>>>
>>>
>>>
>>>
>>> --
>>> С уважением,
>>> Алексей Федотов,
>>> ЗАО «Телеком Экспресс»
>>>
>>
>>
>>
>> --
>> http://xiao-feng.blogspot.com
>>
>
>
>
> --
> С уважением,
> Алексей Федотов,
> ЗАО «Телеком Экспресс»
>
--
http://xiao-feng.blogspot.com
Re: [VM] On-demand class library parsing is ready to commit
Posted by Alexei Fedotov <al...@gmail.com>.
Xiao Feng,
Thank you for explaining.
I get more minor comments on more commented code, ineffective
SetClasspathFromString usage, non-covered unexpected case when pdest >
(*it).second->bytes. One major comment on crossing vm module boundary
still remains. But now I'm happy with the design.
Sorry for being slow.
On Fri, Dec 26, 2008 at 11:40 AM, Xiao-Feng Li <xi...@gmail.com> wrote:
> On Fri, Dec 26, 2008 at 4:03 PM, Alexei Fedotov
> <al...@gmail.com> wrote:
>> Xiao-Feng,
>> Continuing with the server example could you please give me a hint where
>> decision to load swing.jar or not is taken in the patch? My initial
>> perception was that the list of what to load was hardcoded and was not
>> constructed dynamically depending on application.
>
> Alexei, here is the patch code I found:
>
> line 245:
> + // Find which jar exports this package
> + if (pkgName != NULL) {
> + char *boot_class_path =
> env->JavaProperties()->get(VM_BOOT_CLASS_DIR);
> + char *pendingClassPath = NULL;
> + apr_pool_t *tmp_pool;
> + apr_pool_create(&tmp_pool, NULL);
> + while (it != env->pending_jar_set.end()) {
> + pdest = strstr( (*it).second->bytes, pkgName );
> + if (pdest != NULL) {
> + pendingClassPath =
> (char*)STD_MALLOC(strlen(boot_class_path)
> + + strlen((*it).first->bytes) + 1);
> + strcpy(pendingClassPath, boot_class_path);
> + strcat(pendingClassPath, (*it).first->bytes);
> + // Open this found jar, and read all classes
> contained in this jar
> + SetClasspathFromString(pendingClassPath, tmp_pool);
> + // Erase the found jar from pending jar list
> as it has been parsed
> + env->pending_jar_set.erase(it++);
> + STD_FREE(pendingClassPath);
> + } else {
>
> It checks if a JAR has the requested package, then loads it if yes. I
> am not sure if this is what you were asking. (Btw, this is only my
> understanding of his patch. I am not speaking for Wenlong.)
>
> Thanks,
> xiaofeng
>
>> Thanks.
>>
>>
>> On Fri, Dec 26, 2008 at 4:14 AM, Xiao-Feng Li <xi...@gmail.com> wrote:
>>
>>> On Fri, Dec 26, 2008 at 12:49 AM, Alexei Fedotov
>>> <al...@gmail.com> wrote:
>>> > Aleksey,
>>> > I like your conclusion.
>>> >
>>> > Wenlong,
>>> > I'm trying to understand the real life value of the "abstract" startup
>>> > time metric you've suggested. Does Harmony with your patch load
>>> > swing.jar for a server application? Do I understand that loading
>>> > happens, though it happens later compared to VM without your patch? I
>>> > believe that the proper design of delayed loading should answer "no"
>>> > to this question.
>>>
>>> I checked the patch, and I found the answer is indeed "No" as you expected.
>>>
>>> Thanks,
>>> xiaofeng
>>>
>>> > In other words, I appreciate if you describe which real use cases are
>>> > improved by this patch.
>>> > Thanks!
>>>
>>> > On Thu, Dec 25, 2008 at 7:29 PM, Aleksey Shipilev
>>> > <al...@gmail.com> wrote:
>>> >> Ok, here's the catch.
>>> >>
>>> >> bootclasspath.properties is the SortedSet<JARfile>, which enumerates
>>> >> the JARs available for bootclassloader. The set of such the JARs is
>>> >> really stable because modular decomposition of classlib is stable.
>>> >> That's why nobody bothers with automatic generation of it: it only
>>> >> should be updated when new JAR file arrives.
>>> >> Modulelibrarymapping.properties is different on this point, it's the
>>> >> Map<PackageName,JARfile>, which should be updated each time the new
>>> >> *package* is introduced. I'm not talking about java.* packages
>>> >> (they're standardized), rather about org.apache.harmony.*.
>>> >>
>>> >> Automatic generation of this property file gives two advantages:
>>> >> 1. Error-prone. Prevent yourself from hand-messing with mapping and
>>> >> getting spurious ClassNotFoundException. BTW, what's the behaviour in
>>> >> case the mapping is wrong?
>>> >> 2. "Researchful". There're lot of guys around who enjoys the
>>> >> modularity of Harmony classlib and eventually they might want to split
>>> >> the packages even deeper, into smaller pieces. Then automatic
>>> >> generation would enable them to quickly roll-in and experiment with
>>> >> different package layouts and their impact on performance. They could
>>> >> use ordinary bootclasspath.properties, but your feature wouldn't be
>>> >> used by them then ;)
>>> >>
>>> >> That's merely a housekeeping procedure. I believe that anything which
>>> >> could be done more than once, eventually would be done more than once.
>>> >> Hence it should be automated. You say that the file was generated from
>>> >> manifests of JARs, so is it hard to just tie the same tool into DRLVM
>>> >> build process?
>>> >>
>>> >> As for DRLVM-specific, my opinion that this is because the patch:
>>> >> a. breaks the compatibility of classlib (you change
>>> >> bootclasspath.properties, right?) with other VMs.
>>> >> b. treated in DRLVM classloader only.
>>> >>
>>> >> Of course eventually this feature might be used by others, but IMO we
>>> >> should be careful about other guys who use the same classlib. I'd
>>> >> rather wait for some incubation on DRLVM side first.
>>> >>
>>> >> Thanks,
>>> >> Aleksey.
>>> >>
>>> >> On Thu, Dec 25, 2008 at 6:18 PM, Wenlong Li <we...@gmail.com> wrote:
>>> >>> I see. In fact, my file doesn't need track change at the class
>>> >>> granularity. Instead, it only needs know package info provided in the
>>> >>> manifest file. When class is added to a library, do we need change
>>> >>> the manifest as well?
>>> >>>
>>> >>> btw, I guess there is a mis-understanding: my modulelibrarymapping
>>> >>> file only records package info provided by manfiest in each module. It
>>> >>> doesn't relate to each class.
>>> >>>
>>> >>> thx,
>>> >>> Wenlong
>>> >>>
>>> >>> On Thu, Dec 25, 2008 at 10:55 PM, Pavel Pervov <pm...@gmail.com>
>>> wrote:
>>> >>>> Classes are added to class library from time to time. I'm not sure how
>>> >>>> it can be possible to track these changes manually.
>>> >>>>
>>> >>>> WBR,
>>> >>>> Pavel.
>>> >>>>
>>> >>>>
>>> >>>> On Thu, Dec 25, 2008 at 5:09 PM, Wenlong Li <we...@gmail.com>
>>> wrote:
>>> >>>>> Sorry, one more question: bootclasspath.properties is classlib
>>> >>>>> specific file, why we could not make a vm specific file manually?
>>> Just
>>> >>>>> curious to know the possible reason. :)
>>> >>>>>
>>> >>>>> thx,
>>> >>>>> Wenlong
>>> >>>>>
>>> >>>>> On Thu, Dec 25, 2008 at 10:00 PM, Pavel Pervov <pm...@gmail.com>
>>> wrote:
>>> >>>>>> If this would be VM-side automatically produced configuration
>>> file...
>>> >>>>>>
>>> >>>>>> WBR,
>>> >>>>>> Pavel.
>>> >>>>>>
>>> >>>>>> On Thu, Dec 25, 2008 at 4:58 PM, Wenlong Li <we...@gmail.com>
>>> wrote:
>>> >>>>>>> btw, because adding new module is rare case, manually modifying the
>>> >>>>>>> bootclasspath.properties is not an issue?
>>> >>>>>>>
>>> >>>>>>> If so, can I conclude adding another property file with same update
>>> >>>>>>> frequency as bootclasspath would be fine as well?
>>> >>>>>>>
>>> >>>>>>> Pls kindly correct me if my understanding is wrong.
>>> >>>>>>>
>>> >>>>>>> On Thu, Dec 25, 2008 at 9:05 PM, Pavel Pervov <pm...@gmail.com>
>>> wrote:
>>> >>>>>>>> Wenlong,
>>> >>>>>>>>
>>> >>>>>>>> Note, that bootclasspath.properties is only changed on adding new
>>> >>>>>>>> module. This is pretty rare occasion, I believe.
>>> >>>>>>>>
>>> >>>>>>>> WBR,
>>> >>>>>>>> Pavel.
>>> >>>>>>>>
>>> >>>>>>>> On Thu, Dec 25, 2008 at 3:48 PM, Wenlong Li <we...@gmail.com>
>>> wrote:
>>> >>>>>>>>> Thx for your advice. Alexey.
>>> >>>>>>>>>
>>> >>>>>>>>> Here I have one question: do you know how the
>>> bootclasspath.properties
>>> >>>>>>>>> is maintained, manually or automatically?
>>> >>>>>>>>>
>>> >>>>>>>>> Another comment is I would like to treat the patch as DRLVM
>>> specific
>>> >>>>>>>>> optimization, e.g., it targets for improving VM creation time. So
>>> that
>>> >>>>>>>>> is possible to move all updates to DRLVM part to eliminate
>>> potential
>>> >>>>>>>>> modularity and compatibility problem.
>>> >>>>>>>>>
>>> >>>>>>>>> thx,
>>> >>>>>>>>> Wenlong
>>> >>>>>>>>>
>>> >>>>>>>>> On Thu, Dec 25, 2008 at 5:32 PM, Aleksey Shipilev
>>> >>>>>>>>> <al...@gmail.com> wrote:
>>> >>>>>>>>>> Hi, Wenlong.
>>> >>>>>>>>>>
>>> >>>>>>>>>> On Thu, Dec 25, 2008 at 11:49 AM, Wenlong Li <we...@gmail.com>
>>> wrote:
>>> >>>>>>>>>>> btw, Alexey, Let's go back to discuss whether there is a need
>>> to
>>> >>>>>>>>>>> include this feature in Harmony, given 17% performance gain in
>>> Linux
>>> >>>>>>>>>>> when using your methodology. For windows test, I will double
>>> check the
>>> >>>>>>>>>>> backgroud process as you pointed out.
>>> >>>>>>>>>>
>>> >>>>>>>>>> My opinion was already expressed after I had finished the tests
>>> from
>>> >>>>>>>>>> my side: the boost can be achieved in specific conditions, so
>>> whether
>>> >>>>>>>>>> it's worth including into Harmony really depends on how much
>>> mess the
>>> >>>>>>>>>> patch would introduce besides the "performance boost". From what
>>> I
>>> >>>>>>>>>> see, the patch obliges the maintainer to maintain the correct
>>> mapping
>>> >>>>>>>>>> between jars and Java packages. This new feature is also spread
>>> >>>>>>>>>> between Classlib and VM, but it seems like DRLVM specific. In
>>> this
>>> >>>>>>>>>> case I would rather stay without the patch.
>>> >>>>>>>>>>
>>> >>>>>>>>>> Personally (if I'll be committer) I would accept the patch with
>>> two
>>> >>>>>>>>>> serious modifications:
>>> >>>>>>>>>> 1. Stay within DRLVM, do not introduce this feature into
>>> Classlib,
>>> >>>>>>>>>> get the thing tested and evolved on DRLVM side. Otherwise it
>>> might
>>> >>>>>>>>>> break the compatibility with other VMs.
>>> >>>>>>>>>> 2. Make the mapping generated automatically (during build
>>> process?)
>>> >>>>>>>>>> to free the burden for maintainers.
>>> >>>>>>>>>>
>>> >>>>>>>>>> Thanks,
>>> >>>>>>>>>> Aleksey.
>>> >>>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>
>>> >>>>>>>
>>> >>>>>>
>>> >>>>>
>>> >>>>
>>> >>>
>>> >>
>>> >
>>> >
>>> >
>>> > --
>>> > С уважением,
>>> > Алексей Федотов,
>>> > ЗАО «Телеком Экспресс»
>>> >
>>>
>>>
>>>
>>> --
>>> http://xiao-feng.blogspot.com
>>>
>>
>>
>>
>> --
>> С уважением,
>> Алексей Федотов,
>> ЗАО «Телеком Экспресс»
>>
>
>
>
> --
> http://xiao-feng.blogspot.com
>
--
С уважением,
Алексей Федотов,
ЗАО «Телеком Экспресс»
Re: [VM] On-demand class library parsing is ready to commit
Posted by Xiao-Feng Li <xi...@gmail.com>.
On Fri, Dec 26, 2008 at 4:03 PM, Alexei Fedotov
<al...@gmail.com> wrote:
> Xiao-Feng,
> Continuing with the server example could you please give me a hint where
> decision to load swing.jar or not is taken in the patch? My initial
> perception was that the list of what to load was hardcoded and was not
> constructed dynamically depending on application.
Alexei, here is the patch code I found:
line 245:
+ // Find which jar exports this package
+ if (pkgName != NULL) {
+ char *boot_class_path =
env->JavaProperties()->get(VM_BOOT_CLASS_DIR);
+ char *pendingClassPath = NULL;
+ apr_pool_t *tmp_pool;
+ apr_pool_create(&tmp_pool, NULL);
+ while (it != env->pending_jar_set.end()) {
+ pdest = strstr( (*it).second->bytes, pkgName );
+ if (pdest != NULL) {
+ pendingClassPath =
(char*)STD_MALLOC(strlen(boot_class_path)
+ + strlen((*it).first->bytes) + 1);
+ strcpy(pendingClassPath, boot_class_path);
+ strcat(pendingClassPath, (*it).first->bytes);
+ // Open this found jar, and read all classes
contained in this jar
+ SetClasspathFromString(pendingClassPath, tmp_pool);
+ // Erase the found jar from pending jar list
as it has been parsed
+ env->pending_jar_set.erase(it++);
+ STD_FREE(pendingClassPath);
+ } else {
It checks if a JAR has the requested package, then loads it if yes. I
am not sure if this is what you were asking. (Btw, this is only my
understanding of his patch. I am not speaking for Wenlong.)
Thanks,
xiaofeng
> Thanks.
>
>
> On Fri, Dec 26, 2008 at 4:14 AM, Xiao-Feng Li <xi...@gmail.com> wrote:
>
>> On Fri, Dec 26, 2008 at 12:49 AM, Alexei Fedotov
>> <al...@gmail.com> wrote:
>> > Aleksey,
>> > I like your conclusion.
>> >
>> > Wenlong,
>> > I'm trying to understand the real life value of the "abstract" startup
>> > time metric you've suggested. Does Harmony with your patch load
>> > swing.jar for a server application? Do I understand that loading
>> > happens, though it happens later compared to VM without your patch? I
>> > believe that the proper design of delayed loading should answer "no"
>> > to this question.
>>
>> I checked the patch, and I found the answer is indeed "No" as you expected.
>>
>> Thanks,
>> xiaofeng
>>
>> > In other words, I appreciate if you describe which real use cases are
>> > improved by this patch.
>> > Thanks!
>>
>> > On Thu, Dec 25, 2008 at 7:29 PM, Aleksey Shipilev
>> > <al...@gmail.com> wrote:
>> >> Ok, here's the catch.
>> >>
>> >> bootclasspath.properties is the SortedSet<JARfile>, which enumerates
>> >> the JARs available for bootclassloader. The set of such the JARs is
>> >> really stable because modular decomposition of classlib is stable.
>> >> That's why nobody bothers with automatic generation of it: it only
>> >> should be updated when new JAR file arrives.
>> >> Modulelibrarymapping.properties is different on this point, it's the
>> >> Map<PackageName,JARfile>, which should be updated each time the new
>> >> *package* is introduced. I'm not talking about java.* packages
>> >> (they're standardized), rather about org.apache.harmony.*.
>> >>
>> >> Automatic generation of this property file gives two advantages:
>> >> 1. Error-prone. Prevent yourself from hand-messing with mapping and
>> >> getting spurious ClassNotFoundException. BTW, what's the behaviour in
>> >> case the mapping is wrong?
>> >> 2. "Researchful". There're lot of guys around who enjoys the
>> >> modularity of Harmony classlib and eventually they might want to split
>> >> the packages even deeper, into smaller pieces. Then automatic
>> >> generation would enable them to quickly roll-in and experiment with
>> >> different package layouts and their impact on performance. They could
>> >> use ordinary bootclasspath.properties, but your feature wouldn't be
>> >> used by them then ;)
>> >>
>> >> That's merely a housekeeping procedure. I believe that anything which
>> >> could be done more than once, eventually would be done more than once.
>> >> Hence it should be automated. You say that the file was generated from
>> >> manifests of JARs, so is it hard to just tie the same tool into DRLVM
>> >> build process?
>> >>
>> >> As for DRLVM-specific, my opinion that this is because the patch:
>> >> a. breaks the compatibility of classlib (you change
>> >> bootclasspath.properties, right?) with other VMs.
>> >> b. treated in DRLVM classloader only.
>> >>
>> >> Of course eventually this feature might be used by others, but IMO we
>> >> should be careful about other guys who use the same classlib. I'd
>> >> rather wait for some incubation on DRLVM side first.
>> >>
>> >> Thanks,
>> >> Aleksey.
>> >>
>> >> On Thu, Dec 25, 2008 at 6:18 PM, Wenlong Li <we...@gmail.com> wrote:
>> >>> I see. In fact, my file doesn't need track change at the class
>> >>> granularity. Instead, it only needs know package info provided in the
>> >>> manifest file. When class is added to a library, do we need change
>> >>> the manifest as well?
>> >>>
>> >>> btw, I guess there is a mis-understanding: my modulelibrarymapping
>> >>> file only records package info provided by manfiest in each module. It
>> >>> doesn't relate to each class.
>> >>>
>> >>> thx,
>> >>> Wenlong
>> >>>
>> >>> On Thu, Dec 25, 2008 at 10:55 PM, Pavel Pervov <pm...@gmail.com>
>> wrote:
>> >>>> Classes are added to class library from time to time. I'm not sure how
>> >>>> it can be possible to track these changes manually.
>> >>>>
>> >>>> WBR,
>> >>>> Pavel.
>> >>>>
>> >>>>
>> >>>> On Thu, Dec 25, 2008 at 5:09 PM, Wenlong Li <we...@gmail.com>
>> wrote:
>> >>>>> Sorry, one more question: bootclasspath.properties is classlib
>> >>>>> specific file, why we could not make a vm specific file manually?
>> Just
>> >>>>> curious to know the possible reason. :)
>> >>>>>
>> >>>>> thx,
>> >>>>> Wenlong
>> >>>>>
>> >>>>> On Thu, Dec 25, 2008 at 10:00 PM, Pavel Pervov <pm...@gmail.com>
>> wrote:
>> >>>>>> If this would be VM-side automatically produced configuration
>> file...
>> >>>>>>
>> >>>>>> WBR,
>> >>>>>> Pavel.
>> >>>>>>
>> >>>>>> On Thu, Dec 25, 2008 at 4:58 PM, Wenlong Li <we...@gmail.com>
>> wrote:
>> >>>>>>> btw, because adding new module is rare case, manually modifying the
>> >>>>>>> bootclasspath.properties is not an issue?
>> >>>>>>>
>> >>>>>>> If so, can I conclude adding another property file with same update
>> >>>>>>> frequency as bootclasspath would be fine as well?
>> >>>>>>>
>> >>>>>>> Pls kindly correct me if my understanding is wrong.
>> >>>>>>>
>> >>>>>>> On Thu, Dec 25, 2008 at 9:05 PM, Pavel Pervov <pm...@gmail.com>
>> wrote:
>> >>>>>>>> Wenlong,
>> >>>>>>>>
>> >>>>>>>> Note, that bootclasspath.properties is only changed on adding new
>> >>>>>>>> module. This is pretty rare occasion, I believe.
>> >>>>>>>>
>> >>>>>>>> WBR,
>> >>>>>>>> Pavel.
>> >>>>>>>>
>> >>>>>>>> On Thu, Dec 25, 2008 at 3:48 PM, Wenlong Li <we...@gmail.com>
>> wrote:
>> >>>>>>>>> Thx for your advice. Alexey.
>> >>>>>>>>>
>> >>>>>>>>> Here I have one question: do you know how the
>> bootclasspath.properties
>> >>>>>>>>> is maintained, manually or automatically?
>> >>>>>>>>>
>> >>>>>>>>> Another comment is I would like to treat the patch as DRLVM
>> specific
>> >>>>>>>>> optimization, e.g., it targets for improving VM creation time. So
>> that
>> >>>>>>>>> is possible to move all updates to DRLVM part to eliminate
>> potential
>> >>>>>>>>> modularity and compatibility problem.
>> >>>>>>>>>
>> >>>>>>>>> thx,
>> >>>>>>>>> Wenlong
>> >>>>>>>>>
>> >>>>>>>>> On Thu, Dec 25, 2008 at 5:32 PM, Aleksey Shipilev
>> >>>>>>>>> <al...@gmail.com> wrote:
>> >>>>>>>>>> Hi, Wenlong.
>> >>>>>>>>>>
>> >>>>>>>>>> On Thu, Dec 25, 2008 at 11:49 AM, Wenlong Li <we...@gmail.com>
>> wrote:
>> >>>>>>>>>>> btw, Alexey, Let's go back to discuss whether there is a need
>> to
>> >>>>>>>>>>> include this feature in Harmony, given 17% performance gain in
>> Linux
>> >>>>>>>>>>> when using your methodology. For windows test, I will double
>> check the
>> >>>>>>>>>>> backgroud process as you pointed out.
>> >>>>>>>>>>
>> >>>>>>>>>> My opinion was already expressed after I had finished the tests
>> from
>> >>>>>>>>>> my side: the boost can be achieved in specific conditions, so
>> whether
>> >>>>>>>>>> it's worth including into Harmony really depends on how much
>> mess the
>> >>>>>>>>>> patch would introduce besides the "performance boost". From what
>> I
>> >>>>>>>>>> see, the patch obliges the maintainer to maintain the correct
>> mapping
>> >>>>>>>>>> between jars and Java packages. This new feature is also spread
>> >>>>>>>>>> between Classlib and VM, but it seems like DRLVM specific. In
>> this
>> >>>>>>>>>> case I would rather stay without the patch.
>> >>>>>>>>>>
>> >>>>>>>>>> Personally (if I'll be committer) I would accept the patch with
>> two
>> >>>>>>>>>> serious modifications:
>> >>>>>>>>>> 1. Stay within DRLVM, do not introduce this feature into
>> Classlib,
>> >>>>>>>>>> get the thing tested and evolved on DRLVM side. Otherwise it
>> might
>> >>>>>>>>>> break the compatibility with other VMs.
>> >>>>>>>>>> 2. Make the mapping generated automatically (during build
>> process?)
>> >>>>>>>>>> to free the burden for maintainers.
>> >>>>>>>>>>
>> >>>>>>>>>> Thanks,
>> >>>>>>>>>> Aleksey.
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>> >
>> >
>> >
>> > --
>> > С уважением,
>> > Алексей Федотов,
>> > ЗАО «Телеком Экспресс»
>> >
>>
>>
>>
>> --
>> http://xiao-feng.blogspot.com
>>
>
>
>
> --
> С уважением,
> Алексей Федотов,
> ЗАО «Телеком Экспресс»
>
--
http://xiao-feng.blogspot.com
Re: [VM] On-demand class library parsing is ready to commit
Posted by Alexei Fedotov <al...@gmail.com>.
Xiao-Feng,
Continuing with the server example could you please give me a hint where
decision to load swing.jar or not is taken in the patch? My initial
perception was that the list of what to load was hardcoded and was not
constructed dynamically depending on application.
Thanks.
On Fri, Dec 26, 2008 at 4:14 AM, Xiao-Feng Li <xi...@gmail.com> wrote:
> On Fri, Dec 26, 2008 at 12:49 AM, Alexei Fedotov
> <al...@gmail.com> wrote:
> > Aleksey,
> > I like your conclusion.
> >
> > Wenlong,
> > I'm trying to understand the real life value of the "abstract" startup
> > time metric you've suggested. Does Harmony with your patch load
> > swing.jar for a server application? Do I understand that loading
> > happens, though it happens later compared to VM without your patch? I
> > believe that the proper design of delayed loading should answer "no"
> > to this question.
>
> I checked the patch, and I found the answer is indeed "No" as you expected.
>
> Thanks,
> xiaofeng
>
> > In other words, I appreciate if you describe which real use cases are
> > improved by this patch.
> > Thanks!
>
> > On Thu, Dec 25, 2008 at 7:29 PM, Aleksey Shipilev
> > <al...@gmail.com> wrote:
> >> Ok, here's the catch.
> >>
> >> bootclasspath.properties is the SortedSet<JARfile>, which enumerates
> >> the JARs available for bootclassloader. The set of such the JARs is
> >> really stable because modular decomposition of classlib is stable.
> >> That's why nobody bothers with automatic generation of it: it only
> >> should be updated when new JAR file arrives.
> >> Modulelibrarymapping.properties is different on this point, it's the
> >> Map<PackageName,JARfile>, which should be updated each time the new
> >> *package* is introduced. I'm not talking about java.* packages
> >> (they're standardized), rather about org.apache.harmony.*.
> >>
> >> Automatic generation of this property file gives two advantages:
> >> 1. Error-prone. Prevent yourself from hand-messing with mapping and
> >> getting spurious ClassNotFoundException. BTW, what's the behaviour in
> >> case the mapping is wrong?
> >> 2. "Researchful". There're lot of guys around who enjoys the
> >> modularity of Harmony classlib and eventually they might want to split
> >> the packages even deeper, into smaller pieces. Then automatic
> >> generation would enable them to quickly roll-in and experiment with
> >> different package layouts and their impact on performance. They could
> >> use ordinary bootclasspath.properties, but your feature wouldn't be
> >> used by them then ;)
> >>
> >> That's merely a housekeeping procedure. I believe that anything which
> >> could be done more than once, eventually would be done more than once.
> >> Hence it should be automated. You say that the file was generated from
> >> manifests of JARs, so is it hard to just tie the same tool into DRLVM
> >> build process?
> >>
> >> As for DRLVM-specific, my opinion that this is because the patch:
> >> a. breaks the compatibility of classlib (you change
> >> bootclasspath.properties, right?) with other VMs.
> >> b. treated in DRLVM classloader only.
> >>
> >> Of course eventually this feature might be used by others, but IMO we
> >> should be careful about other guys who use the same classlib. I'd
> >> rather wait for some incubation on DRLVM side first.
> >>
> >> Thanks,
> >> Aleksey.
> >>
> >> On Thu, Dec 25, 2008 at 6:18 PM, Wenlong Li <we...@gmail.com> wrote:
> >>> I see. In fact, my file doesn't need track change at the class
> >>> granularity. Instead, it only needs know package info provided in the
> >>> manifest file. When class is added to a library, do we need change
> >>> the manifest as well?
> >>>
> >>> btw, I guess there is a mis-understanding: my modulelibrarymapping
> >>> file only records package info provided by manfiest in each module. It
> >>> doesn't relate to each class.
> >>>
> >>> thx,
> >>> Wenlong
> >>>
> >>> On Thu, Dec 25, 2008 at 10:55 PM, Pavel Pervov <pm...@gmail.com>
> wrote:
> >>>> Classes are added to class library from time to time. I'm not sure how
> >>>> it can be possible to track these changes manually.
> >>>>
> >>>> WBR,
> >>>> Pavel.
> >>>>
> >>>>
> >>>> On Thu, Dec 25, 2008 at 5:09 PM, Wenlong Li <we...@gmail.com>
> wrote:
> >>>>> Sorry, one more question: bootclasspath.properties is classlib
> >>>>> specific file, why we could not make a vm specific file manually?
> Just
> >>>>> curious to know the possible reason. :)
> >>>>>
> >>>>> thx,
> >>>>> Wenlong
> >>>>>
> >>>>> On Thu, Dec 25, 2008 at 10:00 PM, Pavel Pervov <pm...@gmail.com>
> wrote:
> >>>>>> If this would be VM-side automatically produced configuration
> file...
> >>>>>>
> >>>>>> WBR,
> >>>>>> Pavel.
> >>>>>>
> >>>>>> On Thu, Dec 25, 2008 at 4:58 PM, Wenlong Li <we...@gmail.com>
> wrote:
> >>>>>>> btw, because adding new module is rare case, manually modifying the
> >>>>>>> bootclasspath.properties is not an issue?
> >>>>>>>
> >>>>>>> If so, can I conclude adding another property file with same update
> >>>>>>> frequency as bootclasspath would be fine as well?
> >>>>>>>
> >>>>>>> Pls kindly correct me if my understanding is wrong.
> >>>>>>>
> >>>>>>> On Thu, Dec 25, 2008 at 9:05 PM, Pavel Pervov <pm...@gmail.com>
> wrote:
> >>>>>>>> Wenlong,
> >>>>>>>>
> >>>>>>>> Note, that bootclasspath.properties is only changed on adding new
> >>>>>>>> module. This is pretty rare occasion, I believe.
> >>>>>>>>
> >>>>>>>> WBR,
> >>>>>>>> Pavel.
> >>>>>>>>
> >>>>>>>> On Thu, Dec 25, 2008 at 3:48 PM, Wenlong Li <we...@gmail.com>
> wrote:
> >>>>>>>>> Thx for your advice. Alexey.
> >>>>>>>>>
> >>>>>>>>> Here I have one question: do you know how the
> bootclasspath.properties
> >>>>>>>>> is maintained, manually or automatically?
> >>>>>>>>>
> >>>>>>>>> Another comment is I would like to treat the patch as DRLVM
> specific
> >>>>>>>>> optimization, e.g., it targets for improving VM creation time. So
> that
> >>>>>>>>> is possible to move all updates to DRLVM part to eliminate
> potential
> >>>>>>>>> modularity and compatibility problem.
> >>>>>>>>>
> >>>>>>>>> thx,
> >>>>>>>>> Wenlong
> >>>>>>>>>
> >>>>>>>>> On Thu, Dec 25, 2008 at 5:32 PM, Aleksey Shipilev
> >>>>>>>>> <al...@gmail.com> wrote:
> >>>>>>>>>> Hi, Wenlong.
> >>>>>>>>>>
> >>>>>>>>>> On Thu, Dec 25, 2008 at 11:49 AM, Wenlong Li <we...@gmail.com>
> wrote:
> >>>>>>>>>>> btw, Alexey, Let's go back to discuss whether there is a need
> to
> >>>>>>>>>>> include this feature in Harmony, given 17% performance gain in
> Linux
> >>>>>>>>>>> when using your methodology. For windows test, I will double
> check the
> >>>>>>>>>>> backgroud process as you pointed out.
> >>>>>>>>>>
> >>>>>>>>>> My opinion was already expressed after I had finished the tests
> from
> >>>>>>>>>> my side: the boost can be achieved in specific conditions, so
> whether
> >>>>>>>>>> it's worth including into Harmony really depends on how much
> mess the
> >>>>>>>>>> patch would introduce besides the "performance boost". From what
> I
> >>>>>>>>>> see, the patch obliges the maintainer to maintain the correct
> mapping
> >>>>>>>>>> between jars and Java packages. This new feature is also spread
> >>>>>>>>>> between Classlib and VM, but it seems like DRLVM specific. In
> this
> >>>>>>>>>> case I would rather stay without the patch.
> >>>>>>>>>>
> >>>>>>>>>> Personally (if I'll be committer) I would accept the patch with
> two
> >>>>>>>>>> serious modifications:
> >>>>>>>>>> 1. Stay within DRLVM, do not introduce this feature into
> Classlib,
> >>>>>>>>>> get the thing tested and evolved on DRLVM side. Otherwise it
> might
> >>>>>>>>>> break the compatibility with other VMs.
> >>>>>>>>>> 2. Make the mapping generated automatically (during build
> process?)
> >>>>>>>>>> to free the burden for maintainers.
> >>>>>>>>>>
> >>>>>>>>>> Thanks,
> >>>>>>>>>> Aleksey.
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> >
> >
> >
> > --
> > С уважением,
> > Алексей Федотов,
> > ЗАО «Телеком Экспресс»
> >
>
>
>
> --
> http://xiao-feng.blogspot.com
>
--
С уважением,
Алексей Федотов,
ЗАО «Телеком Экспресс»
Re: [VM] On-demand class library parsing is ready to commit
Posted by Xiao-Feng Li <xi...@gmail.com>.
On Fri, Dec 26, 2008 at 12:49 AM, Alexei Fedotov
<al...@gmail.com> wrote:
> Aleksey,
> I like your conclusion.
>
> Wenlong,
> I'm trying to understand the real life value of the "abstract" startup
> time metric you've suggested. Does Harmony with your patch load
> swing.jar for a server application? Do I understand that loading
> happens, though it happens later compared to VM without your patch? I
> believe that the proper design of delayed loading should answer "no"
> to this question.
I checked the patch, and I found the answer is indeed "No" as you expected.
Thanks,
xiaofeng
> In other words, I appreciate if you describe which real use cases are
> improved by this patch.
> Thanks!
> On Thu, Dec 25, 2008 at 7:29 PM, Aleksey Shipilev
> <al...@gmail.com> wrote:
>> Ok, here's the catch.
>>
>> bootclasspath.properties is the SortedSet<JARfile>, which enumerates
>> the JARs available for bootclassloader. The set of such the JARs is
>> really stable because modular decomposition of classlib is stable.
>> That's why nobody bothers with automatic generation of it: it only
>> should be updated when new JAR file arrives.
>> Modulelibrarymapping.properties is different on this point, it's the
>> Map<PackageName,JARfile>, which should be updated each time the new
>> *package* is introduced. I'm not talking about java.* packages
>> (they're standardized), rather about org.apache.harmony.*.
>>
>> Automatic generation of this property file gives two advantages:
>> 1. Error-prone. Prevent yourself from hand-messing with mapping and
>> getting spurious ClassNotFoundException. BTW, what's the behaviour in
>> case the mapping is wrong?
>> 2. "Researchful". There're lot of guys around who enjoys the
>> modularity of Harmony classlib and eventually they might want to split
>> the packages even deeper, into smaller pieces. Then automatic
>> generation would enable them to quickly roll-in and experiment with
>> different package layouts and their impact on performance. They could
>> use ordinary bootclasspath.properties, but your feature wouldn't be
>> used by them then ;)
>>
>> That's merely a housekeeping procedure. I believe that anything which
>> could be done more than once, eventually would be done more than once.
>> Hence it should be automated. You say that the file was generated from
>> manifests of JARs, so is it hard to just tie the same tool into DRLVM
>> build process?
>>
>> As for DRLVM-specific, my opinion that this is because the patch:
>> a. breaks the compatibility of classlib (you change
>> bootclasspath.properties, right?) with other VMs.
>> b. treated in DRLVM classloader only.
>>
>> Of course eventually this feature might be used by others, but IMO we
>> should be careful about other guys who use the same classlib. I'd
>> rather wait for some incubation on DRLVM side first.
>>
>> Thanks,
>> Aleksey.
>>
>> On Thu, Dec 25, 2008 at 6:18 PM, Wenlong Li <we...@gmail.com> wrote:
>>> I see. In fact, my file doesn't need track change at the class
>>> granularity. Instead, it only needs know package info provided in the
>>> manifest file. When class is added to a library, do we need change
>>> the manifest as well?
>>>
>>> btw, I guess there is a mis-understanding: my modulelibrarymapping
>>> file only records package info provided by manfiest in each module. It
>>> doesn't relate to each class.
>>>
>>> thx,
>>> Wenlong
>>>
>>> On Thu, Dec 25, 2008 at 10:55 PM, Pavel Pervov <pm...@gmail.com> wrote:
>>>> Classes are added to class library from time to time. I'm not sure how
>>>> it can be possible to track these changes manually.
>>>>
>>>> WBR,
>>>> Pavel.
>>>>
>>>>
>>>> On Thu, Dec 25, 2008 at 5:09 PM, Wenlong Li <we...@gmail.com> wrote:
>>>>> Sorry, one more question: bootclasspath.properties is classlib
>>>>> specific file, why we could not make a vm specific file manually? Just
>>>>> curious to know the possible reason. :)
>>>>>
>>>>> thx,
>>>>> Wenlong
>>>>>
>>>>> On Thu, Dec 25, 2008 at 10:00 PM, Pavel Pervov <pm...@gmail.com> wrote:
>>>>>> If this would be VM-side automatically produced configuration file...
>>>>>>
>>>>>> WBR,
>>>>>> Pavel.
>>>>>>
>>>>>> On Thu, Dec 25, 2008 at 4:58 PM, Wenlong Li <we...@gmail.com> wrote:
>>>>>>> btw, because adding new module is rare case, manually modifying the
>>>>>>> bootclasspath.properties is not an issue?
>>>>>>>
>>>>>>> If so, can I conclude adding another property file with same update
>>>>>>> frequency as bootclasspath would be fine as well?
>>>>>>>
>>>>>>> Pls kindly correct me if my understanding is wrong.
>>>>>>>
>>>>>>> On Thu, Dec 25, 2008 at 9:05 PM, Pavel Pervov <pm...@gmail.com> wrote:
>>>>>>>> Wenlong,
>>>>>>>>
>>>>>>>> Note, that bootclasspath.properties is only changed on adding new
>>>>>>>> module. This is pretty rare occasion, I believe.
>>>>>>>>
>>>>>>>> WBR,
>>>>>>>> Pavel.
>>>>>>>>
>>>>>>>> On Thu, Dec 25, 2008 at 3:48 PM, Wenlong Li <we...@gmail.com> wrote:
>>>>>>>>> Thx for your advice. Alexey.
>>>>>>>>>
>>>>>>>>> Here I have one question: do you know how the bootclasspath.properties
>>>>>>>>> is maintained, manually or automatically?
>>>>>>>>>
>>>>>>>>> Another comment is I would like to treat the patch as DRLVM specific
>>>>>>>>> optimization, e.g., it targets for improving VM creation time. So that
>>>>>>>>> is possible to move all updates to DRLVM part to eliminate potential
>>>>>>>>> modularity and compatibility problem.
>>>>>>>>>
>>>>>>>>> thx,
>>>>>>>>> Wenlong
>>>>>>>>>
>>>>>>>>> On Thu, Dec 25, 2008 at 5:32 PM, Aleksey Shipilev
>>>>>>>>> <al...@gmail.com> wrote:
>>>>>>>>>> Hi, Wenlong.
>>>>>>>>>>
>>>>>>>>>> On Thu, Dec 25, 2008 at 11:49 AM, Wenlong Li <we...@gmail.com> wrote:
>>>>>>>>>>> btw, Alexey, Let's go back to discuss whether there is a need to
>>>>>>>>>>> include this feature in Harmony, given 17% performance gain in Linux
>>>>>>>>>>> when using your methodology. For windows test, I will double check the
>>>>>>>>>>> backgroud process as you pointed out.
>>>>>>>>>>
>>>>>>>>>> My opinion was already expressed after I had finished the tests from
>>>>>>>>>> my side: the boost can be achieved in specific conditions, so whether
>>>>>>>>>> it's worth including into Harmony really depends on how much mess the
>>>>>>>>>> patch would introduce besides the "performance boost". From what I
>>>>>>>>>> see, the patch obliges the maintainer to maintain the correct mapping
>>>>>>>>>> between jars and Java packages. This new feature is also spread
>>>>>>>>>> between Classlib and VM, but it seems like DRLVM specific. In this
>>>>>>>>>> case I would rather stay without the patch.
>>>>>>>>>>
>>>>>>>>>> Personally (if I'll be committer) I would accept the patch with two
>>>>>>>>>> serious modifications:
>>>>>>>>>> 1. Stay within DRLVM, do not introduce this feature into Classlib,
>>>>>>>>>> get the thing tested and evolved on DRLVM side. Otherwise it might
>>>>>>>>>> break the compatibility with other VMs.
>>>>>>>>>> 2. Make the mapping generated automatically (during build process?)
>>>>>>>>>> to free the burden for maintainers.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Aleksey.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
>
>
> --
> С уважением,
> Алексей Федотов,
> ЗАО «Телеком Экспресс»
>
--
http://xiao-feng.blogspot.com
Re: [VM] On-demand class library parsing is ready to commit
Posted by Alexei Fedotov <al...@gmail.com>.
Aleksey,
I like your conclusion.
Wenlong,
I'm trying to understand the real life value of the "abstract" startup
time metric you've suggested. Does Harmony with your patch load
swing.jar for a server application? Do I understand that loading
happens, though it happens later compared to VM without your patch? I
believe that the proper design of delayed loading should answer "no"
to this question.
In other words, I appreciate if you describe which real use cases are
improved by this patch.
Thanks!
On Thu, Dec 25, 2008 at 7:29 PM, Aleksey Shipilev
<al...@gmail.com> wrote:
> Ok, here's the catch.
>
> bootclasspath.properties is the SortedSet<JARfile>, which enumerates
> the JARs available for bootclassloader. The set of such the JARs is
> really stable because modular decomposition of classlib is stable.
> That's why nobody bothers with automatic generation of it: it only
> should be updated when new JAR file arrives.
> Modulelibrarymapping.properties is different on this point, it's the
> Map<PackageName,JARfile>, which should be updated each time the new
> *package* is introduced. I'm not talking about java.* packages
> (they're standardized), rather about org.apache.harmony.*.
>
> Automatic generation of this property file gives two advantages:
> 1. Error-prone. Prevent yourself from hand-messing with mapping and
> getting spurious ClassNotFoundException. BTW, what's the behaviour in
> case the mapping is wrong?
> 2. "Researchful". There're lot of guys around who enjoys the
> modularity of Harmony classlib and eventually they might want to split
> the packages even deeper, into smaller pieces. Then automatic
> generation would enable them to quickly roll-in and experiment with
> different package layouts and their impact on performance. They could
> use ordinary bootclasspath.properties, but your feature wouldn't be
> used by them then ;)
>
> That's merely a housekeeping procedure. I believe that anything which
> could be done more than once, eventually would be done more than once.
> Hence it should be automated. You say that the file was generated from
> manifests of JARs, so is it hard to just tie the same tool into DRLVM
> build process?
>
> As for DRLVM-specific, my opinion that this is because the patch:
> a. breaks the compatibility of classlib (you change
> bootclasspath.properties, right?) with other VMs.
> b. treated in DRLVM classloader only.
>
> Of course eventually this feature might be used by others, but IMO we
> should be careful about other guys who use the same classlib. I'd
> rather wait for some incubation on DRLVM side first.
>
> Thanks,
> Aleksey.
>
> On Thu, Dec 25, 2008 at 6:18 PM, Wenlong Li <we...@gmail.com> wrote:
>> I see. In fact, my file doesn't need track change at the class
>> granularity. Instead, it only needs know package info provided in the
>> manifest file. When class is added to a library, do we need change
>> the manifest as well?
>>
>> btw, I guess there is a mis-understanding: my modulelibrarymapping
>> file only records package info provided by manfiest in each module. It
>> doesn't relate to each class.
>>
>> thx,
>> Wenlong
>>
>> On Thu, Dec 25, 2008 at 10:55 PM, Pavel Pervov <pm...@gmail.com> wrote:
>>> Classes are added to class library from time to time. I'm not sure how
>>> it can be possible to track these changes manually.
>>>
>>> WBR,
>>> Pavel.
>>>
>>>
>>> On Thu, Dec 25, 2008 at 5:09 PM, Wenlong Li <we...@gmail.com> wrote:
>>>> Sorry, one more question: bootclasspath.properties is classlib
>>>> specific file, why we could not make a vm specific file manually? Just
>>>> curious to know the possible reason. :)
>>>>
>>>> thx,
>>>> Wenlong
>>>>
>>>> On Thu, Dec 25, 2008 at 10:00 PM, Pavel Pervov <pm...@gmail.com> wrote:
>>>>> If this would be VM-side automatically produced configuration file...
>>>>>
>>>>> WBR,
>>>>> Pavel.
>>>>>
>>>>> On Thu, Dec 25, 2008 at 4:58 PM, Wenlong Li <we...@gmail.com> wrote:
>>>>>> btw, because adding new module is rare case, manually modifying the
>>>>>> bootclasspath.properties is not an issue?
>>>>>>
>>>>>> If so, can I conclude adding another property file with same update
>>>>>> frequency as bootclasspath would be fine as well?
>>>>>>
>>>>>> Pls kindly correct me if my understanding is wrong.
>>>>>>
>>>>>> On Thu, Dec 25, 2008 at 9:05 PM, Pavel Pervov <pm...@gmail.com> wrote:
>>>>>>> Wenlong,
>>>>>>>
>>>>>>> Note, that bootclasspath.properties is only changed on adding new
>>>>>>> module. This is pretty rare occasion, I believe.
>>>>>>>
>>>>>>> WBR,
>>>>>>> Pavel.
>>>>>>>
>>>>>>> On Thu, Dec 25, 2008 at 3:48 PM, Wenlong Li <we...@gmail.com> wrote:
>>>>>>>> Thx for your advice. Alexey.
>>>>>>>>
>>>>>>>> Here I have one question: do you know how the bootclasspath.properties
>>>>>>>> is maintained, manually or automatically?
>>>>>>>>
>>>>>>>> Another comment is I would like to treat the patch as DRLVM specific
>>>>>>>> optimization, e.g., it targets for improving VM creation time. So that
>>>>>>>> is possible to move all updates to DRLVM part to eliminate potential
>>>>>>>> modularity and compatibility problem.
>>>>>>>>
>>>>>>>> thx,
>>>>>>>> Wenlong
>>>>>>>>
>>>>>>>> On Thu, Dec 25, 2008 at 5:32 PM, Aleksey Shipilev
>>>>>>>> <al...@gmail.com> wrote:
>>>>>>>>> Hi, Wenlong.
>>>>>>>>>
>>>>>>>>> On Thu, Dec 25, 2008 at 11:49 AM, Wenlong Li <we...@gmail.com> wrote:
>>>>>>>>>> btw, Alexey, Let's go back to discuss whether there is a need to
>>>>>>>>>> include this feature in Harmony, given 17% performance gain in Linux
>>>>>>>>>> when using your methodology. For windows test, I will double check the
>>>>>>>>>> backgroud process as you pointed out.
>>>>>>>>>
>>>>>>>>> My opinion was already expressed after I had finished the tests from
>>>>>>>>> my side: the boost can be achieved in specific conditions, so whether
>>>>>>>>> it's worth including into Harmony really depends on how much mess the
>>>>>>>>> patch would introduce besides the "performance boost". From what I
>>>>>>>>> see, the patch obliges the maintainer to maintain the correct mapping
>>>>>>>>> between jars and Java packages. This new feature is also spread
>>>>>>>>> between Classlib and VM, but it seems like DRLVM specific. In this
>>>>>>>>> case I would rather stay without the patch.
>>>>>>>>>
>>>>>>>>> Personally (if I'll be committer) I would accept the patch with two
>>>>>>>>> serious modifications:
>>>>>>>>> 1. Stay within DRLVM, do not introduce this feature into Classlib,
>>>>>>>>> get the thing tested and evolved on DRLVM side. Otherwise it might
>>>>>>>>> break the compatibility with other VMs.
>>>>>>>>> 2. Make the mapping generated automatically (during build process?)
>>>>>>>>> to free the burden for maintainers.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Aleksey.
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
--
С уважением,
Алексей Федотов,
ЗАО «Телеком Экспресс»
Re: [VM] On-demand class library parsing is ready to commit
Posted by Aleksey Shipilev <al...@gmail.com>.
Ok, here's the catch.
bootclasspath.properties is the SortedSet<JARfile>, which enumerates
the JARs available for bootclassloader. The set of such the JARs is
really stable because modular decomposition of classlib is stable.
That's why nobody bothers with automatic generation of it: it only
should be updated when new JAR file arrives.
Modulelibrarymapping.properties is different on this point, it's the
Map<PackageName,JARfile>, which should be updated each time the new
*package* is introduced. I'm not talking about java.* packages
(they're standardized), rather about org.apache.harmony.*.
Automatic generation of this property file gives two advantages:
1. Error-prone. Prevent yourself from hand-messing with mapping and
getting spurious ClassNotFoundException. BTW, what's the behaviour in
case the mapping is wrong?
2. "Researchful". There're lot of guys around who enjoys the
modularity of Harmony classlib and eventually they might want to split
the packages even deeper, into smaller pieces. Then automatic
generation would enable them to quickly roll-in and experiment with
different package layouts and their impact on performance. They could
use ordinary bootclasspath.properties, but your feature wouldn't be
used by them then ;)
That's merely a housekeeping procedure. I believe that anything which
could be done more than once, eventually would be done more than once.
Hence it should be automated. You say that the file was generated from
manifests of JARs, so is it hard to just tie the same tool into DRLVM
build process?
As for DRLVM-specific, my opinion that this is because the patch:
a. breaks the compatibility of classlib (you change
bootclasspath.properties, right?) with other VMs.
b. treated in DRLVM classloader only.
Of course eventually this feature might be used by others, but IMO we
should be careful about other guys who use the same classlib. I'd
rather wait for some incubation on DRLVM side first.
Thanks,
Aleksey.
On Thu, Dec 25, 2008 at 6:18 PM, Wenlong Li <we...@gmail.com> wrote:
> I see. In fact, my file doesn't need track change at the class
> granularity. Instead, it only needs know package info provided in the
> manifest file. When class is added to a library, do we need change
> the manifest as well?
>
> btw, I guess there is a mis-understanding: my modulelibrarymapping
> file only records package info provided by manfiest in each module. It
> doesn't relate to each class.
>
> thx,
> Wenlong
>
> On Thu, Dec 25, 2008 at 10:55 PM, Pavel Pervov <pm...@gmail.com> wrote:
>> Classes are added to class library from time to time. I'm not sure how
>> it can be possible to track these changes manually.
>>
>> WBR,
>> Pavel.
>>
>>
>> On Thu, Dec 25, 2008 at 5:09 PM, Wenlong Li <we...@gmail.com> wrote:
>>> Sorry, one more question: bootclasspath.properties is classlib
>>> specific file, why we could not make a vm specific file manually? Just
>>> curious to know the possible reason. :)
>>>
>>> thx,
>>> Wenlong
>>>
>>> On Thu, Dec 25, 2008 at 10:00 PM, Pavel Pervov <pm...@gmail.com> wrote:
>>>> If this would be VM-side automatically produced configuration file...
>>>>
>>>> WBR,
>>>> Pavel.
>>>>
>>>> On Thu, Dec 25, 2008 at 4:58 PM, Wenlong Li <we...@gmail.com> wrote:
>>>>> btw, because adding new module is rare case, manually modifying the
>>>>> bootclasspath.properties is not an issue?
>>>>>
>>>>> If so, can I conclude adding another property file with same update
>>>>> frequency as bootclasspath would be fine as well?
>>>>>
>>>>> Pls kindly correct me if my understanding is wrong.
>>>>>
>>>>> On Thu, Dec 25, 2008 at 9:05 PM, Pavel Pervov <pm...@gmail.com> wrote:
>>>>>> Wenlong,
>>>>>>
>>>>>> Note, that bootclasspath.properties is only changed on adding new
>>>>>> module. This is pretty rare occasion, I believe.
>>>>>>
>>>>>> WBR,
>>>>>> Pavel.
>>>>>>
>>>>>> On Thu, Dec 25, 2008 at 3:48 PM, Wenlong Li <we...@gmail.com> wrote:
>>>>>>> Thx for your advice. Alexey.
>>>>>>>
>>>>>>> Here I have one question: do you know how the bootclasspath.properties
>>>>>>> is maintained, manually or automatically?
>>>>>>>
>>>>>>> Another comment is I would like to treat the patch as DRLVM specific
>>>>>>> optimization, e.g., it targets for improving VM creation time. So that
>>>>>>> is possible to move all updates to DRLVM part to eliminate potential
>>>>>>> modularity and compatibility problem.
>>>>>>>
>>>>>>> thx,
>>>>>>> Wenlong
>>>>>>>
>>>>>>> On Thu, Dec 25, 2008 at 5:32 PM, Aleksey Shipilev
>>>>>>> <al...@gmail.com> wrote:
>>>>>>>> Hi, Wenlong.
>>>>>>>>
>>>>>>>> On Thu, Dec 25, 2008 at 11:49 AM, Wenlong Li <we...@gmail.com> wrote:
>>>>>>>>> btw, Alexey, Let's go back to discuss whether there is a need to
>>>>>>>>> include this feature in Harmony, given 17% performance gain in Linux
>>>>>>>>> when using your methodology. For windows test, I will double check the
>>>>>>>>> backgroud process as you pointed out.
>>>>>>>>
>>>>>>>> My opinion was already expressed after I had finished the tests from
>>>>>>>> my side: the boost can be achieved in specific conditions, so whether
>>>>>>>> it's worth including into Harmony really depends on how much mess the
>>>>>>>> patch would introduce besides the "performance boost". From what I
>>>>>>>> see, the patch obliges the maintainer to maintain the correct mapping
>>>>>>>> between jars and Java packages. This new feature is also spread
>>>>>>>> between Classlib and VM, but it seems like DRLVM specific. In this
>>>>>>>> case I would rather stay without the patch.
>>>>>>>>
>>>>>>>> Personally (if I'll be committer) I would accept the patch with two
>>>>>>>> serious modifications:
>>>>>>>> 1. Stay within DRLVM, do not introduce this feature into Classlib,
>>>>>>>> get the thing tested and evolved on DRLVM side. Otherwise it might
>>>>>>>> break the compatibility with other VMs.
>>>>>>>> 2. Make the mapping generated automatically (during build process?)
>>>>>>>> to free the burden for maintainers.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Aleksey.
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: [VM] On-demand class library parsing is ready to commit
Posted by Wenlong Li <we...@gmail.com>.
I see. In fact, my file doesn't need track change at the class
granularity. Instead, it only needs know package info provided in the
manifest file. When class is added to a library, do we need change
the manifest as well?
btw, I guess there is a mis-understanding: my modulelibrarymapping
file only records package info provided by manfiest in each module. It
doesn't relate to each class.
thx,
Wenlong
On Thu, Dec 25, 2008 at 10:55 PM, Pavel Pervov <pm...@gmail.com> wrote:
> Classes are added to class library from time to time. I'm not sure how
> it can be possible to track these changes manually.
>
> WBR,
> Pavel.
>
>
> On Thu, Dec 25, 2008 at 5:09 PM, Wenlong Li <we...@gmail.com> wrote:
>> Sorry, one more question: bootclasspath.properties is classlib
>> specific file, why we could not make a vm specific file manually? Just
>> curious to know the possible reason. :)
>>
>> thx,
>> Wenlong
>>
>> On Thu, Dec 25, 2008 at 10:00 PM, Pavel Pervov <pm...@gmail.com> wrote:
>>> If this would be VM-side automatically produced configuration file...
>>>
>>> WBR,
>>> Pavel.
>>>
>>> On Thu, Dec 25, 2008 at 4:58 PM, Wenlong Li <we...@gmail.com> wrote:
>>>> btw, because adding new module is rare case, manually modifying the
>>>> bootclasspath.properties is not an issue?
>>>>
>>>> If so, can I conclude adding another property file with same update
>>>> frequency as bootclasspath would be fine as well?
>>>>
>>>> Pls kindly correct me if my understanding is wrong.
>>>>
>>>> On Thu, Dec 25, 2008 at 9:05 PM, Pavel Pervov <pm...@gmail.com> wrote:
>>>>> Wenlong,
>>>>>
>>>>> Note, that bootclasspath.properties is only changed on adding new
>>>>> module. This is pretty rare occasion, I believe.
>>>>>
>>>>> WBR,
>>>>> Pavel.
>>>>>
>>>>> On Thu, Dec 25, 2008 at 3:48 PM, Wenlong Li <we...@gmail.com> wrote:
>>>>>> Thx for your advice. Alexey.
>>>>>>
>>>>>> Here I have one question: do you know how the bootclasspath.properties
>>>>>> is maintained, manually or automatically?
>>>>>>
>>>>>> Another comment is I would like to treat the patch as DRLVM specific
>>>>>> optimization, e.g., it targets for improving VM creation time. So that
>>>>>> is possible to move all updates to DRLVM part to eliminate potential
>>>>>> modularity and compatibility problem.
>>>>>>
>>>>>> thx,
>>>>>> Wenlong
>>>>>>
>>>>>> On Thu, Dec 25, 2008 at 5:32 PM, Aleksey Shipilev
>>>>>> <al...@gmail.com> wrote:
>>>>>>> Hi, Wenlong.
>>>>>>>
>>>>>>> On Thu, Dec 25, 2008 at 11:49 AM, Wenlong Li <we...@gmail.com> wrote:
>>>>>>>> btw, Alexey, Let's go back to discuss whether there is a need to
>>>>>>>> include this feature in Harmony, given 17% performance gain in Linux
>>>>>>>> when using your methodology. For windows test, I will double check the
>>>>>>>> backgroud process as you pointed out.
>>>>>>>
>>>>>>> My opinion was already expressed after I had finished the tests from
>>>>>>> my side: the boost can be achieved in specific conditions, so whether
>>>>>>> it's worth including into Harmony really depends on how much mess the
>>>>>>> patch would introduce besides the "performance boost". From what I
>>>>>>> see, the patch obliges the maintainer to maintain the correct mapping
>>>>>>> between jars and Java packages. This new feature is also spread
>>>>>>> between Classlib and VM, but it seems like DRLVM specific. In this
>>>>>>> case I would rather stay without the patch.
>>>>>>>
>>>>>>> Personally (if I'll be committer) I would accept the patch with two
>>>>>>> serious modifications:
>>>>>>> 1. Stay within DRLVM, do not introduce this feature into Classlib,
>>>>>>> get the thing tested and evolved on DRLVM side. Otherwise it might
>>>>>>> break the compatibility with other VMs.
>>>>>>> 2. Make the mapping generated automatically (during build process?)
>>>>>>> to free the burden for maintainers.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Aleksey.
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: [VM] On-demand class library parsing is ready to commit
Posted by Pavel Pervov <pm...@gmail.com>.
Classes are added to class library from time to time. I'm not sure how
it can be possible to track these changes manually.
WBR,
Pavel.
On Thu, Dec 25, 2008 at 5:09 PM, Wenlong Li <we...@gmail.com> wrote:
> Sorry, one more question: bootclasspath.properties is classlib
> specific file, why we could not make a vm specific file manually? Just
> curious to know the possible reason. :)
>
> thx,
> Wenlong
>
> On Thu, Dec 25, 2008 at 10:00 PM, Pavel Pervov <pm...@gmail.com> wrote:
>> If this would be VM-side automatically produced configuration file...
>>
>> WBR,
>> Pavel.
>>
>> On Thu, Dec 25, 2008 at 4:58 PM, Wenlong Li <we...@gmail.com> wrote:
>>> btw, because adding new module is rare case, manually modifying the
>>> bootclasspath.properties is not an issue?
>>>
>>> If so, can I conclude adding another property file with same update
>>> frequency as bootclasspath would be fine as well?
>>>
>>> Pls kindly correct me if my understanding is wrong.
>>>
>>> On Thu, Dec 25, 2008 at 9:05 PM, Pavel Pervov <pm...@gmail.com> wrote:
>>>> Wenlong,
>>>>
>>>> Note, that bootclasspath.properties is only changed on adding new
>>>> module. This is pretty rare occasion, I believe.
>>>>
>>>> WBR,
>>>> Pavel.
>>>>
>>>> On Thu, Dec 25, 2008 at 3:48 PM, Wenlong Li <we...@gmail.com> wrote:
>>>>> Thx for your advice. Alexey.
>>>>>
>>>>> Here I have one question: do you know how the bootclasspath.properties
>>>>> is maintained, manually or automatically?
>>>>>
>>>>> Another comment is I would like to treat the patch as DRLVM specific
>>>>> optimization, e.g., it targets for improving VM creation time. So that
>>>>> is possible to move all updates to DRLVM part to eliminate potential
>>>>> modularity and compatibility problem.
>>>>>
>>>>> thx,
>>>>> Wenlong
>>>>>
>>>>> On Thu, Dec 25, 2008 at 5:32 PM, Aleksey Shipilev
>>>>> <al...@gmail.com> wrote:
>>>>>> Hi, Wenlong.
>>>>>>
>>>>>> On Thu, Dec 25, 2008 at 11:49 AM, Wenlong Li <we...@gmail.com> wrote:
>>>>>>> btw, Alexey, Let's go back to discuss whether there is a need to
>>>>>>> include this feature in Harmony, given 17% performance gain in Linux
>>>>>>> when using your methodology. For windows test, I will double check the
>>>>>>> backgroud process as you pointed out.
>>>>>>
>>>>>> My opinion was already expressed after I had finished the tests from
>>>>>> my side: the boost can be achieved in specific conditions, so whether
>>>>>> it's worth including into Harmony really depends on how much mess the
>>>>>> patch would introduce besides the "performance boost". From what I
>>>>>> see, the patch obliges the maintainer to maintain the correct mapping
>>>>>> between jars and Java packages. This new feature is also spread
>>>>>> between Classlib and VM, but it seems like DRLVM specific. In this
>>>>>> case I would rather stay without the patch.
>>>>>>
>>>>>> Personally (if I'll be committer) I would accept the patch with two
>>>>>> serious modifications:
>>>>>> 1. Stay within DRLVM, do not introduce this feature into Classlib,
>>>>>> get the thing tested and evolved on DRLVM side. Otherwise it might
>>>>>> break the compatibility with other VMs.
>>>>>> 2. Make the mapping generated automatically (during build process?)
>>>>>> to free the burden for maintainers.
>>>>>>
>>>>>> Thanks,
>>>>>> Aleksey.
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: [VM] On-demand class library parsing is ready to commit
Posted by Wenlong Li <we...@gmail.com>.
Sorry, one more question: bootclasspath.properties is classlib
specific file, why we could not make a vm specific file manually? Just
curious to know the possible reason. :)
thx,
Wenlong
On Thu, Dec 25, 2008 at 10:00 PM, Pavel Pervov <pm...@gmail.com> wrote:
> If this would be VM-side automatically produced configuration file...
>
> WBR,
> Pavel.
>
> On Thu, Dec 25, 2008 at 4:58 PM, Wenlong Li <we...@gmail.com> wrote:
>> btw, because adding new module is rare case, manually modifying the
>> bootclasspath.properties is not an issue?
>>
>> If so, can I conclude adding another property file with same update
>> frequency as bootclasspath would be fine as well?
>>
>> Pls kindly correct me if my understanding is wrong.
>>
>> On Thu, Dec 25, 2008 at 9:05 PM, Pavel Pervov <pm...@gmail.com> wrote:
>>> Wenlong,
>>>
>>> Note, that bootclasspath.properties is only changed on adding new
>>> module. This is pretty rare occasion, I believe.
>>>
>>> WBR,
>>> Pavel.
>>>
>>> On Thu, Dec 25, 2008 at 3:48 PM, Wenlong Li <we...@gmail.com> wrote:
>>>> Thx for your advice. Alexey.
>>>>
>>>> Here I have one question: do you know how the bootclasspath.properties
>>>> is maintained, manually or automatically?
>>>>
>>>> Another comment is I would like to treat the patch as DRLVM specific
>>>> optimization, e.g., it targets for improving VM creation time. So that
>>>> is possible to move all updates to DRLVM part to eliminate potential
>>>> modularity and compatibility problem.
>>>>
>>>> thx,
>>>> Wenlong
>>>>
>>>> On Thu, Dec 25, 2008 at 5:32 PM, Aleksey Shipilev
>>>> <al...@gmail.com> wrote:
>>>>> Hi, Wenlong.
>>>>>
>>>>> On Thu, Dec 25, 2008 at 11:49 AM, Wenlong Li <we...@gmail.com> wrote:
>>>>>> btw, Alexey, Let's go back to discuss whether there is a need to
>>>>>> include this feature in Harmony, given 17% performance gain in Linux
>>>>>> when using your methodology. For windows test, I will double check the
>>>>>> backgroud process as you pointed out.
>>>>>
>>>>> My opinion was already expressed after I had finished the tests from
>>>>> my side: the boost can be achieved in specific conditions, so whether
>>>>> it's worth including into Harmony really depends on how much mess the
>>>>> patch would introduce besides the "performance boost". From what I
>>>>> see, the patch obliges the maintainer to maintain the correct mapping
>>>>> between jars and Java packages. This new feature is also spread
>>>>> between Classlib and VM, but it seems like DRLVM specific. In this
>>>>> case I would rather stay without the patch.
>>>>>
>>>>> Personally (if I'll be committer) I would accept the patch with two
>>>>> serious modifications:
>>>>> 1. Stay within DRLVM, do not introduce this feature into Classlib,
>>>>> get the thing tested and evolved on DRLVM side. Otherwise it might
>>>>> break the compatibility with other VMs.
>>>>> 2. Make the mapping generated automatically (during build process?)
>>>>> to free the burden for maintainers.
>>>>>
>>>>> Thanks,
>>>>> Aleksey.
>>>>>
>>>>
>>>
>>
>
Re: [VM] On-demand class library parsing is ready to commit
Posted by Pavel Pervov <pm...@gmail.com>.
If this would be VM-side automatically produced configuration file...
WBR,
Pavel.
On Thu, Dec 25, 2008 at 4:58 PM, Wenlong Li <we...@gmail.com> wrote:
> btw, because adding new module is rare case, manually modifying the
> bootclasspath.properties is not an issue?
>
> If so, can I conclude adding another property file with same update
> frequency as bootclasspath would be fine as well?
>
> Pls kindly correct me if my understanding is wrong.
>
> On Thu, Dec 25, 2008 at 9:05 PM, Pavel Pervov <pm...@gmail.com> wrote:
>> Wenlong,
>>
>> Note, that bootclasspath.properties is only changed on adding new
>> module. This is pretty rare occasion, I believe.
>>
>> WBR,
>> Pavel.
>>
>> On Thu, Dec 25, 2008 at 3:48 PM, Wenlong Li <we...@gmail.com> wrote:
>>> Thx for your advice. Alexey.
>>>
>>> Here I have one question: do you know how the bootclasspath.properties
>>> is maintained, manually or automatically?
>>>
>>> Another comment is I would like to treat the patch as DRLVM specific
>>> optimization, e.g., it targets for improving VM creation time. So that
>>> is possible to move all updates to DRLVM part to eliminate potential
>>> modularity and compatibility problem.
>>>
>>> thx,
>>> Wenlong
>>>
>>> On Thu, Dec 25, 2008 at 5:32 PM, Aleksey Shipilev
>>> <al...@gmail.com> wrote:
>>>> Hi, Wenlong.
>>>>
>>>> On Thu, Dec 25, 2008 at 11:49 AM, Wenlong Li <we...@gmail.com> wrote:
>>>>> btw, Alexey, Let's go back to discuss whether there is a need to
>>>>> include this feature in Harmony, given 17% performance gain in Linux
>>>>> when using your methodology. For windows test, I will double check the
>>>>> backgroud process as you pointed out.
>>>>
>>>> My opinion was already expressed after I had finished the tests from
>>>> my side: the boost can be achieved in specific conditions, so whether
>>>> it's worth including into Harmony really depends on how much mess the
>>>> patch would introduce besides the "performance boost". From what I
>>>> see, the patch obliges the maintainer to maintain the correct mapping
>>>> between jars and Java packages. This new feature is also spread
>>>> between Classlib and VM, but it seems like DRLVM specific. In this
>>>> case I would rather stay without the patch.
>>>>
>>>> Personally (if I'll be committer) I would accept the patch with two
>>>> serious modifications:
>>>> 1. Stay within DRLVM, do not introduce this feature into Classlib,
>>>> get the thing tested and evolved on DRLVM side. Otherwise it might
>>>> break the compatibility with other VMs.
>>>> 2. Make the mapping generated automatically (during build process?)
>>>> to free the burden for maintainers.
>>>>
>>>> Thanks,
>>>> Aleksey.
>>>>
>>>
>>
>
Re: [VM] On-demand class library parsing is ready to commit
Posted by Wenlong Li <we...@gmail.com>.
btw, because adding new module is rare case, manually modifying the
bootclasspath.properties is not an issue?
If so, can I conclude adding another property file with same update
frequency as bootclasspath would be fine as well?
Pls kindly correct me if my understanding is wrong.
On Thu, Dec 25, 2008 at 9:05 PM, Pavel Pervov <pm...@gmail.com> wrote:
> Wenlong,
>
> Note, that bootclasspath.properties is only changed on adding new
> module. This is pretty rare occasion, I believe.
>
> WBR,
> Pavel.
>
> On Thu, Dec 25, 2008 at 3:48 PM, Wenlong Li <we...@gmail.com> wrote:
>> Thx for your advice. Alexey.
>>
>> Here I have one question: do you know how the bootclasspath.properties
>> is maintained, manually or automatically?
>>
>> Another comment is I would like to treat the patch as DRLVM specific
>> optimization, e.g., it targets for improving VM creation time. So that
>> is possible to move all updates to DRLVM part to eliminate potential
>> modularity and compatibility problem.
>>
>> thx,
>> Wenlong
>>
>> On Thu, Dec 25, 2008 at 5:32 PM, Aleksey Shipilev
>> <al...@gmail.com> wrote:
>>> Hi, Wenlong.
>>>
>>> On Thu, Dec 25, 2008 at 11:49 AM, Wenlong Li <we...@gmail.com> wrote:
>>>> btw, Alexey, Let's go back to discuss whether there is a need to
>>>> include this feature in Harmony, given 17% performance gain in Linux
>>>> when using your methodology. For windows test, I will double check the
>>>> backgroud process as you pointed out.
>>>
>>> My opinion was already expressed after I had finished the tests from
>>> my side: the boost can be achieved in specific conditions, so whether
>>> it's worth including into Harmony really depends on how much mess the
>>> patch would introduce besides the "performance boost". From what I
>>> see, the patch obliges the maintainer to maintain the correct mapping
>>> between jars and Java packages. This new feature is also spread
>>> between Classlib and VM, but it seems like DRLVM specific. In this
>>> case I would rather stay without the patch.
>>>
>>> Personally (if I'll be committer) I would accept the patch with two
>>> serious modifications:
>>> 1. Stay within DRLVM, do not introduce this feature into Classlib,
>>> get the thing tested and evolved on DRLVM side. Otherwise it might
>>> break the compatibility with other VMs.
>>> 2. Make the mapping generated automatically (during build process?)
>>> to free the burden for maintainers.
>>>
>>> Thanks,
>>> Aleksey.
>>>
>>
>
Re: [VM] On-demand class library parsing is ready to commit
Posted by Wenlong Li <we...@gmail.com>.
If so, my modulelibrarymapping.properties should also rarely change as
well. If new module is added, both the bootclasspath.properties and
modulelibrarymapping.properties are up to change.
For the modulelibrarymapping.properties, what user should do is just
manually fetching the exported package info from manifest file and put
it into this file.
I just wonder whether modulelibrarymapping.properties should be
automatically produced or manually updated like
bootclasspath.properties.
thx,
Wenlong
On Thu, Dec 25, 2008 at 9:05 PM, Pavel Pervov <pm...@gmail.com> wrote:
> Wenlong,
>
> Note, that bootclasspath.properties is only changed on adding new
> module. This is pretty rare occasion, I believe.
>
> WBR,
> Pavel.
>
> On Thu, Dec 25, 2008 at 3:48 PM, Wenlong Li <we...@gmail.com> wrote:
>> Thx for your advice. Alexey.
>>
>> Here I have one question: do you know how the bootclasspath.properties
>> is maintained, manually or automatically?
>>
>> Another comment is I would like to treat the patch as DRLVM specific
>> optimization, e.g., it targets for improving VM creation time. So that
>> is possible to move all updates to DRLVM part to eliminate potential
>> modularity and compatibility problem.
>>
>> thx,
>> Wenlong
>>
>> On Thu, Dec 25, 2008 at 5:32 PM, Aleksey Shipilev
>> <al...@gmail.com> wrote:
>>> Hi, Wenlong.
>>>
>>> On Thu, Dec 25, 2008 at 11:49 AM, Wenlong Li <we...@gmail.com> wrote:
>>>> btw, Alexey, Let's go back to discuss whether there is a need to
>>>> include this feature in Harmony, given 17% performance gain in Linux
>>>> when using your methodology. For windows test, I will double check the
>>>> backgroud process as you pointed out.
>>>
>>> My opinion was already expressed after I had finished the tests from
>>> my side: the boost can be achieved in specific conditions, so whether
>>> it's worth including into Harmony really depends on how much mess the
>>> patch would introduce besides the "performance boost". From what I
>>> see, the patch obliges the maintainer to maintain the correct mapping
>>> between jars and Java packages. This new feature is also spread
>>> between Classlib and VM, but it seems like DRLVM specific. In this
>>> case I would rather stay without the patch.
>>>
>>> Personally (if I'll be committer) I would accept the patch with two
>>> serious modifications:
>>> 1. Stay within DRLVM, do not introduce this feature into Classlib,
>>> get the thing tested and evolved on DRLVM side. Otherwise it might
>>> break the compatibility with other VMs.
>>> 2. Make the mapping generated automatically (during build process?)
>>> to free the burden for maintainers.
>>>
>>> Thanks,
>>> Aleksey.
>>>
>>
>
Re: [VM] On-demand class library parsing is ready to commit
Posted by Pavel Pervov <pm...@gmail.com>.
Wenlong,
Note, that bootclasspath.properties is only changed on adding new
module. This is pretty rare occasion, I believe.
WBR,
Pavel.
On Thu, Dec 25, 2008 at 3:48 PM, Wenlong Li <we...@gmail.com> wrote:
> Thx for your advice. Alexey.
>
> Here I have one question: do you know how the bootclasspath.properties
> is maintained, manually or automatically?
>
> Another comment is I would like to treat the patch as DRLVM specific
> optimization, e.g., it targets for improving VM creation time. So that
> is possible to move all updates to DRLVM part to eliminate potential
> modularity and compatibility problem.
>
> thx,
> Wenlong
>
> On Thu, Dec 25, 2008 at 5:32 PM, Aleksey Shipilev
> <al...@gmail.com> wrote:
>> Hi, Wenlong.
>>
>> On Thu, Dec 25, 2008 at 11:49 AM, Wenlong Li <we...@gmail.com> wrote:
>>> btw, Alexey, Let's go back to discuss whether there is a need to
>>> include this feature in Harmony, given 17% performance gain in Linux
>>> when using your methodology. For windows test, I will double check the
>>> backgroud process as you pointed out.
>>
>> My opinion was already expressed after I had finished the tests from
>> my side: the boost can be achieved in specific conditions, so whether
>> it's worth including into Harmony really depends on how much mess the
>> patch would introduce besides the "performance boost". From what I
>> see, the patch obliges the maintainer to maintain the correct mapping
>> between jars and Java packages. This new feature is also spread
>> between Classlib and VM, but it seems like DRLVM specific. In this
>> case I would rather stay without the patch.
>>
>> Personally (if I'll be committer) I would accept the patch with two
>> serious modifications:
>> 1. Stay within DRLVM, do not introduce this feature into Classlib,
>> get the thing tested and evolved on DRLVM side. Otherwise it might
>> break the compatibility with other VMs.
>> 2. Make the mapping generated automatically (during build process?)
>> to free the burden for maintainers.
>>
>> Thanks,
>> Aleksey.
>>
>
Re: [VM] On-demand class library parsing is ready to commit
Posted by Wenlong Li <we...@gmail.com>.
Thx for your advice. Alexey.
Here I have one question: do you know how the bootclasspath.properties
is maintained, manually or automatically?
Another comment is I would like to treat the patch as DRLVM specific
optimization, e.g., it targets for improving VM creation time. So that
is possible to move all updates to DRLVM part to eliminate potential
modularity and compatibility problem.
thx,
Wenlong
On Thu, Dec 25, 2008 at 5:32 PM, Aleksey Shipilev
<al...@gmail.com> wrote:
> Hi, Wenlong.
>
> On Thu, Dec 25, 2008 at 11:49 AM, Wenlong Li <we...@gmail.com> wrote:
>> btw, Alexey, Let's go back to discuss whether there is a need to
>> include this feature in Harmony, given 17% performance gain in Linux
>> when using your methodology. For windows test, I will double check the
>> backgroud process as you pointed out.
>
> My opinion was already expressed after I had finished the tests from
> my side: the boost can be achieved in specific conditions, so whether
> it's worth including into Harmony really depends on how much mess the
> patch would introduce besides the "performance boost". From what I
> see, the patch obliges the maintainer to maintain the correct mapping
> between jars and Java packages. This new feature is also spread
> between Classlib and VM, but it seems like DRLVM specific. In this
> case I would rather stay without the patch.
>
> Personally (if I'll be committer) I would accept the patch with two
> serious modifications:
> 1. Stay within DRLVM, do not introduce this feature into Classlib,
> get the thing tested and evolved on DRLVM side. Otherwise it might
> break the compatibility with other VMs.
> 2. Make the mapping generated automatically (during build process?)
> to free the burden for maintainers.
>
> Thanks,
> Aleksey.
>
Re: [VM] On-demand class library parsing is ready to commit
Posted by Aleksey Shipilev <al...@gmail.com>.
Hi, Wenlong.
On Thu, Dec 25, 2008 at 11:49 AM, Wenlong Li <we...@gmail.com> wrote:
> btw, Alexey, Let's go back to discuss whether there is a need to
> include this feature in Harmony, given 17% performance gain in Linux
> when using your methodology. For windows test, I will double check the
> backgroud process as you pointed out.
My opinion was already expressed after I had finished the tests from
my side: the boost can be achieved in specific conditions, so whether
it's worth including into Harmony really depends on how much mess the
patch would introduce besides the "performance boost". From what I
see, the patch obliges the maintainer to maintain the correct mapping
between jars and Java packages. This new feature is also spread
between Classlib and VM, but it seems like DRLVM specific. In this
case I would rather stay without the patch.
Personally (if I'll be committer) I would accept the patch with two
serious modifications:
1. Stay within DRLVM, do not introduce this feature into Classlib,
get the thing tested and evolved on DRLVM side. Otherwise it might
break the compatibility with other VMs.
2. Make the mapping generated automatically (during build process?)
to free the burden for maintainers.
Thanks,
Aleksey.
Re: [VM] On-demand class library parsing is ready to commit
Posted by Wenlong Li <we...@gmail.com>.
btw, Alexey, Let's go back to discuss whether there is a need to
include this feature in Harmony, given 17% performance gain in Linux
when using your methodology. For windows test, I will double check the
backgroud process as you pointed out.
thx,
Wenlong
On Thu, Dec 25, 2008 at 4:27 PM, Aleksey Shipilev
<al...@gmail.com> wrote:
> Hi,
>
> On Thu, Dec 25, 2008 at 11:22 AM, Wenlong Li <we...@gmail.com> wrote:
>> Do you suggest conducting experiments in Linux always?
>> You may restart your windows machine, and then teerminating all
>> running processes and run the Harmony or SUN to see the response time
>> before Hello World appears.
>
> I doubt that's the problem with Windows at all, I bet it's much more
> about exact Windows configuration you working on. 20 sec startup seem
> to be really bad if there no obstacles for JVM to run (like antivirus
> which blocks JARs while loading). I wouldn't trust this machine :)
>
> Thanks,
> Aleksey.
>
Re: [VM] On-demand class library parsing is ready to commit
Posted by Aleksey Shipilev <al...@gmail.com>.
Hi,
On Thu, Dec 25, 2008 at 11:22 AM, Wenlong Li <we...@gmail.com> wrote:
> Do you suggest conducting experiments in Linux always?
> You may restart your windows machine, and then teerminating all
> running processes and run the Harmony or SUN to see the response time
> before Hello World appears.
I doubt that's the problem with Windows at all, I bet it's much more
about exact Windows configuration you working on. 20 sec startup seem
to be really bad if there no obstacles for JVM to run (like antivirus
which blocks JARs while loading). I wouldn't trust this machine :)
Thanks,
Aleksey.
Re: [VM] On-demand class library parsing is ready to commit
Posted by Wenlong Li <we...@gmail.com>.
On Thu, Dec 25, 2008 at 4:06 PM, Aleksey Shipilev
<al...@gmail.com> wrote:
> Hi, Wenlong!
>
> Thanks for doing the experiment. A couple of notes:
>
> On Thu, Dec 25, 2008 at 10:16 AM, Wenlong Li <we...@gmail.com> wrote:
>> Harmony-721077
> Release or debug build?
Release build in all my tests
>
>> The VM destroy module takes around 1 second for both cold and warm-up
>> runnings, but not stable (I issue another JIRA for this problem)
> Finalizers?
I don't know....
>
>> Here is the summary:
>> (d) The performance benefit of this optimization varies with different
>> systems (Alexey sees 17% gain, and I reported 30% here)
> Nope, that's not the difference in performance results, that's the
> difference in the way of treating results, basic formulas are like
> following:
>
> Speedup = optimizedPerformance / baselinePerformance = baselineTime /
> optimizedTime
> Boost = Speedup - 1
>
> In your case:
> a. cold: Boost = 3375/2849 - 1 =~ 0.184 = 18.4%
> b. warm: Boost = 1825/1818 - 1 =~ 0 = 0%
>
>> On my windows
>> system (2.83 GHz Core 2 Quad-core, 3GB main memory, 250GB SATA HD at
>> 7200 RPM, 8MB disk cache, WinXp system, Harmony-713673). This
>> optimization reduces VM creation time from 20+ seconds to 3+ seconds
>> in a cold running case).
> How come?! 20 secs to start HWA on JVM? Is your environment suitable
> for doing startup measurements (e.g. online antiviruses stopped,
> enough physical memory available, etc.)? This result doesn't look
> trustworthy.
Do you suggest conducting experiments in Linux always?
You may restart your windows machine, and then teerminating all
running processes and run the Harmony or SUN to see the response time
before Hello World appears.
>
>> (c) As this optimization is for VM creation module, it doesn't help
>> SPECJVM2008 startup benchmarks (e.g., no performance gain)
> Not obvious enough :) SPECjvm2008 framework invokes another instance
> of JVM each time startup benchmark is done, so it _is_ a good startup
> benchmark. The reason why this patch does not give performance boost
> was already described.
>
>
> Thanks,
> Aleksey.
>
>> On Mon, Dec 22, 2008 at 4:04 AM, Aleksey Shipilev
>> <al...@gmail.com> wrote:
>>> Hi Wenlong,
>>>
>>> I had some performance experiments with your patch. The test system is:
>>> - Pentium D 820 2.8 Ghz / 2 Gb DDR2-667
>>> - WD 3200KS, 320 Gb, 16 Mb cache
>>> - Gentoo Linux x86, 2.6.23
>>> - Harmony r728459
>>> - SPECjvm2008
>>>
>>> To recreate the stressful conditions over and over the simple script
>>> was written [1]. The script invalidates the caches before actually
>>> starting the workload: re-reads the same 64 Mb file a couple of times
>>> to fill out on-HDD cache, invalidating VFS block caches first to make
>>> sure the data is really requested from the disk.
>>>
>>> On HWA [2] these performance results were produced:
>>>
>>> "cold-start" (invalidate caches):
>>> clean: (5.24 +- 0.28) secs
>>> ondemand: (4.49 +- 0.17) secs
>>>
>>> "warm-start" (don't invalidate caches);
>>> clean: (2.82 +- 0.01) secs
>>> ondemand: (2.80 +- 0.02) secs
>>>
>>> That is, on-demand patch does bring +17% (-+9%) improvement on HWA
>>> when running with flushed caches, and does not bring any performance
>>> improvement in warm mode.
>>>
>>> As I mentioned several times, this test does not reflect the real
>>> performance end user would perceive, so I took two SPECjvm2008:startup
>>> benchmarks and run each of them 10x10 times.
>>>
>>> SPECjvm2008:startup.helloworld, "cold start":
>>> clean: (8.93 +- 0.21) ops/min
>>> ondemand: (9.04 +- 0.03) ops/min
>>>
>>> SPECjvm2008:startup.compiler.compiler, "cold start":
>>> clean: (1.44 +- 0.05) ops/min
>>> ondemand: (1.42 +- 0.04) ops/min
>>>
>>> As you can see even in very stressful situation there's no boost. I
>>> would find these performance results unconvincing to change the
>>> infrastructure of boolclasspath resolution. Am I missing something
>>> important?
>>>
>>> Thanks,
>>> Aleksey.
>>>
>>> [1] run.sh
>>> #!/bin/bash
>>>
>>> R=`pwd`
>>>
>>> JAVA=$R/platforms/builds/harmony-release-clean/jdk/jre/bin/java
>>> #JAVA=$R/platforms/builds/harmony-release-ondemand/jdk/jre/bin/java
>>> JAVA_OPTS="-Xmx1024M -Xms1024M"
>>>
>>> for T in `seq 1 10`; do
>>>
>>> echo "*************** EXECUTING ITERATION $T ****************"
>>>
>>> # invalidate HDD caches
>>> # - need to replace all entries in LRU HDD cache
>>> # - flush the kernel VFS cache first to ensure the data
>>> would be read from disk
>>>
>>> echo "Flushing caches"
>>> for I in `seq 1 5`; do
>>> sync
>>> echo 3 > /proc/sys/vm/drop_caches
>>>
>>> dd if=cachekiller.file of=/dev/null > /dev/null 2>&1
>>> done
>>>
>>> echo "Executing."
>>>
>>> # HelloWorld
>>> /usr/bin/time $JAVA $JAVA_OPTS -cp benchmarks/ HelloWorld 2>&1
>>>
>>> # SPECjvm2008
>>> #cd $R/benchmarks/storage/SPECjvm2008
>>> #/usr/bin/time $JAVA $JAVA_OPTS -Djava.awt.headless=true -jar
>>> SPECjvm2008.jar -ikv -i 10 startup.compiler.compiler 2>&1
>>>
>>> echo ""
>>> done
>>>
>>> [2] HelloWorld.java
>>> public class HelloWorld {
>>> public static void main(String[] args) {
>>> System.out.println("Hello, world!");
>>> }
>>> }
>>>
>>>
>>> On Sun, Dec 21, 2008 at 6:02 AM, Wenlong Li <we...@gmail.com> wrote:
>>>> On Sat, Dec 20, 2008 at 7:10 PM, Alexei Fedotov
>>>> <al...@gmail.com> wrote:
>>>>> Wenlong,
>>>>> Thanks for removing the commented code.
>>>>>
>>>>> There are several VMs which make use of the Harmony class library,
>>>>> e.g. Harmony VM, J9, Android Dalvik, etc. Your change is Harmony VM
>>>>> specific, isn't it? If it is, then it's better to keep related changes
>>>>> in the VM module. If it is not, then it might be a good idea to keep
>>>>> the changes in the class library module unless other VMs already has
>>>>> such optimization in their code.
>>>> [Wenlong] Though at this moment, you can think on-demand class parsing
>>>> is a specif optimization from your point of view. I believe it could
>>>> be a general technique, e.g., it can be easily deployed in other
>>>> runtime systems. Current VM also depends on the luniglobal.c in
>>>> working_classlib to get all class libraries/modules. e.g., there is a
>>>> cross-module dependence between classlib and VM. When user wants to
>>>> add new module, they should manually change the
>>>> bootclasspath.properties, while if applying this patch, user should
>>>> revise my added property file instead of the bootclasspath.properties.
>>>> I understand modifying bootclasspath file may be a specification.
>>>>>
>>>>> In any case crossing module boundary would make class library users
>>>>> think more than once or even write some code. Is it technically
>>>>> possible to prepare a patch which does not change module boundaries?
>>>>> What do you think?
>>>> [Wenlong] Yes, it is possible from technical perspective, but a little
>>>> complicated. I can think about it. :)
>>>>
>>>>>
>>>>> As for your performance experiments, which particular test are your
>>>>> measuring? It is bootclasspath-unpretentious "Hello, world", isn't it?
>>>> [Wenlong] My startup means the work executed before running user's
>>>> computation. That is, the vm creation time. I manually add
>>>> instrumentation code for execution time in JNI_CreateJavaVM of
>>>> JNI.cpp. This startup work is common for any benchmarks. My experiment
>>>> was conducted on both Windows and Linux system. Please see my previous
>>>> message about performance gain from this optimization.
>>>>
>>>> Thx,
>>>> Wenlong
>>>>>
>>>>> Thanks!
>>>>>
>>>>> On Sat, Dec 20, 2008 at 2:19 AM, Wenlong Li <we...@gmail.com> wrote:
>>>>>> On Sat, Dec 20, 2008 at 12:42 AM, Alexei Fedotov
>>>>>> <al...@gmail.com> wrote:
>>>>>>> Wenlong,
>>>>>>> Have I missed a discussion of the proposed design? I see that you
>>>>>>> expose a new public interface:
>>>>>>> /**
>>>>>>> * @map the jar with exported package in the pending jar list for
>>>>>>> on-demand jar parsing
>>>>>>> * Key is the jar, and value is the package exported by this jar
>>>>>>> */
>>>>>>> DECLARE_OPEN(void, vm_properties_set_pending_jar, (const char* key,
>>>>>>> const char* value));
>>>>>>>
>>>>>>> Did you mean "Maps" instead of "@map"? Strangely the word "pending"
>>>>>>> disappeared from the name of the wrapping VMI interface
>>>>>>> SetJarPackageMapping . Why should we extend both OPEN and VMI
>>>>>>> interfaces with the same function? Why did you put your code into
>>>>>>> working_classlib/modules/luni/src/main/native/luni/shared/luniglob.c,
>>>>>>> thus introducing another dependency between VM and class library?
>>>>>> [Wenlong] The boot class path is defined in luniglobal.c in Harmony,
>>>>>> and it also has dependence with VM. In my understanding, my patch is
>>>>>> related to boot class path determination, so I also put my code in
>>>>>> luniglobal.c, and use VMI interface to communicate with VM.
>>>>>>
>>>>>>>
>>>>>>> + //rcSetProperty = (*vmInterface)->SetJarPackageMapping
>>>>>>> (vmInterface, jarName, jarValue);
>>>>>>> + /*
>>>>>>> + hymem_free_memory(jarName);
>>>>>>> + hymem_free_memory(jarValue);
>>>>>>> + */
>>>>>>> Should we really commit the commented code?
>>>>>>> Thanks.
>>>>>>
>>>>>> [Wenlong] Please see my latest version of patch in the list. Such
>>>>>> commented code has been removed.
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Dec 19, 2008 at 6:59 PM, Tim Ellison <t....@gmail.com> wrote:
>>>>>>>> I was hoping that somebody else would comment first, so I don't have to
>>>>>>>> be the grumpy one all the time :-)
>>>>>>>>
>>>>>>>> As I said before, this is good prototyping work...
>>>>>>>>
>>>>>>>> Wenlong Li wrote:
>>>>>>>>> I did the pre-commit test on the patch of on-demand class library
>>>>>>>>> parsing (https://issues.apache.org/jira/browse/HARMONY-6039), and it
>>>>>>>>> works well now.
>>>>>>>>> Can Harmony incorporate this feature?
>>>>>>>>
>>>>>>>> I'm not sure it is ready for committing to the head stream yet.
>>>>>>>>
>>>>>>>>> Via on-demand class parsing, we can reduce startup time from 20+
>>>>>>>>> seconds to 3 seconds for cold runing, and 170 ms to 140 ms for warm-up
>>>>>>>>> running on Core 2 Duo with Windows.
>>>>>>>>
>>>>>>>> Can you tell me how to reproduce 20+sec cold start-up? I haven't seen
>>>>>>>> anything like that in my simple tests.
>>>>>>>>
>>>>>>>>> After applying the patch, please note there is some change to add new modules.
>>>>>>>>> (1) If you want to add new modules/libraries, please don't put them in
>>>>>>>>> the bootclasspath.properties file. This file now only saves modules
>>>>>>>>> needed during startup (the VM startup only accesses class libraries in
>>>>>>>>> eight modules)
>>>>>>>>
>>>>>>>> That would break too much. How about creating a new file rather than
>>>>>>>> re-purposing an existing file with different semantics? This file is
>>>>>>>> used by Jikes, IBM VME, the Eclipse plug-in, at least.
>>>>>>>>
>>>>>>>>> (2) For new modules/libraries, please put them in the
>>>>>>>>> modulelibrarymapping.properties file. You should specify the module
>>>>>>>>> name and its exported class library. Here is one example:
>>>>>>>>> math.jar=java.math, where "math.jar" means the module name, and
>>>>>>>>> "java.math" means the class libraries this module exports.
>>>>>>>>
>>>>>>>> As we discussed on another thread, its unclear if the time is spent in
>>>>>>>> following the slow indexing through the classpath/JAR directories, or
>>>>>>>> whether it is speed of loading bytes once we know what we need. I think
>>>>>>>> that it is premature to abandon the JAR manifest data as the principal
>>>>>>>> source of metadata until we understand the problem this solves.
>>>>>>>>
>>>>>>>> Can we measure where the time is spent in the current implementation?
>>>>>>>> I think it will help guide this approach to a better solution.
>>>>>>>> What tools do you recommend for profiling start-up?
>>>>>>>>
>>>>>>>> Regards
>>>>>>>> Tim
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> С уважением,
>>>>>>> Алексей Федотов,
>>>>>>> ЗАО «Телеком Экспресс»
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> С уважением,
>>>>> Алексей Федотов,
>>>>> ЗАО «Телеком Экспресс»
>>>>>
>>>>
>>>
>>
>
Re: [VM] On-demand class library parsing is ready to commit
Posted by Aleksey Shipilev <al...@gmail.com>.
Hi, Wenlong!
Thanks for doing the experiment. A couple of notes:
On Thu, Dec 25, 2008 at 10:16 AM, Wenlong Li <we...@gmail.com> wrote:
> Harmony-721077
Release or debug build?
> The VM destroy module takes around 1 second for both cold and warm-up
> runnings, but not stable (I issue another JIRA for this problem)
Finalizers?
> Here is the summary:
> (d) The performance benefit of this optimization varies with different
> systems (Alexey sees 17% gain, and I reported 30% here)
Nope, that's not the difference in performance results, that's the
difference in the way of treating results, basic formulas are like
following:
Speedup = optimizedPerformance / baselinePerformance = baselineTime /
optimizedTime
Boost = Speedup - 1
In your case:
a. cold: Boost = 3375/2849 - 1 =~ 0.184 = 18.4%
b. warm: Boost = 1825/1818 - 1 =~ 0 = 0%
> On my windows
> system (2.83 GHz Core 2 Quad-core, 3GB main memory, 250GB SATA HD at
> 7200 RPM, 8MB disk cache, WinXp system, Harmony-713673). This
> optimization reduces VM creation time from 20+ seconds to 3+ seconds
> in a cold running case).
How come?! 20 secs to start HWA on JVM? Is your environment suitable
for doing startup measurements (e.g. online antiviruses stopped,
enough physical memory available, etc.)? This result doesn't look
trustworthy.
> (c) As this optimization is for VM creation module, it doesn't help
> SPECJVM2008 startup benchmarks (e.g., no performance gain)
Not obvious enough :) SPECjvm2008 framework invokes another instance
of JVM each time startup benchmark is done, so it _is_ a good startup
benchmark. The reason why this patch does not give performance boost
was already described.
Thanks,
Aleksey.
> On Mon, Dec 22, 2008 at 4:04 AM, Aleksey Shipilev
> <al...@gmail.com> wrote:
>> Hi Wenlong,
>>
>> I had some performance experiments with your patch. The test system is:
>> - Pentium D 820 2.8 Ghz / 2 Gb DDR2-667
>> - WD 3200KS, 320 Gb, 16 Mb cache
>> - Gentoo Linux x86, 2.6.23
>> - Harmony r728459
>> - SPECjvm2008
>>
>> To recreate the stressful conditions over and over the simple script
>> was written [1]. The script invalidates the caches before actually
>> starting the workload: re-reads the same 64 Mb file a couple of times
>> to fill out on-HDD cache, invalidating VFS block caches first to make
>> sure the data is really requested from the disk.
>>
>> On HWA [2] these performance results were produced:
>>
>> "cold-start" (invalidate caches):
>> clean: (5.24 +- 0.28) secs
>> ondemand: (4.49 +- 0.17) secs
>>
>> "warm-start" (don't invalidate caches);
>> clean: (2.82 +- 0.01) secs
>> ondemand: (2.80 +- 0.02) secs
>>
>> That is, on-demand patch does bring +17% (-+9%) improvement on HWA
>> when running with flushed caches, and does not bring any performance
>> improvement in warm mode.
>>
>> As I mentioned several times, this test does not reflect the real
>> performance end user would perceive, so I took two SPECjvm2008:startup
>> benchmarks and run each of them 10x10 times.
>>
>> SPECjvm2008:startup.helloworld, "cold start":
>> clean: (8.93 +- 0.21) ops/min
>> ondemand: (9.04 +- 0.03) ops/min
>>
>> SPECjvm2008:startup.compiler.compiler, "cold start":
>> clean: (1.44 +- 0.05) ops/min
>> ondemand: (1.42 +- 0.04) ops/min
>>
>> As you can see even in very stressful situation there's no boost. I
>> would find these performance results unconvincing to change the
>> infrastructure of boolclasspath resolution. Am I missing something
>> important?
>>
>> Thanks,
>> Aleksey.
>>
>> [1] run.sh
>> #!/bin/bash
>>
>> R=`pwd`
>>
>> JAVA=$R/platforms/builds/harmony-release-clean/jdk/jre/bin/java
>> #JAVA=$R/platforms/builds/harmony-release-ondemand/jdk/jre/bin/java
>> JAVA_OPTS="-Xmx1024M -Xms1024M"
>>
>> for T in `seq 1 10`; do
>>
>> echo "*************** EXECUTING ITERATION $T ****************"
>>
>> # invalidate HDD caches
>> # - need to replace all entries in LRU HDD cache
>> # - flush the kernel VFS cache first to ensure the data
>> would be read from disk
>>
>> echo "Flushing caches"
>> for I in `seq 1 5`; do
>> sync
>> echo 3 > /proc/sys/vm/drop_caches
>>
>> dd if=cachekiller.file of=/dev/null > /dev/null 2>&1
>> done
>>
>> echo "Executing."
>>
>> # HelloWorld
>> /usr/bin/time $JAVA $JAVA_OPTS -cp benchmarks/ HelloWorld 2>&1
>>
>> # SPECjvm2008
>> #cd $R/benchmarks/storage/SPECjvm2008
>> #/usr/bin/time $JAVA $JAVA_OPTS -Djava.awt.headless=true -jar
>> SPECjvm2008.jar -ikv -i 10 startup.compiler.compiler 2>&1
>>
>> echo ""
>> done
>>
>> [2] HelloWorld.java
>> public class HelloWorld {
>> public static void main(String[] args) {
>> System.out.println("Hello, world!");
>> }
>> }
>>
>>
>> On Sun, Dec 21, 2008 at 6:02 AM, Wenlong Li <we...@gmail.com> wrote:
>>> On Sat, Dec 20, 2008 at 7:10 PM, Alexei Fedotov
>>> <al...@gmail.com> wrote:
>>>> Wenlong,
>>>> Thanks for removing the commented code.
>>>>
>>>> There are several VMs which make use of the Harmony class library,
>>>> e.g. Harmony VM, J9, Android Dalvik, etc. Your change is Harmony VM
>>>> specific, isn't it? If it is, then it's better to keep related changes
>>>> in the VM module. If it is not, then it might be a good idea to keep
>>>> the changes in the class library module unless other VMs already has
>>>> such optimization in their code.
>>> [Wenlong] Though at this moment, you can think on-demand class parsing
>>> is a specif optimization from your point of view. I believe it could
>>> be a general technique, e.g., it can be easily deployed in other
>>> runtime systems. Current VM also depends on the luniglobal.c in
>>> working_classlib to get all class libraries/modules. e.g., there is a
>>> cross-module dependence between classlib and VM. When user wants to
>>> add new module, they should manually change the
>>> bootclasspath.properties, while if applying this patch, user should
>>> revise my added property file instead of the bootclasspath.properties.
>>> I understand modifying bootclasspath file may be a specification.
>>>>
>>>> In any case crossing module boundary would make class library users
>>>> think more than once or even write some code. Is it technically
>>>> possible to prepare a patch which does not change module boundaries?
>>>> What do you think?
>>> [Wenlong] Yes, it is possible from technical perspective, but a little
>>> complicated. I can think about it. :)
>>>
>>>>
>>>> As for your performance experiments, which particular test are your
>>>> measuring? It is bootclasspath-unpretentious "Hello, world", isn't it?
>>> [Wenlong] My startup means the work executed before running user's
>>> computation. That is, the vm creation time. I manually add
>>> instrumentation code for execution time in JNI_CreateJavaVM of
>>> JNI.cpp. This startup work is common for any benchmarks. My experiment
>>> was conducted on both Windows and Linux system. Please see my previous
>>> message about performance gain from this optimization.
>>>
>>> Thx,
>>> Wenlong
>>>>
>>>> Thanks!
>>>>
>>>> On Sat, Dec 20, 2008 at 2:19 AM, Wenlong Li <we...@gmail.com> wrote:
>>>>> On Sat, Dec 20, 2008 at 12:42 AM, Alexei Fedotov
>>>>> <al...@gmail.com> wrote:
>>>>>> Wenlong,
>>>>>> Have I missed a discussion of the proposed design? I see that you
>>>>>> expose a new public interface:
>>>>>> /**
>>>>>> * @map the jar with exported package in the pending jar list for
>>>>>> on-demand jar parsing
>>>>>> * Key is the jar, and value is the package exported by this jar
>>>>>> */
>>>>>> DECLARE_OPEN(void, vm_properties_set_pending_jar, (const char* key,
>>>>>> const char* value));
>>>>>>
>>>>>> Did you mean "Maps" instead of "@map"? Strangely the word "pending"
>>>>>> disappeared from the name of the wrapping VMI interface
>>>>>> SetJarPackageMapping . Why should we extend both OPEN and VMI
>>>>>> interfaces with the same function? Why did you put your code into
>>>>>> working_classlib/modules/luni/src/main/native/luni/shared/luniglob.c,
>>>>>> thus introducing another dependency between VM and class library?
>>>>> [Wenlong] The boot class path is defined in luniglobal.c in Harmony,
>>>>> and it also has dependence with VM. In my understanding, my patch is
>>>>> related to boot class path determination, so I also put my code in
>>>>> luniglobal.c, and use VMI interface to communicate with VM.
>>>>>
>>>>>>
>>>>>> + //rcSetProperty = (*vmInterface)->SetJarPackageMapping
>>>>>> (vmInterface, jarName, jarValue);
>>>>>> + /*
>>>>>> + hymem_free_memory(jarName);
>>>>>> + hymem_free_memory(jarValue);
>>>>>> + */
>>>>>> Should we really commit the commented code?
>>>>>> Thanks.
>>>>>
>>>>> [Wenlong] Please see my latest version of patch in the list. Such
>>>>> commented code has been removed.
>>>>>>
>>>>>>
>>>>>> On Fri, Dec 19, 2008 at 6:59 PM, Tim Ellison <t....@gmail.com> wrote:
>>>>>>> I was hoping that somebody else would comment first, so I don't have to
>>>>>>> be the grumpy one all the time :-)
>>>>>>>
>>>>>>> As I said before, this is good prototyping work...
>>>>>>>
>>>>>>> Wenlong Li wrote:
>>>>>>>> I did the pre-commit test on the patch of on-demand class library
>>>>>>>> parsing (https://issues.apache.org/jira/browse/HARMONY-6039), and it
>>>>>>>> works well now.
>>>>>>>> Can Harmony incorporate this feature?
>>>>>>>
>>>>>>> I'm not sure it is ready for committing to the head stream yet.
>>>>>>>
>>>>>>>> Via on-demand class parsing, we can reduce startup time from 20+
>>>>>>>> seconds to 3 seconds for cold runing, and 170 ms to 140 ms for warm-up
>>>>>>>> running on Core 2 Duo with Windows.
>>>>>>>
>>>>>>> Can you tell me how to reproduce 20+sec cold start-up? I haven't seen
>>>>>>> anything like that in my simple tests.
>>>>>>>
>>>>>>>> After applying the patch, please note there is some change to add new modules.
>>>>>>>> (1) If you want to add new modules/libraries, please don't put them in
>>>>>>>> the bootclasspath.properties file. This file now only saves modules
>>>>>>>> needed during startup (the VM startup only accesses class libraries in
>>>>>>>> eight modules)
>>>>>>>
>>>>>>> That would break too much. How about creating a new file rather than
>>>>>>> re-purposing an existing file with different semantics? This file is
>>>>>>> used by Jikes, IBM VME, the Eclipse plug-in, at least.
>>>>>>>
>>>>>>>> (2) For new modules/libraries, please put them in the
>>>>>>>> modulelibrarymapping.properties file. You should specify the module
>>>>>>>> name and its exported class library. Here is one example:
>>>>>>>> math.jar=java.math, where "math.jar" means the module name, and
>>>>>>>> "java.math" means the class libraries this module exports.
>>>>>>>
>>>>>>> As we discussed on another thread, its unclear if the time is spent in
>>>>>>> following the slow indexing through the classpath/JAR directories, or
>>>>>>> whether it is speed of loading bytes once we know what we need. I think
>>>>>>> that it is premature to abandon the JAR manifest data as the principal
>>>>>>> source of metadata until we understand the problem this solves.
>>>>>>>
>>>>>>> Can we measure where the time is spent in the current implementation?
>>>>>>> I think it will help guide this approach to a better solution.
>>>>>>> What tools do you recommend for profiling start-up?
>>>>>>>
>>>>>>> Regards
>>>>>>> Tim
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> С уважением,
>>>>>> Алексей Федотов,
>>>>>> ЗАО «Телеком Экспресс»
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> С уважением,
>>>> Алексей Федотов,
>>>> ЗАО «Телеком Экспресс»
>>>>
>>>
>>
>
Re: [VM] On-demand class library parsing is ready to commit
Posted by Wenlong Li <we...@gmail.com>.
Hey,
I did an experiment using Alexey's script on a Linux machine (Pentium
D - Intel(R) XEON(TM) CPU 2.00GHz, 1GB main memory, 36GB SCSI hard
disk at 10K RPM, 8MB disk cache, Linux OS 2.6.21-1.3194.fc7,
Harmony-721077, HWA(2) micro-bench)
The performance result is
cold running:
clean: 3375 ms, VM_Creation: 2055 ms
on-demand: 2849 ms, VM_creation: 1435 ms
There is a 30% gain for VM creation
warm-up running:
clean: 1825 ms, VM_Creation:454 ms
on-demand: 1818 ms, VM_Creation: 420 ms
There is a 7% gain for VM creation.
The VM destroy module takes around 1 second for both cold and warm-up
runnings, but not stable (I issue another JIRA for this problem)
The overal observation is on-demand parsing helps cold running (30%
gain), and non-negative benefit for warm-up running. This is expected
because such optimization is only for VM creation module by reducing
the amount of work in Jar parsing.
Here is the summary:
(a) The on-demand class library parsing is optimizing the startup work
in VM creation module, and it contributes to ~30% performance gain on
my test bed.
(b) Such optimization doesn't hurt performance for warm-up running
(c) As this optimization is for VM creation module, it doesn't help
SPECJVM2008 startup benchmarks (e.g., no performance gain)
(d) The performance benefit of this optimization varies with different
systems (Alexey sees 17% gain, and I reported 30% here. On my windows
system (2.83 GHz Core 2 Quad-core, 3GB main memory, 250GB SATA HD at
7200 RPM, 8MB disk cache, WinXp system, Harmony-713673). This
optimization reduces VM creation time from 20+ seconds to 3+ seconds
in a cold running case).
(e) For the consideration of modularity, if I move the update from
class_lib to VM, does Harmony want to include this optimization? As
this takes extra effort, I want to hear your comments in advance.
btw, I put my time instrumentation code in
https://issues.apache.org/jira/browse/HARMONY-6039
Thx for your suggestion, all.
Wenlong
On Mon, Dec 22, 2008 at 4:04 AM, Aleksey Shipilev
<al...@gmail.com> wrote:
> Hi Wenlong,
>
> I had some performance experiments with your patch. The test system is:
> - Pentium D 820 2.8 Ghz / 2 Gb DDR2-667
> - WD 3200KS, 320 Gb, 16 Mb cache
> - Gentoo Linux x86, 2.6.23
> - Harmony r728459
> - SPECjvm2008
>
> To recreate the stressful conditions over and over the simple script
> was written [1]. The script invalidates the caches before actually
> starting the workload: re-reads the same 64 Mb file a couple of times
> to fill out on-HDD cache, invalidating VFS block caches first to make
> sure the data is really requested from the disk.
>
> On HWA [2] these performance results were produced:
>
> "cold-start" (invalidate caches):
> clean: (5.24 +- 0.28) secs
> ondemand: (4.49 +- 0.17) secs
>
> "warm-start" (don't invalidate caches);
> clean: (2.82 +- 0.01) secs
> ondemand: (2.80 +- 0.02) secs
>
> That is, on-demand patch does bring +17% (-+9%) improvement on HWA
> when running with flushed caches, and does not bring any performance
> improvement in warm mode.
>
> As I mentioned several times, this test does not reflect the real
> performance end user would perceive, so I took two SPECjvm2008:startup
> benchmarks and run each of them 10x10 times.
>
> SPECjvm2008:startup.helloworld, "cold start":
> clean: (8.93 +- 0.21) ops/min
> ondemand: (9.04 +- 0.03) ops/min
>
> SPECjvm2008:startup.compiler.compiler, "cold start":
> clean: (1.44 +- 0.05) ops/min
> ondemand: (1.42 +- 0.04) ops/min
>
> As you can see even in very stressful situation there's no boost. I
> would find these performance results unconvincing to change the
> infrastructure of boolclasspath resolution. Am I missing something
> important?
>
> Thanks,
> Aleksey.
>
> [1] run.sh
> #!/bin/bash
>
> R=`pwd`
>
> JAVA=$R/platforms/builds/harmony-release-clean/jdk/jre/bin/java
> #JAVA=$R/platforms/builds/harmony-release-ondemand/jdk/jre/bin/java
> JAVA_OPTS="-Xmx1024M -Xms1024M"
>
> for T in `seq 1 10`; do
>
> echo "*************** EXECUTING ITERATION $T ****************"
>
> # invalidate HDD caches
> # - need to replace all entries in LRU HDD cache
> # - flush the kernel VFS cache first to ensure the data
> would be read from disk
>
> echo "Flushing caches"
> for I in `seq 1 5`; do
> sync
> echo 3 > /proc/sys/vm/drop_caches
>
> dd if=cachekiller.file of=/dev/null > /dev/null 2>&1
> done
>
> echo "Executing."
>
> # HelloWorld
> /usr/bin/time $JAVA $JAVA_OPTS -cp benchmarks/ HelloWorld 2>&1
>
> # SPECjvm2008
> #cd $R/benchmarks/storage/SPECjvm2008
> #/usr/bin/time $JAVA $JAVA_OPTS -Djava.awt.headless=true -jar
> SPECjvm2008.jar -ikv -i 10 startup.compiler.compiler 2>&1
>
> echo ""
> done
>
> [2] HelloWorld.java
> public class HelloWorld {
> public static void main(String[] args) {
> System.out.println("Hello, world!");
> }
> }
>
>
> On Sun, Dec 21, 2008 at 6:02 AM, Wenlong Li <we...@gmail.com> wrote:
>> On Sat, Dec 20, 2008 at 7:10 PM, Alexei Fedotov
>> <al...@gmail.com> wrote:
>>> Wenlong,
>>> Thanks for removing the commented code.
>>>
>>> There are several VMs which make use of the Harmony class library,
>>> e.g. Harmony VM, J9, Android Dalvik, etc. Your change is Harmony VM
>>> specific, isn't it? If it is, then it's better to keep related changes
>>> in the VM module. If it is not, then it might be a good idea to keep
>>> the changes in the class library module unless other VMs already has
>>> such optimization in their code.
>> [Wenlong] Though at this moment, you can think on-demand class parsing
>> is a specif optimization from your point of view. I believe it could
>> be a general technique, e.g., it can be easily deployed in other
>> runtime systems. Current VM also depends on the luniglobal.c in
>> working_classlib to get all class libraries/modules. e.g., there is a
>> cross-module dependence between classlib and VM. When user wants to
>> add new module, they should manually change the
>> bootclasspath.properties, while if applying this patch, user should
>> revise my added property file instead of the bootclasspath.properties.
>> I understand modifying bootclasspath file may be a specification.
>>>
>>> In any case crossing module boundary would make class library users
>>> think more than once or even write some code. Is it technically
>>> possible to prepare a patch which does not change module boundaries?
>>> What do you think?
>> [Wenlong] Yes, it is possible from technical perspective, but a little
>> complicated. I can think about it. :)
>>
>>>
>>> As for your performance experiments, which particular test are your
>>> measuring? It is bootclasspath-unpretentious "Hello, world", isn't it?
>> [Wenlong] My startup means the work executed before running user's
>> computation. That is, the vm creation time. I manually add
>> instrumentation code for execution time in JNI_CreateJavaVM of
>> JNI.cpp. This startup work is common for any benchmarks. My experiment
>> was conducted on both Windows and Linux system. Please see my previous
>> message about performance gain from this optimization.
>>
>> Thx,
>> Wenlong
>>>
>>> Thanks!
>>>
>>> On Sat, Dec 20, 2008 at 2:19 AM, Wenlong Li <we...@gmail.com> wrote:
>>>> On Sat, Dec 20, 2008 at 12:42 AM, Alexei Fedotov
>>>> <al...@gmail.com> wrote:
>>>>> Wenlong,
>>>>> Have I missed a discussion of the proposed design? I see that you
>>>>> expose a new public interface:
>>>>> /**
>>>>> * @map the jar with exported package in the pending jar list for
>>>>> on-demand jar parsing
>>>>> * Key is the jar, and value is the package exported by this jar
>>>>> */
>>>>> DECLARE_OPEN(void, vm_properties_set_pending_jar, (const char* key,
>>>>> const char* value));
>>>>>
>>>>> Did you mean "Maps" instead of "@map"? Strangely the word "pending"
>>>>> disappeared from the name of the wrapping VMI interface
>>>>> SetJarPackageMapping . Why should we extend both OPEN and VMI
>>>>> interfaces with the same function? Why did you put your code into
>>>>> working_classlib/modules/luni/src/main/native/luni/shared/luniglob.c,
>>>>> thus introducing another dependency between VM and class library?
>>>> [Wenlong] The boot class path is defined in luniglobal.c in Harmony,
>>>> and it also has dependence with VM. In my understanding, my patch is
>>>> related to boot class path determination, so I also put my code in
>>>> luniglobal.c, and use VMI interface to communicate with VM.
>>>>
>>>>>
>>>>> + //rcSetProperty = (*vmInterface)->SetJarPackageMapping
>>>>> (vmInterface, jarName, jarValue);
>>>>> + /*
>>>>> + hymem_free_memory(jarName);
>>>>> + hymem_free_memory(jarValue);
>>>>> + */
>>>>> Should we really commit the commented code?
>>>>> Thanks.
>>>>
>>>> [Wenlong] Please see my latest version of patch in the list. Such
>>>> commented code has been removed.
>>>>>
>>>>>
>>>>> On Fri, Dec 19, 2008 at 6:59 PM, Tim Ellison <t....@gmail.com> wrote:
>>>>>> I was hoping that somebody else would comment first, so I don't have to
>>>>>> be the grumpy one all the time :-)
>>>>>>
>>>>>> As I said before, this is good prototyping work...
>>>>>>
>>>>>> Wenlong Li wrote:
>>>>>>> I did the pre-commit test on the patch of on-demand class library
>>>>>>> parsing (https://issues.apache.org/jira/browse/HARMONY-6039), and it
>>>>>>> works well now.
>>>>>>> Can Harmony incorporate this feature?
>>>>>>
>>>>>> I'm not sure it is ready for committing to the head stream yet.
>>>>>>
>>>>>>> Via on-demand class parsing, we can reduce startup time from 20+
>>>>>>> seconds to 3 seconds for cold runing, and 170 ms to 140 ms for warm-up
>>>>>>> running on Core 2 Duo with Windows.
>>>>>>
>>>>>> Can you tell me how to reproduce 20+sec cold start-up? I haven't seen
>>>>>> anything like that in my simple tests.
>>>>>>
>>>>>>> After applying the patch, please note there is some change to add new modules.
>>>>>>> (1) If you want to add new modules/libraries, please don't put them in
>>>>>>> the bootclasspath.properties file. This file now only saves modules
>>>>>>> needed during startup (the VM startup only accesses class libraries in
>>>>>>> eight modules)
>>>>>>
>>>>>> That would break too much. How about creating a new file rather than
>>>>>> re-purposing an existing file with different semantics? This file is
>>>>>> used by Jikes, IBM VME, the Eclipse plug-in, at least.
>>>>>>
>>>>>>> (2) For new modules/libraries, please put them in the
>>>>>>> modulelibrarymapping.properties file. You should specify the module
>>>>>>> name and its exported class library. Here is one example:
>>>>>>> math.jar=java.math, where "math.jar" means the module name, and
>>>>>>> "java.math" means the class libraries this module exports.
>>>>>>
>>>>>> As we discussed on another thread, its unclear if the time is spent in
>>>>>> following the slow indexing through the classpath/JAR directories, or
>>>>>> whether it is speed of loading bytes once we know what we need. I think
>>>>>> that it is premature to abandon the JAR manifest data as the principal
>>>>>> source of metadata until we understand the problem this solves.
>>>>>>
>>>>>> Can we measure where the time is spent in the current implementation?
>>>>>> I think it will help guide this approach to a better solution.
>>>>>> What tools do you recommend for profiling start-up?
>>>>>>
>>>>>> Regards
>>>>>> Tim
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> С уважением,
>>>>> Алексей Федотов,
>>>>> ЗАО «Телеком Экспресс»
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> С уважением,
>>> Алексей Федотов,
>>> ЗАО «Телеком Экспресс»
>>>
>>
>
Re: [VM] On-demand class library parsing is ready to commit
Posted by Wenlong Li <we...@gmail.com>.
That is fine. Any comment is welcome. Our goal is the same. :)
btw, this patch is just to optimize the stuffs you pointed out in
previous message. thx
On Tue, Dec 23, 2008 at 9:04 PM, Gregory Shimansky
<gs...@apache.org> wrote:
> Wenlong Li said the following on 23.12.2008 15:59:
>>
>> Gregory,
>>
>> VM creation module includes function call to parse jar file and
>> load&initialize necessary classes stuffs. You can refer to the
>> vm_init1() and vm_init2() for detail.
>
> Ok I see. I didn't read the thread well enough before replying. Sorry for
> confusion.
>
>> On Tue, Dec 23, 2008 at 8:01 PM, Gregory Shimansky
>> <gs...@apache.org> wrote:
>>>
>>> Pavel Pervov said the following on 23.12.2008 10:34:
>>>>
>>>> Guys,
>>>>
>>>> Why you are in need of instrumentation?
>>>> You only want to measure the time spent in JNI_CreateJavaVM method. It
>>>> is very-very simple to create C-program which utilizes invocation API
>>>> to call to JNI_CreateJavaVM and calls to performance counters before
>>>> and after VM creation to calculate that time. This will be exactly the
>>>> "startup-time" according to Wenlong. Then you can run it through the
>>>> script supplied by Aleksey.
>>>>
>>>> If you also want to include finding main class - you can look into our
>>>> launcher and borrow the code from there.
>>>>
>>>> Invocation API rules. ;)
>>>
>>> I don't quite agree. With lazy resolution CreateJavaVM doesn't really
>>> execute most of Java code that is required to load user program and
>>> execute
>>> all of the necessary code of system class loader which parses jar files
>>> and
>>> stuff like that. This code is what takes the most time on startup.
>>>
>>> --
>>> Gregory
>>>
>>>
>
>
> --
> Gregory
>
>
Re: [VM] On-demand class library parsing is ready to commit
Posted by Gregory Shimansky <gs...@apache.org>.
Wenlong Li said the following on 23.12.2008 15:59:
> Gregory,
>
> VM creation module includes function call to parse jar file and
> load&initialize necessary classes stuffs. You can refer to the
> vm_init1() and vm_init2() for detail.
Ok I see. I didn't read the thread well enough before replying. Sorry
for confusion.
> On Tue, Dec 23, 2008 at 8:01 PM, Gregory Shimansky
> <gs...@apache.org> wrote:
>> Pavel Pervov said the following on 23.12.2008 10:34:
>>> Guys,
>>>
>>> Why you are in need of instrumentation?
>>> You only want to measure the time spent in JNI_CreateJavaVM method. It
>>> is very-very simple to create C-program which utilizes invocation API
>>> to call to JNI_CreateJavaVM and calls to performance counters before
>>> and after VM creation to calculate that time. This will be exactly the
>>> "startup-time" according to Wenlong. Then you can run it through the
>>> script supplied by Aleksey.
>>>
>>> If you also want to include finding main class - you can look into our
>>> launcher and borrow the code from there.
>>>
>>> Invocation API rules. ;)
>> I don't quite agree. With lazy resolution CreateJavaVM doesn't really
>> execute most of Java code that is required to load user program and execute
>> all of the necessary code of system class loader which parses jar files and
>> stuff like that. This code is what takes the most time on startup.
>>
>> --
>> Gregory
>>
>>
--
Gregory
Re: [VM] On-demand class library parsing is ready to commit
Posted by Wenlong Li <we...@gmail.com>.
Gregory,
VM creation module includes function call to parse jar file and
load&initialize necessary classes stuffs. You can refer to the
vm_init1() and vm_init2() for detail.
thx,
Wenlong
On Tue, Dec 23, 2008 at 8:01 PM, Gregory Shimansky
<gs...@apache.org> wrote:
> Pavel Pervov said the following on 23.12.2008 10:34:
>>
>> Guys,
>>
>> Why you are in need of instrumentation?
>> You only want to measure the time spent in JNI_CreateJavaVM method. It
>> is very-very simple to create C-program which utilizes invocation API
>> to call to JNI_CreateJavaVM and calls to performance counters before
>> and after VM creation to calculate that time. This will be exactly the
>> "startup-time" according to Wenlong. Then you can run it through the
>> script supplied by Aleksey.
>>
>> If you also want to include finding main class - you can look into our
>> launcher and borrow the code from there.
>>
>> Invocation API rules. ;)
>
> I don't quite agree. With lazy resolution CreateJavaVM doesn't really
> execute most of Java code that is required to load user program and execute
> all of the necessary code of system class loader which parses jar files and
> stuff like that. This code is what takes the most time on startup.
>
> --
> Gregory
>
>
Re: [VM] On-demand class library parsing is ready to commit
Posted by Pavel Pervov <pm...@gmail.com>.
Gregory,
This is my turn to not quite understand.
What we are trying to optimize then?
Pavel.
On Tue, Dec 23, 2008 at 3:01 PM, Gregory Shimansky
<gs...@apache.org> wrote:
> Pavel Pervov said the following on 23.12.2008 10:34:
>>
>> Guys,
>>
>> Why you are in need of instrumentation?
>> You only want to measure the time spent in JNI_CreateJavaVM method. It
>> is very-very simple to create C-program which utilizes invocation API
>> to call to JNI_CreateJavaVM and calls to performance counters before
>> and after VM creation to calculate that time. This will be exactly the
>> "startup-time" according to Wenlong. Then you can run it through the
>> script supplied by Aleksey.
>>
>> If you also want to include finding main class - you can look into our
>> launcher and borrow the code from there.
>>
>> Invocation API rules. ;)
>
> I don't quite agree. With lazy resolution CreateJavaVM doesn't really
> execute most of Java code that is required to load user program and execute
> all of the necessary code of system class loader which parses jar files and
> stuff like that. This code is what takes the most time on startup.
>
> --
> Gregory
>
>
Re: [VM] On-demand class library parsing is ready to commit
Posted by Gregory Shimansky <gs...@apache.org>.
Pavel Pervov said the following on 23.12.2008 10:34:
> Guys,
>
> Why you are in need of instrumentation?
> You only want to measure the time spent in JNI_CreateJavaVM method. It
> is very-very simple to create C-program which utilizes invocation API
> to call to JNI_CreateJavaVM and calls to performance counters before
> and after VM creation to calculate that time. This will be exactly the
> "startup-time" according to Wenlong. Then you can run it through the
> script supplied by Aleksey.
>
> If you also want to include finding main class - you can look into our
> launcher and borrow the code from there.
>
> Invocation API rules. ;)
I don't quite agree. With lazy resolution CreateJavaVM doesn't really
execute most of Java code that is required to load user program and
execute all of the necessary code of system class loader which parses
jar files and stuff like that. This code is what takes the most time on
startup.
--
Gregory
Re: [VM] On-demand class library parsing is ready to commit
Posted by Pavel Pervov <pm...@gmail.com>.
Guys,
Why you are in need of instrumentation?
You only want to measure the time spent in JNI_CreateJavaVM method. It
is very-very simple to create C-program which utilizes invocation API
to call to JNI_CreateJavaVM and calls to performance counters before
and after VM creation to calculate that time. This will be exactly the
"startup-time" according to Wenlong. Then you can run it through the
script supplied by Aleksey.
If you also want to include finding main class - you can look into our
launcher and borrow the code from there.
Invocation API rules. ;)
WBR,
Pavel.
On Tue, Dec 23, 2008 at 3:06 AM, Aleksey Shipilev
<al...@gmail.com> wrote:
> Hi, Wenlong!
>
> That would be terrific.
>
> Can you please do the following runs in both cold and warm modes:
> a. HWA without instrumentation.
> b. HWA with instrumentation which measure the following stages: VM
> creation time, Java main() execution, shutdown sequence.
> c. SPECjvm2008:startup.helloworld without instrumentation.
>
> The instrumentation code should belong in corresponding JIRA issue, as
> the part of prerequisites for testing your patch.
>
> Thanks,
> Aleksey.
>
> On Tue, Dec 23, 2008 at 2:45 AM, Wenlong Li <we...@gmail.com> wrote:
>> Aleksey,
>>
>> Do you mind repeating my experiment in your side? Your code below also
>> measures the VM destroy time, and it could be very long. You can
>> instrument it in destroy_VM. So my suggestion is can you only report
>> the time before executing user's code, e.g., the vm creation time,
>> etc. I can share my time instrumentation code if needed.
>>
>> I want to repeat your experiment in my side.
>>
>> thx,
>> Wenlong
>>
>> On Mon, Dec 22, 2008 at 4:04 AM, Aleksey Shipilev
>> <al...@gmail.com> wrote:
>>> Hi Wenlong,
>>>
>>> I had some performance experiments with your patch. The test system is:
>>> - Pentium D 820 2.8 Ghz / 2 Gb DDR2-667
>>> - WD 3200KS, 320 Gb, 16 Mb cache
>>> - Gentoo Linux x86, 2.6.23
>>> - Harmony r728459
>>> - SPECjvm2008
>>>
>>> To recreate the stressful conditions over and over the simple script
>>> was written [1]. The script invalidates the caches before actually
>>> starting the workload: re-reads the same 64 Mb file a couple of times
>>> to fill out on-HDD cache, invalidating VFS block caches first to make
>>> sure the data is really requested from the disk.
>>>
>>> On HWA [2] these performance results were produced:
>>>
>>> "cold-start" (invalidate caches):
>>> clean: (5.24 +- 0.28) secs
>>> ondemand: (4.49 +- 0.17) secs
>>>
>>> "warm-start" (don't invalidate caches);
>>> clean: (2.82 +- 0.01) secs
>>> ondemand: (2.80 +- 0.02) secs
>>>
>>> That is, on-demand patch does bring +17% (-+9%) improvement on HWA
>>> when running with flushed caches, and does not bring any performance
>>> improvement in warm mode.
>>>
>>> As I mentioned several times, this test does not reflect the real
>>> performance end user would perceive, so I took two SPECjvm2008:startup
>>> benchmarks and run each of them 10x10 times.
>>>
>>> SPECjvm2008:startup.helloworld, "cold start":
>>> clean: (8.93 +- 0.21) ops/min
>>> ondemand: (9.04 +- 0.03) ops/min
>>>
>>> SPECjvm2008:startup.compiler.compiler, "cold start":
>>> clean: (1.44 +- 0.05) ops/min
>>> ondemand: (1.42 +- 0.04) ops/min
>>>
>>> As you can see even in very stressful situation there's no boost. I
>>> would find these performance results unconvincing to change the
>>> infrastructure of boolclasspath resolution. Am I missing something
>>> important?
>>>
>>> Thanks,
>>> Aleksey.
>>>
>>> [1] run.sh
>>> #!/bin/bash
>>>
>>> R=`pwd`
>>>
>>> JAVA=$R/platforms/builds/harmony-release-clean/jdk/jre/bin/java
>>> #JAVA=$R/platforms/builds/harmony-release-ondemand/jdk/jre/bin/java
>>> JAVA_OPTS="-Xmx1024M -Xms1024M"
>>>
>>> for T in `seq 1 10`; do
>>>
>>> echo "*************** EXECUTING ITERATION $T ****************"
>>>
>>> # invalidate HDD caches
>>> # - need to replace all entries in LRU HDD cache
>>> # - flush the kernel VFS cache first to ensure the data
>>> would be read from disk
>>>
>>> echo "Flushing caches"
>>> for I in `seq 1 5`; do
>>> sync
>>> echo 3 > /proc/sys/vm/drop_caches
>>>
>>> dd if=cachekiller.file of=/dev/null > /dev/null 2>&1
>>> done
>>>
>>> echo "Executing."
>>>
>>> # HelloWorld
>>> /usr/bin/time $JAVA $JAVA_OPTS -cp benchmarks/ HelloWorld 2>&1
>>>
>>> # SPECjvm2008
>>> #cd $R/benchmarks/storage/SPECjvm2008
>>> #/usr/bin/time $JAVA $JAVA_OPTS -Djava.awt.headless=true -jar
>>> SPECjvm2008.jar -ikv -i 10 startup.compiler.compiler 2>&1
>>>
>>> echo ""
>>> done
>>>
>>> [2] HelloWorld.java
>>> public class HelloWorld {
>>> public static void main(String[] args) {
>>> System.out.println("Hello, world!");
>>> }
>>> }
>>>
>>>
>>> On Sun, Dec 21, 2008 at 6:02 AM, Wenlong Li <we...@gmail.com> wrote:
>>>> On Sat, Dec 20, 2008 at 7:10 PM, Alexei Fedotov
>>>> <al...@gmail.com> wrote:
>>>>> Wenlong,
>>>>> Thanks for removing the commented code.
>>>>>
>>>>> There are several VMs which make use of the Harmony class library,
>>>>> e.g. Harmony VM, J9, Android Dalvik, etc. Your change is Harmony VM
>>>>> specific, isn't it? If it is, then it's better to keep related changes
>>>>> in the VM module. If it is not, then it might be a good idea to keep
>>>>> the changes in the class library module unless other VMs already has
>>>>> such optimization in their code.
>>>> [Wenlong] Though at this moment, you can think on-demand class parsing
>>>> is a specif optimization from your point of view. I believe it could
>>>> be a general technique, e.g., it can be easily deployed in other
>>>> runtime systems. Current VM also depends on the luniglobal.c in
>>>> working_classlib to get all class libraries/modules. e.g., there is a
>>>> cross-module dependence between classlib and VM. When user wants to
>>>> add new module, they should manually change the
>>>> bootclasspath.properties, while if applying this patch, user should
>>>> revise my added property file instead of the bootclasspath.properties.
>>>> I understand modifying bootclasspath file may be a specification.
>>>>>
>>>>> In any case crossing module boundary would make class library users
>>>>> think more than once or even write some code. Is it technically
>>>>> possible to prepare a patch which does not change module boundaries?
>>>>> What do you think?
>>>> [Wenlong] Yes, it is possible from technical perspective, but a little
>>>> complicated. I can think about it. :)
>>>>
>>>>>
>>>>> As for your performance experiments, which particular test are your
>>>>> measuring? It is bootclasspath-unpretentious "Hello, world", isn't it?
>>>> [Wenlong] My startup means the work executed before running user's
>>>> computation. That is, the vm creation time. I manually add
>>>> instrumentation code for execution time in JNI_CreateJavaVM of
>>>> JNI.cpp. This startup work is common for any benchmarks. My experiment
>>>> was conducted on both Windows and Linux system. Please see my previous
>>>> message about performance gain from this optimization.
>>>>
>>>> Thx,
>>>> Wenlong
>>>>>
>>>>> Thanks!
>>>>>
>>>>> On Sat, Dec 20, 2008 at 2:19 AM, Wenlong Li <we...@gmail.com> wrote:
>>>>>> On Sat, Dec 20, 2008 at 12:42 AM, Alexei Fedotov
>>>>>> <al...@gmail.com> wrote:
>>>>>>> Wenlong,
>>>>>>> Have I missed a discussion of the proposed design? I see that you
>>>>>>> expose a new public interface:
>>>>>>> /**
>>>>>>> * @map the jar with exported package in the pending jar list for
>>>>>>> on-demand jar parsing
>>>>>>> * Key is the jar, and value is the package exported by this jar
>>>>>>> */
>>>>>>> DECLARE_OPEN(void, vm_properties_set_pending_jar, (const char* key,
>>>>>>> const char* value));
>>>>>>>
>>>>>>> Did you mean "Maps" instead of "@map"? Strangely the word "pending"
>>>>>>> disappeared from the name of the wrapping VMI interface
>>>>>>> SetJarPackageMapping . Why should we extend both OPEN and VMI
>>>>>>> interfaces with the same function? Why did you put your code into
>>>>>>> working_classlib/modules/luni/src/main/native/luni/shared/luniglob.c,
>>>>>>> thus introducing another dependency between VM and class library?
>>>>>> [Wenlong] The boot class path is defined in luniglobal.c in Harmony,
>>>>>> and it also has dependence with VM. In my understanding, my patch is
>>>>>> related to boot class path determination, so I also put my code in
>>>>>> luniglobal.c, and use VMI interface to communicate with VM.
>>>>>>
>>>>>>>
>>>>>>> + //rcSetProperty = (*vmInterface)->SetJarPackageMapping
>>>>>>> (vmInterface, jarName, jarValue);
>>>>>>> + /*
>>>>>>> + hymem_free_memory(jarName);
>>>>>>> + hymem_free_memory(jarValue);
>>>>>>> + */
>>>>>>> Should we really commit the commented code?
>>>>>>> Thanks.
>>>>>>
>>>>>> [Wenlong] Please see my latest version of patch in the list. Such
>>>>>> commented code has been removed.
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Dec 19, 2008 at 6:59 PM, Tim Ellison <t....@gmail.com> wrote:
>>>>>>>> I was hoping that somebody else would comment first, so I don't have to
>>>>>>>> be the grumpy one all the time :-)
>>>>>>>>
>>>>>>>> As I said before, this is good prototyping work...
>>>>>>>>
>>>>>>>> Wenlong Li wrote:
>>>>>>>>> I did the pre-commit test on the patch of on-demand class library
>>>>>>>>> parsing (https://issues.apache.org/jira/browse/HARMONY-6039), and it
>>>>>>>>> works well now.
>>>>>>>>> Can Harmony incorporate this feature?
>>>>>>>>
>>>>>>>> I'm not sure it is ready for committing to the head stream yet.
>>>>>>>>
>>>>>>>>> Via on-demand class parsing, we can reduce startup time from 20+
>>>>>>>>> seconds to 3 seconds for cold runing, and 170 ms to 140 ms for warm-up
>>>>>>>>> running on Core 2 Duo with Windows.
>>>>>>>>
>>>>>>>> Can you tell me how to reproduce 20+sec cold start-up? I haven't seen
>>>>>>>> anything like that in my simple tests.
>>>>>>>>
>>>>>>>>> After applying the patch, please note there is some change to add new modules.
>>>>>>>>> (1) If you want to add new modules/libraries, please don't put them in
>>>>>>>>> the bootclasspath.properties file. This file now only saves modules
>>>>>>>>> needed during startup (the VM startup only accesses class libraries in
>>>>>>>>> eight modules)
>>>>>>>>
>>>>>>>> That would break too much. How about creating a new file rather than
>>>>>>>> re-purposing an existing file with different semantics? This file is
>>>>>>>> used by Jikes, IBM VME, the Eclipse plug-in, at least.
>>>>>>>>
>>>>>>>>> (2) For new modules/libraries, please put them in the
>>>>>>>>> modulelibrarymapping.properties file. You should specify the module
>>>>>>>>> name and its exported class library. Here is one example:
>>>>>>>>> math.jar=java.math, where "math.jar" means the module name, and
>>>>>>>>> "java.math" means the class libraries this module exports.
>>>>>>>>
>>>>>>>> As we discussed on another thread, its unclear if the time is spent in
>>>>>>>> following the slow indexing through the classpath/JAR directories, or
>>>>>>>> whether it is speed of loading bytes once we know what we need. I think
>>>>>>>> that it is premature to abandon the JAR manifest data as the principal
>>>>>>>> source of metadata until we understand the problem this solves.
>>>>>>>>
>>>>>>>> Can we measure where the time is spent in the current implementation?
>>>>>>>> I think it will help guide this approach to a better solution.
>>>>>>>> What tools do you recommend for profiling start-up?
>>>>>>>>
>>>>>>>> Regards
>>>>>>>> Tim
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> С уважением,
>>>>>>> Алексей Федотов,
>>>>>>> ЗАО «Телеком Экспресс»
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> С уважением,
>>>>> Алексей Федотов,
>>>>> ЗАО «Телеком Экспресс»
>>>>>
>>>>
>>>
>>
>
Re: [VM] On-demand class library parsing is ready to commit
Posted by Aleksey Shipilev <al...@gmail.com>.
Hi, Wenlong!
That would be terrific.
Can you please do the following runs in both cold and warm modes:
a. HWA without instrumentation.
b. HWA with instrumentation which measure the following stages: VM
creation time, Java main() execution, shutdown sequence.
c. SPECjvm2008:startup.helloworld without instrumentation.
The instrumentation code should belong in corresponding JIRA issue, as
the part of prerequisites for testing your patch.
Thanks,
Aleksey.
On Tue, Dec 23, 2008 at 2:45 AM, Wenlong Li <we...@gmail.com> wrote:
> Aleksey,
>
> Do you mind repeating my experiment in your side? Your code below also
> measures the VM destroy time, and it could be very long. You can
> instrument it in destroy_VM. So my suggestion is can you only report
> the time before executing user's code, e.g., the vm creation time,
> etc. I can share my time instrumentation code if needed.
>
> I want to repeat your experiment in my side.
>
> thx,
> Wenlong
>
> On Mon, Dec 22, 2008 at 4:04 AM, Aleksey Shipilev
> <al...@gmail.com> wrote:
>> Hi Wenlong,
>>
>> I had some performance experiments with your patch. The test system is:
>> - Pentium D 820 2.8 Ghz / 2 Gb DDR2-667
>> - WD 3200KS, 320 Gb, 16 Mb cache
>> - Gentoo Linux x86, 2.6.23
>> - Harmony r728459
>> - SPECjvm2008
>>
>> To recreate the stressful conditions over and over the simple script
>> was written [1]. The script invalidates the caches before actually
>> starting the workload: re-reads the same 64 Mb file a couple of times
>> to fill out on-HDD cache, invalidating VFS block caches first to make
>> sure the data is really requested from the disk.
>>
>> On HWA [2] these performance results were produced:
>>
>> "cold-start" (invalidate caches):
>> clean: (5.24 +- 0.28) secs
>> ondemand: (4.49 +- 0.17) secs
>>
>> "warm-start" (don't invalidate caches);
>> clean: (2.82 +- 0.01) secs
>> ondemand: (2.80 +- 0.02) secs
>>
>> That is, on-demand patch does bring +17% (-+9%) improvement on HWA
>> when running with flushed caches, and does not bring any performance
>> improvement in warm mode.
>>
>> As I mentioned several times, this test does not reflect the real
>> performance end user would perceive, so I took two SPECjvm2008:startup
>> benchmarks and run each of them 10x10 times.
>>
>> SPECjvm2008:startup.helloworld, "cold start":
>> clean: (8.93 +- 0.21) ops/min
>> ondemand: (9.04 +- 0.03) ops/min
>>
>> SPECjvm2008:startup.compiler.compiler, "cold start":
>> clean: (1.44 +- 0.05) ops/min
>> ondemand: (1.42 +- 0.04) ops/min
>>
>> As you can see even in very stressful situation there's no boost. I
>> would find these performance results unconvincing to change the
>> infrastructure of boolclasspath resolution. Am I missing something
>> important?
>>
>> Thanks,
>> Aleksey.
>>
>> [1] run.sh
>> #!/bin/bash
>>
>> R=`pwd`
>>
>> JAVA=$R/platforms/builds/harmony-release-clean/jdk/jre/bin/java
>> #JAVA=$R/platforms/builds/harmony-release-ondemand/jdk/jre/bin/java
>> JAVA_OPTS="-Xmx1024M -Xms1024M"
>>
>> for T in `seq 1 10`; do
>>
>> echo "*************** EXECUTING ITERATION $T ****************"
>>
>> # invalidate HDD caches
>> # - need to replace all entries in LRU HDD cache
>> # - flush the kernel VFS cache first to ensure the data
>> would be read from disk
>>
>> echo "Flushing caches"
>> for I in `seq 1 5`; do
>> sync
>> echo 3 > /proc/sys/vm/drop_caches
>>
>> dd if=cachekiller.file of=/dev/null > /dev/null 2>&1
>> done
>>
>> echo "Executing."
>>
>> # HelloWorld
>> /usr/bin/time $JAVA $JAVA_OPTS -cp benchmarks/ HelloWorld 2>&1
>>
>> # SPECjvm2008
>> #cd $R/benchmarks/storage/SPECjvm2008
>> #/usr/bin/time $JAVA $JAVA_OPTS -Djava.awt.headless=true -jar
>> SPECjvm2008.jar -ikv -i 10 startup.compiler.compiler 2>&1
>>
>> echo ""
>> done
>>
>> [2] HelloWorld.java
>> public class HelloWorld {
>> public static void main(String[] args) {
>> System.out.println("Hello, world!");
>> }
>> }
>>
>>
>> On Sun, Dec 21, 2008 at 6:02 AM, Wenlong Li <we...@gmail.com> wrote:
>>> On Sat, Dec 20, 2008 at 7:10 PM, Alexei Fedotov
>>> <al...@gmail.com> wrote:
>>>> Wenlong,
>>>> Thanks for removing the commented code.
>>>>
>>>> There are several VMs which make use of the Harmony class library,
>>>> e.g. Harmony VM, J9, Android Dalvik, etc. Your change is Harmony VM
>>>> specific, isn't it? If it is, then it's better to keep related changes
>>>> in the VM module. If it is not, then it might be a good idea to keep
>>>> the changes in the class library module unless other VMs already has
>>>> such optimization in their code.
>>> [Wenlong] Though at this moment, you can think on-demand class parsing
>>> is a specif optimization from your point of view. I believe it could
>>> be a general technique, e.g., it can be easily deployed in other
>>> runtime systems. Current VM also depends on the luniglobal.c in
>>> working_classlib to get all class libraries/modules. e.g., there is a
>>> cross-module dependence between classlib and VM. When user wants to
>>> add new module, they should manually change the
>>> bootclasspath.properties, while if applying this patch, user should
>>> revise my added property file instead of the bootclasspath.properties.
>>> I understand modifying bootclasspath file may be a specification.
>>>>
>>>> In any case crossing module boundary would make class library users
>>>> think more than once or even write some code. Is it technically
>>>> possible to prepare a patch which does not change module boundaries?
>>>> What do you think?
>>> [Wenlong] Yes, it is possible from technical perspective, but a little
>>> complicated. I can think about it. :)
>>>
>>>>
>>>> As for your performance experiments, which particular test are your
>>>> measuring? It is bootclasspath-unpretentious "Hello, world", isn't it?
>>> [Wenlong] My startup means the work executed before running user's
>>> computation. That is, the vm creation time. I manually add
>>> instrumentation code for execution time in JNI_CreateJavaVM of
>>> JNI.cpp. This startup work is common for any benchmarks. My experiment
>>> was conducted on both Windows and Linux system. Please see my previous
>>> message about performance gain from this optimization.
>>>
>>> Thx,
>>> Wenlong
>>>>
>>>> Thanks!
>>>>
>>>> On Sat, Dec 20, 2008 at 2:19 AM, Wenlong Li <we...@gmail.com> wrote:
>>>>> On Sat, Dec 20, 2008 at 12:42 AM, Alexei Fedotov
>>>>> <al...@gmail.com> wrote:
>>>>>> Wenlong,
>>>>>> Have I missed a discussion of the proposed design? I see that you
>>>>>> expose a new public interface:
>>>>>> /**
>>>>>> * @map the jar with exported package in the pending jar list for
>>>>>> on-demand jar parsing
>>>>>> * Key is the jar, and value is the package exported by this jar
>>>>>> */
>>>>>> DECLARE_OPEN(void, vm_properties_set_pending_jar, (const char* key,
>>>>>> const char* value));
>>>>>>
>>>>>> Did you mean "Maps" instead of "@map"? Strangely the word "pending"
>>>>>> disappeared from the name of the wrapping VMI interface
>>>>>> SetJarPackageMapping . Why should we extend both OPEN and VMI
>>>>>> interfaces with the same function? Why did you put your code into
>>>>>> working_classlib/modules/luni/src/main/native/luni/shared/luniglob.c,
>>>>>> thus introducing another dependency between VM and class library?
>>>>> [Wenlong] The boot class path is defined in luniglobal.c in Harmony,
>>>>> and it also has dependence with VM. In my understanding, my patch is
>>>>> related to boot class path determination, so I also put my code in
>>>>> luniglobal.c, and use VMI interface to communicate with VM.
>>>>>
>>>>>>
>>>>>> + //rcSetProperty = (*vmInterface)->SetJarPackageMapping
>>>>>> (vmInterface, jarName, jarValue);
>>>>>> + /*
>>>>>> + hymem_free_memory(jarName);
>>>>>> + hymem_free_memory(jarValue);
>>>>>> + */
>>>>>> Should we really commit the commented code?
>>>>>> Thanks.
>>>>>
>>>>> [Wenlong] Please see my latest version of patch in the list. Such
>>>>> commented code has been removed.
>>>>>>
>>>>>>
>>>>>> On Fri, Dec 19, 2008 at 6:59 PM, Tim Ellison <t....@gmail.com> wrote:
>>>>>>> I was hoping that somebody else would comment first, so I don't have to
>>>>>>> be the grumpy one all the time :-)
>>>>>>>
>>>>>>> As I said before, this is good prototyping work...
>>>>>>>
>>>>>>> Wenlong Li wrote:
>>>>>>>> I did the pre-commit test on the patch of on-demand class library
>>>>>>>> parsing (https://issues.apache.org/jira/browse/HARMONY-6039), and it
>>>>>>>> works well now.
>>>>>>>> Can Harmony incorporate this feature?
>>>>>>>
>>>>>>> I'm not sure it is ready for committing to the head stream yet.
>>>>>>>
>>>>>>>> Via on-demand class parsing, we can reduce startup time from 20+
>>>>>>>> seconds to 3 seconds for cold runing, and 170 ms to 140 ms for warm-up
>>>>>>>> running on Core 2 Duo with Windows.
>>>>>>>
>>>>>>> Can you tell me how to reproduce 20+sec cold start-up? I haven't seen
>>>>>>> anything like that in my simple tests.
>>>>>>>
>>>>>>>> After applying the patch, please note there is some change to add new modules.
>>>>>>>> (1) If you want to add new modules/libraries, please don't put them in
>>>>>>>> the bootclasspath.properties file. This file now only saves modules
>>>>>>>> needed during startup (the VM startup only accesses class libraries in
>>>>>>>> eight modules)
>>>>>>>
>>>>>>> That would break too much. How about creating a new file rather than
>>>>>>> re-purposing an existing file with different semantics? This file is
>>>>>>> used by Jikes, IBM VME, the Eclipse plug-in, at least.
>>>>>>>
>>>>>>>> (2) For new modules/libraries, please put them in the
>>>>>>>> modulelibrarymapping.properties file. You should specify the module
>>>>>>>> name and its exported class library. Here is one example:
>>>>>>>> math.jar=java.math, where "math.jar" means the module name, and
>>>>>>>> "java.math" means the class libraries this module exports.
>>>>>>>
>>>>>>> As we discussed on another thread, its unclear if the time is spent in
>>>>>>> following the slow indexing through the classpath/JAR directories, or
>>>>>>> whether it is speed of loading bytes once we know what we need. I think
>>>>>>> that it is premature to abandon the JAR manifest data as the principal
>>>>>>> source of metadata until we understand the problem this solves.
>>>>>>>
>>>>>>> Can we measure where the time is spent in the current implementation?
>>>>>>> I think it will help guide this approach to a better solution.
>>>>>>> What tools do you recommend for profiling start-up?
>>>>>>>
>>>>>>> Regards
>>>>>>> Tim
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> С уважением,
>>>>>> Алексей Федотов,
>>>>>> ЗАО «Телеком Экспресс»
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> С уважением,
>>>> Алексей Федотов,
>>>> ЗАО «Телеком Экспресс»
>>>>
>>>
>>
>
Re: [VM] On-demand class library parsing is ready to commit
Posted by Wenlong Li <we...@gmail.com>.
Aleksey,
Do you mind repeating my experiment in your side? Your code below also
measures the VM destroy time, and it could be very long. You can
instrument it in destroy_VM. So my suggestion is can you only report
the time before executing user's code, e.g., the vm creation time,
etc. I can share my time instrumentation code if needed.
I want to repeat your experiment in my side.
thx,
Wenlong
On Mon, Dec 22, 2008 at 4:04 AM, Aleksey Shipilev
<al...@gmail.com> wrote:
> Hi Wenlong,
>
> I had some performance experiments with your patch. The test system is:
> - Pentium D 820 2.8 Ghz / 2 Gb DDR2-667
> - WD 3200KS, 320 Gb, 16 Mb cache
> - Gentoo Linux x86, 2.6.23
> - Harmony r728459
> - SPECjvm2008
>
> To recreate the stressful conditions over and over the simple script
> was written [1]. The script invalidates the caches before actually
> starting the workload: re-reads the same 64 Mb file a couple of times
> to fill out on-HDD cache, invalidating VFS block caches first to make
> sure the data is really requested from the disk.
>
> On HWA [2] these performance results were produced:
>
> "cold-start" (invalidate caches):
> clean: (5.24 +- 0.28) secs
> ondemand: (4.49 +- 0.17) secs
>
> "warm-start" (don't invalidate caches);
> clean: (2.82 +- 0.01) secs
> ondemand: (2.80 +- 0.02) secs
>
> That is, on-demand patch does bring +17% (-+9%) improvement on HWA
> when running with flushed caches, and does not bring any performance
> improvement in warm mode.
>
> As I mentioned several times, this test does not reflect the real
> performance end user would perceive, so I took two SPECjvm2008:startup
> benchmarks and run each of them 10x10 times.
>
> SPECjvm2008:startup.helloworld, "cold start":
> clean: (8.93 +- 0.21) ops/min
> ondemand: (9.04 +- 0.03) ops/min
>
> SPECjvm2008:startup.compiler.compiler, "cold start":
> clean: (1.44 +- 0.05) ops/min
> ondemand: (1.42 +- 0.04) ops/min
>
> As you can see even in very stressful situation there's no boost. I
> would find these performance results unconvincing to change the
> infrastructure of boolclasspath resolution. Am I missing something
> important?
>
> Thanks,
> Aleksey.
>
> [1] run.sh
> #!/bin/bash
>
> R=`pwd`
>
> JAVA=$R/platforms/builds/harmony-release-clean/jdk/jre/bin/java
> #JAVA=$R/platforms/builds/harmony-release-ondemand/jdk/jre/bin/java
> JAVA_OPTS="-Xmx1024M -Xms1024M"
>
> for T in `seq 1 10`; do
>
> echo "*************** EXECUTING ITERATION $T ****************"
>
> # invalidate HDD caches
> # - need to replace all entries in LRU HDD cache
> # - flush the kernel VFS cache first to ensure the data
> would be read from disk
>
> echo "Flushing caches"
> for I in `seq 1 5`; do
> sync
> echo 3 > /proc/sys/vm/drop_caches
>
> dd if=cachekiller.file of=/dev/null > /dev/null 2>&1
> done
>
> echo "Executing."
>
> # HelloWorld
> /usr/bin/time $JAVA $JAVA_OPTS -cp benchmarks/ HelloWorld 2>&1
>
> # SPECjvm2008
> #cd $R/benchmarks/storage/SPECjvm2008
> #/usr/bin/time $JAVA $JAVA_OPTS -Djava.awt.headless=true -jar
> SPECjvm2008.jar -ikv -i 10 startup.compiler.compiler 2>&1
>
> echo ""
> done
>
> [2] HelloWorld.java
> public class HelloWorld {
> public static void main(String[] args) {
> System.out.println("Hello, world!");
> }
> }
>
>
> On Sun, Dec 21, 2008 at 6:02 AM, Wenlong Li <we...@gmail.com> wrote:
>> On Sat, Dec 20, 2008 at 7:10 PM, Alexei Fedotov
>> <al...@gmail.com> wrote:
>>> Wenlong,
>>> Thanks for removing the commented code.
>>>
>>> There are several VMs which make use of the Harmony class library,
>>> e.g. Harmony VM, J9, Android Dalvik, etc. Your change is Harmony VM
>>> specific, isn't it? If it is, then it's better to keep related changes
>>> in the VM module. If it is not, then it might be a good idea to keep
>>> the changes in the class library module unless other VMs already has
>>> such optimization in their code.
>> [Wenlong] Though at this moment, you can think on-demand class parsing
>> is a specif optimization from your point of view. I believe it could
>> be a general technique, e.g., it can be easily deployed in other
>> runtime systems. Current VM also depends on the luniglobal.c in
>> working_classlib to get all class libraries/modules. e.g., there is a
>> cross-module dependence between classlib and VM. When user wants to
>> add new module, they should manually change the
>> bootclasspath.properties, while if applying this patch, user should
>> revise my added property file instead of the bootclasspath.properties.
>> I understand modifying bootclasspath file may be a specification.
>>>
>>> In any case crossing module boundary would make class library users
>>> think more than once or even write some code. Is it technically
>>> possible to prepare a patch which does not change module boundaries?
>>> What do you think?
>> [Wenlong] Yes, it is possible from technical perspective, but a little
>> complicated. I can think about it. :)
>>
>>>
>>> As for your performance experiments, which particular test are your
>>> measuring? It is bootclasspath-unpretentious "Hello, world", isn't it?
>> [Wenlong] My startup means the work executed before running user's
>> computation. That is, the vm creation time. I manually add
>> instrumentation code for execution time in JNI_CreateJavaVM of
>> JNI.cpp. This startup work is common for any benchmarks. My experiment
>> was conducted on both Windows and Linux system. Please see my previous
>> message about performance gain from this optimization.
>>
>> Thx,
>> Wenlong
>>>
>>> Thanks!
>>>
>>> On Sat, Dec 20, 2008 at 2:19 AM, Wenlong Li <we...@gmail.com> wrote:
>>>> On Sat, Dec 20, 2008 at 12:42 AM, Alexei Fedotov
>>>> <al...@gmail.com> wrote:
>>>>> Wenlong,
>>>>> Have I missed a discussion of the proposed design? I see that you
>>>>> expose a new public interface:
>>>>> /**
>>>>> * @map the jar with exported package in the pending jar list for
>>>>> on-demand jar parsing
>>>>> * Key is the jar, and value is the package exported by this jar
>>>>> */
>>>>> DECLARE_OPEN(void, vm_properties_set_pending_jar, (const char* key,
>>>>> const char* value));
>>>>>
>>>>> Did you mean "Maps" instead of "@map"? Strangely the word "pending"
>>>>> disappeared from the name of the wrapping VMI interface
>>>>> SetJarPackageMapping . Why should we extend both OPEN and VMI
>>>>> interfaces with the same function? Why did you put your code into
>>>>> working_classlib/modules/luni/src/main/native/luni/shared/luniglob.c,
>>>>> thus introducing another dependency between VM and class library?
>>>> [Wenlong] The boot class path is defined in luniglobal.c in Harmony,
>>>> and it also has dependence with VM. In my understanding, my patch is
>>>> related to boot class path determination, so I also put my code in
>>>> luniglobal.c, and use VMI interface to communicate with VM.
>>>>
>>>>>
>>>>> + //rcSetProperty = (*vmInterface)->SetJarPackageMapping
>>>>> (vmInterface, jarName, jarValue);
>>>>> + /*
>>>>> + hymem_free_memory(jarName);
>>>>> + hymem_free_memory(jarValue);
>>>>> + */
>>>>> Should we really commit the commented code?
>>>>> Thanks.
>>>>
>>>> [Wenlong] Please see my latest version of patch in the list. Such
>>>> commented code has been removed.
>>>>>
>>>>>
>>>>> On Fri, Dec 19, 2008 at 6:59 PM, Tim Ellison <t....@gmail.com> wrote:
>>>>>> I was hoping that somebody else would comment first, so I don't have to
>>>>>> be the grumpy one all the time :-)
>>>>>>
>>>>>> As I said before, this is good prototyping work...
>>>>>>
>>>>>> Wenlong Li wrote:
>>>>>>> I did the pre-commit test on the patch of on-demand class library
>>>>>>> parsing (https://issues.apache.org/jira/browse/HARMONY-6039), and it
>>>>>>> works well now.
>>>>>>> Can Harmony incorporate this feature?
>>>>>>
>>>>>> I'm not sure it is ready for committing to the head stream yet.
>>>>>>
>>>>>>> Via on-demand class parsing, we can reduce startup time from 20+
>>>>>>> seconds to 3 seconds for cold runing, and 170 ms to 140 ms for warm-up
>>>>>>> running on Core 2 Duo with Windows.
>>>>>>
>>>>>> Can you tell me how to reproduce 20+sec cold start-up? I haven't seen
>>>>>> anything like that in my simple tests.
>>>>>>
>>>>>>> After applying the patch, please note there is some change to add new modules.
>>>>>>> (1) If you want to add new modules/libraries, please don't put them in
>>>>>>> the bootclasspath.properties file. This file now only saves modules
>>>>>>> needed during startup (the VM startup only accesses class libraries in
>>>>>>> eight modules)
>>>>>>
>>>>>> That would break too much. How about creating a new file rather than
>>>>>> re-purposing an existing file with different semantics? This file is
>>>>>> used by Jikes, IBM VME, the Eclipse plug-in, at least.
>>>>>>
>>>>>>> (2) For new modules/libraries, please put them in the
>>>>>>> modulelibrarymapping.properties file. You should specify the module
>>>>>>> name and its exported class library. Here is one example:
>>>>>>> math.jar=java.math, where "math.jar" means the module name, and
>>>>>>> "java.math" means the class libraries this module exports.
>>>>>>
>>>>>> As we discussed on another thread, its unclear if the time is spent in
>>>>>> following the slow indexing through the classpath/JAR directories, or
>>>>>> whether it is speed of loading bytes once we know what we need. I think
>>>>>> that it is premature to abandon the JAR manifest data as the principal
>>>>>> source of metadata until we understand the problem this solves.
>>>>>>
>>>>>> Can we measure where the time is spent in the current implementation?
>>>>>> I think it will help guide this approach to a better solution.
>>>>>> What tools do you recommend for profiling start-up?
>>>>>>
>>>>>> Regards
>>>>>> Tim
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> С уважением,
>>>>> Алексей Федотов,
>>>>> ЗАО «Телеком Экспресс»
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> С уважением,
>>> Алексей Федотов,
>>> ЗАО «Телеком Экспресс»
>>>
>>
>
Re: [VM] On-demand class library parsing is ready to commit
Posted by Aleksey Shipilev <al...@gmail.com>.
One important thing to realize is, the invocation of main() induces a
waterfall of various Java classes (mostly kernel) to initialize.
Should they be considered in VM startup time? If yes, then the
measurements on empty main() method give just the startup time as
defined by Wenlong. Assuming there's virtually no work on shutdown and
VM just exits.
Thanks,
Aleksey.
On Mon, Dec 22, 2008 at 10:21 AM, Aleksey Shipilev
<al...@gmail.com> wrote:
> Nathan:
>
> Wenlong's definition is "startup time is the time required for VM to
> initialize before entering Java main() method". This one is measured
> by tapping into VM internals.
>
> Aleksey's definition is "startup time is the time needed for VM to
> fully execute main() method one first time". This one is measured by
> standard Java benchmarks.
>
> My impression (backed by Cachegrind/Callgrind profiles) is, there much
> more time spent in initial compilation of Java bytecode than VM
> initialization sequence. This is also demonstrated by running real
> workload through startup scenario: even though there's the boost on
> HWA in "freezing-cold cache conditions", it disappears from startup of
> real workload.
>
> The VM initalization sequence can be a bottleneck but it would come to
> the effect when all other issues are resolved. IMO, the reduction of
> 0.1 msec startup time is neglible and does not worth messing the
> bootclasspath.
>
> Is there another advantages (not performance ones) for doing that? If
> not, I'd rather postpone it.
>
> I'm not saying Wenlong did bad job, he did great, but sometimes the
> consistency gains overcome the performance, especially is the
> performance boost is not general.
>
> Thanks,
> Aleksey.
>
> On Mon, Dec 22, 2008 at 7:58 AM, Wenlong Li <we...@gmail.com> wrote:
>> My startup means the computation needed before executing user's code
>> (in main method) (see [1][2], while Aleksey's opinion is the startup
>> benchmark in SPECJVM2008.
>> [1] http://www.oracle.com/technology/pub/articles/dev2arch/2004/01/jrockit.html
>> [2] http://www.ibm.com/developerworks/java/library/os-ecspy1/
>>
>> On Mon, Dec 22, 2008 at 8:38 AM, Nathan Beyer <nd...@apache.org> wrote:
>>> Can someone give a quick summary of the two different definitions of
>>> "startup" being discussed?
>>>
>>> -Nathan
>>>
>>> On Sun, Dec 21, 2008 at 6:22 PM, Wenlong Li <we...@gmail.com> wrote:
>>>> Aleksey,
>>>>
>>>> Thx for testing this patch, and sharing your experimental result.
>>>> Yes, I think your result would be reasonable. The performance gain of
>>>> this patch varies with different systems.
>>>>
>>>> Again, I would like to say we have different definitions for "startup".
>>>> Maybe I should move the change in classlib module to vm module, so
>>>> that the dependency can be minimized.
>>>>
>>>> thx again for discussion. :)
>>>> wenlong
>>>>
>>>> On Mon, Dec 22, 2008 at 4:04 AM, Aleksey Shipilev
>>>> <al...@gmail.com> wrote:
>>>>> Hi Wenlong,
>>>>>
>>>>> I had some performance experiments with your patch. The test system is:
>>>>> - Pentium D 820 2.8 Ghz / 2 Gb DDR2-667
>>>>> - WD 3200KS, 320 Gb, 16 Mb cache
>>>>> - Gentoo Linux x86, 2.6.23
>>>>> - Harmony r728459
>>>>> - SPECjvm2008
>>>>>
>>>>> To recreate the stressful conditions over and over the simple script
>>>>> was written [1]. The script invalidates the caches before actually
>>>>> starting the workload: re-reads the same 64 Mb file a couple of times
>>>>> to fill out on-HDD cache, invalidating VFS block caches first to make
>>>>> sure the data is really requested from the disk.
>>>>>
>>>>> On HWA [2] these performance results were produced:
>>>>>
>>>>> "cold-start" (invalidate caches):
>>>>> clean: (5.24 +- 0.28) secs
>>>>> ondemand: (4.49 +- 0.17) secs
>>>>>
>>>>> "warm-start" (don't invalidate caches);
>>>>> clean: (2.82 +- 0.01) secs
>>>>> ondemand: (2.80 +- 0.02) secs
>>>>>
>>>>> That is, on-demand patch does bring +17% (-+9%) improvement on HWA
>>>>> when running with flushed caches, and does not bring any performance
>>>>> improvement in warm mode.
>>>>>
>>>>> As I mentioned several times, this test does not reflect the real
>>>>> performance end user would perceive, so I took two SPECjvm2008:startup
>>>>> benchmarks and run each of them 10x10 times.
>>>>>
>>>>> SPECjvm2008:startup.helloworld, "cold start":
>>>>> clean: (8.93 +- 0.21) ops/min
>>>>> ondemand: (9.04 +- 0.03) ops/min
>>>>>
>>>>> SPECjvm2008:startup.compiler.compiler, "cold start":
>>>>> clean: (1.44 +- 0.05) ops/min
>>>>> ondemand: (1.42 +- 0.04) ops/min
>>>>>
>>>>> As you can see even in very stressful situation there's no boost. I
>>>>> would find these performance results unconvincing to change the
>>>>> infrastructure of boolclasspath resolution. Am I missing something
>>>>> important?
>>>>>
>>>>> Thanks,
>>>>> Aleksey.
>>>>>
>>>>> [1] run.sh
>>>>> #!/bin/bash
>>>>>
>>>>> R=`pwd`
>>>>>
>>>>> JAVA=$R/platforms/builds/harmony-release-clean/jdk/jre/bin/java
>>>>> #JAVA=$R/platforms/builds/harmony-release-ondemand/jdk/jre/bin/java
>>>>> JAVA_OPTS="-Xmx1024M -Xms1024M"
>>>>>
>>>>> for T in `seq 1 10`; do
>>>>>
>>>>> echo "*************** EXECUTING ITERATION $T ****************"
>>>>>
>>>>> # invalidate HDD caches
>>>>> # - need to replace all entries in LRU HDD cache
>>>>> # - flush the kernel VFS cache first to ensure the data
>>>>> would be read from disk
>>>>>
>>>>> echo "Flushing caches"
>>>>> for I in `seq 1 5`; do
>>>>> sync
>>>>> echo 3 > /proc/sys/vm/drop_caches
>>>>>
>>>>> dd if=cachekiller.file of=/dev/null > /dev/null 2>&1
>>>>> done
>>>>>
>>>>> echo "Executing."
>>>>>
>>>>> # HelloWorld
>>>>> /usr/bin/time $JAVA $JAVA_OPTS -cp benchmarks/ HelloWorld 2>&1
>>>>>
>>>>> # SPECjvm2008
>>>>> #cd $R/benchmarks/storage/SPECjvm2008
>>>>> #/usr/bin/time $JAVA $JAVA_OPTS -Djava.awt.headless=true -jar
>>>>> SPECjvm2008.jar -ikv -i 10 startup.compiler.compiler 2>&1
>>>>>
>>>>> echo ""
>>>>> done
>>>>>
>>>>> [2] HelloWorld.java
>>>>> public class HelloWorld {
>>>>> public static void main(String[] args) {
>>>>> System.out.println("Hello, world!");
>>>>> }
>>>>> }
>>>>>
>>>>>
>>>>> On Sun, Dec 21, 2008 at 6:02 AM, Wenlong Li <we...@gmail.com> wrote:
>>>>>> On Sat, Dec 20, 2008 at 7:10 PM, Alexei Fedotov
>>>>>> <al...@gmail.com> wrote:
>>>>>>> Wenlong,
>>>>>>> Thanks for removing the commented code.
>>>>>>>
>>>>>>> There are several VMs which make use of the Harmony class library,
>>>>>>> e.g. Harmony VM, J9, Android Dalvik, etc. Your change is Harmony VM
>>>>>>> specific, isn't it? If it is, then it's better to keep related changes
>>>>>>> in the VM module. If it is not, then it might be a good idea to keep
>>>>>>> the changes in the class library module unless other VMs already has
>>>>>>> such optimization in their code.
>>>>>> [Wenlong] Though at this moment, you can think on-demand class parsing
>>>>>> is a specif optimization from your point of view. I believe it could
>>>>>> be a general technique, e.g., it can be easily deployed in other
>>>>>> runtime systems. Current VM also depends on the luniglobal.c in
>>>>>> working_classlib to get all class libraries/modules. e.g., there is a
>>>>>> cross-module dependence between classlib and VM. When user wants to
>>>>>> add new module, they should manually change the
>>>>>> bootclasspath.properties, while if applying this patch, user should
>>>>>> revise my added property file instead of the bootclasspath.properties.
>>>>>> I understand modifying bootclasspath file may be a specification.
>>>>>>>
>>>>>>> In any case crossing module boundary would make class library users
>>>>>>> think more than once or even write some code. Is it technically
>>>>>>> possible to prepare a patch which does not change module boundaries?
>>>>>>> What do you think?
>>>>>> [Wenlong] Yes, it is possible from technical perspective, but a little
>>>>>> complicated. I can think about it. :)
>>>>>>
>>>>>>>
>>>>>>> As for your performance experiments, which particular test are your
>>>>>>> measuring? It is bootclasspath-unpretentious "Hello, world", isn't it?
>>>>>> [Wenlong] My startup means the work executed before running user's
>>>>>> computation. That is, the vm creation time. I manually add
>>>>>> instrumentation code for execution time in JNI_CreateJavaVM of
>>>>>> JNI.cpp. This startup work is common for any benchmarks. My experiment
>>>>>> was conducted on both Windows and Linux system. Please see my previous
>>>>>> message about performance gain from this optimization.
>>>>>>
>>>>>> Thx,
>>>>>> Wenlong
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>> On Sat, Dec 20, 2008 at 2:19 AM, Wenlong Li <we...@gmail.com> wrote:
>>>>>>>> On Sat, Dec 20, 2008 at 12:42 AM, Alexei Fedotov
>>>>>>>> <al...@gmail.com> wrote:
>>>>>>>>> Wenlong,
>>>>>>>>> Have I missed a discussion of the proposed design? I see that you
>>>>>>>>> expose a new public interface:
>>>>>>>>> /**
>>>>>>>>> * @map the jar with exported package in the pending jar list for
>>>>>>>>> on-demand jar parsing
>>>>>>>>> * Key is the jar, and value is the package exported by this jar
>>>>>>>>> */
>>>>>>>>> DECLARE_OPEN(void, vm_properties_set_pending_jar, (const char* key,
>>>>>>>>> const char* value));
>>>>>>>>>
>>>>>>>>> Did you mean "Maps" instead of "@map"? Strangely the word "pending"
>>>>>>>>> disappeared from the name of the wrapping VMI interface
>>>>>>>>> SetJarPackageMapping . Why should we extend both OPEN and VMI
>>>>>>>>> interfaces with the same function? Why did you put your code into
>>>>>>>>> working_classlib/modules/luni/src/main/native/luni/shared/luniglob.c,
>>>>>>>>> thus introducing another dependency between VM and class library?
>>>>>>>> [Wenlong] The boot class path is defined in luniglobal.c in Harmony,
>>>>>>>> and it also has dependence with VM. In my understanding, my patch is
>>>>>>>> related to boot class path determination, so I also put my code in
>>>>>>>> luniglobal.c, and use VMI interface to communicate with VM.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> + //rcSetProperty = (*vmInterface)->SetJarPackageMapping
>>>>>>>>> (vmInterface, jarName, jarValue);
>>>>>>>>> + /*
>>>>>>>>> + hymem_free_memory(jarName);
>>>>>>>>> + hymem_free_memory(jarValue);
>>>>>>>>> + */
>>>>>>>>> Should we really commit the commented code?
>>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>> [Wenlong] Please see my latest version of patch in the list. Such
>>>>>>>> commented code has been removed.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Dec 19, 2008 at 6:59 PM, Tim Ellison <t....@gmail.com> wrote:
>>>>>>>>>> I was hoping that somebody else would comment first, so I don't have to
>>>>>>>>>> be the grumpy one all the time :-)
>>>>>>>>>>
>>>>>>>>>> As I said before, this is good prototyping work...
>>>>>>>>>>
>>>>>>>>>> Wenlong Li wrote:
>>>>>>>>>>> I did the pre-commit test on the patch of on-demand class library
>>>>>>>>>>> parsing (https://issues.apache.org/jira/browse/HARMONY-6039), and it
>>>>>>>>>>> works well now.
>>>>>>>>>>> Can Harmony incorporate this feature?
>>>>>>>>>>
>>>>>>>>>> I'm not sure it is ready for committing to the head stream yet.
>>>>>>>>>>
>>>>>>>>>>> Via on-demand class parsing, we can reduce startup time from 20+
>>>>>>>>>>> seconds to 3 seconds for cold runing, and 170 ms to 140 ms for warm-up
>>>>>>>>>>> running on Core 2 Duo with Windows.
>>>>>>>>>>
>>>>>>>>>> Can you tell me how to reproduce 20+sec cold start-up? I haven't seen
>>>>>>>>>> anything like that in my simple tests.
>>>>>>>>>>
>>>>>>>>>>> After applying the patch, please note there is some change to add new modules.
>>>>>>>>>>> (1) If you want to add new modules/libraries, please don't put them in
>>>>>>>>>>> the bootclasspath.properties file. This file now only saves modules
>>>>>>>>>>> needed during startup (the VM startup only accesses class libraries in
>>>>>>>>>>> eight modules)
>>>>>>>>>>
>>>>>>>>>> That would break too much. How about creating a new file rather than
>>>>>>>>>> re-purposing an existing file with different semantics? This file is
>>>>>>>>>> used by Jikes, IBM VME, the Eclipse plug-in, at least.
>>>>>>>>>>
>>>>>>>>>>> (2) For new modules/libraries, please put them in the
>>>>>>>>>>> modulelibrarymapping.properties file. You should specify the module
>>>>>>>>>>> name and its exported class library. Here is one example:
>>>>>>>>>>> math.jar=java.math, where "math.jar" means the module name, and
>>>>>>>>>>> "java.math" means the class libraries this module exports.
>>>>>>>>>>
>>>>>>>>>> As we discussed on another thread, its unclear if the time is spent in
>>>>>>>>>> following the slow indexing through the classpath/JAR directories, or
>>>>>>>>>> whether it is speed of loading bytes once we know what we need. I think
>>>>>>>>>> that it is premature to abandon the JAR manifest data as the principal
>>>>>>>>>> source of metadata until we understand the problem this solves.
>>>>>>>>>>
>>>>>>>>>> Can we measure where the time is spent in the current implementation?
>>>>>>>>>> I think it will help guide this approach to a better solution.
>>>>>>>>>> What tools do you recommend for profiling start-up?
>>>>>>>>>>
>>>>>>>>>> Regards
>>>>>>>>>> Tim
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> С уважением,
>>>>>>>>> Алексей Федотов,
>>>>>>>>> ЗАО «Телеком Экспресс»
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> С уважением,
>>>>>>> Алексей Федотов,
>>>>>>> ЗАО «Телеком Экспресс»
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: [VM] On-demand class library parsing is ready to commit
Posted by Aleksey Shipilev <al...@gmail.com>.
Nathan:
Wenlong's definition is "startup time is the time required for VM to
initialize before entering Java main() method". This one is measured
by tapping into VM internals.
Aleksey's definition is "startup time is the time needed for VM to
fully execute main() method one first time". This one is measured by
standard Java benchmarks.
My impression (backed by Cachegrind/Callgrind profiles) is, there much
more time spent in initial compilation of Java bytecode than VM
initialization sequence. This is also demonstrated by running real
workload through startup scenario: even though there's the boost on
HWA in "freezing-cold cache conditions", it disappears from startup of
real workload.
The VM initalization sequence can be a bottleneck but it would come to
the effect when all other issues are resolved. IMO, the reduction of
0.1 msec startup time is neglible and does not worth messing the
bootclasspath.
Is there another advantages (not performance ones) for doing that? If
not, I'd rather postpone it.
I'm not saying Wenlong did bad job, he did great, but sometimes the
consistency gains overcome the performance, especially is the
performance boost is not general.
Thanks,
Aleksey.
On Mon, Dec 22, 2008 at 7:58 AM, Wenlong Li <we...@gmail.com> wrote:
> My startup means the computation needed before executing user's code
> (in main method) (see [1][2], while Aleksey's opinion is the startup
> benchmark in SPECJVM2008.
> [1] http://www.oracle.com/technology/pub/articles/dev2arch/2004/01/jrockit.html
> [2] http://www.ibm.com/developerworks/java/library/os-ecspy1/
>
> On Mon, Dec 22, 2008 at 8:38 AM, Nathan Beyer <nd...@apache.org> wrote:
>> Can someone give a quick summary of the two different definitions of
>> "startup" being discussed?
>>
>> -Nathan
>>
>> On Sun, Dec 21, 2008 at 6:22 PM, Wenlong Li <we...@gmail.com> wrote:
>>> Aleksey,
>>>
>>> Thx for testing this patch, and sharing your experimental result.
>>> Yes, I think your result would be reasonable. The performance gain of
>>> this patch varies with different systems.
>>>
>>> Again, I would like to say we have different definitions for "startup".
>>> Maybe I should move the change in classlib module to vm module, so
>>> that the dependency can be minimized.
>>>
>>> thx again for discussion. :)
>>> wenlong
>>>
>>> On Mon, Dec 22, 2008 at 4:04 AM, Aleksey Shipilev
>>> <al...@gmail.com> wrote:
>>>> Hi Wenlong,
>>>>
>>>> I had some performance experiments with your patch. The test system is:
>>>> - Pentium D 820 2.8 Ghz / 2 Gb DDR2-667
>>>> - WD 3200KS, 320 Gb, 16 Mb cache
>>>> - Gentoo Linux x86, 2.6.23
>>>> - Harmony r728459
>>>> - SPECjvm2008
>>>>
>>>> To recreate the stressful conditions over and over the simple script
>>>> was written [1]. The script invalidates the caches before actually
>>>> starting the workload: re-reads the same 64 Mb file a couple of times
>>>> to fill out on-HDD cache, invalidating VFS block caches first to make
>>>> sure the data is really requested from the disk.
>>>>
>>>> On HWA [2] these performance results were produced:
>>>>
>>>> "cold-start" (invalidate caches):
>>>> clean: (5.24 +- 0.28) secs
>>>> ondemand: (4.49 +- 0.17) secs
>>>>
>>>> "warm-start" (don't invalidate caches);
>>>> clean: (2.82 +- 0.01) secs
>>>> ondemand: (2.80 +- 0.02) secs
>>>>
>>>> That is, on-demand patch does bring +17% (-+9%) improvement on HWA
>>>> when running with flushed caches, and does not bring any performance
>>>> improvement in warm mode.
>>>>
>>>> As I mentioned several times, this test does not reflect the real
>>>> performance end user would perceive, so I took two SPECjvm2008:startup
>>>> benchmarks and run each of them 10x10 times.
>>>>
>>>> SPECjvm2008:startup.helloworld, "cold start":
>>>> clean: (8.93 +- 0.21) ops/min
>>>> ondemand: (9.04 +- 0.03) ops/min
>>>>
>>>> SPECjvm2008:startup.compiler.compiler, "cold start":
>>>> clean: (1.44 +- 0.05) ops/min
>>>> ondemand: (1.42 +- 0.04) ops/min
>>>>
>>>> As you can see even in very stressful situation there's no boost. I
>>>> would find these performance results unconvincing to change the
>>>> infrastructure of boolclasspath resolution. Am I missing something
>>>> important?
>>>>
>>>> Thanks,
>>>> Aleksey.
>>>>
>>>> [1] run.sh
>>>> #!/bin/bash
>>>>
>>>> R=`pwd`
>>>>
>>>> JAVA=$R/platforms/builds/harmony-release-clean/jdk/jre/bin/java
>>>> #JAVA=$R/platforms/builds/harmony-release-ondemand/jdk/jre/bin/java
>>>> JAVA_OPTS="-Xmx1024M -Xms1024M"
>>>>
>>>> for T in `seq 1 10`; do
>>>>
>>>> echo "*************** EXECUTING ITERATION $T ****************"
>>>>
>>>> # invalidate HDD caches
>>>> # - need to replace all entries in LRU HDD cache
>>>> # - flush the kernel VFS cache first to ensure the data
>>>> would be read from disk
>>>>
>>>> echo "Flushing caches"
>>>> for I in `seq 1 5`; do
>>>> sync
>>>> echo 3 > /proc/sys/vm/drop_caches
>>>>
>>>> dd if=cachekiller.file of=/dev/null > /dev/null 2>&1
>>>> done
>>>>
>>>> echo "Executing."
>>>>
>>>> # HelloWorld
>>>> /usr/bin/time $JAVA $JAVA_OPTS -cp benchmarks/ HelloWorld 2>&1
>>>>
>>>> # SPECjvm2008
>>>> #cd $R/benchmarks/storage/SPECjvm2008
>>>> #/usr/bin/time $JAVA $JAVA_OPTS -Djava.awt.headless=true -jar
>>>> SPECjvm2008.jar -ikv -i 10 startup.compiler.compiler 2>&1
>>>>
>>>> echo ""
>>>> done
>>>>
>>>> [2] HelloWorld.java
>>>> public class HelloWorld {
>>>> public static void main(String[] args) {
>>>> System.out.println("Hello, world!");
>>>> }
>>>> }
>>>>
>>>>
>>>> On Sun, Dec 21, 2008 at 6:02 AM, Wenlong Li <we...@gmail.com> wrote:
>>>>> On Sat, Dec 20, 2008 at 7:10 PM, Alexei Fedotov
>>>>> <al...@gmail.com> wrote:
>>>>>> Wenlong,
>>>>>> Thanks for removing the commented code.
>>>>>>
>>>>>> There are several VMs which make use of the Harmony class library,
>>>>>> e.g. Harmony VM, J9, Android Dalvik, etc. Your change is Harmony VM
>>>>>> specific, isn't it? If it is, then it's better to keep related changes
>>>>>> in the VM module. If it is not, then it might be a good idea to keep
>>>>>> the changes in the class library module unless other VMs already has
>>>>>> such optimization in their code.
>>>>> [Wenlong] Though at this moment, you can think on-demand class parsing
>>>>> is a specif optimization from your point of view. I believe it could
>>>>> be a general technique, e.g., it can be easily deployed in other
>>>>> runtime systems. Current VM also depends on the luniglobal.c in
>>>>> working_classlib to get all class libraries/modules. e.g., there is a
>>>>> cross-module dependence between classlib and VM. When user wants to
>>>>> add new module, they should manually change the
>>>>> bootclasspath.properties, while if applying this patch, user should
>>>>> revise my added property file instead of the bootclasspath.properties.
>>>>> I understand modifying bootclasspath file may be a specification.
>>>>>>
>>>>>> In any case crossing module boundary would make class library users
>>>>>> think more than once or even write some code. Is it technically
>>>>>> possible to prepare a patch which does not change module boundaries?
>>>>>> What do you think?
>>>>> [Wenlong] Yes, it is possible from technical perspective, but a little
>>>>> complicated. I can think about it. :)
>>>>>
>>>>>>
>>>>>> As for your performance experiments, which particular test are your
>>>>>> measuring? It is bootclasspath-unpretentious "Hello, world", isn't it?
>>>>> [Wenlong] My startup means the work executed before running user's
>>>>> computation. That is, the vm creation time. I manually add
>>>>> instrumentation code for execution time in JNI_CreateJavaVM of
>>>>> JNI.cpp. This startup work is common for any benchmarks. My experiment
>>>>> was conducted on both Windows and Linux system. Please see my previous
>>>>> message about performance gain from this optimization.
>>>>>
>>>>> Thx,
>>>>> Wenlong
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>> On Sat, Dec 20, 2008 at 2:19 AM, Wenlong Li <we...@gmail.com> wrote:
>>>>>>> On Sat, Dec 20, 2008 at 12:42 AM, Alexei Fedotov
>>>>>>> <al...@gmail.com> wrote:
>>>>>>>> Wenlong,
>>>>>>>> Have I missed a discussion of the proposed design? I see that you
>>>>>>>> expose a new public interface:
>>>>>>>> /**
>>>>>>>> * @map the jar with exported package in the pending jar list for
>>>>>>>> on-demand jar parsing
>>>>>>>> * Key is the jar, and value is the package exported by this jar
>>>>>>>> */
>>>>>>>> DECLARE_OPEN(void, vm_properties_set_pending_jar, (const char* key,
>>>>>>>> const char* value));
>>>>>>>>
>>>>>>>> Did you mean "Maps" instead of "@map"? Strangely the word "pending"
>>>>>>>> disappeared from the name of the wrapping VMI interface
>>>>>>>> SetJarPackageMapping . Why should we extend both OPEN and VMI
>>>>>>>> interfaces with the same function? Why did you put your code into
>>>>>>>> working_classlib/modules/luni/src/main/native/luni/shared/luniglob.c,
>>>>>>>> thus introducing another dependency between VM and class library?
>>>>>>> [Wenlong] The boot class path is defined in luniglobal.c in Harmony,
>>>>>>> and it also has dependence with VM. In my understanding, my patch is
>>>>>>> related to boot class path determination, so I also put my code in
>>>>>>> luniglobal.c, and use VMI interface to communicate with VM.
>>>>>>>
>>>>>>>>
>>>>>>>> + //rcSetProperty = (*vmInterface)->SetJarPackageMapping
>>>>>>>> (vmInterface, jarName, jarValue);
>>>>>>>> + /*
>>>>>>>> + hymem_free_memory(jarName);
>>>>>>>> + hymem_free_memory(jarValue);
>>>>>>>> + */
>>>>>>>> Should we really commit the commented code?
>>>>>>>> Thanks.
>>>>>>>
>>>>>>> [Wenlong] Please see my latest version of patch in the list. Such
>>>>>>> commented code has been removed.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Dec 19, 2008 at 6:59 PM, Tim Ellison <t....@gmail.com> wrote:
>>>>>>>>> I was hoping that somebody else would comment first, so I don't have to
>>>>>>>>> be the grumpy one all the time :-)
>>>>>>>>>
>>>>>>>>> As I said before, this is good prototyping work...
>>>>>>>>>
>>>>>>>>> Wenlong Li wrote:
>>>>>>>>>> I did the pre-commit test on the patch of on-demand class library
>>>>>>>>>> parsing (https://issues.apache.org/jira/browse/HARMONY-6039), and it
>>>>>>>>>> works well now.
>>>>>>>>>> Can Harmony incorporate this feature?
>>>>>>>>>
>>>>>>>>> I'm not sure it is ready for committing to the head stream yet.
>>>>>>>>>
>>>>>>>>>> Via on-demand class parsing, we can reduce startup time from 20+
>>>>>>>>>> seconds to 3 seconds for cold runing, and 170 ms to 140 ms for warm-up
>>>>>>>>>> running on Core 2 Duo with Windows.
>>>>>>>>>
>>>>>>>>> Can you tell me how to reproduce 20+sec cold start-up? I haven't seen
>>>>>>>>> anything like that in my simple tests.
>>>>>>>>>
>>>>>>>>>> After applying the patch, please note there is some change to add new modules.
>>>>>>>>>> (1) If you want to add new modules/libraries, please don't put them in
>>>>>>>>>> the bootclasspath.properties file. This file now only saves modules
>>>>>>>>>> needed during startup (the VM startup only accesses class libraries in
>>>>>>>>>> eight modules)
>>>>>>>>>
>>>>>>>>> That would break too much. How about creating a new file rather than
>>>>>>>>> re-purposing an existing file with different semantics? This file is
>>>>>>>>> used by Jikes, IBM VME, the Eclipse plug-in, at least.
>>>>>>>>>
>>>>>>>>>> (2) For new modules/libraries, please put them in the
>>>>>>>>>> modulelibrarymapping.properties file. You should specify the module
>>>>>>>>>> name and its exported class library. Here is one example:
>>>>>>>>>> math.jar=java.math, where "math.jar" means the module name, and
>>>>>>>>>> "java.math" means the class libraries this module exports.
>>>>>>>>>
>>>>>>>>> As we discussed on another thread, its unclear if the time is spent in
>>>>>>>>> following the slow indexing through the classpath/JAR directories, or
>>>>>>>>> whether it is speed of loading bytes once we know what we need. I think
>>>>>>>>> that it is premature to abandon the JAR manifest data as the principal
>>>>>>>>> source of metadata until we understand the problem this solves.
>>>>>>>>>
>>>>>>>>> Can we measure where the time is spent in the current implementation?
>>>>>>>>> I think it will help guide this approach to a better solution.
>>>>>>>>> What tools do you recommend for profiling start-up?
>>>>>>>>>
>>>>>>>>> Regards
>>>>>>>>> Tim
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> С уважением,
>>>>>>>> Алексей Федотов,
>>>>>>>> ЗАО «Телеком Экспресс»
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> С уважением,
>>>>>> Алексей Федотов,
>>>>>> ЗАО «Телеком Экспресс»
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: [VM] On-demand class library parsing is ready to commit
Posted by Wenlong Li <we...@gmail.com>.
My startup means the computation needed before executing user's code
(in main method) (see [1][2], while Aleksey's opinion is the startup
benchmark in SPECJVM2008.
[1] http://www.oracle.com/technology/pub/articles/dev2arch/2004/01/jrockit.html
[2] http://www.ibm.com/developerworks/java/library/os-ecspy1/
On Mon, Dec 22, 2008 at 8:38 AM, Nathan Beyer <nd...@apache.org> wrote:
> Can someone give a quick summary of the two different definitions of
> "startup" being discussed?
>
> -Nathan
>
> On Sun, Dec 21, 2008 at 6:22 PM, Wenlong Li <we...@gmail.com> wrote:
>> Aleksey,
>>
>> Thx for testing this patch, and sharing your experimental result.
>> Yes, I think your result would be reasonable. The performance gain of
>> this patch varies with different systems.
>>
>> Again, I would like to say we have different definitions for "startup".
>> Maybe I should move the change in classlib module to vm module, so
>> that the dependency can be minimized.
>>
>> thx again for discussion. :)
>> wenlong
>>
>> On Mon, Dec 22, 2008 at 4:04 AM, Aleksey Shipilev
>> <al...@gmail.com> wrote:
>>> Hi Wenlong,
>>>
>>> I had some performance experiments with your patch. The test system is:
>>> - Pentium D 820 2.8 Ghz / 2 Gb DDR2-667
>>> - WD 3200KS, 320 Gb, 16 Mb cache
>>> - Gentoo Linux x86, 2.6.23
>>> - Harmony r728459
>>> - SPECjvm2008
>>>
>>> To recreate the stressful conditions over and over the simple script
>>> was written [1]. The script invalidates the caches before actually
>>> starting the workload: re-reads the same 64 Mb file a couple of times
>>> to fill out on-HDD cache, invalidating VFS block caches first to make
>>> sure the data is really requested from the disk.
>>>
>>> On HWA [2] these performance results were produced:
>>>
>>> "cold-start" (invalidate caches):
>>> clean: (5.24 +- 0.28) secs
>>> ondemand: (4.49 +- 0.17) secs
>>>
>>> "warm-start" (don't invalidate caches);
>>> clean: (2.82 +- 0.01) secs
>>> ondemand: (2.80 +- 0.02) secs
>>>
>>> That is, on-demand patch does bring +17% (-+9%) improvement on HWA
>>> when running with flushed caches, and does not bring any performance
>>> improvement in warm mode.
>>>
>>> As I mentioned several times, this test does not reflect the real
>>> performance end user would perceive, so I took two SPECjvm2008:startup
>>> benchmarks and run each of them 10x10 times.
>>>
>>> SPECjvm2008:startup.helloworld, "cold start":
>>> clean: (8.93 +- 0.21) ops/min
>>> ondemand: (9.04 +- 0.03) ops/min
>>>
>>> SPECjvm2008:startup.compiler.compiler, "cold start":
>>> clean: (1.44 +- 0.05) ops/min
>>> ondemand: (1.42 +- 0.04) ops/min
>>>
>>> As you can see even in very stressful situation there's no boost. I
>>> would find these performance results unconvincing to change the
>>> infrastructure of boolclasspath resolution. Am I missing something
>>> important?
>>>
>>> Thanks,
>>> Aleksey.
>>>
>>> [1] run.sh
>>> #!/bin/bash
>>>
>>> R=`pwd`
>>>
>>> JAVA=$R/platforms/builds/harmony-release-clean/jdk/jre/bin/java
>>> #JAVA=$R/platforms/builds/harmony-release-ondemand/jdk/jre/bin/java
>>> JAVA_OPTS="-Xmx1024M -Xms1024M"
>>>
>>> for T in `seq 1 10`; do
>>>
>>> echo "*************** EXECUTING ITERATION $T ****************"
>>>
>>> # invalidate HDD caches
>>> # - need to replace all entries in LRU HDD cache
>>> # - flush the kernel VFS cache first to ensure the data
>>> would be read from disk
>>>
>>> echo "Flushing caches"
>>> for I in `seq 1 5`; do
>>> sync
>>> echo 3 > /proc/sys/vm/drop_caches
>>>
>>> dd if=cachekiller.file of=/dev/null > /dev/null 2>&1
>>> done
>>>
>>> echo "Executing."
>>>
>>> # HelloWorld
>>> /usr/bin/time $JAVA $JAVA_OPTS -cp benchmarks/ HelloWorld 2>&1
>>>
>>> # SPECjvm2008
>>> #cd $R/benchmarks/storage/SPECjvm2008
>>> #/usr/bin/time $JAVA $JAVA_OPTS -Djava.awt.headless=true -jar
>>> SPECjvm2008.jar -ikv -i 10 startup.compiler.compiler 2>&1
>>>
>>> echo ""
>>> done
>>>
>>> [2] HelloWorld.java
>>> public class HelloWorld {
>>> public static void main(String[] args) {
>>> System.out.println("Hello, world!");
>>> }
>>> }
>>>
>>>
>>> On Sun, Dec 21, 2008 at 6:02 AM, Wenlong Li <we...@gmail.com> wrote:
>>>> On Sat, Dec 20, 2008 at 7:10 PM, Alexei Fedotov
>>>> <al...@gmail.com> wrote:
>>>>> Wenlong,
>>>>> Thanks for removing the commented code.
>>>>>
>>>>> There are several VMs which make use of the Harmony class library,
>>>>> e.g. Harmony VM, J9, Android Dalvik, etc. Your change is Harmony VM
>>>>> specific, isn't it? If it is, then it's better to keep related changes
>>>>> in the VM module. If it is not, then it might be a good idea to keep
>>>>> the changes in the class library module unless other VMs already has
>>>>> such optimization in their code.
>>>> [Wenlong] Though at this moment, you can think on-demand class parsing
>>>> is a specif optimization from your point of view. I believe it could
>>>> be a general technique, e.g., it can be easily deployed in other
>>>> runtime systems. Current VM also depends on the luniglobal.c in
>>>> working_classlib to get all class libraries/modules. e.g., there is a
>>>> cross-module dependence between classlib and VM. When user wants to
>>>> add new module, they should manually change the
>>>> bootclasspath.properties, while if applying this patch, user should
>>>> revise my added property file instead of the bootclasspath.properties.
>>>> I understand modifying bootclasspath file may be a specification.
>>>>>
>>>>> In any case crossing module boundary would make class library users
>>>>> think more than once or even write some code. Is it technically
>>>>> possible to prepare a patch which does not change module boundaries?
>>>>> What do you think?
>>>> [Wenlong] Yes, it is possible from technical perspective, but a little
>>>> complicated. I can think about it. :)
>>>>
>>>>>
>>>>> As for your performance experiments, which particular test are your
>>>>> measuring? It is bootclasspath-unpretentious "Hello, world", isn't it?
>>>> [Wenlong] My startup means the work executed before running user's
>>>> computation. That is, the vm creation time. I manually add
>>>> instrumentation code for execution time in JNI_CreateJavaVM of
>>>> JNI.cpp. This startup work is common for any benchmarks. My experiment
>>>> was conducted on both Windows and Linux system. Please see my previous
>>>> message about performance gain from this optimization.
>>>>
>>>> Thx,
>>>> Wenlong
>>>>>
>>>>> Thanks!
>>>>>
>>>>> On Sat, Dec 20, 2008 at 2:19 AM, Wenlong Li <we...@gmail.com> wrote:
>>>>>> On Sat, Dec 20, 2008 at 12:42 AM, Alexei Fedotov
>>>>>> <al...@gmail.com> wrote:
>>>>>>> Wenlong,
>>>>>>> Have I missed a discussion of the proposed design? I see that you
>>>>>>> expose a new public interface:
>>>>>>> /**
>>>>>>> * @map the jar with exported package in the pending jar list for
>>>>>>> on-demand jar parsing
>>>>>>> * Key is the jar, and value is the package exported by this jar
>>>>>>> */
>>>>>>> DECLARE_OPEN(void, vm_properties_set_pending_jar, (const char* key,
>>>>>>> const char* value));
>>>>>>>
>>>>>>> Did you mean "Maps" instead of "@map"? Strangely the word "pending"
>>>>>>> disappeared from the name of the wrapping VMI interface
>>>>>>> SetJarPackageMapping . Why should we extend both OPEN and VMI
>>>>>>> interfaces with the same function? Why did you put your code into
>>>>>>> working_classlib/modules/luni/src/main/native/luni/shared/luniglob.c,
>>>>>>> thus introducing another dependency between VM and class library?
>>>>>> [Wenlong] The boot class path is defined in luniglobal.c in Harmony,
>>>>>> and it also has dependence with VM. In my understanding, my patch is
>>>>>> related to boot class path determination, so I also put my code in
>>>>>> luniglobal.c, and use VMI interface to communicate with VM.
>>>>>>
>>>>>>>
>>>>>>> + //rcSetProperty = (*vmInterface)->SetJarPackageMapping
>>>>>>> (vmInterface, jarName, jarValue);
>>>>>>> + /*
>>>>>>> + hymem_free_memory(jarName);
>>>>>>> + hymem_free_memory(jarValue);
>>>>>>> + */
>>>>>>> Should we really commit the commented code?
>>>>>>> Thanks.
>>>>>>
>>>>>> [Wenlong] Please see my latest version of patch in the list. Such
>>>>>> commented code has been removed.
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Dec 19, 2008 at 6:59 PM, Tim Ellison <t....@gmail.com> wrote:
>>>>>>>> I was hoping that somebody else would comment first, so I don't have to
>>>>>>>> be the grumpy one all the time :-)
>>>>>>>>
>>>>>>>> As I said before, this is good prototyping work...
>>>>>>>>
>>>>>>>> Wenlong Li wrote:
>>>>>>>>> I did the pre-commit test on the patch of on-demand class library
>>>>>>>>> parsing (https://issues.apache.org/jira/browse/HARMONY-6039), and it
>>>>>>>>> works well now.
>>>>>>>>> Can Harmony incorporate this feature?
>>>>>>>>
>>>>>>>> I'm not sure it is ready for committing to the head stream yet.
>>>>>>>>
>>>>>>>>> Via on-demand class parsing, we can reduce startup time from 20+
>>>>>>>>> seconds to 3 seconds for cold runing, and 170 ms to 140 ms for warm-up
>>>>>>>>> running on Core 2 Duo with Windows.
>>>>>>>>
>>>>>>>> Can you tell me how to reproduce 20+sec cold start-up? I haven't seen
>>>>>>>> anything like that in my simple tests.
>>>>>>>>
>>>>>>>>> After applying the patch, please note there is some change to add new modules.
>>>>>>>>> (1) If you want to add new modules/libraries, please don't put them in
>>>>>>>>> the bootclasspath.properties file. This file now only saves modules
>>>>>>>>> needed during startup (the VM startup only accesses class libraries in
>>>>>>>>> eight modules)
>>>>>>>>
>>>>>>>> That would break too much. How about creating a new file rather than
>>>>>>>> re-purposing an existing file with different semantics? This file is
>>>>>>>> used by Jikes, IBM VME, the Eclipse plug-in, at least.
>>>>>>>>
>>>>>>>>> (2) For new modules/libraries, please put them in the
>>>>>>>>> modulelibrarymapping.properties file. You should specify the module
>>>>>>>>> name and its exported class library. Here is one example:
>>>>>>>>> math.jar=java.math, where "math.jar" means the module name, and
>>>>>>>>> "java.math" means the class libraries this module exports.
>>>>>>>>
>>>>>>>> As we discussed on another thread, its unclear if the time is spent in
>>>>>>>> following the slow indexing through the classpath/JAR directories, or
>>>>>>>> whether it is speed of loading bytes once we know what we need. I think
>>>>>>>> that it is premature to abandon the JAR manifest data as the principal
>>>>>>>> source of metadata until we understand the problem this solves.
>>>>>>>>
>>>>>>>> Can we measure where the time is spent in the current implementation?
>>>>>>>> I think it will help guide this approach to a better solution.
>>>>>>>> What tools do you recommend for profiling start-up?
>>>>>>>>
>>>>>>>> Regards
>>>>>>>> Tim
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> С уважением,
>>>>>>> Алексей Федотов,
>>>>>>> ЗАО «Телеком Экспресс»
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> С уважением,
>>>>> Алексей Федотов,
>>>>> ЗАО «Телеком Экспресс»
>>>>>
>>>>
>>>
>>
>
Re: [VM] On-demand class library parsing is ready to commit
Posted by Nathan Beyer <nd...@apache.org>.
Can someone give a quick summary of the two different definitions of
"startup" being discussed?
-Nathan
On Sun, Dec 21, 2008 at 6:22 PM, Wenlong Li <we...@gmail.com> wrote:
> Aleksey,
>
> Thx for testing this patch, and sharing your experimental result.
> Yes, I think your result would be reasonable. The performance gain of
> this patch varies with different systems.
>
> Again, I would like to say we have different definitions for "startup".
> Maybe I should move the change in classlib module to vm module, so
> that the dependency can be minimized.
>
> thx again for discussion. :)
> wenlong
>
> On Mon, Dec 22, 2008 at 4:04 AM, Aleksey Shipilev
> <al...@gmail.com> wrote:
>> Hi Wenlong,
>>
>> I had some performance experiments with your patch. The test system is:
>> - Pentium D 820 2.8 Ghz / 2 Gb DDR2-667
>> - WD 3200KS, 320 Gb, 16 Mb cache
>> - Gentoo Linux x86, 2.6.23
>> - Harmony r728459
>> - SPECjvm2008
>>
>> To recreate the stressful conditions over and over the simple script
>> was written [1]. The script invalidates the caches before actually
>> starting the workload: re-reads the same 64 Mb file a couple of times
>> to fill out on-HDD cache, invalidating VFS block caches first to make
>> sure the data is really requested from the disk.
>>
>> On HWA [2] these performance results were produced:
>>
>> "cold-start" (invalidate caches):
>> clean: (5.24 +- 0.28) secs
>> ondemand: (4.49 +- 0.17) secs
>>
>> "warm-start" (don't invalidate caches);
>> clean: (2.82 +- 0.01) secs
>> ondemand: (2.80 +- 0.02) secs
>>
>> That is, on-demand patch does bring +17% (-+9%) improvement on HWA
>> when running with flushed caches, and does not bring any performance
>> improvement in warm mode.
>>
>> As I mentioned several times, this test does not reflect the real
>> performance end user would perceive, so I took two SPECjvm2008:startup
>> benchmarks and run each of them 10x10 times.
>>
>> SPECjvm2008:startup.helloworld, "cold start":
>> clean: (8.93 +- 0.21) ops/min
>> ondemand: (9.04 +- 0.03) ops/min
>>
>> SPECjvm2008:startup.compiler.compiler, "cold start":
>> clean: (1.44 +- 0.05) ops/min
>> ondemand: (1.42 +- 0.04) ops/min
>>
>> As you can see even in very stressful situation there's no boost. I
>> would find these performance results unconvincing to change the
>> infrastructure of boolclasspath resolution. Am I missing something
>> important?
>>
>> Thanks,
>> Aleksey.
>>
>> [1] run.sh
>> #!/bin/bash
>>
>> R=`pwd`
>>
>> JAVA=$R/platforms/builds/harmony-release-clean/jdk/jre/bin/java
>> #JAVA=$R/platforms/builds/harmony-release-ondemand/jdk/jre/bin/java
>> JAVA_OPTS="-Xmx1024M -Xms1024M"
>>
>> for T in `seq 1 10`; do
>>
>> echo "*************** EXECUTING ITERATION $T ****************"
>>
>> # invalidate HDD caches
>> # - need to replace all entries in LRU HDD cache
>> # - flush the kernel VFS cache first to ensure the data
>> would be read from disk
>>
>> echo "Flushing caches"
>> for I in `seq 1 5`; do
>> sync
>> echo 3 > /proc/sys/vm/drop_caches
>>
>> dd if=cachekiller.file of=/dev/null > /dev/null 2>&1
>> done
>>
>> echo "Executing."
>>
>> # HelloWorld
>> /usr/bin/time $JAVA $JAVA_OPTS -cp benchmarks/ HelloWorld 2>&1
>>
>> # SPECjvm2008
>> #cd $R/benchmarks/storage/SPECjvm2008
>> #/usr/bin/time $JAVA $JAVA_OPTS -Djava.awt.headless=true -jar
>> SPECjvm2008.jar -ikv -i 10 startup.compiler.compiler 2>&1
>>
>> echo ""
>> done
>>
>> [2] HelloWorld.java
>> public class HelloWorld {
>> public static void main(String[] args) {
>> System.out.println("Hello, world!");
>> }
>> }
>>
>>
>> On Sun, Dec 21, 2008 at 6:02 AM, Wenlong Li <we...@gmail.com> wrote:
>>> On Sat, Dec 20, 2008 at 7:10 PM, Alexei Fedotov
>>> <al...@gmail.com> wrote:
>>>> Wenlong,
>>>> Thanks for removing the commented code.
>>>>
>>>> There are several VMs which make use of the Harmony class library,
>>>> e.g. Harmony VM, J9, Android Dalvik, etc. Your change is Harmony VM
>>>> specific, isn't it? If it is, then it's better to keep related changes
>>>> in the VM module. If it is not, then it might be a good idea to keep
>>>> the changes in the class library module unless other VMs already has
>>>> such optimization in their code.
>>> [Wenlong] Though at this moment, you can think on-demand class parsing
>>> is a specif optimization from your point of view. I believe it could
>>> be a general technique, e.g., it can be easily deployed in other
>>> runtime systems. Current VM also depends on the luniglobal.c in
>>> working_classlib to get all class libraries/modules. e.g., there is a
>>> cross-module dependence between classlib and VM. When user wants to
>>> add new module, they should manually change the
>>> bootclasspath.properties, while if applying this patch, user should
>>> revise my added property file instead of the bootclasspath.properties.
>>> I understand modifying bootclasspath file may be a specification.
>>>>
>>>> In any case crossing module boundary would make class library users
>>>> think more than once or even write some code. Is it technically
>>>> possible to prepare a patch which does not change module boundaries?
>>>> What do you think?
>>> [Wenlong] Yes, it is possible from technical perspective, but a little
>>> complicated. I can think about it. :)
>>>
>>>>
>>>> As for your performance experiments, which particular test are your
>>>> measuring? It is bootclasspath-unpretentious "Hello, world", isn't it?
>>> [Wenlong] My startup means the work executed before running user's
>>> computation. That is, the vm creation time. I manually add
>>> instrumentation code for execution time in JNI_CreateJavaVM of
>>> JNI.cpp. This startup work is common for any benchmarks. My experiment
>>> was conducted on both Windows and Linux system. Please see my previous
>>> message about performance gain from this optimization.
>>>
>>> Thx,
>>> Wenlong
>>>>
>>>> Thanks!
>>>>
>>>> On Sat, Dec 20, 2008 at 2:19 AM, Wenlong Li <we...@gmail.com> wrote:
>>>>> On Sat, Dec 20, 2008 at 12:42 AM, Alexei Fedotov
>>>>> <al...@gmail.com> wrote:
>>>>>> Wenlong,
>>>>>> Have I missed a discussion of the proposed design? I see that you
>>>>>> expose a new public interface:
>>>>>> /**
>>>>>> * @map the jar with exported package in the pending jar list for
>>>>>> on-demand jar parsing
>>>>>> * Key is the jar, and value is the package exported by this jar
>>>>>> */
>>>>>> DECLARE_OPEN(void, vm_properties_set_pending_jar, (const char* key,
>>>>>> const char* value));
>>>>>>
>>>>>> Did you mean "Maps" instead of "@map"? Strangely the word "pending"
>>>>>> disappeared from the name of the wrapping VMI interface
>>>>>> SetJarPackageMapping . Why should we extend both OPEN and VMI
>>>>>> interfaces with the same function? Why did you put your code into
>>>>>> working_classlib/modules/luni/src/main/native/luni/shared/luniglob.c,
>>>>>> thus introducing another dependency between VM and class library?
>>>>> [Wenlong] The boot class path is defined in luniglobal.c in Harmony,
>>>>> and it also has dependence with VM. In my understanding, my patch is
>>>>> related to boot class path determination, so I also put my code in
>>>>> luniglobal.c, and use VMI interface to communicate with VM.
>>>>>
>>>>>>
>>>>>> + //rcSetProperty = (*vmInterface)->SetJarPackageMapping
>>>>>> (vmInterface, jarName, jarValue);
>>>>>> + /*
>>>>>> + hymem_free_memory(jarName);
>>>>>> + hymem_free_memory(jarValue);
>>>>>> + */
>>>>>> Should we really commit the commented code?
>>>>>> Thanks.
>>>>>
>>>>> [Wenlong] Please see my latest version of patch in the list. Such
>>>>> commented code has been removed.
>>>>>>
>>>>>>
>>>>>> On Fri, Dec 19, 2008 at 6:59 PM, Tim Ellison <t....@gmail.com> wrote:
>>>>>>> I was hoping that somebody else would comment first, so I don't have to
>>>>>>> be the grumpy one all the time :-)
>>>>>>>
>>>>>>> As I said before, this is good prototyping work...
>>>>>>>
>>>>>>> Wenlong Li wrote:
>>>>>>>> I did the pre-commit test on the patch of on-demand class library
>>>>>>>> parsing (https://issues.apache.org/jira/browse/HARMONY-6039), and it
>>>>>>>> works well now.
>>>>>>>> Can Harmony incorporate this feature?
>>>>>>>
>>>>>>> I'm not sure it is ready for committing to the head stream yet.
>>>>>>>
>>>>>>>> Via on-demand class parsing, we can reduce startup time from 20+
>>>>>>>> seconds to 3 seconds for cold runing, and 170 ms to 140 ms for warm-up
>>>>>>>> running on Core 2 Duo with Windows.
>>>>>>>
>>>>>>> Can you tell me how to reproduce 20+sec cold start-up? I haven't seen
>>>>>>> anything like that in my simple tests.
>>>>>>>
>>>>>>>> After applying the patch, please note there is some change to add new modules.
>>>>>>>> (1) If you want to add new modules/libraries, please don't put them in
>>>>>>>> the bootclasspath.properties file. This file now only saves modules
>>>>>>>> needed during startup (the VM startup only accesses class libraries in
>>>>>>>> eight modules)
>>>>>>>
>>>>>>> That would break too much. How about creating a new file rather than
>>>>>>> re-purposing an existing file with different semantics? This file is
>>>>>>> used by Jikes, IBM VME, the Eclipse plug-in, at least.
>>>>>>>
>>>>>>>> (2) For new modules/libraries, please put them in the
>>>>>>>> modulelibrarymapping.properties file. You should specify the module
>>>>>>>> name and its exported class library. Here is one example:
>>>>>>>> math.jar=java.math, where "math.jar" means the module name, and
>>>>>>>> "java.math" means the class libraries this module exports.
>>>>>>>
>>>>>>> As we discussed on another thread, its unclear if the time is spent in
>>>>>>> following the slow indexing through the classpath/JAR directories, or
>>>>>>> whether it is speed of loading bytes once we know what we need. I think
>>>>>>> that it is premature to abandon the JAR manifest data as the principal
>>>>>>> source of metadata until we understand the problem this solves.
>>>>>>>
>>>>>>> Can we measure where the time is spent in the current implementation?
>>>>>>> I think it will help guide this approach to a better solution.
>>>>>>> What tools do you recommend for profiling start-up?
>>>>>>>
>>>>>>> Regards
>>>>>>> Tim
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> С уважением,
>>>>>> Алексей Федотов,
>>>>>> ЗАО «Телеком Экспресс»
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> С уважением,
>>>> Алексей Федотов,
>>>> ЗАО «Телеком Экспресс»
>>>>
>>>
>>
>
Re: [VM] On-demand class library parsing is ready to commit
Posted by Wenlong Li <we...@gmail.com>.
Aleksey,
Thx for testing this patch, and sharing your experimental result.
Yes, I think your result would be reasonable. The performance gain of
this patch varies with different systems.
Again, I would like to say we have different definitions for "startup".
Maybe I should move the change in classlib module to vm module, so
that the dependency can be minimized.
thx again for discussion. :)
wenlong
On Mon, Dec 22, 2008 at 4:04 AM, Aleksey Shipilev
<al...@gmail.com> wrote:
> Hi Wenlong,
>
> I had some performance experiments with your patch. The test system is:
> - Pentium D 820 2.8 Ghz / 2 Gb DDR2-667
> - WD 3200KS, 320 Gb, 16 Mb cache
> - Gentoo Linux x86, 2.6.23
> - Harmony r728459
> - SPECjvm2008
>
> To recreate the stressful conditions over and over the simple script
> was written [1]. The script invalidates the caches before actually
> starting the workload: re-reads the same 64 Mb file a couple of times
> to fill out on-HDD cache, invalidating VFS block caches first to make
> sure the data is really requested from the disk.
>
> On HWA [2] these performance results were produced:
>
> "cold-start" (invalidate caches):
> clean: (5.24 +- 0.28) secs
> ondemand: (4.49 +- 0.17) secs
>
> "warm-start" (don't invalidate caches);
> clean: (2.82 +- 0.01) secs
> ondemand: (2.80 +- 0.02) secs
>
> That is, on-demand patch does bring +17% (-+9%) improvement on HWA
> when running with flushed caches, and does not bring any performance
> improvement in warm mode.
>
> As I mentioned several times, this test does not reflect the real
> performance end user would perceive, so I took two SPECjvm2008:startup
> benchmarks and run each of them 10x10 times.
>
> SPECjvm2008:startup.helloworld, "cold start":
> clean: (8.93 +- 0.21) ops/min
> ondemand: (9.04 +- 0.03) ops/min
>
> SPECjvm2008:startup.compiler.compiler, "cold start":
> clean: (1.44 +- 0.05) ops/min
> ondemand: (1.42 +- 0.04) ops/min
>
> As you can see even in very stressful situation there's no boost. I
> would find these performance results unconvincing to change the
> infrastructure of boolclasspath resolution. Am I missing something
> important?
>
> Thanks,
> Aleksey.
>
> [1] run.sh
> #!/bin/bash
>
> R=`pwd`
>
> JAVA=$R/platforms/builds/harmony-release-clean/jdk/jre/bin/java
> #JAVA=$R/platforms/builds/harmony-release-ondemand/jdk/jre/bin/java
> JAVA_OPTS="-Xmx1024M -Xms1024M"
>
> for T in `seq 1 10`; do
>
> echo "*************** EXECUTING ITERATION $T ****************"
>
> # invalidate HDD caches
> # - need to replace all entries in LRU HDD cache
> # - flush the kernel VFS cache first to ensure the data
> would be read from disk
>
> echo "Flushing caches"
> for I in `seq 1 5`; do
> sync
> echo 3 > /proc/sys/vm/drop_caches
>
> dd if=cachekiller.file of=/dev/null > /dev/null 2>&1
> done
>
> echo "Executing."
>
> # HelloWorld
> /usr/bin/time $JAVA $JAVA_OPTS -cp benchmarks/ HelloWorld 2>&1
>
> # SPECjvm2008
> #cd $R/benchmarks/storage/SPECjvm2008
> #/usr/bin/time $JAVA $JAVA_OPTS -Djava.awt.headless=true -jar
> SPECjvm2008.jar -ikv -i 10 startup.compiler.compiler 2>&1
>
> echo ""
> done
>
> [2] HelloWorld.java
> public class HelloWorld {
> public static void main(String[] args) {
> System.out.println("Hello, world!");
> }
> }
>
>
> On Sun, Dec 21, 2008 at 6:02 AM, Wenlong Li <we...@gmail.com> wrote:
>> On Sat, Dec 20, 2008 at 7:10 PM, Alexei Fedotov
>> <al...@gmail.com> wrote:
>>> Wenlong,
>>> Thanks for removing the commented code.
>>>
>>> There are several VMs which make use of the Harmony class library,
>>> e.g. Harmony VM, J9, Android Dalvik, etc. Your change is Harmony VM
>>> specific, isn't it? If it is, then it's better to keep related changes
>>> in the VM module. If it is not, then it might be a good idea to keep
>>> the changes in the class library module unless other VMs already has
>>> such optimization in their code.
>> [Wenlong] Though at this moment, you can think on-demand class parsing
>> is a specif optimization from your point of view. I believe it could
>> be a general technique, e.g., it can be easily deployed in other
>> runtime systems. Current VM also depends on the luniglobal.c in
>> working_classlib to get all class libraries/modules. e.g., there is a
>> cross-module dependence between classlib and VM. When user wants to
>> add new module, they should manually change the
>> bootclasspath.properties, while if applying this patch, user should
>> revise my added property file instead of the bootclasspath.properties.
>> I understand modifying bootclasspath file may be a specification.
>>>
>>> In any case crossing module boundary would make class library users
>>> think more than once or even write some code. Is it technically
>>> possible to prepare a patch which does not change module boundaries?
>>> What do you think?
>> [Wenlong] Yes, it is possible from technical perspective, but a little
>> complicated. I can think about it. :)
>>
>>>
>>> As for your performance experiments, which particular test are your
>>> measuring? It is bootclasspath-unpretentious "Hello, world", isn't it?
>> [Wenlong] My startup means the work executed before running user's
>> computation. That is, the vm creation time. I manually add
>> instrumentation code for execution time in JNI_CreateJavaVM of
>> JNI.cpp. This startup work is common for any benchmarks. My experiment
>> was conducted on both Windows and Linux system. Please see my previous
>> message about performance gain from this optimization.
>>
>> Thx,
>> Wenlong
>>>
>>> Thanks!
>>>
>>> On Sat, Dec 20, 2008 at 2:19 AM, Wenlong Li <we...@gmail.com> wrote:
>>>> On Sat, Dec 20, 2008 at 12:42 AM, Alexei Fedotov
>>>> <al...@gmail.com> wrote:
>>>>> Wenlong,
>>>>> Have I missed a discussion of the proposed design? I see that you
>>>>> expose a new public interface:
>>>>> /**
>>>>> * @map the jar with exported package in the pending jar list for
>>>>> on-demand jar parsing
>>>>> * Key is the jar, and value is the package exported by this jar
>>>>> */
>>>>> DECLARE_OPEN(void, vm_properties_set_pending_jar, (const char* key,
>>>>> const char* value));
>>>>>
>>>>> Did you mean "Maps" instead of "@map"? Strangely the word "pending"
>>>>> disappeared from the name of the wrapping VMI interface
>>>>> SetJarPackageMapping . Why should we extend both OPEN and VMI
>>>>> interfaces with the same function? Why did you put your code into
>>>>> working_classlib/modules/luni/src/main/native/luni/shared/luniglob.c,
>>>>> thus introducing another dependency between VM and class library?
>>>> [Wenlong] The boot class path is defined in luniglobal.c in Harmony,
>>>> and it also has dependence with VM. In my understanding, my patch is
>>>> related to boot class path determination, so I also put my code in
>>>> luniglobal.c, and use VMI interface to communicate with VM.
>>>>
>>>>>
>>>>> + //rcSetProperty = (*vmInterface)->SetJarPackageMapping
>>>>> (vmInterface, jarName, jarValue);
>>>>> + /*
>>>>> + hymem_free_memory(jarName);
>>>>> + hymem_free_memory(jarValue);
>>>>> + */
>>>>> Should we really commit the commented code?
>>>>> Thanks.
>>>>
>>>> [Wenlong] Please see my latest version of patch in the list. Such
>>>> commented code has been removed.
>>>>>
>>>>>
>>>>> On Fri, Dec 19, 2008 at 6:59 PM, Tim Ellison <t....@gmail.com> wrote:
>>>>>> I was hoping that somebody else would comment first, so I don't have to
>>>>>> be the grumpy one all the time :-)
>>>>>>
>>>>>> As I said before, this is good prototyping work...
>>>>>>
>>>>>> Wenlong Li wrote:
>>>>>>> I did the pre-commit test on the patch of on-demand class library
>>>>>>> parsing (https://issues.apache.org/jira/browse/HARMONY-6039), and it
>>>>>>> works well now.
>>>>>>> Can Harmony incorporate this feature?
>>>>>>
>>>>>> I'm not sure it is ready for committing to the head stream yet.
>>>>>>
>>>>>>> Via on-demand class parsing, we can reduce startup time from 20+
>>>>>>> seconds to 3 seconds for cold runing, and 170 ms to 140 ms for warm-up
>>>>>>> running on Core 2 Duo with Windows.
>>>>>>
>>>>>> Can you tell me how to reproduce 20+sec cold start-up? I haven't seen
>>>>>> anything like that in my simple tests.
>>>>>>
>>>>>>> After applying the patch, please note there is some change to add new modules.
>>>>>>> (1) If you want to add new modules/libraries, please don't put them in
>>>>>>> the bootclasspath.properties file. This file now only saves modules
>>>>>>> needed during startup (the VM startup only accesses class libraries in
>>>>>>> eight modules)
>>>>>>
>>>>>> That would break too much. How about creating a new file rather than
>>>>>> re-purposing an existing file with different semantics? This file is
>>>>>> used by Jikes, IBM VME, the Eclipse plug-in, at least.
>>>>>>
>>>>>>> (2) For new modules/libraries, please put them in the
>>>>>>> modulelibrarymapping.properties file. You should specify the module
>>>>>>> name and its exported class library. Here is one example:
>>>>>>> math.jar=java.math, where "math.jar" means the module name, and
>>>>>>> "java.math" means the class libraries this module exports.
>>>>>>
>>>>>> As we discussed on another thread, its unclear if the time is spent in
>>>>>> following the slow indexing through the classpath/JAR directories, or
>>>>>> whether it is speed of loading bytes once we know what we need. I think
>>>>>> that it is premature to abandon the JAR manifest data as the principal
>>>>>> source of metadata until we understand the problem this solves.
>>>>>>
>>>>>> Can we measure where the time is spent in the current implementation?
>>>>>> I think it will help guide this approach to a better solution.
>>>>>> What tools do you recommend for profiling start-up?
>>>>>>
>>>>>> Regards
>>>>>> Tim
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> С уважением,
>>>>> Алексей Федотов,
>>>>> ЗАО «Телеком Экспресс»
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> С уважением,
>>> Алексей Федотов,
>>> ЗАО «Телеком Экспресс»
>>>
>>
>
Re: [VM] On-demand class library parsing is ready to commit
Posted by Aleksey Shipilev <al...@gmail.com>.
Hi Wenlong,
I had some performance experiments with your patch. The test system is:
- Pentium D 820 2.8 Ghz / 2 Gb DDR2-667
- WD 3200KS, 320 Gb, 16 Mb cache
- Gentoo Linux x86, 2.6.23
- Harmony r728459
- SPECjvm2008
To recreate the stressful conditions over and over the simple script
was written [1]. The script invalidates the caches before actually
starting the workload: re-reads the same 64 Mb file a couple of times
to fill out on-HDD cache, invalidating VFS block caches first to make
sure the data is really requested from the disk.
On HWA [2] these performance results were produced:
"cold-start" (invalidate caches):
clean: (5.24 +- 0.28) secs
ondemand: (4.49 +- 0.17) secs
"warm-start" (don't invalidate caches);
clean: (2.82 +- 0.01) secs
ondemand: (2.80 +- 0.02) secs
That is, on-demand patch does bring +17% (-+9%) improvement on HWA
when running with flushed caches, and does not bring any performance
improvement in warm mode.
As I mentioned several times, this test does not reflect the real
performance end user would perceive, so I took two SPECjvm2008:startup
benchmarks and run each of them 10x10 times.
SPECjvm2008:startup.helloworld, "cold start":
clean: (8.93 +- 0.21) ops/min
ondemand: (9.04 +- 0.03) ops/min
SPECjvm2008:startup.compiler.compiler, "cold start":
clean: (1.44 +- 0.05) ops/min
ondemand: (1.42 +- 0.04) ops/min
As you can see even in very stressful situation there's no boost. I
would find these performance results unconvincing to change the
infrastructure of boolclasspath resolution. Am I missing something
important?
Thanks,
Aleksey.
[1] run.sh
#!/bin/bash
R=`pwd`
JAVA=$R/platforms/builds/harmony-release-clean/jdk/jre/bin/java
#JAVA=$R/platforms/builds/harmony-release-ondemand/jdk/jre/bin/java
JAVA_OPTS="-Xmx1024M -Xms1024M"
for T in `seq 1 10`; do
echo "*************** EXECUTING ITERATION $T ****************"
# invalidate HDD caches
# - need to replace all entries in LRU HDD cache
# - flush the kernel VFS cache first to ensure the data
would be read from disk
echo "Flushing caches"
for I in `seq 1 5`; do
sync
echo 3 > /proc/sys/vm/drop_caches
dd if=cachekiller.file of=/dev/null > /dev/null 2>&1
done
echo "Executing."
# HelloWorld
/usr/bin/time $JAVA $JAVA_OPTS -cp benchmarks/ HelloWorld 2>&1
# SPECjvm2008
#cd $R/benchmarks/storage/SPECjvm2008
#/usr/bin/time $JAVA $JAVA_OPTS -Djava.awt.headless=true -jar
SPECjvm2008.jar -ikv -i 10 startup.compiler.compiler 2>&1
echo ""
done
[2] HelloWorld.java
public class HelloWorld {
public static void main(String[] args) {
System.out.println("Hello, world!");
}
}
On Sun, Dec 21, 2008 at 6:02 AM, Wenlong Li <we...@gmail.com> wrote:
> On Sat, Dec 20, 2008 at 7:10 PM, Alexei Fedotov
> <al...@gmail.com> wrote:
>> Wenlong,
>> Thanks for removing the commented code.
>>
>> There are several VMs which make use of the Harmony class library,
>> e.g. Harmony VM, J9, Android Dalvik, etc. Your change is Harmony VM
>> specific, isn't it? If it is, then it's better to keep related changes
>> in the VM module. If it is not, then it might be a good idea to keep
>> the changes in the class library module unless other VMs already has
>> such optimization in their code.
> [Wenlong] Though at this moment, you can think on-demand class parsing
> is a specif optimization from your point of view. I believe it could
> be a general technique, e.g., it can be easily deployed in other
> runtime systems. Current VM also depends on the luniglobal.c in
> working_classlib to get all class libraries/modules. e.g., there is a
> cross-module dependence between classlib and VM. When user wants to
> add new module, they should manually change the
> bootclasspath.properties, while if applying this patch, user should
> revise my added property file instead of the bootclasspath.properties.
> I understand modifying bootclasspath file may be a specification.
>>
>> In any case crossing module boundary would make class library users
>> think more than once or even write some code. Is it technically
>> possible to prepare a patch which does not change module boundaries?
>> What do you think?
> [Wenlong] Yes, it is possible from technical perspective, but a little
> complicated. I can think about it. :)
>
>>
>> As for your performance experiments, which particular test are your
>> measuring? It is bootclasspath-unpretentious "Hello, world", isn't it?
> [Wenlong] My startup means the work executed before running user's
> computation. That is, the vm creation time. I manually add
> instrumentation code for execution time in JNI_CreateJavaVM of
> JNI.cpp. This startup work is common for any benchmarks. My experiment
> was conducted on both Windows and Linux system. Please see my previous
> message about performance gain from this optimization.
>
> Thx,
> Wenlong
>>
>> Thanks!
>>
>> On Sat, Dec 20, 2008 at 2:19 AM, Wenlong Li <we...@gmail.com> wrote:
>>> On Sat, Dec 20, 2008 at 12:42 AM, Alexei Fedotov
>>> <al...@gmail.com> wrote:
>>>> Wenlong,
>>>> Have I missed a discussion of the proposed design? I see that you
>>>> expose a new public interface:
>>>> /**
>>>> * @map the jar with exported package in the pending jar list for
>>>> on-demand jar parsing
>>>> * Key is the jar, and value is the package exported by this jar
>>>> */
>>>> DECLARE_OPEN(void, vm_properties_set_pending_jar, (const char* key,
>>>> const char* value));
>>>>
>>>> Did you mean "Maps" instead of "@map"? Strangely the word "pending"
>>>> disappeared from the name of the wrapping VMI interface
>>>> SetJarPackageMapping . Why should we extend both OPEN and VMI
>>>> interfaces with the same function? Why did you put your code into
>>>> working_classlib/modules/luni/src/main/native/luni/shared/luniglob.c,
>>>> thus introducing another dependency between VM and class library?
>>> [Wenlong] The boot class path is defined in luniglobal.c in Harmony,
>>> and it also has dependence with VM. In my understanding, my patch is
>>> related to boot class path determination, so I also put my code in
>>> luniglobal.c, and use VMI interface to communicate with VM.
>>>
>>>>
>>>> + //rcSetProperty = (*vmInterface)->SetJarPackageMapping
>>>> (vmInterface, jarName, jarValue);
>>>> + /*
>>>> + hymem_free_memory(jarName);
>>>> + hymem_free_memory(jarValue);
>>>> + */
>>>> Should we really commit the commented code?
>>>> Thanks.
>>>
>>> [Wenlong] Please see my latest version of patch in the list. Such
>>> commented code has been removed.
>>>>
>>>>
>>>> On Fri, Dec 19, 2008 at 6:59 PM, Tim Ellison <t....@gmail.com> wrote:
>>>>> I was hoping that somebody else would comment first, so I don't have to
>>>>> be the grumpy one all the time :-)
>>>>>
>>>>> As I said before, this is good prototyping work...
>>>>>
>>>>> Wenlong Li wrote:
>>>>>> I did the pre-commit test on the patch of on-demand class library
>>>>>> parsing (https://issues.apache.org/jira/browse/HARMONY-6039), and it
>>>>>> works well now.
>>>>>> Can Harmony incorporate this feature?
>>>>>
>>>>> I'm not sure it is ready for committing to the head stream yet.
>>>>>
>>>>>> Via on-demand class parsing, we can reduce startup time from 20+
>>>>>> seconds to 3 seconds for cold runing, and 170 ms to 140 ms for warm-up
>>>>>> running on Core 2 Duo with Windows.
>>>>>
>>>>> Can you tell me how to reproduce 20+sec cold start-up? I haven't seen
>>>>> anything like that in my simple tests.
>>>>>
>>>>>> After applying the patch, please note there is some change to add new modules.
>>>>>> (1) If you want to add new modules/libraries, please don't put them in
>>>>>> the bootclasspath.properties file. This file now only saves modules
>>>>>> needed during startup (the VM startup only accesses class libraries in
>>>>>> eight modules)
>>>>>
>>>>> That would break too much. How about creating a new file rather than
>>>>> re-purposing an existing file with different semantics? This file is
>>>>> used by Jikes, IBM VME, the Eclipse plug-in, at least.
>>>>>
>>>>>> (2) For new modules/libraries, please put them in the
>>>>>> modulelibrarymapping.properties file. You should specify the module
>>>>>> name and its exported class library. Here is one example:
>>>>>> math.jar=java.math, where "math.jar" means the module name, and
>>>>>> "java.math" means the class libraries this module exports.
>>>>>
>>>>> As we discussed on another thread, its unclear if the time is spent in
>>>>> following the slow indexing through the classpath/JAR directories, or
>>>>> whether it is speed of loading bytes once we know what we need. I think
>>>>> that it is premature to abandon the JAR manifest data as the principal
>>>>> source of metadata until we understand the problem this solves.
>>>>>
>>>>> Can we measure where the time is spent in the current implementation?
>>>>> I think it will help guide this approach to a better solution.
>>>>> What tools do you recommend for profiling start-up?
>>>>>
>>>>> Regards
>>>>> Tim
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> С уважением,
>>>> Алексей Федотов,
>>>> ЗАО «Телеком Экспресс»
>>>>
>>>
>>
>>
>>
>> --
>> С уважением,
>> Алексей Федотов,
>> ЗАО «Телеком Экспресс»
>>
>
Re: [VM] On-demand class library parsing is ready to commit
Posted by Wenlong Li <we...@gmail.com>.
On Sat, Dec 20, 2008 at 7:10 PM, Alexei Fedotov
<al...@gmail.com> wrote:
> Wenlong,
> Thanks for removing the commented code.
>
> There are several VMs which make use of the Harmony class library,
> e.g. Harmony VM, J9, Android Dalvik, etc. Your change is Harmony VM
> specific, isn't it? If it is, then it's better to keep related changes
> in the VM module. If it is not, then it might be a good idea to keep
> the changes in the class library module unless other VMs already has
> such optimization in their code.
[Wenlong] Though at this moment, you can think on-demand class parsing
is a specif optimization from your point of view. I believe it could
be a general technique, e.g., it can be easily deployed in other
runtime systems. Current VM also depends on the luniglobal.c in
working_classlib to get all class libraries/modules. e.g., there is a
cross-module dependence between classlib and VM. When user wants to
add new module, they should manually change the
bootclasspath.properties, while if applying this patch, user should
revise my added property file instead of the bootclasspath.properties.
I understand modifying bootclasspath file may be a specification.
>
> In any case crossing module boundary would make class library users
> think more than once or even write some code. Is it technically
> possible to prepare a patch which does not change module boundaries?
> What do you think?
[Wenlong] Yes, it is possible from technical perspective, but a little
complicated. I can think about it. :)
>
> As for your performance experiments, which particular test are your
> measuring? It is bootclasspath-unpretentious "Hello, world", isn't it?
[Wenlong] My startup means the work executed before running user's
computation. That is, the vm creation time. I manually add
instrumentation code for execution time in JNI_CreateJavaVM of
JNI.cpp. This startup work is common for any benchmarks. My experiment
was conducted on both Windows and Linux system. Please see my previous
message about performance gain from this optimization.
Thx,
Wenlong
>
> Thanks!
>
> On Sat, Dec 20, 2008 at 2:19 AM, Wenlong Li <we...@gmail.com> wrote:
>> On Sat, Dec 20, 2008 at 12:42 AM, Alexei Fedotov
>> <al...@gmail.com> wrote:
>>> Wenlong,
>>> Have I missed a discussion of the proposed design? I see that you
>>> expose a new public interface:
>>> /**
>>> * @map the jar with exported package in the pending jar list for
>>> on-demand jar parsing
>>> * Key is the jar, and value is the package exported by this jar
>>> */
>>> DECLARE_OPEN(void, vm_properties_set_pending_jar, (const char* key,
>>> const char* value));
>>>
>>> Did you mean "Maps" instead of "@map"? Strangely the word "pending"
>>> disappeared from the name of the wrapping VMI interface
>>> SetJarPackageMapping . Why should we extend both OPEN and VMI
>>> interfaces with the same function? Why did you put your code into
>>> working_classlib/modules/luni/src/main/native/luni/shared/luniglob.c,
>>> thus introducing another dependency between VM and class library?
>> [Wenlong] The boot class path is defined in luniglobal.c in Harmony,
>> and it also has dependence with VM. In my understanding, my patch is
>> related to boot class path determination, so I also put my code in
>> luniglobal.c, and use VMI interface to communicate with VM.
>>
>>>
>>> + //rcSetProperty = (*vmInterface)->SetJarPackageMapping
>>> (vmInterface, jarName, jarValue);
>>> + /*
>>> + hymem_free_memory(jarName);
>>> + hymem_free_memory(jarValue);
>>> + */
>>> Should we really commit the commented code?
>>> Thanks.
>>
>> [Wenlong] Please see my latest version of patch in the list. Such
>> commented code has been removed.
>>>
>>>
>>> On Fri, Dec 19, 2008 at 6:59 PM, Tim Ellison <t....@gmail.com> wrote:
>>>> I was hoping that somebody else would comment first, so I don't have to
>>>> be the grumpy one all the time :-)
>>>>
>>>> As I said before, this is good prototyping work...
>>>>
>>>> Wenlong Li wrote:
>>>>> I did the pre-commit test on the patch of on-demand class library
>>>>> parsing (https://issues.apache.org/jira/browse/HARMONY-6039), and it
>>>>> works well now.
>>>>> Can Harmony incorporate this feature?
>>>>
>>>> I'm not sure it is ready for committing to the head stream yet.
>>>>
>>>>> Via on-demand class parsing, we can reduce startup time from 20+
>>>>> seconds to 3 seconds for cold runing, and 170 ms to 140 ms for warm-up
>>>>> running on Core 2 Duo with Windows.
>>>>
>>>> Can you tell me how to reproduce 20+sec cold start-up? I haven't seen
>>>> anything like that in my simple tests.
>>>>
>>>>> After applying the patch, please note there is some change to add new modules.
>>>>> (1) If you want to add new modules/libraries, please don't put them in
>>>>> the bootclasspath.properties file. This file now only saves modules
>>>>> needed during startup (the VM startup only accesses class libraries in
>>>>> eight modules)
>>>>
>>>> That would break too much. How about creating a new file rather than
>>>> re-purposing an existing file with different semantics? This file is
>>>> used by Jikes, IBM VME, the Eclipse plug-in, at least.
>>>>
>>>>> (2) For new modules/libraries, please put them in the
>>>>> modulelibrarymapping.properties file. You should specify the module
>>>>> name and its exported class library. Here is one example:
>>>>> math.jar=java.math, where "math.jar" means the module name, and
>>>>> "java.math" means the class libraries this module exports.
>>>>
>>>> As we discussed on another thread, its unclear if the time is spent in
>>>> following the slow indexing through the classpath/JAR directories, or
>>>> whether it is speed of loading bytes once we know what we need. I think
>>>> that it is premature to abandon the JAR manifest data as the principal
>>>> source of metadata until we understand the problem this solves.
>>>>
>>>> Can we measure where the time is spent in the current implementation?
>>>> I think it will help guide this approach to a better solution.
>>>> What tools do you recommend for profiling start-up?
>>>>
>>>> Regards
>>>> Tim
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> С уважением,
>>> Алексей Федотов,
>>> ЗАО «Телеком Экспресс»
>>>
>>
>
>
>
> --
> С уважением,
> Алексей Федотов,
> ЗАО «Телеком Экспресс»
>
Re: [VM] On-demand class library parsing is ready to commit
Posted by Alexei Fedotov <al...@gmail.com>.
Wenlong,
Thanks for removing the commented code.
There are several VMs which make use of the Harmony class library,
e.g. Harmony VM, J9, Android Dalvik, etc. Your change is Harmony VM
specific, isn't it? If it is, then it's better to keep related changes
in the VM module. If it is not, then it might be a good idea to keep
the changes in the class library module unless other VMs already has
such optimization in their code.
In any case crossing module boundary would make class library users
think more than once or even write some code. Is it technically
possible to prepare a patch which does not change module boundaries?
What do you think?
As for your performance experiments, which particular test are your
measuring? It is bootclasspath-unpretentious "Hello, world", isn't it?
Thanks!
On Sat, Dec 20, 2008 at 2:19 AM, Wenlong Li <we...@gmail.com> wrote:
> On Sat, Dec 20, 2008 at 12:42 AM, Alexei Fedotov
> <al...@gmail.com> wrote:
>> Wenlong,
>> Have I missed a discussion of the proposed design? I see that you
>> expose a new public interface:
>> /**
>> * @map the jar with exported package in the pending jar list for
>> on-demand jar parsing
>> * Key is the jar, and value is the package exported by this jar
>> */
>> DECLARE_OPEN(void, vm_properties_set_pending_jar, (const char* key,
>> const char* value));
>>
>> Did you mean "Maps" instead of "@map"? Strangely the word "pending"
>> disappeared from the name of the wrapping VMI interface
>> SetJarPackageMapping . Why should we extend both OPEN and VMI
>> interfaces with the same function? Why did you put your code into
>> working_classlib/modules/luni/src/main/native/luni/shared/luniglob.c,
>> thus introducing another dependency between VM and class library?
> [Wenlong] The boot class path is defined in luniglobal.c in Harmony,
> and it also has dependence with VM. In my understanding, my patch is
> related to boot class path determination, so I also put my code in
> luniglobal.c, and use VMI interface to communicate with VM.
>
>>
>> + //rcSetProperty = (*vmInterface)->SetJarPackageMapping
>> (vmInterface, jarName, jarValue);
>> + /*
>> + hymem_free_memory(jarName);
>> + hymem_free_memory(jarValue);
>> + */
>> Should we really commit the commented code?
>> Thanks.
>
> [Wenlong] Please see my latest version of patch in the list. Such
> commented code has been removed.
>>
>>
>> On Fri, Dec 19, 2008 at 6:59 PM, Tim Ellison <t....@gmail.com> wrote:
>>> I was hoping that somebody else would comment first, so I don't have to
>>> be the grumpy one all the time :-)
>>>
>>> As I said before, this is good prototyping work...
>>>
>>> Wenlong Li wrote:
>>>> I did the pre-commit test on the patch of on-demand class library
>>>> parsing (https://issues.apache.org/jira/browse/HARMONY-6039), and it
>>>> works well now.
>>>> Can Harmony incorporate this feature?
>>>
>>> I'm not sure it is ready for committing to the head stream yet.
>>>
>>>> Via on-demand class parsing, we can reduce startup time from 20+
>>>> seconds to 3 seconds for cold runing, and 170 ms to 140 ms for warm-up
>>>> running on Core 2 Duo with Windows.
>>>
>>> Can you tell me how to reproduce 20+sec cold start-up? I haven't seen
>>> anything like that in my simple tests.
>>>
>>>> After applying the patch, please note there is some change to add new modules.
>>>> (1) If you want to add new modules/libraries, please don't put them in
>>>> the bootclasspath.properties file. This file now only saves modules
>>>> needed during startup (the VM startup only accesses class libraries in
>>>> eight modules)
>>>
>>> That would break too much. How about creating a new file rather than
>>> re-purposing an existing file with different semantics? This file is
>>> used by Jikes, IBM VME, the Eclipse plug-in, at least.
>>>
>>>> (2) For new modules/libraries, please put them in the
>>>> modulelibrarymapping.properties file. You should specify the module
>>>> name and its exported class library. Here is one example:
>>>> math.jar=java.math, where "math.jar" means the module name, and
>>>> "java.math" means the class libraries this module exports.
>>>
>>> As we discussed on another thread, its unclear if the time is spent in
>>> following the slow indexing through the classpath/JAR directories, or
>>> whether it is speed of loading bytes once we know what we need. I think
>>> that it is premature to abandon the JAR manifest data as the principal
>>> source of metadata until we understand the problem this solves.
>>>
>>> Can we measure where the time is spent in the current implementation?
>>> I think it will help guide this approach to a better solution.
>>> What tools do you recommend for profiling start-up?
>>>
>>> Regards
>>> Tim
>>>
>>>
>>
>>
>>
>> --
>> С уважением,
>> Алексей Федотов,
>> ЗАО «Телеком Экспресс»
>>
>
--
С уважением,
Алексей Федотов,
ЗАО «Телеком Экспресс»
Re: [VM] On-demand class library parsing is ready to commit
Posted by Wenlong Li <we...@gmail.com>.
On Sat, Dec 20, 2008 at 12:42 AM, Alexei Fedotov
<al...@gmail.com> wrote:
> Wenlong,
> Have I missed a discussion of the proposed design? I see that you
> expose a new public interface:
> /**
> * @map the jar with exported package in the pending jar list for
> on-demand jar parsing
> * Key is the jar, and value is the package exported by this jar
> */
> DECLARE_OPEN(void, vm_properties_set_pending_jar, (const char* key,
> const char* value));
>
> Did you mean "Maps" instead of "@map"? Strangely the word "pending"
> disappeared from the name of the wrapping VMI interface
> SetJarPackageMapping . Why should we extend both OPEN and VMI
> interfaces with the same function? Why did you put your code into
> working_classlib/modules/luni/src/main/native/luni/shared/luniglob.c,
> thus introducing another dependency between VM and class library?
[Wenlong] The boot class path is defined in luniglobal.c in Harmony,
and it also has dependence with VM. In my understanding, my patch is
related to boot class path determination, so I also put my code in
luniglobal.c, and use VMI interface to communicate with VM.
>
> + //rcSetProperty = (*vmInterface)->SetJarPackageMapping
> (vmInterface, jarName, jarValue);
> + /*
> + hymem_free_memory(jarName);
> + hymem_free_memory(jarValue);
> + */
> Should we really commit the commented code?
> Thanks.
[Wenlong] Please see my latest version of patch in the list. Such
commented code has been removed.
>
>
> On Fri, Dec 19, 2008 at 6:59 PM, Tim Ellison <t....@gmail.com> wrote:
>> I was hoping that somebody else would comment first, so I don't have to
>> be the grumpy one all the time :-)
>>
>> As I said before, this is good prototyping work...
>>
>> Wenlong Li wrote:
>>> I did the pre-commit test on the patch of on-demand class library
>>> parsing (https://issues.apache.org/jira/browse/HARMONY-6039), and it
>>> works well now.
>>> Can Harmony incorporate this feature?
>>
>> I'm not sure it is ready for committing to the head stream yet.
>>
>>> Via on-demand class parsing, we can reduce startup time from 20+
>>> seconds to 3 seconds for cold runing, and 170 ms to 140 ms for warm-up
>>> running on Core 2 Duo with Windows.
>>
>> Can you tell me how to reproduce 20+sec cold start-up? I haven't seen
>> anything like that in my simple tests.
>>
>>> After applying the patch, please note there is some change to add new modules.
>>> (1) If you want to add new modules/libraries, please don't put them in
>>> the bootclasspath.properties file. This file now only saves modules
>>> needed during startup (the VM startup only accesses class libraries in
>>> eight modules)
>>
>> That would break too much. How about creating a new file rather than
>> re-purposing an existing file with different semantics? This file is
>> used by Jikes, IBM VME, the Eclipse plug-in, at least.
>>
>>> (2) For new modules/libraries, please put them in the
>>> modulelibrarymapping.properties file. You should specify the module
>>> name and its exported class library. Here is one example:
>>> math.jar=java.math, where "math.jar" means the module name, and
>>> "java.math" means the class libraries this module exports.
>>
>> As we discussed on another thread, its unclear if the time is spent in
>> following the slow indexing through the classpath/JAR directories, or
>> whether it is speed of loading bytes once we know what we need. I think
>> that it is premature to abandon the JAR manifest data as the principal
>> source of metadata until we understand the problem this solves.
>>
>> Can we measure where the time is spent in the current implementation?
>> I think it will help guide this approach to a better solution.
>> What tools do you recommend for profiling start-up?
>>
>> Regards
>> Tim
>>
>>
>
>
>
> --
> С уважением,
> Алексей Федотов,
> ЗАО «Телеком Экспресс»
>
Re: [VM] On-demand class library parsing is ready to commit
Posted by Alexei Fedotov <al...@gmail.com>.
Wenlong,
Have I missed a discussion of the proposed design? I see that you
expose a new public interface:
/**
* @map the jar with exported package in the pending jar list for
on-demand jar parsing
* Key is the jar, and value is the package exported by this jar
*/
DECLARE_OPEN(void, vm_properties_set_pending_jar, (const char* key,
const char* value));
Did you mean "Maps" instead of "@map"? Strangely the word "pending"
disappeared from the name of the wrapping VMI interface
SetJarPackageMapping . Why should we extend both OPEN and VMI
interfaces with the same function? Why did you put your code into
working_classlib/modules/luni/src/main/native/luni/shared/luniglob.c,
thus introducing another dependency between VM and class library?
+ //rcSetProperty = (*vmInterface)->SetJarPackageMapping
(vmInterface, jarName, jarValue);
+ /*
+ hymem_free_memory(jarName);
+ hymem_free_memory(jarValue);
+ */
Should we really commit the commented code?
Thanks.
On Fri, Dec 19, 2008 at 6:59 PM, Tim Ellison <t....@gmail.com> wrote:
> I was hoping that somebody else would comment first, so I don't have to
> be the grumpy one all the time :-)
>
> As I said before, this is good prototyping work...
>
> Wenlong Li wrote:
>> I did the pre-commit test on the patch of on-demand class library
>> parsing (https://issues.apache.org/jira/browse/HARMONY-6039), and it
>> works well now.
>> Can Harmony incorporate this feature?
>
> I'm not sure it is ready for committing to the head stream yet.
>
>> Via on-demand class parsing, we can reduce startup time from 20+
>> seconds to 3 seconds for cold runing, and 170 ms to 140 ms for warm-up
>> running on Core 2 Duo with Windows.
>
> Can you tell me how to reproduce 20+sec cold start-up? I haven't seen
> anything like that in my simple tests.
>
>> After applying the patch, please note there is some change to add new modules.
>> (1) If you want to add new modules/libraries, please don't put them in
>> the bootclasspath.properties file. This file now only saves modules
>> needed during startup (the VM startup only accesses class libraries in
>> eight modules)
>
> That would break too much. How about creating a new file rather than
> re-purposing an existing file with different semantics? This file is
> used by Jikes, IBM VME, the Eclipse plug-in, at least.
>
>> (2) For new modules/libraries, please put them in the
>> modulelibrarymapping.properties file. You should specify the module
>> name and its exported class library. Here is one example:
>> math.jar=java.math, where "math.jar" means the module name, and
>> "java.math" means the class libraries this module exports.
>
> As we discussed on another thread, its unclear if the time is spent in
> following the slow indexing through the classpath/JAR directories, or
> whether it is speed of loading bytes once we know what we need. I think
> that it is premature to abandon the JAR manifest data as the principal
> source of metadata until we understand the problem this solves.
>
> Can we measure where the time is spent in the current implementation?
> I think it will help guide this approach to a better solution.
> What tools do you recommend for profiling start-up?
>
> Regards
> Tim
>
>
--
С уважением,
Алексей Федотов,
ЗАО «Телеком Экспресс»
Re: [VM] On-demand class library parsing is ready to commit
Posted by Wenlong Li <we...@gmail.com>.
Tim,
Pls see my embedded comments. :)
On Fri, Dec 19, 2008 at 11:59 PM, Tim Ellison <t....@gmail.com> wrote:
> I was hoping that somebody else would comment first, so I don't have to
> be the grumpy one all the time :-)
>
> As I said before, this is good prototyping work...
>
> Wenlong Li wrote:
>> I did the pre-commit test on the patch of on-demand class library
>> parsing (https://issues.apache.org/jira/browse/HARMONY-6039), and it
>> works well now.
>> Can Harmony incorporate this feature?
>
> I'm not sure it is ready for committing to the head stream yet.
>
>> Via on-demand class parsing, we can reduce startup time from 20+
>> seconds to 3 seconds for cold runing, and 170 ms to 140 ms for warm-up
>> running on Core 2 Duo with Windows.
>
> Can you tell me how to reproduce 20+sec cold start-up? I haven't seen
> anything like that in my simple tests.
[Wenlong] My test was on a Dell machine with Core 2 Quad-core
processor, windows xp system. I forgot to mention on another machine
with Linux system, the cold running time can be reduced from 900 ms to
400 ms. The main difference is the processor frequency and Hard Disk
speedup, where processor frequency contributes very small performance
impact. That means, most of performance improvement is from reducing
hard disk operation, and the performance gain varies with different
Hard disks.
>
>> After applying the patch, please note there is some change to add new modules.
>> (1) If you want to add new modules/libraries, please don't put them in
>> the bootclasspath.properties file. This file now only saves modules
>> needed during startup (the VM startup only accesses class libraries in
>> eight modules)
>
> That would break too much. How about creating a new file rather than
> re-purposing an existing file with different semantics? This file is
> used by Jikes, IBM VME, the Eclipse plug-in, at least.
[Wenlong] Yes, it is possible.
>
>> (2) For new modules/libraries, please put them in the
>> modulelibrarymapping.properties file. You should specify the module
>> name and its exported class library. Here is one example:
>> math.jar=java.math, where "math.jar" means the module name, and
>> "java.math" means the class libraries this module exports.
>
> As we discussed on another thread, its unclear if the time is spent in
> following the slow indexing through the classpath/JAR directories, or
> whether it is speed of loading bytes once we know what we need. I think
> that it is premature to abandon the JAR manifest data as the principal
> source of metadata until we understand the problem this solves.
[Wenlong] The time is mostly in loading data from disk. The index is
predefined when reading boot class path in luniglobal.c
>
> Can we measure where the time is spent in the current implementation?
> I think it will help guide this approach to a better solution.
> What tools do you recommend for profiling start-up?
[Wenlong] I just manually add instrumentation code to collect
execution time data. I don't believe existing tool can help for this
point
>
> Regards
> Tim
>
>
Re: [VM] On-demand class library parsing is ready to commit
Posted by Tim Ellison <t....@gmail.com>.
I was hoping that somebody else would comment first, so I don't have to
be the grumpy one all the time :-)
As I said before, this is good prototyping work...
Wenlong Li wrote:
> I did the pre-commit test on the patch of on-demand class library
> parsing (https://issues.apache.org/jira/browse/HARMONY-6039), and it
> works well now.
> Can Harmony incorporate this feature?
I'm not sure it is ready for committing to the head stream yet.
> Via on-demand class parsing, we can reduce startup time from 20+
> seconds to 3 seconds for cold runing, and 170 ms to 140 ms for warm-up
> running on Core 2 Duo with Windows.
Can you tell me how to reproduce 20+sec cold start-up? I haven't seen
anything like that in my simple tests.
> After applying the patch, please note there is some change to add new modules.
> (1) If you want to add new modules/libraries, please don't put them in
> the bootclasspath.properties file. This file now only saves modules
> needed during startup (the VM startup only accesses class libraries in
> eight modules)
That would break too much. How about creating a new file rather than
re-purposing an existing file with different semantics? This file is
used by Jikes, IBM VME, the Eclipse plug-in, at least.
> (2) For new modules/libraries, please put them in the
> modulelibrarymapping.properties file. You should specify the module
> name and its exported class library. Here is one example:
> math.jar=java.math, where "math.jar" means the module name, and
> "java.math" means the class libraries this module exports.
As we discussed on another thread, its unclear if the time is spent in
following the slow indexing through the classpath/JAR directories, or
whether it is speed of loading bytes once we know what we need. I think
that it is premature to abandon the JAR manifest data as the principal
source of metadata until we understand the problem this solves.
Can we measure where the time is spent in the current implementation?
I think it will help guide this approach to a better solution.
What tools do you recommend for profiling start-up?
Regards
Tim