[classlib][pack200] Status update

[Alex Blewitt <al...@gmail.com>]

I'm still lurking around in the background, but I'm busier than ever
what with having to keep posting at EclipseZone where I'm editor now
... but I'm still managing to plod along with the Pack200 code. The
last patch (1994, IIRC) went down pretty badly; the subclipse plugin
didn't seem to capture all the newly created classes that I had put
together for a bit of refactoring that I did. It's also a nightmare
submitting newly created files; once they've been uploaded to JIRA
into the patch queue, I effectively have to stop work until they're
applied, because there's no way of getting SVN to figure out that the
file called ClassFile.java is the same as the ClassFile.java that's
been added by someone else on my behalf, and I have to overwrite my
local copy to update. Kinda prevents doing any sensible work on newly
created files between submission and application to be honest.

Instead, I plan to sprint (well, jog, at least) to the next stable
point, and then upload a whole wodge of code and leave it for a week
or two to be applied before picking up again. I'm fairly close to
being at that stage, but not quite. Where I am at is being able to
generate (abstract) methods with exceptions and fields with constant
(integer) values, so a bit further forward than last time. On the one
hand, it means that I understand what needs to be done (no mean feat!)
but on the other hand, there's still a fair bit to go. For a start,
I'm going to need to process the actual bytecode for non-abstract
methods (or indeed, any classes) before there's any point in it trying
to be used for real, and there's still a few missing bits (Longs,
Doubles, Floats) from the constant pools. I'm pretty sure Strings
would work, but I've not tried them yet.

Unfortunately, the code is really awful. The Segment.java is getting
on for 1500 lines of code in a single class; and the attribute parsing
contains both duplicated code and isn't as generic as it needs to be
to handle other attributes; and the (in)visible annotations like
@Overrides and similar are going to be a bit of a nightmare ...

My plan is to get the code to a stage where it can start to be useful
for extracting simple classes from a pack file, and then start
generating test data which can be used for regressions. (There's a bit
there at the moment, but it's not exactly a large coverage.) Once I've
done that, I'll start tackling some of the refactoring which will
result in less duplicated code and hopefully something that will be
easier to look after in the future.

Anyway, excuse the long rambling but I've been a bit quiet for a while
and wanted to let people know I was still alive and kicking the code
:-)

Alex.

Re: Re: Re: Re: [classlib][pack200] Status update

[Alex Blewitt <al...@gmail.com>]

On 14/11/06, Alexei Zakharov <al...@gmail.com> wrote:
> Aha, I see. But in JDBC you can at least call
> Class.forName(MyJDBCDriverClass, true, *MyClassLoader*).

Well, you'll also be able to instantiate the Pack200 Harmony code
using direct instantiation like that too. In fact, it will be
necessary to do this if you want to use it to run against a
(non-Harmony) JRE prior to 1.5, because there weren't the factory
classes in those VMs :-)

I'd imagine that most people who end up using the code will be
directly instantiating it to avoid a dependency on Java 1.5, or of
course using it inside Harmony :-)

Alex.

Re: Re: Re: [classlib][pack200] Status update

[Alexei Zakharov <al...@gmail.com>]

> Also note
> that the list of packages that are listed at the URL you gave specify
> the full set of packages that can be overridden, and the pack200
> classes aren't on there :-)

Don't look at this list. Endorsed dirs actually works for any classes
that aren't compiled into VM indeed (in RI). :-)

> The Pack200 stuff is supposed to be like the JDBC stuff, in that you
> should be able to substitute different implementations at a later
> stage.

Aha, I see. But in JDBC you can at least call
Class.forName(MyJDBCDriverClass, true, *MyClassLoader*).

Regards,

2006/11/14, Alex Blewitt <al...@gmail.com>:
> No, it's a bug (see
> http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6489723). Also note
> that the list of packages that are listed at the URL you gave specify
> the full set of packages that can be overridden, and the pack200
> classes aren't on there :-)
>
> The Pack200 stuff is supposed to be like the JDBC stuff, in that you
> should be able to substitute different implementations at a later
> stage. The problem with the current implementation is that the factory
> is loaded via the system classloader rather than the context
> classloader (mainly because it's a static method, I think) and thus
> the bootclasspath is the only one that has been changed.
>
> In any case, ideally you'd want to be able to substitute a different
> implementation without having to do special things on the classpath,
> either via the bootclasspath or endorsed directories, in order to
> deploy it with your own VM or applications.
>
> Alex.
>
> On 13/11/06, Alexei Zakharov <al...@gmail.com> wrote:
> > > Oh, and I've discovered that the default Sun implementation doesn't
> > > actually allow you to replace it with another implementation unless
> > > it's on the boot classpath.
> >
> > Have you tried endorsed dirs "-Djava.endorsed.dirs=..." [1]? Seems it
> > was specially designed for this purpose.
> >
> > [1] http://java.sun.com/j2se/1.5.0/docs/guide/standards/index.html
> >
> > Regards,
> >
> > 2006/11/11, Alex Blewitt <al...@gmail.com>:
> > > A call of frustration at times perhaps :-)
> > >
> > > It's going along. I'm hoping to get to a stage where I can get a
> > > better patch together and get it into the harmony subversion codebase
> > > in the near future. But sometimes it's just slow progress.
> > >
> > > One thing I'd like to get sorted is moving the pack200 code into its
> > > own module in the near future, perhaps after the next patch bomb.
> > >
> > > Oh, and I've discovered that the default Sun implementation doesn't
> > > actually allow you to replace it with another implementation unless
> > > it's on the boot classpath. D'oh! Fortunately, I've raised it as a bug
> > > with Sun:
> > >
> > > http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6489723
> > >
> > > We should ensure that it doesn't happen with Harmony too :-)
> > >
> > > Alex.
> > >
> > > On 10/11/06, Tim Ellison <t....@gmail.com> wrote:
> > > > Thanks for the update Alex.  I assume from this description (and please
> > > > don't take this the wrong way) that you are happy to be left alone to
> > > > work on this for the moment, rather than it be read as a call for help?
> > > >
> > > > Keep up the good work.
> > > > Tim
> > > >
> > > > Alex Blewitt wrote:
> > > > > I'm still lurking around in the background, but I'm busier than ever
> > > > > what with having to keep posting at EclipseZone where I'm editor now
> > > > > ... but I'm still managing to plod along with the Pack200 code. The
> > > > > last patch (1994, IIRC) went down pretty badly; the subclipse plugin
> > > > > didn't seem to capture all the newly created classes that I had put
> > > > > together for a bit of refactoring that I did. It's also a nightmare
> > > > > submitting newly created files; once they've been uploaded to JIRA
> > > > > into the patch queue, I effectively have to stop work until they're
> > > > > applied, because there's no way of getting SVN to figure out that the
> > > > > file called ClassFile.java is the same as the ClassFile.java that's
> > > > > been added by someone else on my behalf, and I have to overwrite my
> > > > > local copy to update. Kinda prevents doing any sensible work on newly
> > > > > created files between submission and application to be honest.
> > > > >
> > > > > Instead, I plan to sprint (well, jog, at least) to the next stable
> > > > > point, and then upload a whole wodge of code and leave it for a week
> > > > > or two to be applied before picking up again. I'm fairly close to
> > > > > being at that stage, but not quite. Where I am at is being able to
> > > > > generate (abstract) methods with exceptions and fields with constant
> > > > > (integer) values, so a bit further forward than last time. On the one
> > > > > hand, it means that I understand what needs to be done (no mean feat!)
> > > > > but on the other hand, there's still a fair bit to go. For a start,
> > > > > I'm going to need to process the actual bytecode for non-abstract
> > > > > methods (or indeed, any classes) before there's any point in it trying
> > > > > to be used for real, and there's still a few missing bits (Longs,
> > > > > Doubles, Floats) from the constant pools. I'm pretty sure Strings
> > > > > would work, but I've not tried them yet.
> > > > >
> > > > > Unfortunately, the code is really awful. The Segment.java is getting
> > > > > on for 1500 lines of code in a single class; and the attribute parsing
> > > > > contains both duplicated code and isn't as generic as it needs to be
> > > > > to handle other attributes; and the (in)visible annotations like
> > > > > @Overrides and similar are going to be a bit of a nightmare ...
> > > > >
> > > > > My plan is to get the code to a stage where it can start to be useful
> > > > > for extracting simple classes from a pack file, and then start
> > > > > generating test data which can be used for regressions. (There's a bit
> > > > > there at the moment, but it's not exactly a large coverage.) Once I've
> > > > > done that, I'll start tackling some of the refactoring which will
> > > > > result in less duplicated code and hopefully something that will be
> > > > > easier to look after in the future.
> > > > >
> > > > > Anyway, excuse the long rambling but I've been a bit quiet for a while
> > > > > and wanted to let people know I was still alive and kicking the code
> > > > > :-)


-- 
Alexei Zakharov,
Intel Enterprise Solutions Software Division

Re: Re: Re: [classlib][pack200] Status update

[Alex Blewitt <al...@gmail.com>]

No, it's a bug (see
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6489723). Also note
that the list of packages that are listed at the URL you gave specify
the full set of packages that can be overridden, and the pack200
classes aren't on there :-)

The Pack200 stuff is supposed to be like the JDBC stuff, in that you
should be able to substitute different implementations at a later
stage. The problem with the current implementation is that the factory
is loaded via the system classloader rather than the context
classloader (mainly because it's a static method, I think) and thus
the bootclasspath is the only one that has been changed.

In any case, ideally you'd want to be able to substitute a different
implementation without having to do special things on the classpath,
either via the bootclasspath or endorsed directories, in order to
deploy it with your own VM or applications.

Alex.

On 13/11/06, Alexei Zakharov <al...@gmail.com> wrote:
> > Oh, and I've discovered that the default Sun implementation doesn't
> > actually allow you to replace it with another implementation unless
> > it's on the boot classpath.
>
> Have you tried endorsed dirs "-Djava.endorsed.dirs=..." [1]? Seems it
> was specially designed for this purpose.
>
> [1] http://java.sun.com/j2se/1.5.0/docs/guide/standards/index.html
>
> Regards,
>
> 2006/11/11, Alex Blewitt <al...@gmail.com>:
> > A call of frustration at times perhaps :-)
> >
> > It's going along. I'm hoping to get to a stage where I can get a
> > better patch together and get it into the harmony subversion codebase
> > in the near future. But sometimes it's just slow progress.
> >
> > One thing I'd like to get sorted is moving the pack200 code into its
> > own module in the near future, perhaps after the next patch bomb.
> >
> > Oh, and I've discovered that the default Sun implementation doesn't
> > actually allow you to replace it with another implementation unless
> > it's on the boot classpath. D'oh! Fortunately, I've raised it as a bug
> > with Sun:
> >
> > http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6489723
> >
> > We should ensure that it doesn't happen with Harmony too :-)
> >
> > Alex.
> >
> > On 10/11/06, Tim Ellison <t....@gmail.com> wrote:
> > > Thanks for the update Alex.  I assume from this description (and please
> > > don't take this the wrong way) that you are happy to be left alone to
> > > work on this for the moment, rather than it be read as a call for help?
> > >
> > > Keep up the good work.
> > > Tim
> > >
> > > Alex Blewitt wrote:
> > > > I'm still lurking around in the background, but I'm busier than ever
> > > > what with having to keep posting at EclipseZone where I'm editor now
> > > > ... but I'm still managing to plod along with the Pack200 code. The
> > > > last patch (1994, IIRC) went down pretty badly; the subclipse plugin
> > > > didn't seem to capture all the newly created classes that I had put
> > > > together for a bit of refactoring that I did. It's also a nightmare
> > > > submitting newly created files; once they've been uploaded to JIRA
> > > > into the patch queue, I effectively have to stop work until they're
> > > > applied, because there's no way of getting SVN to figure out that the
> > > > file called ClassFile.java is the same as the ClassFile.java that's
> > > > been added by someone else on my behalf, and I have to overwrite my
> > > > local copy to update. Kinda prevents doing any sensible work on newly
> > > > created files between submission and application to be honest.
> > > >
> > > > Instead, I plan to sprint (well, jog, at least) to the next stable
> > > > point, and then upload a whole wodge of code and leave it for a week
> > > > or two to be applied before picking up again. I'm fairly close to
> > > > being at that stage, but not quite. Where I am at is being able to
> > > > generate (abstract) methods with exceptions and fields with constant
> > > > (integer) values, so a bit further forward than last time. On the one
> > > > hand, it means that I understand what needs to be done (no mean feat!)
> > > > but on the other hand, there's still a fair bit to go. For a start,
> > > > I'm going to need to process the actual bytecode for non-abstract
> > > > methods (or indeed, any classes) before there's any point in it trying
> > > > to be used for real, and there's still a few missing bits (Longs,
> > > > Doubles, Floats) from the constant pools. I'm pretty sure Strings
> > > > would work, but I've not tried them yet.
> > > >
> > > > Unfortunately, the code is really awful. The Segment.java is getting
> > > > on for 1500 lines of code in a single class; and the attribute parsing
> > > > contains both duplicated code and isn't as generic as it needs to be
> > > > to handle other attributes; and the (in)visible annotations like
> > > > @Overrides and similar are going to be a bit of a nightmare ...
> > > >
> > > > My plan is to get the code to a stage where it can start to be useful
> > > > for extracting simple classes from a pack file, and then start
> > > > generating test data which can be used for regressions. (There's a bit
> > > > there at the moment, but it's not exactly a large coverage.) Once I've
> > > > done that, I'll start tackling some of the refactoring which will
> > > > result in less duplicated code and hopefully something that will be
> > > > easier to look after in the future.
> > > >
> > > > Anyway, excuse the long rambling but I've been a bit quiet for a while
> > > > and wanted to let people know I was still alive and kicking the code
> > > > :-)
>
>
> --
> Alexei Zakharov,
> Intel Enterprise Solutions Software Division
>

Re: Re: [classlib][pack200] Status update

[Alexei Zakharov <al...@gmail.com>]

> Oh, and I've discovered that the default Sun implementation doesn't
> actually allow you to replace it with another implementation unless
> it's on the boot classpath.

Have you tried endorsed dirs "-Djava.endorsed.dirs=..." [1]? Seems it
was specially designed for this purpose.

[1] http://java.sun.com/j2se/1.5.0/docs/guide/standards/index.html

Regards,

2006/11/11, Alex Blewitt <al...@gmail.com>:
> A call of frustration at times perhaps :-)
>
> It's going along. I'm hoping to get to a stage where I can get a
> better patch together and get it into the harmony subversion codebase
> in the near future. But sometimes it's just slow progress.
>
> One thing I'd like to get sorted is moving the pack200 code into its
> own module in the near future, perhaps after the next patch bomb.
>
> Oh, and I've discovered that the default Sun implementation doesn't
> actually allow you to replace it with another implementation unless
> it's on the boot classpath. D'oh! Fortunately, I've raised it as a bug
> with Sun:
>
> http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6489723
>
> We should ensure that it doesn't happen with Harmony too :-)
>
> Alex.
>
> On 10/11/06, Tim Ellison <t....@gmail.com> wrote:
> > Thanks for the update Alex.  I assume from this description (and please
> > don't take this the wrong way) that you are happy to be left alone to
> > work on this for the moment, rather than it be read as a call for help?
> >
> > Keep up the good work.
> > Tim
> >
> > Alex Blewitt wrote:
> > > I'm still lurking around in the background, but I'm busier than ever
> > > what with having to keep posting at EclipseZone where I'm editor now
> > > ... but I'm still managing to plod along with the Pack200 code. The
> > > last patch (1994, IIRC) went down pretty badly; the subclipse plugin
> > > didn't seem to capture all the newly created classes that I had put
> > > together for a bit of refactoring that I did. It's also a nightmare
> > > submitting newly created files; once they've been uploaded to JIRA
> > > into the patch queue, I effectively have to stop work until they're
> > > applied, because there's no way of getting SVN to figure out that the
> > > file called ClassFile.java is the same as the ClassFile.java that's
> > > been added by someone else on my behalf, and I have to overwrite my
> > > local copy to update. Kinda prevents doing any sensible work on newly
> > > created files between submission and application to be honest.
> > >
> > > Instead, I plan to sprint (well, jog, at least) to the next stable
> > > point, and then upload a whole wodge of code and leave it for a week
> > > or two to be applied before picking up again. I'm fairly close to
> > > being at that stage, but not quite. Where I am at is being able to
> > > generate (abstract) methods with exceptions and fields with constant
> > > (integer) values, so a bit further forward than last time. On the one
> > > hand, it means that I understand what needs to be done (no mean feat!)
> > > but on the other hand, there's still a fair bit to go. For a start,
> > > I'm going to need to process the actual bytecode for non-abstract
> > > methods (or indeed, any classes) before there's any point in it trying
> > > to be used for real, and there's still a few missing bits (Longs,
> > > Doubles, Floats) from the constant pools. I'm pretty sure Strings
> > > would work, but I've not tried them yet.
> > >
> > > Unfortunately, the code is really awful. The Segment.java is getting
> > > on for 1500 lines of code in a single class; and the attribute parsing
> > > contains both duplicated code and isn't as generic as it needs to be
> > > to handle other attributes; and the (in)visible annotations like
> > > @Overrides and similar are going to be a bit of a nightmare ...
> > >
> > > My plan is to get the code to a stage where it can start to be useful
> > > for extracting simple classes from a pack file, and then start
> > > generating test data which can be used for regressions. (There's a bit
> > > there at the moment, but it's not exactly a large coverage.) Once I've
> > > done that, I'll start tackling some of the refactoring which will
> > > result in less duplicated code and hopefully something that will be
> > > easier to look after in the future.
> > >
> > > Anyway, excuse the long rambling but I've been a bit quiet for a while
> > > and wanted to let people know I was still alive and kicking the code
> > > :-)


-- 
Alexei Zakharov,
Intel Enterprise Solutions Software Division

Re: Re: [classlib][pack200] Status update

[Alex Blewitt <al...@gmail.com>]

A call of frustration at times perhaps :-)

It's going along. I'm hoping to get to a stage where I can get a
better patch together and get it into the harmony subversion codebase
in the near future. But sometimes it's just slow progress.

One thing I'd like to get sorted is moving the pack200 code into its
own module in the near future, perhaps after the next patch bomb.

Oh, and I've discovered that the default Sun implementation doesn't
actually allow you to replace it with another implementation unless
it's on the boot classpath. D'oh! Fortunately, I've raised it as a bug
with Sun:

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6489723

We should ensure that it doesn't happen with Harmony too :-)

Alex.

On 10/11/06, Tim Ellison <t....@gmail.com> wrote:
> Thanks for the update Alex.  I assume from this description (and please
> don't take this the wrong way) that you are happy to be left alone to
> work on this for the moment, rather than it be read as a call for help?
>
> Keep up the good work.
> Tim
>
> Alex Blewitt wrote:
> > I'm still lurking around in the background, but I'm busier than ever
> > what with having to keep posting at EclipseZone where I'm editor now
> > ... but I'm still managing to plod along with the Pack200 code. The
> > last patch (1994, IIRC) went down pretty badly; the subclipse plugin
> > didn't seem to capture all the newly created classes that I had put
> > together for a bit of refactoring that I did. It's also a nightmare
> > submitting newly created files; once they've been uploaded to JIRA
> > into the patch queue, I effectively have to stop work until they're
> > applied, because there's no way of getting SVN to figure out that the
> > file called ClassFile.java is the same as the ClassFile.java that's
> > been added by someone else on my behalf, and I have to overwrite my
> > local copy to update. Kinda prevents doing any sensible work on newly
> > created files between submission and application to be honest.
> >
> > Instead, I plan to sprint (well, jog, at least) to the next stable
> > point, and then upload a whole wodge of code and leave it for a week
> > or two to be applied before picking up again. I'm fairly close to
> > being at that stage, but not quite. Where I am at is being able to
> > generate (abstract) methods with exceptions and fields with constant
> > (integer) values, so a bit further forward than last time. On the one
> > hand, it means that I understand what needs to be done (no mean feat!)
> > but on the other hand, there's still a fair bit to go. For a start,
> > I'm going to need to process the actual bytecode for non-abstract
> > methods (or indeed, any classes) before there's any point in it trying
> > to be used for real, and there's still a few missing bits (Longs,
> > Doubles, Floats) from the constant pools. I'm pretty sure Strings
> > would work, but I've not tried them yet.
> >
> > Unfortunately, the code is really awful. The Segment.java is getting
> > on for 1500 lines of code in a single class; and the attribute parsing
> > contains both duplicated code and isn't as generic as it needs to be
> > to handle other attributes; and the (in)visible annotations like
> > @Overrides and similar are going to be a bit of a nightmare ...
> >
> > My plan is to get the code to a stage where it can start to be useful
> > for extracting simple classes from a pack file, and then start
> > generating test data which can be used for regressions. (There's a bit
> > there at the moment, but it's not exactly a large coverage.) Once I've
> > done that, I'll start tackling some of the refactoring which will
> > result in less duplicated code and hopefully something that will be
> > easier to look after in the future.
> >
> > Anyway, excuse the long rambling but I've been a bit quiet for a while
> > and wanted to let people know I was still alive and kicking the code
> > :-)
> >
> > Alex.
> >
>
> --
>
> Tim Ellison (t.p.ellison@gmail.com)
> IBM Java technology centre, UK.
>

Re: [classlib][pack200] Status update

[Tim Ellison <t....@gmail.com>]

Thanks for the update Alex.  I assume from this description (and please
don't take this the wrong way) that you are happy to be left alone to
work on this for the moment, rather than it be read as a call for help?

Keep up the good work.
Tim

Alex Blewitt wrote:
> I'm still lurking around in the background, but I'm busier than ever
> what with having to keep posting at EclipseZone where I'm editor now
> ... but I'm still managing to plod along with the Pack200 code. The
> last patch (1994, IIRC) went down pretty badly; the subclipse plugin
> didn't seem to capture all the newly created classes that I had put
> together for a bit of refactoring that I did. It's also a nightmare
> submitting newly created files; once they've been uploaded to JIRA
> into the patch queue, I effectively have to stop work until they're
> applied, because there's no way of getting SVN to figure out that the
> file called ClassFile.java is the same as the ClassFile.java that's
> been added by someone else on my behalf, and I have to overwrite my
> local copy to update. Kinda prevents doing any sensible work on newly
> created files between submission and application to be honest.
> 
> Instead, I plan to sprint (well, jog, at least) to the next stable
> point, and then upload a whole wodge of code and leave it for a week
> or two to be applied before picking up again. I'm fairly close to
> being at that stage, but not quite. Where I am at is being able to
> generate (abstract) methods with exceptions and fields with constant
> (integer) values, so a bit further forward than last time. On the one
> hand, it means that I understand what needs to be done (no mean feat!)
> but on the other hand, there's still a fair bit to go. For a start,
> I'm going to need to process the actual bytecode for non-abstract
> methods (or indeed, any classes) before there's any point in it trying
> to be used for real, and there's still a few missing bits (Longs,
> Doubles, Floats) from the constant pools. I'm pretty sure Strings
> would work, but I've not tried them yet.
> 
> Unfortunately, the code is really awful. The Segment.java is getting
> on for 1500 lines of code in a single class; and the attribute parsing
> contains both duplicated code and isn't as generic as it needs to be
> to handle other attributes; and the (in)visible annotations like
> @Overrides and similar are going to be a bit of a nightmare ...
> 
> My plan is to get the code to a stage where it can start to be useful
> for extracting simple classes from a pack file, and then start
> generating test data which can be used for regressions. (There's a bit
> there at the moment, but it's not exactly a large coverage.) Once I've
> done that, I'll start tackling some of the refactoring which will
> result in less duplicated code and hopefully something that will be
> easier to look after in the future.
> 
> Anyway, excuse the long rambling but I've been a bit quiet for a while
> and wanted to let people know I was still alive and kicking the code
> :-)
> 
> Alex.
> 

-- 

Tim Ellison (t.p.ellison@gmail.com)
IBM Java technology centre, UK.