Re: JCC CompileError -- incorrect generic parameter detection

[Andi Vajda <va...@apache.org>]

  Hi Lukasz,

On Sun, 31 Jul 2011, ?ukasz Jancewicz wrote:

> On Fri, Jul 29, 2011 at 17:09, Andi Vajda <va...@apache.org> wrote:
>> For example, is there a piece of gentyref code that I could 'borrow' (with
>> attribution of course) and include that in the JCC sources to fix this
>> particular problem ?
>
> If you look at this file:
>
> http://code.google.com/p/gentyref/source/browse/src/main/java/com/googlecode/gentyref/GenericTypeReflector.java
>
> you'll see that it's completely independent from the rest of gentyref
> library, so I guess that, technically, you could just copy & paste it
> to JCC.

Not really, no. It depends on the whole thing but that's no big deal. The 
licensing is also compatible since it's Apache 2.0 licensed (like you said 
originally but I missed that, sorry). So it's probably ok to use/include it.

> I don't know much about Apache licensing, so I can't tell you
> if it's legal/appropriate to do so. But the technical possibility
> obviously exists. The advantage of including the library as a whole is
> that any future Java changes (Java 7, etc.) and bugs can be
> potentially taken care of by developers of gentyref.
>
>> This code could be written in Java (and wrapped by JCC for itself)
>
> That's what I did in my patch. I included gentyref.jar in the
> classpath and generated JCC wrappers for it.

So I did a custom build of JCC with that gentyref class wrapped and it does 
fix the problem you encountered but it then no longer compiles Lucene :-(

I get this detailed error message from gentyref:

  jcc.cpp.JavaError: com.googlecode.gentyref.UnresolvedTypeVariableException:
  An exact type is requested, but the type contains a type variable that
  cannot be resolved.
     Variable: A from public org.apache.lucene.util.Attribute
  org.apache.lucene.util.AttributeSource.addAttribute(java.lang.Class)
     Hint: This is usually caused by trying to get an exact type when a
  generic method who's type parameters are not given is involved.

Hacking it a bit, I catch the error and use the original reflection code 
when gentyref fails to see how far I get and I get a bit further but I hit 
more problems with too-specific types being resolved (like array of bool 
into [B). I could probably fix this too but I'm not yet convinced that 
gentyref is actually needed to solve the original problem. It feels like 
gentyref, cool as it is, is actually doing too much.

Clearly, I see the bug you reported but I'm not sure where it is yet. Is it 
in the java.lang.reflect code or is it in jcc itself ? For example, the same 
problem happens if you just define DirectoryEntry as:

  public interface DirectoryEntry extends Entry, Iterable {
  }

But not when I define it thus:

  import java.util.Iterator;
  public interface DirectoryEntry extends Entry, Iterable {
      Iterator iterator();
  }

Or thus:

  import java.util.Iterator;
  public interface DirectoryEntry extends Entry, Iterable<Entry> {
      Iterator<Entry> iterator();
  }

It looks like the absence of an iterator() method definition triggers this.
Maybe all I need to do is make the iterator method code generation a bit 
smarter, like not generate it if it's inherited from above anyway ?
Or see if it's inherited and its return type is overriden by the extends ?

I'm not quite sure yet what to do about this bug...

Andi..

Re: JCC CompileError -- incorrect generic parameter detection

[Andi Vajda <va...@apache.org>]

  Hi Lukasz,

Thank you for the patch. Comments inline.

On Fri, 5 Aug 2011, ?ukasz Jancewicz wrote:

> By the way, I discovered some other issues while using JCC in my project:
>
> 1) In __init__.py of generated code, you set a CLASSPATH variable, but
> then you don't use it anywhere. When I added gentyref.jar to the
> classpath, I was getting ClassDefNotFound errors. Somehow, your
> "classes" dir is found anyway without it. My solution: import
> CLASSPATH from jcc and use env._addClassPath(CLASSPATH) after
> initialising Java VM. I'm not certain that this is the perfect
> solution, but it worked for me.

The CLASSPATH from the generated __init__.py file is used line 400 in 
jcc.cpp and on.

The CLASSPATH from jcc's __init__.py itself was unused indeed since wasn't 
set to the _jcc module and thus not found by jcc.cpp. This may have gone 
unnoticed because jcc itself doesn't normally use any java classes that are 
not in the JRE.

The classes in jcc's 'classes' directory are used by the libjcc runtime lib 
when running jcc-generated modules built with shared mode. I added 
_jcc.CLASSPATH = CLASSPATH to jcc's __init__.py file and classpath is now 
initialized to _jcc.CLASSPATH before parsing command line args instead of 
starting empty.

> 2) I needed to use one class from Tika that was not public. I know
> that I could just modify Tika to make the class public, but I think
> sometimes you don't have access to library source code and want to use
> it anyway. I made a small modification so that if you explicitly enter
> the class name (not the jar containing it) as one of JCC arguments, it
> is exposed even if it's not public.

Thank you for the feature, I integrated that patch.

> 3) you're mixing tabs and spaces in several places in Python code.

Oops, that's not good. Thank you for pointing it out. Fixed.

> I've attached a patch file with these changes. If you feel that any of
> them might improve JCC, please add them.
>
> Thanks again for your help!

You're welcome !

Andi..

Re: JCC CompileError -- incorrect generic parameter detection

[Łukasz Jancewicz <dz...@gmail.com>]

Hi Andi,

On Thu, Aug 4, 2011 at 18:37, Andi Vajda <va...@apache.org> wrote:
> I took another look at this and it seems that the problem could stem from
> another issue: jcc is generating wrappers for methods declared on interfaces
> that are declared as implemented by a class regardless of whether the method
> is, in fact, implemented by the class.

You're right! I applied your patch to JCC trunk and it worked without
problems. Thank you! So, gentyref is unnecessary after all.

By the way, I discovered some other issues while using JCC in my project:

1) In __init__.py of generated code, you set a CLASSPATH variable, but
then you don't use it anywhere. When I added gentyref.jar to the
classpath, I was getting ClassDefNotFound errors. Somehow, your
"classes" dir is found anyway without it. My solution: import
CLASSPATH from jcc and use env._addClassPath(CLASSPATH) after
initialising Java VM. I'm not certain that this is the perfect
solution, but it worked for me.

2) I needed to use one class from Tika that was not public. I know
that I could just modify Tika to make the class public, but I think
sometimes you don't have access to library source code and want to use
it anyway. I made a small modification so that if you explicitly enter
the class name (not the jar containing it) as one of JCC arguments, it
is exposed even if it's not public.

3) you're mixing tabs and spaces in several places in Python code.

I've attached a patch file with these changes. If you feel that any of
them might improve JCC, please add them.

Thanks again for your help!
-- 
Łukasz Jancewicz

Re: JCC CompileError -- incorrect generic parameter detection

[Andi Vajda <va...@apache.org>]

  Hi Lukasz,

On Mon, 1 Aug 2011, Andi Vajda wrote:

> So I did a custom build of JCC with that gentyref class wrapped and it does 
> fix the problem you encountered but it then no longer compiles Lucene :-(
>
> I get this detailed error message from gentyref:
>
> jcc.cpp.JavaError: com.googlecode.gentyref.UnresolvedTypeVariableException:
> An exact type is requested, but the type contains a type variable that
> cannot be resolved.
>    Variable: A from public org.apache.lucene.util.Attribute
> org.apache.lucene.util.AttributeSource.addAttribute(java.lang.Class)
>    Hint: This is usually caused by trying to get an exact type when a
> generic method who's type parameters are not given is involved.
>
> Hacking it a bit, I catch the error and use the original reflection code when 
> gentyref fails to see how far I get and I get a bit further but I hit more 
> problems with too-specific types being resolved (like array of bool into [B). 
> I could probably fix this too but I'm not yet convinced that gentyref is 
> actually needed to solve the original problem. It feels like gentyref, cool 
> as it is, is actually doing too much.
>
> Clearly, I see the bug you reported but I'm not sure where it is yet. Is it 
> in the java.lang.reflect code or is it in jcc itself ? For example, the same 
> problem happens if you just define DirectoryEntry as:
>
> public interface DirectoryEntry extends Entry, Iterable {
> }
>
> But not when I define it thus:
>
> import java.util.Iterator;
> public interface DirectoryEntry extends Entry, Iterable {
>     Iterator iterator();
> }
>
> Or thus:
>
> import java.util.Iterator;
> public interface DirectoryEntry extends Entry, Iterable<Entry> {
>     Iterator<Entry> iterator();
> }
>
> It looks like the absence of an iterator() method definition triggers this.
> Maybe all I need to do is make the iterator method code generation a bit 
> smarter, like not generate it if it's inherited from above anyway ?
> Or see if it's inherited and its return type is overriden by the extends ?
>
> I'm not quite sure yet what to do about this bug...

I took another look at this and it seems that the problem could stem from 
another issue: jcc is generating wrappers for methods declared on interfaces 
that are declared as implemented by a class regardless of whether the method 
is, in fact, implemented by the class.

I don't remember why I wrote it this way originally but it seems to me that 
generating wrappers only for methods with an implementation on a class 
should be sufficient - and taken care by the 'for method in 
cls.getDeclaredMethods() loop above - since the others would be inherited 
anyway. I removed that half page of code and rebuilt PyLucene. All looks 
good.

Could you please try wrapping Tika again with this patch applied to jcc and 
let me know if the result works (and solves the problem you solved with 
gentyref) ?

Thanks !

Andi..