code generation for hash maps

[Benson Margulies <bi...@gmail.com>]

The Trove people use a code generator to get all the variations of
IntByte, IntChar, IntInt, IntLong. Does anyone have strenuous
objections to this sort of thing?

Re: code generation for hash maps

[Jake Mannix <ja...@gmail.com>]

On Mon, Dec 14, 2009 at 5:43 PM, Benson Margulies <bi...@gmail.com>wrote:

> If this is not acceptable, I'll just manually create all the classes.
> It's just easier to keep them in sync if there's one source instead of
> 16 to edit. (The presumption here is that all of 16 classes
> {Byte,Char,Int,Long}x{Byte,Int,Char,Long} are really identical except
> for the data types).
>

{byte,char,int,long,float,double}^2, right?


  -jake

Re: code generation for hash maps

[Benson Margulies <bi...@gmail.com>]

I don't anticipate any big problems in making the default value configurable.

Yes, Trove has the complete cross-product, so matching that sure makes
a code generator attractive.


On Wed, Dec 16, 2009 at 6:18 PM, Ted Dunning <te...@gmail.com> wrote:
> This seems most congenial.  As Jake says, 0 is definitely handier for the
> sparse matrix/vector case and NaN is very nice for the collection case.
>
> On Wed, Dec 16, 2009 at 3:10 PM, Jake Mannix <ja...@gmail.com> wrote:
>
>> Of course, you could also just have the {X}{Y}Map take a default value in
>> the constructor, and return this value whenever contains(k) == false.  The
>> default default
>> value could be zero for integral types, and NaN for floating points.
>>
>
>
>
> --
> Ted Dunning, CTO
> DeepDyve
>

Re: code generation for hash maps

[Benson Margulies <bi...@gmail.com>]

And, in fact, after a bit more thinking, Velocity it is.

On Fri, Dec 18, 2009 at 4:31 PM, Benson Margulies <bi...@gmail.com> wrote:
> 1) velocity spews endless useless log messages whenever you touch it.
> 2) I found an ant task for fmpp that could be easily mutated into a
> maven mojo. I didn't find the equivalent for velocity, but I think I
> got distracted in the process.
>
> I think that the templates in this case will be so simple as to make
> the point moot. In fact, I begin to suspect that either device could
> digest them.
>
> I started writing code to call fmpp, but if I don't find a good answer
> to the puzzle of the moment, I'm going to go read up the velocity doc
> and perhaps change horses.
>
>
> On Fri, Dec 18, 2009 at 2:04 PM, Ted Dunning <te...@gmail.com> wrote:
>> I don't think I know enough to comment intelligently.
>>
>> But I will comment anyway.  :-)
>>
>> I see three options, off-hand,
>>
>> a) velocity (the standard answer, don't know if it fits)
>>
>> b) freemarker (a more fashionable answer, don't know any strong reason it is
>> better than (a))
>>
>> c) something off the wall like python or groovy.
>>
>> Benson,
>>
>> Could you comment on what moves you to like option (b)?  I yearn to be
>> educated.
>>
>> On Fri, Dec 18, 2009 at 4:02 AM, Benson Margulies <bi...@gmail.com>wrote:
>>
>>> Meanwhile, on this thread, anyone object to FreeMarker as the template
>>> language?
>>>
>>> On Fri, Dec 18, 2009 at 7:02 AM, Benson Margulies <bi...@gmail.com>
>>> wrote:
>>> > If you want speed, you want a shortcut for this.
>>> >
>>> > On Fri, Dec 18, 2009 at 2:40 AM, Sean Owen <sr...@gmail.com> wrote:
>>> >> Agree, but isn't the conclusion that the internal map returns NaN, and
>>> >> if desired, the vector converts that to 0.0?
>>> >> In my world of maps, I don't have any case where 'null' should mean '0'.
>>> >>
>>> >> On Thu, Dec 17, 2009 at 2:18 AM, Ted Dunning <te...@gmail.com>
>>> wrote:
>>> >>> This seems most congenial.  As Jake says, 0 is definitely handier for
>>> the
>>> >>> sparse matrix/vector case and NaN is very nice for the collection case.
>>> >>
>>> >
>>>
>>
>>
>>
>> --
>> Ted Dunning, CTO
>> DeepDyve
>>
>

Re: code generation for hash maps

[Benson Margulies <bi...@gmail.com>]

1) velocity spews endless useless log messages whenever you touch it.
2) I found an ant task for fmpp that could be easily mutated into a
maven mojo. I didn't find the equivalent for velocity, but I think I
got distracted in the process.

I think that the templates in this case will be so simple as to make
the point moot. In fact, I begin to suspect that either device could
digest them.

I started writing code to call fmpp, but if I don't find a good answer
to the puzzle of the moment, I'm going to go read up the velocity doc
and perhaps change horses.


On Fri, Dec 18, 2009 at 2:04 PM, Ted Dunning <te...@gmail.com> wrote:
> I don't think I know enough to comment intelligently.
>
> But I will comment anyway.  :-)
>
> I see three options, off-hand,
>
> a) velocity (the standard answer, don't know if it fits)
>
> b) freemarker (a more fashionable answer, don't know any strong reason it is
> better than (a))
>
> c) something off the wall like python or groovy.
>
> Benson,
>
> Could you comment on what moves you to like option (b)?  I yearn to be
> educated.
>
> On Fri, Dec 18, 2009 at 4:02 AM, Benson Margulies <bi...@gmail.com>wrote:
>
>> Meanwhile, on this thread, anyone object to FreeMarker as the template
>> language?
>>
>> On Fri, Dec 18, 2009 at 7:02 AM, Benson Margulies <bi...@gmail.com>
>> wrote:
>> > If you want speed, you want a shortcut for this.
>> >
>> > On Fri, Dec 18, 2009 at 2:40 AM, Sean Owen <sr...@gmail.com> wrote:
>> >> Agree, but isn't the conclusion that the internal map returns NaN, and
>> >> if desired, the vector converts that to 0.0?
>> >> In my world of maps, I don't have any case where 'null' should mean '0'.
>> >>
>> >> On Thu, Dec 17, 2009 at 2:18 AM, Ted Dunning <te...@gmail.com>
>> wrote:
>> >>> This seems most congenial.  As Jake says, 0 is definitely handier for
>> the
>> >>> sparse matrix/vector case and NaN is very nice for the collection case.
>> >>
>> >
>>
>
>
>
> --
> Ted Dunning, CTO
> DeepDyve
>

Re: code generation for hash maps

[Ted Dunning <te...@gmail.com>]

I don't think I know enough to comment intelligently.

But I will comment anyway.  :-)

I see three options, off-hand,

a) velocity (the standard answer, don't know if it fits)

b) freemarker (a more fashionable answer, don't know any strong reason it is
better than (a))

c) something off the wall like python or groovy.

Benson,

Could you comment on what moves you to like option (b)?  I yearn to be
educated.

On Fri, Dec 18, 2009 at 4:02 AM, Benson Margulies <bi...@gmail.com>wrote:

> Meanwhile, on this thread, anyone object to FreeMarker as the template
> language?
>
> On Fri, Dec 18, 2009 at 7:02 AM, Benson Margulies <bi...@gmail.com>
> wrote:
> > If you want speed, you want a shortcut for this.
> >
> > On Fri, Dec 18, 2009 at 2:40 AM, Sean Owen <sr...@gmail.com> wrote:
> >> Agree, but isn't the conclusion that the internal map returns NaN, and
> >> if desired, the vector converts that to 0.0?
> >> In my world of maps, I don't have any case where 'null' should mean '0'.
> >>
> >> On Thu, Dec 17, 2009 at 2:18 AM, Ted Dunning <te...@gmail.com>
> wrote:
> >>> This seems most congenial.  As Jake says, 0 is definitely handier for
> the
> >>> sparse matrix/vector case and NaN is very nice for the collection case.
> >>
> >
>



-- 
Ted Dunning, CTO
DeepDyve

Re: code generation for hash maps

[Benson Margulies <bi...@gmail.com>]

Meanwhile, on this thread, anyone object to FreeMarker as the template language?

On Fri, Dec 18, 2009 at 7:02 AM, Benson Margulies <bi...@gmail.com> wrote:
> If you want speed, you want a shortcut for this.
>
> On Fri, Dec 18, 2009 at 2:40 AM, Sean Owen <sr...@gmail.com> wrote:
>> Agree, but isn't the conclusion that the internal map returns NaN, and
>> if desired, the vector converts that to 0.0?
>> In my world of maps, I don't have any case where 'null' should mean '0'.
>>
>> On Thu, Dec 17, 2009 at 2:18 AM, Ted Dunning <te...@gmail.com> wrote:
>>> This seems most congenial.  As Jake says, 0 is definitely handier for the
>>> sparse matrix/vector case and NaN is very nice for the collection case.
>>
>

Re: code generation for hash maps

[Benson Margulies <bi...@gmail.com>]

If you want speed, you want a shortcut for this.

On Fri, Dec 18, 2009 at 2:40 AM, Sean Owen <sr...@gmail.com> wrote:
> Agree, but isn't the conclusion that the internal map returns NaN, and
> if desired, the vector converts that to 0.0?
> In my world of maps, I don't have any case where 'null' should mean '0'.
>
> On Thu, Dec 17, 2009 at 2:18 AM, Ted Dunning <te...@gmail.com> wrote:
>> This seems most congenial.  As Jake says, 0 is definitely handier for the
>> sparse matrix/vector case and NaN is very nice for the collection case.
>

Re: code generation for hash maps

[Sean Owen <sr...@gmail.com>]

Agree, but isn't the conclusion that the internal map returns NaN, and
if desired, the vector converts that to 0.0?
In my world of maps, I don't have any case where 'null' should mean '0'.

On Thu, Dec 17, 2009 at 2:18 AM, Ted Dunning <te...@gmail.com> wrote:
> This seems most congenial.  As Jake says, 0 is definitely handier for the
> sparse matrix/vector case and NaN is very nice for the collection case.

Re: code generation for hash maps

[Ted Dunning <te...@gmail.com>]

This seems most congenial.  As Jake says, 0 is definitely handier for the
sparse matrix/vector case and NaN is very nice for the collection case.

On Wed, Dec 16, 2009 at 3:10 PM, Jake Mannix <ja...@gmail.com> wrote:

> Of course, you could also just have the {X}{Y}Map take a default value in
> the constructor, and return this value whenever contains(k) == false.  The
> default default
> value could be zero for integral types, and NaN for floating points.
>



-- 
Ted Dunning, CTO
DeepDyve

Re: code generation for hash maps

[Jake Mannix <ja...@gmail.com>]

On Tue, Dec 15, 2009 at 5:26 AM, Benson Margulies <bi...@gmail.com>wrote:

> As to float and double, they may need to be special cases. Ask
> yourself: "What should get(k) return when containsKey(k) is false and
> the payload is double?"
>
> Answer 1: NaN.
> Answer 2: zero, like with the int types.
>

I actually feel that Answer 2 is more correct, if only for the fact that we
currently use
an IntDoubleMap to back our SparseVector impl, and for SparseVectors, we
definitely
want zero as the default.

For general collections, yes, I can see not doing this, but at least in
Mahout, we use
zero-default vectors and matrices.

I guess we could have both, in which case we could autogenerate the zero
case with
the others.

Of course, you could also just have the {X}{Y}Map take a default value in
the constructor,
and return this value whenever contains(k) == false.  The default default
value could be
zero for integral types, and NaN for floating points.

This is maximally flexible at the cost of one additional primitive stored
and no cpu.

Also, what about other primitives?  boolean, short?  Seems like we should
really have
the full complement of all 64 classes, right?

   {boolean}{byte}{char}{short}{int}{long}{float}{double}^2

  -jake

Re: code generation for hash maps

[Benson Margulies <bi...@gmail.com>]

My focus is at the moment on the colt-derived code. I'll report in
when I get that far.

On Tue, Dec 15, 2009 at 10:10 AM, Sean Owen <sr...@gmail.com> wrote:
> There are other solutions that aren't too expensive like maintaining a
> bit set of entries that are valid (which is fast since it initializes
> to all false/0). I've got a fast implementation in the code already.
> I'd prefer to not let the performance consideration convince us to not
> implement the semantics that seem correct -- if we agree that NaN is
> 'correct'.
>
> On Tue, Dec 15, 2009 at 2:57 PM, Benson Margulies <bi...@gmail.com> wrote:
>> I had better be more clear. The problem is that Arrays.fill does use
>> some time. If you just say new double[1000], you've got 0. To get NaN,
>> you then need a fill. Last summer, when I was trying to get an E+M
>> procedure to run in reasonable time, and trying to initialize many
>> arrays  with NaNs to detect various errors, I measured and found that
>> I was getting killed. Now, does that mean that at construction and
>> resize of these hash tables that we can't afford the fill? I won't
>> make that claim without a measurement.
>

Re: code generation for hash maps

[Sean Owen <sr...@gmail.com>]

There are other solutions that aren't too expensive like maintaining a
bit set of entries that are valid (which is fast since it initializes
to all false/0). I've got a fast implementation in the code already.
I'd prefer to not let the performance consideration convince us to not
implement the semantics that seem correct -- if we agree that NaN is
'correct'.

On Tue, Dec 15, 2009 at 2:57 PM, Benson Margulies <bi...@gmail.com> wrote:
> I had better be more clear. The problem is that Arrays.fill does use
> some time. If you just say new double[1000], you've got 0. To get NaN,
> you then need a fill. Last summer, when I was trying to get an E+M
> procedure to run in reasonable time, and trying to initialize many
> arrays  with NaNs to detect various errors, I measured and found that
> I was getting killed. Now, does that mean that at construction and
> resize of these hash tables that we can't afford the fill? I won't
> make that claim without a measurement.

Re: code generation for hash maps

[Benson Margulies <bi...@gmail.com>]

I had better be more clear. The problem is that Arrays.fill does use
some time. If you just say new double[1000], you've got 0. To get NaN,
you then need a fill. Last summer, when I was trying to get an E+M
procedure to run in reasonable time, and trying to initialize many
arrays  with NaNs to detect various errors, I measured and found that
I was getting killed. Now, does that mean that at construction and
resize of these hash tables that we can't afford the fill? I won't
make that claim without a measurement.

On Tue, Dec 15, 2009 at 9:51 AM, Sean Owen <sr...@gmail.com> wrote:
> NaN is my vote too from a correctness standpoint. Really,
> Arrays.fill() is unnaturally slow to fill in NaN? this sounds odd. Is
> it significant?
>
> On Tue, Dec 15, 2009 at 1:26 PM, Benson Margulies <bi...@gmail.com> wrote:
>> My first reaction was to be opposed to checking in generated output,
>> but now I can't really think of a strong objection. A separate maven
>> profile to enable the generator, and off we go.
>>
>> As to float and double, they may need to be special cases. Ask
>> yourself: "What should get(k) return when containsKey(k) is false and
>> the payload is double?"
>>
>> Answer 1: NaN.
>> Answer 2: zero, like with the int types.
>>
>> Answer 1 feels more correct, but I've discovered that the JVM is very
>> slow in filling an array with NaN values, whereas, of course, new
>> delivers a supply of fresh, clean, zeros in a jiffy.
>>
>>
>> On Tue, Dec 15, 2009 at 7:12 AM, Grant Ingersoll <gs...@apache.org> wrote:
>>>
>>> On Dec 14, 2009, at 11:38 PM, Sean Owen wrote:
>>>
>>>> I think both are desirable. Check in the generator and also the
>>>> generated code. It just keeps it simple.
>>>
>>> +1.  This is what Lucene does w/ the QueryParser (JavaCC) and the Ant script detects when the two are out of sync and then regenerates.
>>>
>>>>
>>>> On Tue, Dec 15, 2009 at 1:43 AM, Benson Margulies <bi...@gmail.com> wrote:
>>>>> The idea here is that the templates are checked in. The build process
>>>>> goes template-to-java and then compiles the java. In other words, the
>>>>> java classes are not checked in and editable.
>>>>>
>>>>> If this is not acceptable, I'll just manually create all the classes.
>>>>> It's just easier to keep them in sync if there's one source instead of
>>>>> 16 to edit. (The presumption here is that all of 16 classes
>>>>> {Byte,Char,Int,Long}x{Byte,Int,Char,Long} are really identical except
>>>>> for the data types).
>>>
>>>
>>>
>>
>

Re: code generation for hash maps

[Sean Owen <sr...@gmail.com>]

NaN is my vote too from a correctness standpoint. Really,
Arrays.fill() is unnaturally slow to fill in NaN? this sounds odd. Is
it significant?

On Tue, Dec 15, 2009 at 1:26 PM, Benson Margulies <bi...@gmail.com> wrote:
> My first reaction was to be opposed to checking in generated output,
> but now I can't really think of a strong objection. A separate maven
> profile to enable the generator, and off we go.
>
> As to float and double, they may need to be special cases. Ask
> yourself: "What should get(k) return when containsKey(k) is false and
> the payload is double?"
>
> Answer 1: NaN.
> Answer 2: zero, like with the int types.
>
> Answer 1 feels more correct, but I've discovered that the JVM is very
> slow in filling an array with NaN values, whereas, of course, new
> delivers a supply of fresh, clean, zeros in a jiffy.
>
>
> On Tue, Dec 15, 2009 at 7:12 AM, Grant Ingersoll <gs...@apache.org> wrote:
>>
>> On Dec 14, 2009, at 11:38 PM, Sean Owen wrote:
>>
>>> I think both are desirable. Check in the generator and also the
>>> generated code. It just keeps it simple.
>>
>> +1.  This is what Lucene does w/ the QueryParser (JavaCC) and the Ant script detects when the two are out of sync and then regenerates.
>>
>>>
>>> On Tue, Dec 15, 2009 at 1:43 AM, Benson Margulies <bi...@gmail.com> wrote:
>>>> The idea here is that the templates are checked in. The build process
>>>> goes template-to-java and then compiles the java. In other words, the
>>>> java classes are not checked in and editable.
>>>>
>>>> If this is not acceptable, I'll just manually create all the classes.
>>>> It's just easier to keep them in sync if there's one source instead of
>>>> 16 to edit. (The presumption here is that all of 16 classes
>>>> {Byte,Char,Int,Long}x{Byte,Int,Char,Long} are really identical except
>>>> for the data types).
>>
>>
>>
>

Re: code generation for hash maps

[Benson Margulies <bi...@gmail.com>]

My first reaction was to be opposed to checking in generated output,
but now I can't really think of a strong objection. A separate maven
profile to enable the generator, and off we go.

As to float and double, they may need to be special cases. Ask
yourself: "What should get(k) return when containsKey(k) is false and
the payload is double?"

Answer 1: NaN.
Answer 2: zero, like with the int types.

Answer 1 feels more correct, but I've discovered that the JVM is very
slow in filling an array with NaN values, whereas, of course, new
delivers a supply of fresh, clean, zeros in a jiffy.


On Tue, Dec 15, 2009 at 7:12 AM, Grant Ingersoll <gs...@apache.org> wrote:
>
> On Dec 14, 2009, at 11:38 PM, Sean Owen wrote:
>
>> I think both are desirable. Check in the generator and also the
>> generated code. It just keeps it simple.
>
> +1.  This is what Lucene does w/ the QueryParser (JavaCC) and the Ant script detects when the two are out of sync and then regenerates.
>
>>
>> On Tue, Dec 15, 2009 at 1:43 AM, Benson Margulies <bi...@gmail.com> wrote:
>>> The idea here is that the templates are checked in. The build process
>>> goes template-to-java and then compiles the java. In other words, the
>>> java classes are not checked in and editable.
>>>
>>> If this is not acceptable, I'll just manually create all the classes.
>>> It's just easier to keep them in sync if there's one source instead of
>>> 16 to edit. (The presumption here is that all of 16 classes
>>> {Byte,Char,Int,Long}x{Byte,Int,Char,Long} are really identical except
>>> for the data types).
>
>
>

Re: code generation for hash maps

[Grant Ingersoll <gs...@apache.org>]

On Dec 14, 2009, at 11:38 PM, Sean Owen wrote:

> I think both are desirable. Check in the generator and also the
> generated code. It just keeps it simple.

+1.  This is what Lucene does w/ the QueryParser (JavaCC) and the Ant script detects when the two are out of sync and then regenerates.

> 
> On Tue, Dec 15, 2009 at 1:43 AM, Benson Margulies <bi...@gmail.com> wrote:
>> The idea here is that the templates are checked in. The build process
>> goes template-to-java and then compiles the java. In other words, the
>> java classes are not checked in and editable.
>> 
>> If this is not acceptable, I'll just manually create all the classes.
>> It's just easier to keep them in sync if there's one source instead of
>> 16 to edit. (The presumption here is that all of 16 classes
>> {Byte,Char,Int,Long}x{Byte,Int,Char,Long} are really identical except
>> for the data types).

Re: code generation for hash maps

[Sean Owen <sr...@gmail.com>]

I think both are desirable. Check in the generator and also the
generated code. It just keeps it simple.

On Tue, Dec 15, 2009 at 1:43 AM, Benson Margulies <bi...@gmail.com> wrote:
> The idea here is that the templates are checked in. The build process
> goes template-to-java and then compiles the java. In other words, the
> java classes are not checked in and editable.
>
> If this is not acceptable, I'll just manually create all the classes.
> It's just easier to keep them in sync if there's one source instead of
> 16 to edit. (The presumption here is that all of 16 classes
> {Byte,Char,Int,Long}x{Byte,Int,Char,Long} are really identical except
> for the data types).

Re: code generation for hash maps

[Benson Margulies <bi...@gmail.com>]

The idea here is that the templates are checked in. The build process
goes template-to-java and then compiles the java. In other words, the
java classes are not checked in and editable.

If this is not acceptable, I'll just manually create all the classes.
It's just easier to keep them in sync if there's one source instead of
16 to edit. (The presumption here is that all of 16 classes
{Byte,Char,Int,Long}x{Byte,Int,Char,Long} are really identical except
for the data types).

On Mon, Dec 14, 2009 at 6:58 PM, Ted Dunning <te...@gmail.com> wrote:
> not if there is editable and compilable source of some kind.
>
> On Mon, Dec 14, 2009 at 3:28 PM, Benson Margulies <bi...@gmail.com>wrote:
>
>> The Trove people use a code generator to get all the variations of
>> IntByte, IntChar, IntInt, IntLong. Does anyone have strenuous
>> objections to this sort of thing?
>>
>
>
>
> --
> Ted Dunning, CTO
> DeepDyve
>

Re: code generation for hash maps

[Ted Dunning <te...@gmail.com>]

not if there is editable and compilable source of some kind.

On Mon, Dec 14, 2009 at 3:28 PM, Benson Margulies <bi...@gmail.com>wrote:

> The Trove people use a code generator to get all the variations of
> IntByte, IntChar, IntInt, IntLong. Does anyone have strenuous
> objections to this sort of thing?
>



-- 
Ted Dunning, CTO
DeepDyve