You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@directmemory.apache.org by Noctarius <me...@noctarius.com> on 2012/09/29 14:40:31 UTC
Additional Serializer and raw Buffer access
Hey guys,
Raffaele found out about a project of mine (Lightning) a few weeks
ago. Lightning is a heavy Unsafe and Bytecode generation using
Serializer implementation. He told me that he was interested in
adding something similar to DirectMemory and I would be glad to help
out in this.
Another project I started a few days ago, since it was needed for
work is DirectRingCache. The name not really meets to actual
implementation since it's not yet a ring buffer using cache. I used
this for a pre-serialization simple bytestream cache with
self-growing buffers. It could be nice to have DirectMemory having
raw "buffers" to write to or to read from.
Here are the links from the projects:
https://github.com/noctarius/Lightning
https://github.com/noctarius/direct-ring-cache
It would be nice to help out since I really like the idea of
DirectMemory and since direct-ring-cache is some kind of reinventing
the wheel.
Cheers
Noctarius (Chris)
Re: Additional Serializer and raw Buffer access
Posted by Olivier Lamy <ol...@apache.org>.
2012/9/29 Raffaele P. Guidi <ra...@gmail.com>:
> OK, deal, at least for me ;-) I propose you rename the packages, produce a
> patch for this and the new serializer module (should be simple enough
> starting from an existing one) and, in the meanwhile, apply for ASF
> membership. Is IP clearance needed? I guess yes.
> After this we will come up with a formal vote regarding Noctarius (is this
> your real name?!) allowance in the project team.
+1 from me too.
IMHO ip clearance is not needed. We will just receive a patch from a
contributor (all the code has been done by him AFICS in the git repo).
If that's not the case ip clearance is needed.
At least as it's a huge patch we need a cla from him.
@Noctarius see http://www.apache.org/licenses/ and in section
"Contributor License Agreements" the link to cla from to send to
secretary@.
Once this is done attach your patch to a jira issue
https://issues.apache.org/jira/browse/DIRECTMEMORY
>
> Good times are gonna come :-)
>
> Thanks,
> R
> Il giorno 29/set/2012 17:58, "Olivier Lamy" <ol...@apache.org> ha scritto:
>
>> 2012/9/29 Raffaele P. Guidi <ra...@gmail.com>:
>> > Well we already have a NIO ready interface allowing direct access to DMs
>> > managed bytebuffers but I think this is just half way to what could be
>> > achieved optimally blending serialization and memory allocation together.
>> >
>> > Lightning as a module is of course a good idea and it could easily evolve
>> > as a subproject (for the more experienced asf members: is it a feasible
>> > way?).
>> Nothing prevent to have
>> http://svn.apache.org/repos/asf/directmemory/lightning/trunk
>> with a package like: o.a.d.lightning
>> That will be a subproject.
>> >
>> > Ciao,
>> > R
>> > Il giorno 29/set/2012 17:44, "Noctarius" <me...@noctarius.com> ha scritto:
>> >
>> >> Hey guys,
>> >>
>> >> ok there's no lightning binary available since lightning wasn't
>> >> ready yet for releasing.
>> >>
>> >> For being the only developer it would be no problem to contribute
>> >> the sourcebase for DirectMemory but I'm not sure yet if it wouldn't
>> >> be better to seperate it to be available without using DirectMemory
>> >> itself. I started it as a serializer for cluster synchronization,
>> >> but it would be cool to contribute lightning as a subproject to
>> >> DirectMemory :-)
>> >>
>> >> About the second project I would love to see a public available
>> >> buffer API directly in DirectMemory so that project would be nearly
>> >> needless :-) The only difference I think is the allocation strategy
>> >> my implementation is using against the one DirectMemory has, but I'm
>> >> pretty sure the allocation is extensible ;-)
>> >>
>> >> Like I said, for both projects I'm the only dev so there would be no
>> >> IP problem. So if it's ok to you to not include lightning directly
>> >> in DM I would be glad to contribute to the Apache Foundation :-)
>> >>
>> >> Cheers Chris
>> >>
>> >>
>> >> Am 29.09.2012 16:53, schrieb Raffaele P. Guidi:
>> >> > Ok so it's up to noctarius - your move! ;-) Regarding the new unsafe
>> >> > storage: it's an opt-in feature that can be set with the fluent API
>> (and
>> >> > soon through the conference file).
>> >> >
>> >> > Ciao,
>> >> > R
>> >> > Il giorno 29/set/2012 16:45, "Olivier Lamy" <ol...@apache.org> ha
>> >> scritto:
>> >> >
>> >> >> 2012/9/29 Raffaele P. Guidi <ra...@gmail.com>:
>> >> >>>>> At least for the moment he can provide a patch to be integrated
>> in DM
>> >> >> :-)
>> >> >>>
>> >> >>> Sure, but as lightning is not in any public mvn repo should its
>> code be
>> >> >>> re-published in our svn? Or what?
>> >> >> @Apache we don't care about binaries, only sources are important ! (a
>> >> >> bit theorical for sure but that's it :-) ).
>> >> >> So if Noctarius was the only guy who participate in lightning. He can
>> >> >> just provide a patch we could integrate as a new dm module (note: the
>> >> >> patch must not contains any more copyright and all sources must have
>> >> >> ASF licenses).
>> >> >>
>> >> >> "Copyright 2012 the original author or authors." must be removed.
>> >> >> And BTW package must be changed :-) (com.github is not acceptable
>> @asf
>> >> >> :-) )(@Noctarius are you working for github ? :-) )
>> >> >>
>> >> >> And having him as a committer will be only a matter of voting (we
>> have
>> >> >> a great chair who take cares of administrative stuff :P )
>> >> >>
>> >> >> If some others have participated in the project, we must pass tru an
>> >> >> ip clearance mechanism
>> >> >> (http://incubator.apache.org/ip-clearance/index.html) and all
>> >> >> contributors to lightning must provide a cla. (It it's the case I can
>> >> >> help)
>> >> >>
>> >> >>>
>> >> >>>>> perso I'd like we avoid hard dependency on Unsafe as maybe some
>> use
>> >> >>> other jdks :-)
>> >> >>>
>> >> >>> Well, I believe Unsafe is supported by Sun JDK, OpenJDK, IBM JDK and
>> >> >>> JRockit - and I believe that it is more than enough. Also keep in
>> mind
>> >> >> that
>> >> >>> we already have an alternative Unsafe based memory storage - and
>> >> although
>> >> >>> it's not thoroughly tested for performance it dramaticaly simplifies
>> >> >> code,
>> >> >>> I have great expectations about it.
>> >> >> Me too :-). Yup definitely more simple and faster !
>> >> >> But we must provide a switch off configuration mechanism if some
>> users
>> >> >> don't want to use that (because Unsafe is just "Unsafe" :-) )
>> >> >> And sorry I didn't have a look yet at your changes with using Unsafe.
>> >> >>>
>> >> >>> Cheers,
>> >> >>> R
>> >> >>>
>> >> >>> On Sat, Sep 29, 2012 at 4:03 PM, Olivier Lamy <ol...@apache.org>
>> >> wrote:
>> >> >>>
>> >> >>>> 2012/9/29 Raffaele P. Guidi <ra...@gmail.com>:
>> >> >>>>> What do you think about:
>> >> >>>>> 1) implementing a lightning serialization module
>> >> >>>>> 2) creating a serializer that directly works on a directmemory
>> >> >> provider
>> >> >>>>> ByteBuffer or (maybe better) Unsafe based Pointer?
>> >> >>>>
>> >> >>>> Sounds good (perso I'd like we avoid hard dependency on Unsafe as
>> >> >>>> maybe some use other jdks :-) )
>> >> >>>>>
>> >> >>>>> Now I see lightning is apache licensed and this is fine but I
>> don't
>> >> >> think
>> >> >>>>> it is published in any public maven repo, am I right? We could
>> find a
>> >> >> way
>> >> >>>>> to deal with this; options vary from publishing lightning to the
>> free
>> >> >>>>> sonatype repo, joining the ASF (which is great anyhow!) and
>> >> >> republishing
>> >> >>>>> lightning code in DirectMemory becoming a commiter (which has to
>> >> >> undergo
>> >> >>>> a
>> >> >>>>> PMC vote).
>> >> >>>>
>> >> >>>> At least for the moment he can provide a patch to be integrated in
>> DM
>> >> >> :-)
>> >> >>>>
>> >> >>>>>
>> >> >>>>> I'd like to hear your and the team feelings on this.
>> >> >>>>>
>> >> >>>>> Thanks,
>> >> >>>>> Raffaele
>> >> >>>>>
>> >> >>>>> On Sat, Sep 29, 2012 at 3:27 PM, Noctarius <me...@noctarius.com>
>> wrote:
>> >> >>>>>
>> >> >>>>>> Hey Raffaele,
>> >> >>>>>>
>> >> >>>>>> that's quite similar to what I did at work. We're developing
>> Flash
>> >> >>>>>> online games and using a customized AMF serialization. To support
>> >> >>>>>> async writing of the clients event results I added serialization
>> of
>> >> >>>>>> the components / entities to the players zone calculation and
>> stored
>> >> >>>>>> the pre-serialized bytestream directly to the off-heap (using
>> >> >>>>>> direct-ring-cache implementation). When the client requests the
>> >> >>>>>> results (using long-polling) I just write the pre-serialized
>> data to
>> >> >>>>>> the right position to deserialize it by standard ways on Flash
>> side.
>> >> >>>>>>
>> >> >>>>>> So yeah an seriliaztion to off-heap algorithm would be fine. You
>> can
>> >> >>>>>> avoid using byte arrays and minimalize GC.
>> >> >>>>>>
>> >> >>>>>> Cheers
>> >> >>>>>> Chris
>> >> >>>>>>
>> >> >>>>>> Am 29.09.2012 15:02, schrieb Raffaele P. Guidi:
>> >> >>>>>>> Nice to hear back from you! Yes, I was thinking about creating a
>> >> >> new
>> >> >>>>>> memory
>> >> >>>>>>> storage implementation using Unsafe (and I did it, recently) and
>> >> >>>> also, as
>> >> >>>>>>> DirectMemory relies heavily on serialization (and supports many
>> of
>> >> >>>> them,
>> >> >>>>>>> protostuff, protobuf, msgpack and of course standard
>> >> >> serialization),
>> >> >>>>>> about
>> >> >>>>>>> creating a simple embedded serializer leveraging the same
>> >> >> techniques
>> >> >>>> you
>> >> >>>>>>> used (Unsafe and bytecode generation).
>> >> >>>>>>> The idea with embedding is avoiding to serialize in a byte array
>> >> >> and
>> >> >>>> then
>> >> >>>>>>> moving the byte array to off-heap memory (via Unsafe or
>> >> >> ByteBuffers),
>> >> >>>> and
>> >> >>>>>>> serializing directly into a "managed" off-heap buffer, thus
>> further
>> >> >>>>>>> optimizing heap utilization (less GC).
>> >> >>>>>>>
>> >> >>>>>>> What do you think about it? Does it make any sense to you?
>> >> >>>>>>>
>> >> >>>>>>> Ciao,
>> >> >>>>>>> R
>> >> >>>>>>>
>> >> >>>>>>> On Sat, Sep 29, 2012 at 2:40 PM, Noctarius <me...@noctarius.com>
>> >> >> wrote:
>> >> >>>>>>>
>> >> >>>>>>>> Hey guys,
>> >> >>>>>>>>
>> >> >>>>>>>> Raffaele found out about a project of mine (Lightning) a few
>> weeks
>> >> >>>>>>>> ago. Lightning is a heavy Unsafe and Bytecode generation using
>> >> >>>>>>>> Serializer implementation. He told me that he was interested in
>> >> >>>>>>>> adding something similar to DirectMemory and I would be glad to
>> >> >> help
>> >> >>>>>>>> out in this.
>> >> >>>>>>>>
>> >> >>>>>>>> Another project I started a few days ago, since it was needed
>> for
>> >> >>>>>>>> work is DirectRingCache. The name not really meets to actual
>> >> >>>>>>>> implementation since it's not yet a ring buffer using cache. I
>> >> >> used
>> >> >>>>>>>> this for a pre-serialization simple bytestream cache with
>> >> >>>>>>>> self-growing buffers. It could be nice to have DirectMemory
>> having
>> >> >>>>>>>> raw "buffers" to write to or to read from.
>> >> >>>>>>>>
>> >> >>>>>>>> Here are the links from the projects:
>> >> >>>>>>>> https://github.com/noctarius/Lightning
>> >> >>>>>>>> https://github.com/noctarius/direct-ring-cache
>> >> >>>>>>>>
>> >> >>>>>>>> It would be nice to help out since I really like the idea of
>> >> >>>>>>>> DirectMemory and since direct-ring-cache is some kind of
>> >> >> reinventing
>> >> >>>>>>>> the wheel.
>> >> >>>>>>>>
>> >> >>>>>>>> Cheers
>> >> >>>>>>>> Noctarius (Chris)
>> >> >>>>>>>>
>> >> >>>>>>>
>> >> >>>>>>
>> >> >>>>>>
>> >> >>>>>>
>> >> >>>>
>> >> >>>>
>> >> >>>>
>> >> >>>> --
>> >> >>>> Olivier Lamy
>> >> >>>> Talend: http://coders.talend.com
>> >> >>>> http://twitter.com/olamy | http://linkedin.com/in/olamy
>> >> >>>>
>> >> >>
>> >> >>
>> >> >>
>> >> >> --
>> >> >> Olivier Lamy
>> >> >> Talend: http://coders.talend.com
>> >> >> http://twitter.com/olamy | http://linkedin.com/in/olamy
>> >> >>
>> >> >
>> >>
>>
>>
>>
>> --
>> Olivier Lamy
>> Talend: http://coders.talend.com
>> http://twitter.com/olamy | http://linkedin.com/in/olamy
>>
--
Olivier Lamy
Talend: http://coders.talend.com
http://twitter.com/olamy | http://linkedin.com/in/olamy
Re: Additional Serializer and raw Buffer access
Posted by "Raffaele P. Guidi" <ra...@gmail.com>.
Cool. Hope to see those things in trunk and to hear from you soon.
Cheers,
R
Il giorno 29/set/2012 22:10, "Roman Levenstein" <ro...@gmail.com> ha
scritto:
> Hi,
>
>
>
> On Sat, Sep 29, 2012 at 9:56 PM, Raffaele P. Guidi
> <ra...@gmail.com> wrote:
> > Really nice ideas - we already have an Unsafe based storage (it is rather
> > new) and the idea was exactly to combine the benefits of it with the ones
> > of unsafe serialization. Said that keep in mind that classic DirectMemory
> > is in any case way faster than just using ByteBuffers because it
> > pre-allocates large chunks of memory and then "slices" them for single,
> > smaller items.
>
> Yes, pre-allocation is good. But you should really see the performance
> benefits of writing/reading off-heap memory using Unsafe :-)
> It is way faster, especially for arrays of primitive types. The link I
> mentioned contained some preliminary figures based on my own
> benchmarks.
> Therefore I think even Lightning (or any other serialization lib)
> could benefit from using Unsafe-based off-heap Input/Output streams.
> BTW, my streams support both DirectBuffer-based and
> Unsafe.allocateMemory-based (i.e. pre-allocated) off-heap memory
> access. Therefore, it is a nice fit with DirectMemory.
>
> > And - a Kryo serializer is a rather good idea. Do you feel like working
> on
> > it? Serializers are actually one of the easiest thing to contribute on in
> > DM - and they add immediate value as each one is probably a better fit
> than
> > others in different scenarios.
>
> Yes, I think I could work on it. But first, I'd like to land these
> improvements in Kryo's trunk so that I don't need to work on a branch
> or a fork of Kryo.
>
> Ciao,
> Roman
>
> > Il giorno 29/set/2012 21:16, "Roman Levenstein" <ro...@gmail.com> ha
> > scritto:
> >
> >> Hi,
> >>
> >> I'm just wondering if lightning uses Unsafe only for serializers (i.e.
> >> for reading/writing fields of objects) or also for the input/output
> >> streams used to read/write to/from off-heap memory? Using
> >> DirectBuffers "as is" for accessing off-heap memory is actually rather
> >> slow.
> >>
> >> Let me share some experiences with you:
> >> I have implemented a similar feature for Kryo. It is not in the trunk
> >> yet, as it is being reviewed by Nate, the author of Kryo.
> >> One of the things that I noticed is that using Unsafe only for
> >> serializers is nice, but does not provide much improvement.
> >> At the same time, using Unsafe for input/output streams and
> >> reading/writing directly from/into off-heap memory provides
> >> significant performance benefits.
> >> My patch for Kryo implements both features using Unsafe. If you want
> >> to see more details about the implementation and findings, please have
> >> a look at:
> >> http://code.google.com/p/kryo/issues/detail?id=75
> >>
> >> I hope it will land in Kryo trunk soon. Once it is there, I think it
> >> could be interesting to add a Kryo-based serialization to
> >> DirectMemory. What do you think?
> >> But independent of this, feel free to take any ideas or even the
> >> implementation from the patch. The off-heap memory streams
> >> implementation is rather self-contained and could be reused with other
> >> frameworks besides Kryo.
> >>
> >> Regards,
> >> Roman
> >>
> >> On Sat, Sep 29, 2012 at 8:46 PM, Noctarius <me...@noctarius.com> wrote:
> >> > Actually all dependencies should be AL2 or BSD licensed :-)
> >> >
> >> > Am 29.09.2012 20:42, schrieb Raffaele P. Guidi:
> >> >>>> Hehe well that really sounds like a nice bunch of people.
> >> >>
> >> >> Indeed they are (I'm a newbie as well and try to do my best)
> >> >>
> >> >>>> If lightning will be a sub-part (sub-project) of DM, do I
> >> >>>> need to write
> >> >> an project purposal?
> >> >>
> >> >> Nope, not needed for a sub-project
> >> >>
> >> >>>> Do I need to make any changes to the pom.xml like adding a
> >> >> special parent pom or anything like that?
> >> >>
> >> >> Not for the serializer - just have to take a look at project
> >> >> dependencies - or, better, at their licenses - are they
> >> >> compatible with the ASL 2.0? i.e. a GPL'd library is not a good
> >> >> fit and should be replaced with an apache licensed (or BSD, or
> >> >> MIT...) one if possible. For the integration module is a
> >> >> separate story - you should start off copying one of the other
> >> >> serializers and reusing the same pom and directory structure.
> >> >>
> >> >> Pleased to meet you, Chris :)
> >> >>
> >> >> Ciao, R
> >> >>
> >> >> On Sat, Sep 29, 2012 at 8:24 PM, Noctarius <me...@noctarius.com>
> >> >> wrote:
> >> >>
> >> >>> Hehe well that really sounds like a nice bunch of people.
> >> >>>
> >> >>> Ok to be true I couldn't wait until tomorrow and started
> >> >>> already reading the links. From what I was reading: If
> >> >>> lightning will be a sub-part (sub-project) of DM, do I need
> >> >>> to write an project purposal?
> >> >>>
> >> >>> Do I need to make any changes to the pom.xml like adding a
> >> >>> special parent pom or anything like that?
> >> >>>
> >> >>> In general: there are a lot things to know :-)
> >> >>>
> >> >>> Am 29.09.2012 19:59, schrieb Raffaele P. Guidi:
> >> >>>> Negative part of ASF membership? You get together with a
> >> >>>> lot of geeky, talented people with a fixation for software
> >> >>>> and open source. Oh wait but this is actually nice! :-D Il
> >> >>>> giorno 29/set/2012 19:05, "Olivier Lamy" <ol...@apache.org>
> >> >>>> ha
> >> >>> scritto:
> >> >>>>
> >> >>>>> 2012/9/29 Noctarius <me...@noctarius.com>:
> >> >>>>>> Thanks Olivier for carify, I'll take a look in it
> >> >>>>>> tomorrow but there's just one question left (for now
> >> >>>>>> ;)): What is that vote for becoming a committer? What
> >> >>>>>> if the vote will be negative?
> >> >>>>> The vote is on private list (pmc list for privacy reasons
> >> >>>>> and possible negative stuff being on public lists)
> >> >>>>>> Until now I just used Apache stuff, was never
> >> >>>>>> interested in being part of it so I guess it can be
> >> >>>>>> negative for any reason, can't it?
> >> >>>>> I don't see why it could be negative but suspens ....
> >> >>>>> :-)
> >> >>>>>>
> >> >>>>>> Am 29.09.2012 18:56, schrieb Olivier Lamy:
> >> >>>>>>> 2012/9/29 Noctarius <me...@noctarius.com>:
> >> >>>>>>>> Nope my real name is Christoph Engelbert, but
> >> >>>>>>>> Noctarius is the all time nick :)
> >> >>>>>>>>
> >> >>>>>>>> Renaming the package should be no problem, is it
> >> >>>>>>>> "org.apache.directmemory.lightning" or what would
> >> >>>>>>>> it be?
> >> >>>>>>> fine for me
> >> >>>>>>>> Then there needs to be a change in the license
> >> >>>>>>>> header as Olivier mentioned, that means just remove
> >> >>>>>>>> the first sentence or is there anything more to do
> >> >>>>>>>> (maybe it's easiest thing to just copy the header
> >> >>>>>>>> from DM file ;))?
> >> >>>>>>> yup use same header as DM
> >> >>>>>>>>
> >> >>>>>>>> The CLA is just a form to clarify that the source
> >> >>>>>>>> can be contributed to the Apache Foundation?
> >> >>>>>>> yup correct.
> >> >>>>>>>>
> >> >>>>>>>> The final step will be attaching the patch in form
> >> >>>>>>>> of a huge diff file?
> >> >>>>>>> yes
> >> >>>>>>>>
> >> >>>>>>>> And what is the way to apply for a membership?
> >> >>>>>>>> Never thought about how to do that.
> >> >>>>>>> Read here
> >> >>>>>>> http://apache.org/foundation/how-it-works.html and
> >> >>>>>>> here http://apache.org/foundation/getinvolved.html .
> >> >>>>>>> And feel free to ask any questions :-)
> >> >>>>>>>>
> >> >>>>>>>> Chris
> >> >>>>>>>>
> >> >>>>>>>> Am 29.09.2012 18:23, schrieb Raffaele P. Guidi:
> >> >>>>>>>>> OK, deal, at least for me ;-) I propose you
> >> >>>>>>>>> rename the packages, produce a patch for this and
> >> >>>>>>>>> the new serializer module (should be simple
> >> >>>>>>>>> enough starting from an existing one) and, in the
> >> >>>>>>>>> meanwhile, apply for ASF membership. Is IP
> >> >>>>>>>>> clearance needed? I guess yes. After this we will
> >> >>>>>>>>> come up with a formal vote regarding Noctarius
> >> >>>>>>>>> (is this your real name?!) allowance in the
> >> >>>>>>>>> project team.
> >> >>>>>>>>>
> >> >>>>>>>>> Good times are gonna come :-)
> >> >>>>>>>>>
> >> >>>>>>>>> Thanks, R Il giorno 29/set/2012 17:58, "Olivier
> >> >>>>>>>>> Lamy" <ol...@apache.org> ha scritto:
> >> >>>>>>>>>
> >> >>>>>>>>>> 2012/9/29 Raffaele P. Guidi
> >> >>>>>>>>>> <ra...@gmail.com>:
> >> >>>>>>>>>>> Well we already have a NIO ready interface
> >> >>>>>>>>>>> allowing direct access to DMs managed
> >> >>>>>>>>>>> bytebuffers but I think this is just half way
> >> >>>>>>>>>>> to what could be achieved optimally blending
> >> >>>>>>>>>>> serialization and memory allocation
> >> >>>>>>>>>>> together.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> Lightning as a module is of course a good
> >> >>>>>>>>>>> idea and it could easily evolve as a
> >> >>>>>>>>>>> subproject (for the more experienced asf
> >> >>>>>>>>>>> members: is it a feasible way?).
> >> >>>>>>>>>> Nothing prevent to have
> >> >>>>>>>>>> http://svn.apache.org/repos/asf/directmemory/lightning/trunk
> >> >>>>>>>>>>
> >> >>>>>>>>>>
> >> >>>>>>
> >> >>>>>>>>>>
> >> > with a package like: o.a.d.lightning That will be a
> >> >>>>>>>>>> subproject.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> Ciao, R Il giorno 29/set/2012 17:44,
> >> >>>>>>>>>>> "Noctarius" <me...@noctarius.com> ha scritto:
> >> >>>>>>>>>>>
> >> >>>>>>>>>>>> Hey guys,
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> ok there's no lightning binary available
> >> >>>>>>>>>>>> since lightning wasn't ready yet for
> >> >>>>>>>>>>>> releasing.
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> For being the only developer it would be no
> >> >>>>>>>>>>>> problem to contribute the sourcebase for
> >> >>>>>>>>>>>> DirectMemory but I'm not sure yet if it
> >> >>>>>>>>>>>> wouldn't be better to seperate it to be
> >> >>>>>>>>>>>> available without using DirectMemory
> >> >>>>>>>>>>>> itself. I started it as a serializer for
> >> >>>>>>>>>>>> cluster synchronization, but it would be
> >> >>>>>>>>>>>> cool to contribute lightning as a
> >> >>>>>>>>>>>> subproject to DirectMemory :-)
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> About the second project I would love to
> >> >>>>>>>>>>>> see a public available buffer API directly
> >> >>>>>>>>>>>> in DirectMemory so that project would be
> >> >>>>>>>>>>>> nearly needless :-) The only difference I
> >> >>>>>>>>>>>> think is the allocation strategy my
> >> >>>>>>>>>>>> implementation is using against the one
> >> >>>>>>>>>>>> DirectMemory has, but I'm pretty sure the
> >> >>>>>>>>>>>> allocation is extensible ;-)
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> Like I said, for both projects I'm the only
> >> >>>>>>>>>>>> dev so there would be no IP problem. So if
> >> >>>>>>>>>>>> it's ok to you to not include lightning
> >> >>>>>>>>>>>> directly in DM I would be glad to
> >> >>>>>>>>>>>> contribute to the Apache Foundation :-)
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> Cheers Chris
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> Am 29.09.2012 16:53, schrieb Raffaele P.
> >> >>>>>>>>>>>> Guidi:
> >> >>>>>>>>>>>>> Ok so it's up to noctarius - your move!
> >> >>>>>>>>>>>>> ;-) Regarding the new unsafe storage:
> >> >>>>>>>>>>>>> it's an opt-in feature that can be set
> >> >>>>>>>>>>>>> with the fluent API
> >> >>>>>>>>>> (and
> >> >>>>>>>>>>>>> soon through the conference file).
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>> Ciao, R Il giorno 29/set/2012 16:45,
> >> >>>>>>>>>>>>> "Olivier Lamy" <ol...@apache.org> ha
> >> >>>>>>>>>>>> scritto:
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> 2012/9/29 Raffaele P. Guidi
> >> >>>>>>>>>>>>>> <ra...@gmail.com>:
> >> >>>>>>>>>>>>>>>>> At least for the moment he can
> >> >>>>>>>>>>>>>>>>> provide a patch to be integrated
> >> >>>>>>>>>> in DM
> >> >>>>>>>>>>>>>> :-)
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> Sure, but as lightning is not in any
> >> >>>>>>>>>>>>>>> public mvn repo should its
> >> >>>>>>>>>> code be
> >> >>>>>>>>>>>>>>> re-published in our svn? Or what?
> >> >>>>>>>>>>>>>> @Apache we don't care about binaries,
> >> >>>>>>>>>>>>>> only sources are important ! (a bit
> >> >>>>>>>>>>>>>> theorical for sure but that's it :-) ).
> >> >>>>>>>>>>>>>> So if Noctarius was the only guy who
> >> >>>>>>>>>>>>>> participate in lightning. He can just
> >> >>>>>>>>>>>>>> provide a patch we could integrate as a
> >> >>>>>>>>>>>>>> new dm module (note: the patch must not
> >> >>>>>>>>>>>>>> contains any more copyright and all
> >> >>>>>>>>>>>>>> sources must have ASF licenses).
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> "Copyright 2012 the original author or
> >> >>>>>>>>>>>>>> authors." must be removed. And BTW
> >> >>>>>>>>>>>>>> package must be changed :-) (com.github
> >> >>>>>>>>>>>>>> is not acceptable
> >> >>>>>>>>>> @asf
> >> >>>>>>>>>>>>>> :-) )(@Noctarius are you working for
> >> >>>>>>>>>>>>>> github ? :-) )
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> And having him as a committer will be
> >> >>>>>>>>>>>>>> only a matter of voting (we
> >> >>>>>>>>>> have
> >> >>>>>>>>>>>>>> a great chair who take cares of
> >> >>>>>>>>>>>>>> administrative stuff :P )
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> If some others have participated in the
> >> >>>>>>>>>>>>>> project, we must pass tru an ip
> >> >>>>>>>>>>>>>> clearance mechanism
> >> >>>>>>>>>>>>>> (http://incubator.apache.org/ip-clearance/index.html)
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>
> >> >>>>>>
> >> >>>>>>>>>>>>>>
> >> > and all contributors to lightning must provide a cla.
> >> >>>>>>>>>>>>>> (It it's the case I can help)
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>> perso I'd like we avoid hard
> >> >>>>>>>>>>>>>>>>> dependency on Unsafe as maybe
> >> >>>>>>>>>>>>>>>>> some
> >> >>>>>>>>>> use
> >> >>>>>>>>>>>>>>> other jdks :-)
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> Well, I believe Unsafe is supported
> >> >>>>>>>>>>>>>>> by Sun JDK, OpenJDK, IBM JDK and
> >> >>>>>>>>>>>>>>> JRockit - and I believe that it is
> >> >>>>>>>>>>>>>>> more than enough. Also keep in
> >> >>>>>>>>>> mind
> >> >>>>>>>>>>>>>> that
> >> >>>>>>>>>>>>>>> we already have an alternative Unsafe
> >> >>>>>>>>>>>>>>> based memory storage - and
> >> >>>>>>>>>>>> although
> >> >>>>>>>>>>>>>>> it's not thoroughly tested for
> >> >>>>>>>>>>>>>>> performance it dramaticaly
> >> >>>>>>>>>>>>>>> simplifies
> >> >>>>>>>>>>>>>> code,
> >> >>>>>>>>>>>>>>> I have great expectations about it.
> >> >>>>>>>>>>>>>> Me too :-). Yup definitely more simple
> >> >>>>>>>>>>>>>> and faster ! But we must provide a
> >> >>>>>>>>>>>>>> switch off configuration mechanism if
> >> >>>>>>>>>>>>>> some
> >> >>>>>>>>>> users
> >> >>>>>>>>>>>>>> don't want to use that (because Unsafe
> >> >>>>>>>>>>>>>> is just "Unsafe" :-) ) And sorry I
> >> >>>>>>>>>>>>>> didn't have a look yet at your changes
> >> >>>>>>>>>>>>>> with using Unsafe.
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> Cheers, R
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> On Sat, Sep 29, 2012 at 4:03 PM,
> >> >>>>>>>>>>>>>>> Olivier Lamy <ol...@apache.org>
> >> >>>>>>>>>>>> wrote:
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> 2012/9/29 Raffaele P. Guidi
> >> >>>>>>>>>>>>>>>> <ra...@gmail.com>:
> >> >>>>>>>>>>>>>>>>> What do you think about: 1)
> >> >>>>>>>>>>>>>>>>> implementing a lightning
> >> >>>>>>>>>>>>>>>>> serialization module 2) creating
> >> >>>>>>>>>>>>>>>>> a serializer that directly works
> >> >>>>>>>>>>>>>>>>> on a directmemory
> >> >>>>>>>>>>>>>> provider
> >> >>>>>>>>>>>>>>>>> ByteBuffer or (maybe better)
> >> >>>>>>>>>>>>>>>>> Unsafe based Pointer?
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> Sounds good (perso I'd like we
> >> >>>>>>>>>>>>>>>> avoid hard dependency on Unsafe as
> >> >>>>>>>>>>>>>>>> maybe some use other jdks :-) )
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>> Now I see lightning is apache
> >> >>>>>>>>>>>>>>>>> licensed and this is fine but I
> >> >>>>>>>>>> don't
> >> >>>>>>>>>>>>>> think
> >> >>>>>>>>>>>>>>>>> it is published in any public
> >> >>>>>>>>>>>>>>>>> maven repo, am I right? We could
> >> >>>>>>>>>> find a
> >> >>>>>>>>>>>>>> way
> >> >>>>>>>>>>>>>>>>> to deal with this; options vary
> >> >>>>>>>>>>>>>>>>> from publishing lightning to the
> >> >>>>>>>>>> free
> >> >>>>>>>>>>>>>>>>> sonatype repo, joining the ASF
> >> >>>>>>>>>>>>>>>>> (which is great anyhow!) and
> >> >>>>>>>>>>>>>> republishing
> >> >>>>>>>>>>>>>>>>> lightning code in DirectMemory
> >> >>>>>>>>>>>>>>>>> becoming a commiter (which has
> >> >>>>>>>>>>>>>>>>> to
> >> >>>>>>>>>>>>>> undergo
> >> >>>>>>>>>>>>>>>> a
> >> >>>>>>>>>>>>>>>>> PMC vote).
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> At least for the moment he can
> >> >>>>>>>>>>>>>>>> provide a patch to be integrated
> >> >>>>>>>>>>>>>>>> in
> >> >>>>>>>>>> DM
> >> >>>>>>>>>>>>>> :-)
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>> I'd like to hear your and the
> >> >>>>>>>>>>>>>>>>> team feelings on this.
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>> Thanks, Raffaele
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>> On Sat, Sep 29, 2012 at 3:27 PM,
> >> >>>>>>>>>>>>>>>>> Noctarius <me...@noctarius.com>
> >> >>>>>>>>>> wrote:
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>> Hey Raffaele,
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>> that's quite similar to what I
> >> >>>>>>>>>>>>>>>>>> did at work. We're developing
> >> >>>>>>>>>> Flash
> >> >>>>>>>>>>>>>>>>>> online games and using a
> >> >>>>>>>>>>>>>>>>>> customized AMF serialization.
> >> >>>>>>>>>>>>>>>>>> To support async writing of the
> >> >>>>>>>>>>>>>>>>>> clients event results I added
> >> >>>>>>>>>>>>>>>>>> serialization
> >> >>>>>>>>>> of
> >> >>>>>>>>>>>>>>>>>> the components / entities to
> >> >>>>>>>>>>>>>>>>>> the players zone calculation
> >> >>>>>>>>>>>>>>>>>> and
> >> >>>>>>>>>> stored
> >> >>>>>>>>>>>>>>>>>> the pre-serialized bytestream
> >> >>>>>>>>>>>>>>>>>> directly to the off-heap (using
> >> >>>>>>>>>>>>>>>>>> direct-ring-cache
> >> >>>>>>>>>>>>>>>>>> implementation). When the
> >> >>>>>>>>>>>>>>>>>> client requests the results
> >> >>>>>>>>>>>>>>>>>> (using long-polling) I just
> >> >>>>>>>>>>>>>>>>>> write the pre-serialized
> >> >>>>>>>>>> data to
> >> >>>>>>>>>>>>>>>>>> the right position to
> >> >>>>>>>>>>>>>>>>>> deserialize it by standard ways
> >> >>>>>>>>>>>>>>>>>> on Flash
> >> >>>>>>>>>> side.
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>> So yeah an seriliaztion to
> >> >>>>>>>>>>>>>>>>>> off-heap algorithm would be
> >> >>>>>>>>>>>>>>>>>> fine. You
> >> >>>>>>>>>> can
> >> >>>>>>>>>>>>>>>>>> avoid using byte arrays and
> >> >>>>>>>>>>>>>>>>>> minimalize GC.
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>> Cheers Chris
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>> Am 29.09.2012 15:02, schrieb
> >> >>>>>>>>>>>>>>>>>> Raffaele P. Guidi:
> >> >>>>>>>>>>>>>>>>>>> Nice to hear back from you!
> >> >>>>>>>>>>>>>>>>>>> Yes, I was thinking about
> >> >>>>>>>>>>>>>>>>>>> creating a
> >> >>>>>>>>>>>>>> new
> >> >>>>>>>>>>>>>>>>>> memory
> >> >>>>>>>>>>>>>>>>>>> storage implementation using
> >> >>>>>>>>>>>>>>>>>>> Unsafe (and I did it,
> >> >>>>>>>>>>>>>>>>>>> recently) and
> >> >>>>>>>>>>>>>>>> also, as
> >> >>>>>>>>>>>>>>>>>>> DirectMemory relies heavily
> >> >>>>>>>>>>>>>>>>>>> on serialization (and
> >> >>>>>>>>>>>>>>>>>>> supports many
> >> >>>>>>>>>> of
> >> >>>>>>>>>>>>>>>> them,
> >> >>>>>>>>>>>>>>>>>>> protostuff, protobuf, msgpack
> >> >>>>>>>>>>>>>>>>>>> and of course standard
> >> >>>>>>>>>>>>>> serialization),
> >> >>>>>>>>>>>>>>>>>> about
> >> >>>>>>>>>>>>>>>>>>> creating a simple embedded
> >> >>>>>>>>>>>>>>>>>>> serializer leveraging the
> >> >>>>>>>>>>>>>>>>>>> same
> >> >>>>>>>>>>>>>> techniques
> >> >>>>>>>>>>>>>>>> you
> >> >>>>>>>>>>>>>>>>>>> used (Unsafe and bytecode
> >> >>>>>>>>>>>>>>>>>>> generation). The idea with
> >> >>>>>>>>>>>>>>>>>>> embedding is avoiding to
> >> >>>>>>>>>>>>>>>>>>> serialize in a byte array
> >> >>>>>>>>>>>>>> and
> >> >>>>>>>>>>>>>>>> then
> >> >>>>>>>>>>>>>>>>>>> moving the byte array to
> >> >>>>>>>>>>>>>>>>>>> off-heap memory (via Unsafe
> >> >>>>>>>>>>>>>>>>>>> or
> >> >>>>>>>>>>>>>> ByteBuffers),
> >> >>>>>>>>>>>>>>>> and
> >> >>>>>>>>>>>>>>>>>>> serializing directly into a
> >> >>>>>>>>>>>>>>>>>>> "managed" off-heap buffer,
> >> >>>>>>>>>>>>>>>>>>> thus
> >> >>>>>>>>>> further
> >> >>>>>>>>>>>>>>>>>>> optimizing heap utilization
> >> >>>>>>>>>>>>>>>>>>> (less GC).
> >> >>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>> What do you think about it?
> >> >>>>>>>>>>>>>>>>>>> Does it make any sense to
> >> >>>>>>>>>>>>>>>>>>> you?
> >> >>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>> Ciao, R
> >> >>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>> On Sat, Sep 29, 2012 at 2:40
> >> >>>>>>>>>>>>>>>>>>> PM, Noctarius
> >> >>>>>>>>>>>>>>>>>>> <me...@noctarius.com>
> >> >>>>>>>>>>>>>> wrote:
> >> >>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>> Hey guys,
> >> >>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>> Raffaele found out about a
> >> >>>>>>>>>>>>>>>>>>>> project of mine (Lightning)
> >> >>>>>>>>>>>>>>>>>>>> a few
> >> >>>>>>>>>> weeks
> >> >>>>>>>>>>>>>>>>>>>> ago. Lightning is a heavy
> >> >>>>>>>>>>>>>>>>>>>> Unsafe and Bytecode
> >> >>>>>>>>>>>>>>>>>>>> generation using
> >> >>>>>>>>>>>>>>>>>>>> Serializer implementation.
> >> >>>>>>>>>>>>>>>>>>>> He told me that he was
> >> >>>>>>>>>>>>>>>>>>>> interested in adding
> >> >>>>>>>>>>>>>>>>>>>> something similar to
> >> >>>>>>>>>>>>>>>>>>>> DirectMemory and I would be
> >> >>>>>>>>>>>>>>>>>>>> glad to
> >> >>>>>>>>>>>>>> help
> >> >>>>>>>>>>>>>>>>>>>> out in this.
> >> >>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>> Another project I started a
> >> >>>>>>>>>>>>>>>>>>>> few days ago, since it was
> >> >>>>>>>>>>>>>>>>>>>> needed
> >> >>>>>>>>>> for
> >> >>>>>>>>>>>>>>>>>>>> work is DirectRingCache.
> >> >>>>>>>>>>>>>>>>>>>> The name not really meets
> >> >>>>>>>>>>>>>>>>>>>> to actual implementation
> >> >>>>>>>>>>>>>>>>>>>> since it's not yet a ring
> >> >>>>>>>>>>>>>>>>>>>> buffer using cache. I
> >> >>>>>>>>>>>>>> used
> >> >>>>>>>>>>>>>>>>>>>> this for a
> >> >>>>>>>>>>>>>>>>>>>> pre-serialization simple
> >> >>>>>>>>>>>>>>>>>>>> bytestream cache with
> >> >>>>>>>>>>>>>>>>>>>> self-growing buffers. It
> >> >>>>>>>>>>>>>>>>>>>> could be nice to have
> >> >>>>>>>>>>>>>>>>>>>> DirectMemory
> >> >>>>>>>>>> having
> >> >>>>>>>>>>>>>>>>>>>> raw "buffers" to write to
> >> >>>>>>>>>>>>>>>>>>>> or to read from.
> >> >>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>> Here are the links from
> >> >>>>>>>>>>>>>>>>>>>> the projects:
> >> >>>>>>>>>>>>>>>>>>>> https://github.com/noctarius/Lightning
> >> >>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>
> >> >>>>>>
> >> >>>>>>>>>>>>>>>>>>>>
> >> > https://github.com/noctarius/direct-ring-cache
> >> >>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>> It would be nice to help out since I really like
> >> >>>>>>>> the idea of
> >> >>>>>>>>>>>>>>>>>>>> DirectMemory and since
> >> >>>>>>>>>>>>>>>>>>>> direct-ring-cache is some
> >> >>>>>>>>>>>>>>>>>>>> kind of
> >> >>>>>>>>>>>>>> reinventing
> >> >>>>>>>>>>>>>>>>>>>> the wheel.
> >> >>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>> Cheers Noctarius (Chris)
> >> >>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> -- Olivier Lamy Talend:
> >> >>>>>>>>>>>>>>>> http://coders.talend.com
> >> >>>>>>>>>>>>>>>> http://twitter.com/olamy |
> >> >>>>>>>>>>>>>>>> http://linkedin.com/in/olamy
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> -- Olivier Lamy Talend:
> >> >>>>>>>>>>>>>> http://coders.talend.com
> >> >>>>>>>>>>>>>> http://twitter.com/olamy |
> >> >>>>>>>>>>>>>> http://linkedin.com/in/olamy
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>
> >> >>>>>>>>>>
> >> >>>>>>>>>>
> >> >>>>>>>>>> -- Olivier Lamy Talend:
> >> >>>>>>>>>> http://coders.talend.com
> >> >>>>>>>>>> http://twitter.com/olamy |
> >> >>>>>>>>>> http://linkedin.com/in/olamy
> >> >>>>>>>>>>
> >> >>>>>>>>>
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> -- Olivier Lamy Talend: http://coders.talend.com
> >> >>>>> http://twitter.com/olamy | http://linkedin.com/in/olamy
> >> >>>>>
> >> >>>>
> >> >>>
> >> >>
> >> >
> >>
>
Re: Additional Serializer and raw Buffer access
Posted by Roman Levenstein <ro...@gmail.com>.
Hi,
On Sat, Sep 29, 2012 at 9:56 PM, Raffaele P. Guidi
<ra...@gmail.com> wrote:
> Really nice ideas - we already have an Unsafe based storage (it is rather
> new) and the idea was exactly to combine the benefits of it with the ones
> of unsafe serialization. Said that keep in mind that classic DirectMemory
> is in any case way faster than just using ByteBuffers because it
> pre-allocates large chunks of memory and then "slices" them for single,
> smaller items.
Yes, pre-allocation is good. But you should really see the performance
benefits of writing/reading off-heap memory using Unsafe :-)
It is way faster, especially for arrays of primitive types. The link I
mentioned contained some preliminary figures based on my own
benchmarks.
Therefore I think even Lightning (or any other serialization lib)
could benefit from using Unsafe-based off-heap Input/Output streams.
BTW, my streams support both DirectBuffer-based and
Unsafe.allocateMemory-based (i.e. pre-allocated) off-heap memory
access. Therefore, it is a nice fit with DirectMemory.
> And - a Kryo serializer is a rather good idea. Do you feel like working on
> it? Serializers are actually one of the easiest thing to contribute on in
> DM - and they add immediate value as each one is probably a better fit than
> others in different scenarios.
Yes, I think I could work on it. But first, I'd like to land these
improvements in Kryo's trunk so that I don't need to work on a branch
or a fork of Kryo.
Ciao,
Roman
> Il giorno 29/set/2012 21:16, "Roman Levenstein" <ro...@gmail.com> ha
> scritto:
>
>> Hi,
>>
>> I'm just wondering if lightning uses Unsafe only for serializers (i.e.
>> for reading/writing fields of objects) or also for the input/output
>> streams used to read/write to/from off-heap memory? Using
>> DirectBuffers "as is" for accessing off-heap memory is actually rather
>> slow.
>>
>> Let me share some experiences with you:
>> I have implemented a similar feature for Kryo. It is not in the trunk
>> yet, as it is being reviewed by Nate, the author of Kryo.
>> One of the things that I noticed is that using Unsafe only for
>> serializers is nice, but does not provide much improvement.
>> At the same time, using Unsafe for input/output streams and
>> reading/writing directly from/into off-heap memory provides
>> significant performance benefits.
>> My patch for Kryo implements both features using Unsafe. If you want
>> to see more details about the implementation and findings, please have
>> a look at:
>> http://code.google.com/p/kryo/issues/detail?id=75
>>
>> I hope it will land in Kryo trunk soon. Once it is there, I think it
>> could be interesting to add a Kryo-based serialization to
>> DirectMemory. What do you think?
>> But independent of this, feel free to take any ideas or even the
>> implementation from the patch. The off-heap memory streams
>> implementation is rather self-contained and could be reused with other
>> frameworks besides Kryo.
>>
>> Regards,
>> Roman
>>
>> On Sat, Sep 29, 2012 at 8:46 PM, Noctarius <me...@noctarius.com> wrote:
>> > Actually all dependencies should be AL2 or BSD licensed :-)
>> >
>> > Am 29.09.2012 20:42, schrieb Raffaele P. Guidi:
>> >>>> Hehe well that really sounds like a nice bunch of people.
>> >>
>> >> Indeed they are (I'm a newbie as well and try to do my best)
>> >>
>> >>>> If lightning will be a sub-part (sub-project) of DM, do I
>> >>>> need to write
>> >> an project purposal?
>> >>
>> >> Nope, not needed for a sub-project
>> >>
>> >>>> Do I need to make any changes to the pom.xml like adding a
>> >> special parent pom or anything like that?
>> >>
>> >> Not for the serializer - just have to take a look at project
>> >> dependencies - or, better, at their licenses - are they
>> >> compatible with the ASL 2.0? i.e. a GPL'd library is not a good
>> >> fit and should be replaced with an apache licensed (or BSD, or
>> >> MIT...) one if possible. For the integration module is a
>> >> separate story - you should start off copying one of the other
>> >> serializers and reusing the same pom and directory structure.
>> >>
>> >> Pleased to meet you, Chris :)
>> >>
>> >> Ciao, R
>> >>
>> >> On Sat, Sep 29, 2012 at 8:24 PM, Noctarius <me...@noctarius.com>
>> >> wrote:
>> >>
>> >>> Hehe well that really sounds like a nice bunch of people.
>> >>>
>> >>> Ok to be true I couldn't wait until tomorrow and started
>> >>> already reading the links. From what I was reading: If
>> >>> lightning will be a sub-part (sub-project) of DM, do I need
>> >>> to write an project purposal?
>> >>>
>> >>> Do I need to make any changes to the pom.xml like adding a
>> >>> special parent pom or anything like that?
>> >>>
>> >>> In general: there are a lot things to know :-)
>> >>>
>> >>> Am 29.09.2012 19:59, schrieb Raffaele P. Guidi:
>> >>>> Negative part of ASF membership? You get together with a
>> >>>> lot of geeky, talented people with a fixation for software
>> >>>> and open source. Oh wait but this is actually nice! :-D Il
>> >>>> giorno 29/set/2012 19:05, "Olivier Lamy" <ol...@apache.org>
>> >>>> ha
>> >>> scritto:
>> >>>>
>> >>>>> 2012/9/29 Noctarius <me...@noctarius.com>:
>> >>>>>> Thanks Olivier for carify, I'll take a look in it
>> >>>>>> tomorrow but there's just one question left (for now
>> >>>>>> ;)): What is that vote for becoming a committer? What
>> >>>>>> if the vote will be negative?
>> >>>>> The vote is on private list (pmc list for privacy reasons
>> >>>>> and possible negative stuff being on public lists)
>> >>>>>> Until now I just used Apache stuff, was never
>> >>>>>> interested in being part of it so I guess it can be
>> >>>>>> negative for any reason, can't it?
>> >>>>> I don't see why it could be negative but suspens ....
>> >>>>> :-)
>> >>>>>>
>> >>>>>> Am 29.09.2012 18:56, schrieb Olivier Lamy:
>> >>>>>>> 2012/9/29 Noctarius <me...@noctarius.com>:
>> >>>>>>>> Nope my real name is Christoph Engelbert, but
>> >>>>>>>> Noctarius is the all time nick :)
>> >>>>>>>>
>> >>>>>>>> Renaming the package should be no problem, is it
>> >>>>>>>> "org.apache.directmemory.lightning" or what would
>> >>>>>>>> it be?
>> >>>>>>> fine for me
>> >>>>>>>> Then there needs to be a change in the license
>> >>>>>>>> header as Olivier mentioned, that means just remove
>> >>>>>>>> the first sentence or is there anything more to do
>> >>>>>>>> (maybe it's easiest thing to just copy the header
>> >>>>>>>> from DM file ;))?
>> >>>>>>> yup use same header as DM
>> >>>>>>>>
>> >>>>>>>> The CLA is just a form to clarify that the source
>> >>>>>>>> can be contributed to the Apache Foundation?
>> >>>>>>> yup correct.
>> >>>>>>>>
>> >>>>>>>> The final step will be attaching the patch in form
>> >>>>>>>> of a huge diff file?
>> >>>>>>> yes
>> >>>>>>>>
>> >>>>>>>> And what is the way to apply for a membership?
>> >>>>>>>> Never thought about how to do that.
>> >>>>>>> Read here
>> >>>>>>> http://apache.org/foundation/how-it-works.html and
>> >>>>>>> here http://apache.org/foundation/getinvolved.html .
>> >>>>>>> And feel free to ask any questions :-)
>> >>>>>>>>
>> >>>>>>>> Chris
>> >>>>>>>>
>> >>>>>>>> Am 29.09.2012 18:23, schrieb Raffaele P. Guidi:
>> >>>>>>>>> OK, deal, at least for me ;-) I propose you
>> >>>>>>>>> rename the packages, produce a patch for this and
>> >>>>>>>>> the new serializer module (should be simple
>> >>>>>>>>> enough starting from an existing one) and, in the
>> >>>>>>>>> meanwhile, apply for ASF membership. Is IP
>> >>>>>>>>> clearance needed? I guess yes. After this we will
>> >>>>>>>>> come up with a formal vote regarding Noctarius
>> >>>>>>>>> (is this your real name?!) allowance in the
>> >>>>>>>>> project team.
>> >>>>>>>>>
>> >>>>>>>>> Good times are gonna come :-)
>> >>>>>>>>>
>> >>>>>>>>> Thanks, R Il giorno 29/set/2012 17:58, "Olivier
>> >>>>>>>>> Lamy" <ol...@apache.org> ha scritto:
>> >>>>>>>>>
>> >>>>>>>>>> 2012/9/29 Raffaele P. Guidi
>> >>>>>>>>>> <ra...@gmail.com>:
>> >>>>>>>>>>> Well we already have a NIO ready interface
>> >>>>>>>>>>> allowing direct access to DMs managed
>> >>>>>>>>>>> bytebuffers but I think this is just half way
>> >>>>>>>>>>> to what could be achieved optimally blending
>> >>>>>>>>>>> serialization and memory allocation
>> >>>>>>>>>>> together.
>> >>>>>>>>>>>
>> >>>>>>>>>>> Lightning as a module is of course a good
>> >>>>>>>>>>> idea and it could easily evolve as a
>> >>>>>>>>>>> subproject (for the more experienced asf
>> >>>>>>>>>>> members: is it a feasible way?).
>> >>>>>>>>>> Nothing prevent to have
>> >>>>>>>>>> http://svn.apache.org/repos/asf/directmemory/lightning/trunk
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>
>> >>>>>>>>>>
>> > with a package like: o.a.d.lightning That will be a
>> >>>>>>>>>> subproject.
>> >>>>>>>>>>>
>> >>>>>>>>>>> Ciao, R Il giorno 29/set/2012 17:44,
>> >>>>>>>>>>> "Noctarius" <me...@noctarius.com> ha scritto:
>> >>>>>>>>>>>
>> >>>>>>>>>>>> Hey guys,
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> ok there's no lightning binary available
>> >>>>>>>>>>>> since lightning wasn't ready yet for
>> >>>>>>>>>>>> releasing.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> For being the only developer it would be no
>> >>>>>>>>>>>> problem to contribute the sourcebase for
>> >>>>>>>>>>>> DirectMemory but I'm not sure yet if it
>> >>>>>>>>>>>> wouldn't be better to seperate it to be
>> >>>>>>>>>>>> available without using DirectMemory
>> >>>>>>>>>>>> itself. I started it as a serializer for
>> >>>>>>>>>>>> cluster synchronization, but it would be
>> >>>>>>>>>>>> cool to contribute lightning as a
>> >>>>>>>>>>>> subproject to DirectMemory :-)
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> About the second project I would love to
>> >>>>>>>>>>>> see a public available buffer API directly
>> >>>>>>>>>>>> in DirectMemory so that project would be
>> >>>>>>>>>>>> nearly needless :-) The only difference I
>> >>>>>>>>>>>> think is the allocation strategy my
>> >>>>>>>>>>>> implementation is using against the one
>> >>>>>>>>>>>> DirectMemory has, but I'm pretty sure the
>> >>>>>>>>>>>> allocation is extensible ;-)
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Like I said, for both projects I'm the only
>> >>>>>>>>>>>> dev so there would be no IP problem. So if
>> >>>>>>>>>>>> it's ok to you to not include lightning
>> >>>>>>>>>>>> directly in DM I would be glad to
>> >>>>>>>>>>>> contribute to the Apache Foundation :-)
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Cheers Chris
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Am 29.09.2012 16:53, schrieb Raffaele P.
>> >>>>>>>>>>>> Guidi:
>> >>>>>>>>>>>>> Ok so it's up to noctarius - your move!
>> >>>>>>>>>>>>> ;-) Regarding the new unsafe storage:
>> >>>>>>>>>>>>> it's an opt-in feature that can be set
>> >>>>>>>>>>>>> with the fluent API
>> >>>>>>>>>> (and
>> >>>>>>>>>>>>> soon through the conference file).
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Ciao, R Il giorno 29/set/2012 16:45,
>> >>>>>>>>>>>>> "Olivier Lamy" <ol...@apache.org> ha
>> >>>>>>>>>>>> scritto:
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>> 2012/9/29 Raffaele P. Guidi
>> >>>>>>>>>>>>>> <ra...@gmail.com>:
>> >>>>>>>>>>>>>>>>> At least for the moment he can
>> >>>>>>>>>>>>>>>>> provide a patch to be integrated
>> >>>>>>>>>> in DM
>> >>>>>>>>>>>>>> :-)
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Sure, but as lightning is not in any
>> >>>>>>>>>>>>>>> public mvn repo should its
>> >>>>>>>>>> code be
>> >>>>>>>>>>>>>>> re-published in our svn? Or what?
>> >>>>>>>>>>>>>> @Apache we don't care about binaries,
>> >>>>>>>>>>>>>> only sources are important ! (a bit
>> >>>>>>>>>>>>>> theorical for sure but that's it :-) ).
>> >>>>>>>>>>>>>> So if Noctarius was the only guy who
>> >>>>>>>>>>>>>> participate in lightning. He can just
>> >>>>>>>>>>>>>> provide a patch we could integrate as a
>> >>>>>>>>>>>>>> new dm module (note: the patch must not
>> >>>>>>>>>>>>>> contains any more copyright and all
>> >>>>>>>>>>>>>> sources must have ASF licenses).
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> "Copyright 2012 the original author or
>> >>>>>>>>>>>>>> authors." must be removed. And BTW
>> >>>>>>>>>>>>>> package must be changed :-) (com.github
>> >>>>>>>>>>>>>> is not acceptable
>> >>>>>>>>>> @asf
>> >>>>>>>>>>>>>> :-) )(@Noctarius are you working for
>> >>>>>>>>>>>>>> github ? :-) )
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> And having him as a committer will be
>> >>>>>>>>>>>>>> only a matter of voting (we
>> >>>>>>>>>> have
>> >>>>>>>>>>>>>> a great chair who take cares of
>> >>>>>>>>>>>>>> administrative stuff :P )
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> If some others have participated in the
>> >>>>>>>>>>>>>> project, we must pass tru an ip
>> >>>>>>>>>>>>>> clearance mechanism
>> >>>>>>>>>>>>>> (http://incubator.apache.org/ip-clearance/index.html)
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>
>> >>>>>>>>>>>>>>
>> > and all contributors to lightning must provide a cla.
>> >>>>>>>>>>>>>> (It it's the case I can help)
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> perso I'd like we avoid hard
>> >>>>>>>>>>>>>>>>> dependency on Unsafe as maybe
>> >>>>>>>>>>>>>>>>> some
>> >>>>>>>>>> use
>> >>>>>>>>>>>>>>> other jdks :-)
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Well, I believe Unsafe is supported
>> >>>>>>>>>>>>>>> by Sun JDK, OpenJDK, IBM JDK and
>> >>>>>>>>>>>>>>> JRockit - and I believe that it is
>> >>>>>>>>>>>>>>> more than enough. Also keep in
>> >>>>>>>>>> mind
>> >>>>>>>>>>>>>> that
>> >>>>>>>>>>>>>>> we already have an alternative Unsafe
>> >>>>>>>>>>>>>>> based memory storage - and
>> >>>>>>>>>>>> although
>> >>>>>>>>>>>>>>> it's not thoroughly tested for
>> >>>>>>>>>>>>>>> performance it dramaticaly
>> >>>>>>>>>>>>>>> simplifies
>> >>>>>>>>>>>>>> code,
>> >>>>>>>>>>>>>>> I have great expectations about it.
>> >>>>>>>>>>>>>> Me too :-). Yup definitely more simple
>> >>>>>>>>>>>>>> and faster ! But we must provide a
>> >>>>>>>>>>>>>> switch off configuration mechanism if
>> >>>>>>>>>>>>>> some
>> >>>>>>>>>> users
>> >>>>>>>>>>>>>> don't want to use that (because Unsafe
>> >>>>>>>>>>>>>> is just "Unsafe" :-) ) And sorry I
>> >>>>>>>>>>>>>> didn't have a look yet at your changes
>> >>>>>>>>>>>>>> with using Unsafe.
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Cheers, R
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> On Sat, Sep 29, 2012 at 4:03 PM,
>> >>>>>>>>>>>>>>> Olivier Lamy <ol...@apache.org>
>> >>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> 2012/9/29 Raffaele P. Guidi
>> >>>>>>>>>>>>>>>> <ra...@gmail.com>:
>> >>>>>>>>>>>>>>>>> What do you think about: 1)
>> >>>>>>>>>>>>>>>>> implementing a lightning
>> >>>>>>>>>>>>>>>>> serialization module 2) creating
>> >>>>>>>>>>>>>>>>> a serializer that directly works
>> >>>>>>>>>>>>>>>>> on a directmemory
>> >>>>>>>>>>>>>> provider
>> >>>>>>>>>>>>>>>>> ByteBuffer or (maybe better)
>> >>>>>>>>>>>>>>>>> Unsafe based Pointer?
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> Sounds good (perso I'd like we
>> >>>>>>>>>>>>>>>> avoid hard dependency on Unsafe as
>> >>>>>>>>>>>>>>>> maybe some use other jdks :-) )
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> Now I see lightning is apache
>> >>>>>>>>>>>>>>>>> licensed and this is fine but I
>> >>>>>>>>>> don't
>> >>>>>>>>>>>>>> think
>> >>>>>>>>>>>>>>>>> it is published in any public
>> >>>>>>>>>>>>>>>>> maven repo, am I right? We could
>> >>>>>>>>>> find a
>> >>>>>>>>>>>>>> way
>> >>>>>>>>>>>>>>>>> to deal with this; options vary
>> >>>>>>>>>>>>>>>>> from publishing lightning to the
>> >>>>>>>>>> free
>> >>>>>>>>>>>>>>>>> sonatype repo, joining the ASF
>> >>>>>>>>>>>>>>>>> (which is great anyhow!) and
>> >>>>>>>>>>>>>> republishing
>> >>>>>>>>>>>>>>>>> lightning code in DirectMemory
>> >>>>>>>>>>>>>>>>> becoming a commiter (which has
>> >>>>>>>>>>>>>>>>> to
>> >>>>>>>>>>>>>> undergo
>> >>>>>>>>>>>>>>>> a
>> >>>>>>>>>>>>>>>>> PMC vote).
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> At least for the moment he can
>> >>>>>>>>>>>>>>>> provide a patch to be integrated
>> >>>>>>>>>>>>>>>> in
>> >>>>>>>>>> DM
>> >>>>>>>>>>>>>> :-)
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> I'd like to hear your and the
>> >>>>>>>>>>>>>>>>> team feelings on this.
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> Thanks, Raffaele
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> On Sat, Sep 29, 2012 at 3:27 PM,
>> >>>>>>>>>>>>>>>>> Noctarius <me...@noctarius.com>
>> >>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> Hey Raffaele,
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> that's quite similar to what I
>> >>>>>>>>>>>>>>>>>> did at work. We're developing
>> >>>>>>>>>> Flash
>> >>>>>>>>>>>>>>>>>> online games and using a
>> >>>>>>>>>>>>>>>>>> customized AMF serialization.
>> >>>>>>>>>>>>>>>>>> To support async writing of the
>> >>>>>>>>>>>>>>>>>> clients event results I added
>> >>>>>>>>>>>>>>>>>> serialization
>> >>>>>>>>>> of
>> >>>>>>>>>>>>>>>>>> the components / entities to
>> >>>>>>>>>>>>>>>>>> the players zone calculation
>> >>>>>>>>>>>>>>>>>> and
>> >>>>>>>>>> stored
>> >>>>>>>>>>>>>>>>>> the pre-serialized bytestream
>> >>>>>>>>>>>>>>>>>> directly to the off-heap (using
>> >>>>>>>>>>>>>>>>>> direct-ring-cache
>> >>>>>>>>>>>>>>>>>> implementation). When the
>> >>>>>>>>>>>>>>>>>> client requests the results
>> >>>>>>>>>>>>>>>>>> (using long-polling) I just
>> >>>>>>>>>>>>>>>>>> write the pre-serialized
>> >>>>>>>>>> data to
>> >>>>>>>>>>>>>>>>>> the right position to
>> >>>>>>>>>>>>>>>>>> deserialize it by standard ways
>> >>>>>>>>>>>>>>>>>> on Flash
>> >>>>>>>>>> side.
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> So yeah an seriliaztion to
>> >>>>>>>>>>>>>>>>>> off-heap algorithm would be
>> >>>>>>>>>>>>>>>>>> fine. You
>> >>>>>>>>>> can
>> >>>>>>>>>>>>>>>>>> avoid using byte arrays and
>> >>>>>>>>>>>>>>>>>> minimalize GC.
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> Cheers Chris
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> Am 29.09.2012 15:02, schrieb
>> >>>>>>>>>>>>>>>>>> Raffaele P. Guidi:
>> >>>>>>>>>>>>>>>>>>> Nice to hear back from you!
>> >>>>>>>>>>>>>>>>>>> Yes, I was thinking about
>> >>>>>>>>>>>>>>>>>>> creating a
>> >>>>>>>>>>>>>> new
>> >>>>>>>>>>>>>>>>>> memory
>> >>>>>>>>>>>>>>>>>>> storage implementation using
>> >>>>>>>>>>>>>>>>>>> Unsafe (and I did it,
>> >>>>>>>>>>>>>>>>>>> recently) and
>> >>>>>>>>>>>>>>>> also, as
>> >>>>>>>>>>>>>>>>>>> DirectMemory relies heavily
>> >>>>>>>>>>>>>>>>>>> on serialization (and
>> >>>>>>>>>>>>>>>>>>> supports many
>> >>>>>>>>>> of
>> >>>>>>>>>>>>>>>> them,
>> >>>>>>>>>>>>>>>>>>> protostuff, protobuf, msgpack
>> >>>>>>>>>>>>>>>>>>> and of course standard
>> >>>>>>>>>>>>>> serialization),
>> >>>>>>>>>>>>>>>>>> about
>> >>>>>>>>>>>>>>>>>>> creating a simple embedded
>> >>>>>>>>>>>>>>>>>>> serializer leveraging the
>> >>>>>>>>>>>>>>>>>>> same
>> >>>>>>>>>>>>>> techniques
>> >>>>>>>>>>>>>>>> you
>> >>>>>>>>>>>>>>>>>>> used (Unsafe and bytecode
>> >>>>>>>>>>>>>>>>>>> generation). The idea with
>> >>>>>>>>>>>>>>>>>>> embedding is avoiding to
>> >>>>>>>>>>>>>>>>>>> serialize in a byte array
>> >>>>>>>>>>>>>> and
>> >>>>>>>>>>>>>>>> then
>> >>>>>>>>>>>>>>>>>>> moving the byte array to
>> >>>>>>>>>>>>>>>>>>> off-heap memory (via Unsafe
>> >>>>>>>>>>>>>>>>>>> or
>> >>>>>>>>>>>>>> ByteBuffers),
>> >>>>>>>>>>>>>>>> and
>> >>>>>>>>>>>>>>>>>>> serializing directly into a
>> >>>>>>>>>>>>>>>>>>> "managed" off-heap buffer,
>> >>>>>>>>>>>>>>>>>>> thus
>> >>>>>>>>>> further
>> >>>>>>>>>>>>>>>>>>> optimizing heap utilization
>> >>>>>>>>>>>>>>>>>>> (less GC).
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> What do you think about it?
>> >>>>>>>>>>>>>>>>>>> Does it make any sense to
>> >>>>>>>>>>>>>>>>>>> you?
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> Ciao, R
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> On Sat, Sep 29, 2012 at 2:40
>> >>>>>>>>>>>>>>>>>>> PM, Noctarius
>> >>>>>>>>>>>>>>>>>>> <me...@noctarius.com>
>> >>>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>> Hey guys,
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>> Raffaele found out about a
>> >>>>>>>>>>>>>>>>>>>> project of mine (Lightning)
>> >>>>>>>>>>>>>>>>>>>> a few
>> >>>>>>>>>> weeks
>> >>>>>>>>>>>>>>>>>>>> ago. Lightning is a heavy
>> >>>>>>>>>>>>>>>>>>>> Unsafe and Bytecode
>> >>>>>>>>>>>>>>>>>>>> generation using
>> >>>>>>>>>>>>>>>>>>>> Serializer implementation.
>> >>>>>>>>>>>>>>>>>>>> He told me that he was
>> >>>>>>>>>>>>>>>>>>>> interested in adding
>> >>>>>>>>>>>>>>>>>>>> something similar to
>> >>>>>>>>>>>>>>>>>>>> DirectMemory and I would be
>> >>>>>>>>>>>>>>>>>>>> glad to
>> >>>>>>>>>>>>>> help
>> >>>>>>>>>>>>>>>>>>>> out in this.
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>> Another project I started a
>> >>>>>>>>>>>>>>>>>>>> few days ago, since it was
>> >>>>>>>>>>>>>>>>>>>> needed
>> >>>>>>>>>> for
>> >>>>>>>>>>>>>>>>>>>> work is DirectRingCache.
>> >>>>>>>>>>>>>>>>>>>> The name not really meets
>> >>>>>>>>>>>>>>>>>>>> to actual implementation
>> >>>>>>>>>>>>>>>>>>>> since it's not yet a ring
>> >>>>>>>>>>>>>>>>>>>> buffer using cache. I
>> >>>>>>>>>>>>>> used
>> >>>>>>>>>>>>>>>>>>>> this for a
>> >>>>>>>>>>>>>>>>>>>> pre-serialization simple
>> >>>>>>>>>>>>>>>>>>>> bytestream cache with
>> >>>>>>>>>>>>>>>>>>>> self-growing buffers. It
>> >>>>>>>>>>>>>>>>>>>> could be nice to have
>> >>>>>>>>>>>>>>>>>>>> DirectMemory
>> >>>>>>>>>> having
>> >>>>>>>>>>>>>>>>>>>> raw "buffers" to write to
>> >>>>>>>>>>>>>>>>>>>> or to read from.
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>> Here are the links from
>> >>>>>>>>>>>>>>>>>>>> the projects:
>> >>>>>>>>>>>>>>>>>>>> https://github.com/noctarius/Lightning
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>
>> >>>>>>>>>>>>>>>>>>>>
>> > https://github.com/noctarius/direct-ring-cache
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>> It would be nice to help out since I really like
>> >>>>>>>> the idea of
>> >>>>>>>>>>>>>>>>>>>> DirectMemory and since
>> >>>>>>>>>>>>>>>>>>>> direct-ring-cache is some
>> >>>>>>>>>>>>>>>>>>>> kind of
>> >>>>>>>>>>>>>> reinventing
>> >>>>>>>>>>>>>>>>>>>> the wheel.
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>> Cheers Noctarius (Chris)
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> -- Olivier Lamy Talend:
>> >>>>>>>>>>>>>>>> http://coders.talend.com
>> >>>>>>>>>>>>>>>> http://twitter.com/olamy |
>> >>>>>>>>>>>>>>>> http://linkedin.com/in/olamy
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> -- Olivier Lamy Talend:
>> >>>>>>>>>>>>>> http://coders.talend.com
>> >>>>>>>>>>>>>> http://twitter.com/olamy |
>> >>>>>>>>>>>>>> http://linkedin.com/in/olamy
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> -- Olivier Lamy Talend:
>> >>>>>>>>>> http://coders.talend.com
>> >>>>>>>>>> http://twitter.com/olamy |
>> >>>>>>>>>> http://linkedin.com/in/olamy
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> -- Olivier Lamy Talend: http://coders.talend.com
>> >>>>> http://twitter.com/olamy | http://linkedin.com/in/olamy
>> >>>>>
>> >>>>
>> >>>
>> >>
>> >
>>
Re: Additional Serializer and raw Buffer access
Posted by "Raffaele P. Guidi" <ra...@gmail.com>.
Really nice ideas - we already have an Unsafe based storage (it is rather
new) and the idea was exactly to combine the benefits of it with the ones
of unsafe serialization. Said that keep in mind that classic DirectMemory
is in any case way faster than just using ByteBuffers because it
pre-allocates large chunks of memory and then "slices" them for single,
smaller items.
And - a Kryo serializer is a rather good idea. Do you feel like working on
it? Serializers are actually one of the easiest thing to contribute on in
DM - and they add immediate value as each one is probably a better fit than
others in different scenarios.
Ciao,
R
Il giorno 29/set/2012 21:16, "Roman Levenstein" <ro...@gmail.com> ha
scritto:
> Hi,
>
> I'm just wondering if lightning uses Unsafe only for serializers (i.e.
> for reading/writing fields of objects) or also for the input/output
> streams used to read/write to/from off-heap memory? Using
> DirectBuffers "as is" for accessing off-heap memory is actually rather
> slow.
>
> Let me share some experiences with you:
> I have implemented a similar feature for Kryo. It is not in the trunk
> yet, as it is being reviewed by Nate, the author of Kryo.
> One of the things that I noticed is that using Unsafe only for
> serializers is nice, but does not provide much improvement.
> At the same time, using Unsafe for input/output streams and
> reading/writing directly from/into off-heap memory provides
> significant performance benefits.
> My patch for Kryo implements both features using Unsafe. If you want
> to see more details about the implementation and findings, please have
> a look at:
> http://code.google.com/p/kryo/issues/detail?id=75
>
> I hope it will land in Kryo trunk soon. Once it is there, I think it
> could be interesting to add a Kryo-based serialization to
> DirectMemory. What do you think?
> But independent of this, feel free to take any ideas or even the
> implementation from the patch. The off-heap memory streams
> implementation is rather self-contained and could be reused with other
> frameworks besides Kryo.
>
> Regards,
> Roman
>
> On Sat, Sep 29, 2012 at 8:46 PM, Noctarius <me...@noctarius.com> wrote:
> > Actually all dependencies should be AL2 or BSD licensed :-)
> >
> > Am 29.09.2012 20:42, schrieb Raffaele P. Guidi:
> >>>> Hehe well that really sounds like a nice bunch of people.
> >>
> >> Indeed they are (I'm a newbie as well and try to do my best)
> >>
> >>>> If lightning will be a sub-part (sub-project) of DM, do I
> >>>> need to write
> >> an project purposal?
> >>
> >> Nope, not needed for a sub-project
> >>
> >>>> Do I need to make any changes to the pom.xml like adding a
> >> special parent pom or anything like that?
> >>
> >> Not for the serializer - just have to take a look at project
> >> dependencies - or, better, at their licenses - are they
> >> compatible with the ASL 2.0? i.e. a GPL'd library is not a good
> >> fit and should be replaced with an apache licensed (or BSD, or
> >> MIT...) one if possible. For the integration module is a
> >> separate story - you should start off copying one of the other
> >> serializers and reusing the same pom and directory structure.
> >>
> >> Pleased to meet you, Chris :)
> >>
> >> Ciao, R
> >>
> >> On Sat, Sep 29, 2012 at 8:24 PM, Noctarius <me...@noctarius.com>
> >> wrote:
> >>
> >>> Hehe well that really sounds like a nice bunch of people.
> >>>
> >>> Ok to be true I couldn't wait until tomorrow and started
> >>> already reading the links. From what I was reading: If
> >>> lightning will be a sub-part (sub-project) of DM, do I need
> >>> to write an project purposal?
> >>>
> >>> Do I need to make any changes to the pom.xml like adding a
> >>> special parent pom or anything like that?
> >>>
> >>> In general: there are a lot things to know :-)
> >>>
> >>> Am 29.09.2012 19:59, schrieb Raffaele P. Guidi:
> >>>> Negative part of ASF membership? You get together with a
> >>>> lot of geeky, talented people with a fixation for software
> >>>> and open source. Oh wait but this is actually nice! :-D Il
> >>>> giorno 29/set/2012 19:05, "Olivier Lamy" <ol...@apache.org>
> >>>> ha
> >>> scritto:
> >>>>
> >>>>> 2012/9/29 Noctarius <me...@noctarius.com>:
> >>>>>> Thanks Olivier for carify, I'll take a look in it
> >>>>>> tomorrow but there's just one question left (for now
> >>>>>> ;)): What is that vote for becoming a committer? What
> >>>>>> if the vote will be negative?
> >>>>> The vote is on private list (pmc list for privacy reasons
> >>>>> and possible negative stuff being on public lists)
> >>>>>> Until now I just used Apache stuff, was never
> >>>>>> interested in being part of it so I guess it can be
> >>>>>> negative for any reason, can't it?
> >>>>> I don't see why it could be negative but suspens ....
> >>>>> :-)
> >>>>>>
> >>>>>> Am 29.09.2012 18:56, schrieb Olivier Lamy:
> >>>>>>> 2012/9/29 Noctarius <me...@noctarius.com>:
> >>>>>>>> Nope my real name is Christoph Engelbert, but
> >>>>>>>> Noctarius is the all time nick :)
> >>>>>>>>
> >>>>>>>> Renaming the package should be no problem, is it
> >>>>>>>> "org.apache.directmemory.lightning" or what would
> >>>>>>>> it be?
> >>>>>>> fine for me
> >>>>>>>> Then there needs to be a change in the license
> >>>>>>>> header as Olivier mentioned, that means just remove
> >>>>>>>> the first sentence or is there anything more to do
> >>>>>>>> (maybe it's easiest thing to just copy the header
> >>>>>>>> from DM file ;))?
> >>>>>>> yup use same header as DM
> >>>>>>>>
> >>>>>>>> The CLA is just a form to clarify that the source
> >>>>>>>> can be contributed to the Apache Foundation?
> >>>>>>> yup correct.
> >>>>>>>>
> >>>>>>>> The final step will be attaching the patch in form
> >>>>>>>> of a huge diff file?
> >>>>>>> yes
> >>>>>>>>
> >>>>>>>> And what is the way to apply for a membership?
> >>>>>>>> Never thought about how to do that.
> >>>>>>> Read here
> >>>>>>> http://apache.org/foundation/how-it-works.html and
> >>>>>>> here http://apache.org/foundation/getinvolved.html .
> >>>>>>> And feel free to ask any questions :-)
> >>>>>>>>
> >>>>>>>> Chris
> >>>>>>>>
> >>>>>>>> Am 29.09.2012 18:23, schrieb Raffaele P. Guidi:
> >>>>>>>>> OK, deal, at least for me ;-) I propose you
> >>>>>>>>> rename the packages, produce a patch for this and
> >>>>>>>>> the new serializer module (should be simple
> >>>>>>>>> enough starting from an existing one) and, in the
> >>>>>>>>> meanwhile, apply for ASF membership. Is IP
> >>>>>>>>> clearance needed? I guess yes. After this we will
> >>>>>>>>> come up with a formal vote regarding Noctarius
> >>>>>>>>> (is this your real name?!) allowance in the
> >>>>>>>>> project team.
> >>>>>>>>>
> >>>>>>>>> Good times are gonna come :-)
> >>>>>>>>>
> >>>>>>>>> Thanks, R Il giorno 29/set/2012 17:58, "Olivier
> >>>>>>>>> Lamy" <ol...@apache.org> ha scritto:
> >>>>>>>>>
> >>>>>>>>>> 2012/9/29 Raffaele P. Guidi
> >>>>>>>>>> <ra...@gmail.com>:
> >>>>>>>>>>> Well we already have a NIO ready interface
> >>>>>>>>>>> allowing direct access to DMs managed
> >>>>>>>>>>> bytebuffers but I think this is just half way
> >>>>>>>>>>> to what could be achieved optimally blending
> >>>>>>>>>>> serialization and memory allocation
> >>>>>>>>>>> together.
> >>>>>>>>>>>
> >>>>>>>>>>> Lightning as a module is of course a good
> >>>>>>>>>>> idea and it could easily evolve as a
> >>>>>>>>>>> subproject (for the more experienced asf
> >>>>>>>>>>> members: is it a feasible way?).
> >>>>>>>>>> Nothing prevent to have
> >>>>>>>>>> http://svn.apache.org/repos/asf/directmemory/lightning/trunk
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>
> >>>>>>>>>>
> > with a package like: o.a.d.lightning That will be a
> >>>>>>>>>> subproject.
> >>>>>>>>>>>
> >>>>>>>>>>> Ciao, R Il giorno 29/set/2012 17:44,
> >>>>>>>>>>> "Noctarius" <me...@noctarius.com> ha scritto:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hey guys,
> >>>>>>>>>>>>
> >>>>>>>>>>>> ok there's no lightning binary available
> >>>>>>>>>>>> since lightning wasn't ready yet for
> >>>>>>>>>>>> releasing.
> >>>>>>>>>>>>
> >>>>>>>>>>>> For being the only developer it would be no
> >>>>>>>>>>>> problem to contribute the sourcebase for
> >>>>>>>>>>>> DirectMemory but I'm not sure yet if it
> >>>>>>>>>>>> wouldn't be better to seperate it to be
> >>>>>>>>>>>> available without using DirectMemory
> >>>>>>>>>>>> itself. I started it as a serializer for
> >>>>>>>>>>>> cluster synchronization, but it would be
> >>>>>>>>>>>> cool to contribute lightning as a
> >>>>>>>>>>>> subproject to DirectMemory :-)
> >>>>>>>>>>>>
> >>>>>>>>>>>> About the second project I would love to
> >>>>>>>>>>>> see a public available buffer API directly
> >>>>>>>>>>>> in DirectMemory so that project would be
> >>>>>>>>>>>> nearly needless :-) The only difference I
> >>>>>>>>>>>> think is the allocation strategy my
> >>>>>>>>>>>> implementation is using against the one
> >>>>>>>>>>>> DirectMemory has, but I'm pretty sure the
> >>>>>>>>>>>> allocation is extensible ;-)
> >>>>>>>>>>>>
> >>>>>>>>>>>> Like I said, for both projects I'm the only
> >>>>>>>>>>>> dev so there would be no IP problem. So if
> >>>>>>>>>>>> it's ok to you to not include lightning
> >>>>>>>>>>>> directly in DM I would be glad to
> >>>>>>>>>>>> contribute to the Apache Foundation :-)
> >>>>>>>>>>>>
> >>>>>>>>>>>> Cheers Chris
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Am 29.09.2012 16:53, schrieb Raffaele P.
> >>>>>>>>>>>> Guidi:
> >>>>>>>>>>>>> Ok so it's up to noctarius - your move!
> >>>>>>>>>>>>> ;-) Regarding the new unsafe storage:
> >>>>>>>>>>>>> it's an opt-in feature that can be set
> >>>>>>>>>>>>> with the fluent API
> >>>>>>>>>> (and
> >>>>>>>>>>>>> soon through the conference file).
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Ciao, R Il giorno 29/set/2012 16:45,
> >>>>>>>>>>>>> "Olivier Lamy" <ol...@apache.org> ha
> >>>>>>>>>>>> scritto:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> 2012/9/29 Raffaele P. Guidi
> >>>>>>>>>>>>>> <ra...@gmail.com>:
> >>>>>>>>>>>>>>>>> At least for the moment he can
> >>>>>>>>>>>>>>>>> provide a patch to be integrated
> >>>>>>>>>> in DM
> >>>>>>>>>>>>>> :-)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Sure, but as lightning is not in any
> >>>>>>>>>>>>>>> public mvn repo should its
> >>>>>>>>>> code be
> >>>>>>>>>>>>>>> re-published in our svn? Or what?
> >>>>>>>>>>>>>> @Apache we don't care about binaries,
> >>>>>>>>>>>>>> only sources are important ! (a bit
> >>>>>>>>>>>>>> theorical for sure but that's it :-) ).
> >>>>>>>>>>>>>> So if Noctarius was the only guy who
> >>>>>>>>>>>>>> participate in lightning. He can just
> >>>>>>>>>>>>>> provide a patch we could integrate as a
> >>>>>>>>>>>>>> new dm module (note: the patch must not
> >>>>>>>>>>>>>> contains any more copyright and all
> >>>>>>>>>>>>>> sources must have ASF licenses).
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> "Copyright 2012 the original author or
> >>>>>>>>>>>>>> authors." must be removed. And BTW
> >>>>>>>>>>>>>> package must be changed :-) (com.github
> >>>>>>>>>>>>>> is not acceptable
> >>>>>>>>>> @asf
> >>>>>>>>>>>>>> :-) )(@Noctarius are you working for
> >>>>>>>>>>>>>> github ? :-) )
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> And having him as a committer will be
> >>>>>>>>>>>>>> only a matter of voting (we
> >>>>>>>>>> have
> >>>>>>>>>>>>>> a great chair who take cares of
> >>>>>>>>>>>>>> administrative stuff :P )
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> If some others have participated in the
> >>>>>>>>>>>>>> project, we must pass tru an ip
> >>>>>>>>>>>>>> clearance mechanism
> >>>>>>>>>>>>>> (http://incubator.apache.org/ip-clearance/index.html)
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>
> >>>>>>>>>>>>>>
> > and all contributors to lightning must provide a cla.
> >>>>>>>>>>>>>> (It it's the case I can help)
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> perso I'd like we avoid hard
> >>>>>>>>>>>>>>>>> dependency on Unsafe as maybe
> >>>>>>>>>>>>>>>>> some
> >>>>>>>>>> use
> >>>>>>>>>>>>>>> other jdks :-)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Well, I believe Unsafe is supported
> >>>>>>>>>>>>>>> by Sun JDK, OpenJDK, IBM JDK and
> >>>>>>>>>>>>>>> JRockit - and I believe that it is
> >>>>>>>>>>>>>>> more than enough. Also keep in
> >>>>>>>>>> mind
> >>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>> we already have an alternative Unsafe
> >>>>>>>>>>>>>>> based memory storage - and
> >>>>>>>>>>>> although
> >>>>>>>>>>>>>>> it's not thoroughly tested for
> >>>>>>>>>>>>>>> performance it dramaticaly
> >>>>>>>>>>>>>>> simplifies
> >>>>>>>>>>>>>> code,
> >>>>>>>>>>>>>>> I have great expectations about it.
> >>>>>>>>>>>>>> Me too :-). Yup definitely more simple
> >>>>>>>>>>>>>> and faster ! But we must provide a
> >>>>>>>>>>>>>> switch off configuration mechanism if
> >>>>>>>>>>>>>> some
> >>>>>>>>>> users
> >>>>>>>>>>>>>> don't want to use that (because Unsafe
> >>>>>>>>>>>>>> is just "Unsafe" :-) ) And sorry I
> >>>>>>>>>>>>>> didn't have a look yet at your changes
> >>>>>>>>>>>>>> with using Unsafe.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Cheers, R
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Sat, Sep 29, 2012 at 4:03 PM,
> >>>>>>>>>>>>>>> Olivier Lamy <ol...@apache.org>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> 2012/9/29 Raffaele P. Guidi
> >>>>>>>>>>>>>>>> <ra...@gmail.com>:
> >>>>>>>>>>>>>>>>> What do you think about: 1)
> >>>>>>>>>>>>>>>>> implementing a lightning
> >>>>>>>>>>>>>>>>> serialization module 2) creating
> >>>>>>>>>>>>>>>>> a serializer that directly works
> >>>>>>>>>>>>>>>>> on a directmemory
> >>>>>>>>>>>>>> provider
> >>>>>>>>>>>>>>>>> ByteBuffer or (maybe better)
> >>>>>>>>>>>>>>>>> Unsafe based Pointer?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Sounds good (perso I'd like we
> >>>>>>>>>>>>>>>> avoid hard dependency on Unsafe as
> >>>>>>>>>>>>>>>> maybe some use other jdks :-) )
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Now I see lightning is apache
> >>>>>>>>>>>>>>>>> licensed and this is fine but I
> >>>>>>>>>> don't
> >>>>>>>>>>>>>> think
> >>>>>>>>>>>>>>>>> it is published in any public
> >>>>>>>>>>>>>>>>> maven repo, am I right? We could
> >>>>>>>>>> find a
> >>>>>>>>>>>>>> way
> >>>>>>>>>>>>>>>>> to deal with this; options vary
> >>>>>>>>>>>>>>>>> from publishing lightning to the
> >>>>>>>>>> free
> >>>>>>>>>>>>>>>>> sonatype repo, joining the ASF
> >>>>>>>>>>>>>>>>> (which is great anyhow!) and
> >>>>>>>>>>>>>> republishing
> >>>>>>>>>>>>>>>>> lightning code in DirectMemory
> >>>>>>>>>>>>>>>>> becoming a commiter (which has
> >>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>> undergo
> >>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>> PMC vote).
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> At least for the moment he can
> >>>>>>>>>>>>>>>> provide a patch to be integrated
> >>>>>>>>>>>>>>>> in
> >>>>>>>>>> DM
> >>>>>>>>>>>>>> :-)
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I'd like to hear your and the
> >>>>>>>>>>>>>>>>> team feelings on this.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks, Raffaele
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Sat, Sep 29, 2012 at 3:27 PM,
> >>>>>>>>>>>>>>>>> Noctarius <me...@noctarius.com>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Hey Raffaele,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> that's quite similar to what I
> >>>>>>>>>>>>>>>>>> did at work. We're developing
> >>>>>>>>>> Flash
> >>>>>>>>>>>>>>>>>> online games and using a
> >>>>>>>>>>>>>>>>>> customized AMF serialization.
> >>>>>>>>>>>>>>>>>> To support async writing of the
> >>>>>>>>>>>>>>>>>> clients event results I added
> >>>>>>>>>>>>>>>>>> serialization
> >>>>>>>>>> of
> >>>>>>>>>>>>>>>>>> the components / entities to
> >>>>>>>>>>>>>>>>>> the players zone calculation
> >>>>>>>>>>>>>>>>>> and
> >>>>>>>>>> stored
> >>>>>>>>>>>>>>>>>> the pre-serialized bytestream
> >>>>>>>>>>>>>>>>>> directly to the off-heap (using
> >>>>>>>>>>>>>>>>>> direct-ring-cache
> >>>>>>>>>>>>>>>>>> implementation). When the
> >>>>>>>>>>>>>>>>>> client requests the results
> >>>>>>>>>>>>>>>>>> (using long-polling) I just
> >>>>>>>>>>>>>>>>>> write the pre-serialized
> >>>>>>>>>> data to
> >>>>>>>>>>>>>>>>>> the right position to
> >>>>>>>>>>>>>>>>>> deserialize it by standard ways
> >>>>>>>>>>>>>>>>>> on Flash
> >>>>>>>>>> side.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> So yeah an seriliaztion to
> >>>>>>>>>>>>>>>>>> off-heap algorithm would be
> >>>>>>>>>>>>>>>>>> fine. You
> >>>>>>>>>> can
> >>>>>>>>>>>>>>>>>> avoid using byte arrays and
> >>>>>>>>>>>>>>>>>> minimalize GC.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Cheers Chris
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Am 29.09.2012 15:02, schrieb
> >>>>>>>>>>>>>>>>>> Raffaele P. Guidi:
> >>>>>>>>>>>>>>>>>>> Nice to hear back from you!
> >>>>>>>>>>>>>>>>>>> Yes, I was thinking about
> >>>>>>>>>>>>>>>>>>> creating a
> >>>>>>>>>>>>>> new
> >>>>>>>>>>>>>>>>>> memory
> >>>>>>>>>>>>>>>>>>> storage implementation using
> >>>>>>>>>>>>>>>>>>> Unsafe (and I did it,
> >>>>>>>>>>>>>>>>>>> recently) and
> >>>>>>>>>>>>>>>> also, as
> >>>>>>>>>>>>>>>>>>> DirectMemory relies heavily
> >>>>>>>>>>>>>>>>>>> on serialization (and
> >>>>>>>>>>>>>>>>>>> supports many
> >>>>>>>>>> of
> >>>>>>>>>>>>>>>> them,
> >>>>>>>>>>>>>>>>>>> protostuff, protobuf, msgpack
> >>>>>>>>>>>>>>>>>>> and of course standard
> >>>>>>>>>>>>>> serialization),
> >>>>>>>>>>>>>>>>>> about
> >>>>>>>>>>>>>>>>>>> creating a simple embedded
> >>>>>>>>>>>>>>>>>>> serializer leveraging the
> >>>>>>>>>>>>>>>>>>> same
> >>>>>>>>>>>>>> techniques
> >>>>>>>>>>>>>>>> you
> >>>>>>>>>>>>>>>>>>> used (Unsafe and bytecode
> >>>>>>>>>>>>>>>>>>> generation). The idea with
> >>>>>>>>>>>>>>>>>>> embedding is avoiding to
> >>>>>>>>>>>>>>>>>>> serialize in a byte array
> >>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>> then
> >>>>>>>>>>>>>>>>>>> moving the byte array to
> >>>>>>>>>>>>>>>>>>> off-heap memory (via Unsafe
> >>>>>>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>> ByteBuffers),
> >>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>> serializing directly into a
> >>>>>>>>>>>>>>>>>>> "managed" off-heap buffer,
> >>>>>>>>>>>>>>>>>>> thus
> >>>>>>>>>> further
> >>>>>>>>>>>>>>>>>>> optimizing heap utilization
> >>>>>>>>>>>>>>>>>>> (less GC).
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> What do you think about it?
> >>>>>>>>>>>>>>>>>>> Does it make any sense to
> >>>>>>>>>>>>>>>>>>> you?
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Ciao, R
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On Sat, Sep 29, 2012 at 2:40
> >>>>>>>>>>>>>>>>>>> PM, Noctarius
> >>>>>>>>>>>>>>>>>>> <me...@noctarius.com>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Hey guys,
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Raffaele found out about a
> >>>>>>>>>>>>>>>>>>>> project of mine (Lightning)
> >>>>>>>>>>>>>>>>>>>> a few
> >>>>>>>>>> weeks
> >>>>>>>>>>>>>>>>>>>> ago. Lightning is a heavy
> >>>>>>>>>>>>>>>>>>>> Unsafe and Bytecode
> >>>>>>>>>>>>>>>>>>>> generation using
> >>>>>>>>>>>>>>>>>>>> Serializer implementation.
> >>>>>>>>>>>>>>>>>>>> He told me that he was
> >>>>>>>>>>>>>>>>>>>> interested in adding
> >>>>>>>>>>>>>>>>>>>> something similar to
> >>>>>>>>>>>>>>>>>>>> DirectMemory and I would be
> >>>>>>>>>>>>>>>>>>>> glad to
> >>>>>>>>>>>>>> help
> >>>>>>>>>>>>>>>>>>>> out in this.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Another project I started a
> >>>>>>>>>>>>>>>>>>>> few days ago, since it was
> >>>>>>>>>>>>>>>>>>>> needed
> >>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>> work is DirectRingCache.
> >>>>>>>>>>>>>>>>>>>> The name not really meets
> >>>>>>>>>>>>>>>>>>>> to actual implementation
> >>>>>>>>>>>>>>>>>>>> since it's not yet a ring
> >>>>>>>>>>>>>>>>>>>> buffer using cache. I
> >>>>>>>>>>>>>> used
> >>>>>>>>>>>>>>>>>>>> this for a
> >>>>>>>>>>>>>>>>>>>> pre-serialization simple
> >>>>>>>>>>>>>>>>>>>> bytestream cache with
> >>>>>>>>>>>>>>>>>>>> self-growing buffers. It
> >>>>>>>>>>>>>>>>>>>> could be nice to have
> >>>>>>>>>>>>>>>>>>>> DirectMemory
> >>>>>>>>>> having
> >>>>>>>>>>>>>>>>>>>> raw "buffers" to write to
> >>>>>>>>>>>>>>>>>>>> or to read from.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Here are the links from
> >>>>>>>>>>>>>>>>>>>> the projects:
> >>>>>>>>>>>>>>>>>>>> https://github.com/noctarius/Lightning
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>
> >>>>>>>>>>>>>>>>>>>>
> > https://github.com/noctarius/direct-ring-cache
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>> It would be nice to help out since I really like
> >>>>>>>> the idea of
> >>>>>>>>>>>>>>>>>>>> DirectMemory and since
> >>>>>>>>>>>>>>>>>>>> direct-ring-cache is some
> >>>>>>>>>>>>>>>>>>>> kind of
> >>>>>>>>>>>>>> reinventing
> >>>>>>>>>>>>>>>>>>>> the wheel.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Cheers Noctarius (Chris)
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> -- Olivier Lamy Talend:
> >>>>>>>>>>>>>>>> http://coders.talend.com
> >>>>>>>>>>>>>>>> http://twitter.com/olamy |
> >>>>>>>>>>>>>>>> http://linkedin.com/in/olamy
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> -- Olivier Lamy Talend:
> >>>>>>>>>>>>>> http://coders.talend.com
> >>>>>>>>>>>>>> http://twitter.com/olamy |
> >>>>>>>>>>>>>> http://linkedin.com/in/olamy
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> -- Olivier Lamy Talend:
> >>>>>>>>>> http://coders.talend.com
> >>>>>>>>>> http://twitter.com/olamy |
> >>>>>>>>>> http://linkedin.com/in/olamy
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> -- Olivier Lamy Talend: http://coders.talend.com
> >>>>> http://twitter.com/olamy | http://linkedin.com/in/olamy
> >>>>>
> >>>>
> >>>
> >>
> >
>
Re: Additional Serializer and raw Buffer access
Posted by Noctarius <me...@noctarius.com>.
Am 29.09.2012 21:15, schrieb Roman Levenstein:
> Hi,
>
> I'm just wondering if lightning uses Unsafe only for
> serializers (i.e. for reading/writing fields of objects) or
> also for the input/output streams used to read/write to/from
> off-heap memory? Using DirectBuffers "as is" for accessing
> off-heap memory is actually rather slow.
>
Internally lightning uses DataInput / DataOutput for saving data
that means actually ByteStreams (or whatever is in the backing
Input- / Outputstream is used is utelized.
At the moment lightning has no off heap backend, that was
implemented in an additional project.
Lightning uses a combination of bytecode generation (for
generating the marshaller classes) and Unsafe for field access.
Using Unsafe to directly write to off heap will fasten the
serialization time as well (I think).
> Let me share some experiences with you: I have implemented a
> similar feature for Kryo. It is not in the trunk yet, as it is
> being reviewed by Nate, the author of Kryo. One of the things
> that I noticed is that using Unsafe only for serializers is
> nice, but does not provide much improvement. At the same time,
> using Unsafe for input/output streams and reading/writing
> directly from/into off-heap memory provides significant
> performance benefits. My patch for Kryo implements both
> features using Unsafe. If you want to see more details about
> the implementation and findings, please have a look at:
> http://code.google.com/p/kryo/issues/detail?id=75
>
> I hope it will land in Kryo trunk soon. Once it is there, I
> think it could be interesting to add a Kryo-based serialization
> to DirectMemory. What do you think? But independent of this,
> feel free to take any ideas or even the implementation from the
> patch. The off-heap memory streams implementation is rather
> self-contained and could be reused with other frameworks
> besides Kryo.
>
I guess it would be very interesting to take a look at your
implemention so I'll do so the next days.
> Regards, Roman
>
> On Sat, Sep 29, 2012 at 8:46 PM, Noctarius <me...@noctarius.com>
> wrote:
>> Actually all dependencies should be AL2 or BSD licensed :-)
>>
>> Am 29.09.2012 20:42, schrieb Raffaele P. Guidi:
>>>>> Hehe well that really sounds like a nice bunch of
>>>>> people.
>>>
>>> Indeed they are (I'm a newbie as well and try to do my
>>> best)
>>>
>>>>> If lightning will be a sub-part (sub-project) of DM, do
>>>>> I need to write
>>> an project purposal?
>>>
>>> Nope, not needed for a sub-project
>>>
>>>>> Do I need to make any changes to the pom.xml like
>>>>> adding a
>>> special parent pom or anything like that?
>>>
>>> Not for the serializer - just have to take a look at
>>> project dependencies - or, better, at their licenses - are
>>> they compatible with the ASL 2.0? i.e. a GPL'd library is
>>> not a good fit and should be replaced with an apache
>>> licensed (or BSD, or MIT...) one if possible. For the
>>> integration module is a separate story - you should start
>>> off copying one of the other serializers and reusing the
>>> same pom and directory structure.
>>>
>>> Pleased to meet you, Chris :)
>>>
>>> Ciao, R
>>>
>>> On Sat, Sep 29, 2012 at 8:24 PM, Noctarius
>>> <me...@noctarius.com> wrote:
>>>
>>>> Hehe well that really sounds like a nice bunch of
>>>> people.
>>>>
>>>> Ok to be true I couldn't wait until tomorrow and started
>>>> already reading the links. From what I was reading: If
>>>> lightning will be a sub-part (sub-project) of DM, do I
>>>> need to write an project purposal?
>>>>
>>>> Do I need to make any changes to the pom.xml like adding
>>>> a special parent pom or anything like that?
>>>>
>>>> In general: there are a lot things to know :-)
>>>>
>>>> Am 29.09.2012 19:59, schrieb Raffaele P. Guidi:
>>>>> Negative part of ASF membership? You get together with
>>>>> a lot of geeky, talented people with a fixation for
>>>>> software and open source. Oh wait but this is actually
>>>>> nice! :-D Il giorno 29/set/2012 19:05, "Olivier Lamy"
>>>>> <ol...@apache.org> ha
>>>> scritto:
>>>>>
>>>>>> 2012/9/29 Noctarius <me...@noctarius.com>:
>>>>>>> Thanks Olivier for carify, I'll take a look in it
>>>>>>> tomorrow but there's just one question left (for
>>>>>>> now ;)): What is that vote for becoming a
>>>>>>> committer? What if the vote will be negative?
>>>>>> The vote is on private list (pmc list for privacy
>>>>>> reasons and possible negative stuff being on public
>>>>>> lists)
>>>>>>> Until now I just used Apache stuff, was never
>>>>>>> interested in being part of it so I guess it can
>>>>>>> be negative for any reason, can't it?
>>>>>> I don't see why it could be negative but suspens
>>>>>> .... :-)
>>>>>>>
>>>>>>> Am 29.09.2012 18:56, schrieb Olivier Lamy:
>>>>>>>> 2012/9/29 Noctarius <me...@noctarius.com>:
>>>>>>>>> Nope my real name is Christoph Engelbert, but
>>>>>>>>> Noctarius is the all time nick :)
>>>>>>>>>
>>>>>>>>> Renaming the package should be no problem, is
>>>>>>>>> it "org.apache.directmemory.lightning" or what
>>>>>>>>> would it be?
>>>>>>>> fine for me
>>>>>>>>> Then there needs to be a change in the license
>>>>>>>>> header as Olivier mentioned, that means just
>>>>>>>>> remove the first sentence or is there anything
>>>>>>>>> more to do (maybe it's easiest thing to just
>>>>>>>>> copy the header from DM file ;))?
>>>>>>>> yup use same header as DM
>>>>>>>>>
>>>>>>>>> The CLA is just a form to clarify that the
>>>>>>>>> source can be contributed to the Apache
>>>>>>>>> Foundation?
>>>>>>>> yup correct.
>>>>>>>>>
>>>>>>>>> The final step will be attaching the patch in
>>>>>>>>> form of a huge diff file?
>>>>>>>> yes
>>>>>>>>>
>>>>>>>>> And what is the way to apply for a membership?
>>>>>>>>> Never thought about how to do that.
>>>>>>>> Read here
>>>>>>>> http://apache.org/foundation/how-it-works.html
>>>>>>>> and here
>>>>>>>> http://apache.org/foundation/getinvolved.html .
>>>>>>>> And feel free to ask any questions :-)
>>>>>>>>>
>>>>>>>>> Chris
>>>>>>>>>
>>>>>>>>> Am 29.09.2012 18:23, schrieb Raffaele P.
>>>>>>>>> Guidi:
>>>>>>>>>> OK, deal, at least for me ;-) I propose you
>>>>>>>>>> rename the packages, produce a patch for this
>>>>>>>>>> and the new serializer module (should be
>>>>>>>>>> simple enough starting from an existing one)
>>>>>>>>>> and, in the meanwhile, apply for ASF
>>>>>>>>>> membership. Is IP clearance needed? I guess
>>>>>>>>>> yes. After this we will come up with a formal
>>>>>>>>>> vote regarding Noctarius (is this your real
>>>>>>>>>> name?!) allowance in the project team.
>>>>>>>>>>
>>>>>>>>>> Good times are gonna come :-)
>>>>>>>>>>
>>>>>>>>>> Thanks, R Il giorno 29/set/2012 17:58,
>>>>>>>>>> "Olivier Lamy" <ol...@apache.org> ha
>>>>>>>>>> scritto:
>>>>>>>>>>
>>>>>>>>>>> 2012/9/29 Raffaele P. Guidi
>>>>>>>>>>> <ra...@gmail.com>:
>>>>>>>>>>>> Well we already have a NIO ready
>>>>>>>>>>>> interface allowing direct access to DMs
>>>>>>>>>>>> managed bytebuffers but I think this is
>>>>>>>>>>>> just half way to what could be achieved
>>>>>>>>>>>> optimally blending serialization and
>>>>>>>>>>>> memory allocation together.
>>>>>>>>>>>>
>>>>>>>>>>>> Lightning as a module is of course a
>>>>>>>>>>>> good idea and it could easily evolve as
>>>>>>>>>>>> a subproject (for the more experienced
>>>>>>>>>>>> asf members: is it a feasible way?).
>>>>>>>>>>> Nothing prevent to have
>>>>>>>>>>> http://svn.apache.org/repos/asf/directmemory/lightning/trunk
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>
>>>>>>>>>>>
>>
>>>>>>>>>>>
with a package like: o.a.d.lightning That will be a
>>>>>>>>>>> subproject.
>>>>>>>>>>>>
>>>>>>>>>>>> Ciao, R Il giorno 29/set/2012 17:44,
>>>>>>>>>>>> "Noctarius" <me...@noctarius.com> ha
>>>>>>>>>>>> scritto:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hey guys,
>>>>>>>>>>>>>
>>>>>>>>>>>>> ok there's no lightning binary
>>>>>>>>>>>>> available since lightning wasn't ready
>>>>>>>>>>>>> yet for releasing.
>>>>>>>>>>>>>
>>>>>>>>>>>>> For being the only developer it would
>>>>>>>>>>>>> be no problem to contribute the
>>>>>>>>>>>>> sourcebase for DirectMemory but I'm not
>>>>>>>>>>>>> sure yet if it wouldn't be better to
>>>>>>>>>>>>> seperate it to be available without
>>>>>>>>>>>>> using DirectMemory itself. I started it
>>>>>>>>>>>>> as a serializer for cluster
>>>>>>>>>>>>> synchronization, but it would be cool
>>>>>>>>>>>>> to contribute lightning as a subproject
>>>>>>>>>>>>> to DirectMemory :-)
>>>>>>>>>>>>>
>>>>>>>>>>>>> About the second project I would love
>>>>>>>>>>>>> to see a public available buffer API
>>>>>>>>>>>>> directly in DirectMemory so that
>>>>>>>>>>>>> project would be nearly needless :-)
>>>>>>>>>>>>> The only difference I think is the
>>>>>>>>>>>>> allocation strategy my implementation
>>>>>>>>>>>>> is using against the one DirectMemory
>>>>>>>>>>>>> has, but I'm pretty sure the allocation
>>>>>>>>>>>>> is extensible ;-)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Like I said, for both projects I'm the
>>>>>>>>>>>>> only dev so there would be no IP
>>>>>>>>>>>>> problem. So if it's ok to you to not
>>>>>>>>>>>>> include lightning directly in DM I
>>>>>>>>>>>>> would be glad to contribute to the
>>>>>>>>>>>>> Apache Foundation :-)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers Chris
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Am 29.09.2012 16:53, schrieb Raffaele
>>>>>>>>>>>>> P. Guidi:
>>>>>>>>>>>>>> Ok so it's up to noctarius - your
>>>>>>>>>>>>>> move! ;-) Regarding the new unsafe
>>>>>>>>>>>>>> storage: it's an opt-in feature that
>>>>>>>>>>>>>> can be set with the fluent API
>>>>>>>>>>> (and
>>>>>>>>>>>>>> soon through the conference file).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Ciao, R Il giorno 29/set/2012 16:45,
>>>>>>>>>>>>>> "Olivier Lamy" <ol...@apache.org> ha
>>>>>>>>>>>>> scritto:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2012/9/29 Raffaele P. Guidi
>>>>>>>>>>>>>>> <ra...@gmail.com>:
>>>>>>>>>>>>>>>>>> At least for the moment he
>>>>>>>>>>>>>>>>>> can provide a patch to be
>>>>>>>>>>>>>>>>>> integrated
>>>>>>>>>>> in DM
>>>>>>>>>>>>>>> :-)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Sure, but as lightning is not in
>>>>>>>>>>>>>>>> any public mvn repo should its
>>>>>>>>>>> code be
>>>>>>>>>>>>>>>> re-published in our svn? Or
>>>>>>>>>>>>>>>> what?
>>>>>>>>>>>>>>> @Apache we don't care about
>>>>>>>>>>>>>>> binaries, only sources are
>>>>>>>>>>>>>>> important ! (a bit theorical for
>>>>>>>>>>>>>>> sure but that's it :-) ). So if
>>>>>>>>>>>>>>> Noctarius was the only guy who
>>>>>>>>>>>>>>> participate in lightning. He can
>>>>>>>>>>>>>>> just provide a patch we could
>>>>>>>>>>>>>>> integrate as a new dm module (note:
>>>>>>>>>>>>>>> the patch must not contains any
>>>>>>>>>>>>>>> more copyright and all sources must
>>>>>>>>>>>>>>> have ASF licenses).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> "Copyright 2012 the original author
>>>>>>>>>>>>>>> or authors." must be removed. And
>>>>>>>>>>>>>>> BTW package must be changed :-)
>>>>>>>>>>>>>>> (com.github is not acceptable
>>>>>>>>>>> @asf
>>>>>>>>>>>>>>> :-) )(@Noctarius are you working
>>>>>>>>>>>>>>> for github ? :-) )
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> And having him as a committer will
>>>>>>>>>>>>>>> be only a matter of voting (we
>>>>>>>>>>> have
>>>>>>>>>>>>>>> a great chair who take cares of
>>>>>>>>>>>>>>> administrative stuff :P )
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> If some others have participated in
>>>>>>>>>>>>>>> the project, we must pass tru an
>>>>>>>>>>>>>>> ip clearance mechanism
>>>>>>>>>>>>>>> (http://incubator.apache.org/ip-clearance/index.html)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>
>>>>>>>>>>>>>>>
>>
>>>>>>>>>>>>>>>
and all contributors to lightning must provide a cla.
>>>>>>>>>>>>>>> (It it's the case I can help)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> perso I'd like we avoid hard
>>>>>>>>>>>>>>>>>> dependency on Unsafe as
>>>>>>>>>>>>>>>>>> maybe some
>>>>>>>>>>> use
>>>>>>>>>>>>>>>> other jdks :-)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Well, I believe Unsafe is
>>>>>>>>>>>>>>>> supported by Sun JDK, OpenJDK,
>>>>>>>>>>>>>>>> IBM JDK and JRockit - and I
>>>>>>>>>>>>>>>> believe that it is more than
>>>>>>>>>>>>>>>> enough. Also keep in
>>>>>>>>>>> mind
>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>> we already have an alternative
>>>>>>>>>>>>>>>> Unsafe based memory storage -
>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>> although
>>>>>>>>>>>>>>>> it's not thoroughly tested for
>>>>>>>>>>>>>>>> performance it dramaticaly
>>>>>>>>>>>>>>>> simplifies
>>>>>>>>>>>>>>> code,
>>>>>>>>>>>>>>>> I have great expectations about
>>>>>>>>>>>>>>>> it.
>>>>>>>>>>>>>>> Me too :-). Yup definitely more
>>>>>>>>>>>>>>> simple and faster ! But we must
>>>>>>>>>>>>>>> provide a switch off configuration
>>>>>>>>>>>>>>> mechanism if some
>>>>>>>>>>> users
>>>>>>>>>>>>>>> don't want to use that (because
>>>>>>>>>>>>>>> Unsafe is just "Unsafe" :-) ) And
>>>>>>>>>>>>>>> sorry I didn't have a look yet at
>>>>>>>>>>>>>>> your changes with using Unsafe.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Cheers, R
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Sat, Sep 29, 2012 at 4:03 PM,
>>>>>>>>>>>>>>>> Olivier Lamy <ol...@apache.org>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2012/9/29 Raffaele P. Guidi
>>>>>>>>>>>>>>>>> <ra...@gmail.com>:
>>>>>>>>>>>>>>>>>> What do you think about: 1)
>>>>>>>>>>>>>>>>>> implementing a lightning
>>>>>>>>>>>>>>>>>> serialization module 2)
>>>>>>>>>>>>>>>>>> creating a serializer that
>>>>>>>>>>>>>>>>>> directly works on a
>>>>>>>>>>>>>>>>>> directmemory
>>>>>>>>>>>>>>> provider
>>>>>>>>>>>>>>>>>> ByteBuffer or (maybe better)
>>>>>>>>>>>>>>>>>> Unsafe based Pointer?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Sounds good (perso I'd like we
>>>>>>>>>>>>>>>>> avoid hard dependency on Unsafe
>>>>>>>>>>>>>>>>> as maybe some use other jdks
>>>>>>>>>>>>>>>>> :-) )
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Now I see lightning is
>>>>>>>>>>>>>>>>>> apache licensed and this is
>>>>>>>>>>>>>>>>>> fine but I
>>>>>>>>>>> don't
>>>>>>>>>>>>>>> think
>>>>>>>>>>>>>>>>>> it is published in any
>>>>>>>>>>>>>>>>>> public maven repo, am I
>>>>>>>>>>>>>>>>>> right? We could
>>>>>>>>>>> find a
>>>>>>>>>>>>>>> way
>>>>>>>>>>>>>>>>>> to deal with this; options
>>>>>>>>>>>>>>>>>> vary from publishing
>>>>>>>>>>>>>>>>>> lightning to the
>>>>>>>>>>> free
>>>>>>>>>>>>>>>>>> sonatype repo, joining the
>>>>>>>>>>>>>>>>>> ASF (which is great anyhow!)
>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>> republishing
>>>>>>>>>>>>>>>>>> lightning code in
>>>>>>>>>>>>>>>>>> DirectMemory becoming a
>>>>>>>>>>>>>>>>>> commiter (which has to
>>>>>>>>>>>>>>> undergo
>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>> PMC vote).
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> At least for the moment he can
>>>>>>>>>>>>>>>>> provide a patch to be
>>>>>>>>>>>>>>>>> integrated in
>>>>>>>>>>> DM
>>>>>>>>>>>>>>> :-)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I'd like to hear your and
>>>>>>>>>>>>>>>>>> the team feelings on this.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks, Raffaele
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Sat, Sep 29, 2012 at 3:27
>>>>>>>>>>>>>>>>>> PM, Noctarius
>>>>>>>>>>>>>>>>>> <me...@noctarius.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hey Raffaele,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> that's quite similar to
>>>>>>>>>>>>>>>>>>> what I did at work. We're
>>>>>>>>>>>>>>>>>>> developing
>>>>>>>>>>> Flash
>>>>>>>>>>>>>>>>>>> online games and using a
>>>>>>>>>>>>>>>>>>> customized AMF
>>>>>>>>>>>>>>>>>>> serialization. To support
>>>>>>>>>>>>>>>>>>> async writing of the
>>>>>>>>>>>>>>>>>>> clients event results I
>>>>>>>>>>>>>>>>>>> added serialization
>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>> the components / entities
>>>>>>>>>>>>>>>>>>> to the players zone
>>>>>>>>>>>>>>>>>>> calculation and
>>>>>>>>>>> stored
>>>>>>>>>>>>>>>>>>> the pre-serialized
>>>>>>>>>>>>>>>>>>> bytestream directly to the
>>>>>>>>>>>>>>>>>>> off-heap (using
>>>>>>>>>>>>>>>>>>> direct-ring-cache
>>>>>>>>>>>>>>>>>>> implementation). When the
>>>>>>>>>>>>>>>>>>> client requests the
>>>>>>>>>>>>>>>>>>> results (using
>>>>>>>>>>>>>>>>>>> long-polling) I just write
>>>>>>>>>>>>>>>>>>> the pre-serialized
>>>>>>>>>>> data to
>>>>>>>>>>>>>>>>>>> the right position to
>>>>>>>>>>>>>>>>>>> deserialize it by standard
>>>>>>>>>>>>>>>>>>> ways on Flash
>>>>>>>>>>> side.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> So yeah an seriliaztion to
>>>>>>>>>>>>>>>>>>> off-heap algorithm would
>>>>>>>>>>>>>>>>>>> be fine. You
>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>> avoid using byte arrays
>>>>>>>>>>>>>>>>>>> and minimalize GC.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Cheers Chris
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Am 29.09.2012 15:02,
>>>>>>>>>>>>>>>>>>> schrieb Raffaele P. Guidi:
>>>>>>>>>>>>>>>>>>>> Nice to hear back from
>>>>>>>>>>>>>>>>>>>> you! Yes, I was thinking
>>>>>>>>>>>>>>>>>>>> about creating a
>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>> memory
>>>>>>>>>>>>>>>>>>>> storage implementation
>>>>>>>>>>>>>>>>>>>> using Unsafe (and I did
>>>>>>>>>>>>>>>>>>>> it, recently) and
>>>>>>>>>>>>>>>>> also, as
>>>>>>>>>>>>>>>>>>>> DirectMemory relies
>>>>>>>>>>>>>>>>>>>> heavily on serialization
>>>>>>>>>>>>>>>>>>>> (and supports many
>>>>>>>>>>> of
>>>>>>>>>>>>>>>>> them,
>>>>>>>>>>>>>>>>>>>> protostuff, protobuf,
>>>>>>>>>>>>>>>>>>>> msgpack and of course
>>>>>>>>>>>>>>>>>>>> standard
>>>>>>>>>>>>>>> serialization),
>>>>>>>>>>>>>>>>>>> about
>>>>>>>>>>>>>>>>>>>> creating a simple
>>>>>>>>>>>>>>>>>>>> embedded serializer
>>>>>>>>>>>>>>>>>>>> leveraging the same
>>>>>>>>>>>>>>> techniques
>>>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>>>>>> used (Unsafe and
>>>>>>>>>>>>>>>>>>>> bytecode generation). The
>>>>>>>>>>>>>>>>>>>> idea with embedding is
>>>>>>>>>>>>>>>>>>>> avoiding to serialize in
>>>>>>>>>>>>>>>>>>>> a byte array
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>>>>>>> moving the byte array to
>>>>>>>>>>>>>>>>>>>> off-heap memory (via
>>>>>>>>>>>>>>>>>>>> Unsafe or
>>>>>>>>>>>>>>> ByteBuffers),
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>> serializing directly into
>>>>>>>>>>>>>>>>>>>> a "managed" off-heap
>>>>>>>>>>>>>>>>>>>> buffer, thus
>>>>>>>>>>> further
>>>>>>>>>>>>>>>>>>>> optimizing heap
>>>>>>>>>>>>>>>>>>>> utilization (less GC).
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> What do you think about
>>>>>>>>>>>>>>>>>>>> it? Does it make any
>>>>>>>>>>>>>>>>>>>> sense to you?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Ciao, R
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Sat, Sep 29, 2012 at
>>>>>>>>>>>>>>>>>>>> 2:40 PM, Noctarius
>>>>>>>>>>>>>>>>>>>> <me...@noctarius.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hey guys,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Raffaele found out
>>>>>>>>>>>>>>>>>>>>> about a project of mine
>>>>>>>>>>>>>>>>>>>>> (Lightning) a few
>>>>>>>>>>> weeks
>>>>>>>>>>>>>>>>>>>>> ago. Lightning is a
>>>>>>>>>>>>>>>>>>>>> heavy Unsafe and
>>>>>>>>>>>>>>>>>>>>> Bytecode generation
>>>>>>>>>>>>>>>>>>>>> using Serializer
>>>>>>>>>>>>>>>>>>>>> implementation. He told
>>>>>>>>>>>>>>>>>>>>> me that he was
>>>>>>>>>>>>>>>>>>>>> interested in adding
>>>>>>>>>>>>>>>>>>>>> something similar to
>>>>>>>>>>>>>>>>>>>>> DirectMemory and I
>>>>>>>>>>>>>>>>>>>>> would be glad to
>>>>>>>>>>>>>>> help
>>>>>>>>>>>>>>>>>>>>> out in this.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Another project I
>>>>>>>>>>>>>>>>>>>>> started a few days ago,
>>>>>>>>>>>>>>>>>>>>> since it was needed
>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>> work is
>>>>>>>>>>>>>>>>>>>>> DirectRingCache. The
>>>>>>>>>>>>>>>>>>>>> name not really meets
>>>>>>>>>>>>>>>>>>>>> to actual
>>>>>>>>>>>>>>>>>>>>> implementation since
>>>>>>>>>>>>>>>>>>>>> it's not yet a ring
>>>>>>>>>>>>>>>>>>>>> buffer using cache. I
>>>>>>>>>>>>>>> used
>>>>>>>>>>>>>>>>>>>>> this for a
>>>>>>>>>>>>>>>>>>>>> pre-serialization
>>>>>>>>>>>>>>>>>>>>> simple bytestream cache
>>>>>>>>>>>>>>>>>>>>> with self-growing
>>>>>>>>>>>>>>>>>>>>> buffers. It could be
>>>>>>>>>>>>>>>>>>>>> nice to have
>>>>>>>>>>>>>>>>>>>>> DirectMemory
>>>>>>>>>>> having
>>>>>>>>>>>>>>>>>>>>> raw "buffers" to write
>>>>>>>>>>>>>>>>>>>>> to or to read from.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Here are the links
>>>>>>>>>>>>>>>>>>>>> from the projects:
>>>>>>>>>>>>>>>>>>>>> https://github.com/noctarius/Lightning
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>
>>>>>>>>>>>>>>>>>>>>>
https://github.com/noctarius/direct-ring-cache
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>> It would be nice to help out since I really
>>>>>>>>> like the idea of
>>>>>>>>>>>>>>>>>>>>> DirectMemory and since
>>>>>>>>>>>>>>>>>>>>> direct-ring-cache is
>>>>>>>>>>>>>>>>>>>>> some kind of
>>>>>>>>>>>>>>> reinventing
>>>>>>>>>>>>>>>>>>>>> the wheel.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Cheers Noctarius
>>>>>>>>>>>>>>>>>>>>> (Chris)
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> -- Olivier Lamy Talend:
>>>>>>>>>>>>>>>>> http://coders.talend.com
>>>>>>>>>>>>>>>>> http://twitter.com/olamy |
>>>>>>>>>>>>>>>>> http://linkedin.com/in/olamy
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -- Olivier Lamy Talend:
>>>>>>>>>>>>>>> http://coders.talend.com
>>>>>>>>>>>>>>> http://twitter.com/olamy |
>>>>>>>>>>>>>>> http://linkedin.com/in/olamy
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> -- Olivier Lamy Talend:
>>>>>>>>>>> http://coders.talend.com
>>>>>>>>>>> http://twitter.com/olamy |
>>>>>>>>>>> http://linkedin.com/in/olamy
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> -- Olivier Lamy Talend: http://coders.talend.com
>>>>>> http://twitter.com/olamy |
>>>>>> http://linkedin.com/in/olamy
>>>>>>
>>>>>
>>>>
>>>
>>
--
##############################
# A Digital's Life #
##############################
Nickname: Noctarius
Location: Germany
Meet me at:
Ohloh: http://www.ohloh.net/accounts/noctarius
Web: http://www.noctarius.com
XMPP/Jabber: noctarius@jabber.ccc.de
Re: Additional Serializer and raw Buffer access
Posted by Roman Levenstein <ro...@gmail.com>.
Hi,
I'm just wondering if lightning uses Unsafe only for serializers (i.e.
for reading/writing fields of objects) or also for the input/output
streams used to read/write to/from off-heap memory? Using
DirectBuffers "as is" for accessing off-heap memory is actually rather
slow.
Let me share some experiences with you:
I have implemented a similar feature for Kryo. It is not in the trunk
yet, as it is being reviewed by Nate, the author of Kryo.
One of the things that I noticed is that using Unsafe only for
serializers is nice, but does not provide much improvement.
At the same time, using Unsafe for input/output streams and
reading/writing directly from/into off-heap memory provides
significant performance benefits.
My patch for Kryo implements both features using Unsafe. If you want
to see more details about the implementation and findings, please have
a look at:
http://code.google.com/p/kryo/issues/detail?id=75
I hope it will land in Kryo trunk soon. Once it is there, I think it
could be interesting to add a Kryo-based serialization to
DirectMemory. What do you think?
But independent of this, feel free to take any ideas or even the
implementation from the patch. The off-heap memory streams
implementation is rather self-contained and could be reused with other
frameworks besides Kryo.
Regards,
Roman
On Sat, Sep 29, 2012 at 8:46 PM, Noctarius <me...@noctarius.com> wrote:
> Actually all dependencies should be AL2 or BSD licensed :-)
>
> Am 29.09.2012 20:42, schrieb Raffaele P. Guidi:
>>>> Hehe well that really sounds like a nice bunch of people.
>>
>> Indeed they are (I'm a newbie as well and try to do my best)
>>
>>>> If lightning will be a sub-part (sub-project) of DM, do I
>>>> need to write
>> an project purposal?
>>
>> Nope, not needed for a sub-project
>>
>>>> Do I need to make any changes to the pom.xml like adding a
>> special parent pom or anything like that?
>>
>> Not for the serializer - just have to take a look at project
>> dependencies - or, better, at their licenses - are they
>> compatible with the ASL 2.0? i.e. a GPL'd library is not a good
>> fit and should be replaced with an apache licensed (or BSD, or
>> MIT...) one if possible. For the integration module is a
>> separate story - you should start off copying one of the other
>> serializers and reusing the same pom and directory structure.
>>
>> Pleased to meet you, Chris :)
>>
>> Ciao, R
>>
>> On Sat, Sep 29, 2012 at 8:24 PM, Noctarius <me...@noctarius.com>
>> wrote:
>>
>>> Hehe well that really sounds like a nice bunch of people.
>>>
>>> Ok to be true I couldn't wait until tomorrow and started
>>> already reading the links. From what I was reading: If
>>> lightning will be a sub-part (sub-project) of DM, do I need
>>> to write an project purposal?
>>>
>>> Do I need to make any changes to the pom.xml like adding a
>>> special parent pom or anything like that?
>>>
>>> In general: there are a lot things to know :-)
>>>
>>> Am 29.09.2012 19:59, schrieb Raffaele P. Guidi:
>>>> Negative part of ASF membership? You get together with a
>>>> lot of geeky, talented people with a fixation for software
>>>> and open source. Oh wait but this is actually nice! :-D Il
>>>> giorno 29/set/2012 19:05, "Olivier Lamy" <ol...@apache.org>
>>>> ha
>>> scritto:
>>>>
>>>>> 2012/9/29 Noctarius <me...@noctarius.com>:
>>>>>> Thanks Olivier for carify, I'll take a look in it
>>>>>> tomorrow but there's just one question left (for now
>>>>>> ;)): What is that vote for becoming a committer? What
>>>>>> if the vote will be negative?
>>>>> The vote is on private list (pmc list for privacy reasons
>>>>> and possible negative stuff being on public lists)
>>>>>> Until now I just used Apache stuff, was never
>>>>>> interested in being part of it so I guess it can be
>>>>>> negative for any reason, can't it?
>>>>> I don't see why it could be negative but suspens ....
>>>>> :-)
>>>>>>
>>>>>> Am 29.09.2012 18:56, schrieb Olivier Lamy:
>>>>>>> 2012/9/29 Noctarius <me...@noctarius.com>:
>>>>>>>> Nope my real name is Christoph Engelbert, but
>>>>>>>> Noctarius is the all time nick :)
>>>>>>>>
>>>>>>>> Renaming the package should be no problem, is it
>>>>>>>> "org.apache.directmemory.lightning" or what would
>>>>>>>> it be?
>>>>>>> fine for me
>>>>>>>> Then there needs to be a change in the license
>>>>>>>> header as Olivier mentioned, that means just remove
>>>>>>>> the first sentence or is there anything more to do
>>>>>>>> (maybe it's easiest thing to just copy the header
>>>>>>>> from DM file ;))?
>>>>>>> yup use same header as DM
>>>>>>>>
>>>>>>>> The CLA is just a form to clarify that the source
>>>>>>>> can be contributed to the Apache Foundation?
>>>>>>> yup correct.
>>>>>>>>
>>>>>>>> The final step will be attaching the patch in form
>>>>>>>> of a huge diff file?
>>>>>>> yes
>>>>>>>>
>>>>>>>> And what is the way to apply for a membership?
>>>>>>>> Never thought about how to do that.
>>>>>>> Read here
>>>>>>> http://apache.org/foundation/how-it-works.html and
>>>>>>> here http://apache.org/foundation/getinvolved.html .
>>>>>>> And feel free to ask any questions :-)
>>>>>>>>
>>>>>>>> Chris
>>>>>>>>
>>>>>>>> Am 29.09.2012 18:23, schrieb Raffaele P. Guidi:
>>>>>>>>> OK, deal, at least for me ;-) I propose you
>>>>>>>>> rename the packages, produce a patch for this and
>>>>>>>>> the new serializer module (should be simple
>>>>>>>>> enough starting from an existing one) and, in the
>>>>>>>>> meanwhile, apply for ASF membership. Is IP
>>>>>>>>> clearance needed? I guess yes. After this we will
>>>>>>>>> come up with a formal vote regarding Noctarius
>>>>>>>>> (is this your real name?!) allowance in the
>>>>>>>>> project team.
>>>>>>>>>
>>>>>>>>> Good times are gonna come :-)
>>>>>>>>>
>>>>>>>>> Thanks, R Il giorno 29/set/2012 17:58, "Olivier
>>>>>>>>> Lamy" <ol...@apache.org> ha scritto:
>>>>>>>>>
>>>>>>>>>> 2012/9/29 Raffaele P. Guidi
>>>>>>>>>> <ra...@gmail.com>:
>>>>>>>>>>> Well we already have a NIO ready interface
>>>>>>>>>>> allowing direct access to DMs managed
>>>>>>>>>>> bytebuffers but I think this is just half way
>>>>>>>>>>> to what could be achieved optimally blending
>>>>>>>>>>> serialization and memory allocation
>>>>>>>>>>> together.
>>>>>>>>>>>
>>>>>>>>>>> Lightning as a module is of course a good
>>>>>>>>>>> idea and it could easily evolve as a
>>>>>>>>>>> subproject (for the more experienced asf
>>>>>>>>>>> members: is it a feasible way?).
>>>>>>>>>> Nothing prevent to have
>>>>>>>>>> http://svn.apache.org/repos/asf/directmemory/lightning/trunk
>>>>>>>>>>
>>>>>>>>>>
>>>>>>
>>>>>>>>>>
> with a package like: o.a.d.lightning That will be a
>>>>>>>>>> subproject.
>>>>>>>>>>>
>>>>>>>>>>> Ciao, R Il giorno 29/set/2012 17:44,
>>>>>>>>>>> "Noctarius" <me...@noctarius.com> ha scritto:
>>>>>>>>>>>
>>>>>>>>>>>> Hey guys,
>>>>>>>>>>>>
>>>>>>>>>>>> ok there's no lightning binary available
>>>>>>>>>>>> since lightning wasn't ready yet for
>>>>>>>>>>>> releasing.
>>>>>>>>>>>>
>>>>>>>>>>>> For being the only developer it would be no
>>>>>>>>>>>> problem to contribute the sourcebase for
>>>>>>>>>>>> DirectMemory but I'm not sure yet if it
>>>>>>>>>>>> wouldn't be better to seperate it to be
>>>>>>>>>>>> available without using DirectMemory
>>>>>>>>>>>> itself. I started it as a serializer for
>>>>>>>>>>>> cluster synchronization, but it would be
>>>>>>>>>>>> cool to contribute lightning as a
>>>>>>>>>>>> subproject to DirectMemory :-)
>>>>>>>>>>>>
>>>>>>>>>>>> About the second project I would love to
>>>>>>>>>>>> see a public available buffer API directly
>>>>>>>>>>>> in DirectMemory so that project would be
>>>>>>>>>>>> nearly needless :-) The only difference I
>>>>>>>>>>>> think is the allocation strategy my
>>>>>>>>>>>> implementation is using against the one
>>>>>>>>>>>> DirectMemory has, but I'm pretty sure the
>>>>>>>>>>>> allocation is extensible ;-)
>>>>>>>>>>>>
>>>>>>>>>>>> Like I said, for both projects I'm the only
>>>>>>>>>>>> dev so there would be no IP problem. So if
>>>>>>>>>>>> it's ok to you to not include lightning
>>>>>>>>>>>> directly in DM I would be glad to
>>>>>>>>>>>> contribute to the Apache Foundation :-)
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers Chris
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Am 29.09.2012 16:53, schrieb Raffaele P.
>>>>>>>>>>>> Guidi:
>>>>>>>>>>>>> Ok so it's up to noctarius - your move!
>>>>>>>>>>>>> ;-) Regarding the new unsafe storage:
>>>>>>>>>>>>> it's an opt-in feature that can be set
>>>>>>>>>>>>> with the fluent API
>>>>>>>>>> (and
>>>>>>>>>>>>> soon through the conference file).
>>>>>>>>>>>>>
>>>>>>>>>>>>> Ciao, R Il giorno 29/set/2012 16:45,
>>>>>>>>>>>>> "Olivier Lamy" <ol...@apache.org> ha
>>>>>>>>>>>> scritto:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2012/9/29 Raffaele P. Guidi
>>>>>>>>>>>>>> <ra...@gmail.com>:
>>>>>>>>>>>>>>>>> At least for the moment he can
>>>>>>>>>>>>>>>>> provide a patch to be integrated
>>>>>>>>>> in DM
>>>>>>>>>>>>>> :-)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Sure, but as lightning is not in any
>>>>>>>>>>>>>>> public mvn repo should its
>>>>>>>>>> code be
>>>>>>>>>>>>>>> re-published in our svn? Or what?
>>>>>>>>>>>>>> @Apache we don't care about binaries,
>>>>>>>>>>>>>> only sources are important ! (a bit
>>>>>>>>>>>>>> theorical for sure but that's it :-) ).
>>>>>>>>>>>>>> So if Noctarius was the only guy who
>>>>>>>>>>>>>> participate in lightning. He can just
>>>>>>>>>>>>>> provide a patch we could integrate as a
>>>>>>>>>>>>>> new dm module (note: the patch must not
>>>>>>>>>>>>>> contains any more copyright and all
>>>>>>>>>>>>>> sources must have ASF licenses).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> "Copyright 2012 the original author or
>>>>>>>>>>>>>> authors." must be removed. And BTW
>>>>>>>>>>>>>> package must be changed :-) (com.github
>>>>>>>>>>>>>> is not acceptable
>>>>>>>>>> @asf
>>>>>>>>>>>>>> :-) )(@Noctarius are you working for
>>>>>>>>>>>>>> github ? :-) )
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> And having him as a committer will be
>>>>>>>>>>>>>> only a matter of voting (we
>>>>>>>>>> have
>>>>>>>>>>>>>> a great chair who take cares of
>>>>>>>>>>>>>> administrative stuff :P )
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If some others have participated in the
>>>>>>>>>>>>>> project, we must pass tru an ip
>>>>>>>>>>>>>> clearance mechanism
>>>>>>>>>>>>>> (http://incubator.apache.org/ip-clearance/index.html)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>
>>>>>>>>>>>>>>
> and all contributors to lightning must provide a cla.
>>>>>>>>>>>>>> (It it's the case I can help)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> perso I'd like we avoid hard
>>>>>>>>>>>>>>>>> dependency on Unsafe as maybe
>>>>>>>>>>>>>>>>> some
>>>>>>>>>> use
>>>>>>>>>>>>>>> other jdks :-)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Well, I believe Unsafe is supported
>>>>>>>>>>>>>>> by Sun JDK, OpenJDK, IBM JDK and
>>>>>>>>>>>>>>> JRockit - and I believe that it is
>>>>>>>>>>>>>>> more than enough. Also keep in
>>>>>>>>>> mind
>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>> we already have an alternative Unsafe
>>>>>>>>>>>>>>> based memory storage - and
>>>>>>>>>>>> although
>>>>>>>>>>>>>>> it's not thoroughly tested for
>>>>>>>>>>>>>>> performance it dramaticaly
>>>>>>>>>>>>>>> simplifies
>>>>>>>>>>>>>> code,
>>>>>>>>>>>>>>> I have great expectations about it.
>>>>>>>>>>>>>> Me too :-). Yup definitely more simple
>>>>>>>>>>>>>> and faster ! But we must provide a
>>>>>>>>>>>>>> switch off configuration mechanism if
>>>>>>>>>>>>>> some
>>>>>>>>>> users
>>>>>>>>>>>>>> don't want to use that (because Unsafe
>>>>>>>>>>>>>> is just "Unsafe" :-) ) And sorry I
>>>>>>>>>>>>>> didn't have a look yet at your changes
>>>>>>>>>>>>>> with using Unsafe.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Cheers, R
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Sat, Sep 29, 2012 at 4:03 PM,
>>>>>>>>>>>>>>> Olivier Lamy <ol...@apache.org>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2012/9/29 Raffaele P. Guidi
>>>>>>>>>>>>>>>> <ra...@gmail.com>:
>>>>>>>>>>>>>>>>> What do you think about: 1)
>>>>>>>>>>>>>>>>> implementing a lightning
>>>>>>>>>>>>>>>>> serialization module 2) creating
>>>>>>>>>>>>>>>>> a serializer that directly works
>>>>>>>>>>>>>>>>> on a directmemory
>>>>>>>>>>>>>> provider
>>>>>>>>>>>>>>>>> ByteBuffer or (maybe better)
>>>>>>>>>>>>>>>>> Unsafe based Pointer?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Sounds good (perso I'd like we
>>>>>>>>>>>>>>>> avoid hard dependency on Unsafe as
>>>>>>>>>>>>>>>> maybe some use other jdks :-) )
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Now I see lightning is apache
>>>>>>>>>>>>>>>>> licensed and this is fine but I
>>>>>>>>>> don't
>>>>>>>>>>>>>> think
>>>>>>>>>>>>>>>>> it is published in any public
>>>>>>>>>>>>>>>>> maven repo, am I right? We could
>>>>>>>>>> find a
>>>>>>>>>>>>>> way
>>>>>>>>>>>>>>>>> to deal with this; options vary
>>>>>>>>>>>>>>>>> from publishing lightning to the
>>>>>>>>>> free
>>>>>>>>>>>>>>>>> sonatype repo, joining the ASF
>>>>>>>>>>>>>>>>> (which is great anyhow!) and
>>>>>>>>>>>>>> republishing
>>>>>>>>>>>>>>>>> lightning code in DirectMemory
>>>>>>>>>>>>>>>>> becoming a commiter (which has
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> undergo
>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>> PMC vote).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> At least for the moment he can
>>>>>>>>>>>>>>>> provide a patch to be integrated
>>>>>>>>>>>>>>>> in
>>>>>>>>>> DM
>>>>>>>>>>>>>> :-)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I'd like to hear your and the
>>>>>>>>>>>>>>>>> team feelings on this.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks, Raffaele
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Sat, Sep 29, 2012 at 3:27 PM,
>>>>>>>>>>>>>>>>> Noctarius <me...@noctarius.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hey Raffaele,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> that's quite similar to what I
>>>>>>>>>>>>>>>>>> did at work. We're developing
>>>>>>>>>> Flash
>>>>>>>>>>>>>>>>>> online games and using a
>>>>>>>>>>>>>>>>>> customized AMF serialization.
>>>>>>>>>>>>>>>>>> To support async writing of the
>>>>>>>>>>>>>>>>>> clients event results I added
>>>>>>>>>>>>>>>>>> serialization
>>>>>>>>>> of
>>>>>>>>>>>>>>>>>> the components / entities to
>>>>>>>>>>>>>>>>>> the players zone calculation
>>>>>>>>>>>>>>>>>> and
>>>>>>>>>> stored
>>>>>>>>>>>>>>>>>> the pre-serialized bytestream
>>>>>>>>>>>>>>>>>> directly to the off-heap (using
>>>>>>>>>>>>>>>>>> direct-ring-cache
>>>>>>>>>>>>>>>>>> implementation). When the
>>>>>>>>>>>>>>>>>> client requests the results
>>>>>>>>>>>>>>>>>> (using long-polling) I just
>>>>>>>>>>>>>>>>>> write the pre-serialized
>>>>>>>>>> data to
>>>>>>>>>>>>>>>>>> the right position to
>>>>>>>>>>>>>>>>>> deserialize it by standard ways
>>>>>>>>>>>>>>>>>> on Flash
>>>>>>>>>> side.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> So yeah an seriliaztion to
>>>>>>>>>>>>>>>>>> off-heap algorithm would be
>>>>>>>>>>>>>>>>>> fine. You
>>>>>>>>>> can
>>>>>>>>>>>>>>>>>> avoid using byte arrays and
>>>>>>>>>>>>>>>>>> minimalize GC.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Cheers Chris
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Am 29.09.2012 15:02, schrieb
>>>>>>>>>>>>>>>>>> Raffaele P. Guidi:
>>>>>>>>>>>>>>>>>>> Nice to hear back from you!
>>>>>>>>>>>>>>>>>>> Yes, I was thinking about
>>>>>>>>>>>>>>>>>>> creating a
>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>> memory
>>>>>>>>>>>>>>>>>>> storage implementation using
>>>>>>>>>>>>>>>>>>> Unsafe (and I did it,
>>>>>>>>>>>>>>>>>>> recently) and
>>>>>>>>>>>>>>>> also, as
>>>>>>>>>>>>>>>>>>> DirectMemory relies heavily
>>>>>>>>>>>>>>>>>>> on serialization (and
>>>>>>>>>>>>>>>>>>> supports many
>>>>>>>>>> of
>>>>>>>>>>>>>>>> them,
>>>>>>>>>>>>>>>>>>> protostuff, protobuf, msgpack
>>>>>>>>>>>>>>>>>>> and of course standard
>>>>>>>>>>>>>> serialization),
>>>>>>>>>>>>>>>>>> about
>>>>>>>>>>>>>>>>>>> creating a simple embedded
>>>>>>>>>>>>>>>>>>> serializer leveraging the
>>>>>>>>>>>>>>>>>>> same
>>>>>>>>>>>>>> techniques
>>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>>>>> used (Unsafe and bytecode
>>>>>>>>>>>>>>>>>>> generation). The idea with
>>>>>>>>>>>>>>>>>>> embedding is avoiding to
>>>>>>>>>>>>>>>>>>> serialize in a byte array
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>>>>>> moving the byte array to
>>>>>>>>>>>>>>>>>>> off-heap memory (via Unsafe
>>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>> ByteBuffers),
>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>> serializing directly into a
>>>>>>>>>>>>>>>>>>> "managed" off-heap buffer,
>>>>>>>>>>>>>>>>>>> thus
>>>>>>>>>> further
>>>>>>>>>>>>>>>>>>> optimizing heap utilization
>>>>>>>>>>>>>>>>>>> (less GC).
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> What do you think about it?
>>>>>>>>>>>>>>>>>>> Does it make any sense to
>>>>>>>>>>>>>>>>>>> you?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Ciao, R
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Sat, Sep 29, 2012 at 2:40
>>>>>>>>>>>>>>>>>>> PM, Noctarius
>>>>>>>>>>>>>>>>>>> <me...@noctarius.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hey guys,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Raffaele found out about a
>>>>>>>>>>>>>>>>>>>> project of mine (Lightning)
>>>>>>>>>>>>>>>>>>>> a few
>>>>>>>>>> weeks
>>>>>>>>>>>>>>>>>>>> ago. Lightning is a heavy
>>>>>>>>>>>>>>>>>>>> Unsafe and Bytecode
>>>>>>>>>>>>>>>>>>>> generation using
>>>>>>>>>>>>>>>>>>>> Serializer implementation.
>>>>>>>>>>>>>>>>>>>> He told me that he was
>>>>>>>>>>>>>>>>>>>> interested in adding
>>>>>>>>>>>>>>>>>>>> something similar to
>>>>>>>>>>>>>>>>>>>> DirectMemory and I would be
>>>>>>>>>>>>>>>>>>>> glad to
>>>>>>>>>>>>>> help
>>>>>>>>>>>>>>>>>>>> out in this.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Another project I started a
>>>>>>>>>>>>>>>>>>>> few days ago, since it was
>>>>>>>>>>>>>>>>>>>> needed
>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>> work is DirectRingCache.
>>>>>>>>>>>>>>>>>>>> The name not really meets
>>>>>>>>>>>>>>>>>>>> to actual implementation
>>>>>>>>>>>>>>>>>>>> since it's not yet a ring
>>>>>>>>>>>>>>>>>>>> buffer using cache. I
>>>>>>>>>>>>>> used
>>>>>>>>>>>>>>>>>>>> this for a
>>>>>>>>>>>>>>>>>>>> pre-serialization simple
>>>>>>>>>>>>>>>>>>>> bytestream cache with
>>>>>>>>>>>>>>>>>>>> self-growing buffers. It
>>>>>>>>>>>>>>>>>>>> could be nice to have
>>>>>>>>>>>>>>>>>>>> DirectMemory
>>>>>>>>>> having
>>>>>>>>>>>>>>>>>>>> raw "buffers" to write to
>>>>>>>>>>>>>>>>>>>> or to read from.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Here are the links from
>>>>>>>>>>>>>>>>>>>> the projects:
>>>>>>>>>>>>>>>>>>>> https://github.com/noctarius/Lightning
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>
>>>>>>>>>>>>>>>>>>>>
> https://github.com/noctarius/direct-ring-cache
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>> It would be nice to help out since I really like
>>>>>>>> the idea of
>>>>>>>>>>>>>>>>>>>> DirectMemory and since
>>>>>>>>>>>>>>>>>>>> direct-ring-cache is some
>>>>>>>>>>>>>>>>>>>> kind of
>>>>>>>>>>>>>> reinventing
>>>>>>>>>>>>>>>>>>>> the wheel.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Cheers Noctarius (Chris)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -- Olivier Lamy Talend:
>>>>>>>>>>>>>>>> http://coders.talend.com
>>>>>>>>>>>>>>>> http://twitter.com/olamy |
>>>>>>>>>>>>>>>> http://linkedin.com/in/olamy
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -- Olivier Lamy Talend:
>>>>>>>>>>>>>> http://coders.talend.com
>>>>>>>>>>>>>> http://twitter.com/olamy |
>>>>>>>>>>>>>> http://linkedin.com/in/olamy
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> -- Olivier Lamy Talend:
>>>>>>>>>> http://coders.talend.com
>>>>>>>>>> http://twitter.com/olamy |
>>>>>>>>>> http://linkedin.com/in/olamy
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> -- Olivier Lamy Talend: http://coders.talend.com
>>>>> http://twitter.com/olamy | http://linkedin.com/in/olamy
>>>>>
>>>>
>>>
>>
>
Re: Additional Serializer and raw Buffer access
Posted by "Raffaele P. Guidi" <ra...@gmail.com>.
Don't tell me... worked on github for too long :-D
Il giorno 30/set/2012 13:48, "Noctarius" <me...@noctarius.com> ha scritto:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Ok this is the issue url. Hope the patch is ok because it's been
> long since last used diff files :-)
>
> https://issues.apache.org/jira/browse/DIRECTMEMORY-102
>
> Am 30.09.2012 10:40, schrieb Raffaele P. Guidi:
> > OK let's keep them out then Il giorno 30/set/2012 10:26,
> > "Noctarius" <me...@noctarius.com> ha scritto:
> >
> >> Hey it's me again ;-)
> >>
> >> There are at least two files that are not AL2 licensed but
> >> can be used (I think).
> >>
> >>
> >>
> https://github.com/noctarius/Lightning/tree/master/lightning-core/src/main/java/com/github/lightning/internal/instantiator/jrockit
> >>
> >>
> >>
> If it's not possible to leave them in the sourcetree I'm pretty sure
> >> there will be no problem when we remove them since they are
> >> used for older JRocket versions.
> >>
> >> Am 29.09.2012 21:05, schrieb Raffaele P. Guidi:
> >>> I kinda suspected that... Il giorno 29/set/2012 20:47,
> >>> "Noctarius" <me...@noctarius.com> ha scritto:
> >>>
> >>>> Actually all dependencies should be AL2 or BSD licensed
> >>>> :-)
> >>>>
> >>>> Am 29.09.2012 20:42, schrieb Raffaele P. Guidi:
> >>>>>>> Hehe well that really sounds like a nice bunch of
> >>>>>>> people.
> >>>>>
> >>>>> Indeed they are (I'm a newbie as well and try to do my
> >>>>> best)
> >>>>>
> >>>>>>> If lightning will be a sub-part (sub-project) of
> >>>>>>> DM, do I need to write
> >>>>> an project purposal?
> >>>>>
> >>>>> Nope, not needed for a sub-project
> >>>>>
> >>>>>>> Do I need to make any changes to the pom.xml like
> >>>>>>> adding a
> >>>>> special parent pom or anything like that?
> >>>>>
> >>>>> Not for the serializer - just have to take a look at
> >>>>> project dependencies - or, better, at their licenses -
> >>>>> are they compatible with the ASL 2.0? i.e. a GPL'd
> >>>>> library is not a good fit and should be replaced with
> >>>>> an apache licensed (or BSD, or MIT...) one if possible.
> >>>>> For the integration module is a separate story - you
> >>>>> should start off copying one of the other serializers
> >>>>> and reusing the same pom and directory structure.
> >>>>>
> >>>>> Pleased to meet you, Chris :)
> >>>>>
> >>>>> Ciao, R
> >>>>>
> >>>>> On Sat, Sep 29, 2012 at 8:24 PM, Noctarius
> >>>>> <me...@noctarius.com> wrote:
> >>>>>
> >>>>>> Hehe well that really sounds like a nice bunch of
> >>>>>> people.
> >>>>>>
> >>>>>> Ok to be true I couldn't wait until tomorrow and
> >>>>>> started already reading the links. From what I was
> >>>>>> reading: If lightning will be a sub-part
> >>>>>> (sub-project) of DM, do I need to write an project
> >>>>>> purposal?
> >>>>>>
> >>>>>> Do I need to make any changes to the pom.xml like
> >>>>>> adding a special parent pom or anything like that?
> >>>>>>
> >>>>>> In general: there are a lot things to know :-)
> >>>>>>
> >>>>>> Am 29.09.2012 19:59, schrieb Raffaele P. Guidi:
> >>>>>>> Negative part of ASF membership? You get together
> >>>>>>> with a lot of geeky, talented people with a
> >>>>>>> fixation for software and open source. Oh wait but
> >>>>>>> this is actually nice! :-D Il giorno 29/set/2012
> >>>>>>> 19:05, "Olivier Lamy" <ol...@apache.org> ha
> >>>>>> scritto:
> >>>>>>>
> >>>>>>>> 2012/9/29 Noctarius <me...@noctarius.com>:
> >>>>>>>>> Thanks Olivier for carify, I'll take a look in
> >>>>>>>>> it tomorrow but there's just one question left
> >>>>>>>>> (for now ;)): What is that vote for becoming a
> >>>>>>>>> committer? What if the vote will be negative?
> >>>>>>>> The vote is on private list (pmc list for privacy
> >>>>>>>> reasons and possible negative stuff being on
> >>>>>>>> public lists)
> >>>>>>>>> Until now I just used Apache stuff, was never
> >>>>>>>>> interested in being part of it so I guess it
> >>>>>>>>> can be negative for any reason, can't it?
> >>>>>>>> I don't see why it could be negative but suspens
> >>>>>>>> .... :-)
> >>>>>>>>>
> >>>>>>>>> Am 29.09.2012 18:56, schrieb Olivier Lamy:
> >>>>>>>>>> 2012/9/29 Noctarius <me...@noctarius.com>:
> >>>>>>>>>>> Nope my real name is Christoph Engelbert,
> >>>>>>>>>>> but Noctarius is the all time nick :)
> >>>>>>>>>>>
> >>>>>>>>>>> Renaming the package should be no problem,
> >>>>>>>>>>> is it "org.apache.directmemory.lightning"
> >>>>>>>>>>> or what would it be?
> >>>>>>>>>> fine for me
> >>>>>>>>>>> Then there needs to be a change in the
> >>>>>>>>>>> license header as Olivier mentioned, that
> >>>>>>>>>>> means just remove the first sentence or is
> >>>>>>>>>>> there anything more to do (maybe it's
> >>>>>>>>>>> easiest thing to just copy the header from
> >>>>>>>>>>> DM file ;))?
> >>>>>>>>>> yup use same header as DM
> >>>>>>>>>>>
> >>>>>>>>>>> The CLA is just a form to clarify that the
> >>>>>>>>>>> source can be contributed to the Apache
> >>>>>>>>>>> Foundation?
> >>>>>>>>>> yup correct.
> >>>>>>>>>>>
> >>>>>>>>>>> The final step will be attaching the patch
> >>>>>>>>>>> in form of a huge diff file?
> >>>>>>>>>> yes
> >>>>>>>>>>>
> >>>>>>>>>>> And what is the way to apply for a
> >>>>>>>>>>> membership? Never thought about how to do
> >>>>>>>>>>> that.
> >>>>>>>>>> Read here
> >>>>>>>>>> http://apache.org/foundation/how-it-works.html
> >>>>>>>>>> and here
> >>>>>>>>>> http://apache.org/foundation/getinvolved.html
> >>>>>>>>>> . And feel free to ask any questions :-)
> >>>>>>>>>>>
> >>>>>>>>>>> Chris
> >>>>>>>>>>>
> >>>>>>>>>>> Am 29.09.2012 18:23, schrieb Raffaele P.
> >>>>>>>>>>> Guidi:
> >>>>>>>>>>>> OK, deal, at least for me ;-) I propose
> >>>>>>>>>>>> you rename the packages, produce a patch
> >>>>>>>>>>>> for this and the new serializer module
> >>>>>>>>>>>> (should be simple enough starting from an
> >>>>>>>>>>>> existing one) and, in the meanwhile,
> >>>>>>>>>>>> apply for ASF membership. Is IP clearance
> >>>>>>>>>>>> needed? I guess yes. After this we will
> >>>>>>>>>>>> come up with a formal vote regarding
> >>>>>>>>>>>> Noctarius (is this your real name?!)
> >>>>>>>>>>>> allowance in the project team.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Good times are gonna come :-)
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks, R Il giorno 29/set/2012 17:58,
> >>>>>>>>>>>> "Olivier Lamy" <ol...@apache.org> ha
> >>>>>>>>>>>> scritto:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> 2012/9/29 Raffaele P. Guidi
> >>>>>>>>>>>>> <ra...@gmail.com>:
> >>>>>>>>>>>>>> Well we already have a NIO ready
> >>>>>>>>>>>>>> interface allowing direct access to
> >>>>>>>>>>>>>> DMs managed bytebuffers but I think
> >>>>>>>>>>>>>> this is just half way to what could
> >>>>>>>>>>>>>> be achieved optimally blending
> >>>>>>>>>>>>>> serialization and memory allocation
> >>>>>>>>>>>>>> together.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Lightning as a module is of course a
> >>>>>>>>>>>>>> good idea and it could easily evolve
> >>>>>>>>>>>>>> as a subproject (for the more
> >>>>>>>>>>>>>> experienced asf members: is it a
> >>>>>>>>>>>>>> feasible way?).
> >>>>>>>>>>>>> Nothing prevent to have
> >>>>>>>>>>>>> http://svn.apache.org/repos/asf/directmemory/lightning/trunk
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>>>>
> >>>>
> >>>>>>>>>>>>>
> with a package like: o.a.d.lightning That will be a
> >>>>>>>>>>>>> subproject.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Ciao, R Il giorno 29/set/2012 17:44,
> >>>>>>>>>>>>>> "Noctarius" <me...@noctarius.com> ha
> >>>>>>>>>>>>>> scritto:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hey guys,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> ok there's no lightning binary
> >>>>>>>>>>>>>>> available since lightning wasn't
> >>>>>>>>>>>>>>> ready yet for releasing.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> For being the only developer it
> >>>>>>>>>>>>>>> would be no problem to contribute
> >>>>>>>>>>>>>>> the sourcebase for DirectMemory but
> >>>>>>>>>>>>>>> I'm not sure yet if it wouldn't be
> >>>>>>>>>>>>>>> better to seperate it to be
> >>>>>>>>>>>>>>> available without using
> >>>>>>>>>>>>>>> DirectMemory itself. I started it
> >>>>>>>>>>>>>>> as a serializer for cluster
> >>>>>>>>>>>>>>> synchronization, but it would be
> >>>>>>>>>>>>>>> cool to contribute lightning as a
> >>>>>>>>>>>>>>> subproject to DirectMemory :-)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> About the second project I would
> >>>>>>>>>>>>>>> love to see a public available
> >>>>>>>>>>>>>>> buffer API directly in DirectMemory
> >>>>>>>>>>>>>>> so that project would be nearly
> >>>>>>>>>>>>>>> needless :-) The only difference I
> >>>>>>>>>>>>>>> think is the allocation strategy
> >>>>>>>>>>>>>>> my implementation is using against
> >>>>>>>>>>>>>>> the one DirectMemory has, but I'm
> >>>>>>>>>>>>>>> pretty sure the allocation is
> >>>>>>>>>>>>>>> extensible ;-)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Like I said, for both projects I'm
> >>>>>>>>>>>>>>> the only dev so there would be no
> >>>>>>>>>>>>>>> IP problem. So if it's ok to you to
> >>>>>>>>>>>>>>> not include lightning directly in
> >>>>>>>>>>>>>>> DM I would be glad to contribute to
> >>>>>>>>>>>>>>> the Apache Foundation :-)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Cheers Chris
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Am 29.09.2012 16:53, schrieb
> >>>>>>>>>>>>>>> Raffaele P. Guidi:
> >>>>>>>>>>>>>>>> Ok so it's up to noctarius - your
> >>>>>>>>>>>>>>>> move! ;-) Regarding the new
> >>>>>>>>>>>>>>>> unsafe storage: it's an opt-in
> >>>>>>>>>>>>>>>> feature that can be set with the
> >>>>>>>>>>>>>>>> fluent API
> >>>>>>>>>>>>> (and
> >>>>>>>>>>>>>>>> soon through the conference
> >>>>>>>>>>>>>>>> file).
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Ciao, R Il giorno 29/set/2012
> >>>>>>>>>>>>>>>> 16:45, "Olivier Lamy"
> >>>>>>>>>>>>>>>> <ol...@apache.org> ha
> >>>>>>>>>>>>>>> scritto:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> 2012/9/29 Raffaele P. Guidi
> >>>>>>>>>>>>>>>>> <ra...@gmail.com>:
> >>>>>>>>>>>>>>>>>>>> At least for the moment
> >>>>>>>>>>>>>>>>>>>> he can provide a patch to
> >>>>>>>>>>>>>>>>>>>> be integrated
> >>>>>>>>>>>>> in DM
> >>>>>>>>>>>>>>>>> :-)
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Sure, but as lightning is not
> >>>>>>>>>>>>>>>>>> in any public mvn repo should
> >>>>>>>>>>>>>>>>>> its
> >>>>>>>>>>>>> code be
> >>>>>>>>>>>>>>>>>> re-published in our svn? Or
> >>>>>>>>>>>>>>>>>> what?
> >>>>>>>>>>>>>>>>> @Apache we don't care about
> >>>>>>>>>>>>>>>>> binaries, only sources are
> >>>>>>>>>>>>>>>>> important ! (a bit theorical
> >>>>>>>>>>>>>>>>> for sure but that's it :-) ).
> >>>>>>>>>>>>>>>>> So if Noctarius was the only
> >>>>>>>>>>>>>>>>> guy who participate in
> >>>>>>>>>>>>>>>>> lightning. He can just provide
> >>>>>>>>>>>>>>>>> a patch we could integrate as
> >>>>>>>>>>>>>>>>> a new dm module (note: the
> >>>>>>>>>>>>>>>>> patch must not contains any
> >>>>>>>>>>>>>>>>> more copyright and all sources
> >>>>>>>>>>>>>>>>> must have ASF licenses).
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> "Copyright 2012 the original
> >>>>>>>>>>>>>>>>> author or authors." must be
> >>>>>>>>>>>>>>>>> removed. And BTW package must
> >>>>>>>>>>>>>>>>> be changed :-) (com.github is
> >>>>>>>>>>>>>>>>> not acceptable
> >>>>>>>>>>>>> @asf
> >>>>>>>>>>>>>>>>> :-) )(@Noctarius are you
> >>>>>>>>>>>>>>>>> working for github ? :-) )
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> And having him as a committer
> >>>>>>>>>>>>>>>>> will be only a matter of voting
> >>>>>>>>>>>>>>>>> (we
> >>>>>>>>>>>>> have
> >>>>>>>>>>>>>>>>> a great chair who take cares
> >>>>>>>>>>>>>>>>> of administrative stuff :P )
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> If some others have
> >>>>>>>>>>>>>>>>> participated in the project, we
> >>>>>>>>>>>>>>>>> must pass tru an ip clearance
> >>>>>>>>>>>>>>>>> mechanism
> >>>>>>>>>>>>>>>>> (http://incubator.apache.org/ip-clearance/index.html)
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>
> >>>>>>>>>>>>>>>>>
> and all contributors to lightning must provide a cla.
> >>>>>>>>>>>>>>>>> (It it's the case I can help)
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> perso I'd like we avoid
> >>>>>>>>>>>>>>>>>>>> hard dependency on Unsafe
> >>>>>>>>>>>>>>>>>>>> as maybe some
> >>>>>>>>>>>>> use
> >>>>>>>>>>>>>>>>>> other jdks :-)
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Well, I believe Unsafe is
> >>>>>>>>>>>>>>>>>> supported by Sun JDK,
> >>>>>>>>>>>>>>>>>> OpenJDK, IBM JDK and JRockit
> >>>>>>>>>>>>>>>>>> - and I believe that it is
> >>>>>>>>>>>>>>>>>> more than enough. Also keep
> >>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>> mind
> >>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>> we already have an
> >>>>>>>>>>>>>>>>>> alternative Unsafe based
> >>>>>>>>>>>>>>>>>> memory storage - and
> >>>>>>>>>>>>>>> although
> >>>>>>>>>>>>>>>>>> it's not thoroughly tested
> >>>>>>>>>>>>>>>>>> for performance it
> >>>>>>>>>>>>>>>>>> dramaticaly simplifies
> >>>>>>>>>>>>>>>>> code,
> >>>>>>>>>>>>>>>>>> I have great expectations
> >>>>>>>>>>>>>>>>>> about it.
> >>>>>>>>>>>>>>>>> Me too :-). Yup definitely more
> >>>>>>>>>>>>>>>>> simple and faster ! But we must
> >>>>>>>>>>>>>>>>> provide a switch off
> >>>>>>>>>>>>>>>>> configuration mechanism if
> >>>>>>>>>>>>>>>>> some
> >>>>>>>>>>>>> users
> >>>>>>>>>>>>>>>>> don't want to use that (because
> >>>>>>>>>>>>>>>>> Unsafe is just "Unsafe" :-) )
> >>>>>>>>>>>>>>>>> And sorry I didn't have a look
> >>>>>>>>>>>>>>>>> yet at your changes with using
> >>>>>>>>>>>>>>>>> Unsafe.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Cheers, R
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Sat, Sep 29, 2012 at 4:03
> >>>>>>>>>>>>>>>>>> PM, Olivier Lamy
> >>>>>>>>>>>>>>>>>> <ol...@apache.org>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> 2012/9/29 Raffaele P.
> >>>>>>>>>>>>>>>>>>> Guidi
> >>>>>>>>>>>>>>>>>>> <ra...@gmail.com>:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> What do you think about: 1)
> >>>>>>>>>>>>>>>>>>>> implementing a lightning
> >>>>>>>>>>>>>>>>>>>> serialization module 2)
> >>>>>>>>>>>>>>>>>>>> creating a serializer
> >>>>>>>>>>>>>>>>>>>> that directly works on a
> >>>>>>>>>>>>>>>>>>>> directmemory
> >>>>>>>>>>>>>>>>> provider
> >>>>>>>>>>>>>>>>>>>> ByteBuffer or (maybe
> >>>>>>>>>>>>>>>>>>>> better) Unsafe based
> >>>>>>>>>>>>>>>>>>>> Pointer?
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Sounds good (perso I'd like
> >>>>>>>>>>>>>>>>>>> we avoid hard dependency on
> >>>>>>>>>>>>>>>>>>> Unsafe as maybe some use
> >>>>>>>>>>>>>>>>>>> other jdks :-) )
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Now I see lightning is
> >>>>>>>>>>>>>>>>>>>> apache licensed and this
> >>>>>>>>>>>>>>>>>>>> is fine but I
> >>>>>>>>>>>>> don't
> >>>>>>>>>>>>>>>>> think
> >>>>>>>>>>>>>>>>>>>> it is published in any
> >>>>>>>>>>>>>>>>>>>> public maven repo, am I
> >>>>>>>>>>>>>>>>>>>> right? We could
> >>>>>>>>>>>>> find a
> >>>>>>>>>>>>>>>>> way
> >>>>>>>>>>>>>>>>>>>> to deal with this;
> >>>>>>>>>>>>>>>>>>>> options vary from
> >>>>>>>>>>>>>>>>>>>> publishing lightning to
> >>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>> free
> >>>>>>>>>>>>>>>>>>>> sonatype repo, joining
> >>>>>>>>>>>>>>>>>>>> the ASF (which is great
> >>>>>>>>>>>>>>>>>>>> anyhow!) and
> >>>>>>>>>>>>>>>>> republishing
> >>>>>>>>>>>>>>>>>>>> lightning code in
> >>>>>>>>>>>>>>>>>>>> DirectMemory becoming a
> >>>>>>>>>>>>>>>>>>>> commiter (which has to
> >>>>>>>>>>>>>>>>> undergo
> >>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>> PMC vote).
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> At least for the moment he
> >>>>>>>>>>>>>>>>>>> can provide a patch to be
> >>>>>>>>>>>>>>>>>>> integrated in
> >>>>>>>>>>>>> DM
> >>>>>>>>>>>>>>>>> :-)
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> I'd like to hear your and
> >>>>>>>>>>>>>>>>>>>> the team feelings on
> >>>>>>>>>>>>>>>>>>>> this.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Thanks, Raffaele
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Sat, Sep 29, 2012 at
> >>>>>>>>>>>>>>>>>>>> 3:27 PM, Noctarius
> >>>>>>>>>>>>>>>>>>>> <me...@noctarius.com>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Hey Raffaele,
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> that's quite similar to
> >>>>>>>>>>>>>>>>>>>>> what I did at work.
> >>>>>>>>>>>>>>>>>>>>> We're developing
> >>>>>>>>>>>>> Flash
> >>>>>>>>>>>>>>>>>>>>> online games and using
> >>>>>>>>>>>>>>>>>>>>> a customized AMF
> >>>>>>>>>>>>>>>>>>>>> serialization. To
> >>>>>>>>>>>>>>>>>>>>> support async writing
> >>>>>>>>>>>>>>>>>>>>> of the clients event
> >>>>>>>>>>>>>>>>>>>>> results I added
> >>>>>>>>>>>>>>>>>>>>> serialization
> >>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>> the components /
> >>>>>>>>>>>>>>>>>>>>> entities to the players
> >>>>>>>>>>>>>>>>>>>>> zone calculation and
> >>>>>>>>>>>>> stored
> >>>>>>>>>>>>>>>>>>>>> the pre-serialized
> >>>>>>>>>>>>>>>>>>>>> bytestream directly to
> >>>>>>>>>>>>>>>>>>>>> the off-heap (using
> >>>>>>>>>>>>>>>>>>>>> direct-ring-cache
> >>>>>>>>>>>>>>>>>>>>> implementation). When
> >>>>>>>>>>>>>>>>>>>>> the client requests the
> >>>>>>>>>>>>>>>>>>>>> results (using
> >>>>>>>>>>>>>>>>>>>>> long-polling) I just
> >>>>>>>>>>>>>>>>>>>>> write the
> >>>>>>>>>>>>>>>>>>>>> pre-serialized
> >>>>>>>>>>>>> data to
> >>>>>>>>>>>>>>>>>>>>> the right position to
> >>>>>>>>>>>>>>>>>>>>> deserialize it by
> >>>>>>>>>>>>>>>>>>>>> standard ways on Flash
> >>>>>>>>>>>>> side.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> So yeah an seriliaztion
> >>>>>>>>>>>>>>>>>>>>> to off-heap algorithm
> >>>>>>>>>>>>>>>>>>>>> would be fine. You
> >>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>>>> avoid using byte arrays
> >>>>>>>>>>>>>>>>>>>>> and minimalize GC.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Cheers Chris
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Am 29.09.2012 15:02,
> >>>>>>>>>>>>>>>>>>>>> schrieb Raffaele P.
> >>>>>>>>>>>>>>>>>>>>> Guidi:
> >>>>>>>>>>>>>>>>>>>>>> Nice to hear back
> >>>>>>>>>>>>>>>>>>>>>> from you! Yes, I was
> >>>>>>>>>>>>>>>>>>>>>> thinking about
> >>>>>>>>>>>>>>>>>>>>>> creating a
> >>>>>>>>>>>>>>>>> new
> >>>>>>>>>>>>>>>>>>>>> memory
> >>>>>>>>>>>>>>>>>>>>>> storage
> >>>>>>>>>>>>>>>>>>>>>> implementation using
> >>>>>>>>>>>>>>>>>>>>>> Unsafe (and I did
> >>>>>>>>>>>>>>>>>>>>>> it, recently) and
> >>>>>>>>>>>>>>>>>>> also, as
> >>>>>>>>>>>>>>>>>>>>>> DirectMemory relies
> >>>>>>>>>>>>>>>>>>>>>> heavily on
> >>>>>>>>>>>>>>>>>>>>>> serialization (and
> >>>>>>>>>>>>>>>>>>>>>> supports many
> >>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>> them,
> >>>>>>>>>>>>>>>>>>>>>> protostuff, protobuf,
> >>>>>>>>>>>>>>>>>>>>>> msgpack and of course
> >>>>>>>>>>>>>>>>>>>>>> standard
> >>>>>>>>>>>>>>>>> serialization),
> >>>>>>>>>>>>>>>>>>>>> about
> >>>>>>>>>>>>>>>>>>>>>> creating a simple
> >>>>>>>>>>>>>>>>>>>>>> embedded serializer
> >>>>>>>>>>>>>>>>>>>>>> leveraging the same
> >>>>>>>>>>>>>>>>> techniques
> >>>>>>>>>>>>>>>>>>> you
> >>>>>>>>>>>>>>>>>>>>>> used (Unsafe and
> >>>>>>>>>>>>>>>>>>>>>> bytecode generation).
> >>>>>>>>>>>>>>>>>>>>>> The idea with
> >>>>>>>>>>>>>>>>>>>>>> embedding is avoiding
> >>>>>>>>>>>>>>>>>>>>>> to serialize in a
> >>>>>>>>>>>>>>>>>>>>>> byte array
> >>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>> then
> >>>>>>>>>>>>>>>>>>>>>> moving the byte array
> >>>>>>>>>>>>>>>>>>>>>> to off-heap memory
> >>>>>>>>>>>>>>>>>>>>>> (via Unsafe or
> >>>>>>>>>>>>>>>>> ByteBuffers),
> >>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>> serializing directly
> >>>>>>>>>>>>>>>>>>>>>> into a "managed"
> >>>>>>>>>>>>>>>>>>>>>> off-heap buffer,
> >>>>>>>>>>>>>>>>>>>>>> thus
> >>>>>>>>>>>>> further
> >>>>>>>>>>>>>>>>>>>>>> optimizing heap
> >>>>>>>>>>>>>>>>>>>>>> utilization (less
> >>>>>>>>>>>>>>>>>>>>>> GC).
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> What do you think
> >>>>>>>>>>>>>>>>>>>>>> about it? Does it
> >>>>>>>>>>>>>>>>>>>>>> make any sense to
> >>>>>>>>>>>>>>>>>>>>>> you?
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Ciao, R
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On Sat, Sep 29, 2012
> >>>>>>>>>>>>>>>>>>>>>> at 2:40 PM,
> >>>>>>>>>>>>>>>>>>>>>> Noctarius
> >>>>>>>>>>>>>>>>>>>>>> <me...@noctarius.com>
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Hey guys,
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Raffaele found out
> >>>>>>>>>>>>>>>>>>>>>>> about a project of
> >>>>>>>>>>>>>>>>>>>>>>> mine (Lightning) a
> >>>>>>>>>>>>>>>>>>>>>>> few
> >>>>>>>>>>>>> weeks
> >>>>>>>>>>>>>>>>>>>>>>> ago. Lightning is a
> >>>>>>>>>>>>>>>>>>>>>>> heavy Unsafe and
> >>>>>>>>>>>>>>>>>>>>>>> Bytecode generation
> >>>>>>>>>>>>>>>>>>>>>>> using Serializer
> >>>>>>>>>>>>>>>>>>>>>>> implementation. He
> >>>>>>>>>>>>>>>>>>>>>>> told me that he
> >>>>>>>>>>>>>>>>>>>>>>> was interested in
> >>>>>>>>>>>>>>>>>>>>>>> adding something
> >>>>>>>>>>>>>>>>>>>>>>> similar to
> >>>>>>>>>>>>>>>>>>>>>>> DirectMemory and I
> >>>>>>>>>>>>>>>>>>>>>>> would be glad to
> >>>>>>>>>>>>>>>>> help
> >>>>>>>>>>>>>>>>>>>>>>> out in this.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Another project I
> >>>>>>>>>>>>>>>>>>>>>>> started a few days
> >>>>>>>>>>>>>>>>>>>>>>> ago, since it was
> >>>>>>>>>>>>>>>>>>>>>>> needed
> >>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>> work is
> >>>>>>>>>>>>>>>>>>>>>>> DirectRingCache.
> >>>>>>>>>>>>>>>>>>>>>>> The name not really
> >>>>>>>>>>>>>>>>>>>>>>> meets to actual
> >>>>>>>>>>>>>>>>>>>>>>> implementation
> >>>>>>>>>>>>>>>>>>>>>>> since it's not yet
> >>>>>>>>>>>>>>>>>>>>>>> a ring buffer using
> >>>>>>>>>>>>>>>>>>>>>>> cache. I
> >>>>>>>>>>>>>>>>> used
> >>>>>>>>>>>>>>>>>>>>>>> this for a
> >>>>>>>>>>>>>>>>>>>>>>> pre-serialization
> >>>>>>>>>>>>>>>>>>>>>>> simple bytestream
> >>>>>>>>>>>>>>>>>>>>>>> cache with
> >>>>>>>>>>>>>>>>>>>>>>> self-growing
> >>>>>>>>>>>>>>>>>>>>>>> buffers. It could
> >>>>>>>>>>>>>>>>>>>>>>> be nice to have
> >>>>>>>>>>>>>>>>>>>>>>> DirectMemory
> >>>>>>>>>>>>> having
> >>>>>>>>>>>>>>>>>>>>>>> raw "buffers" to
> >>>>>>>>>>>>>>>>>>>>>>> write to or to read
> >>>>>>>>>>>>>>>>>>>>>>> from.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Here are the links
> >>>>>>>>>>>>>>>>>>>>>>> from the projects:
> >>>>>>>>>>>>>>>>>>>>>>> https://github.com/noctarius/Lightning
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> https://github.com/noctarius/direct-ring-cache
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>> It would be nice to help out since I really
> >>>>>>>>>>> like the idea of
> >>>>>>>>>>>>>>>>>>>>>>> DirectMemory and
> >>>>>>>>>>>>>>>>>>>>>>> since
> >>>>>>>>>>>>>>>>>>>>>>> direct-ring-cache
> >>>>>>>>>>>>>>>>>>>>>>> is some kind of
> >>>>>>>>>>>>>>>>> reinventing
> >>>>>>>>>>>>>>>>>>>>>>> the wheel.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Cheers Noctarius
> >>>>>>>>>>>>>>>>>>>>>>> (Chris)
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> -- Olivier Lamy Talend:
> >>>>>>>>>>>>>>>>>>> http://coders.talend.com
> >>>>>>>>>>>>>>>>>>> http://twitter.com/olamy |
> >>>>>>>>>>>>>>>>>>> http://linkedin.com/in/olamy
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> - -- Olivier Lamy Talend:
> >>>>>>>>>>>>>>>>> http://coders.talend.com
> >>>>>>>>>>>>>>>>> http://twitter.com/olamy |
> >>>>>>>>>>>>>>>>> http://linkedin.com/in/olamy
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> -- Olivier Lamy Talend:
> >>>>>>>>>>>>> http://coders.talend.com
> >>>>>>>>>>>>> http://twitter.com/olamy |
> >>>>>>>>>>>>> http://linkedin.com/in/olamy
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> -- Olivier Lamy Talend: http://coders.talend.com
> >>>>>>>> http://twitter.com/olamy |
> >>>>>>>> http://linkedin.com/in/olamy
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>
> >>
> >
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.12 (MingW32)
> Comment: Using GnuPG with Mozilla - http://www.enigmail.net/
>
> iQIcBAEBAgAGBQJQaDFIAAoJEH/g+YBfahrqvAoP/jGZfWAyHitz48sTKvmk+f2Y
> NvkWELdPUIyhvyY0bpcV/dDv8ih94bkmSU0Tk1N22SJApetUM6PJaa+1QhVHRF7W
> 3x+fDfK9OWiAI/eBpEKjYlUrDzHh72eHmpU1tf9o0dxK+ADIJsvs4cUxtNuoCWx7
> rg6rV67f+r1ba1zfylMC4oG1T9EFLedbmWZQ6xY3TwG/Iplqz5tFg8OghK097PTP
> BYky2keKl/Tfj3HpvpqaORomJsqJLwirDm1YD7ivYN4UL/7QiEUzOtzHwDq+b1jJ
> CgHZMjQlJPDUGxb7zl+xPfi7afOxRBXuZDSb5NJfKRWUM66KISBvhzGTiRg/uMME
> uTqFufFSA1MY/Eg/ZODOmFW+jqth0jgGIwLN1ptcZH3H3IoafpXBhb6eC1lDvSI2
> zkgsvY0boSsyQggzbXuO/z6qCgayl9/XSsaxacMnvLuzIr/I+eJFVHWeYrpcV9Lz
> PE0i89fVih/ZaGijkqpzOd8O3u7v7n1o47Xd8tChNDqPy/gNRUxVnnUmJOtdUpyg
> TC3jNDkkT4WqhgbdiNWO1ypADa5VrSvc2I/O/AVWF6vbEa5uvxov0nCdyqzLM0oB
> UShPtggDH9exKU4XXcEjLfCHlQCtFN315sG9X8TBNxBuptfDrbGwadN7THpjXfj4
> xcDZpJbJQsL96ZsL9E1f
> =jT/+
> -----END PGP SIGNATURE-----
>
Re: Additional Serializer and raw Buffer access
Posted by Noctarius <me...@noctarius.com>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Ok this is the issue url. Hope the patch is ok because it's been
long since last used diff files :-)
https://issues.apache.org/jira/browse/DIRECTMEMORY-102
Am 30.09.2012 10:40, schrieb Raffaele P. Guidi:
> OK let's keep them out then Il giorno 30/set/2012 10:26,
> "Noctarius" <me...@noctarius.com> ha scritto:
>
>> Hey it's me again ;-)
>>
>> There are at least two files that are not AL2 licensed but
>> can be used (I think).
>>
>>
>> https://github.com/noctarius/Lightning/tree/master/lightning-core/src/main/java/com/github/lightning/internal/instantiator/jrockit
>>
>>
>>
If it's not possible to leave them in the sourcetree I'm pretty sure
>> there will be no problem when we remove them since they are
>> used for older JRocket versions.
>>
>> Am 29.09.2012 21:05, schrieb Raffaele P. Guidi:
>>> I kinda suspected that... Il giorno 29/set/2012 20:47,
>>> "Noctarius" <me...@noctarius.com> ha scritto:
>>>
>>>> Actually all dependencies should be AL2 or BSD licensed
>>>> :-)
>>>>
>>>> Am 29.09.2012 20:42, schrieb Raffaele P. Guidi:
>>>>>>> Hehe well that really sounds like a nice bunch of
>>>>>>> people.
>>>>>
>>>>> Indeed they are (I'm a newbie as well and try to do my
>>>>> best)
>>>>>
>>>>>>> If lightning will be a sub-part (sub-project) of
>>>>>>> DM, do I need to write
>>>>> an project purposal?
>>>>>
>>>>> Nope, not needed for a sub-project
>>>>>
>>>>>>> Do I need to make any changes to the pom.xml like
>>>>>>> adding a
>>>>> special parent pom or anything like that?
>>>>>
>>>>> Not for the serializer - just have to take a look at
>>>>> project dependencies - or, better, at their licenses -
>>>>> are they compatible with the ASL 2.0? i.e. a GPL'd
>>>>> library is not a good fit and should be replaced with
>>>>> an apache licensed (or BSD, or MIT...) one if possible.
>>>>> For the integration module is a separate story - you
>>>>> should start off copying one of the other serializers
>>>>> and reusing the same pom and directory structure.
>>>>>
>>>>> Pleased to meet you, Chris :)
>>>>>
>>>>> Ciao, R
>>>>>
>>>>> On Sat, Sep 29, 2012 at 8:24 PM, Noctarius
>>>>> <me...@noctarius.com> wrote:
>>>>>
>>>>>> Hehe well that really sounds like a nice bunch of
>>>>>> people.
>>>>>>
>>>>>> Ok to be true I couldn't wait until tomorrow and
>>>>>> started already reading the links. From what I was
>>>>>> reading: If lightning will be a sub-part
>>>>>> (sub-project) of DM, do I need to write an project
>>>>>> purposal?
>>>>>>
>>>>>> Do I need to make any changes to the pom.xml like
>>>>>> adding a special parent pom or anything like that?
>>>>>>
>>>>>> In general: there are a lot things to know :-)
>>>>>>
>>>>>> Am 29.09.2012 19:59, schrieb Raffaele P. Guidi:
>>>>>>> Negative part of ASF membership? You get together
>>>>>>> with a lot of geeky, talented people with a
>>>>>>> fixation for software and open source. Oh wait but
>>>>>>> this is actually nice! :-D Il giorno 29/set/2012
>>>>>>> 19:05, "Olivier Lamy" <ol...@apache.org> ha
>>>>>> scritto:
>>>>>>>
>>>>>>>> 2012/9/29 Noctarius <me...@noctarius.com>:
>>>>>>>>> Thanks Olivier for carify, I'll take a look in
>>>>>>>>> it tomorrow but there's just one question left
>>>>>>>>> (for now ;)): What is that vote for becoming a
>>>>>>>>> committer? What if the vote will be negative?
>>>>>>>> The vote is on private list (pmc list for privacy
>>>>>>>> reasons and possible negative stuff being on
>>>>>>>> public lists)
>>>>>>>>> Until now I just used Apache stuff, was never
>>>>>>>>> interested in being part of it so I guess it
>>>>>>>>> can be negative for any reason, can't it?
>>>>>>>> I don't see why it could be negative but suspens
>>>>>>>> .... :-)
>>>>>>>>>
>>>>>>>>> Am 29.09.2012 18:56, schrieb Olivier Lamy:
>>>>>>>>>> 2012/9/29 Noctarius <me...@noctarius.com>:
>>>>>>>>>>> Nope my real name is Christoph Engelbert,
>>>>>>>>>>> but Noctarius is the all time nick :)
>>>>>>>>>>>
>>>>>>>>>>> Renaming the package should be no problem,
>>>>>>>>>>> is it "org.apache.directmemory.lightning"
>>>>>>>>>>> or what would it be?
>>>>>>>>>> fine for me
>>>>>>>>>>> Then there needs to be a change in the
>>>>>>>>>>> license header as Olivier mentioned, that
>>>>>>>>>>> means just remove the first sentence or is
>>>>>>>>>>> there anything more to do (maybe it's
>>>>>>>>>>> easiest thing to just copy the header from
>>>>>>>>>>> DM file ;))?
>>>>>>>>>> yup use same header as DM
>>>>>>>>>>>
>>>>>>>>>>> The CLA is just a form to clarify that the
>>>>>>>>>>> source can be contributed to the Apache
>>>>>>>>>>> Foundation?
>>>>>>>>>> yup correct.
>>>>>>>>>>>
>>>>>>>>>>> The final step will be attaching the patch
>>>>>>>>>>> in form of a huge diff file?
>>>>>>>>>> yes
>>>>>>>>>>>
>>>>>>>>>>> And what is the way to apply for a
>>>>>>>>>>> membership? Never thought about how to do
>>>>>>>>>>> that.
>>>>>>>>>> Read here
>>>>>>>>>> http://apache.org/foundation/how-it-works.html
>>>>>>>>>> and here
>>>>>>>>>> http://apache.org/foundation/getinvolved.html
>>>>>>>>>> . And feel free to ask any questions :-)
>>>>>>>>>>>
>>>>>>>>>>> Chris
>>>>>>>>>>>
>>>>>>>>>>> Am 29.09.2012 18:23, schrieb Raffaele P.
>>>>>>>>>>> Guidi:
>>>>>>>>>>>> OK, deal, at least for me ;-) I propose
>>>>>>>>>>>> you rename the packages, produce a patch
>>>>>>>>>>>> for this and the new serializer module
>>>>>>>>>>>> (should be simple enough starting from an
>>>>>>>>>>>> existing one) and, in the meanwhile,
>>>>>>>>>>>> apply for ASF membership. Is IP clearance
>>>>>>>>>>>> needed? I guess yes. After this we will
>>>>>>>>>>>> come up with a formal vote regarding
>>>>>>>>>>>> Noctarius (is this your real name?!)
>>>>>>>>>>>> allowance in the project team.
>>>>>>>>>>>>
>>>>>>>>>>>> Good times are gonna come :-)
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks, R Il giorno 29/set/2012 17:58,
>>>>>>>>>>>> "Olivier Lamy" <ol...@apache.org> ha
>>>>>>>>>>>> scritto:
>>>>>>>>>>>>
>>>>>>>>>>>>> 2012/9/29 Raffaele P. Guidi
>>>>>>>>>>>>> <ra...@gmail.com>:
>>>>>>>>>>>>>> Well we already have a NIO ready
>>>>>>>>>>>>>> interface allowing direct access to
>>>>>>>>>>>>>> DMs managed bytebuffers but I think
>>>>>>>>>>>>>> this is just half way to what could
>>>>>>>>>>>>>> be achieved optimally blending
>>>>>>>>>>>>>> serialization and memory allocation
>>>>>>>>>>>>>> together.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Lightning as a module is of course a
>>>>>>>>>>>>>> good idea and it could easily evolve
>>>>>>>>>>>>>> as a subproject (for the more
>>>>>>>>>>>>>> experienced asf members: is it a
>>>>>>>>>>>>>> feasible way?).
>>>>>>>>>>>>> Nothing prevent to have
>>>>>>>>>>>>> http://svn.apache.org/repos/asf/directmemory/lightning/trunk
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>>>
>>>>
>>>>>>>>>>>>>
with a package like: o.a.d.lightning That will be a
>>>>>>>>>>>>> subproject.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Ciao, R Il giorno 29/set/2012 17:44,
>>>>>>>>>>>>>> "Noctarius" <me...@noctarius.com> ha
>>>>>>>>>>>>>> scritto:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hey guys,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ok there's no lightning binary
>>>>>>>>>>>>>>> available since lightning wasn't
>>>>>>>>>>>>>>> ready yet for releasing.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> For being the only developer it
>>>>>>>>>>>>>>> would be no problem to contribute
>>>>>>>>>>>>>>> the sourcebase for DirectMemory but
>>>>>>>>>>>>>>> I'm not sure yet if it wouldn't be
>>>>>>>>>>>>>>> better to seperate it to be
>>>>>>>>>>>>>>> available without using
>>>>>>>>>>>>>>> DirectMemory itself. I started it
>>>>>>>>>>>>>>> as a serializer for cluster
>>>>>>>>>>>>>>> synchronization, but it would be
>>>>>>>>>>>>>>> cool to contribute lightning as a
>>>>>>>>>>>>>>> subproject to DirectMemory :-)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> About the second project I would
>>>>>>>>>>>>>>> love to see a public available
>>>>>>>>>>>>>>> buffer API directly in DirectMemory
>>>>>>>>>>>>>>> so that project would be nearly
>>>>>>>>>>>>>>> needless :-) The only difference I
>>>>>>>>>>>>>>> think is the allocation strategy
>>>>>>>>>>>>>>> my implementation is using against
>>>>>>>>>>>>>>> the one DirectMemory has, but I'm
>>>>>>>>>>>>>>> pretty sure the allocation is
>>>>>>>>>>>>>>> extensible ;-)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Like I said, for both projects I'm
>>>>>>>>>>>>>>> the only dev so there would be no
>>>>>>>>>>>>>>> IP problem. So if it's ok to you to
>>>>>>>>>>>>>>> not include lightning directly in
>>>>>>>>>>>>>>> DM I would be glad to contribute to
>>>>>>>>>>>>>>> the Apache Foundation :-)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Cheers Chris
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Am 29.09.2012 16:53, schrieb
>>>>>>>>>>>>>>> Raffaele P. Guidi:
>>>>>>>>>>>>>>>> Ok so it's up to noctarius - your
>>>>>>>>>>>>>>>> move! ;-) Regarding the new
>>>>>>>>>>>>>>>> unsafe storage: it's an opt-in
>>>>>>>>>>>>>>>> feature that can be set with the
>>>>>>>>>>>>>>>> fluent API
>>>>>>>>>>>>> (and
>>>>>>>>>>>>>>>> soon through the conference
>>>>>>>>>>>>>>>> file).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Ciao, R Il giorno 29/set/2012
>>>>>>>>>>>>>>>> 16:45, "Olivier Lamy"
>>>>>>>>>>>>>>>> <ol...@apache.org> ha
>>>>>>>>>>>>>>> scritto:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2012/9/29 Raffaele P. Guidi
>>>>>>>>>>>>>>>>> <ra...@gmail.com>:
>>>>>>>>>>>>>>>>>>>> At least for the moment
>>>>>>>>>>>>>>>>>>>> he can provide a patch to
>>>>>>>>>>>>>>>>>>>> be integrated
>>>>>>>>>>>>> in DM
>>>>>>>>>>>>>>>>> :-)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Sure, but as lightning is not
>>>>>>>>>>>>>>>>>> in any public mvn repo should
>>>>>>>>>>>>>>>>>> its
>>>>>>>>>>>>> code be
>>>>>>>>>>>>>>>>>> re-published in our svn? Or
>>>>>>>>>>>>>>>>>> what?
>>>>>>>>>>>>>>>>> @Apache we don't care about
>>>>>>>>>>>>>>>>> binaries, only sources are
>>>>>>>>>>>>>>>>> important ! (a bit theorical
>>>>>>>>>>>>>>>>> for sure but that's it :-) ).
>>>>>>>>>>>>>>>>> So if Noctarius was the only
>>>>>>>>>>>>>>>>> guy who participate in
>>>>>>>>>>>>>>>>> lightning. He can just provide
>>>>>>>>>>>>>>>>> a patch we could integrate as
>>>>>>>>>>>>>>>>> a new dm module (note: the
>>>>>>>>>>>>>>>>> patch must not contains any
>>>>>>>>>>>>>>>>> more copyright and all sources
>>>>>>>>>>>>>>>>> must have ASF licenses).
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> "Copyright 2012 the original
>>>>>>>>>>>>>>>>> author or authors." must be
>>>>>>>>>>>>>>>>> removed. And BTW package must
>>>>>>>>>>>>>>>>> be changed :-) (com.github is
>>>>>>>>>>>>>>>>> not acceptable
>>>>>>>>>>>>> @asf
>>>>>>>>>>>>>>>>> :-) )(@Noctarius are you
>>>>>>>>>>>>>>>>> working for github ? :-) )
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> And having him as a committer
>>>>>>>>>>>>>>>>> will be only a matter of voting
>>>>>>>>>>>>>>>>> (we
>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>> a great chair who take cares
>>>>>>>>>>>>>>>>> of administrative stuff :P )
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> If some others have
>>>>>>>>>>>>>>>>> participated in the project, we
>>>>>>>>>>>>>>>>> must pass tru an ip clearance
>>>>>>>>>>>>>>>>> mechanism
>>>>>>>>>>>>>>>>> (http://incubator.apache.org/ip-clearance/index.html)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>
>>>>>>>>>>>>>>>>>
and all contributors to lightning must provide a cla.
>>>>>>>>>>>>>>>>> (It it's the case I can help)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> perso I'd like we avoid
>>>>>>>>>>>>>>>>>>>> hard dependency on Unsafe
>>>>>>>>>>>>>>>>>>>> as maybe some
>>>>>>>>>>>>> use
>>>>>>>>>>>>>>>>>> other jdks :-)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Well, I believe Unsafe is
>>>>>>>>>>>>>>>>>> supported by Sun JDK,
>>>>>>>>>>>>>>>>>> OpenJDK, IBM JDK and JRockit
>>>>>>>>>>>>>>>>>> - and I believe that it is
>>>>>>>>>>>>>>>>>> more than enough. Also keep
>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>> mind
>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>> we already have an
>>>>>>>>>>>>>>>>>> alternative Unsafe based
>>>>>>>>>>>>>>>>>> memory storage - and
>>>>>>>>>>>>>>> although
>>>>>>>>>>>>>>>>>> it's not thoroughly tested
>>>>>>>>>>>>>>>>>> for performance it
>>>>>>>>>>>>>>>>>> dramaticaly simplifies
>>>>>>>>>>>>>>>>> code,
>>>>>>>>>>>>>>>>>> I have great expectations
>>>>>>>>>>>>>>>>>> about it.
>>>>>>>>>>>>>>>>> Me too :-). Yup definitely more
>>>>>>>>>>>>>>>>> simple and faster ! But we must
>>>>>>>>>>>>>>>>> provide a switch off
>>>>>>>>>>>>>>>>> configuration mechanism if
>>>>>>>>>>>>>>>>> some
>>>>>>>>>>>>> users
>>>>>>>>>>>>>>>>> don't want to use that (because
>>>>>>>>>>>>>>>>> Unsafe is just "Unsafe" :-) )
>>>>>>>>>>>>>>>>> And sorry I didn't have a look
>>>>>>>>>>>>>>>>> yet at your changes with using
>>>>>>>>>>>>>>>>> Unsafe.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Cheers, R
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Sat, Sep 29, 2012 at 4:03
>>>>>>>>>>>>>>>>>> PM, Olivier Lamy
>>>>>>>>>>>>>>>>>> <ol...@apache.org>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2012/9/29 Raffaele P.
>>>>>>>>>>>>>>>>>>> Guidi
>>>>>>>>>>>>>>>>>>> <ra...@gmail.com>:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
What do you think about: 1)
>>>>>>>>>>>>>>>>>>>> implementing a lightning
>>>>>>>>>>>>>>>>>>>> serialization module 2)
>>>>>>>>>>>>>>>>>>>> creating a serializer
>>>>>>>>>>>>>>>>>>>> that directly works on a
>>>>>>>>>>>>>>>>>>>> directmemory
>>>>>>>>>>>>>>>>> provider
>>>>>>>>>>>>>>>>>>>> ByteBuffer or (maybe
>>>>>>>>>>>>>>>>>>>> better) Unsafe based
>>>>>>>>>>>>>>>>>>>> Pointer?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Sounds good (perso I'd like
>>>>>>>>>>>>>>>>>>> we avoid hard dependency on
>>>>>>>>>>>>>>>>>>> Unsafe as maybe some use
>>>>>>>>>>>>>>>>>>> other jdks :-) )
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Now I see lightning is
>>>>>>>>>>>>>>>>>>>> apache licensed and this
>>>>>>>>>>>>>>>>>>>> is fine but I
>>>>>>>>>>>>> don't
>>>>>>>>>>>>>>>>> think
>>>>>>>>>>>>>>>>>>>> it is published in any
>>>>>>>>>>>>>>>>>>>> public maven repo, am I
>>>>>>>>>>>>>>>>>>>> right? We could
>>>>>>>>>>>>> find a
>>>>>>>>>>>>>>>>> way
>>>>>>>>>>>>>>>>>>>> to deal with this;
>>>>>>>>>>>>>>>>>>>> options vary from
>>>>>>>>>>>>>>>>>>>> publishing lightning to
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>> free
>>>>>>>>>>>>>>>>>>>> sonatype repo, joining
>>>>>>>>>>>>>>>>>>>> the ASF (which is great
>>>>>>>>>>>>>>>>>>>> anyhow!) and
>>>>>>>>>>>>>>>>> republishing
>>>>>>>>>>>>>>>>>>>> lightning code in
>>>>>>>>>>>>>>>>>>>> DirectMemory becoming a
>>>>>>>>>>>>>>>>>>>> commiter (which has to
>>>>>>>>>>>>>>>>> undergo
>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>> PMC vote).
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> At least for the moment he
>>>>>>>>>>>>>>>>>>> can provide a patch to be
>>>>>>>>>>>>>>>>>>> integrated in
>>>>>>>>>>>>> DM
>>>>>>>>>>>>>>>>> :-)
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I'd like to hear your and
>>>>>>>>>>>>>>>>>>>> the team feelings on
>>>>>>>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks, Raffaele
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Sat, Sep 29, 2012 at
>>>>>>>>>>>>>>>>>>>> 3:27 PM, Noctarius
>>>>>>>>>>>>>>>>>>>> <me...@noctarius.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hey Raffaele,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> that's quite similar to
>>>>>>>>>>>>>>>>>>>>> what I did at work.
>>>>>>>>>>>>>>>>>>>>> We're developing
>>>>>>>>>>>>> Flash
>>>>>>>>>>>>>>>>>>>>> online games and using
>>>>>>>>>>>>>>>>>>>>> a customized AMF
>>>>>>>>>>>>>>>>>>>>> serialization. To
>>>>>>>>>>>>>>>>>>>>> support async writing
>>>>>>>>>>>>>>>>>>>>> of the clients event
>>>>>>>>>>>>>>>>>>>>> results I added
>>>>>>>>>>>>>>>>>>>>> serialization
>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>> the components /
>>>>>>>>>>>>>>>>>>>>> entities to the players
>>>>>>>>>>>>>>>>>>>>> zone calculation and
>>>>>>>>>>>>> stored
>>>>>>>>>>>>>>>>>>>>> the pre-serialized
>>>>>>>>>>>>>>>>>>>>> bytestream directly to
>>>>>>>>>>>>>>>>>>>>> the off-heap (using
>>>>>>>>>>>>>>>>>>>>> direct-ring-cache
>>>>>>>>>>>>>>>>>>>>> implementation). When
>>>>>>>>>>>>>>>>>>>>> the client requests the
>>>>>>>>>>>>>>>>>>>>> results (using
>>>>>>>>>>>>>>>>>>>>> long-polling) I just
>>>>>>>>>>>>>>>>>>>>> write the
>>>>>>>>>>>>>>>>>>>>> pre-serialized
>>>>>>>>>>>>> data to
>>>>>>>>>>>>>>>>>>>>> the right position to
>>>>>>>>>>>>>>>>>>>>> deserialize it by
>>>>>>>>>>>>>>>>>>>>> standard ways on Flash
>>>>>>>>>>>>> side.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> So yeah an seriliaztion
>>>>>>>>>>>>>>>>>>>>> to off-heap algorithm
>>>>>>>>>>>>>>>>>>>>> would be fine. You
>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>> avoid using byte arrays
>>>>>>>>>>>>>>>>>>>>> and minimalize GC.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Cheers Chris
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Am 29.09.2012 15:02,
>>>>>>>>>>>>>>>>>>>>> schrieb Raffaele P.
>>>>>>>>>>>>>>>>>>>>> Guidi:
>>>>>>>>>>>>>>>>>>>>>> Nice to hear back
>>>>>>>>>>>>>>>>>>>>>> from you! Yes, I was
>>>>>>>>>>>>>>>>>>>>>> thinking about
>>>>>>>>>>>>>>>>>>>>>> creating a
>>>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>>>> memory
>>>>>>>>>>>>>>>>>>>>>> storage
>>>>>>>>>>>>>>>>>>>>>> implementation using
>>>>>>>>>>>>>>>>>>>>>> Unsafe (and I did
>>>>>>>>>>>>>>>>>>>>>> it, recently) and
>>>>>>>>>>>>>>>>>>> also, as
>>>>>>>>>>>>>>>>>>>>>> DirectMemory relies
>>>>>>>>>>>>>>>>>>>>>> heavily on
>>>>>>>>>>>>>>>>>>>>>> serialization (and
>>>>>>>>>>>>>>>>>>>>>> supports many
>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>> them,
>>>>>>>>>>>>>>>>>>>>>> protostuff, protobuf,
>>>>>>>>>>>>>>>>>>>>>> msgpack and of course
>>>>>>>>>>>>>>>>>>>>>> standard
>>>>>>>>>>>>>>>>> serialization),
>>>>>>>>>>>>>>>>>>>>> about
>>>>>>>>>>>>>>>>>>>>>> creating a simple
>>>>>>>>>>>>>>>>>>>>>> embedded serializer
>>>>>>>>>>>>>>>>>>>>>> leveraging the same
>>>>>>>>>>>>>>>>> techniques
>>>>>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>>>>>>>> used (Unsafe and
>>>>>>>>>>>>>>>>>>>>>> bytecode generation).
>>>>>>>>>>>>>>>>>>>>>> The idea with
>>>>>>>>>>>>>>>>>>>>>> embedding is avoiding
>>>>>>>>>>>>>>>>>>>>>> to serialize in a
>>>>>>>>>>>>>>>>>>>>>> byte array
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>>>>>>>>> moving the byte array
>>>>>>>>>>>>>>>>>>>>>> to off-heap memory
>>>>>>>>>>>>>>>>>>>>>> (via Unsafe or
>>>>>>>>>>>>>>>>> ByteBuffers),
>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>> serializing directly
>>>>>>>>>>>>>>>>>>>>>> into a "managed"
>>>>>>>>>>>>>>>>>>>>>> off-heap buffer,
>>>>>>>>>>>>>>>>>>>>>> thus
>>>>>>>>>>>>> further
>>>>>>>>>>>>>>>>>>>>>> optimizing heap
>>>>>>>>>>>>>>>>>>>>>> utilization (less
>>>>>>>>>>>>>>>>>>>>>> GC).
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> What do you think
>>>>>>>>>>>>>>>>>>>>>> about it? Does it
>>>>>>>>>>>>>>>>>>>>>> make any sense to
>>>>>>>>>>>>>>>>>>>>>> you?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Ciao, R
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Sat, Sep 29, 2012
>>>>>>>>>>>>>>>>>>>>>> at 2:40 PM,
>>>>>>>>>>>>>>>>>>>>>> Noctarius
>>>>>>>>>>>>>>>>>>>>>> <me...@noctarius.com>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Hey guys,
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Raffaele found out
>>>>>>>>>>>>>>>>>>>>>>> about a project of
>>>>>>>>>>>>>>>>>>>>>>> mine (Lightning) a
>>>>>>>>>>>>>>>>>>>>>>> few
>>>>>>>>>>>>> weeks
>>>>>>>>>>>>>>>>>>>>>>> ago. Lightning is a
>>>>>>>>>>>>>>>>>>>>>>> heavy Unsafe and
>>>>>>>>>>>>>>>>>>>>>>> Bytecode generation
>>>>>>>>>>>>>>>>>>>>>>> using Serializer
>>>>>>>>>>>>>>>>>>>>>>> implementation. He
>>>>>>>>>>>>>>>>>>>>>>> told me that he
>>>>>>>>>>>>>>>>>>>>>>> was interested in
>>>>>>>>>>>>>>>>>>>>>>> adding something
>>>>>>>>>>>>>>>>>>>>>>> similar to
>>>>>>>>>>>>>>>>>>>>>>> DirectMemory and I
>>>>>>>>>>>>>>>>>>>>>>> would be glad to
>>>>>>>>>>>>>>>>> help
>>>>>>>>>>>>>>>>>>>>>>> out in this.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Another project I
>>>>>>>>>>>>>>>>>>>>>>> started a few days
>>>>>>>>>>>>>>>>>>>>>>> ago, since it was
>>>>>>>>>>>>>>>>>>>>>>> needed
>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>> work is
>>>>>>>>>>>>>>>>>>>>>>> DirectRingCache.
>>>>>>>>>>>>>>>>>>>>>>> The name not really
>>>>>>>>>>>>>>>>>>>>>>> meets to actual
>>>>>>>>>>>>>>>>>>>>>>> implementation
>>>>>>>>>>>>>>>>>>>>>>> since it's not yet
>>>>>>>>>>>>>>>>>>>>>>> a ring buffer using
>>>>>>>>>>>>>>>>>>>>>>> cache. I
>>>>>>>>>>>>>>>>> used
>>>>>>>>>>>>>>>>>>>>>>> this for a
>>>>>>>>>>>>>>>>>>>>>>> pre-serialization
>>>>>>>>>>>>>>>>>>>>>>> simple bytestream
>>>>>>>>>>>>>>>>>>>>>>> cache with
>>>>>>>>>>>>>>>>>>>>>>> self-growing
>>>>>>>>>>>>>>>>>>>>>>> buffers. It could
>>>>>>>>>>>>>>>>>>>>>>> be nice to have
>>>>>>>>>>>>>>>>>>>>>>> DirectMemory
>>>>>>>>>>>>> having
>>>>>>>>>>>>>>>>>>>>>>> raw "buffers" to
>>>>>>>>>>>>>>>>>>>>>>> write to or to read
>>>>>>>>>>>>>>>>>>>>>>> from.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Here are the links
>>>>>>>>>>>>>>>>>>>>>>> from the projects:
>>>>>>>>>>>>>>>>>>>>>>> https://github.com/noctarius/Lightning
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>
>>>>>>>>>>>>>>>>>>>>>>>
https://github.com/noctarius/direct-ring-cache
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>> It would be nice to help out since I really
>>>>>>>>>>> like the idea of
>>>>>>>>>>>>>>>>>>>>>>> DirectMemory and
>>>>>>>>>>>>>>>>>>>>>>> since
>>>>>>>>>>>>>>>>>>>>>>> direct-ring-cache
>>>>>>>>>>>>>>>>>>>>>>> is some kind of
>>>>>>>>>>>>>>>>> reinventing
>>>>>>>>>>>>>>>>>>>>>>> the wheel.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Cheers Noctarius
>>>>>>>>>>>>>>>>>>>>>>> (Chris)
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> -- Olivier Lamy Talend:
>>>>>>>>>>>>>>>>>>> http://coders.talend.com
>>>>>>>>>>>>>>>>>>> http://twitter.com/olamy |
>>>>>>>>>>>>>>>>>>> http://linkedin.com/in/olamy
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
- -- Olivier Lamy Talend:
>>>>>>>>>>>>>>>>> http://coders.talend.com
>>>>>>>>>>>>>>>>> http://twitter.com/olamy |
>>>>>>>>>>>>>>>>> http://linkedin.com/in/olamy
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> -- Olivier Lamy Talend:
>>>>>>>>>>>>> http://coders.talend.com
>>>>>>>>>>>>> http://twitter.com/olamy |
>>>>>>>>>>>>> http://linkedin.com/in/olamy
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> -- Olivier Lamy Talend: http://coders.talend.com
>>>>>>>> http://twitter.com/olamy |
>>>>>>>> http://linkedin.com/in/olamy
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>>
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.12 (MingW32)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/
iQIcBAEBAgAGBQJQaDFIAAoJEH/g+YBfahrqvAoP/jGZfWAyHitz48sTKvmk+f2Y
NvkWELdPUIyhvyY0bpcV/dDv8ih94bkmSU0Tk1N22SJApetUM6PJaa+1QhVHRF7W
3x+fDfK9OWiAI/eBpEKjYlUrDzHh72eHmpU1tf9o0dxK+ADIJsvs4cUxtNuoCWx7
rg6rV67f+r1ba1zfylMC4oG1T9EFLedbmWZQ6xY3TwG/Iplqz5tFg8OghK097PTP
BYky2keKl/Tfj3HpvpqaORomJsqJLwirDm1YD7ivYN4UL/7QiEUzOtzHwDq+b1jJ
CgHZMjQlJPDUGxb7zl+xPfi7afOxRBXuZDSb5NJfKRWUM66KISBvhzGTiRg/uMME
uTqFufFSA1MY/Eg/ZODOmFW+jqth0jgGIwLN1ptcZH3H3IoafpXBhb6eC1lDvSI2
zkgsvY0boSsyQggzbXuO/z6qCgayl9/XSsaxacMnvLuzIr/I+eJFVHWeYrpcV9Lz
PE0i89fVih/ZaGijkqpzOd8O3u7v7n1o47Xd8tChNDqPy/gNRUxVnnUmJOtdUpyg
TC3jNDkkT4WqhgbdiNWO1ypADa5VrSvc2I/O/AVWF6vbEa5uvxov0nCdyqzLM0oB
UShPtggDH9exKU4XXcEjLfCHlQCtFN315sG9X8TBNxBuptfDrbGwadN7THpjXfj4
xcDZpJbJQsL96ZsL9E1f
=jT/+
-----END PGP SIGNATURE-----
Re: Additional Serializer and raw Buffer access
Posted by "Raffaele P. Guidi" <ra...@gmail.com>.
OK let's keep them out then
Il giorno 30/set/2012 10:26, "Noctarius" <me...@noctarius.com> ha scritto:
> Hey it's me again ;-)
>
> There are at least two files that are not AL2 licensed but can be
> used (I think).
>
>
> https://github.com/noctarius/Lightning/tree/master/lightning-core/src/main/java/com/github/lightning/internal/instantiator/jrockit
>
> If it's not possible to leave them in the sourcetree I'm pretty sure
> there will be no problem when we remove them since they are used for
> older JRocket versions.
>
> Am 29.09.2012 21:05, schrieb Raffaele P. Guidi:
> > I kinda suspected that...
> > Il giorno 29/set/2012 20:47, "Noctarius" <me...@noctarius.com> ha scritto:
> >
> >> Actually all dependencies should be AL2 or BSD licensed :-)
> >>
> >> Am 29.09.2012 20:42, schrieb Raffaele P. Guidi:
> >>>>> Hehe well that really sounds like a nice bunch of people.
> >>>
> >>> Indeed they are (I'm a newbie as well and try to do my best)
> >>>
> >>>>> If lightning will be a sub-part (sub-project) of DM, do I
> >>>>> need to write
> >>> an project purposal?
> >>>
> >>> Nope, not needed for a sub-project
> >>>
> >>>>> Do I need to make any changes to the pom.xml like adding a
> >>> special parent pom or anything like that?
> >>>
> >>> Not for the serializer - just have to take a look at project
> >>> dependencies - or, better, at their licenses - are they
> >>> compatible with the ASL 2.0? i.e. a GPL'd library is not a good
> >>> fit and should be replaced with an apache licensed (or BSD, or
> >>> MIT...) one if possible. For the integration module is a
> >>> separate story - you should start off copying one of the other
> >>> serializers and reusing the same pom and directory structure.
> >>>
> >>> Pleased to meet you, Chris :)
> >>>
> >>> Ciao, R
> >>>
> >>> On Sat, Sep 29, 2012 at 8:24 PM, Noctarius <me...@noctarius.com>
> >>> wrote:
> >>>
> >>>> Hehe well that really sounds like a nice bunch of people.
> >>>>
> >>>> Ok to be true I couldn't wait until tomorrow and started
> >>>> already reading the links. From what I was reading: If
> >>>> lightning will be a sub-part (sub-project) of DM, do I need
> >>>> to write an project purposal?
> >>>>
> >>>> Do I need to make any changes to the pom.xml like adding a
> >>>> special parent pom or anything like that?
> >>>>
> >>>> In general: there are a lot things to know :-)
> >>>>
> >>>> Am 29.09.2012 19:59, schrieb Raffaele P. Guidi:
> >>>>> Negative part of ASF membership? You get together with a
> >>>>> lot of geeky, talented people with a fixation for software
> >>>>> and open source. Oh wait but this is actually nice! :-D Il
> >>>>> giorno 29/set/2012 19:05, "Olivier Lamy" <ol...@apache.org>
> >>>>> ha
> >>>> scritto:
> >>>>>
> >>>>>> 2012/9/29 Noctarius <me...@noctarius.com>:
> >>>>>>> Thanks Olivier for carify, I'll take a look in it
> >>>>>>> tomorrow but there's just one question left (for now
> >>>>>>> ;)): What is that vote for becoming a committer? What
> >>>>>>> if the vote will be negative?
> >>>>>> The vote is on private list (pmc list for privacy reasons
> >>>>>> and possible negative stuff being on public lists)
> >>>>>>> Until now I just used Apache stuff, was never
> >>>>>>> interested in being part of it so I guess it can be
> >>>>>>> negative for any reason, can't it?
> >>>>>> I don't see why it could be negative but suspens ....
> >>>>>> :-)
> >>>>>>>
> >>>>>>> Am 29.09.2012 18:56, schrieb Olivier Lamy:
> >>>>>>>> 2012/9/29 Noctarius <me...@noctarius.com>:
> >>>>>>>>> Nope my real name is Christoph Engelbert, but
> >>>>>>>>> Noctarius is the all time nick :)
> >>>>>>>>>
> >>>>>>>>> Renaming the package should be no problem, is it
> >>>>>>>>> "org.apache.directmemory.lightning" or what would
> >>>>>>>>> it be?
> >>>>>>>> fine for me
> >>>>>>>>> Then there needs to be a change in the license
> >>>>>>>>> header as Olivier mentioned, that means just remove
> >>>>>>>>> the first sentence or is there anything more to do
> >>>>>>>>> (maybe it's easiest thing to just copy the header
> >>>>>>>>> from DM file ;))?
> >>>>>>>> yup use same header as DM
> >>>>>>>>>
> >>>>>>>>> The CLA is just a form to clarify that the source
> >>>>>>>>> can be contributed to the Apache Foundation?
> >>>>>>>> yup correct.
> >>>>>>>>>
> >>>>>>>>> The final step will be attaching the patch in form
> >>>>>>>>> of a huge diff file?
> >>>>>>>> yes
> >>>>>>>>>
> >>>>>>>>> And what is the way to apply for a membership?
> >>>>>>>>> Never thought about how to do that.
> >>>>>>>> Read here
> >>>>>>>> http://apache.org/foundation/how-it-works.html and
> >>>>>>>> here http://apache.org/foundation/getinvolved.html .
> >>>>>>>> And feel free to ask any questions :-)
> >>>>>>>>>
> >>>>>>>>> Chris
> >>>>>>>>>
> >>>>>>>>> Am 29.09.2012 18:23, schrieb Raffaele P. Guidi:
> >>>>>>>>>> OK, deal, at least for me ;-) I propose you
> >>>>>>>>>> rename the packages, produce a patch for this and
> >>>>>>>>>> the new serializer module (should be simple
> >>>>>>>>>> enough starting from an existing one) and, in the
> >>>>>>>>>> meanwhile, apply for ASF membership. Is IP
> >>>>>>>>>> clearance needed? I guess yes. After this we will
> >>>>>>>>>> come up with a formal vote regarding Noctarius
> >>>>>>>>>> (is this your real name?!) allowance in the
> >>>>>>>>>> project team.
> >>>>>>>>>>
> >>>>>>>>>> Good times are gonna come :-)
> >>>>>>>>>>
> >>>>>>>>>> Thanks, R Il giorno 29/set/2012 17:58, "Olivier
> >>>>>>>>>> Lamy" <ol...@apache.org> ha scritto:
> >>>>>>>>>>
> >>>>>>>>>>> 2012/9/29 Raffaele P. Guidi
> >>>>>>>>>>> <ra...@gmail.com>:
> >>>>>>>>>>>> Well we already have a NIO ready interface
> >>>>>>>>>>>> allowing direct access to DMs managed
> >>>>>>>>>>>> bytebuffers but I think this is just half way
> >>>>>>>>>>>> to what could be achieved optimally blending
> >>>>>>>>>>>> serialization and memory allocation
> >>>>>>>>>>>> together.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Lightning as a module is of course a good
> >>>>>>>>>>>> idea and it could easily evolve as a
> >>>>>>>>>>>> subproject (for the more experienced asf
> >>>>>>>>>>>> members: is it a feasible way?).
> >>>>>>>>>>> Nothing prevent to have
> >>>>>>>>>>> http://svn.apache.org/repos/asf/directmemory/lightning/trunk
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>
> >>>>>>>>>>>
> >> with a package like: o.a.d.lightning That will be a
> >>>>>>>>>>> subproject.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Ciao, R Il giorno 29/set/2012 17:44,
> >>>>>>>>>>>> "Noctarius" <me...@noctarius.com> ha scritto:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Hey guys,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> ok there's no lightning binary available
> >>>>>>>>>>>>> since lightning wasn't ready yet for
> >>>>>>>>>>>>> releasing.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> For being the only developer it would be no
> >>>>>>>>>>>>> problem to contribute the sourcebase for
> >>>>>>>>>>>>> DirectMemory but I'm not sure yet if it
> >>>>>>>>>>>>> wouldn't be better to seperate it to be
> >>>>>>>>>>>>> available without using DirectMemory
> >>>>>>>>>>>>> itself. I started it as a serializer for
> >>>>>>>>>>>>> cluster synchronization, but it would be
> >>>>>>>>>>>>> cool to contribute lightning as a
> >>>>>>>>>>>>> subproject to DirectMemory :-)
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> About the second project I would love to
> >>>>>>>>>>>>> see a public available buffer API directly
> >>>>>>>>>>>>> in DirectMemory so that project would be
> >>>>>>>>>>>>> nearly needless :-) The only difference I
> >>>>>>>>>>>>> think is the allocation strategy my
> >>>>>>>>>>>>> implementation is using against the one
> >>>>>>>>>>>>> DirectMemory has, but I'm pretty sure the
> >>>>>>>>>>>>> allocation is extensible ;-)
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Like I said, for both projects I'm the only
> >>>>>>>>>>>>> dev so there would be no IP problem. So if
> >>>>>>>>>>>>> it's ok to you to not include lightning
> >>>>>>>>>>>>> directly in DM I would be glad to
> >>>>>>>>>>>>> contribute to the Apache Foundation :-)
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Cheers Chris
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Am 29.09.2012 16:53, schrieb Raffaele P.
> >>>>>>>>>>>>> Guidi:
> >>>>>>>>>>>>>> Ok so it's up to noctarius - your move!
> >>>>>>>>>>>>>> ;-) Regarding the new unsafe storage:
> >>>>>>>>>>>>>> it's an opt-in feature that can be set
> >>>>>>>>>>>>>> with the fluent API
> >>>>>>>>>>> (and
> >>>>>>>>>>>>>> soon through the conference file).
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Ciao, R Il giorno 29/set/2012 16:45,
> >>>>>>>>>>>>>> "Olivier Lamy" <ol...@apache.org> ha
> >>>>>>>>>>>>> scritto:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> 2012/9/29 Raffaele P. Guidi
> >>>>>>>>>>>>>>> <ra...@gmail.com>:
> >>>>>>>>>>>>>>>>>> At least for the moment he can
> >>>>>>>>>>>>>>>>>> provide a patch to be integrated
> >>>>>>>>>>> in DM
> >>>>>>>>>>>>>>> :-)
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Sure, but as lightning is not in any
> >>>>>>>>>>>>>>>> public mvn repo should its
> >>>>>>>>>>> code be
> >>>>>>>>>>>>>>>> re-published in our svn? Or what?
> >>>>>>>>>>>>>>> @Apache we don't care about binaries,
> >>>>>>>>>>>>>>> only sources are important ! (a bit
> >>>>>>>>>>>>>>> theorical for sure but that's it :-) ).
> >>>>>>>>>>>>>>> So if Noctarius was the only guy who
> >>>>>>>>>>>>>>> participate in lightning. He can just
> >>>>>>>>>>>>>>> provide a patch we could integrate as a
> >>>>>>>>>>>>>>> new dm module (note: the patch must not
> >>>>>>>>>>>>>>> contains any more copyright and all
> >>>>>>>>>>>>>>> sources must have ASF licenses).
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> "Copyright 2012 the original author or
> >>>>>>>>>>>>>>> authors." must be removed. And BTW
> >>>>>>>>>>>>>>> package must be changed :-) (com.github
> >>>>>>>>>>>>>>> is not acceptable
> >>>>>>>>>>> @asf
> >>>>>>>>>>>>>>> :-) )(@Noctarius are you working for
> >>>>>>>>>>>>>>> github ? :-) )
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> And having him as a committer will be
> >>>>>>>>>>>>>>> only a matter of voting (we
> >>>>>>>>>>> have
> >>>>>>>>>>>>>>> a great chair who take cares of
> >>>>>>>>>>>>>>> administrative stuff :P )
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> If some others have participated in the
> >>>>>>>>>>>>>>> project, we must pass tru an ip
> >>>>>>>>>>>>>>> clearance mechanism
> >>>>>>>>>>>>>>> (http://incubator.apache.org/ip-clearance/index.html)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>
> >> and all contributors to lightning must provide a cla.
> >>>>>>>>>>>>>>> (It it's the case I can help)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> perso I'd like we avoid hard
> >>>>>>>>>>>>>>>>>> dependency on Unsafe as maybe
> >>>>>>>>>>>>>>>>>> some
> >>>>>>>>>>> use
> >>>>>>>>>>>>>>>> other jdks :-)
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Well, I believe Unsafe is supported
> >>>>>>>>>>>>>>>> by Sun JDK, OpenJDK, IBM JDK and
> >>>>>>>>>>>>>>>> JRockit - and I believe that it is
> >>>>>>>>>>>>>>>> more than enough. Also keep in
> >>>>>>>>>>> mind
> >>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>> we already have an alternative Unsafe
> >>>>>>>>>>>>>>>> based memory storage - and
> >>>>>>>>>>>>> although
> >>>>>>>>>>>>>>>> it's not thoroughly tested for
> >>>>>>>>>>>>>>>> performance it dramaticaly
> >>>>>>>>>>>>>>>> simplifies
> >>>>>>>>>>>>>>> code,
> >>>>>>>>>>>>>>>> I have great expectations about it.
> >>>>>>>>>>>>>>> Me too :-). Yup definitely more simple
> >>>>>>>>>>>>>>> and faster ! But we must provide a
> >>>>>>>>>>>>>>> switch off configuration mechanism if
> >>>>>>>>>>>>>>> some
> >>>>>>>>>>> users
> >>>>>>>>>>>>>>> don't want to use that (because Unsafe
> >>>>>>>>>>>>>>> is just "Unsafe" :-) ) And sorry I
> >>>>>>>>>>>>>>> didn't have a look yet at your changes
> >>>>>>>>>>>>>>> with using Unsafe.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Cheers, R
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Sat, Sep 29, 2012 at 4:03 PM,
> >>>>>>>>>>>>>>>> Olivier Lamy <ol...@apache.org>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> 2012/9/29 Raffaele P. Guidi
> >>>>>>>>>>>>>>>>> <ra...@gmail.com>:
> >>>>>>>>>>>>>>>>>> What do you think about: 1)
> >>>>>>>>>>>>>>>>>> implementing a lightning
> >>>>>>>>>>>>>>>>>> serialization module 2) creating
> >>>>>>>>>>>>>>>>>> a serializer that directly works
> >>>>>>>>>>>>>>>>>> on a directmemory
> >>>>>>>>>>>>>>> provider
> >>>>>>>>>>>>>>>>>> ByteBuffer or (maybe better)
> >>>>>>>>>>>>>>>>>> Unsafe based Pointer?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Sounds good (perso I'd like we
> >>>>>>>>>>>>>>>>> avoid hard dependency on Unsafe as
> >>>>>>>>>>>>>>>>> maybe some use other jdks :-) )
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Now I see lightning is apache
> >>>>>>>>>>>>>>>>>> licensed and this is fine but I
> >>>>>>>>>>> don't
> >>>>>>>>>>>>>>> think
> >>>>>>>>>>>>>>>>>> it is published in any public
> >>>>>>>>>>>>>>>>>> maven repo, am I right? We could
> >>>>>>>>>>> find a
> >>>>>>>>>>>>>>> way
> >>>>>>>>>>>>>>>>>> to deal with this; options vary
> >>>>>>>>>>>>>>>>>> from publishing lightning to the
> >>>>>>>>>>> free
> >>>>>>>>>>>>>>>>>> sonatype repo, joining the ASF
> >>>>>>>>>>>>>>>>>> (which is great anyhow!) and
> >>>>>>>>>>>>>>> republishing
> >>>>>>>>>>>>>>>>>> lightning code in DirectMemory
> >>>>>>>>>>>>>>>>>> becoming a commiter (which has
> >>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>> undergo
> >>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>> PMC vote).
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> At least for the moment he can
> >>>>>>>>>>>>>>>>> provide a patch to be integrated
> >>>>>>>>>>>>>>>>> in
> >>>>>>>>>>> DM
> >>>>>>>>>>>>>>> :-)
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I'd like to hear your and the
> >>>>>>>>>>>>>>>>>> team feelings on this.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks, Raffaele
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Sat, Sep 29, 2012 at 3:27 PM,
> >>>>>>>>>>>>>>>>>> Noctarius <me...@noctarius.com>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Hey Raffaele,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> that's quite similar to what I
> >>>>>>>>>>>>>>>>>>> did at work. We're developing
> >>>>>>>>>>> Flash
> >>>>>>>>>>>>>>>>>>> online games and using a
> >>>>>>>>>>>>>>>>>>> customized AMF serialization.
> >>>>>>>>>>>>>>>>>>> To support async writing of the
> >>>>>>>>>>>>>>>>>>> clients event results I added
> >>>>>>>>>>>>>>>>>>> serialization
> >>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>> the components / entities to
> >>>>>>>>>>>>>>>>>>> the players zone calculation
> >>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>> stored
> >>>>>>>>>>>>>>>>>>> the pre-serialized bytestream
> >>>>>>>>>>>>>>>>>>> directly to the off-heap (using
> >>>>>>>>>>>>>>>>>>> direct-ring-cache
> >>>>>>>>>>>>>>>>>>> implementation). When the
> >>>>>>>>>>>>>>>>>>> client requests the results
> >>>>>>>>>>>>>>>>>>> (using long-polling) I just
> >>>>>>>>>>>>>>>>>>> write the pre-serialized
> >>>>>>>>>>> data to
> >>>>>>>>>>>>>>>>>>> the right position to
> >>>>>>>>>>>>>>>>>>> deserialize it by standard ways
> >>>>>>>>>>>>>>>>>>> on Flash
> >>>>>>>>>>> side.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> So yeah an seriliaztion to
> >>>>>>>>>>>>>>>>>>> off-heap algorithm would be
> >>>>>>>>>>>>>>>>>>> fine. You
> >>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>> avoid using byte arrays and
> >>>>>>>>>>>>>>>>>>> minimalize GC.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Cheers Chris
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Am 29.09.2012 15:02, schrieb
> >>>>>>>>>>>>>>>>>>> Raffaele P. Guidi:
> >>>>>>>>>>>>>>>>>>>> Nice to hear back from you!
> >>>>>>>>>>>>>>>>>>>> Yes, I was thinking about
> >>>>>>>>>>>>>>>>>>>> creating a
> >>>>>>>>>>>>>>> new
> >>>>>>>>>>>>>>>>>>> memory
> >>>>>>>>>>>>>>>>>>>> storage implementation using
> >>>>>>>>>>>>>>>>>>>> Unsafe (and I did it,
> >>>>>>>>>>>>>>>>>>>> recently) and
> >>>>>>>>>>>>>>>>> also, as
> >>>>>>>>>>>>>>>>>>>> DirectMemory relies heavily
> >>>>>>>>>>>>>>>>>>>> on serialization (and
> >>>>>>>>>>>>>>>>>>>> supports many
> >>>>>>>>>>> of
> >>>>>>>>>>>>>>>>> them,
> >>>>>>>>>>>>>>>>>>>> protostuff, protobuf, msgpack
> >>>>>>>>>>>>>>>>>>>> and of course standard
> >>>>>>>>>>>>>>> serialization),
> >>>>>>>>>>>>>>>>>>> about
> >>>>>>>>>>>>>>>>>>>> creating a simple embedded
> >>>>>>>>>>>>>>>>>>>> serializer leveraging the
> >>>>>>>>>>>>>>>>>>>> same
> >>>>>>>>>>>>>>> techniques
> >>>>>>>>>>>>>>>>> you
> >>>>>>>>>>>>>>>>>>>> used (Unsafe and bytecode
> >>>>>>>>>>>>>>>>>>>> generation). The idea with
> >>>>>>>>>>>>>>>>>>>> embedding is avoiding to
> >>>>>>>>>>>>>>>>>>>> serialize in a byte array
> >>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>> then
> >>>>>>>>>>>>>>>>>>>> moving the byte array to
> >>>>>>>>>>>>>>>>>>>> off-heap memory (via Unsafe
> >>>>>>>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>> ByteBuffers),
> >>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>> serializing directly into a
> >>>>>>>>>>>>>>>>>>>> "managed" off-heap buffer,
> >>>>>>>>>>>>>>>>>>>> thus
> >>>>>>>>>>> further
> >>>>>>>>>>>>>>>>>>>> optimizing heap utilization
> >>>>>>>>>>>>>>>>>>>> (less GC).
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> What do you think about it?
> >>>>>>>>>>>>>>>>>>>> Does it make any sense to
> >>>>>>>>>>>>>>>>>>>> you?
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Ciao, R
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Sat, Sep 29, 2012 at 2:40
> >>>>>>>>>>>>>>>>>>>> PM, Noctarius
> >>>>>>>>>>>>>>>>>>>> <me...@noctarius.com>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Hey guys,
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Raffaele found out about a
> >>>>>>>>>>>>>>>>>>>>> project of mine (Lightning)
> >>>>>>>>>>>>>>>>>>>>> a few
> >>>>>>>>>>> weeks
> >>>>>>>>>>>>>>>>>>>>> ago. Lightning is a heavy
> >>>>>>>>>>>>>>>>>>>>> Unsafe and Bytecode
> >>>>>>>>>>>>>>>>>>>>> generation using
> >>>>>>>>>>>>>>>>>>>>> Serializer implementation.
> >>>>>>>>>>>>>>>>>>>>> He told me that he was
> >>>>>>>>>>>>>>>>>>>>> interested in adding
> >>>>>>>>>>>>>>>>>>>>> something similar to
> >>>>>>>>>>>>>>>>>>>>> DirectMemory and I would be
> >>>>>>>>>>>>>>>>>>>>> glad to
> >>>>>>>>>>>>>>> help
> >>>>>>>>>>>>>>>>>>>>> out in this.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Another project I started a
> >>>>>>>>>>>>>>>>>>>>> few days ago, since it was
> >>>>>>>>>>>>>>>>>>>>> needed
> >>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>> work is DirectRingCache.
> >>>>>>>>>>>>>>>>>>>>> The name not really meets
> >>>>>>>>>>>>>>>>>>>>> to actual implementation
> >>>>>>>>>>>>>>>>>>>>> since it's not yet a ring
> >>>>>>>>>>>>>>>>>>>>> buffer using cache. I
> >>>>>>>>>>>>>>> used
> >>>>>>>>>>>>>>>>>>>>> this for a
> >>>>>>>>>>>>>>>>>>>>> pre-serialization simple
> >>>>>>>>>>>>>>>>>>>>> bytestream cache with
> >>>>>>>>>>>>>>>>>>>>> self-growing buffers. It
> >>>>>>>>>>>>>>>>>>>>> could be nice to have
> >>>>>>>>>>>>>>>>>>>>> DirectMemory
> >>>>>>>>>>> having
> >>>>>>>>>>>>>>>>>>>>> raw "buffers" to write to
> >>>>>>>>>>>>>>>>>>>>> or to read from.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Here are the links from
> >>>>>>>>>>>>>>>>>>>>> the projects:
> >>>>>>>>>>>>>>>>>>>>> https://github.com/noctarius/Lightning
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >> https://github.com/noctarius/direct-ring-cache
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>> It would be nice to help out since I really like
> >>>>>>>>> the idea of
> >>>>>>>>>>>>>>>>>>>>> DirectMemory and since
> >>>>>>>>>>>>>>>>>>>>> direct-ring-cache is some
> >>>>>>>>>>>>>>>>>>>>> kind of
> >>>>>>>>>>>>>>> reinventing
> >>>>>>>>>>>>>>>>>>>>> the wheel.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Cheers Noctarius (Chris)
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> -- Olivier Lamy Talend:
> >>>>>>>>>>>>>>>>> http://coders.talend.com
> >>>>>>>>>>>>>>>>> http://twitter.com/olamy |
> >>>>>>>>>>>>>>>>> http://linkedin.com/in/olamy
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> -- Olivier Lamy Talend:
> >>>>>>>>>>>>>>> http://coders.talend.com
> >>>>>>>>>>>>>>> http://twitter.com/olamy |
> >>>>>>>>>>>>>>> http://linkedin.com/in/olamy
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> -- Olivier Lamy Talend:
> >>>>>>>>>>> http://coders.talend.com
> >>>>>>>>>>> http://twitter.com/olamy |
> >>>>>>>>>>> http://linkedin.com/in/olamy
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> -- Olivier Lamy Talend: http://coders.talend.com
> >>>>>> http://twitter.com/olamy | http://linkedin.com/in/olamy
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> >>
> >
>
>
Re: Additional Serializer and raw Buffer access
Posted by Noctarius <me...@noctarius.com>.
Hey it's me again ;-)
There are at least two files that are not AL2 licensed but can be
used (I think).
https://github.com/noctarius/Lightning/tree/master/lightning-core/src/main/java/com/github/lightning/internal/instantiator/jrockit
If it's not possible to leave them in the sourcetree I'm pretty sure
there will be no problem when we remove them since they are used for
older JRocket versions.
Am 29.09.2012 21:05, schrieb Raffaele P. Guidi:
> I kinda suspected that...
> Il giorno 29/set/2012 20:47, "Noctarius" <me...@noctarius.com> ha scritto:
>
>> Actually all dependencies should be AL2 or BSD licensed :-)
>>
>> Am 29.09.2012 20:42, schrieb Raffaele P. Guidi:
>>>>> Hehe well that really sounds like a nice bunch of people.
>>>
>>> Indeed they are (I'm a newbie as well and try to do my best)
>>>
>>>>> If lightning will be a sub-part (sub-project) of DM, do I
>>>>> need to write
>>> an project purposal?
>>>
>>> Nope, not needed for a sub-project
>>>
>>>>> Do I need to make any changes to the pom.xml like adding a
>>> special parent pom or anything like that?
>>>
>>> Not for the serializer - just have to take a look at project
>>> dependencies - or, better, at their licenses - are they
>>> compatible with the ASL 2.0? i.e. a GPL'd library is not a good
>>> fit and should be replaced with an apache licensed (or BSD, or
>>> MIT...) one if possible. For the integration module is a
>>> separate story - you should start off copying one of the other
>>> serializers and reusing the same pom and directory structure.
>>>
>>> Pleased to meet you, Chris :)
>>>
>>> Ciao, R
>>>
>>> On Sat, Sep 29, 2012 at 8:24 PM, Noctarius <me...@noctarius.com>
>>> wrote:
>>>
>>>> Hehe well that really sounds like a nice bunch of people.
>>>>
>>>> Ok to be true I couldn't wait until tomorrow and started
>>>> already reading the links. From what I was reading: If
>>>> lightning will be a sub-part (sub-project) of DM, do I need
>>>> to write an project purposal?
>>>>
>>>> Do I need to make any changes to the pom.xml like adding a
>>>> special parent pom or anything like that?
>>>>
>>>> In general: there are a lot things to know :-)
>>>>
>>>> Am 29.09.2012 19:59, schrieb Raffaele P. Guidi:
>>>>> Negative part of ASF membership? You get together with a
>>>>> lot of geeky, talented people with a fixation for software
>>>>> and open source. Oh wait but this is actually nice! :-D Il
>>>>> giorno 29/set/2012 19:05, "Olivier Lamy" <ol...@apache.org>
>>>>> ha
>>>> scritto:
>>>>>
>>>>>> 2012/9/29 Noctarius <me...@noctarius.com>:
>>>>>>> Thanks Olivier for carify, I'll take a look in it
>>>>>>> tomorrow but there's just one question left (for now
>>>>>>> ;)): What is that vote for becoming a committer? What
>>>>>>> if the vote will be negative?
>>>>>> The vote is on private list (pmc list for privacy reasons
>>>>>> and possible negative stuff being on public lists)
>>>>>>> Until now I just used Apache stuff, was never
>>>>>>> interested in being part of it so I guess it can be
>>>>>>> negative for any reason, can't it?
>>>>>> I don't see why it could be negative but suspens ....
>>>>>> :-)
>>>>>>>
>>>>>>> Am 29.09.2012 18:56, schrieb Olivier Lamy:
>>>>>>>> 2012/9/29 Noctarius <me...@noctarius.com>:
>>>>>>>>> Nope my real name is Christoph Engelbert, but
>>>>>>>>> Noctarius is the all time nick :)
>>>>>>>>>
>>>>>>>>> Renaming the package should be no problem, is it
>>>>>>>>> "org.apache.directmemory.lightning" or what would
>>>>>>>>> it be?
>>>>>>>> fine for me
>>>>>>>>> Then there needs to be a change in the license
>>>>>>>>> header as Olivier mentioned, that means just remove
>>>>>>>>> the first sentence or is there anything more to do
>>>>>>>>> (maybe it's easiest thing to just copy the header
>>>>>>>>> from DM file ;))?
>>>>>>>> yup use same header as DM
>>>>>>>>>
>>>>>>>>> The CLA is just a form to clarify that the source
>>>>>>>>> can be contributed to the Apache Foundation?
>>>>>>>> yup correct.
>>>>>>>>>
>>>>>>>>> The final step will be attaching the patch in form
>>>>>>>>> of a huge diff file?
>>>>>>>> yes
>>>>>>>>>
>>>>>>>>> And what is the way to apply for a membership?
>>>>>>>>> Never thought about how to do that.
>>>>>>>> Read here
>>>>>>>> http://apache.org/foundation/how-it-works.html and
>>>>>>>> here http://apache.org/foundation/getinvolved.html .
>>>>>>>> And feel free to ask any questions :-)
>>>>>>>>>
>>>>>>>>> Chris
>>>>>>>>>
>>>>>>>>> Am 29.09.2012 18:23, schrieb Raffaele P. Guidi:
>>>>>>>>>> OK, deal, at least for me ;-) I propose you
>>>>>>>>>> rename the packages, produce a patch for this and
>>>>>>>>>> the new serializer module (should be simple
>>>>>>>>>> enough starting from an existing one) and, in the
>>>>>>>>>> meanwhile, apply for ASF membership. Is IP
>>>>>>>>>> clearance needed? I guess yes. After this we will
>>>>>>>>>> come up with a formal vote regarding Noctarius
>>>>>>>>>> (is this your real name?!) allowance in the
>>>>>>>>>> project team.
>>>>>>>>>>
>>>>>>>>>> Good times are gonna come :-)
>>>>>>>>>>
>>>>>>>>>> Thanks, R Il giorno 29/set/2012 17:58, "Olivier
>>>>>>>>>> Lamy" <ol...@apache.org> ha scritto:
>>>>>>>>>>
>>>>>>>>>>> 2012/9/29 Raffaele P. Guidi
>>>>>>>>>>> <ra...@gmail.com>:
>>>>>>>>>>>> Well we already have a NIO ready interface
>>>>>>>>>>>> allowing direct access to DMs managed
>>>>>>>>>>>> bytebuffers but I think this is just half way
>>>>>>>>>>>> to what could be achieved optimally blending
>>>>>>>>>>>> serialization and memory allocation
>>>>>>>>>>>> together.
>>>>>>>>>>>>
>>>>>>>>>>>> Lightning as a module is of course a good
>>>>>>>>>>>> idea and it could easily evolve as a
>>>>>>>>>>>> subproject (for the more experienced asf
>>>>>>>>>>>> members: is it a feasible way?).
>>>>>>>>>>> Nothing prevent to have
>>>>>>>>>>> http://svn.apache.org/repos/asf/directmemory/lightning/trunk
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>
>>>>>>>>>>>
>> with a package like: o.a.d.lightning That will be a
>>>>>>>>>>> subproject.
>>>>>>>>>>>>
>>>>>>>>>>>> Ciao, R Il giorno 29/set/2012 17:44,
>>>>>>>>>>>> "Noctarius" <me...@noctarius.com> ha scritto:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hey guys,
>>>>>>>>>>>>>
>>>>>>>>>>>>> ok there's no lightning binary available
>>>>>>>>>>>>> since lightning wasn't ready yet for
>>>>>>>>>>>>> releasing.
>>>>>>>>>>>>>
>>>>>>>>>>>>> For being the only developer it would be no
>>>>>>>>>>>>> problem to contribute the sourcebase for
>>>>>>>>>>>>> DirectMemory but I'm not sure yet if it
>>>>>>>>>>>>> wouldn't be better to seperate it to be
>>>>>>>>>>>>> available without using DirectMemory
>>>>>>>>>>>>> itself. I started it as a serializer for
>>>>>>>>>>>>> cluster synchronization, but it would be
>>>>>>>>>>>>> cool to contribute lightning as a
>>>>>>>>>>>>> subproject to DirectMemory :-)
>>>>>>>>>>>>>
>>>>>>>>>>>>> About the second project I would love to
>>>>>>>>>>>>> see a public available buffer API directly
>>>>>>>>>>>>> in DirectMemory so that project would be
>>>>>>>>>>>>> nearly needless :-) The only difference I
>>>>>>>>>>>>> think is the allocation strategy my
>>>>>>>>>>>>> implementation is using against the one
>>>>>>>>>>>>> DirectMemory has, but I'm pretty sure the
>>>>>>>>>>>>> allocation is extensible ;-)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Like I said, for both projects I'm the only
>>>>>>>>>>>>> dev so there would be no IP problem. So if
>>>>>>>>>>>>> it's ok to you to not include lightning
>>>>>>>>>>>>> directly in DM I would be glad to
>>>>>>>>>>>>> contribute to the Apache Foundation :-)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers Chris
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Am 29.09.2012 16:53, schrieb Raffaele P.
>>>>>>>>>>>>> Guidi:
>>>>>>>>>>>>>> Ok so it's up to noctarius - your move!
>>>>>>>>>>>>>> ;-) Regarding the new unsafe storage:
>>>>>>>>>>>>>> it's an opt-in feature that can be set
>>>>>>>>>>>>>> with the fluent API
>>>>>>>>>>> (and
>>>>>>>>>>>>>> soon through the conference file).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Ciao, R Il giorno 29/set/2012 16:45,
>>>>>>>>>>>>>> "Olivier Lamy" <ol...@apache.org> ha
>>>>>>>>>>>>> scritto:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2012/9/29 Raffaele P. Guidi
>>>>>>>>>>>>>>> <ra...@gmail.com>:
>>>>>>>>>>>>>>>>>> At least for the moment he can
>>>>>>>>>>>>>>>>>> provide a patch to be integrated
>>>>>>>>>>> in DM
>>>>>>>>>>>>>>> :-)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Sure, but as lightning is not in any
>>>>>>>>>>>>>>>> public mvn repo should its
>>>>>>>>>>> code be
>>>>>>>>>>>>>>>> re-published in our svn? Or what?
>>>>>>>>>>>>>>> @Apache we don't care about binaries,
>>>>>>>>>>>>>>> only sources are important ! (a bit
>>>>>>>>>>>>>>> theorical for sure but that's it :-) ).
>>>>>>>>>>>>>>> So if Noctarius was the only guy who
>>>>>>>>>>>>>>> participate in lightning. He can just
>>>>>>>>>>>>>>> provide a patch we could integrate as a
>>>>>>>>>>>>>>> new dm module (note: the patch must not
>>>>>>>>>>>>>>> contains any more copyright and all
>>>>>>>>>>>>>>> sources must have ASF licenses).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> "Copyright 2012 the original author or
>>>>>>>>>>>>>>> authors." must be removed. And BTW
>>>>>>>>>>>>>>> package must be changed :-) (com.github
>>>>>>>>>>>>>>> is not acceptable
>>>>>>>>>>> @asf
>>>>>>>>>>>>>>> :-) )(@Noctarius are you working for
>>>>>>>>>>>>>>> github ? :-) )
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> And having him as a committer will be
>>>>>>>>>>>>>>> only a matter of voting (we
>>>>>>>>>>> have
>>>>>>>>>>>>>>> a great chair who take cares of
>>>>>>>>>>>>>>> administrative stuff :P )
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> If some others have participated in the
>>>>>>>>>>>>>>> project, we must pass tru an ip
>>>>>>>>>>>>>>> clearance mechanism
>>>>>>>>>>>>>>> (http://incubator.apache.org/ip-clearance/index.html)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>
>>>>>>>>>>>>>>>
>> and all contributors to lightning must provide a cla.
>>>>>>>>>>>>>>> (It it's the case I can help)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> perso I'd like we avoid hard
>>>>>>>>>>>>>>>>>> dependency on Unsafe as maybe
>>>>>>>>>>>>>>>>>> some
>>>>>>>>>>> use
>>>>>>>>>>>>>>>> other jdks :-)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Well, I believe Unsafe is supported
>>>>>>>>>>>>>>>> by Sun JDK, OpenJDK, IBM JDK and
>>>>>>>>>>>>>>>> JRockit - and I believe that it is
>>>>>>>>>>>>>>>> more than enough. Also keep in
>>>>>>>>>>> mind
>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>> we already have an alternative Unsafe
>>>>>>>>>>>>>>>> based memory storage - and
>>>>>>>>>>>>> although
>>>>>>>>>>>>>>>> it's not thoroughly tested for
>>>>>>>>>>>>>>>> performance it dramaticaly
>>>>>>>>>>>>>>>> simplifies
>>>>>>>>>>>>>>> code,
>>>>>>>>>>>>>>>> I have great expectations about it.
>>>>>>>>>>>>>>> Me too :-). Yup definitely more simple
>>>>>>>>>>>>>>> and faster ! But we must provide a
>>>>>>>>>>>>>>> switch off configuration mechanism if
>>>>>>>>>>>>>>> some
>>>>>>>>>>> users
>>>>>>>>>>>>>>> don't want to use that (because Unsafe
>>>>>>>>>>>>>>> is just "Unsafe" :-) ) And sorry I
>>>>>>>>>>>>>>> didn't have a look yet at your changes
>>>>>>>>>>>>>>> with using Unsafe.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Cheers, R
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Sat, Sep 29, 2012 at 4:03 PM,
>>>>>>>>>>>>>>>> Olivier Lamy <ol...@apache.org>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2012/9/29 Raffaele P. Guidi
>>>>>>>>>>>>>>>>> <ra...@gmail.com>:
>>>>>>>>>>>>>>>>>> What do you think about: 1)
>>>>>>>>>>>>>>>>>> implementing a lightning
>>>>>>>>>>>>>>>>>> serialization module 2) creating
>>>>>>>>>>>>>>>>>> a serializer that directly works
>>>>>>>>>>>>>>>>>> on a directmemory
>>>>>>>>>>>>>>> provider
>>>>>>>>>>>>>>>>>> ByteBuffer or (maybe better)
>>>>>>>>>>>>>>>>>> Unsafe based Pointer?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Sounds good (perso I'd like we
>>>>>>>>>>>>>>>>> avoid hard dependency on Unsafe as
>>>>>>>>>>>>>>>>> maybe some use other jdks :-) )
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Now I see lightning is apache
>>>>>>>>>>>>>>>>>> licensed and this is fine but I
>>>>>>>>>>> don't
>>>>>>>>>>>>>>> think
>>>>>>>>>>>>>>>>>> it is published in any public
>>>>>>>>>>>>>>>>>> maven repo, am I right? We could
>>>>>>>>>>> find a
>>>>>>>>>>>>>>> way
>>>>>>>>>>>>>>>>>> to deal with this; options vary
>>>>>>>>>>>>>>>>>> from publishing lightning to the
>>>>>>>>>>> free
>>>>>>>>>>>>>>>>>> sonatype repo, joining the ASF
>>>>>>>>>>>>>>>>>> (which is great anyhow!) and
>>>>>>>>>>>>>>> republishing
>>>>>>>>>>>>>>>>>> lightning code in DirectMemory
>>>>>>>>>>>>>>>>>> becoming a commiter (which has
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> undergo
>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>> PMC vote).
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> At least for the moment he can
>>>>>>>>>>>>>>>>> provide a patch to be integrated
>>>>>>>>>>>>>>>>> in
>>>>>>>>>>> DM
>>>>>>>>>>>>>>> :-)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I'd like to hear your and the
>>>>>>>>>>>>>>>>>> team feelings on this.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks, Raffaele
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Sat, Sep 29, 2012 at 3:27 PM,
>>>>>>>>>>>>>>>>>> Noctarius <me...@noctarius.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hey Raffaele,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> that's quite similar to what I
>>>>>>>>>>>>>>>>>>> did at work. We're developing
>>>>>>>>>>> Flash
>>>>>>>>>>>>>>>>>>> online games and using a
>>>>>>>>>>>>>>>>>>> customized AMF serialization.
>>>>>>>>>>>>>>>>>>> To support async writing of the
>>>>>>>>>>>>>>>>>>> clients event results I added
>>>>>>>>>>>>>>>>>>> serialization
>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>> the components / entities to
>>>>>>>>>>>>>>>>>>> the players zone calculation
>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>> stored
>>>>>>>>>>>>>>>>>>> the pre-serialized bytestream
>>>>>>>>>>>>>>>>>>> directly to the off-heap (using
>>>>>>>>>>>>>>>>>>> direct-ring-cache
>>>>>>>>>>>>>>>>>>> implementation). When the
>>>>>>>>>>>>>>>>>>> client requests the results
>>>>>>>>>>>>>>>>>>> (using long-polling) I just
>>>>>>>>>>>>>>>>>>> write the pre-serialized
>>>>>>>>>>> data to
>>>>>>>>>>>>>>>>>>> the right position to
>>>>>>>>>>>>>>>>>>> deserialize it by standard ways
>>>>>>>>>>>>>>>>>>> on Flash
>>>>>>>>>>> side.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> So yeah an seriliaztion to
>>>>>>>>>>>>>>>>>>> off-heap algorithm would be
>>>>>>>>>>>>>>>>>>> fine. You
>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>> avoid using byte arrays and
>>>>>>>>>>>>>>>>>>> minimalize GC.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Cheers Chris
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Am 29.09.2012 15:02, schrieb
>>>>>>>>>>>>>>>>>>> Raffaele P. Guidi:
>>>>>>>>>>>>>>>>>>>> Nice to hear back from you!
>>>>>>>>>>>>>>>>>>>> Yes, I was thinking about
>>>>>>>>>>>>>>>>>>>> creating a
>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>> memory
>>>>>>>>>>>>>>>>>>>> storage implementation using
>>>>>>>>>>>>>>>>>>>> Unsafe (and I did it,
>>>>>>>>>>>>>>>>>>>> recently) and
>>>>>>>>>>>>>>>>> also, as
>>>>>>>>>>>>>>>>>>>> DirectMemory relies heavily
>>>>>>>>>>>>>>>>>>>> on serialization (and
>>>>>>>>>>>>>>>>>>>> supports many
>>>>>>>>>>> of
>>>>>>>>>>>>>>>>> them,
>>>>>>>>>>>>>>>>>>>> protostuff, protobuf, msgpack
>>>>>>>>>>>>>>>>>>>> and of course standard
>>>>>>>>>>>>>>> serialization),
>>>>>>>>>>>>>>>>>>> about
>>>>>>>>>>>>>>>>>>>> creating a simple embedded
>>>>>>>>>>>>>>>>>>>> serializer leveraging the
>>>>>>>>>>>>>>>>>>>> same
>>>>>>>>>>>>>>> techniques
>>>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>>>>>> used (Unsafe and bytecode
>>>>>>>>>>>>>>>>>>>> generation). The idea with
>>>>>>>>>>>>>>>>>>>> embedding is avoiding to
>>>>>>>>>>>>>>>>>>>> serialize in a byte array
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>>>>>>> moving the byte array to
>>>>>>>>>>>>>>>>>>>> off-heap memory (via Unsafe
>>>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>> ByteBuffers),
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>> serializing directly into a
>>>>>>>>>>>>>>>>>>>> "managed" off-heap buffer,
>>>>>>>>>>>>>>>>>>>> thus
>>>>>>>>>>> further
>>>>>>>>>>>>>>>>>>>> optimizing heap utilization
>>>>>>>>>>>>>>>>>>>> (less GC).
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> What do you think about it?
>>>>>>>>>>>>>>>>>>>> Does it make any sense to
>>>>>>>>>>>>>>>>>>>> you?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Ciao, R
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Sat, Sep 29, 2012 at 2:40
>>>>>>>>>>>>>>>>>>>> PM, Noctarius
>>>>>>>>>>>>>>>>>>>> <me...@noctarius.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hey guys,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Raffaele found out about a
>>>>>>>>>>>>>>>>>>>>> project of mine (Lightning)
>>>>>>>>>>>>>>>>>>>>> a few
>>>>>>>>>>> weeks
>>>>>>>>>>>>>>>>>>>>> ago. Lightning is a heavy
>>>>>>>>>>>>>>>>>>>>> Unsafe and Bytecode
>>>>>>>>>>>>>>>>>>>>> generation using
>>>>>>>>>>>>>>>>>>>>> Serializer implementation.
>>>>>>>>>>>>>>>>>>>>> He told me that he was
>>>>>>>>>>>>>>>>>>>>> interested in adding
>>>>>>>>>>>>>>>>>>>>> something similar to
>>>>>>>>>>>>>>>>>>>>> DirectMemory and I would be
>>>>>>>>>>>>>>>>>>>>> glad to
>>>>>>>>>>>>>>> help
>>>>>>>>>>>>>>>>>>>>> out in this.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Another project I started a
>>>>>>>>>>>>>>>>>>>>> few days ago, since it was
>>>>>>>>>>>>>>>>>>>>> needed
>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>> work is DirectRingCache.
>>>>>>>>>>>>>>>>>>>>> The name not really meets
>>>>>>>>>>>>>>>>>>>>> to actual implementation
>>>>>>>>>>>>>>>>>>>>> since it's not yet a ring
>>>>>>>>>>>>>>>>>>>>> buffer using cache. I
>>>>>>>>>>>>>>> used
>>>>>>>>>>>>>>>>>>>>> this for a
>>>>>>>>>>>>>>>>>>>>> pre-serialization simple
>>>>>>>>>>>>>>>>>>>>> bytestream cache with
>>>>>>>>>>>>>>>>>>>>> self-growing buffers. It
>>>>>>>>>>>>>>>>>>>>> could be nice to have
>>>>>>>>>>>>>>>>>>>>> DirectMemory
>>>>>>>>>>> having
>>>>>>>>>>>>>>>>>>>>> raw "buffers" to write to
>>>>>>>>>>>>>>>>>>>>> or to read from.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Here are the links from
>>>>>>>>>>>>>>>>>>>>> the projects:
>>>>>>>>>>>>>>>>>>>>> https://github.com/noctarius/Lightning
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>> https://github.com/noctarius/direct-ring-cache
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>> It would be nice to help out since I really like
>>>>>>>>> the idea of
>>>>>>>>>>>>>>>>>>>>> DirectMemory and since
>>>>>>>>>>>>>>>>>>>>> direct-ring-cache is some
>>>>>>>>>>>>>>>>>>>>> kind of
>>>>>>>>>>>>>>> reinventing
>>>>>>>>>>>>>>>>>>>>> the wheel.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Cheers Noctarius (Chris)
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> -- Olivier Lamy Talend:
>>>>>>>>>>>>>>>>> http://coders.talend.com
>>>>>>>>>>>>>>>>> http://twitter.com/olamy |
>>>>>>>>>>>>>>>>> http://linkedin.com/in/olamy
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -- Olivier Lamy Talend:
>>>>>>>>>>>>>>> http://coders.talend.com
>>>>>>>>>>>>>>> http://twitter.com/olamy |
>>>>>>>>>>>>>>> http://linkedin.com/in/olamy
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> -- Olivier Lamy Talend:
>>>>>>>>>>> http://coders.talend.com
>>>>>>>>>>> http://twitter.com/olamy |
>>>>>>>>>>> http://linkedin.com/in/olamy
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> -- Olivier Lamy Talend: http://coders.talend.com
>>>>>> http://twitter.com/olamy | http://linkedin.com/in/olamy
>>>>>>
>>>>>
>>>>
>>>
>>
>>
>
Re: Additional Serializer and raw Buffer access
Posted by "Raffaele P. Guidi" <ra...@gmail.com>.
I usually work with git-svn and git at least keeps it locally
Il giorno 30/set/2012 09:37, "Noctarius" <me...@noctarius.com> ha scritto:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Morning,
>
> is there any way to preserve the commit history? I don't think so.
>
> Cheers Chris
>
> Am 29.09.2012 21:05, schrieb Raffaele P. Guidi:
> > I kinda suspected that... Il giorno 29/set/2012 20:47,
> > "Noctarius" <me...@noctarius.com> ha scritto:
> >
> >> Actually all dependencies should be AL2 or BSD licensed :-)
> >>
> >> Am 29.09.2012 20:42, schrieb Raffaele P. Guidi:
> >>>>> Hehe well that really sounds like a nice bunch of
> >>>>> people.
> >>>
> >>> Indeed they are (I'm a newbie as well and try to do my
> >>> best)
> >>>
> >>>>> If lightning will be a sub-part (sub-project) of DM, do
> >>>>> I need to write
> >>> an project purposal?
> >>>
> >>> Nope, not needed for a sub-project
> >>>
> >>>>> Do I need to make any changes to the pom.xml like
> >>>>> adding a
> >>> special parent pom or anything like that?
> >>>
> >>> Not for the serializer - just have to take a look at
> >>> project dependencies - or, better, at their licenses - are
> >>> they compatible with the ASL 2.0? i.e. a GPL'd library is
> >>> not a good fit and should be replaced with an apache
> >>> licensed (or BSD, or MIT...) one if possible. For the
> >>> integration module is a separate story - you should start
> >>> off copying one of the other serializers and reusing the
> >>> same pom and directory structure.
> >>>
> >>> Pleased to meet you, Chris :)
> >>>
> >>> Ciao, R
> >>>
> >>> On Sat, Sep 29, 2012 at 8:24 PM, Noctarius
> >>> <me...@noctarius.com> wrote:
> >>>
> >>>> Hehe well that really sounds like a nice bunch of
> >>>> people.
> >>>>
> >>>> Ok to be true I couldn't wait until tomorrow and started
> >>>> already reading the links. From what I was reading: If
> >>>> lightning will be a sub-part (sub-project) of DM, do I
> >>>> need to write an project purposal?
> >>>>
> >>>> Do I need to make any changes to the pom.xml like adding
> >>>> a special parent pom or anything like that?
> >>>>
> >>>> In general: there are a lot things to know :-)
> >>>>
> >>>> Am 29.09.2012 19:59, schrieb Raffaele P. Guidi:
> >>>>> Negative part of ASF membership? You get together with
> >>>>> a lot of geeky, talented people with a fixation for
> >>>>> software and open source. Oh wait but this is actually
> >>>>> nice! :-D Il giorno 29/set/2012 19:05, "Olivier Lamy"
> >>>>> <ol...@apache.org> ha
> >>>> scritto:
> >>>>>
> >>>>>> 2012/9/29 Noctarius <me...@noctarius.com>:
> >>>>>>> Thanks Olivier for carify, I'll take a look in it
> >>>>>>> tomorrow but there's just one question left (for
> >>>>>>> now ;)): What is that vote for becoming a
> >>>>>>> committer? What if the vote will be negative?
> >>>>>> The vote is on private list (pmc list for privacy
> >>>>>> reasons and possible negative stuff being on public
> >>>>>> lists)
> >>>>>>> Until now I just used Apache stuff, was never
> >>>>>>> interested in being part of it so I guess it can
> >>>>>>> be negative for any reason, can't it?
> >>>>>> I don't see why it could be negative but suspens
> >>>>>> .... :-)
> >>>>>>>
> >>>>>>> Am 29.09.2012 18:56, schrieb Olivier Lamy:
> >>>>>>>> 2012/9/29 Noctarius <me...@noctarius.com>:
> >>>>>>>>> Nope my real name is Christoph Engelbert, but
> >>>>>>>>> Noctarius is the all time nick :)
> >>>>>>>>>
> >>>>>>>>> Renaming the package should be no problem, is
> >>>>>>>>> it "org.apache.directmemory.lightning" or what
> >>>>>>>>> would it be?
> >>>>>>>> fine for me
> >>>>>>>>> Then there needs to be a change in the license
> >>>>>>>>> header as Olivier mentioned, that means just
> >>>>>>>>> remove the first sentence or is there anything
> >>>>>>>>> more to do (maybe it's easiest thing to just
> >>>>>>>>> copy the header from DM file ;))?
> >>>>>>>> yup use same header as DM
> >>>>>>>>>
> >>>>>>>>> The CLA is just a form to clarify that the
> >>>>>>>>> source can be contributed to the Apache
> >>>>>>>>> Foundation?
> >>>>>>>> yup correct.
> >>>>>>>>>
> >>>>>>>>> The final step will be attaching the patch in
> >>>>>>>>> form of a huge diff file?
> >>>>>>>> yes
> >>>>>>>>>
> >>>>>>>>> And what is the way to apply for a membership?
> >>>>>>>>> Never thought about how to do that.
> >>>>>>>> Read here
> >>>>>>>> http://apache.org/foundation/how-it-works.html
> >>>>>>>> and here
> >>>>>>>> http://apache.org/foundation/getinvolved.html .
> >>>>>>>> And feel free to ask any questions :-)
> >>>>>>>>>
> >>>>>>>>> Chris
> >>>>>>>>>
> >>>>>>>>> Am 29.09.2012 18:23, schrieb Raffaele P.
> >>>>>>>>> Guidi:
> >>>>>>>>>> OK, deal, at least for me ;-) I propose you
> >>>>>>>>>> rename the packages, produce a patch for this
> >>>>>>>>>> and the new serializer module (should be
> >>>>>>>>>> simple enough starting from an existing one)
> >>>>>>>>>> and, in the meanwhile, apply for ASF
> >>>>>>>>>> membership. Is IP clearance needed? I guess
> >>>>>>>>>> yes. After this we will come up with a formal
> >>>>>>>>>> vote regarding Noctarius (is this your real
> >>>>>>>>>> name?!) allowance in the project team.
> >>>>>>>>>>
> >>>>>>>>>> Good times are gonna come :-)
> >>>>>>>>>>
> >>>>>>>>>> Thanks, R Il giorno 29/set/2012 17:58,
> >>>>>>>>>> "Olivier Lamy" <ol...@apache.org> ha
> >>>>>>>>>> scritto:
> >>>>>>>>>>
> >>>>>>>>>>> 2012/9/29 Raffaele P. Guidi
> >>>>>>>>>>> <ra...@gmail.com>:
> >>>>>>>>>>>> Well we already have a NIO ready
> >>>>>>>>>>>> interface allowing direct access to DMs
> >>>>>>>>>>>> managed bytebuffers but I think this is
> >>>>>>>>>>>> just half way to what could be achieved
> >>>>>>>>>>>> optimally blending serialization and
> >>>>>>>>>>>> memory allocation together.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Lightning as a module is of course a
> >>>>>>>>>>>> good idea and it could easily evolve as
> >>>>>>>>>>>> a subproject (for the more experienced
> >>>>>>>>>>>> asf members: is it a feasible way?).
> >>>>>>>>>>> Nothing prevent to have
> >>>>>>>>>>> http://svn.apache.org/repos/asf/directmemory/lightning/trunk
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>
> >>>>>>>>>>>
> >>
> >>>>>>>>>>>
> with a package like: o.a.d.lightning That will be a
> >>>>>>>>>>> subproject.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Ciao, R Il giorno 29/set/2012 17:44,
> >>>>>>>>>>>> "Noctarius" <me...@noctarius.com> ha
> >>>>>>>>>>>> scritto:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Hey guys,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> ok there's no lightning binary
> >>>>>>>>>>>>> available since lightning wasn't ready
> >>>>>>>>>>>>> yet for releasing.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> For being the only developer it would
> >>>>>>>>>>>>> be no problem to contribute the
> >>>>>>>>>>>>> sourcebase for DirectMemory but I'm not
> >>>>>>>>>>>>> sure yet if it wouldn't be better to
> >>>>>>>>>>>>> seperate it to be available without
> >>>>>>>>>>>>> using DirectMemory itself. I started it
> >>>>>>>>>>>>> as a serializer for cluster
> >>>>>>>>>>>>> synchronization, but it would be cool
> >>>>>>>>>>>>> to contribute lightning as a subproject
> >>>>>>>>>>>>> to DirectMemory :-)
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> About the second project I would love
> >>>>>>>>>>>>> to see a public available buffer API
> >>>>>>>>>>>>> directly in DirectMemory so that
> >>>>>>>>>>>>> project would be nearly needless :-)
> >>>>>>>>>>>>> The only difference I think is the
> >>>>>>>>>>>>> allocation strategy my implementation
> >>>>>>>>>>>>> is using against the one DirectMemory
> >>>>>>>>>>>>> has, but I'm pretty sure the allocation
> >>>>>>>>>>>>> is extensible ;-)
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Like I said, for both projects I'm the
> >>>>>>>>>>>>> only dev so there would be no IP
> >>>>>>>>>>>>> problem. So if it's ok to you to not
> >>>>>>>>>>>>> include lightning directly in DM I
> >>>>>>>>>>>>> would be glad to contribute to the
> >>>>>>>>>>>>> Apache Foundation :-)
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Cheers Chris
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Am 29.09.2012 16:53, schrieb Raffaele
> >>>>>>>>>>>>> P. Guidi:
> >>>>>>>>>>>>>> Ok so it's up to noctarius - your
> >>>>>>>>>>>>>> move! ;-) Regarding the new unsafe
> >>>>>>>>>>>>>> storage: it's an opt-in feature that
> >>>>>>>>>>>>>> can be set with the fluent API
> >>>>>>>>>>> (and
> >>>>>>>>>>>>>> soon through the conference file).
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Ciao, R Il giorno 29/set/2012 16:45,
> >>>>>>>>>>>>>> "Olivier Lamy" <ol...@apache.org> ha
> >>>>>>>>>>>>> scritto:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> 2012/9/29 Raffaele P. Guidi
> >>>>>>>>>>>>>>> <ra...@gmail.com>:
> >>>>>>>>>>>>>>>>>> At least for the moment he
> >>>>>>>>>>>>>>>>>> can provide a patch to be
> >>>>>>>>>>>>>>>>>> integrated
> >>>>>>>>>>> in DM
> >>>>>>>>>>>>>>> :-)
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Sure, but as lightning is not in
> >>>>>>>>>>>>>>>> any public mvn repo should its
> >>>>>>>>>>> code be
> >>>>>>>>>>>>>>>> re-published in our svn? Or
> >>>>>>>>>>>>>>>> what?
> >>>>>>>>>>>>>>> @Apache we don't care about
> >>>>>>>>>>>>>>> binaries, only sources are
> >>>>>>>>>>>>>>> important ! (a bit theorical for
> >>>>>>>>>>>>>>> sure but that's it :-) ). So if
> >>>>>>>>>>>>>>> Noctarius was the only guy who
> >>>>>>>>>>>>>>> participate in lightning. He can
> >>>>>>>>>>>>>>> just provide a patch we could
> >>>>>>>>>>>>>>> integrate as a new dm module (note:
> >>>>>>>>>>>>>>> the patch must not contains any
> >>>>>>>>>>>>>>> more copyright and all sources must
> >>>>>>>>>>>>>>> have ASF licenses).
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> "Copyright 2012 the original author
> >>>>>>>>>>>>>>> or authors." must be removed. And
> >>>>>>>>>>>>>>> BTW package must be changed :-)
> >>>>>>>>>>>>>>> (com.github is not acceptable
> >>>>>>>>>>> @asf
> >>>>>>>>>>>>>>> :-) )(@Noctarius are you working
> >>>>>>>>>>>>>>> for github ? :-) )
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> And having him as a committer will
> >>>>>>>>>>>>>>> be only a matter of voting (we
> >>>>>>>>>>> have
> >>>>>>>>>>>>>>> a great chair who take cares of
> >>>>>>>>>>>>>>> administrative stuff :P )
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> If some others have participated in
> >>>>>>>>>>>>>>> the project, we must pass tru an
> >>>>>>>>>>>>>>> ip clearance mechanism
> >>>>>>>>>>>>>>> (http://incubator.apache.org/ip-clearance/index.html)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>
> >>
> >>>>>>>>>>>>>>>
> and all contributors to lightning must provide a cla.
> >>>>>>>>>>>>>>> (It it's the case I can help)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> perso I'd like we avoid hard
> >>>>>>>>>>>>>>>>>> dependency on Unsafe as
> >>>>>>>>>>>>>>>>>> maybe some
> >>>>>>>>>>> use
> >>>>>>>>>>>>>>>> other jdks :-)
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Well, I believe Unsafe is
> >>>>>>>>>>>>>>>> supported by Sun JDK, OpenJDK,
> >>>>>>>>>>>>>>>> IBM JDK and JRockit - and I
> >>>>>>>>>>>>>>>> believe that it is more than
> >>>>>>>>>>>>>>>> enough. Also keep in
> >>>>>>>>>>> mind
> >>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>> we already have an alternative
> >>>>>>>>>>>>>>>> Unsafe based memory storage -
> >>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>> although
> >>>>>>>>>>>>>>>> it's not thoroughly tested for
> >>>>>>>>>>>>>>>> performance it dramaticaly
> >>>>>>>>>>>>>>>> simplifies
> >>>>>>>>>>>>>>> code,
> >>>>>>>>>>>>>>>> I have great expectations about
> >>>>>>>>>>>>>>>> it.
> >>>>>>>>>>>>>>> Me too :-). Yup definitely more
> >>>>>>>>>>>>>>> simple and faster ! But we must
> >>>>>>>>>>>>>>> provide a switch off configuration
> >>>>>>>>>>>>>>> mechanism if some
> >>>>>>>>>>> users
> >>>>>>>>>>>>>>> don't want to use that (because
> >>>>>>>>>>>>>>> Unsafe is just "Unsafe" :-) ) And
> >>>>>>>>>>>>>>> sorry I didn't have a look yet at
> >>>>>>>>>>>>>>> your changes with using Unsafe.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Cheers, R
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Sat, Sep 29, 2012 at 4:03 PM,
> >>>>>>>>>>>>>>>> Olivier Lamy <ol...@apache.org>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> 2012/9/29 Raffaele P. Guidi
> >>>>>>>>>>>>>>>>> <ra...@gmail.com>:
> >>>>>>>>>>>>>>>>>> What do you think about: 1)
> >>>>>>>>>>>>>>>>>> implementing a lightning
> >>>>>>>>>>>>>>>>>> serialization module 2)
> >>>>>>>>>>>>>>>>>> creating a serializer that
> >>>>>>>>>>>>>>>>>> directly works on a
> >>>>>>>>>>>>>>>>>> directmemory
> >>>>>>>>>>>>>>> provider
> >>>>>>>>>>>>>>>>>> ByteBuffer or (maybe better)
> >>>>>>>>>>>>>>>>>> Unsafe based Pointer?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Sounds good (perso I'd like we
> >>>>>>>>>>>>>>>>> avoid hard dependency on Unsafe
> >>>>>>>>>>>>>>>>> as maybe some use other jdks
> >>>>>>>>>>>>>>>>> :-) )
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Now I see lightning is
> >>>>>>>>>>>>>>>>>> apache licensed and this is
> >>>>>>>>>>>>>>>>>> fine but I
> >>>>>>>>>>> don't
> >>>>>>>>>>>>>>> think
> >>>>>>>>>>>>>>>>>> it is published in any
> >>>>>>>>>>>>>>>>>> public maven repo, am I
> >>>>>>>>>>>>>>>>>> right? We could
> >>>>>>>>>>> find a
> >>>>>>>>>>>>>>> way
> >>>>>>>>>>>>>>>>>> to deal with this; options
> >>>>>>>>>>>>>>>>>> vary from publishing
> >>>>>>>>>>>>>>>>>> lightning to the
> >>>>>>>>>>> free
> >>>>>>>>>>>>>>>>>> sonatype repo, joining the
> >>>>>>>>>>>>>>>>>> ASF (which is great anyhow!)
> >>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>> republishing
> >>>>>>>>>>>>>>>>>> lightning code in
> >>>>>>>>>>>>>>>>>> DirectMemory becoming a
> >>>>>>>>>>>>>>>>>> commiter (which has to
> >>>>>>>>>>>>>>> undergo
> >>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>> PMC vote).
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> At least for the moment he can
> >>>>>>>>>>>>>>>>> provide a patch to be
> >>>>>>>>>>>>>>>>> integrated in
> >>>>>>>>>>> DM
> >>>>>>>>>>>>>>> :-)
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I'd like to hear your and
> >>>>>>>>>>>>>>>>>> the team feelings on this.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks, Raffaele
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Sat, Sep 29, 2012 at 3:27
> >>>>>>>>>>>>>>>>>> PM, Noctarius
> >>>>>>>>>>>>>>>>>> <me...@noctarius.com>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Hey Raffaele,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> that's quite similar to
> >>>>>>>>>>>>>>>>>>> what I did at work. We're
> >>>>>>>>>>>>>>>>>>> developing
> >>>>>>>>>>> Flash
> >>>>>>>>>>>>>>>>>>> online games and using a
> >>>>>>>>>>>>>>>>>>> customized AMF
> >>>>>>>>>>>>>>>>>>> serialization. To support
> >>>>>>>>>>>>>>>>>>> async writing of the
> >>>>>>>>>>>>>>>>>>> clients event results I
> >>>>>>>>>>>>>>>>>>> added serialization
> >>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>> the components / entities
> >>>>>>>>>>>>>>>>>>> to the players zone
> >>>>>>>>>>>>>>>>>>> calculation and
> >>>>>>>>>>> stored
> >>>>>>>>>>>>>>>>>>> the pre-serialized
> >>>>>>>>>>>>>>>>>>> bytestream directly to the
> >>>>>>>>>>>>>>>>>>> off-heap (using
> >>>>>>>>>>>>>>>>>>> direct-ring-cache
> >>>>>>>>>>>>>>>>>>> implementation). When the
> >>>>>>>>>>>>>>>>>>> client requests the
> >>>>>>>>>>>>>>>>>>> results (using
> >>>>>>>>>>>>>>>>>>> long-polling) I just write
> >>>>>>>>>>>>>>>>>>> the pre-serialized
> >>>>>>>>>>> data to
> >>>>>>>>>>>>>>>>>>> the right position to
> >>>>>>>>>>>>>>>>>>> deserialize it by standard
> >>>>>>>>>>>>>>>>>>> ways on Flash
> >>>>>>>>>>> side.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> So yeah an seriliaztion to
> >>>>>>>>>>>>>>>>>>> off-heap algorithm would
> >>>>>>>>>>>>>>>>>>> be fine. You
> >>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>> avoid using byte arrays
> >>>>>>>>>>>>>>>>>>> and minimalize GC.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Cheers Chris
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Am 29.09.2012 15:02,
> >>>>>>>>>>>>>>>>>>> schrieb Raffaele P. Guidi:
> >>>>>>>>>>>>>>>>>>>> Nice to hear back from
> >>>>>>>>>>>>>>>>>>>> you! Yes, I was thinking
> >>>>>>>>>>>>>>>>>>>> about creating a
> >>>>>>>>>>>>>>> new
> >>>>>>>>>>>>>>>>>>> memory
> >>>>>>>>>>>>>>>>>>>> storage implementation
> >>>>>>>>>>>>>>>>>>>> using Unsafe (and I did
> >>>>>>>>>>>>>>>>>>>> it, recently) and
> >>>>>>>>>>>>>>>>> also, as
> >>>>>>>>>>>>>>>>>>>> DirectMemory relies
> >>>>>>>>>>>>>>>>>>>> heavily on serialization
> >>>>>>>>>>>>>>>>>>>> (and supports many
> >>>>>>>>>>> of
> >>>>>>>>>>>>>>>>> them,
> >>>>>>>>>>>>>>>>>>>> protostuff, protobuf,
> >>>>>>>>>>>>>>>>>>>> msgpack and of course
> >>>>>>>>>>>>>>>>>>>> standard
> >>>>>>>>>>>>>>> serialization),
> >>>>>>>>>>>>>>>>>>> about
> >>>>>>>>>>>>>>>>>>>> creating a simple
> >>>>>>>>>>>>>>>>>>>> embedded serializer
> >>>>>>>>>>>>>>>>>>>> leveraging the same
> >>>>>>>>>>>>>>> techniques
> >>>>>>>>>>>>>>>>> you
> >>>>>>>>>>>>>>>>>>>> used (Unsafe and
> >>>>>>>>>>>>>>>>>>>> bytecode generation). The
> >>>>>>>>>>>>>>>>>>>> idea with embedding is
> >>>>>>>>>>>>>>>>>>>> avoiding to serialize in
> >>>>>>>>>>>>>>>>>>>> a byte array
> >>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>> then
> >>>>>>>>>>>>>>>>>>>> moving the byte array to
> >>>>>>>>>>>>>>>>>>>> off-heap memory (via
> >>>>>>>>>>>>>>>>>>>> Unsafe or
> >>>>>>>>>>>>>>> ByteBuffers),
> >>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>> serializing directly into
> >>>>>>>>>>>>>>>>>>>> a "managed" off-heap
> >>>>>>>>>>>>>>>>>>>> buffer, thus
> >>>>>>>>>>> further
> >>>>>>>>>>>>>>>>>>>> optimizing heap
> >>>>>>>>>>>>>>>>>>>> utilization (less GC).
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> What do you think about
> >>>>>>>>>>>>>>>>>>>> it? Does it make any
> >>>>>>>>>>>>>>>>>>>> sense to you?
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Ciao, R
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Sat, Sep 29, 2012 at
> >>>>>>>>>>>>>>>>>>>> 2:40 PM, Noctarius
> >>>>>>>>>>>>>>>>>>>> <me...@noctarius.com>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Hey guys,
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Raffaele found out
> >>>>>>>>>>>>>>>>>>>>> about a project of mine
> >>>>>>>>>>>>>>>>>>>>> (Lightning) a few
> >>>>>>>>>>> weeks
> >>>>>>>>>>>>>>>>>>>>> ago. Lightning is a
> >>>>>>>>>>>>>>>>>>>>> heavy Unsafe and
> >>>>>>>>>>>>>>>>>>>>> Bytecode generation
> >>>>>>>>>>>>>>>>>>>>> using Serializer
> >>>>>>>>>>>>>>>>>>>>> implementation. He told
> >>>>>>>>>>>>>>>>>>>>> me that he was
> >>>>>>>>>>>>>>>>>>>>> interested in adding
> >>>>>>>>>>>>>>>>>>>>> something similar to
> >>>>>>>>>>>>>>>>>>>>> DirectMemory and I
> >>>>>>>>>>>>>>>>>>>>> would be glad to
> >>>>>>>>>>>>>>> help
> >>>>>>>>>>>>>>>>>>>>> out in this.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Another project I
> >>>>>>>>>>>>>>>>>>>>> started a few days ago,
> >>>>>>>>>>>>>>>>>>>>> since it was needed
> >>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>> work is
> >>>>>>>>>>>>>>>>>>>>> DirectRingCache. The
> >>>>>>>>>>>>>>>>>>>>> name not really meets
> >>>>>>>>>>>>>>>>>>>>> to actual
> >>>>>>>>>>>>>>>>>>>>> implementation since
> >>>>>>>>>>>>>>>>>>>>> it's not yet a ring
> >>>>>>>>>>>>>>>>>>>>> buffer using cache. I
> >>>>>>>>>>>>>>> used
> >>>>>>>>>>>>>>>>>>>>> this for a
> >>>>>>>>>>>>>>>>>>>>> pre-serialization
> >>>>>>>>>>>>>>>>>>>>> simple bytestream cache
> >>>>>>>>>>>>>>>>>>>>> with self-growing
> >>>>>>>>>>>>>>>>>>>>> buffers. It could be
> >>>>>>>>>>>>>>>>>>>>> nice to have
> >>>>>>>>>>>>>>>>>>>>> DirectMemory
> >>>>>>>>>>> having
> >>>>>>>>>>>>>>>>>>>>> raw "buffers" to write
> >>>>>>>>>>>>>>>>>>>>> to or to read from.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Here are the links
> >>>>>>>>>>>>>>>>>>>>> from the projects:
> >>>>>>>>>>>>>>>>>>>>> https://github.com/noctarius/Lightning
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>
> >>>>>>>>>>>>>>>>>>>>>
> https://github.com/noctarius/direct-ring-cache
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>> It would be nice to help out since I really
> >>>>>>>>> like the idea of
> >>>>>>>>>>>>>>>>>>>>> DirectMemory and since
> >>>>>>>>>>>>>>>>>>>>> direct-ring-cache is
> >>>>>>>>>>>>>>>>>>>>> some kind of
> >>>>>>>>>>>>>>> reinventing
> >>>>>>>>>>>>>>>>>>>>> the wheel.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Cheers Noctarius
> >>>>>>>>>>>>>>>>>>>>> (Chris)
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> -- Olivier Lamy Talend:
> >>>>>>>>>>>>>>>>> http://coders.talend.com
> >>>>>>>>>>>>>>>>> http://twitter.com/olamy |
> >>>>>>>>>>>>>>>>> http://linkedin.com/in/olamy
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> -- Olivier Lamy Talend:
> >>>>>>>>>>>>>>> http://coders.talend.com
> >>>>>>>>>>>>>>> http://twitter.com/olamy |
> >>>>>>>>>>>>>>> http://linkedin.com/in/olamy
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> -- Olivier Lamy Talend:
> >>>>>>>>>>> http://coders.talend.com
> >>>>>>>>>>> http://twitter.com/olamy |
> >>>>>>>>>>> http://linkedin.com/in/olamy
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> -- Olivier Lamy Talend: http://coders.talend.com
> >>>>>> http://twitter.com/olamy |
> >>>>>> http://linkedin.com/in/olamy
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> >>
> >
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.12 (MingW32)
> Comment: Using GnuPG with Mozilla - http://www.enigmail.net/
>
> iQIcBAEBAgAGBQJQZ/ZZAAoJEH/g+YBfahrqbIUQALEwBddCj1OY7z2QYgUDze6u
> t2RfhWdWpArExVG4vof46xwJ6RmF5T6muIjl3MoR8B6dG46QeBxv3bPq0RmUkcAF
> U8USSMnC9w6Qr6DCjCwjqLb7dnUBgNGqLIMdIIePX7+UxnOOsQ9l4Ext0Vsz5xn3
> 2qkmy7nxiuBkox9OJlXlB8Nt//3LgHRi3iIuBpZ6S4GEwILIQ/UT0WuGKFx6+Sa7
> bOwtSnViERwYNUiFth8O3SS4KmCsEHwWijV6vd+/jxVGhFqMSzxhj5++KzhCe+GZ
> /WavR3rrvmZot9Mmpc4wDWj+6l6PXQrIXqxZDy9SrV7619r0Mh/SvbAKasP+/uMb
> XLjQ/eZoAXRJ34G2l9l3q33lqwv7GK0+zcmgbgX6qun1eKUMuTR/08qfxu1UZO/k
> MMAEZKfT50sNiEDHFKqyGLk8hgTh4nvoCPwxNZBWbezgIFg+8NdigdAlgIxWb+59
> HwTaFtGmbZYkje0dx6WkoNXgLUsLLGOAq+rSbn4Pz3594hb7leJOIgGKUXA1/Eak
> XoXFaXR/eg9ICuzrgN17Apz7GqN4HT3HxmGk6h+J6jkLTFOfyK5J6WQpP0hEXU5O
> 7dH6q0Zc/Af1J8eT2IGUrbbbwRWwXxIBUEQocon4VoDa219gmev/fxrRlL1xuB9f
> 7LS2c0gONRWWdl7M5n8+
> =/dDf
> -----END PGP SIGNATURE-----
>
Re: Additional Serializer and raw Buffer access
Posted by Noctarius <me...@noctarius.com>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Morning,
is there any way to preserve the commit history? I don't think so.
Cheers Chris
Am 29.09.2012 21:05, schrieb Raffaele P. Guidi:
> I kinda suspected that... Il giorno 29/set/2012 20:47,
> "Noctarius" <me...@noctarius.com> ha scritto:
>
>> Actually all dependencies should be AL2 or BSD licensed :-)
>>
>> Am 29.09.2012 20:42, schrieb Raffaele P. Guidi:
>>>>> Hehe well that really sounds like a nice bunch of
>>>>> people.
>>>
>>> Indeed they are (I'm a newbie as well and try to do my
>>> best)
>>>
>>>>> If lightning will be a sub-part (sub-project) of DM, do
>>>>> I need to write
>>> an project purposal?
>>>
>>> Nope, not needed for a sub-project
>>>
>>>>> Do I need to make any changes to the pom.xml like
>>>>> adding a
>>> special parent pom or anything like that?
>>>
>>> Not for the serializer - just have to take a look at
>>> project dependencies - or, better, at their licenses - are
>>> they compatible with the ASL 2.0? i.e. a GPL'd library is
>>> not a good fit and should be replaced with an apache
>>> licensed (or BSD, or MIT...) one if possible. For the
>>> integration module is a separate story - you should start
>>> off copying one of the other serializers and reusing the
>>> same pom and directory structure.
>>>
>>> Pleased to meet you, Chris :)
>>>
>>> Ciao, R
>>>
>>> On Sat, Sep 29, 2012 at 8:24 PM, Noctarius
>>> <me...@noctarius.com> wrote:
>>>
>>>> Hehe well that really sounds like a nice bunch of
>>>> people.
>>>>
>>>> Ok to be true I couldn't wait until tomorrow and started
>>>> already reading the links. From what I was reading: If
>>>> lightning will be a sub-part (sub-project) of DM, do I
>>>> need to write an project purposal?
>>>>
>>>> Do I need to make any changes to the pom.xml like adding
>>>> a special parent pom or anything like that?
>>>>
>>>> In general: there are a lot things to know :-)
>>>>
>>>> Am 29.09.2012 19:59, schrieb Raffaele P. Guidi:
>>>>> Negative part of ASF membership? You get together with
>>>>> a lot of geeky, talented people with a fixation for
>>>>> software and open source. Oh wait but this is actually
>>>>> nice! :-D Il giorno 29/set/2012 19:05, "Olivier Lamy"
>>>>> <ol...@apache.org> ha
>>>> scritto:
>>>>>
>>>>>> 2012/9/29 Noctarius <me...@noctarius.com>:
>>>>>>> Thanks Olivier for carify, I'll take a look in it
>>>>>>> tomorrow but there's just one question left (for
>>>>>>> now ;)): What is that vote for becoming a
>>>>>>> committer? What if the vote will be negative?
>>>>>> The vote is on private list (pmc list for privacy
>>>>>> reasons and possible negative stuff being on public
>>>>>> lists)
>>>>>>> Until now I just used Apache stuff, was never
>>>>>>> interested in being part of it so I guess it can
>>>>>>> be negative for any reason, can't it?
>>>>>> I don't see why it could be negative but suspens
>>>>>> .... :-)
>>>>>>>
>>>>>>> Am 29.09.2012 18:56, schrieb Olivier Lamy:
>>>>>>>> 2012/9/29 Noctarius <me...@noctarius.com>:
>>>>>>>>> Nope my real name is Christoph Engelbert, but
>>>>>>>>> Noctarius is the all time nick :)
>>>>>>>>>
>>>>>>>>> Renaming the package should be no problem, is
>>>>>>>>> it "org.apache.directmemory.lightning" or what
>>>>>>>>> would it be?
>>>>>>>> fine for me
>>>>>>>>> Then there needs to be a change in the license
>>>>>>>>> header as Olivier mentioned, that means just
>>>>>>>>> remove the first sentence or is there anything
>>>>>>>>> more to do (maybe it's easiest thing to just
>>>>>>>>> copy the header from DM file ;))?
>>>>>>>> yup use same header as DM
>>>>>>>>>
>>>>>>>>> The CLA is just a form to clarify that the
>>>>>>>>> source can be contributed to the Apache
>>>>>>>>> Foundation?
>>>>>>>> yup correct.
>>>>>>>>>
>>>>>>>>> The final step will be attaching the patch in
>>>>>>>>> form of a huge diff file?
>>>>>>>> yes
>>>>>>>>>
>>>>>>>>> And what is the way to apply for a membership?
>>>>>>>>> Never thought about how to do that.
>>>>>>>> Read here
>>>>>>>> http://apache.org/foundation/how-it-works.html
>>>>>>>> and here
>>>>>>>> http://apache.org/foundation/getinvolved.html .
>>>>>>>> And feel free to ask any questions :-)
>>>>>>>>>
>>>>>>>>> Chris
>>>>>>>>>
>>>>>>>>> Am 29.09.2012 18:23, schrieb Raffaele P.
>>>>>>>>> Guidi:
>>>>>>>>>> OK, deal, at least for me ;-) I propose you
>>>>>>>>>> rename the packages, produce a patch for this
>>>>>>>>>> and the new serializer module (should be
>>>>>>>>>> simple enough starting from an existing one)
>>>>>>>>>> and, in the meanwhile, apply for ASF
>>>>>>>>>> membership. Is IP clearance needed? I guess
>>>>>>>>>> yes. After this we will come up with a formal
>>>>>>>>>> vote regarding Noctarius (is this your real
>>>>>>>>>> name?!) allowance in the project team.
>>>>>>>>>>
>>>>>>>>>> Good times are gonna come :-)
>>>>>>>>>>
>>>>>>>>>> Thanks, R Il giorno 29/set/2012 17:58,
>>>>>>>>>> "Olivier Lamy" <ol...@apache.org> ha
>>>>>>>>>> scritto:
>>>>>>>>>>
>>>>>>>>>>> 2012/9/29 Raffaele P. Guidi
>>>>>>>>>>> <ra...@gmail.com>:
>>>>>>>>>>>> Well we already have a NIO ready
>>>>>>>>>>>> interface allowing direct access to DMs
>>>>>>>>>>>> managed bytebuffers but I think this is
>>>>>>>>>>>> just half way to what could be achieved
>>>>>>>>>>>> optimally blending serialization and
>>>>>>>>>>>> memory allocation together.
>>>>>>>>>>>>
>>>>>>>>>>>> Lightning as a module is of course a
>>>>>>>>>>>> good idea and it could easily evolve as
>>>>>>>>>>>> a subproject (for the more experienced
>>>>>>>>>>>> asf members: is it a feasible way?).
>>>>>>>>>>> Nothing prevent to have
>>>>>>>>>>> http://svn.apache.org/repos/asf/directmemory/lightning/trunk
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>
>>>>>>>>>>>
>>
>>>>>>>>>>>
with a package like: o.a.d.lightning That will be a
>>>>>>>>>>> subproject.
>>>>>>>>>>>>
>>>>>>>>>>>> Ciao, R Il giorno 29/set/2012 17:44,
>>>>>>>>>>>> "Noctarius" <me...@noctarius.com> ha
>>>>>>>>>>>> scritto:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hey guys,
>>>>>>>>>>>>>
>>>>>>>>>>>>> ok there's no lightning binary
>>>>>>>>>>>>> available since lightning wasn't ready
>>>>>>>>>>>>> yet for releasing.
>>>>>>>>>>>>>
>>>>>>>>>>>>> For being the only developer it would
>>>>>>>>>>>>> be no problem to contribute the
>>>>>>>>>>>>> sourcebase for DirectMemory but I'm not
>>>>>>>>>>>>> sure yet if it wouldn't be better to
>>>>>>>>>>>>> seperate it to be available without
>>>>>>>>>>>>> using DirectMemory itself. I started it
>>>>>>>>>>>>> as a serializer for cluster
>>>>>>>>>>>>> synchronization, but it would be cool
>>>>>>>>>>>>> to contribute lightning as a subproject
>>>>>>>>>>>>> to DirectMemory :-)
>>>>>>>>>>>>>
>>>>>>>>>>>>> About the second project I would love
>>>>>>>>>>>>> to see a public available buffer API
>>>>>>>>>>>>> directly in DirectMemory so that
>>>>>>>>>>>>> project would be nearly needless :-)
>>>>>>>>>>>>> The only difference I think is the
>>>>>>>>>>>>> allocation strategy my implementation
>>>>>>>>>>>>> is using against the one DirectMemory
>>>>>>>>>>>>> has, but I'm pretty sure the allocation
>>>>>>>>>>>>> is extensible ;-)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Like I said, for both projects I'm the
>>>>>>>>>>>>> only dev so there would be no IP
>>>>>>>>>>>>> problem. So if it's ok to you to not
>>>>>>>>>>>>> include lightning directly in DM I
>>>>>>>>>>>>> would be glad to contribute to the
>>>>>>>>>>>>> Apache Foundation :-)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers Chris
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Am 29.09.2012 16:53, schrieb Raffaele
>>>>>>>>>>>>> P. Guidi:
>>>>>>>>>>>>>> Ok so it's up to noctarius - your
>>>>>>>>>>>>>> move! ;-) Regarding the new unsafe
>>>>>>>>>>>>>> storage: it's an opt-in feature that
>>>>>>>>>>>>>> can be set with the fluent API
>>>>>>>>>>> (and
>>>>>>>>>>>>>> soon through the conference file).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Ciao, R Il giorno 29/set/2012 16:45,
>>>>>>>>>>>>>> "Olivier Lamy" <ol...@apache.org> ha
>>>>>>>>>>>>> scritto:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2012/9/29 Raffaele P. Guidi
>>>>>>>>>>>>>>> <ra...@gmail.com>:
>>>>>>>>>>>>>>>>>> At least for the moment he
>>>>>>>>>>>>>>>>>> can provide a patch to be
>>>>>>>>>>>>>>>>>> integrated
>>>>>>>>>>> in DM
>>>>>>>>>>>>>>> :-)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Sure, but as lightning is not in
>>>>>>>>>>>>>>>> any public mvn repo should its
>>>>>>>>>>> code be
>>>>>>>>>>>>>>>> re-published in our svn? Or
>>>>>>>>>>>>>>>> what?
>>>>>>>>>>>>>>> @Apache we don't care about
>>>>>>>>>>>>>>> binaries, only sources are
>>>>>>>>>>>>>>> important ! (a bit theorical for
>>>>>>>>>>>>>>> sure but that's it :-) ). So if
>>>>>>>>>>>>>>> Noctarius was the only guy who
>>>>>>>>>>>>>>> participate in lightning. He can
>>>>>>>>>>>>>>> just provide a patch we could
>>>>>>>>>>>>>>> integrate as a new dm module (note:
>>>>>>>>>>>>>>> the patch must not contains any
>>>>>>>>>>>>>>> more copyright and all sources must
>>>>>>>>>>>>>>> have ASF licenses).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> "Copyright 2012 the original author
>>>>>>>>>>>>>>> or authors." must be removed. And
>>>>>>>>>>>>>>> BTW package must be changed :-)
>>>>>>>>>>>>>>> (com.github is not acceptable
>>>>>>>>>>> @asf
>>>>>>>>>>>>>>> :-) )(@Noctarius are you working
>>>>>>>>>>>>>>> for github ? :-) )
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> And having him as a committer will
>>>>>>>>>>>>>>> be only a matter of voting (we
>>>>>>>>>>> have
>>>>>>>>>>>>>>> a great chair who take cares of
>>>>>>>>>>>>>>> administrative stuff :P )
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> If some others have participated in
>>>>>>>>>>>>>>> the project, we must pass tru an
>>>>>>>>>>>>>>> ip clearance mechanism
>>>>>>>>>>>>>>> (http://incubator.apache.org/ip-clearance/index.html)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>
>>>>>>>>>>>>>>>
>>
>>>>>>>>>>>>>>>
and all contributors to lightning must provide a cla.
>>>>>>>>>>>>>>> (It it's the case I can help)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> perso I'd like we avoid hard
>>>>>>>>>>>>>>>>>> dependency on Unsafe as
>>>>>>>>>>>>>>>>>> maybe some
>>>>>>>>>>> use
>>>>>>>>>>>>>>>> other jdks :-)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Well, I believe Unsafe is
>>>>>>>>>>>>>>>> supported by Sun JDK, OpenJDK,
>>>>>>>>>>>>>>>> IBM JDK and JRockit - and I
>>>>>>>>>>>>>>>> believe that it is more than
>>>>>>>>>>>>>>>> enough. Also keep in
>>>>>>>>>>> mind
>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>> we already have an alternative
>>>>>>>>>>>>>>>> Unsafe based memory storage -
>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>> although
>>>>>>>>>>>>>>>> it's not thoroughly tested for
>>>>>>>>>>>>>>>> performance it dramaticaly
>>>>>>>>>>>>>>>> simplifies
>>>>>>>>>>>>>>> code,
>>>>>>>>>>>>>>>> I have great expectations about
>>>>>>>>>>>>>>>> it.
>>>>>>>>>>>>>>> Me too :-). Yup definitely more
>>>>>>>>>>>>>>> simple and faster ! But we must
>>>>>>>>>>>>>>> provide a switch off configuration
>>>>>>>>>>>>>>> mechanism if some
>>>>>>>>>>> users
>>>>>>>>>>>>>>> don't want to use that (because
>>>>>>>>>>>>>>> Unsafe is just "Unsafe" :-) ) And
>>>>>>>>>>>>>>> sorry I didn't have a look yet at
>>>>>>>>>>>>>>> your changes with using Unsafe.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Cheers, R
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Sat, Sep 29, 2012 at 4:03 PM,
>>>>>>>>>>>>>>>> Olivier Lamy <ol...@apache.org>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2012/9/29 Raffaele P. Guidi
>>>>>>>>>>>>>>>>> <ra...@gmail.com>:
>>>>>>>>>>>>>>>>>> What do you think about: 1)
>>>>>>>>>>>>>>>>>> implementing a lightning
>>>>>>>>>>>>>>>>>> serialization module 2)
>>>>>>>>>>>>>>>>>> creating a serializer that
>>>>>>>>>>>>>>>>>> directly works on a
>>>>>>>>>>>>>>>>>> directmemory
>>>>>>>>>>>>>>> provider
>>>>>>>>>>>>>>>>>> ByteBuffer or (maybe better)
>>>>>>>>>>>>>>>>>> Unsafe based Pointer?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Sounds good (perso I'd like we
>>>>>>>>>>>>>>>>> avoid hard dependency on Unsafe
>>>>>>>>>>>>>>>>> as maybe some use other jdks
>>>>>>>>>>>>>>>>> :-) )
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Now I see lightning is
>>>>>>>>>>>>>>>>>> apache licensed and this is
>>>>>>>>>>>>>>>>>> fine but I
>>>>>>>>>>> don't
>>>>>>>>>>>>>>> think
>>>>>>>>>>>>>>>>>> it is published in any
>>>>>>>>>>>>>>>>>> public maven repo, am I
>>>>>>>>>>>>>>>>>> right? We could
>>>>>>>>>>> find a
>>>>>>>>>>>>>>> way
>>>>>>>>>>>>>>>>>> to deal with this; options
>>>>>>>>>>>>>>>>>> vary from publishing
>>>>>>>>>>>>>>>>>> lightning to the
>>>>>>>>>>> free
>>>>>>>>>>>>>>>>>> sonatype repo, joining the
>>>>>>>>>>>>>>>>>> ASF (which is great anyhow!)
>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>> republishing
>>>>>>>>>>>>>>>>>> lightning code in
>>>>>>>>>>>>>>>>>> DirectMemory becoming a
>>>>>>>>>>>>>>>>>> commiter (which has to
>>>>>>>>>>>>>>> undergo
>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>> PMC vote).
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> At least for the moment he can
>>>>>>>>>>>>>>>>> provide a patch to be
>>>>>>>>>>>>>>>>> integrated in
>>>>>>>>>>> DM
>>>>>>>>>>>>>>> :-)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I'd like to hear your and
>>>>>>>>>>>>>>>>>> the team feelings on this.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks, Raffaele
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Sat, Sep 29, 2012 at 3:27
>>>>>>>>>>>>>>>>>> PM, Noctarius
>>>>>>>>>>>>>>>>>> <me...@noctarius.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hey Raffaele,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> that's quite similar to
>>>>>>>>>>>>>>>>>>> what I did at work. We're
>>>>>>>>>>>>>>>>>>> developing
>>>>>>>>>>> Flash
>>>>>>>>>>>>>>>>>>> online games and using a
>>>>>>>>>>>>>>>>>>> customized AMF
>>>>>>>>>>>>>>>>>>> serialization. To support
>>>>>>>>>>>>>>>>>>> async writing of the
>>>>>>>>>>>>>>>>>>> clients event results I
>>>>>>>>>>>>>>>>>>> added serialization
>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>> the components / entities
>>>>>>>>>>>>>>>>>>> to the players zone
>>>>>>>>>>>>>>>>>>> calculation and
>>>>>>>>>>> stored
>>>>>>>>>>>>>>>>>>> the pre-serialized
>>>>>>>>>>>>>>>>>>> bytestream directly to the
>>>>>>>>>>>>>>>>>>> off-heap (using
>>>>>>>>>>>>>>>>>>> direct-ring-cache
>>>>>>>>>>>>>>>>>>> implementation). When the
>>>>>>>>>>>>>>>>>>> client requests the
>>>>>>>>>>>>>>>>>>> results (using
>>>>>>>>>>>>>>>>>>> long-polling) I just write
>>>>>>>>>>>>>>>>>>> the pre-serialized
>>>>>>>>>>> data to
>>>>>>>>>>>>>>>>>>> the right position to
>>>>>>>>>>>>>>>>>>> deserialize it by standard
>>>>>>>>>>>>>>>>>>> ways on Flash
>>>>>>>>>>> side.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> So yeah an seriliaztion to
>>>>>>>>>>>>>>>>>>> off-heap algorithm would
>>>>>>>>>>>>>>>>>>> be fine. You
>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>> avoid using byte arrays
>>>>>>>>>>>>>>>>>>> and minimalize GC.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Cheers Chris
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Am 29.09.2012 15:02,
>>>>>>>>>>>>>>>>>>> schrieb Raffaele P. Guidi:
>>>>>>>>>>>>>>>>>>>> Nice to hear back from
>>>>>>>>>>>>>>>>>>>> you! Yes, I was thinking
>>>>>>>>>>>>>>>>>>>> about creating a
>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>> memory
>>>>>>>>>>>>>>>>>>>> storage implementation
>>>>>>>>>>>>>>>>>>>> using Unsafe (and I did
>>>>>>>>>>>>>>>>>>>> it, recently) and
>>>>>>>>>>>>>>>>> also, as
>>>>>>>>>>>>>>>>>>>> DirectMemory relies
>>>>>>>>>>>>>>>>>>>> heavily on serialization
>>>>>>>>>>>>>>>>>>>> (and supports many
>>>>>>>>>>> of
>>>>>>>>>>>>>>>>> them,
>>>>>>>>>>>>>>>>>>>> protostuff, protobuf,
>>>>>>>>>>>>>>>>>>>> msgpack and of course
>>>>>>>>>>>>>>>>>>>> standard
>>>>>>>>>>>>>>> serialization),
>>>>>>>>>>>>>>>>>>> about
>>>>>>>>>>>>>>>>>>>> creating a simple
>>>>>>>>>>>>>>>>>>>> embedded serializer
>>>>>>>>>>>>>>>>>>>> leveraging the same
>>>>>>>>>>>>>>> techniques
>>>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>>>>>> used (Unsafe and
>>>>>>>>>>>>>>>>>>>> bytecode generation). The
>>>>>>>>>>>>>>>>>>>> idea with embedding is
>>>>>>>>>>>>>>>>>>>> avoiding to serialize in
>>>>>>>>>>>>>>>>>>>> a byte array
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>>>>>>> moving the byte array to
>>>>>>>>>>>>>>>>>>>> off-heap memory (via
>>>>>>>>>>>>>>>>>>>> Unsafe or
>>>>>>>>>>>>>>> ByteBuffers),
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>> serializing directly into
>>>>>>>>>>>>>>>>>>>> a "managed" off-heap
>>>>>>>>>>>>>>>>>>>> buffer, thus
>>>>>>>>>>> further
>>>>>>>>>>>>>>>>>>>> optimizing heap
>>>>>>>>>>>>>>>>>>>> utilization (less GC).
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> What do you think about
>>>>>>>>>>>>>>>>>>>> it? Does it make any
>>>>>>>>>>>>>>>>>>>> sense to you?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Ciao, R
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Sat, Sep 29, 2012 at
>>>>>>>>>>>>>>>>>>>> 2:40 PM, Noctarius
>>>>>>>>>>>>>>>>>>>> <me...@noctarius.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hey guys,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Raffaele found out
>>>>>>>>>>>>>>>>>>>>> about a project of mine
>>>>>>>>>>>>>>>>>>>>> (Lightning) a few
>>>>>>>>>>> weeks
>>>>>>>>>>>>>>>>>>>>> ago. Lightning is a
>>>>>>>>>>>>>>>>>>>>> heavy Unsafe and
>>>>>>>>>>>>>>>>>>>>> Bytecode generation
>>>>>>>>>>>>>>>>>>>>> using Serializer
>>>>>>>>>>>>>>>>>>>>> implementation. He told
>>>>>>>>>>>>>>>>>>>>> me that he was
>>>>>>>>>>>>>>>>>>>>> interested in adding
>>>>>>>>>>>>>>>>>>>>> something similar to
>>>>>>>>>>>>>>>>>>>>> DirectMemory and I
>>>>>>>>>>>>>>>>>>>>> would be glad to
>>>>>>>>>>>>>>> help
>>>>>>>>>>>>>>>>>>>>> out in this.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Another project I
>>>>>>>>>>>>>>>>>>>>> started a few days ago,
>>>>>>>>>>>>>>>>>>>>> since it was needed
>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>> work is
>>>>>>>>>>>>>>>>>>>>> DirectRingCache. The
>>>>>>>>>>>>>>>>>>>>> name not really meets
>>>>>>>>>>>>>>>>>>>>> to actual
>>>>>>>>>>>>>>>>>>>>> implementation since
>>>>>>>>>>>>>>>>>>>>> it's not yet a ring
>>>>>>>>>>>>>>>>>>>>> buffer using cache. I
>>>>>>>>>>>>>>> used
>>>>>>>>>>>>>>>>>>>>> this for a
>>>>>>>>>>>>>>>>>>>>> pre-serialization
>>>>>>>>>>>>>>>>>>>>> simple bytestream cache
>>>>>>>>>>>>>>>>>>>>> with self-growing
>>>>>>>>>>>>>>>>>>>>> buffers. It could be
>>>>>>>>>>>>>>>>>>>>> nice to have
>>>>>>>>>>>>>>>>>>>>> DirectMemory
>>>>>>>>>>> having
>>>>>>>>>>>>>>>>>>>>> raw "buffers" to write
>>>>>>>>>>>>>>>>>>>>> to or to read from.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Here are the links
>>>>>>>>>>>>>>>>>>>>> from the projects:
>>>>>>>>>>>>>>>>>>>>> https://github.com/noctarius/Lightning
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>
>>>>>>>>>>>>>>>>>>>>>
https://github.com/noctarius/direct-ring-cache
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>> It would be nice to help out since I really
>>>>>>>>> like the idea of
>>>>>>>>>>>>>>>>>>>>> DirectMemory and since
>>>>>>>>>>>>>>>>>>>>> direct-ring-cache is
>>>>>>>>>>>>>>>>>>>>> some kind of
>>>>>>>>>>>>>>> reinventing
>>>>>>>>>>>>>>>>>>>>> the wheel.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Cheers Noctarius
>>>>>>>>>>>>>>>>>>>>> (Chris)
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> -- Olivier Lamy Talend:
>>>>>>>>>>>>>>>>> http://coders.talend.com
>>>>>>>>>>>>>>>>> http://twitter.com/olamy |
>>>>>>>>>>>>>>>>> http://linkedin.com/in/olamy
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -- Olivier Lamy Talend:
>>>>>>>>>>>>>>> http://coders.talend.com
>>>>>>>>>>>>>>> http://twitter.com/olamy |
>>>>>>>>>>>>>>> http://linkedin.com/in/olamy
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> -- Olivier Lamy Talend:
>>>>>>>>>>> http://coders.talend.com
>>>>>>>>>>> http://twitter.com/olamy |
>>>>>>>>>>> http://linkedin.com/in/olamy
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> -- Olivier Lamy Talend: http://coders.talend.com
>>>>>> http://twitter.com/olamy |
>>>>>> http://linkedin.com/in/olamy
>>>>>>
>>>>>
>>>>
>>>
>>
>>
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.12 (MingW32)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/
iQIcBAEBAgAGBQJQZ/ZZAAoJEH/g+YBfahrqbIUQALEwBddCj1OY7z2QYgUDze6u
t2RfhWdWpArExVG4vof46xwJ6RmF5T6muIjl3MoR8B6dG46QeBxv3bPq0RmUkcAF
U8USSMnC9w6Qr6DCjCwjqLb7dnUBgNGqLIMdIIePX7+UxnOOsQ9l4Ext0Vsz5xn3
2qkmy7nxiuBkox9OJlXlB8Nt//3LgHRi3iIuBpZ6S4GEwILIQ/UT0WuGKFx6+Sa7
bOwtSnViERwYNUiFth8O3SS4KmCsEHwWijV6vd+/jxVGhFqMSzxhj5++KzhCe+GZ
/WavR3rrvmZot9Mmpc4wDWj+6l6PXQrIXqxZDy9SrV7619r0Mh/SvbAKasP+/uMb
XLjQ/eZoAXRJ34G2l9l3q33lqwv7GK0+zcmgbgX6qun1eKUMuTR/08qfxu1UZO/k
MMAEZKfT50sNiEDHFKqyGLk8hgTh4nvoCPwxNZBWbezgIFg+8NdigdAlgIxWb+59
HwTaFtGmbZYkje0dx6WkoNXgLUsLLGOAq+rSbn4Pz3594hb7leJOIgGKUXA1/Eak
XoXFaXR/eg9ICuzrgN17Apz7GqN4HT3HxmGk6h+J6jkLTFOfyK5J6WQpP0hEXU5O
7dH6q0Zc/Af1J8eT2IGUrbbbwRWwXxIBUEQocon4VoDa219gmev/fxrRlL1xuB9f
7LS2c0gONRWWdl7M5n8+
=/dDf
-----END PGP SIGNATURE-----
Re: Additional Serializer and raw Buffer access
Posted by "Raffaele P. Guidi" <ra...@gmail.com>.
I kinda suspected that...
Il giorno 29/set/2012 20:47, "Noctarius" <me...@noctarius.com> ha scritto:
> Actually all dependencies should be AL2 or BSD licensed :-)
>
> Am 29.09.2012 20:42, schrieb Raffaele P. Guidi:
> >>> Hehe well that really sounds like a nice bunch of people.
> >
> > Indeed they are (I'm a newbie as well and try to do my best)
> >
> >>> If lightning will be a sub-part (sub-project) of DM, do I
> >>> need to write
> > an project purposal?
> >
> > Nope, not needed for a sub-project
> >
> >>> Do I need to make any changes to the pom.xml like adding a
> > special parent pom or anything like that?
> >
> > Not for the serializer - just have to take a look at project
> > dependencies - or, better, at their licenses - are they
> > compatible with the ASL 2.0? i.e. a GPL'd library is not a good
> > fit and should be replaced with an apache licensed (or BSD, or
> > MIT...) one if possible. For the integration module is a
> > separate story - you should start off copying one of the other
> > serializers and reusing the same pom and directory structure.
> >
> > Pleased to meet you, Chris :)
> >
> > Ciao, R
> >
> > On Sat, Sep 29, 2012 at 8:24 PM, Noctarius <me...@noctarius.com>
> > wrote:
> >
> >> Hehe well that really sounds like a nice bunch of people.
> >>
> >> Ok to be true I couldn't wait until tomorrow and started
> >> already reading the links. From what I was reading: If
> >> lightning will be a sub-part (sub-project) of DM, do I need
> >> to write an project purposal?
> >>
> >> Do I need to make any changes to the pom.xml like adding a
> >> special parent pom or anything like that?
> >>
> >> In general: there are a lot things to know :-)
> >>
> >> Am 29.09.2012 19:59, schrieb Raffaele P. Guidi:
> >>> Negative part of ASF membership? You get together with a
> >>> lot of geeky, talented people with a fixation for software
> >>> and open source. Oh wait but this is actually nice! :-D Il
> >>> giorno 29/set/2012 19:05, "Olivier Lamy" <ol...@apache.org>
> >>> ha
> >> scritto:
> >>>
> >>>> 2012/9/29 Noctarius <me...@noctarius.com>:
> >>>>> Thanks Olivier for carify, I'll take a look in it
> >>>>> tomorrow but there's just one question left (for now
> >>>>> ;)): What is that vote for becoming a committer? What
> >>>>> if the vote will be negative?
> >>>> The vote is on private list (pmc list for privacy reasons
> >>>> and possible negative stuff being on public lists)
> >>>>> Until now I just used Apache stuff, was never
> >>>>> interested in being part of it so I guess it can be
> >>>>> negative for any reason, can't it?
> >>>> I don't see why it could be negative but suspens ....
> >>>> :-)
> >>>>>
> >>>>> Am 29.09.2012 18:56, schrieb Olivier Lamy:
> >>>>>> 2012/9/29 Noctarius <me...@noctarius.com>:
> >>>>>>> Nope my real name is Christoph Engelbert, but
> >>>>>>> Noctarius is the all time nick :)
> >>>>>>>
> >>>>>>> Renaming the package should be no problem, is it
> >>>>>>> "org.apache.directmemory.lightning" or what would
> >>>>>>> it be?
> >>>>>> fine for me
> >>>>>>> Then there needs to be a change in the license
> >>>>>>> header as Olivier mentioned, that means just remove
> >>>>>>> the first sentence or is there anything more to do
> >>>>>>> (maybe it's easiest thing to just copy the header
> >>>>>>> from DM file ;))?
> >>>>>> yup use same header as DM
> >>>>>>>
> >>>>>>> The CLA is just a form to clarify that the source
> >>>>>>> can be contributed to the Apache Foundation?
> >>>>>> yup correct.
> >>>>>>>
> >>>>>>> The final step will be attaching the patch in form
> >>>>>>> of a huge diff file?
> >>>>>> yes
> >>>>>>>
> >>>>>>> And what is the way to apply for a membership?
> >>>>>>> Never thought about how to do that.
> >>>>>> Read here
> >>>>>> http://apache.org/foundation/how-it-works.html and
> >>>>>> here http://apache.org/foundation/getinvolved.html .
> >>>>>> And feel free to ask any questions :-)
> >>>>>>>
> >>>>>>> Chris
> >>>>>>>
> >>>>>>> Am 29.09.2012 18:23, schrieb Raffaele P. Guidi:
> >>>>>>>> OK, deal, at least for me ;-) I propose you
> >>>>>>>> rename the packages, produce a patch for this and
> >>>>>>>> the new serializer module (should be simple
> >>>>>>>> enough starting from an existing one) and, in the
> >>>>>>>> meanwhile, apply for ASF membership. Is IP
> >>>>>>>> clearance needed? I guess yes. After this we will
> >>>>>>>> come up with a formal vote regarding Noctarius
> >>>>>>>> (is this your real name?!) allowance in the
> >>>>>>>> project team.
> >>>>>>>>
> >>>>>>>> Good times are gonna come :-)
> >>>>>>>>
> >>>>>>>> Thanks, R Il giorno 29/set/2012 17:58, "Olivier
> >>>>>>>> Lamy" <ol...@apache.org> ha scritto:
> >>>>>>>>
> >>>>>>>>> 2012/9/29 Raffaele P. Guidi
> >>>>>>>>> <ra...@gmail.com>:
> >>>>>>>>>> Well we already have a NIO ready interface
> >>>>>>>>>> allowing direct access to DMs managed
> >>>>>>>>>> bytebuffers but I think this is just half way
> >>>>>>>>>> to what could be achieved optimally blending
> >>>>>>>>>> serialization and memory allocation
> >>>>>>>>>> together.
> >>>>>>>>>>
> >>>>>>>>>> Lightning as a module is of course a good
> >>>>>>>>>> idea and it could easily evolve as a
> >>>>>>>>>> subproject (for the more experienced asf
> >>>>>>>>>> members: is it a feasible way?).
> >>>>>>>>> Nothing prevent to have
> >>>>>>>>> http://svn.apache.org/repos/asf/directmemory/lightning/trunk
> >>>>>>>>>
> >>>>>>>>>
> >>>>>
> >>>>>>>>>
> with a package like: o.a.d.lightning That will be a
> >>>>>>>>> subproject.
> >>>>>>>>>>
> >>>>>>>>>> Ciao, R Il giorno 29/set/2012 17:44,
> >>>>>>>>>> "Noctarius" <me...@noctarius.com> ha scritto:
> >>>>>>>>>>
> >>>>>>>>>>> Hey guys,
> >>>>>>>>>>>
> >>>>>>>>>>> ok there's no lightning binary available
> >>>>>>>>>>> since lightning wasn't ready yet for
> >>>>>>>>>>> releasing.
> >>>>>>>>>>>
> >>>>>>>>>>> For being the only developer it would be no
> >>>>>>>>>>> problem to contribute the sourcebase for
> >>>>>>>>>>> DirectMemory but I'm not sure yet if it
> >>>>>>>>>>> wouldn't be better to seperate it to be
> >>>>>>>>>>> available without using DirectMemory
> >>>>>>>>>>> itself. I started it as a serializer for
> >>>>>>>>>>> cluster synchronization, but it would be
> >>>>>>>>>>> cool to contribute lightning as a
> >>>>>>>>>>> subproject to DirectMemory :-)
> >>>>>>>>>>>
> >>>>>>>>>>> About the second project I would love to
> >>>>>>>>>>> see a public available buffer API directly
> >>>>>>>>>>> in DirectMemory so that project would be
> >>>>>>>>>>> nearly needless :-) The only difference I
> >>>>>>>>>>> think is the allocation strategy my
> >>>>>>>>>>> implementation is using against the one
> >>>>>>>>>>> DirectMemory has, but I'm pretty sure the
> >>>>>>>>>>> allocation is extensible ;-)
> >>>>>>>>>>>
> >>>>>>>>>>> Like I said, for both projects I'm the only
> >>>>>>>>>>> dev so there would be no IP problem. So if
> >>>>>>>>>>> it's ok to you to not include lightning
> >>>>>>>>>>> directly in DM I would be glad to
> >>>>>>>>>>> contribute to the Apache Foundation :-)
> >>>>>>>>>>>
> >>>>>>>>>>> Cheers Chris
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Am 29.09.2012 16:53, schrieb Raffaele P.
> >>>>>>>>>>> Guidi:
> >>>>>>>>>>>> Ok so it's up to noctarius - your move!
> >>>>>>>>>>>> ;-) Regarding the new unsafe storage:
> >>>>>>>>>>>> it's an opt-in feature that can be set
> >>>>>>>>>>>> with the fluent API
> >>>>>>>>> (and
> >>>>>>>>>>>> soon through the conference file).
> >>>>>>>>>>>>
> >>>>>>>>>>>> Ciao, R Il giorno 29/set/2012 16:45,
> >>>>>>>>>>>> "Olivier Lamy" <ol...@apache.org> ha
> >>>>>>>>>>> scritto:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> 2012/9/29 Raffaele P. Guidi
> >>>>>>>>>>>>> <ra...@gmail.com>:
> >>>>>>>>>>>>>>>> At least for the moment he can
> >>>>>>>>>>>>>>>> provide a patch to be integrated
> >>>>>>>>> in DM
> >>>>>>>>>>>>> :-)
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Sure, but as lightning is not in any
> >>>>>>>>>>>>>> public mvn repo should its
> >>>>>>>>> code be
> >>>>>>>>>>>>>> re-published in our svn? Or what?
> >>>>>>>>>>>>> @Apache we don't care about binaries,
> >>>>>>>>>>>>> only sources are important ! (a bit
> >>>>>>>>>>>>> theorical for sure but that's it :-) ).
> >>>>>>>>>>>>> So if Noctarius was the only guy who
> >>>>>>>>>>>>> participate in lightning. He can just
> >>>>>>>>>>>>> provide a patch we could integrate as a
> >>>>>>>>>>>>> new dm module (note: the patch must not
> >>>>>>>>>>>>> contains any more copyright and all
> >>>>>>>>>>>>> sources must have ASF licenses).
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> "Copyright 2012 the original author or
> >>>>>>>>>>>>> authors." must be removed. And BTW
> >>>>>>>>>>>>> package must be changed :-) (com.github
> >>>>>>>>>>>>> is not acceptable
> >>>>>>>>> @asf
> >>>>>>>>>>>>> :-) )(@Noctarius are you working for
> >>>>>>>>>>>>> github ? :-) )
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> And having him as a committer will be
> >>>>>>>>>>>>> only a matter of voting (we
> >>>>>>>>> have
> >>>>>>>>>>>>> a great chair who take cares of
> >>>>>>>>>>>>> administrative stuff :P )
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> If some others have participated in the
> >>>>>>>>>>>>> project, we must pass tru an ip
> >>>>>>>>>>>>> clearance mechanism
> >>>>>>>>>>>>> (http://incubator.apache.org/ip-clearance/index.html)
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>
> >>>>>>>>>>>>>
> and all contributors to lightning must provide a cla.
> >>>>>>>>>>>>> (It it's the case I can help)
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> perso I'd like we avoid hard
> >>>>>>>>>>>>>>>> dependency on Unsafe as maybe
> >>>>>>>>>>>>>>>> some
> >>>>>>>>> use
> >>>>>>>>>>>>>> other jdks :-)
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Well, I believe Unsafe is supported
> >>>>>>>>>>>>>> by Sun JDK, OpenJDK, IBM JDK and
> >>>>>>>>>>>>>> JRockit - and I believe that it is
> >>>>>>>>>>>>>> more than enough. Also keep in
> >>>>>>>>> mind
> >>>>>>>>>>>>> that
> >>>>>>>>>>>>>> we already have an alternative Unsafe
> >>>>>>>>>>>>>> based memory storage - and
> >>>>>>>>>>> although
> >>>>>>>>>>>>>> it's not thoroughly tested for
> >>>>>>>>>>>>>> performance it dramaticaly
> >>>>>>>>>>>>>> simplifies
> >>>>>>>>>>>>> code,
> >>>>>>>>>>>>>> I have great expectations about it.
> >>>>>>>>>>>>> Me too :-). Yup definitely more simple
> >>>>>>>>>>>>> and faster ! But we must provide a
> >>>>>>>>>>>>> switch off configuration mechanism if
> >>>>>>>>>>>>> some
> >>>>>>>>> users
> >>>>>>>>>>>>> don't want to use that (because Unsafe
> >>>>>>>>>>>>> is just "Unsafe" :-) ) And sorry I
> >>>>>>>>>>>>> didn't have a look yet at your changes
> >>>>>>>>>>>>> with using Unsafe.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Cheers, R
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Sat, Sep 29, 2012 at 4:03 PM,
> >>>>>>>>>>>>>> Olivier Lamy <ol...@apache.org>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> 2012/9/29 Raffaele P. Guidi
> >>>>>>>>>>>>>>> <ra...@gmail.com>:
> >>>>>>>>>>>>>>>> What do you think about: 1)
> >>>>>>>>>>>>>>>> implementing a lightning
> >>>>>>>>>>>>>>>> serialization module 2) creating
> >>>>>>>>>>>>>>>> a serializer that directly works
> >>>>>>>>>>>>>>>> on a directmemory
> >>>>>>>>>>>>> provider
> >>>>>>>>>>>>>>>> ByteBuffer or (maybe better)
> >>>>>>>>>>>>>>>> Unsafe based Pointer?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Sounds good (perso I'd like we
> >>>>>>>>>>>>>>> avoid hard dependency on Unsafe as
> >>>>>>>>>>>>>>> maybe some use other jdks :-) )
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Now I see lightning is apache
> >>>>>>>>>>>>>>>> licensed and this is fine but I
> >>>>>>>>> don't
> >>>>>>>>>>>>> think
> >>>>>>>>>>>>>>>> it is published in any public
> >>>>>>>>>>>>>>>> maven repo, am I right? We could
> >>>>>>>>> find a
> >>>>>>>>>>>>> way
> >>>>>>>>>>>>>>>> to deal with this; options vary
> >>>>>>>>>>>>>>>> from publishing lightning to the
> >>>>>>>>> free
> >>>>>>>>>>>>>>>> sonatype repo, joining the ASF
> >>>>>>>>>>>>>>>> (which is great anyhow!) and
> >>>>>>>>>>>>> republishing
> >>>>>>>>>>>>>>>> lightning code in DirectMemory
> >>>>>>>>>>>>>>>> becoming a commiter (which has
> >>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>> undergo
> >>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>> PMC vote).
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> At least for the moment he can
> >>>>>>>>>>>>>>> provide a patch to be integrated
> >>>>>>>>>>>>>>> in
> >>>>>>>>> DM
> >>>>>>>>>>>>> :-)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I'd like to hear your and the
> >>>>>>>>>>>>>>>> team feelings on this.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks, Raffaele
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Sat, Sep 29, 2012 at 3:27 PM,
> >>>>>>>>>>>>>>>> Noctarius <me...@noctarius.com>
> >>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hey Raffaele,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> that's quite similar to what I
> >>>>>>>>>>>>>>>>> did at work. We're developing
> >>>>>>>>> Flash
> >>>>>>>>>>>>>>>>> online games and using a
> >>>>>>>>>>>>>>>>> customized AMF serialization.
> >>>>>>>>>>>>>>>>> To support async writing of the
> >>>>>>>>>>>>>>>>> clients event results I added
> >>>>>>>>>>>>>>>>> serialization
> >>>>>>>>> of
> >>>>>>>>>>>>>>>>> the components / entities to
> >>>>>>>>>>>>>>>>> the players zone calculation
> >>>>>>>>>>>>>>>>> and
> >>>>>>>>> stored
> >>>>>>>>>>>>>>>>> the pre-serialized bytestream
> >>>>>>>>>>>>>>>>> directly to the off-heap (using
> >>>>>>>>>>>>>>>>> direct-ring-cache
> >>>>>>>>>>>>>>>>> implementation). When the
> >>>>>>>>>>>>>>>>> client requests the results
> >>>>>>>>>>>>>>>>> (using long-polling) I just
> >>>>>>>>>>>>>>>>> write the pre-serialized
> >>>>>>>>> data to
> >>>>>>>>>>>>>>>>> the right position to
> >>>>>>>>>>>>>>>>> deserialize it by standard ways
> >>>>>>>>>>>>>>>>> on Flash
> >>>>>>>>> side.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> So yeah an seriliaztion to
> >>>>>>>>>>>>>>>>> off-heap algorithm would be
> >>>>>>>>>>>>>>>>> fine. You
> >>>>>>>>> can
> >>>>>>>>>>>>>>>>> avoid using byte arrays and
> >>>>>>>>>>>>>>>>> minimalize GC.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Cheers Chris
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Am 29.09.2012 15:02, schrieb
> >>>>>>>>>>>>>>>>> Raffaele P. Guidi:
> >>>>>>>>>>>>>>>>>> Nice to hear back from you!
> >>>>>>>>>>>>>>>>>> Yes, I was thinking about
> >>>>>>>>>>>>>>>>>> creating a
> >>>>>>>>>>>>> new
> >>>>>>>>>>>>>>>>> memory
> >>>>>>>>>>>>>>>>>> storage implementation using
> >>>>>>>>>>>>>>>>>> Unsafe (and I did it,
> >>>>>>>>>>>>>>>>>> recently) and
> >>>>>>>>>>>>>>> also, as
> >>>>>>>>>>>>>>>>>> DirectMemory relies heavily
> >>>>>>>>>>>>>>>>>> on serialization (and
> >>>>>>>>>>>>>>>>>> supports many
> >>>>>>>>> of
> >>>>>>>>>>>>>>> them,
> >>>>>>>>>>>>>>>>>> protostuff, protobuf, msgpack
> >>>>>>>>>>>>>>>>>> and of course standard
> >>>>>>>>>>>>> serialization),
> >>>>>>>>>>>>>>>>> about
> >>>>>>>>>>>>>>>>>> creating a simple embedded
> >>>>>>>>>>>>>>>>>> serializer leveraging the
> >>>>>>>>>>>>>>>>>> same
> >>>>>>>>>>>>> techniques
> >>>>>>>>>>>>>>> you
> >>>>>>>>>>>>>>>>>> used (Unsafe and bytecode
> >>>>>>>>>>>>>>>>>> generation). The idea with
> >>>>>>>>>>>>>>>>>> embedding is avoiding to
> >>>>>>>>>>>>>>>>>> serialize in a byte array
> >>>>>>>>>>>>> and
> >>>>>>>>>>>>>>> then
> >>>>>>>>>>>>>>>>>> moving the byte array to
> >>>>>>>>>>>>>>>>>> off-heap memory (via Unsafe
> >>>>>>>>>>>>>>>>>> or
> >>>>>>>>>>>>> ByteBuffers),
> >>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>> serializing directly into a
> >>>>>>>>>>>>>>>>>> "managed" off-heap buffer,
> >>>>>>>>>>>>>>>>>> thus
> >>>>>>>>> further
> >>>>>>>>>>>>>>>>>> optimizing heap utilization
> >>>>>>>>>>>>>>>>>> (less GC).
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> What do you think about it?
> >>>>>>>>>>>>>>>>>> Does it make any sense to
> >>>>>>>>>>>>>>>>>> you?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Ciao, R
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Sat, Sep 29, 2012 at 2:40
> >>>>>>>>>>>>>>>>>> PM, Noctarius
> >>>>>>>>>>>>>>>>>> <me...@noctarius.com>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Hey guys,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Raffaele found out about a
> >>>>>>>>>>>>>>>>>>> project of mine (Lightning)
> >>>>>>>>>>>>>>>>>>> a few
> >>>>>>>>> weeks
> >>>>>>>>>>>>>>>>>>> ago. Lightning is a heavy
> >>>>>>>>>>>>>>>>>>> Unsafe and Bytecode
> >>>>>>>>>>>>>>>>>>> generation using
> >>>>>>>>>>>>>>>>>>> Serializer implementation.
> >>>>>>>>>>>>>>>>>>> He told me that he was
> >>>>>>>>>>>>>>>>>>> interested in adding
> >>>>>>>>>>>>>>>>>>> something similar to
> >>>>>>>>>>>>>>>>>>> DirectMemory and I would be
> >>>>>>>>>>>>>>>>>>> glad to
> >>>>>>>>>>>>> help
> >>>>>>>>>>>>>>>>>>> out in this.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Another project I started a
> >>>>>>>>>>>>>>>>>>> few days ago, since it was
> >>>>>>>>>>>>>>>>>>> needed
> >>>>>>>>> for
> >>>>>>>>>>>>>>>>>>> work is DirectRingCache.
> >>>>>>>>>>>>>>>>>>> The name not really meets
> >>>>>>>>>>>>>>>>>>> to actual implementation
> >>>>>>>>>>>>>>>>>>> since it's not yet a ring
> >>>>>>>>>>>>>>>>>>> buffer using cache. I
> >>>>>>>>>>>>> used
> >>>>>>>>>>>>>>>>>>> this for a
> >>>>>>>>>>>>>>>>>>> pre-serialization simple
> >>>>>>>>>>>>>>>>>>> bytestream cache with
> >>>>>>>>>>>>>>>>>>> self-growing buffers. It
> >>>>>>>>>>>>>>>>>>> could be nice to have
> >>>>>>>>>>>>>>>>>>> DirectMemory
> >>>>>>>>> having
> >>>>>>>>>>>>>>>>>>> raw "buffers" to write to
> >>>>>>>>>>>>>>>>>>> or to read from.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Here are the links from
> >>>>>>>>>>>>>>>>>>> the projects:
> >>>>>>>>>>>>>>>>>>> https://github.com/noctarius/Lightning
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>
> >>>>>>>>>>>>>>>>>>>
> https://github.com/noctarius/direct-ring-cache
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>> It would be nice to help out since I really like
> >>>>>>> the idea of
> >>>>>>>>>>>>>>>>>>> DirectMemory and since
> >>>>>>>>>>>>>>>>>>> direct-ring-cache is some
> >>>>>>>>>>>>>>>>>>> kind of
> >>>>>>>>>>>>> reinventing
> >>>>>>>>>>>>>>>>>>> the wheel.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Cheers Noctarius (Chris)
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> -- Olivier Lamy Talend:
> >>>>>>>>>>>>>>> http://coders.talend.com
> >>>>>>>>>>>>>>> http://twitter.com/olamy |
> >>>>>>>>>>>>>>> http://linkedin.com/in/olamy
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> -- Olivier Lamy Talend:
> >>>>>>>>>>>>> http://coders.talend.com
> >>>>>>>>>>>>> http://twitter.com/olamy |
> >>>>>>>>>>>>> http://linkedin.com/in/olamy
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> -- Olivier Lamy Talend:
> >>>>>>>>> http://coders.talend.com
> >>>>>>>>> http://twitter.com/olamy |
> >>>>>>>>> http://linkedin.com/in/olamy
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>
> >>>>
> >>>>
> >>>> -- Olivier Lamy Talend: http://coders.talend.com
> >>>> http://twitter.com/olamy | http://linkedin.com/in/olamy
> >>>>
> >>>
> >>
> >
>
>
Re: Additional Serializer and raw Buffer access
Posted by Noctarius <me...@noctarius.com>.
Actually all dependencies should be AL2 or BSD licensed :-)
Am 29.09.2012 20:42, schrieb Raffaele P. Guidi:
>>> Hehe well that really sounds like a nice bunch of people.
>
> Indeed they are (I'm a newbie as well and try to do my best)
>
>>> If lightning will be a sub-part (sub-project) of DM, do I
>>> need to write
> an project purposal?
>
> Nope, not needed for a sub-project
>
>>> Do I need to make any changes to the pom.xml like adding a
> special parent pom or anything like that?
>
> Not for the serializer - just have to take a look at project
> dependencies - or, better, at their licenses - are they
> compatible with the ASL 2.0? i.e. a GPL'd library is not a good
> fit and should be replaced with an apache licensed (or BSD, or
> MIT...) one if possible. For the integration module is a
> separate story - you should start off copying one of the other
> serializers and reusing the same pom and directory structure.
>
> Pleased to meet you, Chris :)
>
> Ciao, R
>
> On Sat, Sep 29, 2012 at 8:24 PM, Noctarius <me...@noctarius.com>
> wrote:
>
>> Hehe well that really sounds like a nice bunch of people.
>>
>> Ok to be true I couldn't wait until tomorrow and started
>> already reading the links. From what I was reading: If
>> lightning will be a sub-part (sub-project) of DM, do I need
>> to write an project purposal?
>>
>> Do I need to make any changes to the pom.xml like adding a
>> special parent pom or anything like that?
>>
>> In general: there are a lot things to know :-)
>>
>> Am 29.09.2012 19:59, schrieb Raffaele P. Guidi:
>>> Negative part of ASF membership? You get together with a
>>> lot of geeky, talented people with a fixation for software
>>> and open source. Oh wait but this is actually nice! :-D Il
>>> giorno 29/set/2012 19:05, "Olivier Lamy" <ol...@apache.org>
>>> ha
>> scritto:
>>>
>>>> 2012/9/29 Noctarius <me...@noctarius.com>:
>>>>> Thanks Olivier for carify, I'll take a look in it
>>>>> tomorrow but there's just one question left (for now
>>>>> ;)): What is that vote for becoming a committer? What
>>>>> if the vote will be negative?
>>>> The vote is on private list (pmc list for privacy reasons
>>>> and possible negative stuff being on public lists)
>>>>> Until now I just used Apache stuff, was never
>>>>> interested in being part of it so I guess it can be
>>>>> negative for any reason, can't it?
>>>> I don't see why it could be negative but suspens ....
>>>> :-)
>>>>>
>>>>> Am 29.09.2012 18:56, schrieb Olivier Lamy:
>>>>>> 2012/9/29 Noctarius <me...@noctarius.com>:
>>>>>>> Nope my real name is Christoph Engelbert, but
>>>>>>> Noctarius is the all time nick :)
>>>>>>>
>>>>>>> Renaming the package should be no problem, is it
>>>>>>> "org.apache.directmemory.lightning" or what would
>>>>>>> it be?
>>>>>> fine for me
>>>>>>> Then there needs to be a change in the license
>>>>>>> header as Olivier mentioned, that means just remove
>>>>>>> the first sentence or is there anything more to do
>>>>>>> (maybe it's easiest thing to just copy the header
>>>>>>> from DM file ;))?
>>>>>> yup use same header as DM
>>>>>>>
>>>>>>> The CLA is just a form to clarify that the source
>>>>>>> can be contributed to the Apache Foundation?
>>>>>> yup correct.
>>>>>>>
>>>>>>> The final step will be attaching the patch in form
>>>>>>> of a huge diff file?
>>>>>> yes
>>>>>>>
>>>>>>> And what is the way to apply for a membership?
>>>>>>> Never thought about how to do that.
>>>>>> Read here
>>>>>> http://apache.org/foundation/how-it-works.html and
>>>>>> here http://apache.org/foundation/getinvolved.html .
>>>>>> And feel free to ask any questions :-)
>>>>>>>
>>>>>>> Chris
>>>>>>>
>>>>>>> Am 29.09.2012 18:23, schrieb Raffaele P. Guidi:
>>>>>>>> OK, deal, at least for me ;-) I propose you
>>>>>>>> rename the packages, produce a patch for this and
>>>>>>>> the new serializer module (should be simple
>>>>>>>> enough starting from an existing one) and, in the
>>>>>>>> meanwhile, apply for ASF membership. Is IP
>>>>>>>> clearance needed? I guess yes. After this we will
>>>>>>>> come up with a formal vote regarding Noctarius
>>>>>>>> (is this your real name?!) allowance in the
>>>>>>>> project team.
>>>>>>>>
>>>>>>>> Good times are gonna come :-)
>>>>>>>>
>>>>>>>> Thanks, R Il giorno 29/set/2012 17:58, "Olivier
>>>>>>>> Lamy" <ol...@apache.org> ha scritto:
>>>>>>>>
>>>>>>>>> 2012/9/29 Raffaele P. Guidi
>>>>>>>>> <ra...@gmail.com>:
>>>>>>>>>> Well we already have a NIO ready interface
>>>>>>>>>> allowing direct access to DMs managed
>>>>>>>>>> bytebuffers but I think this is just half way
>>>>>>>>>> to what could be achieved optimally blending
>>>>>>>>>> serialization and memory allocation
>>>>>>>>>> together.
>>>>>>>>>>
>>>>>>>>>> Lightning as a module is of course a good
>>>>>>>>>> idea and it could easily evolve as a
>>>>>>>>>> subproject (for the more experienced asf
>>>>>>>>>> members: is it a feasible way?).
>>>>>>>>> Nothing prevent to have
>>>>>>>>> http://svn.apache.org/repos/asf/directmemory/lightning/trunk
>>>>>>>>>
>>>>>>>>>
>>>>>
>>>>>>>>>
with a package like: o.a.d.lightning That will be a
>>>>>>>>> subproject.
>>>>>>>>>>
>>>>>>>>>> Ciao, R Il giorno 29/set/2012 17:44,
>>>>>>>>>> "Noctarius" <me...@noctarius.com> ha scritto:
>>>>>>>>>>
>>>>>>>>>>> Hey guys,
>>>>>>>>>>>
>>>>>>>>>>> ok there's no lightning binary available
>>>>>>>>>>> since lightning wasn't ready yet for
>>>>>>>>>>> releasing.
>>>>>>>>>>>
>>>>>>>>>>> For being the only developer it would be no
>>>>>>>>>>> problem to contribute the sourcebase for
>>>>>>>>>>> DirectMemory but I'm not sure yet if it
>>>>>>>>>>> wouldn't be better to seperate it to be
>>>>>>>>>>> available without using DirectMemory
>>>>>>>>>>> itself. I started it as a serializer for
>>>>>>>>>>> cluster synchronization, but it would be
>>>>>>>>>>> cool to contribute lightning as a
>>>>>>>>>>> subproject to DirectMemory :-)
>>>>>>>>>>>
>>>>>>>>>>> About the second project I would love to
>>>>>>>>>>> see a public available buffer API directly
>>>>>>>>>>> in DirectMemory so that project would be
>>>>>>>>>>> nearly needless :-) The only difference I
>>>>>>>>>>> think is the allocation strategy my
>>>>>>>>>>> implementation is using against the one
>>>>>>>>>>> DirectMemory has, but I'm pretty sure the
>>>>>>>>>>> allocation is extensible ;-)
>>>>>>>>>>>
>>>>>>>>>>> Like I said, for both projects I'm the only
>>>>>>>>>>> dev so there would be no IP problem. So if
>>>>>>>>>>> it's ok to you to not include lightning
>>>>>>>>>>> directly in DM I would be glad to
>>>>>>>>>>> contribute to the Apache Foundation :-)
>>>>>>>>>>>
>>>>>>>>>>> Cheers Chris
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Am 29.09.2012 16:53, schrieb Raffaele P.
>>>>>>>>>>> Guidi:
>>>>>>>>>>>> Ok so it's up to noctarius - your move!
>>>>>>>>>>>> ;-) Regarding the new unsafe storage:
>>>>>>>>>>>> it's an opt-in feature that can be set
>>>>>>>>>>>> with the fluent API
>>>>>>>>> (and
>>>>>>>>>>>> soon through the conference file).
>>>>>>>>>>>>
>>>>>>>>>>>> Ciao, R Il giorno 29/set/2012 16:45,
>>>>>>>>>>>> "Olivier Lamy" <ol...@apache.org> ha
>>>>>>>>>>> scritto:
>>>>>>>>>>>>
>>>>>>>>>>>>> 2012/9/29 Raffaele P. Guidi
>>>>>>>>>>>>> <ra...@gmail.com>:
>>>>>>>>>>>>>>>> At least for the moment he can
>>>>>>>>>>>>>>>> provide a patch to be integrated
>>>>>>>>> in DM
>>>>>>>>>>>>> :-)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Sure, but as lightning is not in any
>>>>>>>>>>>>>> public mvn repo should its
>>>>>>>>> code be
>>>>>>>>>>>>>> re-published in our svn? Or what?
>>>>>>>>>>>>> @Apache we don't care about binaries,
>>>>>>>>>>>>> only sources are important ! (a bit
>>>>>>>>>>>>> theorical for sure but that's it :-) ).
>>>>>>>>>>>>> So if Noctarius was the only guy who
>>>>>>>>>>>>> participate in lightning. He can just
>>>>>>>>>>>>> provide a patch we could integrate as a
>>>>>>>>>>>>> new dm module (note: the patch must not
>>>>>>>>>>>>> contains any more copyright and all
>>>>>>>>>>>>> sources must have ASF licenses).
>>>>>>>>>>>>>
>>>>>>>>>>>>> "Copyright 2012 the original author or
>>>>>>>>>>>>> authors." must be removed. And BTW
>>>>>>>>>>>>> package must be changed :-) (com.github
>>>>>>>>>>>>> is not acceptable
>>>>>>>>> @asf
>>>>>>>>>>>>> :-) )(@Noctarius are you working for
>>>>>>>>>>>>> github ? :-) )
>>>>>>>>>>>>>
>>>>>>>>>>>>> And having him as a committer will be
>>>>>>>>>>>>> only a matter of voting (we
>>>>>>>>> have
>>>>>>>>>>>>> a great chair who take cares of
>>>>>>>>>>>>> administrative stuff :P )
>>>>>>>>>>>>>
>>>>>>>>>>>>> If some others have participated in the
>>>>>>>>>>>>> project, we must pass tru an ip
>>>>>>>>>>>>> clearance mechanism
>>>>>>>>>>>>> (http://incubator.apache.org/ip-clearance/index.html)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>
>>>>>>>>>>>>>
and all contributors to lightning must provide a cla.
>>>>>>>>>>>>> (It it's the case I can help)
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> perso I'd like we avoid hard
>>>>>>>>>>>>>>>> dependency on Unsafe as maybe
>>>>>>>>>>>>>>>> some
>>>>>>>>> use
>>>>>>>>>>>>>> other jdks :-)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Well, I believe Unsafe is supported
>>>>>>>>>>>>>> by Sun JDK, OpenJDK, IBM JDK and
>>>>>>>>>>>>>> JRockit - and I believe that it is
>>>>>>>>>>>>>> more than enough. Also keep in
>>>>>>>>> mind
>>>>>>>>>>>>> that
>>>>>>>>>>>>>> we already have an alternative Unsafe
>>>>>>>>>>>>>> based memory storage - and
>>>>>>>>>>> although
>>>>>>>>>>>>>> it's not thoroughly tested for
>>>>>>>>>>>>>> performance it dramaticaly
>>>>>>>>>>>>>> simplifies
>>>>>>>>>>>>> code,
>>>>>>>>>>>>>> I have great expectations about it.
>>>>>>>>>>>>> Me too :-). Yup definitely more simple
>>>>>>>>>>>>> and faster ! But we must provide a
>>>>>>>>>>>>> switch off configuration mechanism if
>>>>>>>>>>>>> some
>>>>>>>>> users
>>>>>>>>>>>>> don't want to use that (because Unsafe
>>>>>>>>>>>>> is just "Unsafe" :-) ) And sorry I
>>>>>>>>>>>>> didn't have a look yet at your changes
>>>>>>>>>>>>> with using Unsafe.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Cheers, R
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Sat, Sep 29, 2012 at 4:03 PM,
>>>>>>>>>>>>>> Olivier Lamy <ol...@apache.org>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2012/9/29 Raffaele P. Guidi
>>>>>>>>>>>>>>> <ra...@gmail.com>:
>>>>>>>>>>>>>>>> What do you think about: 1)
>>>>>>>>>>>>>>>> implementing a lightning
>>>>>>>>>>>>>>>> serialization module 2) creating
>>>>>>>>>>>>>>>> a serializer that directly works
>>>>>>>>>>>>>>>> on a directmemory
>>>>>>>>>>>>> provider
>>>>>>>>>>>>>>>> ByteBuffer or (maybe better)
>>>>>>>>>>>>>>>> Unsafe based Pointer?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Sounds good (perso I'd like we
>>>>>>>>>>>>>>> avoid hard dependency on Unsafe as
>>>>>>>>>>>>>>> maybe some use other jdks :-) )
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Now I see lightning is apache
>>>>>>>>>>>>>>>> licensed and this is fine but I
>>>>>>>>> don't
>>>>>>>>>>>>> think
>>>>>>>>>>>>>>>> it is published in any public
>>>>>>>>>>>>>>>> maven repo, am I right? We could
>>>>>>>>> find a
>>>>>>>>>>>>> way
>>>>>>>>>>>>>>>> to deal with this; options vary
>>>>>>>>>>>>>>>> from publishing lightning to the
>>>>>>>>> free
>>>>>>>>>>>>>>>> sonatype repo, joining the ASF
>>>>>>>>>>>>>>>> (which is great anyhow!) and
>>>>>>>>>>>>> republishing
>>>>>>>>>>>>>>>> lightning code in DirectMemory
>>>>>>>>>>>>>>>> becoming a commiter (which has
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>> undergo
>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>> PMC vote).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> At least for the moment he can
>>>>>>>>>>>>>>> provide a patch to be integrated
>>>>>>>>>>>>>>> in
>>>>>>>>> DM
>>>>>>>>>>>>> :-)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'd like to hear your and the
>>>>>>>>>>>>>>>> team feelings on this.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks, Raffaele
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Sat, Sep 29, 2012 at 3:27 PM,
>>>>>>>>>>>>>>>> Noctarius <me...@noctarius.com>
>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hey Raffaele,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> that's quite similar to what I
>>>>>>>>>>>>>>>>> did at work. We're developing
>>>>>>>>> Flash
>>>>>>>>>>>>>>>>> online games and using a
>>>>>>>>>>>>>>>>> customized AMF serialization.
>>>>>>>>>>>>>>>>> To support async writing of the
>>>>>>>>>>>>>>>>> clients event results I added
>>>>>>>>>>>>>>>>> serialization
>>>>>>>>> of
>>>>>>>>>>>>>>>>> the components / entities to
>>>>>>>>>>>>>>>>> the players zone calculation
>>>>>>>>>>>>>>>>> and
>>>>>>>>> stored
>>>>>>>>>>>>>>>>> the pre-serialized bytestream
>>>>>>>>>>>>>>>>> directly to the off-heap (using
>>>>>>>>>>>>>>>>> direct-ring-cache
>>>>>>>>>>>>>>>>> implementation). When the
>>>>>>>>>>>>>>>>> client requests the results
>>>>>>>>>>>>>>>>> (using long-polling) I just
>>>>>>>>>>>>>>>>> write the pre-serialized
>>>>>>>>> data to
>>>>>>>>>>>>>>>>> the right position to
>>>>>>>>>>>>>>>>> deserialize it by standard ways
>>>>>>>>>>>>>>>>> on Flash
>>>>>>>>> side.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> So yeah an seriliaztion to
>>>>>>>>>>>>>>>>> off-heap algorithm would be
>>>>>>>>>>>>>>>>> fine. You
>>>>>>>>> can
>>>>>>>>>>>>>>>>> avoid using byte arrays and
>>>>>>>>>>>>>>>>> minimalize GC.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Cheers Chris
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Am 29.09.2012 15:02, schrieb
>>>>>>>>>>>>>>>>> Raffaele P. Guidi:
>>>>>>>>>>>>>>>>>> Nice to hear back from you!
>>>>>>>>>>>>>>>>>> Yes, I was thinking about
>>>>>>>>>>>>>>>>>> creating a
>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>> memory
>>>>>>>>>>>>>>>>>> storage implementation using
>>>>>>>>>>>>>>>>>> Unsafe (and I did it,
>>>>>>>>>>>>>>>>>> recently) and
>>>>>>>>>>>>>>> also, as
>>>>>>>>>>>>>>>>>> DirectMemory relies heavily
>>>>>>>>>>>>>>>>>> on serialization (and
>>>>>>>>>>>>>>>>>> supports many
>>>>>>>>> of
>>>>>>>>>>>>>>> them,
>>>>>>>>>>>>>>>>>> protostuff, protobuf, msgpack
>>>>>>>>>>>>>>>>>> and of course standard
>>>>>>>>>>>>> serialization),
>>>>>>>>>>>>>>>>> about
>>>>>>>>>>>>>>>>>> creating a simple embedded
>>>>>>>>>>>>>>>>>> serializer leveraging the
>>>>>>>>>>>>>>>>>> same
>>>>>>>>>>>>> techniques
>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>>>> used (Unsafe and bytecode
>>>>>>>>>>>>>>>>>> generation). The idea with
>>>>>>>>>>>>>>>>>> embedding is avoiding to
>>>>>>>>>>>>>>>>>> serialize in a byte array
>>>>>>>>>>>>> and
>>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>>>>> moving the byte array to
>>>>>>>>>>>>>>>>>> off-heap memory (via Unsafe
>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>> ByteBuffers),
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>> serializing directly into a
>>>>>>>>>>>>>>>>>> "managed" off-heap buffer,
>>>>>>>>>>>>>>>>>> thus
>>>>>>>>> further
>>>>>>>>>>>>>>>>>> optimizing heap utilization
>>>>>>>>>>>>>>>>>> (less GC).
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> What do you think about it?
>>>>>>>>>>>>>>>>>> Does it make any sense to
>>>>>>>>>>>>>>>>>> you?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Ciao, R
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Sat, Sep 29, 2012 at 2:40
>>>>>>>>>>>>>>>>>> PM, Noctarius
>>>>>>>>>>>>>>>>>> <me...@noctarius.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hey guys,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Raffaele found out about a
>>>>>>>>>>>>>>>>>>> project of mine (Lightning)
>>>>>>>>>>>>>>>>>>> a few
>>>>>>>>> weeks
>>>>>>>>>>>>>>>>>>> ago. Lightning is a heavy
>>>>>>>>>>>>>>>>>>> Unsafe and Bytecode
>>>>>>>>>>>>>>>>>>> generation using
>>>>>>>>>>>>>>>>>>> Serializer implementation.
>>>>>>>>>>>>>>>>>>> He told me that he was
>>>>>>>>>>>>>>>>>>> interested in adding
>>>>>>>>>>>>>>>>>>> something similar to
>>>>>>>>>>>>>>>>>>> DirectMemory and I would be
>>>>>>>>>>>>>>>>>>> glad to
>>>>>>>>>>>>> help
>>>>>>>>>>>>>>>>>>> out in this.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Another project I started a
>>>>>>>>>>>>>>>>>>> few days ago, since it was
>>>>>>>>>>>>>>>>>>> needed
>>>>>>>>> for
>>>>>>>>>>>>>>>>>>> work is DirectRingCache.
>>>>>>>>>>>>>>>>>>> The name not really meets
>>>>>>>>>>>>>>>>>>> to actual implementation
>>>>>>>>>>>>>>>>>>> since it's not yet a ring
>>>>>>>>>>>>>>>>>>> buffer using cache. I
>>>>>>>>>>>>> used
>>>>>>>>>>>>>>>>>>> this for a
>>>>>>>>>>>>>>>>>>> pre-serialization simple
>>>>>>>>>>>>>>>>>>> bytestream cache with
>>>>>>>>>>>>>>>>>>> self-growing buffers. It
>>>>>>>>>>>>>>>>>>> could be nice to have
>>>>>>>>>>>>>>>>>>> DirectMemory
>>>>>>>>> having
>>>>>>>>>>>>>>>>>>> raw "buffers" to write to
>>>>>>>>>>>>>>>>>>> or to read from.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Here are the links from
>>>>>>>>>>>>>>>>>>> the projects:
>>>>>>>>>>>>>>>>>>> https://github.com/noctarius/Lightning
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>
>>>>>>>>>>>>>>>>>>>
https://github.com/noctarius/direct-ring-cache
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>> It would be nice to help out since I really like
>>>>>>> the idea of
>>>>>>>>>>>>>>>>>>> DirectMemory and since
>>>>>>>>>>>>>>>>>>> direct-ring-cache is some
>>>>>>>>>>>>>>>>>>> kind of
>>>>>>>>>>>>> reinventing
>>>>>>>>>>>>>>>>>>> the wheel.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Cheers Noctarius (Chris)
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -- Olivier Lamy Talend:
>>>>>>>>>>>>>>> http://coders.talend.com
>>>>>>>>>>>>>>> http://twitter.com/olamy |
>>>>>>>>>>>>>>> http://linkedin.com/in/olamy
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> -- Olivier Lamy Talend:
>>>>>>>>>>>>> http://coders.talend.com
>>>>>>>>>>>>> http://twitter.com/olamy |
>>>>>>>>>>>>> http://linkedin.com/in/olamy
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -- Olivier Lamy Talend:
>>>>>>>>> http://coders.talend.com
>>>>>>>>> http://twitter.com/olamy |
>>>>>>>>> http://linkedin.com/in/olamy
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>>
>>>>
>>>> -- Olivier Lamy Talend: http://coders.talend.com
>>>> http://twitter.com/olamy | http://linkedin.com/in/olamy
>>>>
>>>
>>
>
Re: Additional Serializer and raw Buffer access
Posted by "Raffaele P. Guidi" <ra...@gmail.com>.
>> Hehe well that really sounds like a nice bunch of people.
Indeed they are (I'm a newbie as well and try to do my best)
>> If lightning will be a sub-part (sub-project) of DM, do I need to write
an project purposal?
Nope, not needed for a sub-project
>> Do I need to make any changes to the pom.xml like adding a
special parent pom or anything like that?
Not for the serializer - just have to take a look at project dependencies -
or, better, at their licenses - are they compatible with the ASL 2.0? i.e.
a GPL'd library is not a good fit and should be replaced with an apache
licensed (or BSD, or MIT...) one if possible. For the integration module is
a separate story - you should start off copying one of the other
serializers and reusing the same pom and directory structure.
Pleased to meet you, Chris :)
Ciao,
R
On Sat, Sep 29, 2012 at 8:24 PM, Noctarius <me...@noctarius.com> wrote:
> Hehe well that really sounds like a nice bunch of people.
>
> Ok to be true I couldn't wait until tomorrow and started already
> reading the links.
> From what I was reading:
> If lightning will be a sub-part (sub-project) of DM, do I need to
> write an project purposal?
>
> Do I need to make any changes to the pom.xml like adding a special
> parent pom or anything like that?
>
> In general: there are a lot things to know :-)
>
> Am 29.09.2012 19:59, schrieb Raffaele P. Guidi:
> > Negative part of ASF membership? You get together with a lot of geeky,
> > talented people with a fixation for software and open source. Oh wait but
> > this is actually nice! :-D
> > Il giorno 29/set/2012 19:05, "Olivier Lamy" <ol...@apache.org> ha
> scritto:
> >
> >> 2012/9/29 Noctarius <me...@noctarius.com>:
> >>> Thanks Olivier for carify, I'll take a look in it tomorrow but
> >>> there's just one question left (for now ;)):
> >>> What is that vote for becoming a committer? What if the vote will
> >>> be negative?
> >> The vote is on private list (pmc list for privacy reasons and possible
> >> negative stuff being on public lists)
> >>> Until now I just used Apache stuff, was never interested in being
> >>> part of it so I guess it can be negative for any reason, can't it?
> >> I don't see why it could be negative but suspens .... :-)
> >>>
> >>> Am 29.09.2012 18:56, schrieb Olivier Lamy:
> >>>> 2012/9/29 Noctarius <me...@noctarius.com>:
> >>>>> Nope my real name is Christoph Engelbert, but Noctarius is
> >>>>> the all time nick :)
> >>>>>
> >>>>> Renaming the package should be no problem, is it
> >>>>> "org.apache.directmemory.lightning" or what would it be?
> >>>> fine for me
> >>>>> Then there needs to be a change in the license header as
> >>>>> Olivier mentioned, that means just remove the first sentence
> >>>>> or is there anything more to do (maybe it's easiest thing to
> >>>>> just copy the header from DM file ;))?
> >>>> yup use same header as DM
> >>>>>
> >>>>> The CLA is just a form to clarify that the source can be
> >>>>> contributed to the Apache Foundation?
> >>>> yup correct.
> >>>>>
> >>>>> The final step will be attaching the patch in form of a huge
> >>>>> diff file?
> >>>> yes
> >>>>>
> >>>>> And what is the way to apply for a membership? Never thought
> >>>>> about how to do that.
> >>>> Read here http://apache.org/foundation/how-it-works.html and
> >>>> here http://apache.org/foundation/getinvolved.html . And feel
> >>>> free to ask any questions :-)
> >>>>>
> >>>>> Chris
> >>>>>
> >>>>> Am 29.09.2012 18:23, schrieb Raffaele P. Guidi:
> >>>>>> OK, deal, at least for me ;-) I propose you rename the
> >>>>>> packages, produce a patch for this and the new serializer
> >>>>>> module (should be simple enough starting from an existing
> >>>>>> one) and, in the meanwhile, apply for ASF membership. Is
> >>>>>> IP clearance needed? I guess yes. After this we will come
> >>>>>> up with a formal vote regarding Noctarius (is this your
> >>>>>> real name?!) allowance in the project team.
> >>>>>>
> >>>>>> Good times are gonna come :-)
> >>>>>>
> >>>>>> Thanks, R Il giorno 29/set/2012 17:58, "Olivier Lamy"
> >>>>>> <ol...@apache.org> ha scritto:
> >>>>>>
> >>>>>>> 2012/9/29 Raffaele P. Guidi
> >>>>>>> <ra...@gmail.com>:
> >>>>>>>> Well we already have a NIO ready interface allowing
> >>>>>>>> direct access to DMs managed bytebuffers but I think
> >>>>>>>> this is just half way to what could be achieved
> >>>>>>>> optimally blending serialization and memory allocation
> >>>>>>>> together.
> >>>>>>>>
> >>>>>>>> Lightning as a module is of course a good idea and it
> >>>>>>>> could easily evolve as a subproject (for the more
> >>>>>>>> experienced asf members: is it a feasible way?).
> >>>>>>> Nothing prevent to have
> >>>>>>> http://svn.apache.org/repos/asf/directmemory/lightning/trunk
> >>>>>>>
> >>>>>>>
> >>> with a package like: o.a.d.lightning That will be a
> >>>>>>> subproject.
> >>>>>>>>
> >>>>>>>> Ciao, R Il giorno 29/set/2012 17:44, "Noctarius"
> >>>>>>>> <me...@noctarius.com> ha scritto:
> >>>>>>>>
> >>>>>>>>> Hey guys,
> >>>>>>>>>
> >>>>>>>>> ok there's no lightning binary available since
> >>>>>>>>> lightning wasn't ready yet for releasing.
> >>>>>>>>>
> >>>>>>>>> For being the only developer it would be no problem
> >>>>>>>>> to contribute the sourcebase for DirectMemory but I'm
> >>>>>>>>> not sure yet if it wouldn't be better to seperate it
> >>>>>>>>> to be available without using DirectMemory itself. I
> >>>>>>>>> started it as a serializer for cluster
> >>>>>>>>> synchronization, but it would be cool to contribute
> >>>>>>>>> lightning as a subproject to DirectMemory :-)
> >>>>>>>>>
> >>>>>>>>> About the second project I would love to see a
> >>>>>>>>> public available buffer API directly in DirectMemory
> >>>>>>>>> so that project would be nearly needless :-) The only
> >>>>>>>>> difference I think is the allocation strategy my
> >>>>>>>>> implementation is using against the one DirectMemory
> >>>>>>>>> has, but I'm pretty sure the allocation is extensible
> >>>>>>>>> ;-)
> >>>>>>>>>
> >>>>>>>>> Like I said, for both projects I'm the only dev so
> >>>>>>>>> there would be no IP problem. So if it's ok to you to
> >>>>>>>>> not include lightning directly in DM I would be glad
> >>>>>>>>> to contribute to the Apache Foundation :-)
> >>>>>>>>>
> >>>>>>>>> Cheers Chris
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Am 29.09.2012 16:53, schrieb Raffaele P. Guidi:
> >>>>>>>>>> Ok so it's up to noctarius - your move! ;-)
> >>>>>>>>>> Regarding the new unsafe storage: it's an opt-in
> >>>>>>>>>> feature that can be set with the fluent API
> >>>>>>> (and
> >>>>>>>>>> soon through the conference file).
> >>>>>>>>>>
> >>>>>>>>>> Ciao, R Il giorno 29/set/2012 16:45, "Olivier
> >>>>>>>>>> Lamy" <ol...@apache.org> ha
> >>>>>>>>> scritto:
> >>>>>>>>>>
> >>>>>>>>>>> 2012/9/29 Raffaele P. Guidi
> >>>>>>>>>>> <ra...@gmail.com>:
> >>>>>>>>>>>>>> At least for the moment he can provide a
> >>>>>>>>>>>>>> patch to be integrated
> >>>>>>> in DM
> >>>>>>>>>>> :-)
> >>>>>>>>>>>>
> >>>>>>>>>>>> Sure, but as lightning is not in any public
> >>>>>>>>>>>> mvn repo should its
> >>>>>>> code be
> >>>>>>>>>>>> re-published in our svn? Or what?
> >>>>>>>>>>> @Apache we don't care about binaries, only
> >>>>>>>>>>> sources are important ! (a bit theorical for sure
> >>>>>>>>>>> but that's it :-) ). So if Noctarius was the only
> >>>>>>>>>>> guy who participate in lightning. He can just
> >>>>>>>>>>> provide a patch we could integrate as a new dm
> >>>>>>>>>>> module (note: the patch must not contains any
> >>>>>>>>>>> more copyright and all sources must have ASF
> >>>>>>>>>>> licenses).
> >>>>>>>>>>>
> >>>>>>>>>>> "Copyright 2012 the original author or authors."
> >>>>>>>>>>> must be removed. And BTW package must be changed
> >>>>>>>>>>> :-) (com.github is not acceptable
> >>>>>>> @asf
> >>>>>>>>>>> :-) )(@Noctarius are you working for github ? :-)
> >>>>>>>>>>> )
> >>>>>>>>>>>
> >>>>>>>>>>> And having him as a committer will be only a
> >>>>>>>>>>> matter of voting (we
> >>>>>>> have
> >>>>>>>>>>> a great chair who take cares of administrative
> >>>>>>>>>>> stuff :P )
> >>>>>>>>>>>
> >>>>>>>>>>> If some others have participated in the project,
> >>>>>>>>>>> we must pass tru an ip clearance mechanism
> >>>>>>>>>>> (http://incubator.apache.org/ip-clearance/index.html)
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>> and all contributors to lightning must provide a cla.
> >>>>>>>>>>> (It it's the case I can help)
> >>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>> perso I'd like we avoid hard dependency on
> >>>>>>>>>>>>>> Unsafe as maybe some
> >>>>>>> use
> >>>>>>>>>>>> other jdks :-)
> >>>>>>>>>>>>
> >>>>>>>>>>>> Well, I believe Unsafe is supported by Sun
> >>>>>>>>>>>> JDK, OpenJDK, IBM JDK and JRockit - and I
> >>>>>>>>>>>> believe that it is more than enough. Also keep
> >>>>>>>>>>>> in
> >>>>>>> mind
> >>>>>>>>>>> that
> >>>>>>>>>>>> we already have an alternative Unsafe based
> >>>>>>>>>>>> memory storage - and
> >>>>>>>>> although
> >>>>>>>>>>>> it's not thoroughly tested for performance it
> >>>>>>>>>>>> dramaticaly simplifies
> >>>>>>>>>>> code,
> >>>>>>>>>>>> I have great expectations about it.
> >>>>>>>>>>> Me too :-). Yup definitely more simple and faster
> >>>>>>>>>>> ! But we must provide a switch off configuration
> >>>>>>>>>>> mechanism if some
> >>>>>>> users
> >>>>>>>>>>> don't want to use that (because Unsafe is just
> >>>>>>>>>>> "Unsafe" :-) ) And sorry I didn't have a look yet
> >>>>>>>>>>> at your changes with using Unsafe.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Cheers, R
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Sat, Sep 29, 2012 at 4:03 PM, Olivier Lamy
> >>>>>>>>>>>> <ol...@apache.org>
> >>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> 2012/9/29 Raffaele P. Guidi
> >>>>>>>>>>>>> <ra...@gmail.com>:
> >>>>>>>>>>>>>> What do you think about: 1) implementing a
> >>>>>>>>>>>>>> lightning serialization module 2) creating
> >>>>>>>>>>>>>> a serializer that directly works on a
> >>>>>>>>>>>>>> directmemory
> >>>>>>>>>>> provider
> >>>>>>>>>>>>>> ByteBuffer or (maybe better) Unsafe based
> >>>>>>>>>>>>>> Pointer?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Sounds good (perso I'd like we avoid hard
> >>>>>>>>>>>>> dependency on Unsafe as maybe some use other
> >>>>>>>>>>>>> jdks :-) )
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Now I see lightning is apache licensed and
> >>>>>>>>>>>>>> this is fine but I
> >>>>>>> don't
> >>>>>>>>>>> think
> >>>>>>>>>>>>>> it is published in any public maven repo,
> >>>>>>>>>>>>>> am I right? We could
> >>>>>>> find a
> >>>>>>>>>>> way
> >>>>>>>>>>>>>> to deal with this; options vary from
> >>>>>>>>>>>>>> publishing lightning to the
> >>>>>>> free
> >>>>>>>>>>>>>> sonatype repo, joining the ASF (which is
> >>>>>>>>>>>>>> great anyhow!) and
> >>>>>>>>>>> republishing
> >>>>>>>>>>>>>> lightning code in DirectMemory becoming a
> >>>>>>>>>>>>>> commiter (which has to
> >>>>>>>>>>> undergo
> >>>>>>>>>>>>> a
> >>>>>>>>>>>>>> PMC vote).
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> At least for the moment he can provide a
> >>>>>>>>>>>>> patch to be integrated in
> >>>>>>> DM
> >>>>>>>>>>> :-)
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I'd like to hear your and the team feelings
> >>>>>>>>>>>>>> on this.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks, Raffaele
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Sat, Sep 29, 2012 at 3:27 PM, Noctarius
> >>>>>>>>>>>>>> <me...@noctarius.com>
> >>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hey Raffaele,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> that's quite similar to what I did at
> >>>>>>>>>>>>>>> work. We're developing
> >>>>>>> Flash
> >>>>>>>>>>>>>>> online games and using a customized AMF
> >>>>>>>>>>>>>>> serialization. To support async writing
> >>>>>>>>>>>>>>> of the clients event results I added
> >>>>>>>>>>>>>>> serialization
> >>>>>>> of
> >>>>>>>>>>>>>>> the components / entities to the players
> >>>>>>>>>>>>>>> zone calculation and
> >>>>>>> stored
> >>>>>>>>>>>>>>> the pre-serialized bytestream directly to
> >>>>>>>>>>>>>>> the off-heap (using direct-ring-cache
> >>>>>>>>>>>>>>> implementation). When the client
> >>>>>>>>>>>>>>> requests the results (using long-polling)
> >>>>>>>>>>>>>>> I just write the pre-serialized
> >>>>>>> data to
> >>>>>>>>>>>>>>> the right position to deserialize it by
> >>>>>>>>>>>>>>> standard ways on Flash
> >>>>>>> side.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> So yeah an seriliaztion to off-heap
> >>>>>>>>>>>>>>> algorithm would be fine. You
> >>>>>>> can
> >>>>>>>>>>>>>>> avoid using byte arrays and minimalize
> >>>>>>>>>>>>>>> GC.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Cheers Chris
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Am 29.09.2012 15:02, schrieb Raffaele P.
> >>>>>>>>>>>>>>> Guidi:
> >>>>>>>>>>>>>>>> Nice to hear back from you! Yes, I was
> >>>>>>>>>>>>>>>> thinking about creating a
> >>>>>>>>>>> new
> >>>>>>>>>>>>>>> memory
> >>>>>>>>>>>>>>>> storage implementation using Unsafe
> >>>>>>>>>>>>>>>> (and I did it, recently) and
> >>>>>>>>>>>>> also, as
> >>>>>>>>>>>>>>>> DirectMemory relies heavily on
> >>>>>>>>>>>>>>>> serialization (and supports many
> >>>>>>> of
> >>>>>>>>>>>>> them,
> >>>>>>>>>>>>>>>> protostuff, protobuf, msgpack and of
> >>>>>>>>>>>>>>>> course standard
> >>>>>>>>>>> serialization),
> >>>>>>>>>>>>>>> about
> >>>>>>>>>>>>>>>> creating a simple embedded serializer
> >>>>>>>>>>>>>>>> leveraging the same
> >>>>>>>>>>> techniques
> >>>>>>>>>>>>> you
> >>>>>>>>>>>>>>>> used (Unsafe and bytecode generation).
> >>>>>>>>>>>>>>>> The idea with embedding is avoiding to
> >>>>>>>>>>>>>>>> serialize in a byte array
> >>>>>>>>>>> and
> >>>>>>>>>>>>> then
> >>>>>>>>>>>>>>>> moving the byte array to off-heap
> >>>>>>>>>>>>>>>> memory (via Unsafe or
> >>>>>>>>>>> ByteBuffers),
> >>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>> serializing directly into a "managed"
> >>>>>>>>>>>>>>>> off-heap buffer, thus
> >>>>>>> further
> >>>>>>>>>>>>>>>> optimizing heap utilization (less GC).
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> What do you think about it? Does it
> >>>>>>>>>>>>>>>> make any sense to you?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Ciao, R
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Sat, Sep 29, 2012 at 2:40 PM,
> >>>>>>>>>>>>>>>> Noctarius <me...@noctarius.com>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hey guys,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Raffaele found out about a project
> >>>>>>>>>>>>>>>>> of mine (Lightning) a few
> >>>>>>> weeks
> >>>>>>>>>>>>>>>>> ago. Lightning is a heavy Unsafe and
> >>>>>>>>>>>>>>>>> Bytecode generation using Serializer
> >>>>>>>>>>>>>>>>> implementation. He told me that he
> >>>>>>>>>>>>>>>>> was interested in adding something
> >>>>>>>>>>>>>>>>> similar to DirectMemory and I would
> >>>>>>>>>>>>>>>>> be glad to
> >>>>>>>>>>> help
> >>>>>>>>>>>>>>>>> out in this.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Another project I started a few days
> >>>>>>>>>>>>>>>>> ago, since it was needed
> >>>>>>> for
> >>>>>>>>>>>>>>>>> work is DirectRingCache. The name
> >>>>>>>>>>>>>>>>> not really meets to actual
> >>>>>>>>>>>>>>>>> implementation since it's not yet a
> >>>>>>>>>>>>>>>>> ring buffer using cache. I
> >>>>>>>>>>> used
> >>>>>>>>>>>>>>>>> this for a pre-serialization simple
> >>>>>>>>>>>>>>>>> bytestream cache with self-growing
> >>>>>>>>>>>>>>>>> buffers. It could be nice to have
> >>>>>>>>>>>>>>>>> DirectMemory
> >>>>>>> having
> >>>>>>>>>>>>>>>>> raw "buffers" to write to or to read
> >>>>>>>>>>>>>>>>> from.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Here are the links from the
> >>>>>>>>>>>>>>>>> projects:
> >>>>>>>>>>>>>>>>> https://github.com/noctarius/Lightning
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>> https://github.com/noctarius/direct-ring-cache
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>> It would be nice to help out since I really like the idea of
> >>>>>>>>>>>>>>>>> DirectMemory and since
> >>>>>>>>>>>>>>>>> direct-ring-cache is some kind of
> >>>>>>>>>>> reinventing
> >>>>>>>>>>>>>>>>> the wheel.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Cheers Noctarius (Chris)
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> -- Olivier Lamy Talend:
> >>>>>>>>>>>>> http://coders.talend.com
> >>>>>>>>>>>>> http://twitter.com/olamy |
> >>>>>>>>>>>>> http://linkedin.com/in/olamy
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> -- Olivier Lamy Talend: http://coders.talend.com
> >>>>>>>>>>> http://twitter.com/olamy |
> >>>>>>>>>>> http://linkedin.com/in/olamy
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> -- Olivier Lamy Talend: http://coders.talend.com
> >>>>>>> http://twitter.com/olamy | http://linkedin.com/in/olamy
> >>>>>>>
> >>>>>>
> >>>>
> >>>>
> >>>>
> >>
> >>
> >>
> >> --
> >> Olivier Lamy
> >> Talend: http://coders.talend.com
> >> http://twitter.com/olamy | http://linkedin.com/in/olamy
> >>
> >
>
Re: Additional Serializer and raw Buffer access
Posted by Noctarius <me...@noctarius.com>.
Hehe well that really sounds like a nice bunch of people.
Ok to be true I couldn't wait until tomorrow and started already
reading the links.
>From what I was reading:
If lightning will be a sub-part (sub-project) of DM, do I need to
write an project purposal?
Do I need to make any changes to the pom.xml like adding a special
parent pom or anything like that?
In general: there are a lot things to know :-)
Am 29.09.2012 19:59, schrieb Raffaele P. Guidi:
> Negative part of ASF membership? You get together with a lot of geeky,
> talented people with a fixation for software and open source. Oh wait but
> this is actually nice! :-D
> Il giorno 29/set/2012 19:05, "Olivier Lamy" <ol...@apache.org> ha scritto:
>
>> 2012/9/29 Noctarius <me...@noctarius.com>:
>>> Thanks Olivier for carify, I'll take a look in it tomorrow but
>>> there's just one question left (for now ;)):
>>> What is that vote for becoming a committer? What if the vote will
>>> be negative?
>> The vote is on private list (pmc list for privacy reasons and possible
>> negative stuff being on public lists)
>>> Until now I just used Apache stuff, was never interested in being
>>> part of it so I guess it can be negative for any reason, can't it?
>> I don't see why it could be negative but suspens .... :-)
>>>
>>> Am 29.09.2012 18:56, schrieb Olivier Lamy:
>>>> 2012/9/29 Noctarius <me...@noctarius.com>:
>>>>> Nope my real name is Christoph Engelbert, but Noctarius is
>>>>> the all time nick :)
>>>>>
>>>>> Renaming the package should be no problem, is it
>>>>> "org.apache.directmemory.lightning" or what would it be?
>>>> fine for me
>>>>> Then there needs to be a change in the license header as
>>>>> Olivier mentioned, that means just remove the first sentence
>>>>> or is there anything more to do (maybe it's easiest thing to
>>>>> just copy the header from DM file ;))?
>>>> yup use same header as DM
>>>>>
>>>>> The CLA is just a form to clarify that the source can be
>>>>> contributed to the Apache Foundation?
>>>> yup correct.
>>>>>
>>>>> The final step will be attaching the patch in form of a huge
>>>>> diff file?
>>>> yes
>>>>>
>>>>> And what is the way to apply for a membership? Never thought
>>>>> about how to do that.
>>>> Read here http://apache.org/foundation/how-it-works.html and
>>>> here http://apache.org/foundation/getinvolved.html . And feel
>>>> free to ask any questions :-)
>>>>>
>>>>> Chris
>>>>>
>>>>> Am 29.09.2012 18:23, schrieb Raffaele P. Guidi:
>>>>>> OK, deal, at least for me ;-) I propose you rename the
>>>>>> packages, produce a patch for this and the new serializer
>>>>>> module (should be simple enough starting from an existing
>>>>>> one) and, in the meanwhile, apply for ASF membership. Is
>>>>>> IP clearance needed? I guess yes. After this we will come
>>>>>> up with a formal vote regarding Noctarius (is this your
>>>>>> real name?!) allowance in the project team.
>>>>>>
>>>>>> Good times are gonna come :-)
>>>>>>
>>>>>> Thanks, R Il giorno 29/set/2012 17:58, "Olivier Lamy"
>>>>>> <ol...@apache.org> ha scritto:
>>>>>>
>>>>>>> 2012/9/29 Raffaele P. Guidi
>>>>>>> <ra...@gmail.com>:
>>>>>>>> Well we already have a NIO ready interface allowing
>>>>>>>> direct access to DMs managed bytebuffers but I think
>>>>>>>> this is just half way to what could be achieved
>>>>>>>> optimally blending serialization and memory allocation
>>>>>>>> together.
>>>>>>>>
>>>>>>>> Lightning as a module is of course a good idea and it
>>>>>>>> could easily evolve as a subproject (for the more
>>>>>>>> experienced asf members: is it a feasible way?).
>>>>>>> Nothing prevent to have
>>>>>>> http://svn.apache.org/repos/asf/directmemory/lightning/trunk
>>>>>>>
>>>>>>>
>>> with a package like: o.a.d.lightning That will be a
>>>>>>> subproject.
>>>>>>>>
>>>>>>>> Ciao, R Il giorno 29/set/2012 17:44, "Noctarius"
>>>>>>>> <me...@noctarius.com> ha scritto:
>>>>>>>>
>>>>>>>>> Hey guys,
>>>>>>>>>
>>>>>>>>> ok there's no lightning binary available since
>>>>>>>>> lightning wasn't ready yet for releasing.
>>>>>>>>>
>>>>>>>>> For being the only developer it would be no problem
>>>>>>>>> to contribute the sourcebase for DirectMemory but I'm
>>>>>>>>> not sure yet if it wouldn't be better to seperate it
>>>>>>>>> to be available without using DirectMemory itself. I
>>>>>>>>> started it as a serializer for cluster
>>>>>>>>> synchronization, but it would be cool to contribute
>>>>>>>>> lightning as a subproject to DirectMemory :-)
>>>>>>>>>
>>>>>>>>> About the second project I would love to see a
>>>>>>>>> public available buffer API directly in DirectMemory
>>>>>>>>> so that project would be nearly needless :-) The only
>>>>>>>>> difference I think is the allocation strategy my
>>>>>>>>> implementation is using against the one DirectMemory
>>>>>>>>> has, but I'm pretty sure the allocation is extensible
>>>>>>>>> ;-)
>>>>>>>>>
>>>>>>>>> Like I said, for both projects I'm the only dev so
>>>>>>>>> there would be no IP problem. So if it's ok to you to
>>>>>>>>> not include lightning directly in DM I would be glad
>>>>>>>>> to contribute to the Apache Foundation :-)
>>>>>>>>>
>>>>>>>>> Cheers Chris
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Am 29.09.2012 16:53, schrieb Raffaele P. Guidi:
>>>>>>>>>> Ok so it's up to noctarius - your move! ;-)
>>>>>>>>>> Regarding the new unsafe storage: it's an opt-in
>>>>>>>>>> feature that can be set with the fluent API
>>>>>>> (and
>>>>>>>>>> soon through the conference file).
>>>>>>>>>>
>>>>>>>>>> Ciao, R Il giorno 29/set/2012 16:45, "Olivier
>>>>>>>>>> Lamy" <ol...@apache.org> ha
>>>>>>>>> scritto:
>>>>>>>>>>
>>>>>>>>>>> 2012/9/29 Raffaele P. Guidi
>>>>>>>>>>> <ra...@gmail.com>:
>>>>>>>>>>>>>> At least for the moment he can provide a
>>>>>>>>>>>>>> patch to be integrated
>>>>>>> in DM
>>>>>>>>>>> :-)
>>>>>>>>>>>>
>>>>>>>>>>>> Sure, but as lightning is not in any public
>>>>>>>>>>>> mvn repo should its
>>>>>>> code be
>>>>>>>>>>>> re-published in our svn? Or what?
>>>>>>>>>>> @Apache we don't care about binaries, only
>>>>>>>>>>> sources are important ! (a bit theorical for sure
>>>>>>>>>>> but that's it :-) ). So if Noctarius was the only
>>>>>>>>>>> guy who participate in lightning. He can just
>>>>>>>>>>> provide a patch we could integrate as a new dm
>>>>>>>>>>> module (note: the patch must not contains any
>>>>>>>>>>> more copyright and all sources must have ASF
>>>>>>>>>>> licenses).
>>>>>>>>>>>
>>>>>>>>>>> "Copyright 2012 the original author or authors."
>>>>>>>>>>> must be removed. And BTW package must be changed
>>>>>>>>>>> :-) (com.github is not acceptable
>>>>>>> @asf
>>>>>>>>>>> :-) )(@Noctarius are you working for github ? :-)
>>>>>>>>>>> )
>>>>>>>>>>>
>>>>>>>>>>> And having him as a committer will be only a
>>>>>>>>>>> matter of voting (we
>>>>>>> have
>>>>>>>>>>> a great chair who take cares of administrative
>>>>>>>>>>> stuff :P )
>>>>>>>>>>>
>>>>>>>>>>> If some others have participated in the project,
>>>>>>>>>>> we must pass tru an ip clearance mechanism
>>>>>>>>>>> (http://incubator.apache.org/ip-clearance/index.html)
>>>>>>>>>>>
>>>>>>>>>>>
>>> and all contributors to lightning must provide a cla.
>>>>>>>>>>> (It it's the case I can help)
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>> perso I'd like we avoid hard dependency on
>>>>>>>>>>>>>> Unsafe as maybe some
>>>>>>> use
>>>>>>>>>>>> other jdks :-)
>>>>>>>>>>>>
>>>>>>>>>>>> Well, I believe Unsafe is supported by Sun
>>>>>>>>>>>> JDK, OpenJDK, IBM JDK and JRockit - and I
>>>>>>>>>>>> believe that it is more than enough. Also keep
>>>>>>>>>>>> in
>>>>>>> mind
>>>>>>>>>>> that
>>>>>>>>>>>> we already have an alternative Unsafe based
>>>>>>>>>>>> memory storage - and
>>>>>>>>> although
>>>>>>>>>>>> it's not thoroughly tested for performance it
>>>>>>>>>>>> dramaticaly simplifies
>>>>>>>>>>> code,
>>>>>>>>>>>> I have great expectations about it.
>>>>>>>>>>> Me too :-). Yup definitely more simple and faster
>>>>>>>>>>> ! But we must provide a switch off configuration
>>>>>>>>>>> mechanism if some
>>>>>>> users
>>>>>>>>>>> don't want to use that (because Unsafe is just
>>>>>>>>>>> "Unsafe" :-) ) And sorry I didn't have a look yet
>>>>>>>>>>> at your changes with using Unsafe.
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers, R
>>>>>>>>>>>>
>>>>>>>>>>>> On Sat, Sep 29, 2012 at 4:03 PM, Olivier Lamy
>>>>>>>>>>>> <ol...@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> 2012/9/29 Raffaele P. Guidi
>>>>>>>>>>>>> <ra...@gmail.com>:
>>>>>>>>>>>>>> What do you think about: 1) implementing a
>>>>>>>>>>>>>> lightning serialization module 2) creating
>>>>>>>>>>>>>> a serializer that directly works on a
>>>>>>>>>>>>>> directmemory
>>>>>>>>>>> provider
>>>>>>>>>>>>>> ByteBuffer or (maybe better) Unsafe based
>>>>>>>>>>>>>> Pointer?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Sounds good (perso I'd like we avoid hard
>>>>>>>>>>>>> dependency on Unsafe as maybe some use other
>>>>>>>>>>>>> jdks :-) )
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Now I see lightning is apache licensed and
>>>>>>>>>>>>>> this is fine but I
>>>>>>> don't
>>>>>>>>>>> think
>>>>>>>>>>>>>> it is published in any public maven repo,
>>>>>>>>>>>>>> am I right? We could
>>>>>>> find a
>>>>>>>>>>> way
>>>>>>>>>>>>>> to deal with this; options vary from
>>>>>>>>>>>>>> publishing lightning to the
>>>>>>> free
>>>>>>>>>>>>>> sonatype repo, joining the ASF (which is
>>>>>>>>>>>>>> great anyhow!) and
>>>>>>>>>>> republishing
>>>>>>>>>>>>>> lightning code in DirectMemory becoming a
>>>>>>>>>>>>>> commiter (which has to
>>>>>>>>>>> undergo
>>>>>>>>>>>>> a
>>>>>>>>>>>>>> PMC vote).
>>>>>>>>>>>>>
>>>>>>>>>>>>> At least for the moment he can provide a
>>>>>>>>>>>>> patch to be integrated in
>>>>>>> DM
>>>>>>>>>>> :-)
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'd like to hear your and the team feelings
>>>>>>>>>>>>>> on this.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks, Raffaele
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Sat, Sep 29, 2012 at 3:27 PM, Noctarius
>>>>>>>>>>>>>> <me...@noctarius.com>
>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hey Raffaele,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> that's quite similar to what I did at
>>>>>>>>>>>>>>> work. We're developing
>>>>>>> Flash
>>>>>>>>>>>>>>> online games and using a customized AMF
>>>>>>>>>>>>>>> serialization. To support async writing
>>>>>>>>>>>>>>> of the clients event results I added
>>>>>>>>>>>>>>> serialization
>>>>>>> of
>>>>>>>>>>>>>>> the components / entities to the players
>>>>>>>>>>>>>>> zone calculation and
>>>>>>> stored
>>>>>>>>>>>>>>> the pre-serialized bytestream directly to
>>>>>>>>>>>>>>> the off-heap (using direct-ring-cache
>>>>>>>>>>>>>>> implementation). When the client
>>>>>>>>>>>>>>> requests the results (using long-polling)
>>>>>>>>>>>>>>> I just write the pre-serialized
>>>>>>> data to
>>>>>>>>>>>>>>> the right position to deserialize it by
>>>>>>>>>>>>>>> standard ways on Flash
>>>>>>> side.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> So yeah an seriliaztion to off-heap
>>>>>>>>>>>>>>> algorithm would be fine. You
>>>>>>> can
>>>>>>>>>>>>>>> avoid using byte arrays and minimalize
>>>>>>>>>>>>>>> GC.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Cheers Chris
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Am 29.09.2012 15:02, schrieb Raffaele P.
>>>>>>>>>>>>>>> Guidi:
>>>>>>>>>>>>>>>> Nice to hear back from you! Yes, I was
>>>>>>>>>>>>>>>> thinking about creating a
>>>>>>>>>>> new
>>>>>>>>>>>>>>> memory
>>>>>>>>>>>>>>>> storage implementation using Unsafe
>>>>>>>>>>>>>>>> (and I did it, recently) and
>>>>>>>>>>>>> also, as
>>>>>>>>>>>>>>>> DirectMemory relies heavily on
>>>>>>>>>>>>>>>> serialization (and supports many
>>>>>>> of
>>>>>>>>>>>>> them,
>>>>>>>>>>>>>>>> protostuff, protobuf, msgpack and of
>>>>>>>>>>>>>>>> course standard
>>>>>>>>>>> serialization),
>>>>>>>>>>>>>>> about
>>>>>>>>>>>>>>>> creating a simple embedded serializer
>>>>>>>>>>>>>>>> leveraging the same
>>>>>>>>>>> techniques
>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>> used (Unsafe and bytecode generation).
>>>>>>>>>>>>>>>> The idea with embedding is avoiding to
>>>>>>>>>>>>>>>> serialize in a byte array
>>>>>>>>>>> and
>>>>>>>>>>>>> then
>>>>>>>>>>>>>>>> moving the byte array to off-heap
>>>>>>>>>>>>>>>> memory (via Unsafe or
>>>>>>>>>>> ByteBuffers),
>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>> serializing directly into a "managed"
>>>>>>>>>>>>>>>> off-heap buffer, thus
>>>>>>> further
>>>>>>>>>>>>>>>> optimizing heap utilization (less GC).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> What do you think about it? Does it
>>>>>>>>>>>>>>>> make any sense to you?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Ciao, R
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Sat, Sep 29, 2012 at 2:40 PM,
>>>>>>>>>>>>>>>> Noctarius <me...@noctarius.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hey guys,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Raffaele found out about a project
>>>>>>>>>>>>>>>>> of mine (Lightning) a few
>>>>>>> weeks
>>>>>>>>>>>>>>>>> ago. Lightning is a heavy Unsafe and
>>>>>>>>>>>>>>>>> Bytecode generation using Serializer
>>>>>>>>>>>>>>>>> implementation. He told me that he
>>>>>>>>>>>>>>>>> was interested in adding something
>>>>>>>>>>>>>>>>> similar to DirectMemory and I would
>>>>>>>>>>>>>>>>> be glad to
>>>>>>>>>>> help
>>>>>>>>>>>>>>>>> out in this.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Another project I started a few days
>>>>>>>>>>>>>>>>> ago, since it was needed
>>>>>>> for
>>>>>>>>>>>>>>>>> work is DirectRingCache. The name
>>>>>>>>>>>>>>>>> not really meets to actual
>>>>>>>>>>>>>>>>> implementation since it's not yet a
>>>>>>>>>>>>>>>>> ring buffer using cache. I
>>>>>>>>>>> used
>>>>>>>>>>>>>>>>> this for a pre-serialization simple
>>>>>>>>>>>>>>>>> bytestream cache with self-growing
>>>>>>>>>>>>>>>>> buffers. It could be nice to have
>>>>>>>>>>>>>>>>> DirectMemory
>>>>>>> having
>>>>>>>>>>>>>>>>> raw "buffers" to write to or to read
>>>>>>>>>>>>>>>>> from.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Here are the links from the
>>>>>>>>>>>>>>>>> projects:
>>>>>>>>>>>>>>>>> https://github.com/noctarius/Lightning
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>> https://github.com/noctarius/direct-ring-cache
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>> It would be nice to help out since I really like the idea of
>>>>>>>>>>>>>>>>> DirectMemory and since
>>>>>>>>>>>>>>>>> direct-ring-cache is some kind of
>>>>>>>>>>> reinventing
>>>>>>>>>>>>>>>>> the wheel.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Cheers Noctarius (Chris)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> -- Olivier Lamy Talend:
>>>>>>>>>>>>> http://coders.talend.com
>>>>>>>>>>>>> http://twitter.com/olamy |
>>>>>>>>>>>>> http://linkedin.com/in/olamy
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> -- Olivier Lamy Talend: http://coders.talend.com
>>>>>>>>>>> http://twitter.com/olamy |
>>>>>>>>>>> http://linkedin.com/in/olamy
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> -- Olivier Lamy Talend: http://coders.talend.com
>>>>>>> http://twitter.com/olamy | http://linkedin.com/in/olamy
>>>>>>>
>>>>>>
>>>>
>>>>
>>>>
>>
>>
>>
>> --
>> Olivier Lamy
>> Talend: http://coders.talend.com
>> http://twitter.com/olamy | http://linkedin.com/in/olamy
>>
>
Re: Additional Serializer and raw Buffer access
Posted by "Raffaele P. Guidi" <ra...@gmail.com>.
Negative part of ASF membership? You get together with a lot of geeky,
talented people with a fixation for software and open source. Oh wait but
this is actually nice! :-D
Il giorno 29/set/2012 19:05, "Olivier Lamy" <ol...@apache.org> ha scritto:
> 2012/9/29 Noctarius <me...@noctarius.com>:
> > Thanks Olivier for carify, I'll take a look in it tomorrow but
> > there's just one question left (for now ;)):
> > What is that vote for becoming a committer? What if the vote will
> > be negative?
> The vote is on private list (pmc list for privacy reasons and possible
> negative stuff being on public lists)
> > Until now I just used Apache stuff, was never interested in being
> > part of it so I guess it can be negative for any reason, can't it?
> I don't see why it could be negative but suspens .... :-)
> >
> > Am 29.09.2012 18:56, schrieb Olivier Lamy:
> >> 2012/9/29 Noctarius <me...@noctarius.com>:
> >>> Nope my real name is Christoph Engelbert, but Noctarius is
> >>> the all time nick :)
> >>>
> >>> Renaming the package should be no problem, is it
> >>> "org.apache.directmemory.lightning" or what would it be?
> >> fine for me
> >>> Then there needs to be a change in the license header as
> >>> Olivier mentioned, that means just remove the first sentence
> >>> or is there anything more to do (maybe it's easiest thing to
> >>> just copy the header from DM file ;))?
> >> yup use same header as DM
> >>>
> >>> The CLA is just a form to clarify that the source can be
> >>> contributed to the Apache Foundation?
> >> yup correct.
> >>>
> >>> The final step will be attaching the patch in form of a huge
> >>> diff file?
> >> yes
> >>>
> >>> And what is the way to apply for a membership? Never thought
> >>> about how to do that.
> >> Read here http://apache.org/foundation/how-it-works.html and
> >> here http://apache.org/foundation/getinvolved.html . And feel
> >> free to ask any questions :-)
> >>>
> >>> Chris
> >>>
> >>> Am 29.09.2012 18:23, schrieb Raffaele P. Guidi:
> >>>> OK, deal, at least for me ;-) I propose you rename the
> >>>> packages, produce a patch for this and the new serializer
> >>>> module (should be simple enough starting from an existing
> >>>> one) and, in the meanwhile, apply for ASF membership. Is
> >>>> IP clearance needed? I guess yes. After this we will come
> >>>> up with a formal vote regarding Noctarius (is this your
> >>>> real name?!) allowance in the project team.
> >>>>
> >>>> Good times are gonna come :-)
> >>>>
> >>>> Thanks, R Il giorno 29/set/2012 17:58, "Olivier Lamy"
> >>>> <ol...@apache.org> ha scritto:
> >>>>
> >>>>> 2012/9/29 Raffaele P. Guidi
> >>>>> <ra...@gmail.com>:
> >>>>>> Well we already have a NIO ready interface allowing
> >>>>>> direct access to DMs managed bytebuffers but I think
> >>>>>> this is just half way to what could be achieved
> >>>>>> optimally blending serialization and memory allocation
> >>>>>> together.
> >>>>>>
> >>>>>> Lightning as a module is of course a good idea and it
> >>>>>> could easily evolve as a subproject (for the more
> >>>>>> experienced asf members: is it a feasible way?).
> >>>>> Nothing prevent to have
> >>>>> http://svn.apache.org/repos/asf/directmemory/lightning/trunk
> >>>>>
> >>>>>
> > with a package like: o.a.d.lightning That will be a
> >>>>> subproject.
> >>>>>>
> >>>>>> Ciao, R Il giorno 29/set/2012 17:44, "Noctarius"
> >>>>>> <me...@noctarius.com> ha scritto:
> >>>>>>
> >>>>>>> Hey guys,
> >>>>>>>
> >>>>>>> ok there's no lightning binary available since
> >>>>>>> lightning wasn't ready yet for releasing.
> >>>>>>>
> >>>>>>> For being the only developer it would be no problem
> >>>>>>> to contribute the sourcebase for DirectMemory but I'm
> >>>>>>> not sure yet if it wouldn't be better to seperate it
> >>>>>>> to be available without using DirectMemory itself. I
> >>>>>>> started it as a serializer for cluster
> >>>>>>> synchronization, but it would be cool to contribute
> >>>>>>> lightning as a subproject to DirectMemory :-)
> >>>>>>>
> >>>>>>> About the second project I would love to see a
> >>>>>>> public available buffer API directly in DirectMemory
> >>>>>>> so that project would be nearly needless :-) The only
> >>>>>>> difference I think is the allocation strategy my
> >>>>>>> implementation is using against the one DirectMemory
> >>>>>>> has, but I'm pretty sure the allocation is extensible
> >>>>>>> ;-)
> >>>>>>>
> >>>>>>> Like I said, for both projects I'm the only dev so
> >>>>>>> there would be no IP problem. So if it's ok to you to
> >>>>>>> not include lightning directly in DM I would be glad
> >>>>>>> to contribute to the Apache Foundation :-)
> >>>>>>>
> >>>>>>> Cheers Chris
> >>>>>>>
> >>>>>>>
> >>>>>>> Am 29.09.2012 16:53, schrieb Raffaele P. Guidi:
> >>>>>>>> Ok so it's up to noctarius - your move! ;-)
> >>>>>>>> Regarding the new unsafe storage: it's an opt-in
> >>>>>>>> feature that can be set with the fluent API
> >>>>> (and
> >>>>>>>> soon through the conference file).
> >>>>>>>>
> >>>>>>>> Ciao, R Il giorno 29/set/2012 16:45, "Olivier
> >>>>>>>> Lamy" <ol...@apache.org> ha
> >>>>>>> scritto:
> >>>>>>>>
> >>>>>>>>> 2012/9/29 Raffaele P. Guidi
> >>>>>>>>> <ra...@gmail.com>:
> >>>>>>>>>>>> At least for the moment he can provide a
> >>>>>>>>>>>> patch to be integrated
> >>>>> in DM
> >>>>>>>>> :-)
> >>>>>>>>>>
> >>>>>>>>>> Sure, but as lightning is not in any public
> >>>>>>>>>> mvn repo should its
> >>>>> code be
> >>>>>>>>>> re-published in our svn? Or what?
> >>>>>>>>> @Apache we don't care about binaries, only
> >>>>>>>>> sources are important ! (a bit theorical for sure
> >>>>>>>>> but that's it :-) ). So if Noctarius was the only
> >>>>>>>>> guy who participate in lightning. He can just
> >>>>>>>>> provide a patch we could integrate as a new dm
> >>>>>>>>> module (note: the patch must not contains any
> >>>>>>>>> more copyright and all sources must have ASF
> >>>>>>>>> licenses).
> >>>>>>>>>
> >>>>>>>>> "Copyright 2012 the original author or authors."
> >>>>>>>>> must be removed. And BTW package must be changed
> >>>>>>>>> :-) (com.github is not acceptable
> >>>>> @asf
> >>>>>>>>> :-) )(@Noctarius are you working for github ? :-)
> >>>>>>>>> )
> >>>>>>>>>
> >>>>>>>>> And having him as a committer will be only a
> >>>>>>>>> matter of voting (we
> >>>>> have
> >>>>>>>>> a great chair who take cares of administrative
> >>>>>>>>> stuff :P )
> >>>>>>>>>
> >>>>>>>>> If some others have participated in the project,
> >>>>>>>>> we must pass tru an ip clearance mechanism
> >>>>>>>>> (http://incubator.apache.org/ip-clearance/index.html)
> >>>>>>>>>
> >>>>>>>>>
> > and all contributors to lightning must provide a cla.
> >>>>>>>>> (It it's the case I can help)
> >>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>> perso I'd like we avoid hard dependency on
> >>>>>>>>>>>> Unsafe as maybe some
> >>>>> use
> >>>>>>>>>> other jdks :-)
> >>>>>>>>>>
> >>>>>>>>>> Well, I believe Unsafe is supported by Sun
> >>>>>>>>>> JDK, OpenJDK, IBM JDK and JRockit - and I
> >>>>>>>>>> believe that it is more than enough. Also keep
> >>>>>>>>>> in
> >>>>> mind
> >>>>>>>>> that
> >>>>>>>>>> we already have an alternative Unsafe based
> >>>>>>>>>> memory storage - and
> >>>>>>> although
> >>>>>>>>>> it's not thoroughly tested for performance it
> >>>>>>>>>> dramaticaly simplifies
> >>>>>>>>> code,
> >>>>>>>>>> I have great expectations about it.
> >>>>>>>>> Me too :-). Yup definitely more simple and faster
> >>>>>>>>> ! But we must provide a switch off configuration
> >>>>>>>>> mechanism if some
> >>>>> users
> >>>>>>>>> don't want to use that (because Unsafe is just
> >>>>>>>>> "Unsafe" :-) ) And sorry I didn't have a look yet
> >>>>>>>>> at your changes with using Unsafe.
> >>>>>>>>>>
> >>>>>>>>>> Cheers, R
> >>>>>>>>>>
> >>>>>>>>>> On Sat, Sep 29, 2012 at 4:03 PM, Olivier Lamy
> >>>>>>>>>> <ol...@apache.org>
> >>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> 2012/9/29 Raffaele P. Guidi
> >>>>>>>>>>> <ra...@gmail.com>:
> >>>>>>>>>>>> What do you think about: 1) implementing a
> >>>>>>>>>>>> lightning serialization module 2) creating
> >>>>>>>>>>>> a serializer that directly works on a
> >>>>>>>>>>>> directmemory
> >>>>>>>>> provider
> >>>>>>>>>>>> ByteBuffer or (maybe better) Unsafe based
> >>>>>>>>>>>> Pointer?
> >>>>>>>>>>>
> >>>>>>>>>>> Sounds good (perso I'd like we avoid hard
> >>>>>>>>>>> dependency on Unsafe as maybe some use other
> >>>>>>>>>>> jdks :-) )
> >>>>>>>>>>>>
> >>>>>>>>>>>> Now I see lightning is apache licensed and
> >>>>>>>>>>>> this is fine but I
> >>>>> don't
> >>>>>>>>> think
> >>>>>>>>>>>> it is published in any public maven repo,
> >>>>>>>>>>>> am I right? We could
> >>>>> find a
> >>>>>>>>> way
> >>>>>>>>>>>> to deal with this; options vary from
> >>>>>>>>>>>> publishing lightning to the
> >>>>> free
> >>>>>>>>>>>> sonatype repo, joining the ASF (which is
> >>>>>>>>>>>> great anyhow!) and
> >>>>>>>>> republishing
> >>>>>>>>>>>> lightning code in DirectMemory becoming a
> >>>>>>>>>>>> commiter (which has to
> >>>>>>>>> undergo
> >>>>>>>>>>> a
> >>>>>>>>>>>> PMC vote).
> >>>>>>>>>>>
> >>>>>>>>>>> At least for the moment he can provide a
> >>>>>>>>>>> patch to be integrated in
> >>>>> DM
> >>>>>>>>> :-)
> >>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> I'd like to hear your and the team feelings
> >>>>>>>>>>>> on this.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks, Raffaele
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Sat, Sep 29, 2012 at 3:27 PM, Noctarius
> >>>>>>>>>>>> <me...@noctarius.com>
> >>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Hey Raffaele,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> that's quite similar to what I did at
> >>>>>>>>>>>>> work. We're developing
> >>>>> Flash
> >>>>>>>>>>>>> online games and using a customized AMF
> >>>>>>>>>>>>> serialization. To support async writing
> >>>>>>>>>>>>> of the clients event results I added
> >>>>>>>>>>>>> serialization
> >>>>> of
> >>>>>>>>>>>>> the components / entities to the players
> >>>>>>>>>>>>> zone calculation and
> >>>>> stored
> >>>>>>>>>>>>> the pre-serialized bytestream directly to
> >>>>>>>>>>>>> the off-heap (using direct-ring-cache
> >>>>>>>>>>>>> implementation). When the client
> >>>>>>>>>>>>> requests the results (using long-polling)
> >>>>>>>>>>>>> I just write the pre-serialized
> >>>>> data to
> >>>>>>>>>>>>> the right position to deserialize it by
> >>>>>>>>>>>>> standard ways on Flash
> >>>>> side.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> So yeah an seriliaztion to off-heap
> >>>>>>>>>>>>> algorithm would be fine. You
> >>>>> can
> >>>>>>>>>>>>> avoid using byte arrays and minimalize
> >>>>>>>>>>>>> GC.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Cheers Chris
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Am 29.09.2012 15:02, schrieb Raffaele P.
> >>>>>>>>>>>>> Guidi:
> >>>>>>>>>>>>>> Nice to hear back from you! Yes, I was
> >>>>>>>>>>>>>> thinking about creating a
> >>>>>>>>> new
> >>>>>>>>>>>>> memory
> >>>>>>>>>>>>>> storage implementation using Unsafe
> >>>>>>>>>>>>>> (and I did it, recently) and
> >>>>>>>>>>> also, as
> >>>>>>>>>>>>>> DirectMemory relies heavily on
> >>>>>>>>>>>>>> serialization (and supports many
> >>>>> of
> >>>>>>>>>>> them,
> >>>>>>>>>>>>>> protostuff, protobuf, msgpack and of
> >>>>>>>>>>>>>> course standard
> >>>>>>>>> serialization),
> >>>>>>>>>>>>> about
> >>>>>>>>>>>>>> creating a simple embedded serializer
> >>>>>>>>>>>>>> leveraging the same
> >>>>>>>>> techniques
> >>>>>>>>>>> you
> >>>>>>>>>>>>>> used (Unsafe and bytecode generation).
> >>>>>>>>>>>>>> The idea with embedding is avoiding to
> >>>>>>>>>>>>>> serialize in a byte array
> >>>>>>>>> and
> >>>>>>>>>>> then
> >>>>>>>>>>>>>> moving the byte array to off-heap
> >>>>>>>>>>>>>> memory (via Unsafe or
> >>>>>>>>> ByteBuffers),
> >>>>>>>>>>> and
> >>>>>>>>>>>>>> serializing directly into a "managed"
> >>>>>>>>>>>>>> off-heap buffer, thus
> >>>>> further
> >>>>>>>>>>>>>> optimizing heap utilization (less GC).
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> What do you think about it? Does it
> >>>>>>>>>>>>>> make any sense to you?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Ciao, R
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Sat, Sep 29, 2012 at 2:40 PM,
> >>>>>>>>>>>>>> Noctarius <me...@noctarius.com>
> >>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hey guys,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Raffaele found out about a project
> >>>>>>>>>>>>>>> of mine (Lightning) a few
> >>>>> weeks
> >>>>>>>>>>>>>>> ago. Lightning is a heavy Unsafe and
> >>>>>>>>>>>>>>> Bytecode generation using Serializer
> >>>>>>>>>>>>>>> implementation. He told me that he
> >>>>>>>>>>>>>>> was interested in adding something
> >>>>>>>>>>>>>>> similar to DirectMemory and I would
> >>>>>>>>>>>>>>> be glad to
> >>>>>>>>> help
> >>>>>>>>>>>>>>> out in this.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Another project I started a few days
> >>>>>>>>>>>>>>> ago, since it was needed
> >>>>> for
> >>>>>>>>>>>>>>> work is DirectRingCache. The name
> >>>>>>>>>>>>>>> not really meets to actual
> >>>>>>>>>>>>>>> implementation since it's not yet a
> >>>>>>>>>>>>>>> ring buffer using cache. I
> >>>>>>>>> used
> >>>>>>>>>>>>>>> this for a pre-serialization simple
> >>>>>>>>>>>>>>> bytestream cache with self-growing
> >>>>>>>>>>>>>>> buffers. It could be nice to have
> >>>>>>>>>>>>>>> DirectMemory
> >>>>> having
> >>>>>>>>>>>>>>> raw "buffers" to write to or to read
> >>>>>>>>>>>>>>> from.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Here are the links from the
> >>>>>>>>>>>>>>> projects:
> >>>>>>>>>>>>>>> https://github.com/noctarius/Lightning
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> > https://github.com/noctarius/direct-ring-cache
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>> It would be nice to help out since I really like the idea of
> >>>>>>>>>>>>>>> DirectMemory and since
> >>>>>>>>>>>>>>> direct-ring-cache is some kind of
> >>>>>>>>> reinventing
> >>>>>>>>>>>>>>> the wheel.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Cheers Noctarius (Chris)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> -- Olivier Lamy Talend:
> >>>>>>>>>>> http://coders.talend.com
> >>>>>>>>>>> http://twitter.com/olamy |
> >>>>>>>>>>> http://linkedin.com/in/olamy
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> -- Olivier Lamy Talend: http://coders.talend.com
> >>>>>>>>> http://twitter.com/olamy |
> >>>>>>>>> http://linkedin.com/in/olamy
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> -- Olivier Lamy Talend: http://coders.talend.com
> >>>>> http://twitter.com/olamy | http://linkedin.com/in/olamy
> >>>>>
> >>>>
> >>
> >>
> >>
>
>
>
> --
> Olivier Lamy
> Talend: http://coders.talend.com
> http://twitter.com/olamy | http://linkedin.com/in/olamy
>
Re: Additional Serializer and raw Buffer access
Posted by Olivier Lamy <ol...@apache.org>.
2012/9/29 Noctarius <me...@noctarius.com>:
> Thanks Olivier for carify, I'll take a look in it tomorrow but
> there's just one question left (for now ;)):
> What is that vote for becoming a committer? What if the vote will
> be negative?
The vote is on private list (pmc list for privacy reasons and possible
negative stuff being on public lists)
> Until now I just used Apache stuff, was never interested in being
> part of it so I guess it can be negative for any reason, can't it?
I don't see why it could be negative but suspens .... :-)
>
> Am 29.09.2012 18:56, schrieb Olivier Lamy:
>> 2012/9/29 Noctarius <me...@noctarius.com>:
>>> Nope my real name is Christoph Engelbert, but Noctarius is
>>> the all time nick :)
>>>
>>> Renaming the package should be no problem, is it
>>> "org.apache.directmemory.lightning" or what would it be?
>> fine for me
>>> Then there needs to be a change in the license header as
>>> Olivier mentioned, that means just remove the first sentence
>>> or is there anything more to do (maybe it's easiest thing to
>>> just copy the header from DM file ;))?
>> yup use same header as DM
>>>
>>> The CLA is just a form to clarify that the source can be
>>> contributed to the Apache Foundation?
>> yup correct.
>>>
>>> The final step will be attaching the patch in form of a huge
>>> diff file?
>> yes
>>>
>>> And what is the way to apply for a membership? Never thought
>>> about how to do that.
>> Read here http://apache.org/foundation/how-it-works.html and
>> here http://apache.org/foundation/getinvolved.html . And feel
>> free to ask any questions :-)
>>>
>>> Chris
>>>
>>> Am 29.09.2012 18:23, schrieb Raffaele P. Guidi:
>>>> OK, deal, at least for me ;-) I propose you rename the
>>>> packages, produce a patch for this and the new serializer
>>>> module (should be simple enough starting from an existing
>>>> one) and, in the meanwhile, apply for ASF membership. Is
>>>> IP clearance needed? I guess yes. After this we will come
>>>> up with a formal vote regarding Noctarius (is this your
>>>> real name?!) allowance in the project team.
>>>>
>>>> Good times are gonna come :-)
>>>>
>>>> Thanks, R Il giorno 29/set/2012 17:58, "Olivier Lamy"
>>>> <ol...@apache.org> ha scritto:
>>>>
>>>>> 2012/9/29 Raffaele P. Guidi
>>>>> <ra...@gmail.com>:
>>>>>> Well we already have a NIO ready interface allowing
>>>>>> direct access to DMs managed bytebuffers but I think
>>>>>> this is just half way to what could be achieved
>>>>>> optimally blending serialization and memory allocation
>>>>>> together.
>>>>>>
>>>>>> Lightning as a module is of course a good idea and it
>>>>>> could easily evolve as a subproject (for the more
>>>>>> experienced asf members: is it a feasible way?).
>>>>> Nothing prevent to have
>>>>> http://svn.apache.org/repos/asf/directmemory/lightning/trunk
>>>>>
>>>>>
> with a package like: o.a.d.lightning That will be a
>>>>> subproject.
>>>>>>
>>>>>> Ciao, R Il giorno 29/set/2012 17:44, "Noctarius"
>>>>>> <me...@noctarius.com> ha scritto:
>>>>>>
>>>>>>> Hey guys,
>>>>>>>
>>>>>>> ok there's no lightning binary available since
>>>>>>> lightning wasn't ready yet for releasing.
>>>>>>>
>>>>>>> For being the only developer it would be no problem
>>>>>>> to contribute the sourcebase for DirectMemory but I'm
>>>>>>> not sure yet if it wouldn't be better to seperate it
>>>>>>> to be available without using DirectMemory itself. I
>>>>>>> started it as a serializer for cluster
>>>>>>> synchronization, but it would be cool to contribute
>>>>>>> lightning as a subproject to DirectMemory :-)
>>>>>>>
>>>>>>> About the second project I would love to see a
>>>>>>> public available buffer API directly in DirectMemory
>>>>>>> so that project would be nearly needless :-) The only
>>>>>>> difference I think is the allocation strategy my
>>>>>>> implementation is using against the one DirectMemory
>>>>>>> has, but I'm pretty sure the allocation is extensible
>>>>>>> ;-)
>>>>>>>
>>>>>>> Like I said, for both projects I'm the only dev so
>>>>>>> there would be no IP problem. So if it's ok to you to
>>>>>>> not include lightning directly in DM I would be glad
>>>>>>> to contribute to the Apache Foundation :-)
>>>>>>>
>>>>>>> Cheers Chris
>>>>>>>
>>>>>>>
>>>>>>> Am 29.09.2012 16:53, schrieb Raffaele P. Guidi:
>>>>>>>> Ok so it's up to noctarius - your move! ;-)
>>>>>>>> Regarding the new unsafe storage: it's an opt-in
>>>>>>>> feature that can be set with the fluent API
>>>>> (and
>>>>>>>> soon through the conference file).
>>>>>>>>
>>>>>>>> Ciao, R Il giorno 29/set/2012 16:45, "Olivier
>>>>>>>> Lamy" <ol...@apache.org> ha
>>>>>>> scritto:
>>>>>>>>
>>>>>>>>> 2012/9/29 Raffaele P. Guidi
>>>>>>>>> <ra...@gmail.com>:
>>>>>>>>>>>> At least for the moment he can provide a
>>>>>>>>>>>> patch to be integrated
>>>>> in DM
>>>>>>>>> :-)
>>>>>>>>>>
>>>>>>>>>> Sure, but as lightning is not in any public
>>>>>>>>>> mvn repo should its
>>>>> code be
>>>>>>>>>> re-published in our svn? Or what?
>>>>>>>>> @Apache we don't care about binaries, only
>>>>>>>>> sources are important ! (a bit theorical for sure
>>>>>>>>> but that's it :-) ). So if Noctarius was the only
>>>>>>>>> guy who participate in lightning. He can just
>>>>>>>>> provide a patch we could integrate as a new dm
>>>>>>>>> module (note: the patch must not contains any
>>>>>>>>> more copyright and all sources must have ASF
>>>>>>>>> licenses).
>>>>>>>>>
>>>>>>>>> "Copyright 2012 the original author or authors."
>>>>>>>>> must be removed. And BTW package must be changed
>>>>>>>>> :-) (com.github is not acceptable
>>>>> @asf
>>>>>>>>> :-) )(@Noctarius are you working for github ? :-)
>>>>>>>>> )
>>>>>>>>>
>>>>>>>>> And having him as a committer will be only a
>>>>>>>>> matter of voting (we
>>>>> have
>>>>>>>>> a great chair who take cares of administrative
>>>>>>>>> stuff :P )
>>>>>>>>>
>>>>>>>>> If some others have participated in the project,
>>>>>>>>> we must pass tru an ip clearance mechanism
>>>>>>>>> (http://incubator.apache.org/ip-clearance/index.html)
>>>>>>>>>
>>>>>>>>>
> and all contributors to lightning must provide a cla.
>>>>>>>>> (It it's the case I can help)
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>> perso I'd like we avoid hard dependency on
>>>>>>>>>>>> Unsafe as maybe some
>>>>> use
>>>>>>>>>> other jdks :-)
>>>>>>>>>>
>>>>>>>>>> Well, I believe Unsafe is supported by Sun
>>>>>>>>>> JDK, OpenJDK, IBM JDK and JRockit - and I
>>>>>>>>>> believe that it is more than enough. Also keep
>>>>>>>>>> in
>>>>> mind
>>>>>>>>> that
>>>>>>>>>> we already have an alternative Unsafe based
>>>>>>>>>> memory storage - and
>>>>>>> although
>>>>>>>>>> it's not thoroughly tested for performance it
>>>>>>>>>> dramaticaly simplifies
>>>>>>>>> code,
>>>>>>>>>> I have great expectations about it.
>>>>>>>>> Me too :-). Yup definitely more simple and faster
>>>>>>>>> ! But we must provide a switch off configuration
>>>>>>>>> mechanism if some
>>>>> users
>>>>>>>>> don't want to use that (because Unsafe is just
>>>>>>>>> "Unsafe" :-) ) And sorry I didn't have a look yet
>>>>>>>>> at your changes with using Unsafe.
>>>>>>>>>>
>>>>>>>>>> Cheers, R
>>>>>>>>>>
>>>>>>>>>> On Sat, Sep 29, 2012 at 4:03 PM, Olivier Lamy
>>>>>>>>>> <ol...@apache.org>
>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> 2012/9/29 Raffaele P. Guidi
>>>>>>>>>>> <ra...@gmail.com>:
>>>>>>>>>>>> What do you think about: 1) implementing a
>>>>>>>>>>>> lightning serialization module 2) creating
>>>>>>>>>>>> a serializer that directly works on a
>>>>>>>>>>>> directmemory
>>>>>>>>> provider
>>>>>>>>>>>> ByteBuffer or (maybe better) Unsafe based
>>>>>>>>>>>> Pointer?
>>>>>>>>>>>
>>>>>>>>>>> Sounds good (perso I'd like we avoid hard
>>>>>>>>>>> dependency on Unsafe as maybe some use other
>>>>>>>>>>> jdks :-) )
>>>>>>>>>>>>
>>>>>>>>>>>> Now I see lightning is apache licensed and
>>>>>>>>>>>> this is fine but I
>>>>> don't
>>>>>>>>> think
>>>>>>>>>>>> it is published in any public maven repo,
>>>>>>>>>>>> am I right? We could
>>>>> find a
>>>>>>>>> way
>>>>>>>>>>>> to deal with this; options vary from
>>>>>>>>>>>> publishing lightning to the
>>>>> free
>>>>>>>>>>>> sonatype repo, joining the ASF (which is
>>>>>>>>>>>> great anyhow!) and
>>>>>>>>> republishing
>>>>>>>>>>>> lightning code in DirectMemory becoming a
>>>>>>>>>>>> commiter (which has to
>>>>>>>>> undergo
>>>>>>>>>>> a
>>>>>>>>>>>> PMC vote).
>>>>>>>>>>>
>>>>>>>>>>> At least for the moment he can provide a
>>>>>>>>>>> patch to be integrated in
>>>>> DM
>>>>>>>>> :-)
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I'd like to hear your and the team feelings
>>>>>>>>>>>> on this.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks, Raffaele
>>>>>>>>>>>>
>>>>>>>>>>>> On Sat, Sep 29, 2012 at 3:27 PM, Noctarius
>>>>>>>>>>>> <me...@noctarius.com>
>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hey Raffaele,
>>>>>>>>>>>>>
>>>>>>>>>>>>> that's quite similar to what I did at
>>>>>>>>>>>>> work. We're developing
>>>>> Flash
>>>>>>>>>>>>> online games and using a customized AMF
>>>>>>>>>>>>> serialization. To support async writing
>>>>>>>>>>>>> of the clients event results I added
>>>>>>>>>>>>> serialization
>>>>> of
>>>>>>>>>>>>> the components / entities to the players
>>>>>>>>>>>>> zone calculation and
>>>>> stored
>>>>>>>>>>>>> the pre-serialized bytestream directly to
>>>>>>>>>>>>> the off-heap (using direct-ring-cache
>>>>>>>>>>>>> implementation). When the client
>>>>>>>>>>>>> requests the results (using long-polling)
>>>>>>>>>>>>> I just write the pre-serialized
>>>>> data to
>>>>>>>>>>>>> the right position to deserialize it by
>>>>>>>>>>>>> standard ways on Flash
>>>>> side.
>>>>>>>>>>>>>
>>>>>>>>>>>>> So yeah an seriliaztion to off-heap
>>>>>>>>>>>>> algorithm would be fine. You
>>>>> can
>>>>>>>>>>>>> avoid using byte arrays and minimalize
>>>>>>>>>>>>> GC.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers Chris
>>>>>>>>>>>>>
>>>>>>>>>>>>> Am 29.09.2012 15:02, schrieb Raffaele P.
>>>>>>>>>>>>> Guidi:
>>>>>>>>>>>>>> Nice to hear back from you! Yes, I was
>>>>>>>>>>>>>> thinking about creating a
>>>>>>>>> new
>>>>>>>>>>>>> memory
>>>>>>>>>>>>>> storage implementation using Unsafe
>>>>>>>>>>>>>> (and I did it, recently) and
>>>>>>>>>>> also, as
>>>>>>>>>>>>>> DirectMemory relies heavily on
>>>>>>>>>>>>>> serialization (and supports many
>>>>> of
>>>>>>>>>>> them,
>>>>>>>>>>>>>> protostuff, protobuf, msgpack and of
>>>>>>>>>>>>>> course standard
>>>>>>>>> serialization),
>>>>>>>>>>>>> about
>>>>>>>>>>>>>> creating a simple embedded serializer
>>>>>>>>>>>>>> leveraging the same
>>>>>>>>> techniques
>>>>>>>>>>> you
>>>>>>>>>>>>>> used (Unsafe and bytecode generation).
>>>>>>>>>>>>>> The idea with embedding is avoiding to
>>>>>>>>>>>>>> serialize in a byte array
>>>>>>>>> and
>>>>>>>>>>> then
>>>>>>>>>>>>>> moving the byte array to off-heap
>>>>>>>>>>>>>> memory (via Unsafe or
>>>>>>>>> ByteBuffers),
>>>>>>>>>>> and
>>>>>>>>>>>>>> serializing directly into a "managed"
>>>>>>>>>>>>>> off-heap buffer, thus
>>>>> further
>>>>>>>>>>>>>> optimizing heap utilization (less GC).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> What do you think about it? Does it
>>>>>>>>>>>>>> make any sense to you?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Ciao, R
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Sat, Sep 29, 2012 at 2:40 PM,
>>>>>>>>>>>>>> Noctarius <me...@noctarius.com>
>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hey guys,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Raffaele found out about a project
>>>>>>>>>>>>>>> of mine (Lightning) a few
>>>>> weeks
>>>>>>>>>>>>>>> ago. Lightning is a heavy Unsafe and
>>>>>>>>>>>>>>> Bytecode generation using Serializer
>>>>>>>>>>>>>>> implementation. He told me that he
>>>>>>>>>>>>>>> was interested in adding something
>>>>>>>>>>>>>>> similar to DirectMemory and I would
>>>>>>>>>>>>>>> be glad to
>>>>>>>>> help
>>>>>>>>>>>>>>> out in this.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Another project I started a few days
>>>>>>>>>>>>>>> ago, since it was needed
>>>>> for
>>>>>>>>>>>>>>> work is DirectRingCache. The name
>>>>>>>>>>>>>>> not really meets to actual
>>>>>>>>>>>>>>> implementation since it's not yet a
>>>>>>>>>>>>>>> ring buffer using cache. I
>>>>>>>>> used
>>>>>>>>>>>>>>> this for a pre-serialization simple
>>>>>>>>>>>>>>> bytestream cache with self-growing
>>>>>>>>>>>>>>> buffers. It could be nice to have
>>>>>>>>>>>>>>> DirectMemory
>>>>> having
>>>>>>>>>>>>>>> raw "buffers" to write to or to read
>>>>>>>>>>>>>>> from.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Here are the links from the
>>>>>>>>>>>>>>> projects:
>>>>>>>>>>>>>>> https://github.com/noctarius/Lightning
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
> https://github.com/noctarius/direct-ring-cache
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>> It would be nice to help out since I really like the idea of
>>>>>>>>>>>>>>> DirectMemory and since
>>>>>>>>>>>>>>> direct-ring-cache is some kind of
>>>>>>>>> reinventing
>>>>>>>>>>>>>>> the wheel.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Cheers Noctarius (Chris)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> -- Olivier Lamy Talend:
>>>>>>>>>>> http://coders.talend.com
>>>>>>>>>>> http://twitter.com/olamy |
>>>>>>>>>>> http://linkedin.com/in/olamy
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -- Olivier Lamy Talend: http://coders.talend.com
>>>>>>>>> http://twitter.com/olamy |
>>>>>>>>> http://linkedin.com/in/olamy
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> -- Olivier Lamy Talend: http://coders.talend.com
>>>>> http://twitter.com/olamy | http://linkedin.com/in/olamy
>>>>>
>>>>
>>
>>
>>
--
Olivier Lamy
Talend: http://coders.talend.com
http://twitter.com/olamy | http://linkedin.com/in/olamy
Re: Additional Serializer and raw Buffer access
Posted by Olivier Lamy <ol...@apache.org>.
2012/9/29 Noctarius <me...@noctarius.com>:
> PS: Links below won't work for me "connection cancelled". Is there
> a problem with the servers?
I noticed too network issues for people in eu :-(
>
> Am 29.09.2012 19:00, schrieb Noctarius:
>> Thanks Olivier for carify, I'll take a look in it tomorrow but
>> there's just one question left (for now ;)): What is that vote
>> for becoming a committer? What if the vote will be negative?
>> Until now I just used Apache stuff, was never interested in
>> being part of it so I guess it can be negative for any reason,
>> can't it?
>>
>> Am 29.09.2012 18:56, schrieb Olivier Lamy:
>>> 2012/9/29 Noctarius <me...@noctarius.com>:
>>>> Nope my real name is Christoph Engelbert, but Noctarius is
>>>> the all time nick :)
>>>>
>>>> Renaming the package should be no problem, is it
>>>> "org.apache.directmemory.lightning" or what would it be?
>>> fine for me
>>>> Then there needs to be a change in the license header as
>>>> Olivier mentioned, that means just remove the first
>>>> sentence or is there anything more to do (maybe it's
>>>> easiest thing to just copy the header from DM file ;))?
>>> yup use same header as DM
>>>>
>>>> The CLA is just a form to clarify that the source can be
>>>> contributed to the Apache Foundation?
>>> yup correct.
>>>>
>>>> The final step will be attaching the patch in form of a
>>>> huge diff file?
>>> yes
>>>>
>>>> And what is the way to apply for a membership? Never
>>>> thought about how to do that.
>>> Read here http://apache.org/foundation/how-it-works.html and
>>> here http://apache.org/foundation/getinvolved.html . And
>>> feel free to ask any questions :-)
>>>>
>>>> Chris
>>>>
>>>> Am 29.09.2012 18:23, schrieb Raffaele P. Guidi:
>>>>> OK, deal, at least for me ;-) I propose you rename the
>>>>> packages, produce a patch for this and the new serializer
>>>>> module (should be simple enough starting from an
>>>>> existing one) and, in the meanwhile, apply for ASF
>>>>> membership. Is IP clearance needed? I guess yes. After
>>>>> this we will come up with a formal vote regarding
>>>>> Noctarius (is this your real name?!) allowance in the
>>>>> project team.
>>>>>
>>>>> Good times are gonna come :-)
>>>>>
>>>>> Thanks, R Il giorno 29/set/2012 17:58, "Olivier Lamy"
>>>>> <ol...@apache.org> ha scritto:
>>>>>
>>>>>> 2012/9/29 Raffaele P. Guidi
>>>>>> <ra...@gmail.com>:
>>>>>>> Well we already have a NIO ready interface allowing
>>>>>>> direct access to DMs managed bytebuffers but I think
>>>>>>> this is just half way to what could be achieved
>>>>>>> optimally blending serialization and memory
>>>>>>> allocation together.
>>>>>>>
>>>>>>> Lightning as a module is of course a good idea and
>>>>>>> it could easily evolve as a subproject (for the more
>>>>>>> experienced asf members: is it a feasible way?).
>>>>>> Nothing prevent to have
>>>>>> http://svn.apache.org/repos/asf/directmemory/lightning/trunk
>>>>>>
>>>>>>
>>
>>>>>>
> with a package like: o.a.d.lightning That will be a
>>>>>> subproject.
>>>>>>>
>>>>>>> Ciao, R Il giorno 29/set/2012 17:44, "Noctarius"
>>>>>>> <me...@noctarius.com> ha scritto:
>>>>>>>
>>>>>>>> Hey guys,
>>>>>>>>
>>>>>>>> ok there's no lightning binary available since
>>>>>>>> lightning wasn't ready yet for releasing.
>>>>>>>>
>>>>>>>> For being the only developer it would be no
>>>>>>>> problem to contribute the sourcebase for
>>>>>>>> DirectMemory but I'm not sure yet if it wouldn't be
>>>>>>>> better to seperate it to be available without using
>>>>>>>> DirectMemory itself. I started it as a serializer
>>>>>>>> for cluster synchronization, but it would be cool
>>>>>>>> to contribute lightning as a subproject to
>>>>>>>> DirectMemory :-)
>>>>>>>>
>>>>>>>> About the second project I would love to see a
>>>>>>>> public available buffer API directly in
>>>>>>>> DirectMemory so that project would be nearly
>>>>>>>> needless :-) The only difference I think is the
>>>>>>>> allocation strategy my implementation is using
>>>>>>>> against the one DirectMemory has, but I'm pretty
>>>>>>>> sure the allocation is extensible ;-)
>>>>>>>>
>>>>>>>> Like I said, for both projects I'm the only dev so
>>>>>>>> there would be no IP problem. So if it's ok to you
>>>>>>>> to not include lightning directly in DM I would be
>>>>>>>> glad to contribute to the Apache Foundation :-)
>>>>>>>>
>>>>>>>> Cheers Chris
>>>>>>>>
>>>>>>>>
>>>>>>>> Am 29.09.2012 16:53, schrieb Raffaele P. Guidi:
>>>>>>>>> Ok so it's up to noctarius - your move! ;-)
>>>>>>>>> Regarding the new unsafe storage: it's an opt-in
>>>>>>>>> feature that can be set with the fluent API
>>>>>> (and
>>>>>>>>> soon through the conference file).
>>>>>>>>>
>>>>>>>>> Ciao, R Il giorno 29/set/2012 16:45, "Olivier
>>>>>>>>> Lamy" <ol...@apache.org> ha
>>>>>>>> scritto:
>>>>>>>>>
>>>>>>>>>> 2012/9/29 Raffaele P. Guidi
>>>>>>>>>> <ra...@gmail.com>:
>>>>>>>>>>>>> At least for the moment he can provide a
>>>>>>>>>>>>> patch to be integrated
>>>>>> in DM
>>>>>>>>>> :-)
>>>>>>>>>>>
>>>>>>>>>>> Sure, but as lightning is not in any public
>>>>>>>>>>> mvn repo should its
>>>>>> code be
>>>>>>>>>>> re-published in our svn? Or what?
>>>>>>>>>> @Apache we don't care about binaries, only
>>>>>>>>>> sources are important ! (a bit theorical for
>>>>>>>>>> sure but that's it :-) ). So if Noctarius was
>>>>>>>>>> the only guy who participate in lightning. He
>>>>>>>>>> can just provide a patch we could integrate as
>>>>>>>>>> a new dm module (note: the patch must not
>>>>>>>>>> contains any more copyright and all sources
>>>>>>>>>> must have ASF licenses).
>>>>>>>>>>
>>>>>>>>>> "Copyright 2012 the original author or
>>>>>>>>>> authors." must be removed. And BTW package must
>>>>>>>>>> be changed :-) (com.github is not acceptable
>>>>>> @asf
>>>>>>>>>> :-) )(@Noctarius are you working for github ?
>>>>>>>>>> :-) )
>>>>>>>>>>
>>>>>>>>>> And having him as a committer will be only a
>>>>>>>>>> matter of voting (we
>>>>>> have
>>>>>>>>>> a great chair who take cares of administrative
>>>>>>>>>> stuff :P )
>>>>>>>>>>
>>>>>>>>>> If some others have participated in the
>>>>>>>>>> project, we must pass tru an ip clearance
>>>>>>>>>> mechanism
>>>>>>>>>> (http://incubator.apache.org/ip-clearance/index.html)
>>>>>>>>>>
>>>>>>>>>>
>>
>>>>>>>>>>
> and all contributors to lightning must provide a cla.
>>>>>>>>>> (It it's the case I can help)
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>> perso I'd like we avoid hard dependency
>>>>>>>>>>>>> on Unsafe as maybe some
>>>>>> use
>>>>>>>>>>> other jdks :-)
>>>>>>>>>>>
>>>>>>>>>>> Well, I believe Unsafe is supported by Sun
>>>>>>>>>>> JDK, OpenJDK, IBM JDK and JRockit - and I
>>>>>>>>>>> believe that it is more than enough. Also
>>>>>>>>>>> keep in
>>>>>> mind
>>>>>>>>>> that
>>>>>>>>>>> we already have an alternative Unsafe based
>>>>>>>>>>> memory storage - and
>>>>>>>> although
>>>>>>>>>>> it's not thoroughly tested for performance it
>>>>>>>>>>> dramaticaly simplifies
>>>>>>>>>> code,
>>>>>>>>>>> I have great expectations about it.
>>>>>>>>>> Me too :-). Yup definitely more simple and
>>>>>>>>>> faster ! But we must provide a switch off
>>>>>>>>>> configuration mechanism if some
>>>>>> users
>>>>>>>>>> don't want to use that (because Unsafe is just
>>>>>>>>>> "Unsafe" :-) ) And sorry I didn't have a look
>>>>>>>>>> yet at your changes with using Unsafe.
>>>>>>>>>>>
>>>>>>>>>>> Cheers, R
>>>>>>>>>>>
>>>>>>>>>>> On Sat, Sep 29, 2012 at 4:03 PM, Olivier Lamy
>>>>>>>>>>> <ol...@apache.org>
>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> 2012/9/29 Raffaele P. Guidi
>>>>>>>>>>>> <ra...@gmail.com>:
>>>>>>>>>>>>> What do you think about: 1) implementing
>>>>>>>>>>>>> a lightning serialization module 2)
>>>>>>>>>>>>> creating a serializer that directly works
>>>>>>>>>>>>> on a directmemory
>>>>>>>>>> provider
>>>>>>>>>>>>> ByteBuffer or (maybe better) Unsafe based
>>>>>>>>>>>>> Pointer?
>>>>>>>>>>>>
>>>>>>>>>>>> Sounds good (perso I'd like we avoid hard
>>>>>>>>>>>> dependency on Unsafe as maybe some use
>>>>>>>>>>>> other jdks :-) )
>>>>>>>>>>>>>
>>>>>>>>>>>>> Now I see lightning is apache licensed
>>>>>>>>>>>>> and this is fine but I
>>>>>> don't
>>>>>>>>>> think
>>>>>>>>>>>>> it is published in any public maven
>>>>>>>>>>>>> repo, am I right? We could
>>>>>> find a
>>>>>>>>>> way
>>>>>>>>>>>>> to deal with this; options vary from
>>>>>>>>>>>>> publishing lightning to the
>>>>>> free
>>>>>>>>>>>>> sonatype repo, joining the ASF (which
>>>>>>>>>>>>> is great anyhow!) and
>>>>>>>>>> republishing
>>>>>>>>>>>>> lightning code in DirectMemory becoming a
>>>>>>>>>>>>> commiter (which has to
>>>>>>>>>> undergo
>>>>>>>>>>>> a
>>>>>>>>>>>>> PMC vote).
>>>>>>>>>>>>
>>>>>>>>>>>> At least for the moment he can provide a
>>>>>>>>>>>> patch to be integrated in
>>>>>> DM
>>>>>>>>>> :-)
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'd like to hear your and the team
>>>>>>>>>>>>> feelings on this.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks, Raffaele
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Sat, Sep 29, 2012 at 3:27 PM,
>>>>>>>>>>>>> Noctarius <me...@noctarius.com>
>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hey Raffaele,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> that's quite similar to what I did at
>>>>>>>>>>>>>> work. We're developing
>>>>>> Flash
>>>>>>>>>>>>>> online games and using a customized AMF
>>>>>>>>>>>>>> serialization. To support async
>>>>>>>>>>>>>> writing of the clients event results I
>>>>>>>>>>>>>> added serialization
>>>>>> of
>>>>>>>>>>>>>> the components / entities to the
>>>>>>>>>>>>>> players zone calculation and
>>>>>> stored
>>>>>>>>>>>>>> the pre-serialized bytestream directly
>>>>>>>>>>>>>> to the off-heap (using
>>>>>>>>>>>>>> direct-ring-cache implementation). When
>>>>>>>>>>>>>> the client requests the results (using
>>>>>>>>>>>>>> long-polling) I just write the
>>>>>>>>>>>>>> pre-serialized
>>>>>> data to
>>>>>>>>>>>>>> the right position to deserialize it by
>>>>>>>>>>>>>> standard ways on Flash
>>>>>> side.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> So yeah an seriliaztion to off-heap
>>>>>>>>>>>>>> algorithm would be fine. You
>>>>>> can
>>>>>>>>>>>>>> avoid using byte arrays and minimalize
>>>>>>>>>>>>>> GC.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Cheers Chris
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Am 29.09.2012 15:02, schrieb Raffaele
>>>>>>>>>>>>>> P. Guidi:
>>>>>>>>>>>>>>> Nice to hear back from you! Yes, I
>>>>>>>>>>>>>>> was thinking about creating a
>>>>>>>>>> new
>>>>>>>>>>>>>> memory
>>>>>>>>>>>>>>> storage implementation using Unsafe
>>>>>>>>>>>>>>> (and I did it, recently) and
>>>>>>>>>>>> also, as
>>>>>>>>>>>>>>> DirectMemory relies heavily on
>>>>>>>>>>>>>>> serialization (and supports many
>>>>>> of
>>>>>>>>>>>> them,
>>>>>>>>>>>>>>> protostuff, protobuf, msgpack and of
>>>>>>>>>>>>>>> course standard
>>>>>>>>>> serialization),
>>>>>>>>>>>>>> about
>>>>>>>>>>>>>>> creating a simple embedded serializer
>>>>>>>>>>>>>>> leveraging the same
>>>>>>>>>> techniques
>>>>>>>>>>>> you
>>>>>>>>>>>>>>> used (Unsafe and bytecode
>>>>>>>>>>>>>>> generation). The idea with embedding
>>>>>>>>>>>>>>> is avoiding to serialize in a byte
>>>>>>>>>>>>>>> array
>>>>>>>>>> and
>>>>>>>>>>>> then
>>>>>>>>>>>>>>> moving the byte array to off-heap
>>>>>>>>>>>>>>> memory (via Unsafe or
>>>>>>>>>> ByteBuffers),
>>>>>>>>>>>> and
>>>>>>>>>>>>>>> serializing directly into a "managed"
>>>>>>>>>>>>>>> off-heap buffer, thus
>>>>>> further
>>>>>>>>>>>>>>> optimizing heap utilization (less
>>>>>>>>>>>>>>> GC).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> What do you think about it? Does it
>>>>>>>>>>>>>>> make any sense to you?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Ciao, R
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Sat, Sep 29, 2012 at 2:40 PM,
>>>>>>>>>>>>>>> Noctarius <me...@noctarius.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hey guys,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Raffaele found out about a project
>>>>>>>>>>>>>>>> of mine (Lightning) a few
>>>>>> weeks
>>>>>>>>>>>>>>>> ago. Lightning is a heavy Unsafe
>>>>>>>>>>>>>>>> and Bytecode generation using
>>>>>>>>>>>>>>>> Serializer implementation. He told
>>>>>>>>>>>>>>>> me that he was interested in adding
>>>>>>>>>>>>>>>> something similar to DirectMemory
>>>>>>>>>>>>>>>> and I would be glad to
>>>>>>>>>> help
>>>>>>>>>>>>>>>> out in this.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Another project I started a few
>>>>>>>>>>>>>>>> days ago, since it was needed
>>>>>> for
>>>>>>>>>>>>>>>> work is DirectRingCache. The name
>>>>>>>>>>>>>>>> not really meets to actual
>>>>>>>>>>>>>>>> implementation since it's not yet
>>>>>>>>>>>>>>>> a ring buffer using cache. I
>>>>>>>>>> used
>>>>>>>>>>>>>>>> this for a pre-serialization simple
>>>>>>>>>>>>>>>> bytestream cache with self-growing
>>>>>>>>>>>>>>>> buffers. It could be nice to have
>>>>>>>>>>>>>>>> DirectMemory
>>>>>> having
>>>>>>>>>>>>>>>> raw "buffers" to write to or to
>>>>>>>>>>>>>>>> read from.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Here are the links from the
>>>>>>>>>>>>>>>> projects:
>>>>>>>>>>>>>>>> https://github.com/noctarius/Lightning
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>
>>>>>>>>>>>>>>>>
> https://github.com/noctarius/direct-ring-cache
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>> It would be nice to help out since I really like the idea
>>>> of
>>>>>>>>>>>>>>>> DirectMemory and since
>>>>>>>>>>>>>>>> direct-ring-cache is some kind of
>>>>>>>>>> reinventing
>>>>>>>>>>>>>>>> the wheel.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Cheers Noctarius (Chris)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> -- Olivier Lamy Talend:
>>>>>>>>>>>> http://coders.talend.com
>>>>>>>>>>>> http://twitter.com/olamy |
>>>>>>>>>>>> http://linkedin.com/in/olamy
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> -- Olivier Lamy Talend:
>>>>>>>>>> http://coders.talend.com
>>>>>>>>>> http://twitter.com/olamy |
>>>>>>>>>> http://linkedin.com/in/olamy
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> -- Olivier Lamy Talend: http://coders.talend.com
>>>>>> http://twitter.com/olamy |
>>>>>> http://linkedin.com/in/olamy
>>>>>>
>>>>>
>>>
>>>
>>>
>
--
Olivier Lamy
Talend: http://coders.talend.com
http://twitter.com/olamy | http://linkedin.com/in/olamy
Re: Additional Serializer and raw Buffer access
Posted by Noctarius <me...@noctarius.com>.
PS: Links below won't work for me "connection cancelled". Is there
a problem with the servers?
Am 29.09.2012 19:00, schrieb Noctarius:
> Thanks Olivier for carify, I'll take a look in it tomorrow but
> there's just one question left (for now ;)): What is that vote
> for becoming a committer? What if the vote will be negative?
> Until now I just used Apache stuff, was never interested in
> being part of it so I guess it can be negative for any reason,
> can't it?
>
> Am 29.09.2012 18:56, schrieb Olivier Lamy:
>> 2012/9/29 Noctarius <me...@noctarius.com>:
>>> Nope my real name is Christoph Engelbert, but Noctarius is
>>> the all time nick :)
>>>
>>> Renaming the package should be no problem, is it
>>> "org.apache.directmemory.lightning" or what would it be?
>> fine for me
>>> Then there needs to be a change in the license header as
>>> Olivier mentioned, that means just remove the first
>>> sentence or is there anything more to do (maybe it's
>>> easiest thing to just copy the header from DM file ;))?
>> yup use same header as DM
>>>
>>> The CLA is just a form to clarify that the source can be
>>> contributed to the Apache Foundation?
>> yup correct.
>>>
>>> The final step will be attaching the patch in form of a
>>> huge diff file?
>> yes
>>>
>>> And what is the way to apply for a membership? Never
>>> thought about how to do that.
>> Read here http://apache.org/foundation/how-it-works.html and
>> here http://apache.org/foundation/getinvolved.html . And
>> feel free to ask any questions :-)
>>>
>>> Chris
>>>
>>> Am 29.09.2012 18:23, schrieb Raffaele P. Guidi:
>>>> OK, deal, at least for me ;-) I propose you rename the
>>>> packages, produce a patch for this and the new serializer
>>>> module (should be simple enough starting from an
>>>> existing one) and, in the meanwhile, apply for ASF
>>>> membership. Is IP clearance needed? I guess yes. After
>>>> this we will come up with a formal vote regarding
>>>> Noctarius (is this your real name?!) allowance in the
>>>> project team.
>>>>
>>>> Good times are gonna come :-)
>>>>
>>>> Thanks, R Il giorno 29/set/2012 17:58, "Olivier Lamy"
>>>> <ol...@apache.org> ha scritto:
>>>>
>>>>> 2012/9/29 Raffaele P. Guidi
>>>>> <ra...@gmail.com>:
>>>>>> Well we already have a NIO ready interface allowing
>>>>>> direct access to DMs managed bytebuffers but I think
>>>>>> this is just half way to what could be achieved
>>>>>> optimally blending serialization and memory
>>>>>> allocation together.
>>>>>>
>>>>>> Lightning as a module is of course a good idea and
>>>>>> it could easily evolve as a subproject (for the more
>>>>>> experienced asf members: is it a feasible way?).
>>>>> Nothing prevent to have
>>>>> http://svn.apache.org/repos/asf/directmemory/lightning/trunk
>>>>>
>>>>>
>
>>>>>
with a package like: o.a.d.lightning That will be a
>>>>> subproject.
>>>>>>
>>>>>> Ciao, R Il giorno 29/set/2012 17:44, "Noctarius"
>>>>>> <me...@noctarius.com> ha scritto:
>>>>>>
>>>>>>> Hey guys,
>>>>>>>
>>>>>>> ok there's no lightning binary available since
>>>>>>> lightning wasn't ready yet for releasing.
>>>>>>>
>>>>>>> For being the only developer it would be no
>>>>>>> problem to contribute the sourcebase for
>>>>>>> DirectMemory but I'm not sure yet if it wouldn't be
>>>>>>> better to seperate it to be available without using
>>>>>>> DirectMemory itself. I started it as a serializer
>>>>>>> for cluster synchronization, but it would be cool
>>>>>>> to contribute lightning as a subproject to
>>>>>>> DirectMemory :-)
>>>>>>>
>>>>>>> About the second project I would love to see a
>>>>>>> public available buffer API directly in
>>>>>>> DirectMemory so that project would be nearly
>>>>>>> needless :-) The only difference I think is the
>>>>>>> allocation strategy my implementation is using
>>>>>>> against the one DirectMemory has, but I'm pretty
>>>>>>> sure the allocation is extensible ;-)
>>>>>>>
>>>>>>> Like I said, for both projects I'm the only dev so
>>>>>>> there would be no IP problem. So if it's ok to you
>>>>>>> to not include lightning directly in DM I would be
>>>>>>> glad to contribute to the Apache Foundation :-)
>>>>>>>
>>>>>>> Cheers Chris
>>>>>>>
>>>>>>>
>>>>>>> Am 29.09.2012 16:53, schrieb Raffaele P. Guidi:
>>>>>>>> Ok so it's up to noctarius - your move! ;-)
>>>>>>>> Regarding the new unsafe storage: it's an opt-in
>>>>>>>> feature that can be set with the fluent API
>>>>> (and
>>>>>>>> soon through the conference file).
>>>>>>>>
>>>>>>>> Ciao, R Il giorno 29/set/2012 16:45, "Olivier
>>>>>>>> Lamy" <ol...@apache.org> ha
>>>>>>> scritto:
>>>>>>>>
>>>>>>>>> 2012/9/29 Raffaele P. Guidi
>>>>>>>>> <ra...@gmail.com>:
>>>>>>>>>>>> At least for the moment he can provide a
>>>>>>>>>>>> patch to be integrated
>>>>> in DM
>>>>>>>>> :-)
>>>>>>>>>>
>>>>>>>>>> Sure, but as lightning is not in any public
>>>>>>>>>> mvn repo should its
>>>>> code be
>>>>>>>>>> re-published in our svn? Or what?
>>>>>>>>> @Apache we don't care about binaries, only
>>>>>>>>> sources are important ! (a bit theorical for
>>>>>>>>> sure but that's it :-) ). So if Noctarius was
>>>>>>>>> the only guy who participate in lightning. He
>>>>>>>>> can just provide a patch we could integrate as
>>>>>>>>> a new dm module (note: the patch must not
>>>>>>>>> contains any more copyright and all sources
>>>>>>>>> must have ASF licenses).
>>>>>>>>>
>>>>>>>>> "Copyright 2012 the original author or
>>>>>>>>> authors." must be removed. And BTW package must
>>>>>>>>> be changed :-) (com.github is not acceptable
>>>>> @asf
>>>>>>>>> :-) )(@Noctarius are you working for github ?
>>>>>>>>> :-) )
>>>>>>>>>
>>>>>>>>> And having him as a committer will be only a
>>>>>>>>> matter of voting (we
>>>>> have
>>>>>>>>> a great chair who take cares of administrative
>>>>>>>>> stuff :P )
>>>>>>>>>
>>>>>>>>> If some others have participated in the
>>>>>>>>> project, we must pass tru an ip clearance
>>>>>>>>> mechanism
>>>>>>>>> (http://incubator.apache.org/ip-clearance/index.html)
>>>>>>>>>
>>>>>>>>>
>
>>>>>>>>>
and all contributors to lightning must provide a cla.
>>>>>>>>> (It it's the case I can help)
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>> perso I'd like we avoid hard dependency
>>>>>>>>>>>> on Unsafe as maybe some
>>>>> use
>>>>>>>>>> other jdks :-)
>>>>>>>>>>
>>>>>>>>>> Well, I believe Unsafe is supported by Sun
>>>>>>>>>> JDK, OpenJDK, IBM JDK and JRockit - and I
>>>>>>>>>> believe that it is more than enough. Also
>>>>>>>>>> keep in
>>>>> mind
>>>>>>>>> that
>>>>>>>>>> we already have an alternative Unsafe based
>>>>>>>>>> memory storage - and
>>>>>>> although
>>>>>>>>>> it's not thoroughly tested for performance it
>>>>>>>>>> dramaticaly simplifies
>>>>>>>>> code,
>>>>>>>>>> I have great expectations about it.
>>>>>>>>> Me too :-). Yup definitely more simple and
>>>>>>>>> faster ! But we must provide a switch off
>>>>>>>>> configuration mechanism if some
>>>>> users
>>>>>>>>> don't want to use that (because Unsafe is just
>>>>>>>>> "Unsafe" :-) ) And sorry I didn't have a look
>>>>>>>>> yet at your changes with using Unsafe.
>>>>>>>>>>
>>>>>>>>>> Cheers, R
>>>>>>>>>>
>>>>>>>>>> On Sat, Sep 29, 2012 at 4:03 PM, Olivier Lamy
>>>>>>>>>> <ol...@apache.org>
>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> 2012/9/29 Raffaele P. Guidi
>>>>>>>>>>> <ra...@gmail.com>:
>>>>>>>>>>>> What do you think about: 1) implementing
>>>>>>>>>>>> a lightning serialization module 2)
>>>>>>>>>>>> creating a serializer that directly works
>>>>>>>>>>>> on a directmemory
>>>>>>>>> provider
>>>>>>>>>>>> ByteBuffer or (maybe better) Unsafe based
>>>>>>>>>>>> Pointer?
>>>>>>>>>>>
>>>>>>>>>>> Sounds good (perso I'd like we avoid hard
>>>>>>>>>>> dependency on Unsafe as maybe some use
>>>>>>>>>>> other jdks :-) )
>>>>>>>>>>>>
>>>>>>>>>>>> Now I see lightning is apache licensed
>>>>>>>>>>>> and this is fine but I
>>>>> don't
>>>>>>>>> think
>>>>>>>>>>>> it is published in any public maven
>>>>>>>>>>>> repo, am I right? We could
>>>>> find a
>>>>>>>>> way
>>>>>>>>>>>> to deal with this; options vary from
>>>>>>>>>>>> publishing lightning to the
>>>>> free
>>>>>>>>>>>> sonatype repo, joining the ASF (which
>>>>>>>>>>>> is great anyhow!) and
>>>>>>>>> republishing
>>>>>>>>>>>> lightning code in DirectMemory becoming a
>>>>>>>>>>>> commiter (which has to
>>>>>>>>> undergo
>>>>>>>>>>> a
>>>>>>>>>>>> PMC vote).
>>>>>>>>>>>
>>>>>>>>>>> At least for the moment he can provide a
>>>>>>>>>>> patch to be integrated in
>>>>> DM
>>>>>>>>> :-)
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I'd like to hear your and the team
>>>>>>>>>>>> feelings on this.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks, Raffaele
>>>>>>>>>>>>
>>>>>>>>>>>> On Sat, Sep 29, 2012 at 3:27 PM,
>>>>>>>>>>>> Noctarius <me...@noctarius.com>
>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hey Raffaele,
>>>>>>>>>>>>>
>>>>>>>>>>>>> that's quite similar to what I did at
>>>>>>>>>>>>> work. We're developing
>>>>> Flash
>>>>>>>>>>>>> online games and using a customized AMF
>>>>>>>>>>>>> serialization. To support async
>>>>>>>>>>>>> writing of the clients event results I
>>>>>>>>>>>>> added serialization
>>>>> of
>>>>>>>>>>>>> the components / entities to the
>>>>>>>>>>>>> players zone calculation and
>>>>> stored
>>>>>>>>>>>>> the pre-serialized bytestream directly
>>>>>>>>>>>>> to the off-heap (using
>>>>>>>>>>>>> direct-ring-cache implementation). When
>>>>>>>>>>>>> the client requests the results (using
>>>>>>>>>>>>> long-polling) I just write the
>>>>>>>>>>>>> pre-serialized
>>>>> data to
>>>>>>>>>>>>> the right position to deserialize it by
>>>>>>>>>>>>> standard ways on Flash
>>>>> side.
>>>>>>>>>>>>>
>>>>>>>>>>>>> So yeah an seriliaztion to off-heap
>>>>>>>>>>>>> algorithm would be fine. You
>>>>> can
>>>>>>>>>>>>> avoid using byte arrays and minimalize
>>>>>>>>>>>>> GC.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers Chris
>>>>>>>>>>>>>
>>>>>>>>>>>>> Am 29.09.2012 15:02, schrieb Raffaele
>>>>>>>>>>>>> P. Guidi:
>>>>>>>>>>>>>> Nice to hear back from you! Yes, I
>>>>>>>>>>>>>> was thinking about creating a
>>>>>>>>> new
>>>>>>>>>>>>> memory
>>>>>>>>>>>>>> storage implementation using Unsafe
>>>>>>>>>>>>>> (and I did it, recently) and
>>>>>>>>>>> also, as
>>>>>>>>>>>>>> DirectMemory relies heavily on
>>>>>>>>>>>>>> serialization (and supports many
>>>>> of
>>>>>>>>>>> them,
>>>>>>>>>>>>>> protostuff, protobuf, msgpack and of
>>>>>>>>>>>>>> course standard
>>>>>>>>> serialization),
>>>>>>>>>>>>> about
>>>>>>>>>>>>>> creating a simple embedded serializer
>>>>>>>>>>>>>> leveraging the same
>>>>>>>>> techniques
>>>>>>>>>>> you
>>>>>>>>>>>>>> used (Unsafe and bytecode
>>>>>>>>>>>>>> generation). The idea with embedding
>>>>>>>>>>>>>> is avoiding to serialize in a byte
>>>>>>>>>>>>>> array
>>>>>>>>> and
>>>>>>>>>>> then
>>>>>>>>>>>>>> moving the byte array to off-heap
>>>>>>>>>>>>>> memory (via Unsafe or
>>>>>>>>> ByteBuffers),
>>>>>>>>>>> and
>>>>>>>>>>>>>> serializing directly into a "managed"
>>>>>>>>>>>>>> off-heap buffer, thus
>>>>> further
>>>>>>>>>>>>>> optimizing heap utilization (less
>>>>>>>>>>>>>> GC).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> What do you think about it? Does it
>>>>>>>>>>>>>> make any sense to you?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Ciao, R
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Sat, Sep 29, 2012 at 2:40 PM,
>>>>>>>>>>>>>> Noctarius <me...@noctarius.com>
>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hey guys,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Raffaele found out about a project
>>>>>>>>>>>>>>> of mine (Lightning) a few
>>>>> weeks
>>>>>>>>>>>>>>> ago. Lightning is a heavy Unsafe
>>>>>>>>>>>>>>> and Bytecode generation using
>>>>>>>>>>>>>>> Serializer implementation. He told
>>>>>>>>>>>>>>> me that he was interested in adding
>>>>>>>>>>>>>>> something similar to DirectMemory
>>>>>>>>>>>>>>> and I would be glad to
>>>>>>>>> help
>>>>>>>>>>>>>>> out in this.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Another project I started a few
>>>>>>>>>>>>>>> days ago, since it was needed
>>>>> for
>>>>>>>>>>>>>>> work is DirectRingCache. The name
>>>>>>>>>>>>>>> not really meets to actual
>>>>>>>>>>>>>>> implementation since it's not yet
>>>>>>>>>>>>>>> a ring buffer using cache. I
>>>>>>>>> used
>>>>>>>>>>>>>>> this for a pre-serialization simple
>>>>>>>>>>>>>>> bytestream cache with self-growing
>>>>>>>>>>>>>>> buffers. It could be nice to have
>>>>>>>>>>>>>>> DirectMemory
>>>>> having
>>>>>>>>>>>>>>> raw "buffers" to write to or to
>>>>>>>>>>>>>>> read from.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Here are the links from the
>>>>>>>>>>>>>>> projects:
>>>>>>>>>>>>>>> https://github.com/noctarius/Lightning
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>
>>>>>>>>>>>>>>>
https://github.com/noctarius/direct-ring-cache
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>> It would be nice to help out since I really like the idea
>>> of
>>>>>>>>>>>>>>> DirectMemory and since
>>>>>>>>>>>>>>> direct-ring-cache is some kind of
>>>>>>>>> reinventing
>>>>>>>>>>>>>>> the wheel.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Cheers Noctarius (Chris)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> -- Olivier Lamy Talend:
>>>>>>>>>>> http://coders.talend.com
>>>>>>>>>>> http://twitter.com/olamy |
>>>>>>>>>>> http://linkedin.com/in/olamy
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -- Olivier Lamy Talend:
>>>>>>>>> http://coders.talend.com
>>>>>>>>> http://twitter.com/olamy |
>>>>>>>>> http://linkedin.com/in/olamy
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> -- Olivier Lamy Talend: http://coders.talend.com
>>>>> http://twitter.com/olamy |
>>>>> http://linkedin.com/in/olamy
>>>>>
>>>>
>>
>>
>>
Re: Additional Serializer and raw Buffer access
Posted by Noctarius <me...@noctarius.com>.
Thanks Olivier for carify, I'll take a look in it tomorrow but
there's just one question left (for now ;)):
What is that vote for becoming a committer? What if the vote will
be negative?
Until now I just used Apache stuff, was never interested in being
part of it so I guess it can be negative for any reason, can't it?
Am 29.09.2012 18:56, schrieb Olivier Lamy:
> 2012/9/29 Noctarius <me...@noctarius.com>:
>> Nope my real name is Christoph Engelbert, but Noctarius is
>> the all time nick :)
>>
>> Renaming the package should be no problem, is it
>> "org.apache.directmemory.lightning" or what would it be?
> fine for me
>> Then there needs to be a change in the license header as
>> Olivier mentioned, that means just remove the first sentence
>> or is there anything more to do (maybe it's easiest thing to
>> just copy the header from DM file ;))?
> yup use same header as DM
>>
>> The CLA is just a form to clarify that the source can be
>> contributed to the Apache Foundation?
> yup correct.
>>
>> The final step will be attaching the patch in form of a huge
>> diff file?
> yes
>>
>> And what is the way to apply for a membership? Never thought
>> about how to do that.
> Read here http://apache.org/foundation/how-it-works.html and
> here http://apache.org/foundation/getinvolved.html . And feel
> free to ask any questions :-)
>>
>> Chris
>>
>> Am 29.09.2012 18:23, schrieb Raffaele P. Guidi:
>>> OK, deal, at least for me ;-) I propose you rename the
>>> packages, produce a patch for this and the new serializer
>>> module (should be simple enough starting from an existing
>>> one) and, in the meanwhile, apply for ASF membership. Is
>>> IP clearance needed? I guess yes. After this we will come
>>> up with a formal vote regarding Noctarius (is this your
>>> real name?!) allowance in the project team.
>>>
>>> Good times are gonna come :-)
>>>
>>> Thanks, R Il giorno 29/set/2012 17:58, "Olivier Lamy"
>>> <ol...@apache.org> ha scritto:
>>>
>>>> 2012/9/29 Raffaele P. Guidi
>>>> <ra...@gmail.com>:
>>>>> Well we already have a NIO ready interface allowing
>>>>> direct access to DMs managed bytebuffers but I think
>>>>> this is just half way to what could be achieved
>>>>> optimally blending serialization and memory allocation
>>>>> together.
>>>>>
>>>>> Lightning as a module is of course a good idea and it
>>>>> could easily evolve as a subproject (for the more
>>>>> experienced asf members: is it a feasible way?).
>>>> Nothing prevent to have
>>>> http://svn.apache.org/repos/asf/directmemory/lightning/trunk
>>>>
>>>>
with a package like: o.a.d.lightning That will be a
>>>> subproject.
>>>>>
>>>>> Ciao, R Il giorno 29/set/2012 17:44, "Noctarius"
>>>>> <me...@noctarius.com> ha scritto:
>>>>>
>>>>>> Hey guys,
>>>>>>
>>>>>> ok there's no lightning binary available since
>>>>>> lightning wasn't ready yet for releasing.
>>>>>>
>>>>>> For being the only developer it would be no problem
>>>>>> to contribute the sourcebase for DirectMemory but I'm
>>>>>> not sure yet if it wouldn't be better to seperate it
>>>>>> to be available without using DirectMemory itself. I
>>>>>> started it as a serializer for cluster
>>>>>> synchronization, but it would be cool to contribute
>>>>>> lightning as a subproject to DirectMemory :-)
>>>>>>
>>>>>> About the second project I would love to see a
>>>>>> public available buffer API directly in DirectMemory
>>>>>> so that project would be nearly needless :-) The only
>>>>>> difference I think is the allocation strategy my
>>>>>> implementation is using against the one DirectMemory
>>>>>> has, but I'm pretty sure the allocation is extensible
>>>>>> ;-)
>>>>>>
>>>>>> Like I said, for both projects I'm the only dev so
>>>>>> there would be no IP problem. So if it's ok to you to
>>>>>> not include lightning directly in DM I would be glad
>>>>>> to contribute to the Apache Foundation :-)
>>>>>>
>>>>>> Cheers Chris
>>>>>>
>>>>>>
>>>>>> Am 29.09.2012 16:53, schrieb Raffaele P. Guidi:
>>>>>>> Ok so it's up to noctarius - your move! ;-)
>>>>>>> Regarding the new unsafe storage: it's an opt-in
>>>>>>> feature that can be set with the fluent API
>>>> (and
>>>>>>> soon through the conference file).
>>>>>>>
>>>>>>> Ciao, R Il giorno 29/set/2012 16:45, "Olivier
>>>>>>> Lamy" <ol...@apache.org> ha
>>>>>> scritto:
>>>>>>>
>>>>>>>> 2012/9/29 Raffaele P. Guidi
>>>>>>>> <ra...@gmail.com>:
>>>>>>>>>>> At least for the moment he can provide a
>>>>>>>>>>> patch to be integrated
>>>> in DM
>>>>>>>> :-)
>>>>>>>>>
>>>>>>>>> Sure, but as lightning is not in any public
>>>>>>>>> mvn repo should its
>>>> code be
>>>>>>>>> re-published in our svn? Or what?
>>>>>>>> @Apache we don't care about binaries, only
>>>>>>>> sources are important ! (a bit theorical for sure
>>>>>>>> but that's it :-) ). So if Noctarius was the only
>>>>>>>> guy who participate in lightning. He can just
>>>>>>>> provide a patch we could integrate as a new dm
>>>>>>>> module (note: the patch must not contains any
>>>>>>>> more copyright and all sources must have ASF
>>>>>>>> licenses).
>>>>>>>>
>>>>>>>> "Copyright 2012 the original author or authors."
>>>>>>>> must be removed. And BTW package must be changed
>>>>>>>> :-) (com.github is not acceptable
>>>> @asf
>>>>>>>> :-) )(@Noctarius are you working for github ? :-)
>>>>>>>> )
>>>>>>>>
>>>>>>>> And having him as a committer will be only a
>>>>>>>> matter of voting (we
>>>> have
>>>>>>>> a great chair who take cares of administrative
>>>>>>>> stuff :P )
>>>>>>>>
>>>>>>>> If some others have participated in the project,
>>>>>>>> we must pass tru an ip clearance mechanism
>>>>>>>> (http://incubator.apache.org/ip-clearance/index.html)
>>>>>>>>
>>>>>>>>
and all contributors to lightning must provide a cla.
>>>>>>>> (It it's the case I can help)
>>>>>>>>
>>>>>>>>>
>>>>>>>>>>> perso I'd like we avoid hard dependency on
>>>>>>>>>>> Unsafe as maybe some
>>>> use
>>>>>>>>> other jdks :-)
>>>>>>>>>
>>>>>>>>> Well, I believe Unsafe is supported by Sun
>>>>>>>>> JDK, OpenJDK, IBM JDK and JRockit - and I
>>>>>>>>> believe that it is more than enough. Also keep
>>>>>>>>> in
>>>> mind
>>>>>>>> that
>>>>>>>>> we already have an alternative Unsafe based
>>>>>>>>> memory storage - and
>>>>>> although
>>>>>>>>> it's not thoroughly tested for performance it
>>>>>>>>> dramaticaly simplifies
>>>>>>>> code,
>>>>>>>>> I have great expectations about it.
>>>>>>>> Me too :-). Yup definitely more simple and faster
>>>>>>>> ! But we must provide a switch off configuration
>>>>>>>> mechanism if some
>>>> users
>>>>>>>> don't want to use that (because Unsafe is just
>>>>>>>> "Unsafe" :-) ) And sorry I didn't have a look yet
>>>>>>>> at your changes with using Unsafe.
>>>>>>>>>
>>>>>>>>> Cheers, R
>>>>>>>>>
>>>>>>>>> On Sat, Sep 29, 2012 at 4:03 PM, Olivier Lamy
>>>>>>>>> <ol...@apache.org>
>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> 2012/9/29 Raffaele P. Guidi
>>>>>>>>>> <ra...@gmail.com>:
>>>>>>>>>>> What do you think about: 1) implementing a
>>>>>>>>>>> lightning serialization module 2) creating
>>>>>>>>>>> a serializer that directly works on a
>>>>>>>>>>> directmemory
>>>>>>>> provider
>>>>>>>>>>> ByteBuffer or (maybe better) Unsafe based
>>>>>>>>>>> Pointer?
>>>>>>>>>>
>>>>>>>>>> Sounds good (perso I'd like we avoid hard
>>>>>>>>>> dependency on Unsafe as maybe some use other
>>>>>>>>>> jdks :-) )
>>>>>>>>>>>
>>>>>>>>>>> Now I see lightning is apache licensed and
>>>>>>>>>>> this is fine but I
>>>> don't
>>>>>>>> think
>>>>>>>>>>> it is published in any public maven repo,
>>>>>>>>>>> am I right? We could
>>>> find a
>>>>>>>> way
>>>>>>>>>>> to deal with this; options vary from
>>>>>>>>>>> publishing lightning to the
>>>> free
>>>>>>>>>>> sonatype repo, joining the ASF (which is
>>>>>>>>>>> great anyhow!) and
>>>>>>>> republishing
>>>>>>>>>>> lightning code in DirectMemory becoming a
>>>>>>>>>>> commiter (which has to
>>>>>>>> undergo
>>>>>>>>>> a
>>>>>>>>>>> PMC vote).
>>>>>>>>>>
>>>>>>>>>> At least for the moment he can provide a
>>>>>>>>>> patch to be integrated in
>>>> DM
>>>>>>>> :-)
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I'd like to hear your and the team feelings
>>>>>>>>>>> on this.
>>>>>>>>>>>
>>>>>>>>>>> Thanks, Raffaele
>>>>>>>>>>>
>>>>>>>>>>> On Sat, Sep 29, 2012 at 3:27 PM, Noctarius
>>>>>>>>>>> <me...@noctarius.com>
>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hey Raffaele,
>>>>>>>>>>>>
>>>>>>>>>>>> that's quite similar to what I did at
>>>>>>>>>>>> work. We're developing
>>>> Flash
>>>>>>>>>>>> online games and using a customized AMF
>>>>>>>>>>>> serialization. To support async writing
>>>>>>>>>>>> of the clients event results I added
>>>>>>>>>>>> serialization
>>>> of
>>>>>>>>>>>> the components / entities to the players
>>>>>>>>>>>> zone calculation and
>>>> stored
>>>>>>>>>>>> the pre-serialized bytestream directly to
>>>>>>>>>>>> the off-heap (using direct-ring-cache
>>>>>>>>>>>> implementation). When the client
>>>>>>>>>>>> requests the results (using long-polling)
>>>>>>>>>>>> I just write the pre-serialized
>>>> data to
>>>>>>>>>>>> the right position to deserialize it by
>>>>>>>>>>>> standard ways on Flash
>>>> side.
>>>>>>>>>>>>
>>>>>>>>>>>> So yeah an seriliaztion to off-heap
>>>>>>>>>>>> algorithm would be fine. You
>>>> can
>>>>>>>>>>>> avoid using byte arrays and minimalize
>>>>>>>>>>>> GC.
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers Chris
>>>>>>>>>>>>
>>>>>>>>>>>> Am 29.09.2012 15:02, schrieb Raffaele P.
>>>>>>>>>>>> Guidi:
>>>>>>>>>>>>> Nice to hear back from you! Yes, I was
>>>>>>>>>>>>> thinking about creating a
>>>>>>>> new
>>>>>>>>>>>> memory
>>>>>>>>>>>>> storage implementation using Unsafe
>>>>>>>>>>>>> (and I did it, recently) and
>>>>>>>>>> also, as
>>>>>>>>>>>>> DirectMemory relies heavily on
>>>>>>>>>>>>> serialization (and supports many
>>>> of
>>>>>>>>>> them,
>>>>>>>>>>>>> protostuff, protobuf, msgpack and of
>>>>>>>>>>>>> course standard
>>>>>>>> serialization),
>>>>>>>>>>>> about
>>>>>>>>>>>>> creating a simple embedded serializer
>>>>>>>>>>>>> leveraging the same
>>>>>>>> techniques
>>>>>>>>>> you
>>>>>>>>>>>>> used (Unsafe and bytecode generation).
>>>>>>>>>>>>> The idea with embedding is avoiding to
>>>>>>>>>>>>> serialize in a byte array
>>>>>>>> and
>>>>>>>>>> then
>>>>>>>>>>>>> moving the byte array to off-heap
>>>>>>>>>>>>> memory (via Unsafe or
>>>>>>>> ByteBuffers),
>>>>>>>>>> and
>>>>>>>>>>>>> serializing directly into a "managed"
>>>>>>>>>>>>> off-heap buffer, thus
>>>> further
>>>>>>>>>>>>> optimizing heap utilization (less GC).
>>>>>>>>>>>>>
>>>>>>>>>>>>> What do you think about it? Does it
>>>>>>>>>>>>> make any sense to you?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Ciao, R
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Sat, Sep 29, 2012 at 2:40 PM,
>>>>>>>>>>>>> Noctarius <me...@noctarius.com>
>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hey guys,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Raffaele found out about a project
>>>>>>>>>>>>>> of mine (Lightning) a few
>>>> weeks
>>>>>>>>>>>>>> ago. Lightning is a heavy Unsafe and
>>>>>>>>>>>>>> Bytecode generation using Serializer
>>>>>>>>>>>>>> implementation. He told me that he
>>>>>>>>>>>>>> was interested in adding something
>>>>>>>>>>>>>> similar to DirectMemory and I would
>>>>>>>>>>>>>> be glad to
>>>>>>>> help
>>>>>>>>>>>>>> out in this.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Another project I started a few days
>>>>>>>>>>>>>> ago, since it was needed
>>>> for
>>>>>>>>>>>>>> work is DirectRingCache. The name
>>>>>>>>>>>>>> not really meets to actual
>>>>>>>>>>>>>> implementation since it's not yet a
>>>>>>>>>>>>>> ring buffer using cache. I
>>>>>>>> used
>>>>>>>>>>>>>> this for a pre-serialization simple
>>>>>>>>>>>>>> bytestream cache with self-growing
>>>>>>>>>>>>>> buffers. It could be nice to have
>>>>>>>>>>>>>> DirectMemory
>>>> having
>>>>>>>>>>>>>> raw "buffers" to write to or to read
>>>>>>>>>>>>>> from.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Here are the links from the
>>>>>>>>>>>>>> projects:
>>>>>>>>>>>>>> https://github.com/noctarius/Lightning
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
https://github.com/noctarius/direct-ring-cache
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>> It would be nice to help out since I really like the idea of
>>>>>>>>>>>>>> DirectMemory and since
>>>>>>>>>>>>>> direct-ring-cache is some kind of
>>>>>>>> reinventing
>>>>>>>>>>>>>> the wheel.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Cheers Noctarius (Chris)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> -- Olivier Lamy Talend:
>>>>>>>>>> http://coders.talend.com
>>>>>>>>>> http://twitter.com/olamy |
>>>>>>>>>> http://linkedin.com/in/olamy
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> -- Olivier Lamy Talend: http://coders.talend.com
>>>>>>>> http://twitter.com/olamy |
>>>>>>>> http://linkedin.com/in/olamy
>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>>
>>>>
>>>> -- Olivier Lamy Talend: http://coders.talend.com
>>>> http://twitter.com/olamy | http://linkedin.com/in/olamy
>>>>
>>>
>
>
>
Re: Additional Serializer and raw Buffer access
Posted by Olivier Lamy <ol...@apache.org>.
2012/9/29 Noctarius <me...@noctarius.com>:
> Nope my real name is Christoph Engelbert, but Noctarius is the all
> time nick :)
>
> Renaming the package should be no problem, is it
> "org.apache.directmemory.lightning" or what would it be?
fine for me
> Then there needs to be a change in the license header as Olivier
> mentioned, that means just remove the first sentence or is there
> anything more to do (maybe it's easiest thing to just copy the
> header from DM file ;))?
yup use same header as DM
>
> The CLA is just a form to clarify that the source can be
> contributed to the Apache Foundation?
yup correct.
>
> The final step will be attaching the patch in form of a huge diff
> file?
yes
>
> And what is the way to apply for a membership? Never thought about
> how to do that.
Read here http://apache.org/foundation/how-it-works.html and here
http://apache.org/foundation/getinvolved.html .
And feel free to ask any questions :-)
>
> Chris
>
> Am 29.09.2012 18:23, schrieb Raffaele P. Guidi:
>> OK, deal, at least for me ;-) I propose you rename the
>> packages, produce a patch for this and the new serializer
>> module (should be simple enough starting from an existing one)
>> and, in the meanwhile, apply for ASF membership. Is IP
>> clearance needed? I guess yes. After this we will come up with
>> a formal vote regarding Noctarius (is this your real name?!)
>> allowance in the project team.
>>
>> Good times are gonna come :-)
>>
>> Thanks, R Il giorno 29/set/2012 17:58, "Olivier Lamy"
>> <ol...@apache.org> ha scritto:
>>
>>> 2012/9/29 Raffaele P. Guidi <ra...@gmail.com>:
>>>> Well we already have a NIO ready interface allowing direct
>>>> access to DMs managed bytebuffers but I think this is just
>>>> half way to what could be achieved optimally blending
>>>> serialization and memory allocation together.
>>>>
>>>> Lightning as a module is of course a good idea and it could
>>>> easily evolve as a subproject (for the more experienced asf
>>>> members: is it a feasible way?).
>>> Nothing prevent to have
>>> http://svn.apache.org/repos/asf/directmemory/lightning/trunk
>>> with a package like: o.a.d.lightning That will be a
>>> subproject.
>>>>
>>>> Ciao, R Il giorno 29/set/2012 17:44, "Noctarius"
>>>> <me...@noctarius.com> ha scritto:
>>>>
>>>>> Hey guys,
>>>>>
>>>>> ok there's no lightning binary available since lightning
>>>>> wasn't ready yet for releasing.
>>>>>
>>>>> For being the only developer it would be no problem to
>>>>> contribute the sourcebase for DirectMemory but I'm not
>>>>> sure yet if it wouldn't be better to seperate it to be
>>>>> available without using DirectMemory itself. I started it
>>>>> as a serializer for cluster synchronization, but it would
>>>>> be cool to contribute lightning as a subproject to
>>>>> DirectMemory :-)
>>>>>
>>>>> About the second project I would love to see a public
>>>>> available buffer API directly in DirectMemory so that
>>>>> project would be nearly needless :-) The only difference
>>>>> I think is the allocation strategy my implementation is
>>>>> using against the one DirectMemory has, but I'm pretty
>>>>> sure the allocation is extensible ;-)
>>>>>
>>>>> Like I said, for both projects I'm the only dev so there
>>>>> would be no IP problem. So if it's ok to you to not
>>>>> include lightning directly in DM I would be glad to
>>>>> contribute to the Apache Foundation :-)
>>>>>
>>>>> Cheers Chris
>>>>>
>>>>>
>>>>> Am 29.09.2012 16:53, schrieb Raffaele P. Guidi:
>>>>>> Ok so it's up to noctarius - your move! ;-) Regarding
>>>>>> the new unsafe storage: it's an opt-in feature that can
>>>>>> be set with the fluent API
>>> (and
>>>>>> soon through the conference file).
>>>>>>
>>>>>> Ciao, R Il giorno 29/set/2012 16:45, "Olivier Lamy"
>>>>>> <ol...@apache.org> ha
>>>>> scritto:
>>>>>>
>>>>>>> 2012/9/29 Raffaele P. Guidi
>>>>>>> <ra...@gmail.com>:
>>>>>>>>>> At least for the moment he can provide a patch
>>>>>>>>>> to be integrated
>>> in DM
>>>>>>> :-)
>>>>>>>>
>>>>>>>> Sure, but as lightning is not in any public mvn
>>>>>>>> repo should its
>>> code be
>>>>>>>> re-published in our svn? Or what?
>>>>>>> @Apache we don't care about binaries, only sources
>>>>>>> are important ! (a bit theorical for sure but that's
>>>>>>> it :-) ). So if Noctarius was the only guy who
>>>>>>> participate in lightning. He can just provide a patch
>>>>>>> we could integrate as a new dm module (note: the
>>>>>>> patch must not contains any more copyright and all
>>>>>>> sources must have ASF licenses).
>>>>>>>
>>>>>>> "Copyright 2012 the original author or authors." must
>>>>>>> be removed. And BTW package must be changed :-)
>>>>>>> (com.github is not acceptable
>>> @asf
>>>>>>> :-) )(@Noctarius are you working for github ? :-) )
>>>>>>>
>>>>>>> And having him as a committer will be only a matter
>>>>>>> of voting (we
>>> have
>>>>>>> a great chair who take cares of administrative stuff
>>>>>>> :P )
>>>>>>>
>>>>>>> If some others have participated in the project, we
>>>>>>> must pass tru an ip clearance mechanism
>>>>>>> (http://incubator.apache.org/ip-clearance/index.html)
>>>>>>> and all contributors to lightning must provide a cla.
>>>>>>> (It it's the case I can help)
>>>>>>>
>>>>>>>>
>>>>>>>>>> perso I'd like we avoid hard dependency on
>>>>>>>>>> Unsafe as maybe some
>>> use
>>>>>>>> other jdks :-)
>>>>>>>>
>>>>>>>> Well, I believe Unsafe is supported by Sun JDK,
>>>>>>>> OpenJDK, IBM JDK and JRockit - and I believe that
>>>>>>>> it is more than enough. Also keep in
>>> mind
>>>>>>> that
>>>>>>>> we already have an alternative Unsafe based memory
>>>>>>>> storage - and
>>>>> although
>>>>>>>> it's not thoroughly tested for performance it
>>>>>>>> dramaticaly simplifies
>>>>>>> code,
>>>>>>>> I have great expectations about it.
>>>>>>> Me too :-). Yup definitely more simple and faster !
>>>>>>> But we must provide a switch off configuration
>>>>>>> mechanism if some
>>> users
>>>>>>> don't want to use that (because Unsafe is just
>>>>>>> "Unsafe" :-) ) And sorry I didn't have a look yet at
>>>>>>> your changes with using Unsafe.
>>>>>>>>
>>>>>>>> Cheers, R
>>>>>>>>
>>>>>>>> On Sat, Sep 29, 2012 at 4:03 PM, Olivier Lamy
>>>>>>>> <ol...@apache.org>
>>>>> wrote:
>>>>>>>>
>>>>>>>>> 2012/9/29 Raffaele P. Guidi
>>>>>>>>> <ra...@gmail.com>:
>>>>>>>>>> What do you think about: 1) implementing a
>>>>>>>>>> lightning serialization module 2) creating a
>>>>>>>>>> serializer that directly works on a
>>>>>>>>>> directmemory
>>>>>>> provider
>>>>>>>>>> ByteBuffer or (maybe better) Unsafe based
>>>>>>>>>> Pointer?
>>>>>>>>>
>>>>>>>>> Sounds good (perso I'd like we avoid hard
>>>>>>>>> dependency on Unsafe as maybe some use other jdks
>>>>>>>>> :-) )
>>>>>>>>>>
>>>>>>>>>> Now I see lightning is apache licensed and this
>>>>>>>>>> is fine but I
>>> don't
>>>>>>> think
>>>>>>>>>> it is published in any public maven repo, am I
>>>>>>>>>> right? We could
>>> find a
>>>>>>> way
>>>>>>>>>> to deal with this; options vary from publishing
>>>>>>>>>> lightning to the
>>> free
>>>>>>>>>> sonatype repo, joining the ASF (which is great
>>>>>>>>>> anyhow!) and
>>>>>>> republishing
>>>>>>>>>> lightning code in DirectMemory becoming a
>>>>>>>>>> commiter (which has to
>>>>>>> undergo
>>>>>>>>> a
>>>>>>>>>> PMC vote).
>>>>>>>>>
>>>>>>>>> At least for the moment he can provide a patch to
>>>>>>>>> be integrated in
>>> DM
>>>>>>> :-)
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I'd like to hear your and the team feelings on
>>>>>>>>>> this.
>>>>>>>>>>
>>>>>>>>>> Thanks, Raffaele
>>>>>>>>>>
>>>>>>>>>> On Sat, Sep 29, 2012 at 3:27 PM, Noctarius
>>>>>>>>>> <me...@noctarius.com>
>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hey Raffaele,
>>>>>>>>>>>
>>>>>>>>>>> that's quite similar to what I did at work.
>>>>>>>>>>> We're developing
>>> Flash
>>>>>>>>>>> online games and using a customized AMF
>>>>>>>>>>> serialization. To support async writing of
>>>>>>>>>>> the clients event results I added
>>>>>>>>>>> serialization
>>> of
>>>>>>>>>>> the components / entities to the players zone
>>>>>>>>>>> calculation and
>>> stored
>>>>>>>>>>> the pre-serialized bytestream directly to the
>>>>>>>>>>> off-heap (using direct-ring-cache
>>>>>>>>>>> implementation). When the client requests
>>>>>>>>>>> the results (using long-polling) I just write
>>>>>>>>>>> the pre-serialized
>>> data to
>>>>>>>>>>> the right position to deserialize it by
>>>>>>>>>>> standard ways on Flash
>>> side.
>>>>>>>>>>>
>>>>>>>>>>> So yeah an seriliaztion to off-heap algorithm
>>>>>>>>>>> would be fine. You
>>> can
>>>>>>>>>>> avoid using byte arrays and minimalize GC.
>>>>>>>>>>>
>>>>>>>>>>> Cheers Chris
>>>>>>>>>>>
>>>>>>>>>>> Am 29.09.2012 15:02, schrieb Raffaele P.
>>>>>>>>>>> Guidi:
>>>>>>>>>>>> Nice to hear back from you! Yes, I was
>>>>>>>>>>>> thinking about creating a
>>>>>>> new
>>>>>>>>>>> memory
>>>>>>>>>>>> storage implementation using Unsafe (and I
>>>>>>>>>>>> did it, recently) and
>>>>>>>>> also, as
>>>>>>>>>>>> DirectMemory relies heavily on
>>>>>>>>>>>> serialization (and supports many
>>> of
>>>>>>>>> them,
>>>>>>>>>>>> protostuff, protobuf, msgpack and of course
>>>>>>>>>>>> standard
>>>>>>> serialization),
>>>>>>>>>>> about
>>>>>>>>>>>> creating a simple embedded serializer
>>>>>>>>>>>> leveraging the same
>>>>>>> techniques
>>>>>>>>> you
>>>>>>>>>>>> used (Unsafe and bytecode generation). The
>>>>>>>>>>>> idea with embedding is avoiding to
>>>>>>>>>>>> serialize in a byte array
>>>>>>> and
>>>>>>>>> then
>>>>>>>>>>>> moving the byte array to off-heap memory
>>>>>>>>>>>> (via Unsafe or
>>>>>>> ByteBuffers),
>>>>>>>>> and
>>>>>>>>>>>> serializing directly into a "managed"
>>>>>>>>>>>> off-heap buffer, thus
>>> further
>>>>>>>>>>>> optimizing heap utilization (less GC).
>>>>>>>>>>>>
>>>>>>>>>>>> What do you think about it? Does it make
>>>>>>>>>>>> any sense to you?
>>>>>>>>>>>>
>>>>>>>>>>>> Ciao, R
>>>>>>>>>>>>
>>>>>>>>>>>> On Sat, Sep 29, 2012 at 2:40 PM, Noctarius
>>>>>>>>>>>> <me...@noctarius.com>
>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hey guys,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Raffaele found out about a project of
>>>>>>>>>>>>> mine (Lightning) a few
>>> weeks
>>>>>>>>>>>>> ago. Lightning is a heavy Unsafe and
>>>>>>>>>>>>> Bytecode generation using Serializer
>>>>>>>>>>>>> implementation. He told me that he was
>>>>>>>>>>>>> interested in adding something similar to
>>>>>>>>>>>>> DirectMemory and I would be glad to
>>>>>>> help
>>>>>>>>>>>>> out in this.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Another project I started a few days ago,
>>>>>>>>>>>>> since it was needed
>>> for
>>>>>>>>>>>>> work is DirectRingCache. The name not
>>>>>>>>>>>>> really meets to actual implementation
>>>>>>>>>>>>> since it's not yet a ring buffer using
>>>>>>>>>>>>> cache. I
>>>>>>> used
>>>>>>>>>>>>> this for a pre-serialization simple
>>>>>>>>>>>>> bytestream cache with self-growing
>>>>>>>>>>>>> buffers. It could be nice to have
>>>>>>>>>>>>> DirectMemory
>>> having
>>>>>>>>>>>>> raw "buffers" to write to or to read
>>>>>>>>>>>>> from.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Here are the links from the projects:
>>>>>>>>>>>>> https://github.com/noctarius/Lightning
>>>>>>>>>>>>> https://github.com/noctarius/direct-ring-cache
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
> It would be nice to help out since I really like the idea of
>>>>>>>>>>>>> DirectMemory and since direct-ring-cache
>>>>>>>>>>>>> is some kind of
>>>>>>> reinventing
>>>>>>>>>>>>> the wheel.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers Noctarius (Chris)
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -- Olivier Lamy Talend: http://coders.talend.com
>>>>>>>>> http://twitter.com/olamy |
>>>>>>>>> http://linkedin.com/in/olamy
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> -- Olivier Lamy Talend: http://coders.talend.com
>>>>>>> http://twitter.com/olamy |
>>>>>>> http://linkedin.com/in/olamy
>>>>>>>
>>>>>>
>>>>>
>>>
>>>
>>>
>>> -- Olivier Lamy Talend: http://coders.talend.com
>>> http://twitter.com/olamy | http://linkedin.com/in/olamy
>>>
>>
--
Olivier Lamy
Talend: http://coders.talend.com
http://twitter.com/olamy | http://linkedin.com/in/olamy
Re: Additional Serializer and raw Buffer access
Posted by Noctarius <me...@noctarius.com>.
Nope my real name is Christoph Engelbert, but Noctarius is the all
time nick :)
Renaming the package should be no problem, is it
"org.apache.directmemory.lightning" or what would it be?
Then there needs to be a change in the license header as Olivier
mentioned, that means just remove the first sentence or is there
anything more to do (maybe it's easiest thing to just copy the
header from DM file ;))?
The CLA is just a form to clarify that the source can be
contributed to the Apache Foundation?
The final step will be attaching the patch in form of a huge diff
file?
And what is the way to apply for a membership? Never thought about
how to do that.
Chris
Am 29.09.2012 18:23, schrieb Raffaele P. Guidi:
> OK, deal, at least for me ;-) I propose you rename the
> packages, produce a patch for this and the new serializer
> module (should be simple enough starting from an existing one)
> and, in the meanwhile, apply for ASF membership. Is IP
> clearance needed? I guess yes. After this we will come up with
> a formal vote regarding Noctarius (is this your real name?!)
> allowance in the project team.
>
> Good times are gonna come :-)
>
> Thanks, R Il giorno 29/set/2012 17:58, "Olivier Lamy"
> <ol...@apache.org> ha scritto:
>
>> 2012/9/29 Raffaele P. Guidi <ra...@gmail.com>:
>>> Well we already have a NIO ready interface allowing direct
>>> access to DMs managed bytebuffers but I think this is just
>>> half way to what could be achieved optimally blending
>>> serialization and memory allocation together.
>>>
>>> Lightning as a module is of course a good idea and it could
>>> easily evolve as a subproject (for the more experienced asf
>>> members: is it a feasible way?).
>> Nothing prevent to have
>> http://svn.apache.org/repos/asf/directmemory/lightning/trunk
>> with a package like: o.a.d.lightning That will be a
>> subproject.
>>>
>>> Ciao, R Il giorno 29/set/2012 17:44, "Noctarius"
>>> <me...@noctarius.com> ha scritto:
>>>
>>>> Hey guys,
>>>>
>>>> ok there's no lightning binary available since lightning
>>>> wasn't ready yet for releasing.
>>>>
>>>> For being the only developer it would be no problem to
>>>> contribute the sourcebase for DirectMemory but I'm not
>>>> sure yet if it wouldn't be better to seperate it to be
>>>> available without using DirectMemory itself. I started it
>>>> as a serializer for cluster synchronization, but it would
>>>> be cool to contribute lightning as a subproject to
>>>> DirectMemory :-)
>>>>
>>>> About the second project I would love to see a public
>>>> available buffer API directly in DirectMemory so that
>>>> project would be nearly needless :-) The only difference
>>>> I think is the allocation strategy my implementation is
>>>> using against the one DirectMemory has, but I'm pretty
>>>> sure the allocation is extensible ;-)
>>>>
>>>> Like I said, for both projects I'm the only dev so there
>>>> would be no IP problem. So if it's ok to you to not
>>>> include lightning directly in DM I would be glad to
>>>> contribute to the Apache Foundation :-)
>>>>
>>>> Cheers Chris
>>>>
>>>>
>>>> Am 29.09.2012 16:53, schrieb Raffaele P. Guidi:
>>>>> Ok so it's up to noctarius - your move! ;-) Regarding
>>>>> the new unsafe storage: it's an opt-in feature that can
>>>>> be set with the fluent API
>> (and
>>>>> soon through the conference file).
>>>>>
>>>>> Ciao, R Il giorno 29/set/2012 16:45, "Olivier Lamy"
>>>>> <ol...@apache.org> ha
>>>> scritto:
>>>>>
>>>>>> 2012/9/29 Raffaele P. Guidi
>>>>>> <ra...@gmail.com>:
>>>>>>>>> At least for the moment he can provide a patch
>>>>>>>>> to be integrated
>> in DM
>>>>>> :-)
>>>>>>>
>>>>>>> Sure, but as lightning is not in any public mvn
>>>>>>> repo should its
>> code be
>>>>>>> re-published in our svn? Or what?
>>>>>> @Apache we don't care about binaries, only sources
>>>>>> are important ! (a bit theorical for sure but that's
>>>>>> it :-) ). So if Noctarius was the only guy who
>>>>>> participate in lightning. He can just provide a patch
>>>>>> we could integrate as a new dm module (note: the
>>>>>> patch must not contains any more copyright and all
>>>>>> sources must have ASF licenses).
>>>>>>
>>>>>> "Copyright 2012 the original author or authors." must
>>>>>> be removed. And BTW package must be changed :-)
>>>>>> (com.github is not acceptable
>> @asf
>>>>>> :-) )(@Noctarius are you working for github ? :-) )
>>>>>>
>>>>>> And having him as a committer will be only a matter
>>>>>> of voting (we
>> have
>>>>>> a great chair who take cares of administrative stuff
>>>>>> :P )
>>>>>>
>>>>>> If some others have participated in the project, we
>>>>>> must pass tru an ip clearance mechanism
>>>>>> (http://incubator.apache.org/ip-clearance/index.html)
>>>>>> and all contributors to lightning must provide a cla.
>>>>>> (It it's the case I can help)
>>>>>>
>>>>>>>
>>>>>>>>> perso I'd like we avoid hard dependency on
>>>>>>>>> Unsafe as maybe some
>> use
>>>>>>> other jdks :-)
>>>>>>>
>>>>>>> Well, I believe Unsafe is supported by Sun JDK,
>>>>>>> OpenJDK, IBM JDK and JRockit - and I believe that
>>>>>>> it is more than enough. Also keep in
>> mind
>>>>>> that
>>>>>>> we already have an alternative Unsafe based memory
>>>>>>> storage - and
>>>> although
>>>>>>> it's not thoroughly tested for performance it
>>>>>>> dramaticaly simplifies
>>>>>> code,
>>>>>>> I have great expectations about it.
>>>>>> Me too :-). Yup definitely more simple and faster !
>>>>>> But we must provide a switch off configuration
>>>>>> mechanism if some
>> users
>>>>>> don't want to use that (because Unsafe is just
>>>>>> "Unsafe" :-) ) And sorry I didn't have a look yet at
>>>>>> your changes with using Unsafe.
>>>>>>>
>>>>>>> Cheers, R
>>>>>>>
>>>>>>> On Sat, Sep 29, 2012 at 4:03 PM, Olivier Lamy
>>>>>>> <ol...@apache.org>
>>>> wrote:
>>>>>>>
>>>>>>>> 2012/9/29 Raffaele P. Guidi
>>>>>>>> <ra...@gmail.com>:
>>>>>>>>> What do you think about: 1) implementing a
>>>>>>>>> lightning serialization module 2) creating a
>>>>>>>>> serializer that directly works on a
>>>>>>>>> directmemory
>>>>>> provider
>>>>>>>>> ByteBuffer or (maybe better) Unsafe based
>>>>>>>>> Pointer?
>>>>>>>>
>>>>>>>> Sounds good (perso I'd like we avoid hard
>>>>>>>> dependency on Unsafe as maybe some use other jdks
>>>>>>>> :-) )
>>>>>>>>>
>>>>>>>>> Now I see lightning is apache licensed and this
>>>>>>>>> is fine but I
>> don't
>>>>>> think
>>>>>>>>> it is published in any public maven repo, am I
>>>>>>>>> right? We could
>> find a
>>>>>> way
>>>>>>>>> to deal with this; options vary from publishing
>>>>>>>>> lightning to the
>> free
>>>>>>>>> sonatype repo, joining the ASF (which is great
>>>>>>>>> anyhow!) and
>>>>>> republishing
>>>>>>>>> lightning code in DirectMemory becoming a
>>>>>>>>> commiter (which has to
>>>>>> undergo
>>>>>>>> a
>>>>>>>>> PMC vote).
>>>>>>>>
>>>>>>>> At least for the moment he can provide a patch to
>>>>>>>> be integrated in
>> DM
>>>>>> :-)
>>>>>>>>
>>>>>>>>>
>>>>>>>>> I'd like to hear your and the team feelings on
>>>>>>>>> this.
>>>>>>>>>
>>>>>>>>> Thanks, Raffaele
>>>>>>>>>
>>>>>>>>> On Sat, Sep 29, 2012 at 3:27 PM, Noctarius
>>>>>>>>> <me...@noctarius.com>
>> wrote:
>>>>>>>>>
>>>>>>>>>> Hey Raffaele,
>>>>>>>>>>
>>>>>>>>>> that's quite similar to what I did at work.
>>>>>>>>>> We're developing
>> Flash
>>>>>>>>>> online games and using a customized AMF
>>>>>>>>>> serialization. To support async writing of
>>>>>>>>>> the clients event results I added
>>>>>>>>>> serialization
>> of
>>>>>>>>>> the components / entities to the players zone
>>>>>>>>>> calculation and
>> stored
>>>>>>>>>> the pre-serialized bytestream directly to the
>>>>>>>>>> off-heap (using direct-ring-cache
>>>>>>>>>> implementation). When the client requests
>>>>>>>>>> the results (using long-polling) I just write
>>>>>>>>>> the pre-serialized
>> data to
>>>>>>>>>> the right position to deserialize it by
>>>>>>>>>> standard ways on Flash
>> side.
>>>>>>>>>>
>>>>>>>>>> So yeah an seriliaztion to off-heap algorithm
>>>>>>>>>> would be fine. You
>> can
>>>>>>>>>> avoid using byte arrays and minimalize GC.
>>>>>>>>>>
>>>>>>>>>> Cheers Chris
>>>>>>>>>>
>>>>>>>>>> Am 29.09.2012 15:02, schrieb Raffaele P.
>>>>>>>>>> Guidi:
>>>>>>>>>>> Nice to hear back from you! Yes, I was
>>>>>>>>>>> thinking about creating a
>>>>>> new
>>>>>>>>>> memory
>>>>>>>>>>> storage implementation using Unsafe (and I
>>>>>>>>>>> did it, recently) and
>>>>>>>> also, as
>>>>>>>>>>> DirectMemory relies heavily on
>>>>>>>>>>> serialization (and supports many
>> of
>>>>>>>> them,
>>>>>>>>>>> protostuff, protobuf, msgpack and of course
>>>>>>>>>>> standard
>>>>>> serialization),
>>>>>>>>>> about
>>>>>>>>>>> creating a simple embedded serializer
>>>>>>>>>>> leveraging the same
>>>>>> techniques
>>>>>>>> you
>>>>>>>>>>> used (Unsafe and bytecode generation). The
>>>>>>>>>>> idea with embedding is avoiding to
>>>>>>>>>>> serialize in a byte array
>>>>>> and
>>>>>>>> then
>>>>>>>>>>> moving the byte array to off-heap memory
>>>>>>>>>>> (via Unsafe or
>>>>>> ByteBuffers),
>>>>>>>> and
>>>>>>>>>>> serializing directly into a "managed"
>>>>>>>>>>> off-heap buffer, thus
>> further
>>>>>>>>>>> optimizing heap utilization (less GC).
>>>>>>>>>>>
>>>>>>>>>>> What do you think about it? Does it make
>>>>>>>>>>> any sense to you?
>>>>>>>>>>>
>>>>>>>>>>> Ciao, R
>>>>>>>>>>>
>>>>>>>>>>> On Sat, Sep 29, 2012 at 2:40 PM, Noctarius
>>>>>>>>>>> <me...@noctarius.com>
>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hey guys,
>>>>>>>>>>>>
>>>>>>>>>>>> Raffaele found out about a project of
>>>>>>>>>>>> mine (Lightning) a few
>> weeks
>>>>>>>>>>>> ago. Lightning is a heavy Unsafe and
>>>>>>>>>>>> Bytecode generation using Serializer
>>>>>>>>>>>> implementation. He told me that he was
>>>>>>>>>>>> interested in adding something similar to
>>>>>>>>>>>> DirectMemory and I would be glad to
>>>>>> help
>>>>>>>>>>>> out in this.
>>>>>>>>>>>>
>>>>>>>>>>>> Another project I started a few days ago,
>>>>>>>>>>>> since it was needed
>> for
>>>>>>>>>>>> work is DirectRingCache. The name not
>>>>>>>>>>>> really meets to actual implementation
>>>>>>>>>>>> since it's not yet a ring buffer using
>>>>>>>>>>>> cache. I
>>>>>> used
>>>>>>>>>>>> this for a pre-serialization simple
>>>>>>>>>>>> bytestream cache with self-growing
>>>>>>>>>>>> buffers. It could be nice to have
>>>>>>>>>>>> DirectMemory
>> having
>>>>>>>>>>>> raw "buffers" to write to or to read
>>>>>>>>>>>> from.
>>>>>>>>>>>>
>>>>>>>>>>>> Here are the links from the projects:
>>>>>>>>>>>> https://github.com/noctarius/Lightning
>>>>>>>>>>>> https://github.com/noctarius/direct-ring-cache
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
It would be nice to help out since I really like the idea of
>>>>>>>>>>>> DirectMemory and since direct-ring-cache
>>>>>>>>>>>> is some kind of
>>>>>> reinventing
>>>>>>>>>>>> the wheel.
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers Noctarius (Chris)
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> -- Olivier Lamy Talend: http://coders.talend.com
>>>>>>>> http://twitter.com/olamy |
>>>>>>>> http://linkedin.com/in/olamy
>>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> -- Olivier Lamy Talend: http://coders.talend.com
>>>>>> http://twitter.com/olamy |
>>>>>> http://linkedin.com/in/olamy
>>>>>>
>>>>>
>>>>
>>
>>
>>
>> -- Olivier Lamy Talend: http://coders.talend.com
>> http://twitter.com/olamy | http://linkedin.com/in/olamy
>>
>
Re: Additional Serializer and raw Buffer access
Posted by "Raffaele P. Guidi" <ra...@gmail.com>.
OK, deal, at least for me ;-) I propose you rename the packages, produce a
patch for this and the new serializer module (should be simple enough
starting from an existing one) and, in the meanwhile, apply for ASF
membership. Is IP clearance needed? I guess yes.
After this we will come up with a formal vote regarding Noctarius (is this
your real name?!) allowance in the project team.
Good times are gonna come :-)
Thanks,
R
Il giorno 29/set/2012 17:58, "Olivier Lamy" <ol...@apache.org> ha scritto:
> 2012/9/29 Raffaele P. Guidi <ra...@gmail.com>:
> > Well we already have a NIO ready interface allowing direct access to DMs
> > managed bytebuffers but I think this is just half way to what could be
> > achieved optimally blending serialization and memory allocation together.
> >
> > Lightning as a module is of course a good idea and it could easily evolve
> > as a subproject (for the more experienced asf members: is it a feasible
> > way?).
> Nothing prevent to have
> http://svn.apache.org/repos/asf/directmemory/lightning/trunk
> with a package like: o.a.d.lightning
> That will be a subproject.
> >
> > Ciao,
> > R
> > Il giorno 29/set/2012 17:44, "Noctarius" <me...@noctarius.com> ha scritto:
> >
> >> Hey guys,
> >>
> >> ok there's no lightning binary available since lightning wasn't
> >> ready yet for releasing.
> >>
> >> For being the only developer it would be no problem to contribute
> >> the sourcebase for DirectMemory but I'm not sure yet if it wouldn't
> >> be better to seperate it to be available without using DirectMemory
> >> itself. I started it as a serializer for cluster synchronization,
> >> but it would be cool to contribute lightning as a subproject to
> >> DirectMemory :-)
> >>
> >> About the second project I would love to see a public available
> >> buffer API directly in DirectMemory so that project would be nearly
> >> needless :-) The only difference I think is the allocation strategy
> >> my implementation is using against the one DirectMemory has, but I'm
> >> pretty sure the allocation is extensible ;-)
> >>
> >> Like I said, for both projects I'm the only dev so there would be no
> >> IP problem. So if it's ok to you to not include lightning directly
> >> in DM I would be glad to contribute to the Apache Foundation :-)
> >>
> >> Cheers Chris
> >>
> >>
> >> Am 29.09.2012 16:53, schrieb Raffaele P. Guidi:
> >> > Ok so it's up to noctarius - your move! ;-) Regarding the new unsafe
> >> > storage: it's an opt-in feature that can be set with the fluent API
> (and
> >> > soon through the conference file).
> >> >
> >> > Ciao,
> >> > R
> >> > Il giorno 29/set/2012 16:45, "Olivier Lamy" <ol...@apache.org> ha
> >> scritto:
> >> >
> >> >> 2012/9/29 Raffaele P. Guidi <ra...@gmail.com>:
> >> >>>>> At least for the moment he can provide a patch to be integrated
> in DM
> >> >> :-)
> >> >>>
> >> >>> Sure, but as lightning is not in any public mvn repo should its
> code be
> >> >>> re-published in our svn? Or what?
> >> >> @Apache we don't care about binaries, only sources are important ! (a
> >> >> bit theorical for sure but that's it :-) ).
> >> >> So if Noctarius was the only guy who participate in lightning. He can
> >> >> just provide a patch we could integrate as a new dm module (note: the
> >> >> patch must not contains any more copyright and all sources must have
> >> >> ASF licenses).
> >> >>
> >> >> "Copyright 2012 the original author or authors." must be removed.
> >> >> And BTW package must be changed :-) (com.github is not acceptable
> @asf
> >> >> :-) )(@Noctarius are you working for github ? :-) )
> >> >>
> >> >> And having him as a committer will be only a matter of voting (we
> have
> >> >> a great chair who take cares of administrative stuff :P )
> >> >>
> >> >> If some others have participated in the project, we must pass tru an
> >> >> ip clearance mechanism
> >> >> (http://incubator.apache.org/ip-clearance/index.html) and all
> >> >> contributors to lightning must provide a cla. (It it's the case I can
> >> >> help)
> >> >>
> >> >>>
> >> >>>>> perso I'd like we avoid hard dependency on Unsafe as maybe some
> use
> >> >>> other jdks :-)
> >> >>>
> >> >>> Well, I believe Unsafe is supported by Sun JDK, OpenJDK, IBM JDK and
> >> >>> JRockit - and I believe that it is more than enough. Also keep in
> mind
> >> >> that
> >> >>> we already have an alternative Unsafe based memory storage - and
> >> although
> >> >>> it's not thoroughly tested for performance it dramaticaly simplifies
> >> >> code,
> >> >>> I have great expectations about it.
> >> >> Me too :-). Yup definitely more simple and faster !
> >> >> But we must provide a switch off configuration mechanism if some
> users
> >> >> don't want to use that (because Unsafe is just "Unsafe" :-) )
> >> >> And sorry I didn't have a look yet at your changes with using Unsafe.
> >> >>>
> >> >>> Cheers,
> >> >>> R
> >> >>>
> >> >>> On Sat, Sep 29, 2012 at 4:03 PM, Olivier Lamy <ol...@apache.org>
> >> wrote:
> >> >>>
> >> >>>> 2012/9/29 Raffaele P. Guidi <ra...@gmail.com>:
> >> >>>>> What do you think about:
> >> >>>>> 1) implementing a lightning serialization module
> >> >>>>> 2) creating a serializer that directly works on a directmemory
> >> >> provider
> >> >>>>> ByteBuffer or (maybe better) Unsafe based Pointer?
> >> >>>>
> >> >>>> Sounds good (perso I'd like we avoid hard dependency on Unsafe as
> >> >>>> maybe some use other jdks :-) )
> >> >>>>>
> >> >>>>> Now I see lightning is apache licensed and this is fine but I
> don't
> >> >> think
> >> >>>>> it is published in any public maven repo, am I right? We could
> find a
> >> >> way
> >> >>>>> to deal with this; options vary from publishing lightning to the
> free
> >> >>>>> sonatype repo, joining the ASF (which is great anyhow!) and
> >> >> republishing
> >> >>>>> lightning code in DirectMemory becoming a commiter (which has to
> >> >> undergo
> >> >>>> a
> >> >>>>> PMC vote).
> >> >>>>
> >> >>>> At least for the moment he can provide a patch to be integrated in
> DM
> >> >> :-)
> >> >>>>
> >> >>>>>
> >> >>>>> I'd like to hear your and the team feelings on this.
> >> >>>>>
> >> >>>>> Thanks,
> >> >>>>> Raffaele
> >> >>>>>
> >> >>>>> On Sat, Sep 29, 2012 at 3:27 PM, Noctarius <me...@noctarius.com>
> wrote:
> >> >>>>>
> >> >>>>>> Hey Raffaele,
> >> >>>>>>
> >> >>>>>> that's quite similar to what I did at work. We're developing
> Flash
> >> >>>>>> online games and using a customized AMF serialization. To support
> >> >>>>>> async writing of the clients event results I added serialization
> of
> >> >>>>>> the components / entities to the players zone calculation and
> stored
> >> >>>>>> the pre-serialized bytestream directly to the off-heap (using
> >> >>>>>> direct-ring-cache implementation). When the client requests the
> >> >>>>>> results (using long-polling) I just write the pre-serialized
> data to
> >> >>>>>> the right position to deserialize it by standard ways on Flash
> side.
> >> >>>>>>
> >> >>>>>> So yeah an seriliaztion to off-heap algorithm would be fine. You
> can
> >> >>>>>> avoid using byte arrays and minimalize GC.
> >> >>>>>>
> >> >>>>>> Cheers
> >> >>>>>> Chris
> >> >>>>>>
> >> >>>>>> Am 29.09.2012 15:02, schrieb Raffaele P. Guidi:
> >> >>>>>>> Nice to hear back from you! Yes, I was thinking about creating a
> >> >> new
> >> >>>>>> memory
> >> >>>>>>> storage implementation using Unsafe (and I did it, recently) and
> >> >>>> also, as
> >> >>>>>>> DirectMemory relies heavily on serialization (and supports many
> of
> >> >>>> them,
> >> >>>>>>> protostuff, protobuf, msgpack and of course standard
> >> >> serialization),
> >> >>>>>> about
> >> >>>>>>> creating a simple embedded serializer leveraging the same
> >> >> techniques
> >> >>>> you
> >> >>>>>>> used (Unsafe and bytecode generation).
> >> >>>>>>> The idea with embedding is avoiding to serialize in a byte array
> >> >> and
> >> >>>> then
> >> >>>>>>> moving the byte array to off-heap memory (via Unsafe or
> >> >> ByteBuffers),
> >> >>>> and
> >> >>>>>>> serializing directly into a "managed" off-heap buffer, thus
> further
> >> >>>>>>> optimizing heap utilization (less GC).
> >> >>>>>>>
> >> >>>>>>> What do you think about it? Does it make any sense to you?
> >> >>>>>>>
> >> >>>>>>> Ciao,
> >> >>>>>>> R
> >> >>>>>>>
> >> >>>>>>> On Sat, Sep 29, 2012 at 2:40 PM, Noctarius <me...@noctarius.com>
> >> >> wrote:
> >> >>>>>>>
> >> >>>>>>>> Hey guys,
> >> >>>>>>>>
> >> >>>>>>>> Raffaele found out about a project of mine (Lightning) a few
> weeks
> >> >>>>>>>> ago. Lightning is a heavy Unsafe and Bytecode generation using
> >> >>>>>>>> Serializer implementation. He told me that he was interested in
> >> >>>>>>>> adding something similar to DirectMemory and I would be glad to
> >> >> help
> >> >>>>>>>> out in this.
> >> >>>>>>>>
> >> >>>>>>>> Another project I started a few days ago, since it was needed
> for
> >> >>>>>>>> work is DirectRingCache. The name not really meets to actual
> >> >>>>>>>> implementation since it's not yet a ring buffer using cache. I
> >> >> used
> >> >>>>>>>> this for a pre-serialization simple bytestream cache with
> >> >>>>>>>> self-growing buffers. It could be nice to have DirectMemory
> having
> >> >>>>>>>> raw "buffers" to write to or to read from.
> >> >>>>>>>>
> >> >>>>>>>> Here are the links from the projects:
> >> >>>>>>>> https://github.com/noctarius/Lightning
> >> >>>>>>>> https://github.com/noctarius/direct-ring-cache
> >> >>>>>>>>
> >> >>>>>>>> It would be nice to help out since I really like the idea of
> >> >>>>>>>> DirectMemory and since direct-ring-cache is some kind of
> >> >> reinventing
> >> >>>>>>>> the wheel.
> >> >>>>>>>>
> >> >>>>>>>> Cheers
> >> >>>>>>>> Noctarius (Chris)
> >> >>>>>>>>
> >> >>>>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>> --
> >> >>>> Olivier Lamy
> >> >>>> Talend: http://coders.talend.com
> >> >>>> http://twitter.com/olamy | http://linkedin.com/in/olamy
> >> >>>>
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Olivier Lamy
> >> >> Talend: http://coders.talend.com
> >> >> http://twitter.com/olamy | http://linkedin.com/in/olamy
> >> >>
> >> >
> >>
>
>
>
> --
> Olivier Lamy
> Talend: http://coders.talend.com
> http://twitter.com/olamy | http://linkedin.com/in/olamy
>
Re: Additional Serializer and raw Buffer access
Posted by Noctarius <me...@noctarius.com>.
If that is possible there's no problem from my side. It would be
an honor to give the Apache Foundation something instead of only
taking code :-)
PS: I'm pretty sure there's a way to make lightning again a little
bit faster by removing boxing on primitives by using
invokedynamic. I started it but left the idea behind to finish the
public api first.
Am 29.09.2012 17:58, schrieb Olivier Lamy:
> 2012/9/29 Raffaele P. Guidi <ra...@gmail.com>:
>> Well we already have a NIO ready interface allowing direct
>> access to DMs managed bytebuffers but I think this is just
>> half way to what could be achieved optimally blending
>> serialization and memory allocation together.
>>
>> Lightning as a module is of course a good idea and it could
>> easily evolve as a subproject (for the more experienced asf
>> members: is it a feasible way?).
> Nothing prevent to have
> http://svn.apache.org/repos/asf/directmemory/lightning/trunk
> with a package like: o.a.d.lightning That will be a
> subproject.
>>
>> Ciao, R Il giorno 29/set/2012 17:44, "Noctarius"
>> <me...@noctarius.com> ha scritto:
>>
>>> Hey guys,
>>>
>>> ok there's no lightning binary available since lightning
>>> wasn't ready yet for releasing.
>>>
>>> For being the only developer it would be no problem to
>>> contribute the sourcebase for DirectMemory but I'm not sure
>>> yet if it wouldn't be better to seperate it to be available
>>> without using DirectMemory itself. I started it as a
>>> serializer for cluster synchronization, but it would be
>>> cool to contribute lightning as a subproject to
>>> DirectMemory :-)
>>>
>>> About the second project I would love to see a public
>>> available buffer API directly in DirectMemory so that
>>> project would be nearly needless :-) The only difference I
>>> think is the allocation strategy my implementation is using
>>> against the one DirectMemory has, but I'm pretty sure the
>>> allocation is extensible ;-)
>>>
>>> Like I said, for both projects I'm the only dev so there
>>> would be no IP problem. So if it's ok to you to not include
>>> lightning directly in DM I would be glad to contribute to
>>> the Apache Foundation :-)
>>>
>>> Cheers Chris
>>>
>>>
>>> Am 29.09.2012 16:53, schrieb Raffaele P. Guidi:
>>>> Ok so it's up to noctarius - your move! ;-) Regarding the
>>>> new unsafe storage: it's an opt-in feature that can be
>>>> set with the fluent API (and soon through the conference
>>>> file).
>>>>
>>>> Ciao, R Il giorno 29/set/2012 16:45, "Olivier Lamy"
>>>> <ol...@apache.org> ha
>>> scritto:
>>>>
>>>>> 2012/9/29 Raffaele P. Guidi
>>>>> <ra...@gmail.com>:
>>>>>>>> At least for the moment he can provide a patch to
>>>>>>>> be integrated in DM
>>>>> :-)
>>>>>>
>>>>>> Sure, but as lightning is not in any public mvn repo
>>>>>> should its code be re-published in our svn? Or what?
>>>>> @Apache we don't care about binaries, only sources are
>>>>> important ! (a bit theorical for sure but that's it :-)
>>>>> ). So if Noctarius was the only guy who participate in
>>>>> lightning. He can just provide a patch we could
>>>>> integrate as a new dm module (note: the patch must not
>>>>> contains any more copyright and all sources must have
>>>>> ASF licenses).
>>>>>
>>>>> "Copyright 2012 the original author or authors." must
>>>>> be removed. And BTW package must be changed :-)
>>>>> (com.github is not acceptable @asf :-) )(@Noctarius are
>>>>> you working for github ? :-) )
>>>>>
>>>>> And having him as a committer will be only a matter of
>>>>> voting (we have a great chair who take cares of
>>>>> administrative stuff :P )
>>>>>
>>>>> If some others have participated in the project, we
>>>>> must pass tru an ip clearance mechanism
>>>>> (http://incubator.apache.org/ip-clearance/index.html)
>>>>> and all contributors to lightning must provide a cla.
>>>>> (It it's the case I can help)
>>>>>
>>>>>>
>>>>>>>> perso I'd like we avoid hard dependency on Unsafe
>>>>>>>> as maybe some use
>>>>>> other jdks :-)
>>>>>>
>>>>>> Well, I believe Unsafe is supported by Sun JDK,
>>>>>> OpenJDK, IBM JDK and JRockit - and I believe that it
>>>>>> is more than enough. Also keep in mind
>>>>> that
>>>>>> we already have an alternative Unsafe based memory
>>>>>> storage - and
>>> although
>>>>>> it's not thoroughly tested for performance it
>>>>>> dramaticaly simplifies
>>>>> code,
>>>>>> I have great expectations about it.
>>>>> Me too :-). Yup definitely more simple and faster ! But
>>>>> we must provide a switch off configuration mechanism if
>>>>> some users don't want to use that (because Unsafe is
>>>>> just "Unsafe" :-) ) And sorry I didn't have a look yet
>>>>> at your changes with using Unsafe.
>>>>>>
>>>>>> Cheers, R
>>>>>>
>>>>>> On Sat, Sep 29, 2012 at 4:03 PM, Olivier Lamy
>>>>>> <ol...@apache.org>
>>> wrote:
>>>>>>
>>>>>>> 2012/9/29 Raffaele P. Guidi
>>>>>>> <ra...@gmail.com>:
>>>>>>>> What do you think about: 1) implementing a
>>>>>>>> lightning serialization module 2) creating a
>>>>>>>> serializer that directly works on a directmemory
>>>>> provider
>>>>>>>> ByteBuffer or (maybe better) Unsafe based
>>>>>>>> Pointer?
>>>>>>>
>>>>>>> Sounds good (perso I'd like we avoid hard
>>>>>>> dependency on Unsafe as maybe some use other jdks
>>>>>>> :-) )
>>>>>>>>
>>>>>>>> Now I see lightning is apache licensed and this
>>>>>>>> is fine but I don't
>>>>> think
>>>>>>>> it is published in any public maven repo, am I
>>>>>>>> right? We could find a
>>>>> way
>>>>>>>> to deal with this; options vary from publishing
>>>>>>>> lightning to the free sonatype repo, joining the
>>>>>>>> ASF (which is great anyhow!) and
>>>>> republishing
>>>>>>>> lightning code in DirectMemory becoming a
>>>>>>>> commiter (which has to
>>>>> undergo
>>>>>>> a
>>>>>>>> PMC vote).
>>>>>>>
>>>>>>> At least for the moment he can provide a patch to
>>>>>>> be integrated in DM
>>>>> :-)
>>>>>>>
>>>>>>>>
>>>>>>>> I'd like to hear your and the team feelings on
>>>>>>>> this.
>>>>>>>>
>>>>>>>> Thanks, Raffaele
>>>>>>>>
>>>>>>>> On Sat, Sep 29, 2012 at 3:27 PM, Noctarius
>>>>>>>> <me...@noctarius.com> wrote:
>>>>>>>>
>>>>>>>>> Hey Raffaele,
>>>>>>>>>
>>>>>>>>> that's quite similar to what I did at work.
>>>>>>>>> We're developing Flash online games and using a
>>>>>>>>> customized AMF serialization. To support async
>>>>>>>>> writing of the clients event results I added
>>>>>>>>> serialization of the components / entities to
>>>>>>>>> the players zone calculation and stored the
>>>>>>>>> pre-serialized bytestream directly to the
>>>>>>>>> off-heap (using direct-ring-cache
>>>>>>>>> implementation). When the client requests the
>>>>>>>>> results (using long-polling) I just write the
>>>>>>>>> pre-serialized data to the right position to
>>>>>>>>> deserialize it by standard ways on Flash side.
>>>>>>>>>
>>>>>>>>> So yeah an seriliaztion to off-heap algorithm
>>>>>>>>> would be fine. You can avoid using byte arrays
>>>>>>>>> and minimalize GC.
>>>>>>>>>
>>>>>>>>> Cheers Chris
>>>>>>>>>
>>>>>>>>> Am 29.09.2012 15:02, schrieb Raffaele P.
>>>>>>>>> Guidi:
>>>>>>>>>> Nice to hear back from you! Yes, I was
>>>>>>>>>> thinking about creating a
>>>>> new
>>>>>>>>> memory
>>>>>>>>>> storage implementation using Unsafe (and I
>>>>>>>>>> did it, recently) and
>>>>>>> also, as
>>>>>>>>>> DirectMemory relies heavily on serialization
>>>>>>>>>> (and supports many of
>>>>>>> them,
>>>>>>>>>> protostuff, protobuf, msgpack and of course
>>>>>>>>>> standard
>>>>> serialization),
>>>>>>>>> about
>>>>>>>>>> creating a simple embedded serializer
>>>>>>>>>> leveraging the same
>>>>> techniques
>>>>>>> you
>>>>>>>>>> used (Unsafe and bytecode generation). The
>>>>>>>>>> idea with embedding is avoiding to serialize
>>>>>>>>>> in a byte array
>>>>> and
>>>>>>> then
>>>>>>>>>> moving the byte array to off-heap memory (via
>>>>>>>>>> Unsafe or
>>>>> ByteBuffers),
>>>>>>> and
>>>>>>>>>> serializing directly into a "managed"
>>>>>>>>>> off-heap buffer, thus further optimizing heap
>>>>>>>>>> utilization (less GC).
>>>>>>>>>>
>>>>>>>>>> What do you think about it? Does it make any
>>>>>>>>>> sense to you?
>>>>>>>>>>
>>>>>>>>>> Ciao, R
>>>>>>>>>>
>>>>>>>>>> On Sat, Sep 29, 2012 at 2:40 PM, Noctarius
>>>>>>>>>> <me...@noctarius.com>
>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hey guys,
>>>>>>>>>>>
>>>>>>>>>>> Raffaele found out about a project of mine
>>>>>>>>>>> (Lightning) a few weeks ago. Lightning is a
>>>>>>>>>>> heavy Unsafe and Bytecode generation using
>>>>>>>>>>> Serializer implementation. He told me that
>>>>>>>>>>> he was interested in adding something
>>>>>>>>>>> similar to DirectMemory and I would be glad
>>>>>>>>>>> to
>>>>> help
>>>>>>>>>>> out in this.
>>>>>>>>>>>
>>>>>>>>>>> Another project I started a few days ago,
>>>>>>>>>>> since it was needed for work is
>>>>>>>>>>> DirectRingCache. The name not really meets
>>>>>>>>>>> to actual implementation since it's not yet
>>>>>>>>>>> a ring buffer using cache. I
>>>>> used
>>>>>>>>>>> this for a pre-serialization simple
>>>>>>>>>>> bytestream cache with self-growing buffers.
>>>>>>>>>>> It could be nice to have DirectMemory
>>>>>>>>>>> having raw "buffers" to write to or to read
>>>>>>>>>>> from.
>>>>>>>>>>>
>>>>>>>>>>> Here are the links from the projects:
>>>>>>>>>>> https://github.com/noctarius/Lightning
>>>>>>>>>>> https://github.com/noctarius/direct-ring-cache
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
It would be nice to help out since I really like the idea of
>>>>>>>>>>> DirectMemory and since direct-ring-cache is
>>>>>>>>>>> some kind of
>>>>> reinventing
>>>>>>>>>>> the wheel.
>>>>>>>>>>>
>>>>>>>>>>> Cheers Noctarius (Chris)
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> -- Olivier Lamy Talend: http://coders.talend.com
>>>>>>> http://twitter.com/olamy |
>>>>>>> http://linkedin.com/in/olamy
>>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> -- Olivier Lamy Talend: http://coders.talend.com
>>>>> http://twitter.com/olamy |
>>>>> http://linkedin.com/in/olamy
>>>>>
>>>>
>>>
>
>
>
Re: Additional Serializer and raw Buffer access
Posted by Olivier Lamy <ol...@apache.org>.
2012/9/29 Raffaele P. Guidi <ra...@gmail.com>:
> Well we already have a NIO ready interface allowing direct access to DMs
> managed bytebuffers but I think this is just half way to what could be
> achieved optimally blending serialization and memory allocation together.
>
> Lightning as a module is of course a good idea and it could easily evolve
> as a subproject (for the more experienced asf members: is it a feasible
> way?).
Nothing prevent to have
http://svn.apache.org/repos/asf/directmemory/lightning/trunk
with a package like: o.a.d.lightning
That will be a subproject.
>
> Ciao,
> R
> Il giorno 29/set/2012 17:44, "Noctarius" <me...@noctarius.com> ha scritto:
>
>> Hey guys,
>>
>> ok there's no lightning binary available since lightning wasn't
>> ready yet for releasing.
>>
>> For being the only developer it would be no problem to contribute
>> the sourcebase for DirectMemory but I'm not sure yet if it wouldn't
>> be better to seperate it to be available without using DirectMemory
>> itself. I started it as a serializer for cluster synchronization,
>> but it would be cool to contribute lightning as a subproject to
>> DirectMemory :-)
>>
>> About the second project I would love to see a public available
>> buffer API directly in DirectMemory so that project would be nearly
>> needless :-) The only difference I think is the allocation strategy
>> my implementation is using against the one DirectMemory has, but I'm
>> pretty sure the allocation is extensible ;-)
>>
>> Like I said, for both projects I'm the only dev so there would be no
>> IP problem. So if it's ok to you to not include lightning directly
>> in DM I would be glad to contribute to the Apache Foundation :-)
>>
>> Cheers Chris
>>
>>
>> Am 29.09.2012 16:53, schrieb Raffaele P. Guidi:
>> > Ok so it's up to noctarius - your move! ;-) Regarding the new unsafe
>> > storage: it's an opt-in feature that can be set with the fluent API (and
>> > soon through the conference file).
>> >
>> > Ciao,
>> > R
>> > Il giorno 29/set/2012 16:45, "Olivier Lamy" <ol...@apache.org> ha
>> scritto:
>> >
>> >> 2012/9/29 Raffaele P. Guidi <ra...@gmail.com>:
>> >>>>> At least for the moment he can provide a patch to be integrated in DM
>> >> :-)
>> >>>
>> >>> Sure, but as lightning is not in any public mvn repo should its code be
>> >>> re-published in our svn? Or what?
>> >> @Apache we don't care about binaries, only sources are important ! (a
>> >> bit theorical for sure but that's it :-) ).
>> >> So if Noctarius was the only guy who participate in lightning. He can
>> >> just provide a patch we could integrate as a new dm module (note: the
>> >> patch must not contains any more copyright and all sources must have
>> >> ASF licenses).
>> >>
>> >> "Copyright 2012 the original author or authors." must be removed.
>> >> And BTW package must be changed :-) (com.github is not acceptable @asf
>> >> :-) )(@Noctarius are you working for github ? :-) )
>> >>
>> >> And having him as a committer will be only a matter of voting (we have
>> >> a great chair who take cares of administrative stuff :P )
>> >>
>> >> If some others have participated in the project, we must pass tru an
>> >> ip clearance mechanism
>> >> (http://incubator.apache.org/ip-clearance/index.html) and all
>> >> contributors to lightning must provide a cla. (It it's the case I can
>> >> help)
>> >>
>> >>>
>> >>>>> perso I'd like we avoid hard dependency on Unsafe as maybe some use
>> >>> other jdks :-)
>> >>>
>> >>> Well, I believe Unsafe is supported by Sun JDK, OpenJDK, IBM JDK and
>> >>> JRockit - and I believe that it is more than enough. Also keep in mind
>> >> that
>> >>> we already have an alternative Unsafe based memory storage - and
>> although
>> >>> it's not thoroughly tested for performance it dramaticaly simplifies
>> >> code,
>> >>> I have great expectations about it.
>> >> Me too :-). Yup definitely more simple and faster !
>> >> But we must provide a switch off configuration mechanism if some users
>> >> don't want to use that (because Unsafe is just "Unsafe" :-) )
>> >> And sorry I didn't have a look yet at your changes with using Unsafe.
>> >>>
>> >>> Cheers,
>> >>> R
>> >>>
>> >>> On Sat, Sep 29, 2012 at 4:03 PM, Olivier Lamy <ol...@apache.org>
>> wrote:
>> >>>
>> >>>> 2012/9/29 Raffaele P. Guidi <ra...@gmail.com>:
>> >>>>> What do you think about:
>> >>>>> 1) implementing a lightning serialization module
>> >>>>> 2) creating a serializer that directly works on a directmemory
>> >> provider
>> >>>>> ByteBuffer or (maybe better) Unsafe based Pointer?
>> >>>>
>> >>>> Sounds good (perso I'd like we avoid hard dependency on Unsafe as
>> >>>> maybe some use other jdks :-) )
>> >>>>>
>> >>>>> Now I see lightning is apache licensed and this is fine but I don't
>> >> think
>> >>>>> it is published in any public maven repo, am I right? We could find a
>> >> way
>> >>>>> to deal with this; options vary from publishing lightning to the free
>> >>>>> sonatype repo, joining the ASF (which is great anyhow!) and
>> >> republishing
>> >>>>> lightning code in DirectMemory becoming a commiter (which has to
>> >> undergo
>> >>>> a
>> >>>>> PMC vote).
>> >>>>
>> >>>> At least for the moment he can provide a patch to be integrated in DM
>> >> :-)
>> >>>>
>> >>>>>
>> >>>>> I'd like to hear your and the team feelings on this.
>> >>>>>
>> >>>>> Thanks,
>> >>>>> Raffaele
>> >>>>>
>> >>>>> On Sat, Sep 29, 2012 at 3:27 PM, Noctarius <me...@noctarius.com> wrote:
>> >>>>>
>> >>>>>> Hey Raffaele,
>> >>>>>>
>> >>>>>> that's quite similar to what I did at work. We're developing Flash
>> >>>>>> online games and using a customized AMF serialization. To support
>> >>>>>> async writing of the clients event results I added serialization of
>> >>>>>> the components / entities to the players zone calculation and stored
>> >>>>>> the pre-serialized bytestream directly to the off-heap (using
>> >>>>>> direct-ring-cache implementation). When the client requests the
>> >>>>>> results (using long-polling) I just write the pre-serialized data to
>> >>>>>> the right position to deserialize it by standard ways on Flash side.
>> >>>>>>
>> >>>>>> So yeah an seriliaztion to off-heap algorithm would be fine. You can
>> >>>>>> avoid using byte arrays and minimalize GC.
>> >>>>>>
>> >>>>>> Cheers
>> >>>>>> Chris
>> >>>>>>
>> >>>>>> Am 29.09.2012 15:02, schrieb Raffaele P. Guidi:
>> >>>>>>> Nice to hear back from you! Yes, I was thinking about creating a
>> >> new
>> >>>>>> memory
>> >>>>>>> storage implementation using Unsafe (and I did it, recently) and
>> >>>> also, as
>> >>>>>>> DirectMemory relies heavily on serialization (and supports many of
>> >>>> them,
>> >>>>>>> protostuff, protobuf, msgpack and of course standard
>> >> serialization),
>> >>>>>> about
>> >>>>>>> creating a simple embedded serializer leveraging the same
>> >> techniques
>> >>>> you
>> >>>>>>> used (Unsafe and bytecode generation).
>> >>>>>>> The idea with embedding is avoiding to serialize in a byte array
>> >> and
>> >>>> then
>> >>>>>>> moving the byte array to off-heap memory (via Unsafe or
>> >> ByteBuffers),
>> >>>> and
>> >>>>>>> serializing directly into a "managed" off-heap buffer, thus further
>> >>>>>>> optimizing heap utilization (less GC).
>> >>>>>>>
>> >>>>>>> What do you think about it? Does it make any sense to you?
>> >>>>>>>
>> >>>>>>> Ciao,
>> >>>>>>> R
>> >>>>>>>
>> >>>>>>> On Sat, Sep 29, 2012 at 2:40 PM, Noctarius <me...@noctarius.com>
>> >> wrote:
>> >>>>>>>
>> >>>>>>>> Hey guys,
>> >>>>>>>>
>> >>>>>>>> Raffaele found out about a project of mine (Lightning) a few weeks
>> >>>>>>>> ago. Lightning is a heavy Unsafe and Bytecode generation using
>> >>>>>>>> Serializer implementation. He told me that he was interested in
>> >>>>>>>> adding something similar to DirectMemory and I would be glad to
>> >> help
>> >>>>>>>> out in this.
>> >>>>>>>>
>> >>>>>>>> Another project I started a few days ago, since it was needed for
>> >>>>>>>> work is DirectRingCache. The name not really meets to actual
>> >>>>>>>> implementation since it's not yet a ring buffer using cache. I
>> >> used
>> >>>>>>>> this for a pre-serialization simple bytestream cache with
>> >>>>>>>> self-growing buffers. It could be nice to have DirectMemory having
>> >>>>>>>> raw "buffers" to write to or to read from.
>> >>>>>>>>
>> >>>>>>>> Here are the links from the projects:
>> >>>>>>>> https://github.com/noctarius/Lightning
>> >>>>>>>> https://github.com/noctarius/direct-ring-cache
>> >>>>>>>>
>> >>>>>>>> It would be nice to help out since I really like the idea of
>> >>>>>>>> DirectMemory and since direct-ring-cache is some kind of
>> >> reinventing
>> >>>>>>>> the wheel.
>> >>>>>>>>
>> >>>>>>>> Cheers
>> >>>>>>>> Noctarius (Chris)
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> --
>> >>>> Olivier Lamy
>> >>>> Talend: http://coders.talend.com
>> >>>> http://twitter.com/olamy | http://linkedin.com/in/olamy
>> >>>>
>> >>
>> >>
>> >>
>> >> --
>> >> Olivier Lamy
>> >> Talend: http://coders.talend.com
>> >> http://twitter.com/olamy | http://linkedin.com/in/olamy
>> >>
>> >
>>
--
Olivier Lamy
Talend: http://coders.talend.com
http://twitter.com/olamy | http://linkedin.com/in/olamy
Re: Additional Serializer and raw Buffer access
Posted by "Raffaele P. Guidi" <ra...@gmail.com>.
Well we already have a NIO ready interface allowing direct access to DMs
managed bytebuffers but I think this is just half way to what could be
achieved optimally blending serialization and memory allocation together.
Lightning as a module is of course a good idea and it could easily evolve
as a subproject (for the more experienced asf members: is it a feasible
way?).
Ciao,
R
Il giorno 29/set/2012 17:44, "Noctarius" <me...@noctarius.com> ha scritto:
> Hey guys,
>
> ok there's no lightning binary available since lightning wasn't
> ready yet for releasing.
>
> For being the only developer it would be no problem to contribute
> the sourcebase for DirectMemory but I'm not sure yet if it wouldn't
> be better to seperate it to be available without using DirectMemory
> itself. I started it as a serializer for cluster synchronization,
> but it would be cool to contribute lightning as a subproject to
> DirectMemory :-)
>
> About the second project I would love to see a public available
> buffer API directly in DirectMemory so that project would be nearly
> needless :-) The only difference I think is the allocation strategy
> my implementation is using against the one DirectMemory has, but I'm
> pretty sure the allocation is extensible ;-)
>
> Like I said, for both projects I'm the only dev so there would be no
> IP problem. So if it's ok to you to not include lightning directly
> in DM I would be glad to contribute to the Apache Foundation :-)
>
> Cheers Chris
>
>
> Am 29.09.2012 16:53, schrieb Raffaele P. Guidi:
> > Ok so it's up to noctarius - your move! ;-) Regarding the new unsafe
> > storage: it's an opt-in feature that can be set with the fluent API (and
> > soon through the conference file).
> >
> > Ciao,
> > R
> > Il giorno 29/set/2012 16:45, "Olivier Lamy" <ol...@apache.org> ha
> scritto:
> >
> >> 2012/9/29 Raffaele P. Guidi <ra...@gmail.com>:
> >>>>> At least for the moment he can provide a patch to be integrated in DM
> >> :-)
> >>>
> >>> Sure, but as lightning is not in any public mvn repo should its code be
> >>> re-published in our svn? Or what?
> >> @Apache we don't care about binaries, only sources are important ! (a
> >> bit theorical for sure but that's it :-) ).
> >> So if Noctarius was the only guy who participate in lightning. He can
> >> just provide a patch we could integrate as a new dm module (note: the
> >> patch must not contains any more copyright and all sources must have
> >> ASF licenses).
> >>
> >> "Copyright 2012 the original author or authors." must be removed.
> >> And BTW package must be changed :-) (com.github is not acceptable @asf
> >> :-) )(@Noctarius are you working for github ? :-) )
> >>
> >> And having him as a committer will be only a matter of voting (we have
> >> a great chair who take cares of administrative stuff :P )
> >>
> >> If some others have participated in the project, we must pass tru an
> >> ip clearance mechanism
> >> (http://incubator.apache.org/ip-clearance/index.html) and all
> >> contributors to lightning must provide a cla. (It it's the case I can
> >> help)
> >>
> >>>
> >>>>> perso I'd like we avoid hard dependency on Unsafe as maybe some use
> >>> other jdks :-)
> >>>
> >>> Well, I believe Unsafe is supported by Sun JDK, OpenJDK, IBM JDK and
> >>> JRockit - and I believe that it is more than enough. Also keep in mind
> >> that
> >>> we already have an alternative Unsafe based memory storage - and
> although
> >>> it's not thoroughly tested for performance it dramaticaly simplifies
> >> code,
> >>> I have great expectations about it.
> >> Me too :-). Yup definitely more simple and faster !
> >> But we must provide a switch off configuration mechanism if some users
> >> don't want to use that (because Unsafe is just "Unsafe" :-) )
> >> And sorry I didn't have a look yet at your changes with using Unsafe.
> >>>
> >>> Cheers,
> >>> R
> >>>
> >>> On Sat, Sep 29, 2012 at 4:03 PM, Olivier Lamy <ol...@apache.org>
> wrote:
> >>>
> >>>> 2012/9/29 Raffaele P. Guidi <ra...@gmail.com>:
> >>>>> What do you think about:
> >>>>> 1) implementing a lightning serialization module
> >>>>> 2) creating a serializer that directly works on a directmemory
> >> provider
> >>>>> ByteBuffer or (maybe better) Unsafe based Pointer?
> >>>>
> >>>> Sounds good (perso I'd like we avoid hard dependency on Unsafe as
> >>>> maybe some use other jdks :-) )
> >>>>>
> >>>>> Now I see lightning is apache licensed and this is fine but I don't
> >> think
> >>>>> it is published in any public maven repo, am I right? We could find a
> >> way
> >>>>> to deal with this; options vary from publishing lightning to the free
> >>>>> sonatype repo, joining the ASF (which is great anyhow!) and
> >> republishing
> >>>>> lightning code in DirectMemory becoming a commiter (which has to
> >> undergo
> >>>> a
> >>>>> PMC vote).
> >>>>
> >>>> At least for the moment he can provide a patch to be integrated in DM
> >> :-)
> >>>>
> >>>>>
> >>>>> I'd like to hear your and the team feelings on this.
> >>>>>
> >>>>> Thanks,
> >>>>> Raffaele
> >>>>>
> >>>>> On Sat, Sep 29, 2012 at 3:27 PM, Noctarius <me...@noctarius.com> wrote:
> >>>>>
> >>>>>> Hey Raffaele,
> >>>>>>
> >>>>>> that's quite similar to what I did at work. We're developing Flash
> >>>>>> online games and using a customized AMF serialization. To support
> >>>>>> async writing of the clients event results I added serialization of
> >>>>>> the components / entities to the players zone calculation and stored
> >>>>>> the pre-serialized bytestream directly to the off-heap (using
> >>>>>> direct-ring-cache implementation). When the client requests the
> >>>>>> results (using long-polling) I just write the pre-serialized data to
> >>>>>> the right position to deserialize it by standard ways on Flash side.
> >>>>>>
> >>>>>> So yeah an seriliaztion to off-heap algorithm would be fine. You can
> >>>>>> avoid using byte arrays and minimalize GC.
> >>>>>>
> >>>>>> Cheers
> >>>>>> Chris
> >>>>>>
> >>>>>> Am 29.09.2012 15:02, schrieb Raffaele P. Guidi:
> >>>>>>> Nice to hear back from you! Yes, I was thinking about creating a
> >> new
> >>>>>> memory
> >>>>>>> storage implementation using Unsafe (and I did it, recently) and
> >>>> also, as
> >>>>>>> DirectMemory relies heavily on serialization (and supports many of
> >>>> them,
> >>>>>>> protostuff, protobuf, msgpack and of course standard
> >> serialization),
> >>>>>> about
> >>>>>>> creating a simple embedded serializer leveraging the same
> >> techniques
> >>>> you
> >>>>>>> used (Unsafe and bytecode generation).
> >>>>>>> The idea with embedding is avoiding to serialize in a byte array
> >> and
> >>>> then
> >>>>>>> moving the byte array to off-heap memory (via Unsafe or
> >> ByteBuffers),
> >>>> and
> >>>>>>> serializing directly into a "managed" off-heap buffer, thus further
> >>>>>>> optimizing heap utilization (less GC).
> >>>>>>>
> >>>>>>> What do you think about it? Does it make any sense to you?
> >>>>>>>
> >>>>>>> Ciao,
> >>>>>>> R
> >>>>>>>
> >>>>>>> On Sat, Sep 29, 2012 at 2:40 PM, Noctarius <me...@noctarius.com>
> >> wrote:
> >>>>>>>
> >>>>>>>> Hey guys,
> >>>>>>>>
> >>>>>>>> Raffaele found out about a project of mine (Lightning) a few weeks
> >>>>>>>> ago. Lightning is a heavy Unsafe and Bytecode generation using
> >>>>>>>> Serializer implementation. He told me that he was interested in
> >>>>>>>> adding something similar to DirectMemory and I would be glad to
> >> help
> >>>>>>>> out in this.
> >>>>>>>>
> >>>>>>>> Another project I started a few days ago, since it was needed for
> >>>>>>>> work is DirectRingCache. The name not really meets to actual
> >>>>>>>> implementation since it's not yet a ring buffer using cache. I
> >> used
> >>>>>>>> this for a pre-serialization simple bytestream cache with
> >>>>>>>> self-growing buffers. It could be nice to have DirectMemory having
> >>>>>>>> raw "buffers" to write to or to read from.
> >>>>>>>>
> >>>>>>>> Here are the links from the projects:
> >>>>>>>> https://github.com/noctarius/Lightning
> >>>>>>>> https://github.com/noctarius/direct-ring-cache
> >>>>>>>>
> >>>>>>>> It would be nice to help out since I really like the idea of
> >>>>>>>> DirectMemory and since direct-ring-cache is some kind of
> >> reinventing
> >>>>>>>> the wheel.
> >>>>>>>>
> >>>>>>>> Cheers
> >>>>>>>> Noctarius (Chris)
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Olivier Lamy
> >>>> Talend: http://coders.talend.com
> >>>> http://twitter.com/olamy | http://linkedin.com/in/olamy
> >>>>
> >>
> >>
> >>
> >> --
> >> Olivier Lamy
> >> Talend: http://coders.talend.com
> >> http://twitter.com/olamy | http://linkedin.com/in/olamy
> >>
> >
>
Re: Additional Serializer and raw Buffer access
Posted by Noctarius <me...@noctarius.com>.
Hey guys,
ok there's no lightning binary available since lightning wasn't
ready yet for releasing.
For being the only developer it would be no problem to contribute
the sourcebase for DirectMemory but I'm not sure yet if it wouldn't
be better to seperate it to be available without using DirectMemory
itself. I started it as a serializer for cluster synchronization,
but it would be cool to contribute lightning as a subproject to
DirectMemory :-)
About the second project I would love to see a public available
buffer API directly in DirectMemory so that project would be nearly
needless :-) The only difference I think is the allocation strategy
my implementation is using against the one DirectMemory has, but I'm
pretty sure the allocation is extensible ;-)
Like I said, for both projects I'm the only dev so there would be no
IP problem. So if it's ok to you to not include lightning directly
in DM I would be glad to contribute to the Apache Foundation :-)
Cheers Chris
Am 29.09.2012 16:53, schrieb Raffaele P. Guidi:
> Ok so it's up to noctarius - your move! ;-) Regarding the new unsafe
> storage: it's an opt-in feature that can be set with the fluent API (and
> soon through the conference file).
>
> Ciao,
> R
> Il giorno 29/set/2012 16:45, "Olivier Lamy" <ol...@apache.org> ha scritto:
>
>> 2012/9/29 Raffaele P. Guidi <ra...@gmail.com>:
>>>>> At least for the moment he can provide a patch to be integrated in DM
>> :-)
>>>
>>> Sure, but as lightning is not in any public mvn repo should its code be
>>> re-published in our svn? Or what?
>> @Apache we don't care about binaries, only sources are important ! (a
>> bit theorical for sure but that's it :-) ).
>> So if Noctarius was the only guy who participate in lightning. He can
>> just provide a patch we could integrate as a new dm module (note: the
>> patch must not contains any more copyright and all sources must have
>> ASF licenses).
>>
>> "Copyright 2012 the original author or authors." must be removed.
>> And BTW package must be changed :-) (com.github is not acceptable @asf
>> :-) )(@Noctarius are you working for github ? :-) )
>>
>> And having him as a committer will be only a matter of voting (we have
>> a great chair who take cares of administrative stuff :P )
>>
>> If some others have participated in the project, we must pass tru an
>> ip clearance mechanism
>> (http://incubator.apache.org/ip-clearance/index.html) and all
>> contributors to lightning must provide a cla. (It it's the case I can
>> help)
>>
>>>
>>>>> perso I'd like we avoid hard dependency on Unsafe as maybe some use
>>> other jdks :-)
>>>
>>> Well, I believe Unsafe is supported by Sun JDK, OpenJDK, IBM JDK and
>>> JRockit - and I believe that it is more than enough. Also keep in mind
>> that
>>> we already have an alternative Unsafe based memory storage - and although
>>> it's not thoroughly tested for performance it dramaticaly simplifies
>> code,
>>> I have great expectations about it.
>> Me too :-). Yup definitely more simple and faster !
>> But we must provide a switch off configuration mechanism if some users
>> don't want to use that (because Unsafe is just "Unsafe" :-) )
>> And sorry I didn't have a look yet at your changes with using Unsafe.
>>>
>>> Cheers,
>>> R
>>>
>>> On Sat, Sep 29, 2012 at 4:03 PM, Olivier Lamy <ol...@apache.org> wrote:
>>>
>>>> 2012/9/29 Raffaele P. Guidi <ra...@gmail.com>:
>>>>> What do you think about:
>>>>> 1) implementing a lightning serialization module
>>>>> 2) creating a serializer that directly works on a directmemory
>> provider
>>>>> ByteBuffer or (maybe better) Unsafe based Pointer?
>>>>
>>>> Sounds good (perso I'd like we avoid hard dependency on Unsafe as
>>>> maybe some use other jdks :-) )
>>>>>
>>>>> Now I see lightning is apache licensed and this is fine but I don't
>> think
>>>>> it is published in any public maven repo, am I right? We could find a
>> way
>>>>> to deal with this; options vary from publishing lightning to the free
>>>>> sonatype repo, joining the ASF (which is great anyhow!) and
>> republishing
>>>>> lightning code in DirectMemory becoming a commiter (which has to
>> undergo
>>>> a
>>>>> PMC vote).
>>>>
>>>> At least for the moment he can provide a patch to be integrated in DM
>> :-)
>>>>
>>>>>
>>>>> I'd like to hear your and the team feelings on this.
>>>>>
>>>>> Thanks,
>>>>> Raffaele
>>>>>
>>>>> On Sat, Sep 29, 2012 at 3:27 PM, Noctarius <me...@noctarius.com> wrote:
>>>>>
>>>>>> Hey Raffaele,
>>>>>>
>>>>>> that's quite similar to what I did at work. We're developing Flash
>>>>>> online games and using a customized AMF serialization. To support
>>>>>> async writing of the clients event results I added serialization of
>>>>>> the components / entities to the players zone calculation and stored
>>>>>> the pre-serialized bytestream directly to the off-heap (using
>>>>>> direct-ring-cache implementation). When the client requests the
>>>>>> results (using long-polling) I just write the pre-serialized data to
>>>>>> the right position to deserialize it by standard ways on Flash side.
>>>>>>
>>>>>> So yeah an seriliaztion to off-heap algorithm would be fine. You can
>>>>>> avoid using byte arrays and minimalize GC.
>>>>>>
>>>>>> Cheers
>>>>>> Chris
>>>>>>
>>>>>> Am 29.09.2012 15:02, schrieb Raffaele P. Guidi:
>>>>>>> Nice to hear back from you! Yes, I was thinking about creating a
>> new
>>>>>> memory
>>>>>>> storage implementation using Unsafe (and I did it, recently) and
>>>> also, as
>>>>>>> DirectMemory relies heavily on serialization (and supports many of
>>>> them,
>>>>>>> protostuff, protobuf, msgpack and of course standard
>> serialization),
>>>>>> about
>>>>>>> creating a simple embedded serializer leveraging the same
>> techniques
>>>> you
>>>>>>> used (Unsafe and bytecode generation).
>>>>>>> The idea with embedding is avoiding to serialize in a byte array
>> and
>>>> then
>>>>>>> moving the byte array to off-heap memory (via Unsafe or
>> ByteBuffers),
>>>> and
>>>>>>> serializing directly into a "managed" off-heap buffer, thus further
>>>>>>> optimizing heap utilization (less GC).
>>>>>>>
>>>>>>> What do you think about it? Does it make any sense to you?
>>>>>>>
>>>>>>> Ciao,
>>>>>>> R
>>>>>>>
>>>>>>> On Sat, Sep 29, 2012 at 2:40 PM, Noctarius <me...@noctarius.com>
>> wrote:
>>>>>>>
>>>>>>>> Hey guys,
>>>>>>>>
>>>>>>>> Raffaele found out about a project of mine (Lightning) a few weeks
>>>>>>>> ago. Lightning is a heavy Unsafe and Bytecode generation using
>>>>>>>> Serializer implementation. He told me that he was interested in
>>>>>>>> adding something similar to DirectMemory and I would be glad to
>> help
>>>>>>>> out in this.
>>>>>>>>
>>>>>>>> Another project I started a few days ago, since it was needed for
>>>>>>>> work is DirectRingCache. The name not really meets to actual
>>>>>>>> implementation since it's not yet a ring buffer using cache. I
>> used
>>>>>>>> this for a pre-serialization simple bytestream cache with
>>>>>>>> self-growing buffers. It could be nice to have DirectMemory having
>>>>>>>> raw "buffers" to write to or to read from.
>>>>>>>>
>>>>>>>> Here are the links from the projects:
>>>>>>>> https://github.com/noctarius/Lightning
>>>>>>>> https://github.com/noctarius/direct-ring-cache
>>>>>>>>
>>>>>>>> It would be nice to help out since I really like the idea of
>>>>>>>> DirectMemory and since direct-ring-cache is some kind of
>> reinventing
>>>>>>>> the wheel.
>>>>>>>>
>>>>>>>> Cheers
>>>>>>>> Noctarius (Chris)
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Olivier Lamy
>>>> Talend: http://coders.talend.com
>>>> http://twitter.com/olamy | http://linkedin.com/in/olamy
>>>>
>>
>>
>>
>> --
>> Olivier Lamy
>> Talend: http://coders.talend.com
>> http://twitter.com/olamy | http://linkedin.com/in/olamy
>>
>
Re: Additional Serializer and raw Buffer access
Posted by "Raffaele P. Guidi" <ra...@gmail.com>.
Ok so it's up to noctarius - your move! ;-) Regarding the new unsafe
storage: it's an opt-in feature that can be set with the fluent API (and
soon through the conference file).
Ciao,
R
Il giorno 29/set/2012 16:45, "Olivier Lamy" <ol...@apache.org> ha scritto:
> 2012/9/29 Raffaele P. Guidi <ra...@gmail.com>:
> >>> At least for the moment he can provide a patch to be integrated in DM
> :-)
> >
> > Sure, but as lightning is not in any public mvn repo should its code be
> > re-published in our svn? Or what?
> @Apache we don't care about binaries, only sources are important ! (a
> bit theorical for sure but that's it :-) ).
> So if Noctarius was the only guy who participate in lightning. He can
> just provide a patch we could integrate as a new dm module (note: the
> patch must not contains any more copyright and all sources must have
> ASF licenses).
>
> "Copyright 2012 the original author or authors." must be removed.
> And BTW package must be changed :-) (com.github is not acceptable @asf
> :-) )(@Noctarius are you working for github ? :-) )
>
> And having him as a committer will be only a matter of voting (we have
> a great chair who take cares of administrative stuff :P )
>
> If some others have participated in the project, we must pass tru an
> ip clearance mechanism
> (http://incubator.apache.org/ip-clearance/index.html) and all
> contributors to lightning must provide a cla. (It it's the case I can
> help)
>
> >
> >>> perso I'd like we avoid hard dependency on Unsafe as maybe some use
> > other jdks :-)
> >
> > Well, I believe Unsafe is supported by Sun JDK, OpenJDK, IBM JDK and
> > JRockit - and I believe that it is more than enough. Also keep in mind
> that
> > we already have an alternative Unsafe based memory storage - and although
> > it's not thoroughly tested for performance it dramaticaly simplifies
> code,
> > I have great expectations about it.
> Me too :-). Yup definitely more simple and faster !
> But we must provide a switch off configuration mechanism if some users
> don't want to use that (because Unsafe is just "Unsafe" :-) )
> And sorry I didn't have a look yet at your changes with using Unsafe.
> >
> > Cheers,
> > R
> >
> > On Sat, Sep 29, 2012 at 4:03 PM, Olivier Lamy <ol...@apache.org> wrote:
> >
> >> 2012/9/29 Raffaele P. Guidi <ra...@gmail.com>:
> >> > What do you think about:
> >> > 1) implementing a lightning serialization module
> >> > 2) creating a serializer that directly works on a directmemory
> provider
> >> > ByteBuffer or (maybe better) Unsafe based Pointer?
> >>
> >> Sounds good (perso I'd like we avoid hard dependency on Unsafe as
> >> maybe some use other jdks :-) )
> >> >
> >> > Now I see lightning is apache licensed and this is fine but I don't
> think
> >> > it is published in any public maven repo, am I right? We could find a
> way
> >> > to deal with this; options vary from publishing lightning to the free
> >> > sonatype repo, joining the ASF (which is great anyhow!) and
> republishing
> >> > lightning code in DirectMemory becoming a commiter (which has to
> undergo
> >> a
> >> > PMC vote).
> >>
> >> At least for the moment he can provide a patch to be integrated in DM
> :-)
> >>
> >> >
> >> > I'd like to hear your and the team feelings on this.
> >> >
> >> > Thanks,
> >> > Raffaele
> >> >
> >> > On Sat, Sep 29, 2012 at 3:27 PM, Noctarius <me...@noctarius.com> wrote:
> >> >
> >> >> Hey Raffaele,
> >> >>
> >> >> that's quite similar to what I did at work. We're developing Flash
> >> >> online games and using a customized AMF serialization. To support
> >> >> async writing of the clients event results I added serialization of
> >> >> the components / entities to the players zone calculation and stored
> >> >> the pre-serialized bytestream directly to the off-heap (using
> >> >> direct-ring-cache implementation). When the client requests the
> >> >> results (using long-polling) I just write the pre-serialized data to
> >> >> the right position to deserialize it by standard ways on Flash side.
> >> >>
> >> >> So yeah an seriliaztion to off-heap algorithm would be fine. You can
> >> >> avoid using byte arrays and minimalize GC.
> >> >>
> >> >> Cheers
> >> >> Chris
> >> >>
> >> >> Am 29.09.2012 15:02, schrieb Raffaele P. Guidi:
> >> >> > Nice to hear back from you! Yes, I was thinking about creating a
> new
> >> >> memory
> >> >> > storage implementation using Unsafe (and I did it, recently) and
> >> also, as
> >> >> > DirectMemory relies heavily on serialization (and supports many of
> >> them,
> >> >> > protostuff, protobuf, msgpack and of course standard
> serialization),
> >> >> about
> >> >> > creating a simple embedded serializer leveraging the same
> techniques
> >> you
> >> >> > used (Unsafe and bytecode generation).
> >> >> > The idea with embedding is avoiding to serialize in a byte array
> and
> >> then
> >> >> > moving the byte array to off-heap memory (via Unsafe or
> ByteBuffers),
> >> and
> >> >> > serializing directly into a "managed" off-heap buffer, thus further
> >> >> > optimizing heap utilization (less GC).
> >> >> >
> >> >> > What do you think about it? Does it make any sense to you?
> >> >> >
> >> >> > Ciao,
> >> >> > R
> >> >> >
> >> >> > On Sat, Sep 29, 2012 at 2:40 PM, Noctarius <me...@noctarius.com>
> wrote:
> >> >> >
> >> >> >> Hey guys,
> >> >> >>
> >> >> >> Raffaele found out about a project of mine (Lightning) a few weeks
> >> >> >> ago. Lightning is a heavy Unsafe and Bytecode generation using
> >> >> >> Serializer implementation. He told me that he was interested in
> >> >> >> adding something similar to DirectMemory and I would be glad to
> help
> >> >> >> out in this.
> >> >> >>
> >> >> >> Another project I started a few days ago, since it was needed for
> >> >> >> work is DirectRingCache. The name not really meets to actual
> >> >> >> implementation since it's not yet a ring buffer using cache. I
> used
> >> >> >> this for a pre-serialization simple bytestream cache with
> >> >> >> self-growing buffers. It could be nice to have DirectMemory having
> >> >> >> raw "buffers" to write to or to read from.
> >> >> >>
> >> >> >> Here are the links from the projects:
> >> >> >> https://github.com/noctarius/Lightning
> >> >> >> https://github.com/noctarius/direct-ring-cache
> >> >> >>
> >> >> >> It would be nice to help out since I really like the idea of
> >> >> >> DirectMemory and since direct-ring-cache is some kind of
> reinventing
> >> >> >> the wheel.
> >> >> >>
> >> >> >> Cheers
> >> >> >> Noctarius (Chris)
> >> >> >>
> >> >> >
> >> >>
> >> >>
> >> >>
> >>
> >>
> >>
> >> --
> >> Olivier Lamy
> >> Talend: http://coders.talend.com
> >> http://twitter.com/olamy | http://linkedin.com/in/olamy
> >>
>
>
>
> --
> Olivier Lamy
> Talend: http://coders.talend.com
> http://twitter.com/olamy | http://linkedin.com/in/olamy
>
Re: Additional Serializer and raw Buffer access
Posted by Olivier Lamy <ol...@apache.org>.
2012/9/29 Raffaele P. Guidi <ra...@gmail.com>:
>>> At least for the moment he can provide a patch to be integrated in DM :-)
>
> Sure, but as lightning is not in any public mvn repo should its code be
> re-published in our svn? Or what?
@Apache we don't care about binaries, only sources are important ! (a
bit theorical for sure but that's it :-) ).
So if Noctarius was the only guy who participate in lightning. He can
just provide a patch we could integrate as a new dm module (note: the
patch must not contains any more copyright and all sources must have
ASF licenses).
"Copyright 2012 the original author or authors." must be removed.
And BTW package must be changed :-) (com.github is not acceptable @asf
:-) )(@Noctarius are you working for github ? :-) )
And having him as a committer will be only a matter of voting (we have
a great chair who take cares of administrative stuff :P )
If some others have participated in the project, we must pass tru an
ip clearance mechanism
(http://incubator.apache.org/ip-clearance/index.html) and all
contributors to lightning must provide a cla. (It it's the case I can
help)
>
>>> perso I'd like we avoid hard dependency on Unsafe as maybe some use
> other jdks :-)
>
> Well, I believe Unsafe is supported by Sun JDK, OpenJDK, IBM JDK and
> JRockit - and I believe that it is more than enough. Also keep in mind that
> we already have an alternative Unsafe based memory storage - and although
> it's not thoroughly tested for performance it dramaticaly simplifies code,
> I have great expectations about it.
Me too :-). Yup definitely more simple and faster !
But we must provide a switch off configuration mechanism if some users
don't want to use that (because Unsafe is just "Unsafe" :-) )
And sorry I didn't have a look yet at your changes with using Unsafe.
>
> Cheers,
> R
>
> On Sat, Sep 29, 2012 at 4:03 PM, Olivier Lamy <ol...@apache.org> wrote:
>
>> 2012/9/29 Raffaele P. Guidi <ra...@gmail.com>:
>> > What do you think about:
>> > 1) implementing a lightning serialization module
>> > 2) creating a serializer that directly works on a directmemory provider
>> > ByteBuffer or (maybe better) Unsafe based Pointer?
>>
>> Sounds good (perso I'd like we avoid hard dependency on Unsafe as
>> maybe some use other jdks :-) )
>> >
>> > Now I see lightning is apache licensed and this is fine but I don't think
>> > it is published in any public maven repo, am I right? We could find a way
>> > to deal with this; options vary from publishing lightning to the free
>> > sonatype repo, joining the ASF (which is great anyhow!) and republishing
>> > lightning code in DirectMemory becoming a commiter (which has to undergo
>> a
>> > PMC vote).
>>
>> At least for the moment he can provide a patch to be integrated in DM :-)
>>
>> >
>> > I'd like to hear your and the team feelings on this.
>> >
>> > Thanks,
>> > Raffaele
>> >
>> > On Sat, Sep 29, 2012 at 3:27 PM, Noctarius <me...@noctarius.com> wrote:
>> >
>> >> Hey Raffaele,
>> >>
>> >> that's quite similar to what I did at work. We're developing Flash
>> >> online games and using a customized AMF serialization. To support
>> >> async writing of the clients event results I added serialization of
>> >> the components / entities to the players zone calculation and stored
>> >> the pre-serialized bytestream directly to the off-heap (using
>> >> direct-ring-cache implementation). When the client requests the
>> >> results (using long-polling) I just write the pre-serialized data to
>> >> the right position to deserialize it by standard ways on Flash side.
>> >>
>> >> So yeah an seriliaztion to off-heap algorithm would be fine. You can
>> >> avoid using byte arrays and minimalize GC.
>> >>
>> >> Cheers
>> >> Chris
>> >>
>> >> Am 29.09.2012 15:02, schrieb Raffaele P. Guidi:
>> >> > Nice to hear back from you! Yes, I was thinking about creating a new
>> >> memory
>> >> > storage implementation using Unsafe (and I did it, recently) and
>> also, as
>> >> > DirectMemory relies heavily on serialization (and supports many of
>> them,
>> >> > protostuff, protobuf, msgpack and of course standard serialization),
>> >> about
>> >> > creating a simple embedded serializer leveraging the same techniques
>> you
>> >> > used (Unsafe and bytecode generation).
>> >> > The idea with embedding is avoiding to serialize in a byte array and
>> then
>> >> > moving the byte array to off-heap memory (via Unsafe or ByteBuffers),
>> and
>> >> > serializing directly into a "managed" off-heap buffer, thus further
>> >> > optimizing heap utilization (less GC).
>> >> >
>> >> > What do you think about it? Does it make any sense to you?
>> >> >
>> >> > Ciao,
>> >> > R
>> >> >
>> >> > On Sat, Sep 29, 2012 at 2:40 PM, Noctarius <me...@noctarius.com> wrote:
>> >> >
>> >> >> Hey guys,
>> >> >>
>> >> >> Raffaele found out about a project of mine (Lightning) a few weeks
>> >> >> ago. Lightning is a heavy Unsafe and Bytecode generation using
>> >> >> Serializer implementation. He told me that he was interested in
>> >> >> adding something similar to DirectMemory and I would be glad to help
>> >> >> out in this.
>> >> >>
>> >> >> Another project I started a few days ago, since it was needed for
>> >> >> work is DirectRingCache. The name not really meets to actual
>> >> >> implementation since it's not yet a ring buffer using cache. I used
>> >> >> this for a pre-serialization simple bytestream cache with
>> >> >> self-growing buffers. It could be nice to have DirectMemory having
>> >> >> raw "buffers" to write to or to read from.
>> >> >>
>> >> >> Here are the links from the projects:
>> >> >> https://github.com/noctarius/Lightning
>> >> >> https://github.com/noctarius/direct-ring-cache
>> >> >>
>> >> >> It would be nice to help out since I really like the idea of
>> >> >> DirectMemory and since direct-ring-cache is some kind of reinventing
>> >> >> the wheel.
>> >> >>
>> >> >> Cheers
>> >> >> Noctarius (Chris)
>> >> >>
>> >> >
>> >>
>> >>
>> >>
>>
>>
>>
>> --
>> Olivier Lamy
>> Talend: http://coders.talend.com
>> http://twitter.com/olamy | http://linkedin.com/in/olamy
>>
--
Olivier Lamy
Talend: http://coders.talend.com
http://twitter.com/olamy | http://linkedin.com/in/olamy
Re: Additional Serializer and raw Buffer access
Posted by "Raffaele P. Guidi" <ra...@gmail.com>.
>> At least for the moment he can provide a patch to be integrated in DM :-)
Sure, but as lightning is not in any public mvn repo should its code be
re-published in our svn? Or what?
>> perso I'd like we avoid hard dependency on Unsafe as maybe some use
other jdks :-)
Well, I believe Unsafe is supported by Sun JDK, OpenJDK, IBM JDK and
JRockit - and I believe that it is more than enough. Also keep in mind that
we already have an alternative Unsafe based memory storage - and although
it's not thoroughly tested for performance it dramaticaly simplifies code,
I have great expectations about it.
Cheers,
R
On Sat, Sep 29, 2012 at 4:03 PM, Olivier Lamy <ol...@apache.org> wrote:
> 2012/9/29 Raffaele P. Guidi <ra...@gmail.com>:
> > What do you think about:
> > 1) implementing a lightning serialization module
> > 2) creating a serializer that directly works on a directmemory provider
> > ByteBuffer or (maybe better) Unsafe based Pointer?
>
> Sounds good (perso I'd like we avoid hard dependency on Unsafe as
> maybe some use other jdks :-) )
> >
> > Now I see lightning is apache licensed and this is fine but I don't think
> > it is published in any public maven repo, am I right? We could find a way
> > to deal with this; options vary from publishing lightning to the free
> > sonatype repo, joining the ASF (which is great anyhow!) and republishing
> > lightning code in DirectMemory becoming a commiter (which has to undergo
> a
> > PMC vote).
>
> At least for the moment he can provide a patch to be integrated in DM :-)
>
> >
> > I'd like to hear your and the team feelings on this.
> >
> > Thanks,
> > Raffaele
> >
> > On Sat, Sep 29, 2012 at 3:27 PM, Noctarius <me...@noctarius.com> wrote:
> >
> >> Hey Raffaele,
> >>
> >> that's quite similar to what I did at work. We're developing Flash
> >> online games and using a customized AMF serialization. To support
> >> async writing of the clients event results I added serialization of
> >> the components / entities to the players zone calculation and stored
> >> the pre-serialized bytestream directly to the off-heap (using
> >> direct-ring-cache implementation). When the client requests the
> >> results (using long-polling) I just write the pre-serialized data to
> >> the right position to deserialize it by standard ways on Flash side.
> >>
> >> So yeah an seriliaztion to off-heap algorithm would be fine. You can
> >> avoid using byte arrays and minimalize GC.
> >>
> >> Cheers
> >> Chris
> >>
> >> Am 29.09.2012 15:02, schrieb Raffaele P. Guidi:
> >> > Nice to hear back from you! Yes, I was thinking about creating a new
> >> memory
> >> > storage implementation using Unsafe (and I did it, recently) and
> also, as
> >> > DirectMemory relies heavily on serialization (and supports many of
> them,
> >> > protostuff, protobuf, msgpack and of course standard serialization),
> >> about
> >> > creating a simple embedded serializer leveraging the same techniques
> you
> >> > used (Unsafe and bytecode generation).
> >> > The idea with embedding is avoiding to serialize in a byte array and
> then
> >> > moving the byte array to off-heap memory (via Unsafe or ByteBuffers),
> and
> >> > serializing directly into a "managed" off-heap buffer, thus further
> >> > optimizing heap utilization (less GC).
> >> >
> >> > What do you think about it? Does it make any sense to you?
> >> >
> >> > Ciao,
> >> > R
> >> >
> >> > On Sat, Sep 29, 2012 at 2:40 PM, Noctarius <me...@noctarius.com> wrote:
> >> >
> >> >> Hey guys,
> >> >>
> >> >> Raffaele found out about a project of mine (Lightning) a few weeks
> >> >> ago. Lightning is a heavy Unsafe and Bytecode generation using
> >> >> Serializer implementation. He told me that he was interested in
> >> >> adding something similar to DirectMemory and I would be glad to help
> >> >> out in this.
> >> >>
> >> >> Another project I started a few days ago, since it was needed for
> >> >> work is DirectRingCache. The name not really meets to actual
> >> >> implementation since it's not yet a ring buffer using cache. I used
> >> >> this for a pre-serialization simple bytestream cache with
> >> >> self-growing buffers. It could be nice to have DirectMemory having
> >> >> raw "buffers" to write to or to read from.
> >> >>
> >> >> Here are the links from the projects:
> >> >> https://github.com/noctarius/Lightning
> >> >> https://github.com/noctarius/direct-ring-cache
> >> >>
> >> >> It would be nice to help out since I really like the idea of
> >> >> DirectMemory and since direct-ring-cache is some kind of reinventing
> >> >> the wheel.
> >> >>
> >> >> Cheers
> >> >> Noctarius (Chris)
> >> >>
> >> >
> >>
> >>
> >>
>
>
>
> --
> Olivier Lamy
> Talend: http://coders.talend.com
> http://twitter.com/olamy | http://linkedin.com/in/olamy
>
Re: Additional Serializer and raw Buffer access
Posted by Olivier Lamy <ol...@apache.org>.
2012/9/29 Raffaele P. Guidi <ra...@gmail.com>:
> What do you think about:
> 1) implementing a lightning serialization module
> 2) creating a serializer that directly works on a directmemory provider
> ByteBuffer or (maybe better) Unsafe based Pointer?
Sounds good (perso I'd like we avoid hard dependency on Unsafe as
maybe some use other jdks :-) )
>
> Now I see lightning is apache licensed and this is fine but I don't think
> it is published in any public maven repo, am I right? We could find a way
> to deal with this; options vary from publishing lightning to the free
> sonatype repo, joining the ASF (which is great anyhow!) and republishing
> lightning code in DirectMemory becoming a commiter (which has to undergo a
> PMC vote).
At least for the moment he can provide a patch to be integrated in DM :-)
>
> I'd like to hear your and the team feelings on this.
>
> Thanks,
> Raffaele
>
> On Sat, Sep 29, 2012 at 3:27 PM, Noctarius <me...@noctarius.com> wrote:
>
>> Hey Raffaele,
>>
>> that's quite similar to what I did at work. We're developing Flash
>> online games and using a customized AMF serialization. To support
>> async writing of the clients event results I added serialization of
>> the components / entities to the players zone calculation and stored
>> the pre-serialized bytestream directly to the off-heap (using
>> direct-ring-cache implementation). When the client requests the
>> results (using long-polling) I just write the pre-serialized data to
>> the right position to deserialize it by standard ways on Flash side.
>>
>> So yeah an seriliaztion to off-heap algorithm would be fine. You can
>> avoid using byte arrays and minimalize GC.
>>
>> Cheers
>> Chris
>>
>> Am 29.09.2012 15:02, schrieb Raffaele P. Guidi:
>> > Nice to hear back from you! Yes, I was thinking about creating a new
>> memory
>> > storage implementation using Unsafe (and I did it, recently) and also, as
>> > DirectMemory relies heavily on serialization (and supports many of them,
>> > protostuff, protobuf, msgpack and of course standard serialization),
>> about
>> > creating a simple embedded serializer leveraging the same techniques you
>> > used (Unsafe and bytecode generation).
>> > The idea with embedding is avoiding to serialize in a byte array and then
>> > moving the byte array to off-heap memory (via Unsafe or ByteBuffers), and
>> > serializing directly into a "managed" off-heap buffer, thus further
>> > optimizing heap utilization (less GC).
>> >
>> > What do you think about it? Does it make any sense to you?
>> >
>> > Ciao,
>> > R
>> >
>> > On Sat, Sep 29, 2012 at 2:40 PM, Noctarius <me...@noctarius.com> wrote:
>> >
>> >> Hey guys,
>> >>
>> >> Raffaele found out about a project of mine (Lightning) a few weeks
>> >> ago. Lightning is a heavy Unsafe and Bytecode generation using
>> >> Serializer implementation. He told me that he was interested in
>> >> adding something similar to DirectMemory and I would be glad to help
>> >> out in this.
>> >>
>> >> Another project I started a few days ago, since it was needed for
>> >> work is DirectRingCache. The name not really meets to actual
>> >> implementation since it's not yet a ring buffer using cache. I used
>> >> this for a pre-serialization simple bytestream cache with
>> >> self-growing buffers. It could be nice to have DirectMemory having
>> >> raw "buffers" to write to or to read from.
>> >>
>> >> Here are the links from the projects:
>> >> https://github.com/noctarius/Lightning
>> >> https://github.com/noctarius/direct-ring-cache
>> >>
>> >> It would be nice to help out since I really like the idea of
>> >> DirectMemory and since direct-ring-cache is some kind of reinventing
>> >> the wheel.
>> >>
>> >> Cheers
>> >> Noctarius (Chris)
>> >>
>> >
>>
>>
>>
--
Olivier Lamy
Talend: http://coders.talend.com
http://twitter.com/olamy | http://linkedin.com/in/olamy
Re: Additional Serializer and raw Buffer access
Posted by "Raffaele P. Guidi" <ra...@gmail.com>.
What do you think about:
1) implementing a lightning serialization module
2) creating a serializer that directly works on a directmemory provider
ByteBuffer or (maybe better) Unsafe based Pointer?
Now I see lightning is apache licensed and this is fine but I don't think
it is published in any public maven repo, am I right? We could find a way
to deal with this; options vary from publishing lightning to the free
sonatype repo, joining the ASF (which is great anyhow!) and republishing
lightning code in DirectMemory becoming a commiter (which has to undergo a
PMC vote).
I'd like to hear your and the team feelings on this.
Thanks,
Raffaele
On Sat, Sep 29, 2012 at 3:27 PM, Noctarius <me...@noctarius.com> wrote:
> Hey Raffaele,
>
> that's quite similar to what I did at work. We're developing Flash
> online games and using a customized AMF serialization. To support
> async writing of the clients event results I added serialization of
> the components / entities to the players zone calculation and stored
> the pre-serialized bytestream directly to the off-heap (using
> direct-ring-cache implementation). When the client requests the
> results (using long-polling) I just write the pre-serialized data to
> the right position to deserialize it by standard ways on Flash side.
>
> So yeah an seriliaztion to off-heap algorithm would be fine. You can
> avoid using byte arrays and minimalize GC.
>
> Cheers
> Chris
>
> Am 29.09.2012 15:02, schrieb Raffaele P. Guidi:
> > Nice to hear back from you! Yes, I was thinking about creating a new
> memory
> > storage implementation using Unsafe (and I did it, recently) and also, as
> > DirectMemory relies heavily on serialization (and supports many of them,
> > protostuff, protobuf, msgpack and of course standard serialization),
> about
> > creating a simple embedded serializer leveraging the same techniques you
> > used (Unsafe and bytecode generation).
> > The idea with embedding is avoiding to serialize in a byte array and then
> > moving the byte array to off-heap memory (via Unsafe or ByteBuffers), and
> > serializing directly into a "managed" off-heap buffer, thus further
> > optimizing heap utilization (less GC).
> >
> > What do you think about it? Does it make any sense to you?
> >
> > Ciao,
> > R
> >
> > On Sat, Sep 29, 2012 at 2:40 PM, Noctarius <me...@noctarius.com> wrote:
> >
> >> Hey guys,
> >>
> >> Raffaele found out about a project of mine (Lightning) a few weeks
> >> ago. Lightning is a heavy Unsafe and Bytecode generation using
> >> Serializer implementation. He told me that he was interested in
> >> adding something similar to DirectMemory and I would be glad to help
> >> out in this.
> >>
> >> Another project I started a few days ago, since it was needed for
> >> work is DirectRingCache. The name not really meets to actual
> >> implementation since it's not yet a ring buffer using cache. I used
> >> this for a pre-serialization simple bytestream cache with
> >> self-growing buffers. It could be nice to have DirectMemory having
> >> raw "buffers" to write to or to read from.
> >>
> >> Here are the links from the projects:
> >> https://github.com/noctarius/Lightning
> >> https://github.com/noctarius/direct-ring-cache
> >>
> >> It would be nice to help out since I really like the idea of
> >> DirectMemory and since direct-ring-cache is some kind of reinventing
> >> the wheel.
> >>
> >> Cheers
> >> Noctarius (Chris)
> >>
> >
>
>
>
Re: Additional Serializer and raw Buffer access
Posted by Noctarius <me...@noctarius.com>.
Hey Raffaele,
that's quite similar to what I did at work. We're developing Flash
online games and using a customized AMF serialization. To support
async writing of the clients event results I added serialization of
the components / entities to the players zone calculation and stored
the pre-serialized bytestream directly to the off-heap (using
direct-ring-cache implementation). When the client requests the
results (using long-polling) I just write the pre-serialized data to
the right position to deserialize it by standard ways on Flash side.
So yeah an seriliaztion to off-heap algorithm would be fine. You can
avoid using byte arrays and minimalize GC.
Cheers
Chris
Am 29.09.2012 15:02, schrieb Raffaele P. Guidi:
> Nice to hear back from you! Yes, I was thinking about creating a new memory
> storage implementation using Unsafe (and I did it, recently) and also, as
> DirectMemory relies heavily on serialization (and supports many of them,
> protostuff, protobuf, msgpack and of course standard serialization), about
> creating a simple embedded serializer leveraging the same techniques you
> used (Unsafe and bytecode generation).
> The idea with embedding is avoiding to serialize in a byte array and then
> moving the byte array to off-heap memory (via Unsafe or ByteBuffers), and
> serializing directly into a "managed" off-heap buffer, thus further
> optimizing heap utilization (less GC).
>
> What do you think about it? Does it make any sense to you?
>
> Ciao,
> R
>
> On Sat, Sep 29, 2012 at 2:40 PM, Noctarius <me...@noctarius.com> wrote:
>
>> Hey guys,
>>
>> Raffaele found out about a project of mine (Lightning) a few weeks
>> ago. Lightning is a heavy Unsafe and Bytecode generation using
>> Serializer implementation. He told me that he was interested in
>> adding something similar to DirectMemory and I would be glad to help
>> out in this.
>>
>> Another project I started a few days ago, since it was needed for
>> work is DirectRingCache. The name not really meets to actual
>> implementation since it's not yet a ring buffer using cache. I used
>> this for a pre-serialization simple bytestream cache with
>> self-growing buffers. It could be nice to have DirectMemory having
>> raw "buffers" to write to or to read from.
>>
>> Here are the links from the projects:
>> https://github.com/noctarius/Lightning
>> https://github.com/noctarius/direct-ring-cache
>>
>> It would be nice to help out since I really like the idea of
>> DirectMemory and since direct-ring-cache is some kind of reinventing
>> the wheel.
>>
>> Cheers
>> Noctarius (Chris)
>>
>
Re: Additional Serializer and raw Buffer access
Posted by "Raffaele P. Guidi" <ra...@gmail.com>.
Nice to hear back from you! Yes, I was thinking about creating a new memory
storage implementation using Unsafe (and I did it, recently) and also, as
DirectMemory relies heavily on serialization (and supports many of them,
protostuff, protobuf, msgpack and of course standard serialization), about
creating a simple embedded serializer leveraging the same techniques you
used (Unsafe and bytecode generation).
The idea with embedding is avoiding to serialize in a byte array and then
moving the byte array to off-heap memory (via Unsafe or ByteBuffers), and
serializing directly into a "managed" off-heap buffer, thus further
optimizing heap utilization (less GC).
What do you think about it? Does it make any sense to you?
Ciao,
R
On Sat, Sep 29, 2012 at 2:40 PM, Noctarius <me...@noctarius.com> wrote:
> Hey guys,
>
> Raffaele found out about a project of mine (Lightning) a few weeks
> ago. Lightning is a heavy Unsafe and Bytecode generation using
> Serializer implementation. He told me that he was interested in
> adding something similar to DirectMemory and I would be glad to help
> out in this.
>
> Another project I started a few days ago, since it was needed for
> work is DirectRingCache. The name not really meets to actual
> implementation since it's not yet a ring buffer using cache. I used
> this for a pre-serialization simple bytestream cache with
> self-growing buffers. It could be nice to have DirectMemory having
> raw "buffers" to write to or to read from.
>
> Here are the links from the projects:
> https://github.com/noctarius/Lightning
> https://github.com/noctarius/direct-ring-cache
>
> It would be nice to help out since I really like the idea of
> DirectMemory and since direct-ring-cache is some kind of reinventing
> the wheel.
>
> Cheers
> Noctarius (Chris)
>