You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hama.apache.org by "Edward J. Yoon" <ed...@apache.org> on 2012/12/13 00:17:28 UTC
Create common module
Hi,
I propose to create common module to share common classes among all modules.
At the moment, the main reason is to support VertexInputReader. As I
mentioned here[1], there's no way to integrate partitioner without
merging graph to core or adding graph own partitioner. If we create
common module, it'll all be clear. I'm not maven and java expert. If
I'm wrong, please correct me.
1. https://issues.apache.org/jira/browse/HAMA-531?focusedCommentId=13483009&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13483009
--
Best Regards, Edward J. Yoon
@eddieyoon
Re: Create common module
Posted by "Edward J. Yoon" <ed...@apache.org>.
I'll move reusable math-related writable classes in graph, ml, matrix
examples to o.a.h.commons.math package of commons module.
If you have other opinion, pls let me know.
On Mon, Dec 17, 2012 at 8:06 PM, Edward J. Yoon <ed...@apache.org> wrote:
> Oh, .... I just noticed that why you asked that question today. On
> second thoughts, converter is not good idea. Partitioner is just a
> partitioner, Vertex should be parsed at GraphJobRunner.
>
> Leave it to me.
>
> On Sun, Dec 16, 2012 at 9:57 AM, Edward J. Yoon <ed...@apache.org> wrote:
>>>> implementation. Now my question is, if the input to be read from HBase, do
>>>> we write every record from HBase to sequential file format? I am a little
>>>> confused here.
>>
>> In NoSQLs cases, they have own partitions based on ranges of key. If
>> the key is vertex and PartitioningRunner can be skipped, user can use
>> default partitions of table without re-writing splits.
>>
>> On Sun, Dec 16, 2012 at 8:50 AM, Edward J. Yoon <ed...@apache.org> wrote:
>>> +1 Looks great!
>>>
>>> On Sun, Dec 16, 2012 at 7:20 AM, Suraj Menon <me...@gmail.com> wrote:
>>>> +1 for hama-io, +0 for hama-commons. Can you look at the patch for HAMA-700
>>>> and let me know if you have any issues going forward in that direction.
>>>> Basically it includes a raw format to intermediate binary format
>>>> conversion, that could be overridden as shown in VertexInputReader
>>>> implementation. Now my question is, if the input to be read from HBase, do
>>>> we write every record from HBase to sequential file format? I am a little
>>>> confused here.
>>>>
>>>> Thanks,
>>>> Suraj
>>>>
>>>> On Fri, Dec 14, 2012 at 2:19 AM, Tommaso Teofili
>>>> <to...@gmail.com>wrote:
>>>>
>>>>> thinking about it a bit further, maybe a separate module for IO related
>>>>> stuff may be worth, so hama-io instead of hama-commons.
>>>>> Whatever it is I'd like to avoid duplicated code.
>>>>> Tommaso
>>>>>
>>>>>
>>>>> 2012/12/13 Edward J. Yoon <ed...@apache.org>
>>>>>
>>>>> > Otherwise, we have to copy VertexInputReader class to core module or
>>>>> > add own partitioner to graph package.
>>>>> >
>>>>> > In fact, I don't want to put any graph-related code to core module.
>>>>> > But if we want to support VertexInputReader, I think this is best.
>>>>> >
>>>>> > On Fri, Dec 14, 2012 at 4:22 AM, Apurv Verma <da...@gmail.com> wrote:
>>>>> > > +0 I am not against but I think that would be an overkill for now.
>>>>> > >
>>>>> > > --
>>>>> > > Regards,
>>>>> > > Apurv Verma
>>>>> > >
>>>>> > >
>>>>> > >
>>>>> > >
>>>>> > > On Thu, Dec 13, 2012 at 12:32 PM, Tommaso Teofili
>>>>> > > <to...@gmail.com> wrote:
>>>>> > >> that sounds good to me, +1 for a hama-commons module.
>>>>> > >>
>>>>> > >> Tommaso
>>>>> > >>
>>>>> > >>
>>>>> > >> 2012/12/13 Edward J. Yoon <ed...@apache.org>
>>>>> > >>
>>>>> > >>> Hi,
>>>>> > >>>
>>>>> > >>> I propose to create common module to share common classes among all
>>>>> > >>> modules.
>>>>> > >>>
>>>>> > >>> At the moment, the main reason is to support VertexInputReader. As I
>>>>> > >>> mentioned here[1], there's no way to integrate partitioner without
>>>>> > >>> merging graph to core or adding graph own partitioner. If we create
>>>>> > >>> common module, it'll all be clear. I'm not maven and java expert. If
>>>>> > >>> I'm wrong, please correct me.
>>>>> > >>>
>>>>> > >>> 1.
>>>>> > >>>
>>>>> >
>>>>> https://issues.apache.org/jira/browse/HAMA-531?focusedCommentId=13483009&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13483009
>>>>> > >>>
>>>>> > >>> --
>>>>> > >>> Best Regards, Edward J. Yoon
>>>>> > >>> @eddieyoon
>>>>> > >>>
>>>>> >
>>>>> >
>>>>> >
>>>>> > --
>>>>> > Best Regards, Edward J. Yoon
>>>>> > @eddieyoon
>>>>> >
>>>>>
>>>
>>>
>>>
>>> --
>>> Best Regards, Edward J. Yoon
>>> @eddieyoon
>>
>>
>>
>> --
>> Best Regards, Edward J. Yoon
>> @eddieyoon
>
>
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon
--
Best Regards, Edward J. Yoon
@eddieyoon
Re: Create common module
Posted by "Edward J. Yoon" <ed...@apache.org>.
Oh, .... I just noticed that why you asked that question today. On
second thoughts, converter is not good idea. Partitioner is just a
partitioner, Vertex should be parsed at GraphJobRunner.
Leave it to me.
On Sun, Dec 16, 2012 at 9:57 AM, Edward J. Yoon <ed...@apache.org> wrote:
>>> implementation. Now my question is, if the input to be read from HBase, do
>>> we write every record from HBase to sequential file format? I am a little
>>> confused here.
>
> In NoSQLs cases, they have own partitions based on ranges of key. If
> the key is vertex and PartitioningRunner can be skipped, user can use
> default partitions of table without re-writing splits.
>
> On Sun, Dec 16, 2012 at 8:50 AM, Edward J. Yoon <ed...@apache.org> wrote:
>> +1 Looks great!
>>
>> On Sun, Dec 16, 2012 at 7:20 AM, Suraj Menon <me...@gmail.com> wrote:
>>> +1 for hama-io, +0 for hama-commons. Can you look at the patch for HAMA-700
>>> and let me know if you have any issues going forward in that direction.
>>> Basically it includes a raw format to intermediate binary format
>>> conversion, that could be overridden as shown in VertexInputReader
>>> implementation. Now my question is, if the input to be read from HBase, do
>>> we write every record from HBase to sequential file format? I am a little
>>> confused here.
>>>
>>> Thanks,
>>> Suraj
>>>
>>> On Fri, Dec 14, 2012 at 2:19 AM, Tommaso Teofili
>>> <to...@gmail.com>wrote:
>>>
>>>> thinking about it a bit further, maybe a separate module for IO related
>>>> stuff may be worth, so hama-io instead of hama-commons.
>>>> Whatever it is I'd like to avoid duplicated code.
>>>> Tommaso
>>>>
>>>>
>>>> 2012/12/13 Edward J. Yoon <ed...@apache.org>
>>>>
>>>> > Otherwise, we have to copy VertexInputReader class to core module or
>>>> > add own partitioner to graph package.
>>>> >
>>>> > In fact, I don't want to put any graph-related code to core module.
>>>> > But if we want to support VertexInputReader, I think this is best.
>>>> >
>>>> > On Fri, Dec 14, 2012 at 4:22 AM, Apurv Verma <da...@gmail.com> wrote:
>>>> > > +0 I am not against but I think that would be an overkill for now.
>>>> > >
>>>> > > --
>>>> > > Regards,
>>>> > > Apurv Verma
>>>> > >
>>>> > >
>>>> > >
>>>> > >
>>>> > > On Thu, Dec 13, 2012 at 12:32 PM, Tommaso Teofili
>>>> > > <to...@gmail.com> wrote:
>>>> > >> that sounds good to me, +1 for a hama-commons module.
>>>> > >>
>>>> > >> Tommaso
>>>> > >>
>>>> > >>
>>>> > >> 2012/12/13 Edward J. Yoon <ed...@apache.org>
>>>> > >>
>>>> > >>> Hi,
>>>> > >>>
>>>> > >>> I propose to create common module to share common classes among all
>>>> > >>> modules.
>>>> > >>>
>>>> > >>> At the moment, the main reason is to support VertexInputReader. As I
>>>> > >>> mentioned here[1], there's no way to integrate partitioner without
>>>> > >>> merging graph to core or adding graph own partitioner. If we create
>>>> > >>> common module, it'll all be clear. I'm not maven and java expert. If
>>>> > >>> I'm wrong, please correct me.
>>>> > >>>
>>>> > >>> 1.
>>>> > >>>
>>>> >
>>>> https://issues.apache.org/jira/browse/HAMA-531?focusedCommentId=13483009&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13483009
>>>> > >>>
>>>> > >>> --
>>>> > >>> Best Regards, Edward J. Yoon
>>>> > >>> @eddieyoon
>>>> > >>>
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > Best Regards, Edward J. Yoon
>>>> > @eddieyoon
>>>> >
>>>>
>>
>>
>>
>> --
>> Best Regards, Edward J. Yoon
>> @eddieyoon
>
>
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon
--
Best Regards, Edward J. Yoon
@eddieyoon
Re: Create common module
Posted by "Edward J. Yoon" <ed...@apache.org>.
>> implementation. Now my question is, if the input to be read from HBase, do
>> we write every record from HBase to sequential file format? I am a little
>> confused here.
In NoSQLs cases, they have own partitions based on ranges of key. If
the key is vertex and PartitioningRunner can be skipped, user can use
default partitions of table without re-writing splits.
On Sun, Dec 16, 2012 at 8:50 AM, Edward J. Yoon <ed...@apache.org> wrote:
> +1 Looks great!
>
> On Sun, Dec 16, 2012 at 7:20 AM, Suraj Menon <me...@gmail.com> wrote:
>> +1 for hama-io, +0 for hama-commons. Can you look at the patch for HAMA-700
>> and let me know if you have any issues going forward in that direction.
>> Basically it includes a raw format to intermediate binary format
>> conversion, that could be overridden as shown in VertexInputReader
>> implementation. Now my question is, if the input to be read from HBase, do
>> we write every record from HBase to sequential file format? I am a little
>> confused here.
>>
>> Thanks,
>> Suraj
>>
>> On Fri, Dec 14, 2012 at 2:19 AM, Tommaso Teofili
>> <to...@gmail.com>wrote:
>>
>>> thinking about it a bit further, maybe a separate module for IO related
>>> stuff may be worth, so hama-io instead of hama-commons.
>>> Whatever it is I'd like to avoid duplicated code.
>>> Tommaso
>>>
>>>
>>> 2012/12/13 Edward J. Yoon <ed...@apache.org>
>>>
>>> > Otherwise, we have to copy VertexInputReader class to core module or
>>> > add own partitioner to graph package.
>>> >
>>> > In fact, I don't want to put any graph-related code to core module.
>>> > But if we want to support VertexInputReader, I think this is best.
>>> >
>>> > On Fri, Dec 14, 2012 at 4:22 AM, Apurv Verma <da...@gmail.com> wrote:
>>> > > +0 I am not against but I think that would be an overkill for now.
>>> > >
>>> > > --
>>> > > Regards,
>>> > > Apurv Verma
>>> > >
>>> > >
>>> > >
>>> > >
>>> > > On Thu, Dec 13, 2012 at 12:32 PM, Tommaso Teofili
>>> > > <to...@gmail.com> wrote:
>>> > >> that sounds good to me, +1 for a hama-commons module.
>>> > >>
>>> > >> Tommaso
>>> > >>
>>> > >>
>>> > >> 2012/12/13 Edward J. Yoon <ed...@apache.org>
>>> > >>
>>> > >>> Hi,
>>> > >>>
>>> > >>> I propose to create common module to share common classes among all
>>> > >>> modules.
>>> > >>>
>>> > >>> At the moment, the main reason is to support VertexInputReader. As I
>>> > >>> mentioned here[1], there's no way to integrate partitioner without
>>> > >>> merging graph to core or adding graph own partitioner. If we create
>>> > >>> common module, it'll all be clear. I'm not maven and java expert. If
>>> > >>> I'm wrong, please correct me.
>>> > >>>
>>> > >>> 1.
>>> > >>>
>>> >
>>> https://issues.apache.org/jira/browse/HAMA-531?focusedCommentId=13483009&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13483009
>>> > >>>
>>> > >>> --
>>> > >>> Best Regards, Edward J. Yoon
>>> > >>> @eddieyoon
>>> > >>>
>>> >
>>> >
>>> >
>>> > --
>>> > Best Regards, Edward J. Yoon
>>> > @eddieyoon
>>> >
>>>
>
>
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon
--
Best Regards, Edward J. Yoon
@eddieyoon
Re: Create common module
Posted by "Edward J. Yoon" <ed...@apache.org>.
+1 Looks great!
On Sun, Dec 16, 2012 at 7:20 AM, Suraj Menon <me...@gmail.com> wrote:
> +1 for hama-io, +0 for hama-commons. Can you look at the patch for HAMA-700
> and let me know if you have any issues going forward in that direction.
> Basically it includes a raw format to intermediate binary format
> conversion, that could be overridden as shown in VertexInputReader
> implementation. Now my question is, if the input to be read from HBase, do
> we write every record from HBase to sequential file format? I am a little
> confused here.
>
> Thanks,
> Suraj
>
> On Fri, Dec 14, 2012 at 2:19 AM, Tommaso Teofili
> <to...@gmail.com>wrote:
>
>> thinking about it a bit further, maybe a separate module for IO related
>> stuff may be worth, so hama-io instead of hama-commons.
>> Whatever it is I'd like to avoid duplicated code.
>> Tommaso
>>
>>
>> 2012/12/13 Edward J. Yoon <ed...@apache.org>
>>
>> > Otherwise, we have to copy VertexInputReader class to core module or
>> > add own partitioner to graph package.
>> >
>> > In fact, I don't want to put any graph-related code to core module.
>> > But if we want to support VertexInputReader, I think this is best.
>> >
>> > On Fri, Dec 14, 2012 at 4:22 AM, Apurv Verma <da...@gmail.com> wrote:
>> > > +0 I am not against but I think that would be an overkill for now.
>> > >
>> > > --
>> > > Regards,
>> > > Apurv Verma
>> > >
>> > >
>> > >
>> > >
>> > > On Thu, Dec 13, 2012 at 12:32 PM, Tommaso Teofili
>> > > <to...@gmail.com> wrote:
>> > >> that sounds good to me, +1 for a hama-commons module.
>> > >>
>> > >> Tommaso
>> > >>
>> > >>
>> > >> 2012/12/13 Edward J. Yoon <ed...@apache.org>
>> > >>
>> > >>> Hi,
>> > >>>
>> > >>> I propose to create common module to share common classes among all
>> > >>> modules.
>> > >>>
>> > >>> At the moment, the main reason is to support VertexInputReader. As I
>> > >>> mentioned here[1], there's no way to integrate partitioner without
>> > >>> merging graph to core or adding graph own partitioner. If we create
>> > >>> common module, it'll all be clear. I'm not maven and java expert. If
>> > >>> I'm wrong, please correct me.
>> > >>>
>> > >>> 1.
>> > >>>
>> >
>> https://issues.apache.org/jira/browse/HAMA-531?focusedCommentId=13483009&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13483009
>> > >>>
>> > >>> --
>> > >>> Best Regards, Edward J. Yoon
>> > >>> @eddieyoon
>> > >>>
>> >
>> >
>> >
>> > --
>> > Best Regards, Edward J. Yoon
>> > @eddieyoon
>> >
>>
--
Best Regards, Edward J. Yoon
@eddieyoon
Re: Create common module
Posted by Suraj Menon <me...@gmail.com>.
+1 for hama-io, +0 for hama-commons. Can you look at the patch for HAMA-700
and let me know if you have any issues going forward in that direction.
Basically it includes a raw format to intermediate binary format
conversion, that could be overridden as shown in VertexInputReader
implementation. Now my question is, if the input to be read from HBase, do
we write every record from HBase to sequential file format? I am a little
confused here.
Thanks,
Suraj
On Fri, Dec 14, 2012 at 2:19 AM, Tommaso Teofili
<to...@gmail.com>wrote:
> thinking about it a bit further, maybe a separate module for IO related
> stuff may be worth, so hama-io instead of hama-commons.
> Whatever it is I'd like to avoid duplicated code.
> Tommaso
>
>
> 2012/12/13 Edward J. Yoon <ed...@apache.org>
>
> > Otherwise, we have to copy VertexInputReader class to core module or
> > add own partitioner to graph package.
> >
> > In fact, I don't want to put any graph-related code to core module.
> > But if we want to support VertexInputReader, I think this is best.
> >
> > On Fri, Dec 14, 2012 at 4:22 AM, Apurv Verma <da...@gmail.com> wrote:
> > > +0 I am not against but I think that would be an overkill for now.
> > >
> > > --
> > > Regards,
> > > Apurv Verma
> > >
> > >
> > >
> > >
> > > On Thu, Dec 13, 2012 at 12:32 PM, Tommaso Teofili
> > > <to...@gmail.com> wrote:
> > >> that sounds good to me, +1 for a hama-commons module.
> > >>
> > >> Tommaso
> > >>
> > >>
> > >> 2012/12/13 Edward J. Yoon <ed...@apache.org>
> > >>
> > >>> Hi,
> > >>>
> > >>> I propose to create common module to share common classes among all
> > >>> modules.
> > >>>
> > >>> At the moment, the main reason is to support VertexInputReader. As I
> > >>> mentioned here[1], there's no way to integrate partitioner without
> > >>> merging graph to core or adding graph own partitioner. If we create
> > >>> common module, it'll all be clear. I'm not maven and java expert. If
> > >>> I'm wrong, please correct me.
> > >>>
> > >>> 1.
> > >>>
> >
> https://issues.apache.org/jira/browse/HAMA-531?focusedCommentId=13483009&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13483009
> > >>>
> > >>> --
> > >>> Best Regards, Edward J. Yoon
> > >>> @eddieyoon
> > >>>
> >
> >
> >
> > --
> > Best Regards, Edward J. Yoon
> > @eddieyoon
> >
>
Re: Create common module
Posted by Tommaso Teofili <to...@gmail.com>.
thinking about it a bit further, maybe a separate module for IO related
stuff may be worth, so hama-io instead of hama-commons.
Whatever it is I'd like to avoid duplicated code.
Tommaso
2012/12/13 Edward J. Yoon <ed...@apache.org>
> Otherwise, we have to copy VertexInputReader class to core module or
> add own partitioner to graph package.
>
> In fact, I don't want to put any graph-related code to core module.
> But if we want to support VertexInputReader, I think this is best.
>
> On Fri, Dec 14, 2012 at 4:22 AM, Apurv Verma <da...@gmail.com> wrote:
> > +0 I am not against but I think that would be an overkill for now.
> >
> > --
> > Regards,
> > Apurv Verma
> >
> >
> >
> >
> > On Thu, Dec 13, 2012 at 12:32 PM, Tommaso Teofili
> > <to...@gmail.com> wrote:
> >> that sounds good to me, +1 for a hama-commons module.
> >>
> >> Tommaso
> >>
> >>
> >> 2012/12/13 Edward J. Yoon <ed...@apache.org>
> >>
> >>> Hi,
> >>>
> >>> I propose to create common module to share common classes among all
> >>> modules.
> >>>
> >>> At the moment, the main reason is to support VertexInputReader. As I
> >>> mentioned here[1], there's no way to integrate partitioner without
> >>> merging graph to core or adding graph own partitioner. If we create
> >>> common module, it'll all be clear. I'm not maven and java expert. If
> >>> I'm wrong, please correct me.
> >>>
> >>> 1.
> >>>
> https://issues.apache.org/jira/browse/HAMA-531?focusedCommentId=13483009&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13483009
> >>>
> >>> --
> >>> Best Regards, Edward J. Yoon
> >>> @eddieyoon
> >>>
>
>
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon
>
Re: Create common module
Posted by "Edward J. Yoon" <ed...@apache.org>.
Otherwise, we have to copy VertexInputReader class to core module or
add own partitioner to graph package.
In fact, I don't want to put any graph-related code to core module.
But if we want to support VertexInputReader, I think this is best.
On Fri, Dec 14, 2012 at 4:22 AM, Apurv Verma <da...@gmail.com> wrote:
> +0 I am not against but I think that would be an overkill for now.
>
> --
> Regards,
> Apurv Verma
>
>
>
>
> On Thu, Dec 13, 2012 at 12:32 PM, Tommaso Teofili
> <to...@gmail.com> wrote:
>> that sounds good to me, +1 for a hama-commons module.
>>
>> Tommaso
>>
>>
>> 2012/12/13 Edward J. Yoon <ed...@apache.org>
>>
>>> Hi,
>>>
>>> I propose to create common module to share common classes among all
>>> modules.
>>>
>>> At the moment, the main reason is to support VertexInputReader. As I
>>> mentioned here[1], there's no way to integrate partitioner without
>>> merging graph to core or adding graph own partitioner. If we create
>>> common module, it'll all be clear. I'm not maven and java expert. If
>>> I'm wrong, please correct me.
>>>
>>> 1.
>>> https://issues.apache.org/jira/browse/HAMA-531?focusedCommentId=13483009&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13483009
>>>
>>> --
>>> Best Regards, Edward J. Yoon
>>> @eddieyoon
>>>
--
Best Regards, Edward J. Yoon
@eddieyoon
Re: Create common module
Posted by Apurv Verma <da...@gmail.com>.
+0 I am not against but I think that would be an overkill for now.
--
Regards,
Apurv Verma
On Thu, Dec 13, 2012 at 12:32 PM, Tommaso Teofili
<to...@gmail.com> wrote:
> that sounds good to me, +1 for a hama-commons module.
>
> Tommaso
>
>
> 2012/12/13 Edward J. Yoon <ed...@apache.org>
>
>> Hi,
>>
>> I propose to create common module to share common classes among all
>> modules.
>>
>> At the moment, the main reason is to support VertexInputReader. As I
>> mentioned here[1], there's no way to integrate partitioner without
>> merging graph to core or adding graph own partitioner. If we create
>> common module, it'll all be clear. I'm not maven and java expert. If
>> I'm wrong, please correct me.
>>
>> 1.
>> https://issues.apache.org/jira/browse/HAMA-531?focusedCommentId=13483009&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13483009
>>
>> --
>> Best Regards, Edward J. Yoon
>> @eddieyoon
>>
Re: Create common module
Posted by Tommaso Teofili <to...@gmail.com>.
that sounds good to me, +1 for a hama-commons module.
Tommaso
2012/12/13 Edward J. Yoon <ed...@apache.org>
> Hi,
>
> I propose to create common module to share common classes among all
> modules.
>
> At the moment, the main reason is to support VertexInputReader. As I
> mentioned here[1], there's no way to integrate partitioner without
> merging graph to core or adding graph own partitioner. If we create
> common module, it'll all be clear. I'm not maven and java expert. If
> I'm wrong, please correct me.
>
> 1.
> https://issues.apache.org/jira/browse/HAMA-531?focusedCommentId=13483009&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13483009
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon
>