You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@giraph.apache.org by Benjamin Heitmann <be...@deri.org> on 2012/05/10 18:43:08 UTC
Running munge maven plugin from Eclipse? (slightly off-topic)
Hello,
since giraph requires the maven munge plugin in order to have correct source,
and since google did not find any existing solution, I though it might be okay to ask this here:
How can I tell eclipse or the m2e maven eclipse plugin to execute the munge maven plugin so that I have
giraph sources which actually compile in eclipse without errors ?
Right now, I need to edit the java source files which are broken due to the munge syntax manually.
However, I need to do this every time those files get changed.
Is there an easier way to do this ?
cheers, Benjamin.
Re: Running munge maven plugin from Eclipse? (slightly off-topic)
Posted by Jakob Homan <jg...@gmail.com>.
Three or four small patches is equivalent to three or four branches,
one for each version of Hadoop. That seems like a pretty big hassle
over the long term.
On Thu, May 31, 2012 at 11:28 AM, Paolo Castagna
<ca...@googlemail.com> wrote:
> Jakob Homan wrote:
>>> While I see no problem in replacing the Hadoop RPC with the Netty implementation
>>> Avery contributed, I am not 100% sure about the implications in relation to
>>> secure vs. non-secure versions of Hadoop.
>> Those running netty in a secure cluster need to know we're totally
>> secure. Agreed that we need a better story in this situation.
>>
>>
>>> If changes to support different versions of Hadoop cannot be factored out,
>>> another option (IMHO better than the current situation) would be to maintain two
>>> or three small patches that people who want to use Giraph with a different
>>> version of Hadoop can eventually apply.
>> Munging needs to go, the sooner the better; shimming may be in the
>> future, although that's not ideal either. One option would be to not
>> support older versions aggressively, particularly with cross-version
>> rpc compatibility coming soon. But I'd like to avoid a roll-your-own
>> approach; we need to officially support and have versions for
>> different Hadoops.
>>
>> -jg
>
> Jakob, given the fact that the changes to support multiple version of Hadoop in
> Giraph are minimal, do you think it would be possible to maintain three or four
> small patch files in the Giraph trunk and use those instead of munging and/or
> shimming?
>
> We can configure Jenkins to apply each patch and test at each commit.
>
> This is unusual and I would not propose it if the patches weren't very small and
> not willing to change much.
>
> Paolo
>
Re: Running munge maven plugin from Eclipse? (slightly off-topic)
Posted by Paolo Castagna <ca...@googlemail.com>.
Jakob Homan wrote:
>> While I see no problem in replacing the Hadoop RPC with the Netty implementation
>> Avery contributed, I am not 100% sure about the implications in relation to
>> secure vs. non-secure versions of Hadoop.
> Those running netty in a secure cluster need to know we're totally
> secure. Agreed that we need a better story in this situation.
>
>
>> If changes to support different versions of Hadoop cannot be factored out,
>> another option (IMHO better than the current situation) would be to maintain two
>> or three small patches that people who want to use Giraph with a different
>> version of Hadoop can eventually apply.
> Munging needs to go, the sooner the better; shimming may be in the
> future, although that's not ideal either. One option would be to not
> support older versions aggressively, particularly with cross-version
> rpc compatibility coming soon. But I'd like to avoid a roll-your-own
> approach; we need to officially support and have versions for
> different Hadoops.
>
> -jg
Jakob, given the fact that the changes to support multiple version of Hadoop in
Giraph are minimal, do you think it would be possible to maintain three or four
small patch files in the Giraph trunk and use those instead of munging and/or
shimming?
We can configure Jenkins to apply each patch and test at each commit.
This is unusual and I would not propose it if the patches weren't very small and
not willing to change much.
Paolo
Re: Running munge maven plugin from Eclipse? (slightly off-topic)
Posted by Claudio Martella <cl...@gmail.com>.
Talking yesterday at the workshop with the other committers and users it
seams obvious that we should really get rid of the munge headache ASAP. I
wish I had a solution though, probably branching looks the best right now.
On Thursday, June 7, 2012, Avery Ching wrote:
> Regarding the netty secureness, we can of course add this to netty (i.e.
> ssl is built in to netty).
>
> Avery
>
> On 5/31/12 10:53 AM, Jakob Homan wrote:
>
>> While I see no problem in replacing the Hadoop RPC with the Netty
>>> implementation
>>> Avery contributed, I am not 100% sure about the implications in relation
>>> to
>>> secure vs. non-secure versions of Hadoop.
>>>
>> Those running netty in a secure cluster need to know we're totally
>> secure. Agreed that we need a better story in this situation.
>>
>>
>> If changes to support different versions of Hadoop cannot be factored
>>> out,
>>> another option (IMHO better than the current situation) would be to
>>> maintain two
>>> or three small patches that people who want to use Giraph with a
>>> different
>>> version of Hadoop can eventually apply.
>>>
>> Munging needs to go, the sooner the better; shimming may be in the
>> future, although that's not ideal either. One option would be to not
>> support older versions aggressively, particularly with cross-version
>> rpc compatibility coming soon. But I'd like to avoid a roll-your-own
>> approach; we need to officially support and have versions for
>> different Hadoops.
>>
>> -jg
>>
>
>
--
Claudio Martella
claudio.martella@gmail.com
Running munge maven plugin from Eclipse? (slightly off-topic)
Posted by Claudio Martella <cl...@gmail.com>.
Talking yesterday at the workshop with the other committers and users it
seams obvious that we should really get rid of the munge headache ASAP. I
wish I had a solution though, probably branching looks the best right now.
On Thursday, June 7, 2012, Avery Ching wrote:
> Regarding the netty secureness, we can of course add this to netty (i.e.
> ssl is built in to netty).
>
> Avery
>
> On 5/31/12 10:53 AM, Jakob Homan wrote:
>
>> While I see no problem in replacing the Hadoop RPC with the Netty
>>> implementation
>>> Avery contributed, I am not 100% sure about the implications in relation
>>> to
>>> secure vs. non-secure versions of Hadoop.
>>>
>> Those running netty in a secure cluster need to know we're totally
>> secure. Agreed that we need a better story in this situation.
>>
>>
>> If changes to support different versions of Hadoop cannot be factored
>>> out,
>>> another option (IMHO better than the current situation) would be to
>>> maintain two
>>> or three small patches that people who want to use Giraph with a
>>> different
>>> version of Hadoop can eventually apply.
>>>
>> Munging needs to go, the sooner the better; shimming may be in the
>> future, although that's not ideal either. One option would be to not
>> support older versions aggressively, particularly with cross-version
>> rpc compatibility coming soon. But I'd like to avoid a roll-your-own
>> approach; we need to officially support and have versions for
>> different Hadoops.
>>
>> -jg
>>
>
>
--
Claudio Martella
claudio.martella@gmail.com
Re: Running munge maven plugin from Eclipse? (slightly off-topic)
Posted by Avery Ching <ac...@apache.org>.
Regarding the netty secureness, we can of course add this to netty (i.e.
ssl is built in to netty).
Avery
On 5/31/12 10:53 AM, Jakob Homan wrote:
>> While I see no problem in replacing the Hadoop RPC with the Netty implementation
>> Avery contributed, I am not 100% sure about the implications in relation to
>> secure vs. non-secure versions of Hadoop.
> Those running netty in a secure cluster need to know we're totally
> secure. Agreed that we need a better story in this situation.
>
>
>> If changes to support different versions of Hadoop cannot be factored out,
>> another option (IMHO better than the current situation) would be to maintain two
>> or three small patches that people who want to use Giraph with a different
>> version of Hadoop can eventually apply.
> Munging needs to go, the sooner the better; shimming may be in the
> future, although that's not ideal either. One option would be to not
> support older versions aggressively, particularly with cross-version
> rpc compatibility coming soon. But I'd like to avoid a roll-your-own
> approach; we need to officially support and have versions for
> different Hadoops.
>
> -jg
Re: Running munge maven plugin from Eclipse? (slightly off-topic)
Posted by Jakob Homan <jg...@gmail.com>.
> While I see no problem in replacing the Hadoop RPC with the Netty implementation
> Avery contributed, I am not 100% sure about the implications in relation to
> secure vs. non-secure versions of Hadoop.
Those running netty in a secure cluster need to know we're totally
secure. Agreed that we need a better story in this situation.
> If changes to support different versions of Hadoop cannot be factored out,
> another option (IMHO better than the current situation) would be to maintain two
> or three small patches that people who want to use Giraph with a different
> version of Hadoop can eventually apply.
Munging needs to go, the sooner the better; shimming may be in the
future, although that's not ideal either. One option would be to not
support older versions aggressively, particularly with cross-version
rpc compatibility coming soon. But I'd like to avoid a roll-your-own
approach; we need to officially support and have versions for
different Hadoops.
-jg
Re: Running munge maven plugin from Eclipse? (slightly off-topic)
Posted by Paolo Castagna <ca...@googlemail.com>.
Benjamin Heitmann wrote:
> On 25 May 2012, at 08:19, Avery Ching wrote:
>
>> Actually, I suspect that we can remove munge if we get rid of the Hadoop RPC
implementation and only use the Netty one. Any thoughts on that?
>
> +1
>
> That would be great.
>
> Its hard to say if the munge maven plugin is actively maintained/supported,
there does not really seem to be a web site for it AFAIK...
Hi Benjamin, Sebastian, Avery,
there is GIRAPH-196 to track progress on this:
https://issues.apache.org/jira/browse/GIRAPH-196 (initial patch available)
While I see no problem in replacing the Hadoop RPC with the Netty implementation
Avery contributed, I am not 100% sure about the implications in relation to
secure vs. non-secure versions of Hadoop.
In particular, the removal of BspPolicyProvider.java and BspTokenSelector.java
(that might be a mistake).
>From my point of view, not being able to work with the support of an IDE as
normal (I use Eclipse) is quite a limitation. In particular when you need to
refactor some code. I work and make my changes with Eclipse, as usual, but those
are on the generated code. Then you need to manually port/copy them to the
'real' sources in src/ and you do not have the help of SVN to tell you what's
changed since generated sources are in target and (rightly so) not under version
control.
If changes to support different versions of Hadoop cannot be factored out,
another option (IMHO better than the current situation) would be to maintain two
or three small patches that people who want to use Giraph with a different
version of Hadoop can eventually apply.
What do you think?
Paolo
Re: Running munge maven plugin from Eclipse? (slightly off-topic)
Posted by Benjamin Heitmann <be...@deri.org>.
On 25 May 2012, at 08:19, Avery Ching wrote:
> Actually, I suspect that we can remove munge if we get rid of the Hadoop RPC implementation and only use the Netty one. Any thoughts on that?
+1
That would be great.
Its hard to say if the munge maven plugin is actively maintained/supported, there does not really seem to be a web site for it AFAIK...
Re: Running munge maven plugin from Eclipse? (slightly off-topic)
Posted by Sebastian Schelter <ss...@apache.org>.
+1
That would be awesome.
I cannot execute the tests in IntelliJ anymore since we integrated the
munge plugin...
Best,
Sebastian
On 25.05.2012 09:19, Avery Ching wrote:
> Actually, I suspect that we can remove munge if we get rid of the Hadoop
> RPC implementation and only use the Netty one. Any thoughts on that?
>
> Avery
>
> On 5/15/12 9:21 AM, Paolo Castagna wrote:
>> Benjamin Heitmann wrote:
>>> Hello,
>>>
>>> since giraph requires the maven munge plugin in order to have correct
>>> source,
>>> and since google did not find any existing solution, I though it
>>> might be okay to ask this here:
>>>
>>> How can I tell eclipse or the m2e maven eclipse plugin to execute the
>>> munge maven plugin so that I have
>>> giraph sources which actually compile in eclipse without errors ?
>>>
>>> Right now, I need to edit the java source files which are broken due
>>> to the munge syntax manually.
>>> However, I need to do this every time those files get changed.
>>>
>>> Is there an easier way to do this ?
>> Hi Benjamin,
>> I do not have an answer/solution to your question.
>> Just wanted to say I experience same pain.
>>
>> The goal of supporting all possible variants/version of Hadoop is
>> great... but
>> it comes with a cost and pain.
>>
>> Paolo
>>
>>>
>>> cheers, Benjamin.
>
Re: Running munge maven plugin from Eclipse? (slightly off-topic)
Posted by Avery Ching <ac...@apache.org>.
Actually, I suspect that we can remove munge if we get rid of the Hadoop
RPC implementation and only use the Netty one. Any thoughts on that?
Avery
On 5/15/12 9:21 AM, Paolo Castagna wrote:
> Benjamin Heitmann wrote:
>> Hello,
>>
>> since giraph requires the maven munge plugin in order to have correct source,
>> and since google did not find any existing solution, I though it might be okay to ask this here:
>>
>> How can I tell eclipse or the m2e maven eclipse plugin to execute the munge maven plugin so that I have
>> giraph sources which actually compile in eclipse without errors ?
>>
>> Right now, I need to edit the java source files which are broken due to the munge syntax manually.
>> However, I need to do this every time those files get changed.
>>
>> Is there an easier way to do this ?
> Hi Benjamin,
> I do not have an answer/solution to your question.
> Just wanted to say I experience same pain.
>
> The goal of supporting all possible variants/version of Hadoop is great... but
> it comes with a cost and pain.
>
> Paolo
>
>>
>> cheers, Benjamin.
Re: Running munge maven plugin from Eclipse? (slightly off-topic)
Posted by Paolo Castagna <ca...@googlemail.com>.
Benjamin Heitmann wrote:
> Hello,
>
> since giraph requires the maven munge plugin in order to have correct source,
> and since google did not find any existing solution, I though it might be okay to ask this here:
>
> How can I tell eclipse or the m2e maven eclipse plugin to execute the munge maven plugin so that I have
> giraph sources which actually compile in eclipse without errors ?
>
> Right now, I need to edit the java source files which are broken due to the munge syntax manually.
> However, I need to do this every time those files get changed.
>
> Is there an easier way to do this ?
Hi Benjamin,
I do not have an answer/solution to your question.
Just wanted to say I experience same pain.
The goal of supporting all possible variants/version of Hadoop is great... but
it comes with a cost and pain.
Paolo
>
>
> cheers, Benjamin.