You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Josh Elser <el...@apache.org> on 2018/03/15 19:03:09 UTC
[DISCUSS] Compilation on $ARCH which protoc doesn't support
I'm faced with what I think is a tricky issue. I'm trying to help folks
to compile HBase on PPC. The protobuf community does not publish a PPC
binary for protoc which means that the HBase build can't generate our
protobuf files. Protobuf supports: x86/x86_64 for linux, windows, and
osx in 3.3.0 (an aarch_64 starting in 3.5.0).
I'm trying to think through any/all solutions:
1. Get protobuf folks to build/publish protoc for PPC.
2. Commit our generated java files to the repository and provide an
option to skip compilation on unsupported ARCHs.
3. Try to find/create some non-official PPC support ala
https://github.com/os72/protoc-jar
4. Do nothing
Evaluating these, #1 and #3 are a "later" solution. #2 puts some pain on
developers to git-add these files if they are modified. Maybe not a lot
of pain since protocol changes should be infrequent at this point?
There seems to be some stalled work for #1 via
https://github.com/google/protobuf/pull/3506
What do people think? I'm probably between a rock and a hard place to
get this work one way or another in the immediate. If there's support
for this in the community, I'd prefer to work towards that.
- Josh
Re: [DISCUSS] Compilation on $ARCH which protoc doesn't support
Posted by Josh Elser <el...@apache.org>.
Coming full circle: https://github.com/google/protobuf/pull/4411
I got some traction from some acquaintances who actually know ppc64le.
Hopefully we can just drag our feet long enough and will require nothing
from our end :)
On 3/16/18 1:27 PM, Josh Elser wrote:
> Hey Duo,
>
> Yeah, that's #1 that I outlined. See also the link to the pull request
> which has stalled since last Fall. My C++ skills are beyond rusty, and I
> wouldn't know where to start with PPC in general.
>
> I've reached out to some folks in the hopes of getting someone who can
> drive it home as, like you allude, this would be ideal and require no
> change from us in HBase.
>
> On 3/15/18 8:52 PM, 张铎(Duo Zhang) wrote:
>> IIRC there is an issue on compiling hbase on arm, and the guy there faced
>> the same problem. Seems the final solution is that they filed an issue
>> for
>> the protobuf project and now there is a pre complied binary for arm64
>> available in the central maven repo...
>>
>> http://central.maven.org/maven2/com/google/protobuf/protoc/3.5.1-1/
>>
>> There is a 'protoc-3.5.1-1-linux-aarch_64.exe'
>>
>> So is it possible to do the same thing for ppc?
>>
>> 2018-03-16 4:09 GMT+08:00 Josh Elser <el...@apache.org>:
>>
>>>
>>>
>>> On 3/15/18 3:45 PM, Stack wrote:
>>>
>>>> On Thu, Mar 15, 2018 at 12:03 PM, Josh Elser <el...@apache.org> wrote:
>>>>
>>>> I'm faced with what I think is a tricky issue. I'm trying to help
>>>> folks to
>>>>> compile HBase on PPC. The protobuf community does not publish a PPC
>>>>> binary
>>>>> for protoc which means that the HBase build can't generate our
>>>>> protobuf
>>>>> files. Protobuf supports: x86/x86_64 for linux, windows, and osx in
>>>>> 3.3.0
>>>>> (an aarch_64 starting in 3.5.0).
>>>>>
>>>>> I'm trying to think through any/all solutions:
>>>>>
>>>>> 1. Get protobuf folks to build/publish protoc for PPC.
>>>>> 2. Commit our generated java files to the repository and provide an
>>>>> option
>>>>> to skip compilation on unsupported ARCHs.
>>>>> 3. Try to find/create some non-official PPC support ala
>>>>> https://github.com/os72/protoc-jar
>>>>> 4. Do nothing
>>>>>
>>>>> Evaluating these, #1 and #3 are a "later" solution. #2 puts some
>>>>> pain on
>>>>> developers to git-add these files if they are modified. Maybe not a
>>>>> lot
>>>>> of
>>>>> pain since protocol changes should be infrequent at this point?
>>>>>
>>>>>
>>>>> We used to do this but after some work the nice protobuf-maven-plugin
>>>> that
>>>> provides platform-appropriate protoc made it so we could skip having
>>>> to do
>>>> this halving (IIRC) the size of our checkout.
>>>>
>>>> Have a PPC profile that messes w/ the module list? (Claim on the
>>>> profiles
>>>> doc page is that its possible[1]). Exclude hbase-protocol* modules
>>>> if os
>>>> finds that the platform is PPC? (Activating a profile based off os is
>>>> common usage). Have the PPC profile depend on published versions of
>>>> these
>>>> jars? Would something like this do Josh?
>>>>
>>>> St.Ack
>>>> 1. http://maven.apache.org/guides/introduction/introduction-to-
>>>> profiles.html
>>>>
>>>
>>> Interesting... I had the idea, but brushed it off because I didn't think
>>> there was a way to accomplish it realistically. Your solution makes me
>>> think it might work.
>>>
>>> I remember the checked-in-compiled-code version (and much prefer what we
>>> have now, sadly). Let me poke at your suggestion and see what I can
>>> frankenstein together.
>>>
>>> Thanks!
>>>
>>>
>>
Re: [DISCUSS] Compilation on $ARCH which protoc doesn't support
Posted by Josh Elser <el...@apache.org>.
Hey Duo,
Yeah, that's #1 that I outlined. See also the link to the pull request
which has stalled since last Fall. My C++ skills are beyond rusty, and I
wouldn't know where to start with PPC in general.
I've reached out to some folks in the hopes of getting someone who can
drive it home as, like you allude, this would be ideal and require no
change from us in HBase.
On 3/15/18 8:52 PM, 张铎(Duo Zhang) wrote:
> IIRC there is an issue on compiling hbase on arm, and the guy there faced
> the same problem. Seems the final solution is that they filed an issue for
> the protobuf project and now there is a pre complied binary for arm64
> available in the central maven repo...
>
> http://central.maven.org/maven2/com/google/protobuf/protoc/3.5.1-1/
>
> There is a 'protoc-3.5.1-1-linux-aarch_64.exe'
>
> So is it possible to do the same thing for ppc?
>
> 2018-03-16 4:09 GMT+08:00 Josh Elser <el...@apache.org>:
>
>>
>>
>> On 3/15/18 3:45 PM, Stack wrote:
>>
>>> On Thu, Mar 15, 2018 at 12:03 PM, Josh Elser <el...@apache.org> wrote:
>>>
>>> I'm faced with what I think is a tricky issue. I'm trying to help folks to
>>>> compile HBase on PPC. The protobuf community does not publish a PPC
>>>> binary
>>>> for protoc which means that the HBase build can't generate our protobuf
>>>> files. Protobuf supports: x86/x86_64 for linux, windows, and osx in 3.3.0
>>>> (an aarch_64 starting in 3.5.0).
>>>>
>>>> I'm trying to think through any/all solutions:
>>>>
>>>> 1. Get protobuf folks to build/publish protoc for PPC.
>>>> 2. Commit our generated java files to the repository and provide an
>>>> option
>>>> to skip compilation on unsupported ARCHs.
>>>> 3. Try to find/create some non-official PPC support ala
>>>> https://github.com/os72/protoc-jar
>>>> 4. Do nothing
>>>>
>>>> Evaluating these, #1 and #3 are a "later" solution. #2 puts some pain on
>>>> developers to git-add these files if they are modified. Maybe not a lot
>>>> of
>>>> pain since protocol changes should be infrequent at this point?
>>>>
>>>>
>>>> We used to do this but after some work the nice protobuf-maven-plugin
>>> that
>>> provides platform-appropriate protoc made it so we could skip having to do
>>> this halving (IIRC) the size of our checkout.
>>>
>>> Have a PPC profile that messes w/ the module list? (Claim on the profiles
>>> doc page is that its possible[1]). Exclude hbase-protocol* modules if os
>>> finds that the platform is PPC? (Activating a profile based off os is
>>> common usage). Have the PPC profile depend on published versions of these
>>> jars? Would something like this do Josh?
>>>
>>> St.Ack
>>> 1. http://maven.apache.org/guides/introduction/introduction-to-
>>> profiles.html
>>>
>>
>> Interesting... I had the idea, but brushed it off because I didn't think
>> there was a way to accomplish it realistically. Your solution makes me
>> think it might work.
>>
>> I remember the checked-in-compiled-code version (and much prefer what we
>> have now, sadly). Let me poke at your suggestion and see what I can
>> frankenstein together.
>>
>> Thanks!
>>
>>
>
Re: [DISCUSS] Compilation on $ARCH which protoc doesn't support
Posted by "张铎 (Duo Zhang)" <pa...@gmail.com>.
IIRC there is an issue on compiling hbase on arm, and the guy there faced
the same problem. Seems the final solution is that they filed an issue for
the protobuf project and now there is a pre complied binary for arm64
available in the central maven repo...
http://central.maven.org/maven2/com/google/protobuf/protoc/3.5.1-1/
There is a 'protoc-3.5.1-1-linux-aarch_64.exe'
So is it possible to do the same thing for ppc?
2018-03-16 4:09 GMT+08:00 Josh Elser <el...@apache.org>:
>
>
> On 3/15/18 3:45 PM, Stack wrote:
>
>> On Thu, Mar 15, 2018 at 12:03 PM, Josh Elser <el...@apache.org> wrote:
>>
>> I'm faced with what I think is a tricky issue. I'm trying to help folks to
>>> compile HBase on PPC. The protobuf community does not publish a PPC
>>> binary
>>> for protoc which means that the HBase build can't generate our protobuf
>>> files. Protobuf supports: x86/x86_64 for linux, windows, and osx in 3.3.0
>>> (an aarch_64 starting in 3.5.0).
>>>
>>> I'm trying to think through any/all solutions:
>>>
>>> 1. Get protobuf folks to build/publish protoc for PPC.
>>> 2. Commit our generated java files to the repository and provide an
>>> option
>>> to skip compilation on unsupported ARCHs.
>>> 3. Try to find/create some non-official PPC support ala
>>> https://github.com/os72/protoc-jar
>>> 4. Do nothing
>>>
>>> Evaluating these, #1 and #3 are a "later" solution. #2 puts some pain on
>>> developers to git-add these files if they are modified. Maybe not a lot
>>> of
>>> pain since protocol changes should be infrequent at this point?
>>>
>>>
>>> We used to do this but after some work the nice protobuf-maven-plugin
>> that
>> provides platform-appropriate protoc made it so we could skip having to do
>> this halving (IIRC) the size of our checkout.
>>
>> Have a PPC profile that messes w/ the module list? (Claim on the profiles
>> doc page is that its possible[1]). Exclude hbase-protocol* modules if os
>> finds that the platform is PPC? (Activating a profile based off os is
>> common usage). Have the PPC profile depend on published versions of these
>> jars? Would something like this do Josh?
>>
>> St.Ack
>> 1. http://maven.apache.org/guides/introduction/introduction-to-
>> profiles.html
>>
>
> Interesting... I had the idea, but brushed it off because I didn't think
> there was a way to accomplish it realistically. Your solution makes me
> think it might work.
>
> I remember the checked-in-compiled-code version (and much prefer what we
> have now, sadly). Let me poke at your suggestion and see what I can
> frankenstein together.
>
> Thanks!
>
>
Re: [DISCUSS] Compilation on $ARCH which protoc doesn't support
Posted by Josh Elser <el...@apache.org>.
On 3/15/18 3:45 PM, Stack wrote:
> On Thu, Mar 15, 2018 at 12:03 PM, Josh Elser <el...@apache.org> wrote:
>
>> I'm faced with what I think is a tricky issue. I'm trying to help folks to
>> compile HBase on PPC. The protobuf community does not publish a PPC binary
>> for protoc which means that the HBase build can't generate our protobuf
>> files. Protobuf supports: x86/x86_64 for linux, windows, and osx in 3.3.0
>> (an aarch_64 starting in 3.5.0).
>>
>> I'm trying to think through any/all solutions:
>>
>> 1. Get protobuf folks to build/publish protoc for PPC.
>> 2. Commit our generated java files to the repository and provide an option
>> to skip compilation on unsupported ARCHs.
>> 3. Try to find/create some non-official PPC support ala
>> https://github.com/os72/protoc-jar
>> 4. Do nothing
>>
>> Evaluating these, #1 and #3 are a "later" solution. #2 puts some pain on
>> developers to git-add these files if they are modified. Maybe not a lot of
>> pain since protocol changes should be infrequent at this point?
>>
>>
> We used to do this but after some work the nice protobuf-maven-plugin that
> provides platform-appropriate protoc made it so we could skip having to do
> this halving (IIRC) the size of our checkout.
>
> Have a PPC profile that messes w/ the module list? (Claim on the profiles
> doc page is that its possible[1]). Exclude hbase-protocol* modules if os
> finds that the platform is PPC? (Activating a profile based off os is
> common usage). Have the PPC profile depend on published versions of these
> jars? Would something like this do Josh?
>
> St.Ack
> 1. http://maven.apache.org/guides/introduction/introduction-to-profiles.html
Interesting... I had the idea, but brushed it off because I didn't think
there was a way to accomplish it realistically. Your solution makes me
think it might work.
I remember the checked-in-compiled-code version (and much prefer what we
have now, sadly). Let me poke at your suggestion and see what I can
frankenstein together.
Thanks!
Re: [DISCUSS] Compilation on $ARCH which protoc doesn't support
Posted by Stack <st...@duboce.net>.
On Thu, Mar 15, 2018 at 12:03 PM, Josh Elser <el...@apache.org> wrote:
> I'm faced with what I think is a tricky issue. I'm trying to help folks to
> compile HBase on PPC. The protobuf community does not publish a PPC binary
> for protoc which means that the HBase build can't generate our protobuf
> files. Protobuf supports: x86/x86_64 for linux, windows, and osx in 3.3.0
> (an aarch_64 starting in 3.5.0).
>
> I'm trying to think through any/all solutions:
>
> 1. Get protobuf folks to build/publish protoc for PPC.
> 2. Commit our generated java files to the repository and provide an option
> to skip compilation on unsupported ARCHs.
> 3. Try to find/create some non-official PPC support ala
> https://github.com/os72/protoc-jar
> 4. Do nothing
>
> Evaluating these, #1 and #3 are a "later" solution. #2 puts some pain on
> developers to git-add these files if they are modified. Maybe not a lot of
> pain since protocol changes should be infrequent at this point?
>
>
We used to do this but after some work the nice protobuf-maven-plugin that
provides platform-appropriate protoc made it so we could skip having to do
this halving (IIRC) the size of our checkout.
Have a PPC profile that messes w/ the module list? (Claim on the profiles
doc page is that its possible[1]). Exclude hbase-protocol* modules if os
finds that the platform is PPC? (Activating a profile based off os is
common usage). Have the PPC profile depend on published versions of these
jars? Would something like this do Josh?
St.Ack
1. http://maven.apache.org/guides/introduction/introduction-to-profiles.html
> There seems to be some stalled work for #1 via
> https://github.com/google/protobuf/pull/3506
>
> What do people think? I'm probably between a rock and a hard place to get
> this work one way or another in the immediate. If there's support for this
> in the community, I'd prefer to work towards that.
>
> - Josh
>