You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Steve Loughran <st...@cloudera.com> on 2026/03/08 14:58:19 UTC
differences between x86 and aarch JAR artifacts
I spent a couple of hours this w/e getting claude to answer the question
for me "do the x86 and aarch JAR files differ?
The 3.4.3 release ended up using the arm build as its source of jar files
because using the cloud x86 vm I was using for release produced multiple
staging repos in apache nexus, something suggested to be VPN/IP address
related: nexus saw requests coming in from different source IP addresses
and so assigned the artifacts to different repos.
I was curious about whether the binaries were different, and also whether
it'd be possible to detect and defend against malicious release managers.
That is: if I'd added a back door into the code, would it have been
detected.
Here then is my auditor
https://github.com/steveloughran/auditor
And here are the results.
https://gist.github.com/steveloughran/d3c9ad6a718bfec68085b08584ae414e
The main issue to flag is that in hadoop common, the protobuf classes are
somehow different. leveldbjni is different too, which is interesting but
not too concerning, though "auditor" does flag the different code is
looking at the system environment so extra suspicious.
I doubt thats new; just something we've never noticed before. Whatever the
native protoc compilers are doing, they seem to be generating different
classes, even with the same shaded protobuf being used.
Re: differences between x86 and aarch JAR artifacts
Posted by Cheng Pan <pa...@gmail.com>.
I opened HADOOP-19843 (https://github.com/apache/hadoop/pull/8350)
to regenerate it, could you please take a look?
Thanks,
Cheng Pan
> On Mar 18, 2026, at 02:01, Steve Loughran <st...@cloudera.com> wrote:
>
> I don't know, I really don't know.
>
> I think regenerating from x86 is best
>
> On Mon, 16 Mar 2026 at 09:34, Cheng Pan <pa...@gmail.com> wrote:
>
>> Hi Steve,
>>
>> So what should we do for the ARM pb2 code for 3.5.0? Delete it or
>> regenerate it?
>>
>> Thanks,
>> Cheng Pan
>>
>>
>>
>>> On Mar 10, 2026, at 22:22, Edward Capriolo <ed...@gmail.com>
>> wrote:
>>>
>>> Hello.
>>>
>>> These jars are the bane of my existence.
>>> 1) The way hadoop gets packaged the audit tools that look at pom
>>> dependencies claim linkages to proto 2.5.0, this makes hadoop spark etc,
>>> all the downstream complain and audit owners look for remidiation..
>>>
>>> 2) when i build on alpine, maven tries to download the plugins that
>>> download protobuf2.5
>>> 2a) because of this I sometimes need to compile and install and ancient
>>> protoc on my system
>>>
>>> As you may know I have tried to introduce a newer generation protobuf
>>> plugin. It is working but not without challenges
>>>
>>> https://issues.apache.org/jira/browse/YARN-11930
>>>
>>> Once the Java 8 dominio topples we should knock out proto bug 3.5.0 next
>>>
>>>
>>> On Mon, Mar 9, 2026 at 8:02 AM Cheng Pan <pa...@gmail.com> wrote:
>>>
>>>> I checked the trunk branch, the x86 generated pb source code does not
>>>> match the shipped version under arm-java.
>>>>
>>>> On M1 macOS, you can run `CPU_ARCH=x86_64 ./start-build-env.sh` to
>>>> leverage Rosetta2 to emulate x86 platform with 50% performance.
>>>>
>>>> Then compile the project(at least the hadoop-common module) and use
>> `diff`
>>>> command to compare those two files
>>>> ```
>>>> $ ./mvnw clean install -DskipTests -pl
>> hadoop-common-project/hadoop-common
>>>> -am
>>>> $ diff
>>>>
>> hadoop-common-project/hadoop-common/src/main/arm-java/org/apache/hadoop/ipc/protobuf/ProtobufRpcEngineProtos.java
>>>>
>> hadoop-common-project/hadoop-common/target/generated-sources/java/org/apache/hadoop/ipc/protobuf/ProtobufRpcEngineProtos.java
>>>> ```
>>>>
>>>> Here is my generated ProtobufRpcEngineProtos.java
>>>>
>>>> https://gist.github.com/pan3793/0ffd495736c9373a9d3c24a354eb6997
>>>>
>>>>
>>>> From the JIRA, seems this only affected Ozone, and is likely already
>> fixed
>>>> by Ozone. So, safe to remove?
>>>>
>>>> Thanks,
>>>> Cheng Pan
>>>>
>>>>
>>>>
>>
>>
Re: differences between x86 and aarch JAR artifacts
Posted by Steve Loughran <st...@cloudera.com>.
I don't know, I really don't know.
I think regenerating from x86 is best
On Mon, 16 Mar 2026 at 09:34, Cheng Pan <pa...@gmail.com> wrote:
> Hi Steve,
>
> So what should we do for the ARM pb2 code for 3.5.0? Delete it or
> regenerate it?
>
> Thanks,
> Cheng Pan
>
>
>
> > On Mar 10, 2026, at 22:22, Edward Capriolo <ed...@gmail.com>
> wrote:
> >
> > Hello.
> >
> > These jars are the bane of my existence.
> > 1) The way hadoop gets packaged the audit tools that look at pom
> > dependencies claim linkages to proto 2.5.0, this makes hadoop spark etc,
> > all the downstream complain and audit owners look for remidiation..
> >
> > 2) when i build on alpine, maven tries to download the plugins that
> > download protobuf2.5
> > 2a) because of this I sometimes need to compile and install and ancient
> > protoc on my system
> >
> > As you may know I have tried to introduce a newer generation protobuf
> > plugin. It is working but not without challenges
> >
> > https://issues.apache.org/jira/browse/YARN-11930
> >
> > Once the Java 8 dominio topples we should knock out proto bug 3.5.0 next
> >
> >
> > On Mon, Mar 9, 2026 at 8:02 AM Cheng Pan <pa...@gmail.com> wrote:
> >
> >> I checked the trunk branch, the x86 generated pb source code does not
> >> match the shipped version under arm-java.
> >>
> >> On M1 macOS, you can run `CPU_ARCH=x86_64 ./start-build-env.sh` to
> >> leverage Rosetta2 to emulate x86 platform with 50% performance.
> >>
> >> Then compile the project(at least the hadoop-common module) and use
> `diff`
> >> command to compare those two files
> >> ```
> >> $ ./mvnw clean install -DskipTests -pl
> hadoop-common-project/hadoop-common
> >> -am
> >> $ diff
> >>
> hadoop-common-project/hadoop-common/src/main/arm-java/org/apache/hadoop/ipc/protobuf/ProtobufRpcEngineProtos.java
> >>
> hadoop-common-project/hadoop-common/target/generated-sources/java/org/apache/hadoop/ipc/protobuf/ProtobufRpcEngineProtos.java
> >> ```
> >>
> >> Here is my generated ProtobufRpcEngineProtos.java
> >>
> >> https://gist.github.com/pan3793/0ffd495736c9373a9d3c24a354eb6997
> >>
> >>
> >> From the JIRA, seems this only affected Ozone, and is likely already
> fixed
> >> by Ozone. So, safe to remove?
> >>
> >> Thanks,
> >> Cheng Pan
> >>
> >>
> >>
>
>
Re: differences between x86 and aarch JAR artifacts
Posted by Cheng Pan <pa...@gmail.com>.
Hi Steve,
So what should we do for the ARM pb2 code for 3.5.0? Delete it or regenerate it?
Thanks,
Cheng Pan
> On Mar 10, 2026, at 22:22, Edward Capriolo <ed...@gmail.com> wrote:
>
> Hello.
>
> These jars are the bane of my existence.
> 1) The way hadoop gets packaged the audit tools that look at pom
> dependencies claim linkages to proto 2.5.0, this makes hadoop spark etc,
> all the downstream complain and audit owners look for remidiation..
>
> 2) when i build on alpine, maven tries to download the plugins that
> download protobuf2.5
> 2a) because of this I sometimes need to compile and install and ancient
> protoc on my system
>
> As you may know I have tried to introduce a newer generation protobuf
> plugin. It is working but not without challenges
>
> https://issues.apache.org/jira/browse/YARN-11930
>
> Once the Java 8 dominio topples we should knock out proto bug 3.5.0 next
>
>
> On Mon, Mar 9, 2026 at 8:02 AM Cheng Pan <pa...@gmail.com> wrote:
>
>> I checked the trunk branch, the x86 generated pb source code does not
>> match the shipped version under arm-java.
>>
>> On M1 macOS, you can run `CPU_ARCH=x86_64 ./start-build-env.sh` to
>> leverage Rosetta2 to emulate x86 platform with 50% performance.
>>
>> Then compile the project(at least the hadoop-common module) and use `diff`
>> command to compare those two files
>> ```
>> $ ./mvnw clean install -DskipTests -pl hadoop-common-project/hadoop-common
>> -am
>> $ diff
>> hadoop-common-project/hadoop-common/src/main/arm-java/org/apache/hadoop/ipc/protobuf/ProtobufRpcEngineProtos.java
>> hadoop-common-project/hadoop-common/target/generated-sources/java/org/apache/hadoop/ipc/protobuf/ProtobufRpcEngineProtos.java
>> ```
>>
>> Here is my generated ProtobufRpcEngineProtos.java
>>
>> https://gist.github.com/pan3793/0ffd495736c9373a9d3c24a354eb6997
>>
>>
>> From the JIRA, seems this only affected Ozone, and is likely already fixed
>> by Ozone. So, safe to remove?
>>
>> Thanks,
>> Cheng Pan
>>
>>
>>
Re: differences between x86 and aarch JAR artifacts
Posted by Edward Capriolo <ed...@gmail.com>.
Hello.
These jars are the bane of my existence.
1) The way hadoop gets packaged the audit tools that look at pom
dependencies claim linkages to proto 2.5.0, this makes hadoop spark etc,
all the downstream complain and audit owners look for remidiation..
2) when i build on alpine, maven tries to download the plugins that
download protobuf2.5
2a) because of this I sometimes need to compile and install and ancient
protoc on my system
As you may know I have tried to introduce a newer generation protobuf
plugin. It is working but not without challenges
https://issues.apache.org/jira/browse/YARN-11930
Once the Java 8 dominio topples we should knock out proto bug 3.5.0 next
On Mon, Mar 9, 2026 at 8:02 AM Cheng Pan <pa...@gmail.com> wrote:
> I checked the trunk branch, the x86 generated pb source code does not
> match the shipped version under arm-java.
>
> On M1 macOS, you can run `CPU_ARCH=x86_64 ./start-build-env.sh` to
> leverage Rosetta2 to emulate x86 platform with 50% performance.
>
> Then compile the project(at least the hadoop-common module) and use `diff`
> command to compare those two files
> ```
> $ ./mvnw clean install -DskipTests -pl hadoop-common-project/hadoop-common
> -am
> $ diff
> hadoop-common-project/hadoop-common/src/main/arm-java/org/apache/hadoop/ipc/protobuf/ProtobufRpcEngineProtos.java
> hadoop-common-project/hadoop-common/target/generated-sources/java/org/apache/hadoop/ipc/protobuf/ProtobufRpcEngineProtos.java
> ```
>
> Here is my generated ProtobufRpcEngineProtos.java
>
> https://gist.github.com/pan3793/0ffd495736c9373a9d3c24a354eb6997
>
>
> From the JIRA, seems this only affected Ozone, and is likely already fixed
> by Ozone. So, safe to remove?
>
> Thanks,
> Cheng Pan
>
>
>
Re: differences between x86 and aarch JAR artifacts
Posted by Cheng Pan <pa...@gmail.com>.
I checked the trunk branch, the x86 generated pb source code does not match the shipped version under arm-java.
On M1 macOS, you can run `CPU_ARCH=x86_64 ./start-build-env.sh` to leverage Rosetta2 to emulate x86 platform with 50% performance.
Then compile the project(at least the hadoop-common module) and use `diff` command to compare those two files
```
$ ./mvnw clean install -DskipTests -pl hadoop-common-project/hadoop-common -am
$ diff hadoop-common-project/hadoop-common/src/main/arm-java/org/apache/hadoop/ipc/protobuf/ProtobufRpcEngineProtos.java hadoop-common-project/hadoop-common/target/generated-sources/java/org/apache/hadoop/ipc/protobuf/ProtobufRpcEngineProtos.java
```
Here is my generated ProtobufRpcEngineProtos.java
https://gist.github.com/pan3793/0ffd495736c9373a9d3c24a354eb6997
From the JIRA, seems this only affected Ozone, and is likely already fixed by Ozone. So, safe to remove?
Thanks,
Cheng Pan
Re: differences between x86 and aarch JAR artifacts
Posted by Steve Loughran <st...@cloudera.com>.
So, what do we do here? the normal ProtobufRpcEngineProtos file is
autogenerated, and presumably shaded.
looks like it it came with the restoration of protobuf 2.5 support
https://github.com/apache/hadoop/pull/2026#issuecomment-638060134
*ProtobufRpcEngineProto.java is generated using protobuf 2.5.0, which does
not available for arm. So I have added the generated
ProtobufRpcEngineProto.java in a separare source directory in
hadoop-common\src\main\arm-java and adding this to sources in case of arm.
But Re-generating using protobuf-maven-plugin in case of x86 platform as
before.*
This would probably only surfaces with HBase 1 compatibility, which is why
it hasn't been noticed.
1. Do we care it is broken?
2. if so: how to fix
3. if not; how to remove?
On Mon, 9 Mar 2026 at 10:41, Steve Loughran <st...@cloudera.com> wrote:
> Ah, so we have different versions of the arm and normal protos? and 3.4.3
> maven has the arm versions? joy.
>
> I was thinking we needed to a dependency update release anyway....
>
> On Sun, 8 Mar 2026 at 16:35, Cheng Pan <pa...@gmail.com> wrote:
>
>> The different protobuf classes are not generated on-the-fly on ARM build,
>> it’s likely that the tracked source code is out-of-date.
>>
>>
>> https://github.com/apache/hadoop/blob/branch-3.4.3/hadoop-common-project/hadoop-common/src/main/arm-java/org/apache/hadoop/ipc/protobuf/ProtobufRpcEngineProtos.java
>>
>> Thanks,
>> Cheng Pan
>>
>>
>>
>> On Mar 8, 2026, at 22:58, Steve Loughran <st...@cloudera.com> wrote:
>>
>> I spent a couple of hours this w/e getting claude to answer the question
>> for me "do the x86 and aarch JAR files differ?
>> The 3.4.3 release ended up using the arm build as its source of jar files
>> because using the cloud x86 vm I was using for release produced multiple
>> staging repos in apache nexus, something suggested to be VPN/IP address
>> related: nexus saw requests coming in from different source IP addresses
>> and so assigned the artifacts to different repos.
>>
>> I was curious about whether the binaries were different, and also whether
>> it'd be possible to detect and defend against malicious release managers.
>> That is: if I'd added a back door into the code, would it have been
>> detected.
>>
>> Here then is my auditor
>> https://github.com/steveloughran/auditor
>>
>> And here are the results.
>> https://gist.github.com/steveloughran/d3c9ad6a718bfec68085b08584ae414e
>>
>> The main issue to flag is that in hadoop common, the protobuf classes are
>> somehow different. leveldbjni is different too, which is interesting but
>> not too concerning, though "auditor" does flag the different code is
>> looking at the system environment so extra suspicious.
>>
>> I doubt thats new; just something we've never noticed before. Whatever the
>> native protoc compilers are doing, they seem to be generating different
>> classes, even with the same shaded protobuf being used.
>>
>>
>>
>>
>>
>> Thanks,
>> Cheng Pan
>>
>
Re: differences between x86 and aarch JAR artifacts
Posted by Steve Loughran <st...@cloudera.com>.
Ah, so we have different versions of the arm and normal protos? and 3.4.3
maven has the arm versions? joy.
I was thinking we needed to a dependency update release anyway....
On Sun, 8 Mar 2026 at 16:35, Cheng Pan <pa...@gmail.com> wrote:
> The different protobuf classes are not generated on-the-fly on ARM build,
> it’s likely that the tracked source code is out-of-date.
>
>
> https://github.com/apache/hadoop/blob/branch-3.4.3/hadoop-common-project/hadoop-common/src/main/arm-java/org/apache/hadoop/ipc/protobuf/ProtobufRpcEngineProtos.java
>
> Thanks,
> Cheng Pan
>
>
>
> On Mar 8, 2026, at 22:58, Steve Loughran <st...@cloudera.com> wrote:
>
> I spent a couple of hours this w/e getting claude to answer the question
> for me "do the x86 and aarch JAR files differ?
> The 3.4.3 release ended up using the arm build as its source of jar files
> because using the cloud x86 vm I was using for release produced multiple
> staging repos in apache nexus, something suggested to be VPN/IP address
> related: nexus saw requests coming in from different source IP addresses
> and so assigned the artifacts to different repos.
>
> I was curious about whether the binaries were different, and also whether
> it'd be possible to detect and defend against malicious release managers.
> That is: if I'd added a back door into the code, would it have been
> detected.
>
> Here then is my auditor
> https://github.com/steveloughran/auditor
>
> And here are the results.
> https://gist.github.com/steveloughran/d3c9ad6a718bfec68085b08584ae414e
>
> The main issue to flag is that in hadoop common, the protobuf classes are
> somehow different. leveldbjni is different too, which is interesting but
> not too concerning, though "auditor" does flag the different code is
> looking at the system environment so extra suspicious.
>
> I doubt thats new; just something we've never noticed before. Whatever the
> native protoc compilers are doing, they seem to be generating different
> classes, even with the same shaded protobuf being used.
>
>
>
>
>
> Thanks,
> Cheng Pan
>
Re: differences between x86 and aarch JAR artifacts
Posted by Cheng Pan <pa...@gmail.com>.
The different protobuf classes are not generated on-the-fly on ARM build,
it’s likely that the tracked source code is out-of-date.
https://github.com/apache/hadoop/blob/branch-3.4.3/hadoop-common-project/hadoop-common/src/main/arm-java/org/apache/hadoop/ipc/protobuf/ProtobufRpcEngineProtos.java
Thanks,
Cheng Pan
On Mar 8, 2026, at 22:58, Steve Loughran <st...@cloudera.com> wrote:
I spent a couple of hours this w/e getting claude to answer the question
for me "do the x86 and aarch JAR files differ?
The 3.4.3 release ended up using the arm build as its source of jar files
because using the cloud x86 vm I was using for release produced multiple
staging repos in apache nexus, something suggested to be VPN/IP address
related: nexus saw requests coming in from different source IP addresses
and so assigned the artifacts to different repos.
I was curious about whether the binaries were different, and also whether
it'd be possible to detect and defend against malicious release managers.
That is: if I'd added a back door into the code, would it have been
detected.
Here then is my auditor
https://github.com/steveloughran/auditor
And here are the results.
https://gist.github.com/steveloughran/d3c9ad6a718bfec68085b08584ae414e
The main issue to flag is that in hadoop common, the protobuf classes are
somehow different. leveldbjni is different too, which is interesting but
not too concerning, though "auditor" does flag the different code is
looking at the system environment so extra suspicious.
I doubt thats new; just something we've never noticed before. Whatever the
native protoc compilers are doing, they seem to be generating different
classes, even with the same shaded protobuf being used.
Thanks,
Cheng Pan