Protocol Buffer mismatch version between Impala and Apache Hadoop

[Alessio Annuzzi <al...@gmail.com>]

Hello,

I'm trying to understand better the connection between impala and the
filesystem hdfs.

I compiled impala 3.4.0 using the apache hadoop libraries ( not cdh
components).
When I add the Apache Hadoop libhdfs.so to the LD_LIBRARY_PATH, I get the
following FATAL error (crash of my impala service )

*[libprotobuf FATAL google/protobuf/stubs/common.cc:79] This program was
compiled against version 2.5.0 of the Protocol Buffer runtime library,
which is not compatible with the installed version (3.5.1).  Contact the
program author for an update.  If you compiled the program yourself, make
sure that your headers are from the same version of Protocol Buffers as
your link-time library.  (Version verification failed in
"/root/workspace/build_fileclient/fs/proto/security.pb.cc
<http://security.pb.cc>".)*

I tried to change the version of protocol buffer installed on my host and
add it to LD_RUN_PATH/LD_PRELOAD/LD_LIBRARY_PATH, but nothing changes

I compiled again impala using the cdh components ( default toolchain ) and
I don't face up this issue.

I noticed that:

   - Impala is built with 3.5.1 version of protobuf (
   https://github.com/apache/impala/blob/3.4.0/bin/impala-config.sh#L134 )
   - Apache Hadoop cannot be compiled with a protobuf version higher than
   2.5.0 ( https://issues.apache.org/jira/browse/HADOOP-13363 )


How can I manage this issue using Apache Hadoop libraries ? Is impala not
compatible with the Apache Hadoop ?
Are Java Native Libraries of CDH Hadoop built using the 3.5.1 version of
protobuf ?

In case I would like to add a connection to another storage like kudu,
should I provide the libkuduclient.so compiled with the same version of
protobuf used in impala ?

Thanks a lot
Alessio

Re: Protocol Buffer mismatch version between Impala and Apache Hadoop

[Tim Armstrong <ta...@cloudera.com>]

> How can I manage this issue using Apache Hadoop libraries ? Is impala not
compatible with the Apache Hadoop ?
We've focused so far on building against CDH and CDP. Building against a
set of pure Apache dependencies is a valid goal, but nobody has stepped up
to maintain that (there are some challenges in harmonizing versions of
everything given the large number of projects we depend on).

We do build directly against Apache Kudu (via native-toolchain so the
library and compiler versions match Impala). You need to build against a
Kudu built that way, then to make sure it can find libkuduclient.so at
runtime (i.e. `ldd path/to/impala` should report the correct
libkuduclient.so after you've set LD_LIBRARY_PATH or whatever you do).

I don't know what the solution to the build issue is but I can probably
point out a few things that are relevant.

   - libhdfs doesn't use protobuf afaict, so it's not clear why this is
   happening -
   https://github.com/apache/hadoop/tree/a55d6bba71c81c1c4e9d8cd11f55c78f10a548b0/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs
   - We always statically link libhdfs - there's no runtime dependency on
   libhdfs.so in a typical Impala build. I.e. if you point the Impala build at
   a hadoop build with a libhdfs.a and it finds it successfully, that code
   will be linked into the Impala binary and you don't need to do anything
   with libhdfs.so. The only thing the Impala build really needs here are
   hdfs.h and libhdfs.a

One thing that wasn't clear to me was whether you got the services running
in the Impala development environment after building.
start-impala-cluster.py can start the services with all of the library
dependencies set up correctly (bin/run-jvm-binary.sh is the thing that
actually sets up LD_LIBRARY_PATH and CLASSPATH). If you can't run the
binaries in-place in the development environment, that suggests to me that
something went wrong with the build. If you *can* run them, then that
suggests that they built ok, but that something is wrong with how you're
running them.

On Fri, May 22, 2020 at 11:41 AM Alessio Annuzzi <al...@gmail.com>
wrote:

> Hello,
>
> I'm trying to understand better the connection between impala and the
> filesystem hdfs.
>
> I compiled impala 3.4.0 using the apache hadoop libraries ( not cdh
> components).
> When I add the Apache Hadoop libhdfs.so to the LD_LIBRARY_PATH, I get the
> following FATAL error (crash of my impala service )
>
> *[libprotobuf FATAL google/protobuf/stubs/common.cc:79] This program was
> compiled against version 2.5.0 of the Protocol Buffer runtime library,
> which is not compatible with the installed version (3.5.1).  Contact the
> program author for an update.  If you compiled the program yourself, make
> sure that your headers are from the same version of Protocol Buffers as
> your link-time library.  (Version verification failed in
> "/root/workspace/build_fileclient/fs/proto/security.pb.cc
> <http://security.pb.cc>".)*
>
> I tried to change the version of protocol buffer installed on my host and
> add it to LD_RUN_PATH/LD_PRELOAD/LD_LIBRARY_PATH, but nothing changes
>
> I compiled again impala using the cdh components ( default toolchain )
> and I don't face up this issue.
>
> I noticed that:
>
>    - Impala is built with 3.5.1 version of protobuf (
>    https://github.com/apache/impala/blob/3.4.0/bin/impala-config.sh#L134 )
>    - Apache Hadoop cannot be compiled with a protobuf version higher than
>    2.5.0 ( https://issues.apache.org/jira/browse/HADOOP-13363 )
>
>
> How can I manage this issue using Apache Hadoop libraries ? Is impala not
> compatible with the Apache Hadoop ?
> Are Java Native Libraries of CDH Hadoop built using the 3.5.1 version of
> protobuf ?
>
> In case I would like to add a connection to another storage like kudu,
> should I provide the libkuduclient.so compiled with the same version of
> protobuf used in impala ?
>
> Thanks a lot
> Alessio
>