You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@impala.apache.org by Anup Tiwari <an...@gmail.com> on 2018/01/23 12:52:16 UTC
Install Impala on Apache Hadoop and Hive
Hi All
I want to install impala on my production environment(distributed) :-
*1. Os Details :* CentOS Linux release 7.2.1511 (Core)
*2.* Apache Hadoop 2.8.0
*3.* Apache Hive 2.1.1
I didn't found any link on mail, please suggest if it is possible or not
and if possible then please share some links.
Thanks in advance!!
Regards,
Anup Tiwari
Re: Install Impala on Apache Hadoop and Hive
Posted by Zoltan Borok-Nagy <bo...@cloudera.com>.
Hi Anup,
apache-impala-2.11.0.tar.gz is a source release of Impala, were you able to
build it?
https://cwiki.apache.org/confluence/display/IMPALA/Building+Impala shows
the build instructions.
Seems you need to install a package that contains the lsb_release command.
This link might help to do that:
https://www.unixmen.com/linux-troubleshooting-fix-lsb_release-command-found-centos/
Also, don't just execute impala-config.sh, you need to source it, so if you
are using bash, issue the following:
source bin/impala-config.sh
BR,
Zoltan
On Mon, Feb 5, 2018 at 9:09 AM, Anup Tiwari <an...@gmail.com> wrote:
> Hi All,
>
> I am new to impala so i am getting errors while i am setting impala.
>
> I have downloaded latest impala release from "http://www.apache.org/dyn/
> closer.cgi?action=download&filename=impala/2.11.0/apache-
> impala-2.11.0.tar.gz".
> I have extracted this in a folder and followed description mentioned in
> below link :-
>
> https://impala.apache.org/docs/build/html/topics/impala_
> processes.html#processes
>
> As mentioned in above link "*To start the Impala state store and Impala
> from the command line or a script, you can either use the service command
> or you can start the daemons directly through the impalad, statestored, and
> catalogd executables.*" So i searched for executable and found
> "start-impalad.sh"(which i think is correct one, let me know if i have
> missed this) but when i am executing this , i am getting below *errors* :-
>
> bin/start-impalad.sh: line 89: /admin: No such file or directory
> bin/start-impalad.sh: line 106: /opt/apache-s/apache-impala/
> apache-impala-2.11.0/be/build/latest/service/impalad: No such file or
> directory
>
> Also when i executed ./impala-config.sh then i got below *error* :-
>
> Unable to find the lsb_release command. Please ensure it is available in
> your PATH.
> ./impala-config.sh: line 265: return: can only `return' from a function or
> sourced script
> lsb_release command failed, output was: ./impala-config.sh: line 267:
> lsb_release: command not found
> ./impala-config.sh: line 270: return: can only `return' from a function or
> sourced script
>
>
> *Questions :-*
> 1. When we setup Hive/Hadoop/Drill etc then we have certain steps for
> setting HIVE_HOME/HADOOP_HOME/DRILL_HOME and also we specify certain
> setting in conf folder. Do we have similar documentation which can specify
> step by step setup.
>
>
> Regards,
> Anup Tiwari
>
> On Tue, Jan 23, 2018 at 8:56 PM, Jim Apple <jb...@cloudera.com> wrote:
>
>> Ubuntu 14.04 and 16.04 are the OSes most Impala developers use, but
>> many Impala users use CentOS 7.
>>
>> Here is part of the docs on tuning:
>> https://impala.apache.org/docs/build/html/topics/impala_performance.html
>>
>> Here is some tuning info that was produced by Cloudera:
>> http://blog.cloudera.com/wp-content/uploads/2017/03/Impala-
>> Cookbook-01-20173.pdf
>>
>> On Tue, Jan 23, 2018 at 6:53 AM, Anup Tiwari <an...@gmail.com>
>> wrote:
>> > Hi Vincent,
>> >
>> > I found this link which says Supported Operating Systems are Ubuntu
>> 14.04,
>> > 16.04 so does this mean it won't work on centos 7.
>> >
>> > Also can you send some link which can help us in tuning as well.
>> >
>> > Regards,
>> > Anup Tiwari
>> >
>> > On Tue, Jan 23, 2018 at 7:26 PM, Vincent Tran <vt...@cloudera.com>
>> wrote:
>> >>
>> >> Hi Anup,
>> >>
>> >> Is this the link you are looking for?
>> >>
>> >> https://impala.apache.org/docs/build/html/topics/impala_install.html
>> >>
>> >> On Tue, Jan 23, 2018 at 7:52 AM, Anup Tiwari <an...@gmail.com>
>> >> wrote:
>> >>>
>> >>> Hi All
>> >>>
>> >>> I want to install impala on my production environment(distributed) :-
>> >>>
>> >>> 1. Os Details : CentOS Linux release 7.2.1511 (Core)
>> >>> 2. Apache Hadoop 2.8.0
>> >>> 3. Apache Hive 2.1.1
>> >>>
>> >>> I didn't found any link on mail, please suggest if it is possible or
>> not
>> >>> and if possible then please share some links.
>> >>>
>> >>> Thanks in advance!!
>> >>>
>> >>> Regards,
>> >>> Anup Tiwari
>> >>
>> >>
>> >>
>> >>
>> >> --
>> >> Vincent T. Tran
>> >> Customer Operations Engineer
>> >> Cloudera, Inc.
>> >
>> >
>>
>
>
Re: Install Impala on Apache Hadoop and Hive
Posted by Tim Armstrong <ta...@cloudera.com>.
I agree with Jim. We're happy to try to help, but this is a community
mailing list and relies on people taking time away from other personal or
work activities to answer questions.
Please help us to help you. The easier you make it for us to understand the
exact steps you took and the full output you got, the more likely we are to
be able to help you out. There's a limit to the amount of time I'm going to
spend trying to reconstruct the steps you took to get into the state where
you got that error message.
- Tim
On Thu, Feb 8, 2018 at 9:02 AM, Jim Apple <jb...@cloudera.com> wrote:
> There were 18 hours between your last email and the ping you sent 4
> hours ago. Sometimes it can take a little while for someone from user@
> to respond, especially on trickier issues. I hope you will be able to
> find some patience with the community when this happens in the future.
>
> This looked to me like a problem with the linker flags:
>
> https://stackoverflow.com/questions/1662909/undefined-
> reference-to-pthread-create-in-linux
>
> That said, I looked a bit through the code and wasn't able to track
> down the exact origin of this compilation and check to see why it is
> using -lpthreads vs. -lpthread vs -pthread. What did you find when you
> looked?
>
> On Thu, Feb 8, 2018 at 4:35 AM, Anup Tiwari <an...@gmail.com>
> wrote:
> > Hi,
> >
> > Any update on this?
> >
> > Regards,
> > Anup Tiwari
> >
> > On Wed, Feb 7, 2018 at 11:43 PM, Anup Tiwari <an...@gmail.com>
> wrote:
> >>
> >> Hi,
> >>
> >> I have executed below commands
> >>
> >> Build Command :
> >> ${IMPALA_HOME}/buildall.sh -noclean -skiptests -build_shared_libs
> -format
> >>
> >> Please refer trail mail for details or steps.
> >>
> >> On 07-Feb-2018 10:51 PM, "Tim Armstrong" <ta...@cloudera.com>
> wrote:
> >>
> >> What command did you run to kick off the build? It looks like you're
> >> missing the HDFS dependency from the expected location, but that should
> be
> >> automatically downloaded if you run buildall.sh with the default
> settings.
> >>
> >> On Wed, Feb 7, 2018 at 6:30 AM, Anup Tiwari <an...@gmail.com>
> >> wrote:
> >>>
> >>> Hi Vincent,
> >>>
> >>> I have installed all packages/modules mentioned in
> >>> https://cwiki.apache.org/confluence/display/IMPALA/
> Impala+Build+Prerequisites
> >>> and now getting below error while building and the one which you have
> >>> mention is for Ubuntu 14.04 as per docs and i am using centos7 as
> mention in
> >>> trail mail :-
> >>>
> >>> -- Added shared library dependency zlib:
> >>> /opt/apache-s/apache-impala/apache-impala-2.11.0/
> toolchain/zlib-1.2.8/lib/libz.so
> >>> CMake Error at cmake_modules/FindHDFS.cmake:78 (message):
> >>> HDFS includes and libraries NOT found. HDFS_INCLUDE_DIR-NOTFOUND, )
> >>> Call Stack (most recent call first):
> >>> CMakeLists.txt:183 (find_package)
> >>> -- Configuring incomplete, errors occurred!
> >>> See also
> >>> "/opt/apache-s/apache-impala/apache-impala-2.11.0/
> CMakeFiles/CMakeOutput.log".
> >>> See also
> >>> "/opt/apache-s/apache-impala/apache-impala-2.11.0/
> CMakeFiles/CMakeError.log".
> >>> Error in
> >>> /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/make_impala.sh
> at line
> >>> 161: cmake . ${CMAKE_ARGS[@]}
> >>>
> >>>
> >>> OUTPUT of CMakeError.log :-
> >>>
> >>> Determining if the pthread_create exist failed with the following
> output:
> >>> Change Dir:
> >>> /opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/CMakeTmp
> >>>
> >>> Run Build Command:"/usr/bin/gmake" "cmTC_18658/fast"
> >>> /usr/bin/gmake -f CMakeFiles/cmTC_18658.dir/build.make
> >>> CMakeFiles/cmTC_18658.dir/build
> >>> gmake[1]: Entering directory
> >>> `/opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/CMakeTmp'
> >>> Building C object CMakeFiles/cmTC_18658.dir/CheckSymbolExists.c.o
> >>>
> >>> /opt/apache-s/apache-impala/apache-impala-2.11.0/
> toolchain/gcc-4.9.2/bin/gcc
> >>> -o CMakeFiles/cmTC_18658.dir/CheckSymbolExists.c.o -c
> >>> /opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/CMakeTmp/
> CheckSymbolExists.c
> >>> Linking C executable cmTC_18658
> >>>
> >>> /opt/apache-s/apache-impala/native-toolchain/build/cmake-
> 3.8.2-p1/bin/cmake
> >>> -E cmake_link_script CMakeFiles/cmTC_18658.dir/link.txt --verbose=1
> >>>
> >>> /opt/apache-s/apache-impala/apache-impala-2.11.0/
> toolchain/gcc-4.9.2/bin/gcc
> >>> -rdynamic CMakeFiles/cmTC_18658.dir/CheckSymbolExists.c.o -o
> cmTC_18658
> >>> CMakeFiles/cmTC_18658.dir/CheckSymbolExists.c.o: In function `main':
> >>> CheckSymbolExists.c:(.text+0x16): undefined reference to
> `pthread_create'
> >>> collect2: error: ld returned 1 exit status
> >>> gmake[1]: *** [cmTC_18658] Error 1
> >>> gmake[1]: Leaving directory
> >>> `/opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/CMakeTmp'
> >>> gmake: *** [cmTC_18658/fast] Error 2
> >>>
> >>> File
> >>> /opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/CMakeTmp/
> CheckSymbolExists.c:
> >>> /* */
> >>> #include <pthread.h>
> >>>
> >>> int main(int argc, char** argv)
> >>> {
> >>> (void)argv;
> >>> #ifndef pthread_create
> >>> return ((int*)(&pthread_create))[argc];
> >>> #else
> >>> (void)argc;
> >>> return 0;
> >>> #endif
> >>> }
> >>>
> >>> Determining if the function pthread_create exists in the pthreads
> failed
> >>> with the following output:
> >>> Change Dir:
> >>> /opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/CMakeTmp
> >>>
> >>> Run Build Command:"/usr/bin/gmake" "cmTC_81779/fast"
> >>> /usr/bin/gmake -f CMakeFiles/cmTC_81779.dir/build.make
> >>> CMakeFiles/cmTC_81779.dir/build
> >>> gmake[1]: Entering directory
> >>> `/opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/CMakeTmp'
> >>> Building C object CMakeFiles/cmTC_81779.dir/CheckFunctionExists.c.o
> >>>
> >>> /opt/apache-s/apache-impala/apache-impala-2.11.0/
> toolchain/gcc-4.9.2/bin/gcc
> >>> -DCHECK_FUNCTION_EXISTS=pthread_create -o
> >>> CMakeFiles/cmTC_81779.dir/CheckFunctionExists.c.o -c
> >>> /opt/apache-s/apache-impala/native-toolchain/build/cmake-
> 3.8.2-p1/share/cmake-3.8/Modules/CheckFunctionExists.c
> >>> Linking C executable cmTC_81779
> >>>
> >>> /opt/apache-s/apache-impala/native-toolchain/build/cmake-
> 3.8.2-p1/bin/cmake
> >>> -E cmake_link_script CMakeFiles/cmTC_81779.dir/link.txt --verbose=1
> >>>
> >>> /opt/apache-s/apache-impala/apache-impala-2.11.0/
> toolchain/gcc-4.9.2/bin/gcc
> >>> -DCHECK_FUNCTION_EXISTS=pthread_create -rdynamic
> >>> CMakeFiles/cmTC_81779.dir/CheckFunctionExists.c.o -o cmTC_81779
> -lpthreads
> >>> /usr/bin/ld: cannot find -lpthreads
> >>> collect2: error: ld returned 1 exit status
> >>> gmake[1]: *** [cmTC_81779] Error 1
> >>> gmake[1]: Leaving directory
> >>> `/opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/CMakeTmp'
> >>> gmake: *** [cmTC_81779/fast] Error 2
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> Regards,
> >>> Anup Tiwari
> >>>
> >>> On Tue, Feb 6, 2018 at 6:07 PM, Vincent Tran <vt...@cloudera.com>
> wrote:
> >>>>
> >>>> Hey Anup,
> >>>>
> >>>> Lools like ypu may be missing some prereq packages. This particular
> one
> >>>> is libsasl2-dev
> >>>>
> >>>>
> >>>>
> >>>> On Feb 6, 2018 5:12 AM, "Anup Tiwari" <an...@gmail.com> wrote:
> >>>>>
> >>>>> I found this link
> >>>>> http://mail-archives.apache.org/mod_mbox/impala-dev/
> 201708.mbox/%3CCADZba-Td-wXRqLRxctvFM2mY44cYJhkX-JSs4=
> uPQCnFDhoDyA@mail.gmail.com%3E
> >>>>> which says
> >>>>>
> >>>>> Do the following:
> >>>>>
> >>>>> rm -rf ${IMPALA_TOOLCHAIN}/gflags-${IMPALA_GFLAGS_VERSION}
> >>>>> . bin/impala-config.sh
> >>>>> bin/bootstrap_toolchain.py
> >>>>>
> >>>>> But post executing bootstrap command ; i am getting below error :-
> >>>>>
> >>>>> Traceback (most recent call last):
> >>>>> File
> >>>>> "/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../
> infra/python/bootstrap_virtualenv.py",
> >>>>> line 367, in <module>
> >>>>> if install_compiled_deps_if_possible():
> >>>>> File
> >>>>> "/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../
> infra/python/bootstrap_virtualenv.py",
> >>>>> line 216, in install_compiled_deps_if_possible
> >>>>> exec_pip_install(["-r", COMPILED_REQS_PATH], cc=cc, env=env)
> >>>>> File
> >>>>> "/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../
> infra/python/bootstrap_virtualenv.py",
> >>>>> line 145, in exec_pip_install
> >>>>> "file://%s" % urllib.pathname2url(os.path.abspath(DEPS_DIR))] +
> >>>>> args, env=env)
> >>>>> File
> >>>>> "/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../
> infra/python/bootstrap_virtualenv.py",
> >>>>> line 96, in exec_cmd
> >>>>> % (args, output))
> >>>>> Exception: Command returned non-zero status
> >>>>> Command:
> >>>>> ['/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../
> infra/python/env/bin/python',
> >>>>> '/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../
> infra/python/env/bin/pip',
> >>>>> 'install', '--no-binary', '--no-index', '--no-cache-dir',
> '--find-links',
> >>>>> 'file:///opt/apache-s/apache-impala/apache-impala-2.11.0/
> infra/python/deps',
> >>>>> '-r',
> >>>>> '/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../
> infra/python/deps/compiled-requirements.txt']
> >>>>> Output: You are using pip version 7.1.0, however version 9.0.1 is
> >>>>> available.
> >>>>> You should consider upgrading via the 'pip install --upgrade pip'
> >>>>> command.
> >>>>> Requirement already satisfied (use --upgrade to upgrade):
> >>>>> argparse==1.4.0 in ./infra/python/env/lib/python2.7/site-packages
> (from -r
> >>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../
> infra/python/deps/compiled-requirements.txt
> >>>>> (line 1))
> >>>>> Requirement already satisfied (use --upgrade to upgrade):
> >>>>> Fabric==1.10.2 in ./infra/python/env/lib/python2.7/site-packages
> (from -r
> >>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../
> infra/python/deps/compiled-requirements.txt
> >>>>> (line 2))
> >>>>> Requirement already satisfied (use --upgrade to upgrade):
> >>>>> paramiko==1.15.2 in ./infra/python/env/lib/python2.7/site-packages
> (from -r
> >>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../
> infra/python/deps/compiled-requirements.txt
> >>>>> (line 3))
> >>>>> Requirement already satisfied (use --upgrade to upgrade): ecdsa==0.13
> >>>>> in ./infra/python/env/lib/python2.7/site-packages (from -r
> >>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../
> infra/python/deps/compiled-requirements.txt
> >>>>> (line 4))
> >>>>> Requirement already satisfied (use --upgrade to upgrade):
> >>>>> pycrypto==2.6.1 in ./infra/python/env/lib/python2.7/site-packages
> (from -r
> >>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../
> infra/python/deps/compiled-requirements.txt
> >>>>> (line 5))
> >>>>> Requirement already satisfied (use --upgrade to upgrade):
> >>>>> impyla==0.14.0 in ./infra/python/env/lib/python2.7/site-packages
> (from -r
> >>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../
> infra/python/deps/compiled-requirements.txt
> >>>>> (line 6))
> >>>>> Requirement already satisfied (use --upgrade to upgrade):
> >>>>> bitarray==0.8.1 in ./infra/python/env/lib/python2.7/site-packages
> (from -r
> >>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../
> infra/python/deps/compiled-requirements.txt
> >>>>> (line 7))
> >>>>> Collecting sasl==0.1.3 (from -r
> >>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../
> infra/python/deps/compiled-requirements.txt
> >>>>> (line 8))
> >>>>> Requirement already satisfied (use --upgrade to upgrade): six==1.9.0
> in
> >>>>> ./infra/python/env/lib/python2.7/site-packages (from -r
> >>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../
> infra/python/deps/compiled-requirements.txt
> >>>>> (line 9))
> >>>>> Collecting thrift==0.9.0 (from -r
> >>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../
> infra/python/deps/compiled-requirements.txt
> >>>>> (line 10))
> >>>>> Collecting thrift-sasl==0.1.0 (from -r
> >>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../
> infra/python/deps/compiled-requirements.txt
> >>>>> (line 11))
> >>>>> Collecting psutil==0.7.1 (from -r
> >>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../
> infra/python/deps/compiled-requirements.txt
> >>>>> (line 12))
> >>>>> Collecting Cython==0.23.4 (from -r
> >>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../
> infra/python/deps/compiled-requirements.txt
> >>>>> (line 13))
> >>>>> Collecting numpy==1.10.4 (from -r
> >>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../
> infra/python/deps/compiled-requirements.txt
> >>>>> (line 14))
> >>>>> Installing collected packages: sasl, thrift, thrift-sasl, psutil,
> >>>>> Cython, numpy
> >>>>> Running setup.py install for sasl
> >>>>> Complete output from command
> >>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../
> infra/python/env/bin/python
> >>>>> -c "import setuptools,
> >>>>> tokenize;__file__='/tmp/pip-build-MTPxSs/sasl/setup.py';
> exec(compile(getattr(tokenize,
> >>>>> 'open', open)(__file__).read().replace('\r\n', '\n'), __file__,
> 'exec'))"
> >>>>> install --record /tmp/pip-EHfJDn-record/install-record.txt
> >>>>> --single-version-externally-managed --compile --install-headers
> >>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../
> infra/python/env/include/site/python2.7/sasl:
> >>>>> running install
> >>>>> running build
> >>>>> running build_py
> >>>>> creating build
> >>>>> creating build/lib.linux-x86_64-2.7
> >>>>> creating build/lib.linux-x86_64-2.7/sasl
> >>>>> copying sasl/__init__.py -> build/lib.linux-x86_64-2.7/sasl
> >>>>> copying sasl/saslwrapper.py -> build/lib.linux-x86_64-2.7/sasl
> >>>>> running egg_info
> >>>>> writing sasl.egg-info/PKG-INFO
> >>>>> writing top-level names to sasl.egg-info/top_level.txt
> >>>>> writing dependency_links to sasl.egg-info/dependency_links.txt
> >>>>> warning: manifest_maker: standard file '-c' not found
> >>>>>
> >>>>> reading manifest file 'sasl.egg-info/SOURCES.txt'
> >>>>> reading manifest template 'MANIFEST.in'
> >>>>> writing manifest file 'sasl.egg-info/SOURCES.txt'
> >>>>> running build_ext
> >>>>> building '_saslwrapper' extension
> >>>>> creating build/temp.linux-x86_64-2.7
> >>>>> creating build/temp.linux-x86_64-2.7/sasl
> >>>>>
> >>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/
> toolchain/gcc-4.9.2/bin/gcc
> >>>>> -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2
> -fexceptions
> >>>>> -fstack-protector-strong --param=ssp-buffer-size=4
> -grecord-gcc-switches
> >>>>> -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g
> -pipe -Wall
> >>>>> -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong
> >>>>> --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic
> >>>>> -D_GNU_SOURCE -fPIC -fwrapv -fPIC -Isasl -I/usr/include/python2.7 -c
> >>>>> sasl/saslwrapper.cpp -o build/temp.linux-x86_64-2.7/
> sasl/saslwrapper.o
> >>>>> sasl/saslwrapper.cpp:21:23: fatal error: sasl/sasl.h: No such
> file
> >>>>> or directory
> >>>>> #include <sasl/sasl.h>
> >>>>> ^
> >>>>> compilation terminated.
> >>>>> error: command
> >>>>> '/opt/apache-s/apache-impala/apache-impala-2.11.0/
> toolchain/gcc-4.9.2/bin/gcc'
> >>>>> failed with exit status 1
> >>>>>
> >>>>> ----------------------------------------
> >>>>> Command
> >>>>> "/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../
> infra/python/env/bin/python
> >>>>> -c "import setuptools,
> >>>>> tokenize;__file__='/tmp/pip-build-MTPxSs/sasl/setup.py';
> exec(compile(getattr(tokenize,
> >>>>> 'open', open)(__file__).read().replace('\r\n', '\n'), __file__,
> 'exec'))"
> >>>>> install --record /tmp/pip-EHfJDn-record/install-record.txt
> >>>>> --single-version-externally-managed --compile --install-headers
> >>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../
> infra/python/env/include/site/python2.7/sasl"
> >>>>> failed with error code 1 in /tmp/pip-build-MTPxSs/sasl
> >>>>>
> >>>>> Error in
> >>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/impala-python
> at line
> >>>>> 28:
> >>>>>
> >>>>>
> >>>>>
> >>>>> Regards,
> >>>>> Anup Tiwari
> >>>>>
> >>>>> On Tue, Feb 6, 2018 at 2:52 PM, Anup Tiwari <an...@gmail.com>
> >>>>> wrote:
> >>>>>>
> >>>>>> Hi Zoltan,
> >>>>>>
> >>>>>> I followed the steps mention in
> >>>>>> https://cwiki.apache.org/confluence/display/IMPALA/Building+Impala
> and when
> >>>>>> i executed below command , i got error after some time :-
> >>>>>>
> >>>>>> Build Command :
> >>>>>> ${IMPALA_HOME}/buildall.sh -noclean -skiptests -build_shared_libs
> >>>>>> -format
> >>>>>>
> >>>>>> Error :
> >>>>>> -- ----------> Adding thirdparty library cyrus_sasl. <----------
> >>>>>> CMake Error at CMakeLists.txt:136 (message):
> >>>>>> Library 'cyrus_sasl' has neither shared nor static library files
> >>>>>> Call Stack (most recent call first):
> >>>>>> CMakeLists.txt:227 (IMPALA_ADD_THIRDPARTY_LIB)
> >>>>>>
> >>>>>> -- Configuring incomplete, errors occurred!
> >>>>>> See also
> >>>>>> "/opt/apache-s/apache-impala/apache-impala-2.11.0/
> CMakeFiles/CMakeOutput.log".
> >>>>>> See also
> >>>>>> "/opt/apache-s/apache-impala/apache-impala-2.11.0/
> CMakeFiles/CMakeError.log".
> >>>>>> Error in
> >>>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/make_impala.sh
> at line
> >>>>>> 161: cmake . ${CMAKE_ARGS[@]}
> >>>>>>
> >>>>>>
> >>>>>> Logs in "CMakeOutput.log" :-
> >>>>>>
> >>>>>> Determining if the function pthread_create exists in the pthread
> >>>>>> passed with the following output:
> >>>>>> Change Dir:
> >>>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/
> CMakeFiles/CMakeTmp
> >>>>>>
> >>>>>> Run Build Command:"/usr/bin/gmake" "cmTC_ee4f0/fast"
> >>>>>> /usr/bin/gmake -f CMakeFiles/cmTC_ee4f0.dir/build.make
> >>>>>> CMakeFiles/cmTC_ee4f0.dir/build
> >>>>>> gmake[1]: Entering directory
> >>>>>> `/opt/apache-s/apache-impala/apache-impala-2.11.0/
> CMakeFiles/CMakeTmp'
> >>>>>> Building C object CMakeFiles/cmTC_ee4f0.dir/CheckFunctionExists.c.o
> >>>>>>
> >>>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/
> toolchain/gcc-4.9.2/bin/gcc
> >>>>>> -DCHECK_FUNCTION_EXISTS=pthread_create -o
> >>>>>> CMakeFiles/cmTC_ee4f0.dir/CheckFunctionExists.c.o -c
> >>>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/
> toolchain/cmake-3.8.2-p1/share/cmake-3.8/Modules/CheckFunctionExists.c
> >>>>>> Linking C executable cmTC_ee4f0
> >>>>>>
> >>>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/
> toolchain/cmake-3.8.2-p1/bin/cmake
> >>>>>> -E cmake_link_script CMakeFiles/cmTC_ee4f0.dir/link.txt --verbose=1
> >>>>>>
> >>>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/
> toolchain/gcc-4.9.2/bin/gcc
> >>>>>> -DCHECK_FUNCTION_EXISTS=pthread_create -rdynamic
> >>>>>> CMakeFiles/cmTC_ee4f0.dir/CheckFunctionExists.c.o -o cmTC_ee4f0
> -lpthread
> >>>>>> gmake[1]: Leaving directory
> >>>>>> `/opt/apache-s/apache-impala/apache-impala-2.11.0/
> CMakeFiles/CMakeTmp'
> >>>>>>
> >>>>>> Logs in "CMakeError.log" :-
> >>>>>>
> >>>>>> Determining if the function pthread_create exists in the pthreads
> >>>>>> failed with the following output:
> >>>>>> Change Dir:
> >>>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/
> CMakeFiles/CMakeTmp
> >>>>>>
> >>>>>> Run Build Command:"/usr/bin/gmake" "cmTC_18bf3/fast"
> >>>>>> /usr/bin/gmake -f CMakeFiles/cmTC_18bf3.dir/build.make
> >>>>>> CMakeFiles/cmTC_18bf3.dir/build
> >>>>>> gmake[1]: Entering directory
> >>>>>> `/opt/apache-s/apache-impala/apache-impala-2.11.0/
> CMakeFiles/CMakeTmp'
> >>>>>> Building C object CMakeFiles/cmTC_18bf3.dir/CheckFunctionExists.c.o
> >>>>>>
> >>>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/
> toolchain/gcc-4.9.2/bin/gcc
> >>>>>> -DCHECK_FUNCTION_EXISTS=pthread_create -o
> >>>>>> CMakeFiles/cmTC_18bf3.dir/CheckFunctionExists.c.o -c
> >>>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/
> toolchain/cmake-3.8.2-p1/share/cmake-3.8/Modules/CheckFunctionExists.c
> >>>>>> Linking C executable cmTC_18bf3
> >>>>>>
> >>>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/
> toolchain/cmake-3.8.2-p1/bin/cmake
> >>>>>> -E cmake_link_script CMakeFiles/cmTC_18bf3.dir/link.txt --verbose=1
> >>>>>>
> >>>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/
> toolchain/gcc-4.9.2/bin/gcc
> >>>>>> -DCHECK_FUNCTION_EXISTS=pthread_create -rdynamic
> >>>>>> CMakeFiles/cmTC_18bf3.dir/CheckFunctionExists.c.o -o cmTC_18bf3
> -lpthreads
> >>>>>> /usr/bin/ld: cannot find -lpthreads
> >>>>>> collect2: error: ld returned 1 exit status
> >>>>>> gmake[1]: *** [cmTC_18bf3] Error 1
> >>>>>> gmake[1]: Leaving directory
> >>>>>> `/opt/apache-s/apache-impala/apache-impala-2.11.0/
> CMakeFiles/CMakeTmp'
> >>>>>> gmake: *** [cmTC_18bf3/fast] Error 2
> >>>>>>
> >>>>>> Also it seems that this is related to toolchain error so i followed
> >>>>>> this link :
> >>>>>> https://cwiki.apache.org/confluence/display/IMPALA/
> Building+native-toolchain+from+scratch+and+using+with+Impala
> >>>>>> where i first builded native-toolchain and then followed step
> mention in
> >>>>>> "Using custom toolchain with Impala" but while creating symbolic
> link i got
> >>>>>> error "ln: ‘./gcc-4.9.2’: cannot overwrite directory".
> >>>>>>
> >>>>>> @xinran, i guess post building impala, i will have to perform steps
> >>>>>> mentioned by you.. correct me if i am wrong?
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> Regards,
> >>>>>> Anup Tiwari
> >>>>>>
> >>>>>> On Mon, Feb 5, 2018 at 11:14 PM, Xinran Yu Tinney
> >>>>>> <yu...@gmail.com> wrote:
> >>>>>>>
> >>>>>>> Hi Anup,
> >>>>>>> Did you also source the config.sh? Run the following before
> >>>>>>> impala-shell.sh:
> >>>>>>>
> >>>>>>> source ~/Impala/bin/impala-config.sh
> >>>>>>>
> >>>>>>> ${IMPALA_HOME}/buildall.sh -notests -build_shared_libs
> >>>>>>> -start_minicluster
> >>>>>>>
> >>>>>>> ${IMPALA_HOME}/bin/start-impala-cluster.py
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> 2018-02-05 10:07 GMT-06:00 Michael Brown <mi...@cloudera.com>:
> >>>>>>>>
> >>>>>>>> Hello,
> >>>>>>>>
> >>>>>>>> Please see:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> https://cwiki.apache.org/confluence/display/IMPALA/
> Bootstrapping+an+Impala+Development+Environment+From+Scratch
> >>>>>>>> https://cwiki.apache.org/confluence/display/IMPALA/
> Building+Impala
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Mon, Feb 5, 2018 at 12:09 AM, Anup Tiwari
> >>>>>>>> <an...@gmail.com> wrote:
> >>>>>>>>>
> >>>>>>>>> Hi All,
> >>>>>>>>>
> >>>>>>>>> I am new to impala so i am getting errors while i am setting
> >>>>>>>>> impala.
> >>>>>>>>>
> >>>>>>>>> I have downloaded latest impala release from
> >>>>>>>>> "http://www.apache.org/dyn/closer.cgi?action=download&
> filename=impala/2.11.0/apache-impala-2.11.0.tar.gz".
> >>>>>>>>> I have extracted this in a folder and followed description
> >>>>>>>>> mentioned in below link :-
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> https://impala.apache.org/docs/build/html/topics/impala_
> processes.html#processes
> >>>>>>>>>
> >>>>>>>>> As mentioned in above link "To start the Impala state store and
> >>>>>>>>> Impala from the command line or a script, you can either use the
> service
> >>>>>>>>> command or you can start the daemons directly through the
> impalad,
> >>>>>>>>> statestored, and catalogd executables." So i searched for
> executable and
> >>>>>>>>> found "start-impalad.sh"(which i think is correct one, let me
> know if i have
> >>>>>>>>> missed this) but when i am executing this , i am getting below
> errors :-
> >>>>>>>>>
> >>>>>>>>> bin/start-impalad.sh: line 89: /admin: No such file or directory
> >>>>>>>>> bin/start-impalad.sh: line 106:
> >>>>>>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/be/build/
> latest/service/impalad:
> >>>>>>>>> No such file or directory
> >>>>>>>>>
> >>>>>>>>> Also when i executed ./impala-config.sh then i got below error :-
> >>>>>>>>>
> >>>>>>>>> Unable to find the lsb_release command. Please ensure it is
> >>>>>>>>> available in your PATH.
> >>>>>>>>> ./impala-config.sh: line 265: return: can only `return' from a
> >>>>>>>>> function or sourced script
> >>>>>>>>> lsb_release command failed, output was: ./impala-config.sh: line
> >>>>>>>>> 267: lsb_release: command not found
> >>>>>>>>> ./impala-config.sh: line 270: return: can only `return' from a
> >>>>>>>>> function or sourced script
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Questions :-
> >>>>>>>>> 1. When we setup Hive/Hadoop/Drill etc then we have certain steps
> >>>>>>>>> for setting HIVE_HOME/HADOOP_HOME/DRILL_HOME and also we
> specify certain
> >>>>>>>>> setting in conf folder. Do we have similar documentation which
> can specify
> >>>>>>>>> step by step setup.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Regards,
> >>>>>>>>> Anup Tiwari
> >>>>>>>>>
> >>>>>>>>> On Tue, Jan 23, 2018 at 8:56 PM, Jim Apple <jbapple@cloudera.com
> >
> >>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>> Ubuntu 14.04 and 16.04 are the OSes most Impala developers use,
> >>>>>>>>>> but
> >>>>>>>>>> many Impala users use CentOS 7.
> >>>>>>>>>>
> >>>>>>>>>> Here is part of the docs on tuning:
> >>>>>>>>>>
> >>>>>>>>>> https://impala.apache.org/docs/build/html/topics/impala_
> performance.html
> >>>>>>>>>>
> >>>>>>>>>> Here is some tuning info that was produced by Cloudera:
> >>>>>>>>>>
> >>>>>>>>>> http://blog.cloudera.com/wp-content/uploads/2017/03/
> Impala-Cookbook-01-20173.pdf
> >>>>>>>>>>
> >>>>>>>>>> On Tue, Jan 23, 2018 at 6:53 AM, Anup Tiwari
> >>>>>>>>>> <an...@gmail.com> wrote:
> >>>>>>>>>> > Hi Vincent,
> >>>>>>>>>> >
> >>>>>>>>>> > I found this link which says Supported Operating Systems are
> >>>>>>>>>> > Ubuntu 14.04,
> >>>>>>>>>> > 16.04 so does this mean it won't work on centos 7.
> >>>>>>>>>> >
> >>>>>>>>>> > Also can you send some link which can help us in tuning as
> well.
> >>>>>>>>>> >
> >>>>>>>>>> > Regards,
> >>>>>>>>>> > Anup Tiwari
> >>>>>>>>>> >
> >>>>>>>>>> > On Tue, Jan 23, 2018 at 7:26 PM, Vincent Tran
> >>>>>>>>>> > <vt...@cloudera.com> wrote:
> >>>>>>>>>> >>
> >>>>>>>>>> >> Hi Anup,
> >>>>>>>>>> >>
> >>>>>>>>>> >> Is this the link you are looking for?
> >>>>>>>>>> >>
> >>>>>>>>>> >>
> >>>>>>>>>> >> https://impala.apache.org/docs/build/html/topics/impala_
> install.html
> >>>>>>>>>> >>
> >>>>>>>>>> >> On Tue, Jan 23, 2018 at 7:52 AM, Anup Tiwari
> >>>>>>>>>> >> <an...@gmail.com>
> >>>>>>>>>> >> wrote:
> >>>>>>>>>> >>>
> >>>>>>>>>> >>> Hi All
> >>>>>>>>>> >>>
> >>>>>>>>>> >>> I want to install impala on my production
> >>>>>>>>>> >>> environment(distributed) :-
> >>>>>>>>>> >>>
> >>>>>>>>>> >>> 1. Os Details : CentOS Linux release 7.2.1511 (Core)
> >>>>>>>>>> >>> 2. Apache Hadoop 2.8.0
> >>>>>>>>>> >>> 3. Apache Hive 2.1.1
> >>>>>>>>>> >>>
> >>>>>>>>>> >>> I didn't found any link on mail, please suggest if it is
> >>>>>>>>>> >>> possible or not
> >>>>>>>>>> >>> and if possible then please share some links.
> >>>>>>>>>> >>>
> >>>>>>>>>> >>> Thanks in advance!!
> >>>>>>>>>> >>>
> >>>>>>>>>> >>> Regards,
> >>>>>>>>>> >>> Anup Tiwari
> >>>>>>>>>> >>
> >>>>>>>>>> >>
> >>>>>>>>>> >>
> >>>>>>>>>> >>
> >>>>>>>>>> >> --
> >>>>>>>>>> >> Vincent T. Tran
> >>>>>>>>>> >> Customer Operations Engineer
> >>>>>>>>>> >> Cloudera, Inc.
> >>>>>>>>>> >
> >>>>>>>>>> >
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>
> >>
> >>
> >
>
Re: Install Impala on Apache Hadoop and Hive
Posted by Jim Apple <jb...@cloudera.com>.
There were 18 hours between your last email and the ping you sent 4
hours ago. Sometimes it can take a little while for someone from user@
to respond, especially on trickier issues. I hope you will be able to
find some patience with the community when this happens in the future.
This looked to me like a problem with the linker flags:
https://stackoverflow.com/questions/1662909/undefined-reference-to-pthread-create-in-linux
That said, I looked a bit through the code and wasn't able to track
down the exact origin of this compilation and check to see why it is
using -lpthreads vs. -lpthread vs -pthread. What did you find when you
looked?
On Thu, Feb 8, 2018 at 4:35 AM, Anup Tiwari <an...@gmail.com> wrote:
> Hi,
>
> Any update on this?
>
> Regards,
> Anup Tiwari
>
> On Wed, Feb 7, 2018 at 11:43 PM, Anup Tiwari <an...@gmail.com> wrote:
>>
>> Hi,
>>
>> I have executed below commands
>>
>> Build Command :
>> ${IMPALA_HOME}/buildall.sh -noclean -skiptests -build_shared_libs -format
>>
>> Please refer trail mail for details or steps.
>>
>> On 07-Feb-2018 10:51 PM, "Tim Armstrong" <ta...@cloudera.com> wrote:
>>
>> What command did you run to kick off the build? It looks like you're
>> missing the HDFS dependency from the expected location, but that should be
>> automatically downloaded if you run buildall.sh with the default settings.
>>
>> On Wed, Feb 7, 2018 at 6:30 AM, Anup Tiwari <an...@gmail.com>
>> wrote:
>>>
>>> Hi Vincent,
>>>
>>> I have installed all packages/modules mentioned in
>>> https://cwiki.apache.org/confluence/display/IMPALA/Impala+Build+Prerequisites
>>> and now getting below error while building and the one which you have
>>> mention is for Ubuntu 14.04 as per docs and i am using centos7 as mention in
>>> trail mail :-
>>>
>>> -- Added shared library dependency zlib:
>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/zlib-1.2.8/lib/libz.so
>>> CMake Error at cmake_modules/FindHDFS.cmake:78 (message):
>>> HDFS includes and libraries NOT found. HDFS_INCLUDE_DIR-NOTFOUND, )
>>> Call Stack (most recent call first):
>>> CMakeLists.txt:183 (find_package)
>>> -- Configuring incomplete, errors occurred!
>>> See also
>>> "/opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/CMakeOutput.log".
>>> See also
>>> "/opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/CMakeError.log".
>>> Error in
>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/make_impala.sh at line
>>> 161: cmake . ${CMAKE_ARGS[@]}
>>>
>>>
>>> OUTPUT of CMakeError.log :-
>>>
>>> Determining if the pthread_create exist failed with the following output:
>>> Change Dir:
>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/CMakeTmp
>>>
>>> Run Build Command:"/usr/bin/gmake" "cmTC_18658/fast"
>>> /usr/bin/gmake -f CMakeFiles/cmTC_18658.dir/build.make
>>> CMakeFiles/cmTC_18658.dir/build
>>> gmake[1]: Entering directory
>>> `/opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/CMakeTmp'
>>> Building C object CMakeFiles/cmTC_18658.dir/CheckSymbolExists.c.o
>>>
>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
>>> -o CMakeFiles/cmTC_18658.dir/CheckSymbolExists.c.o -c
>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/CMakeTmp/CheckSymbolExists.c
>>> Linking C executable cmTC_18658
>>>
>>> /opt/apache-s/apache-impala/native-toolchain/build/cmake-3.8.2-p1/bin/cmake
>>> -E cmake_link_script CMakeFiles/cmTC_18658.dir/link.txt --verbose=1
>>>
>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
>>> -rdynamic CMakeFiles/cmTC_18658.dir/CheckSymbolExists.c.o -o cmTC_18658
>>> CMakeFiles/cmTC_18658.dir/CheckSymbolExists.c.o: In function `main':
>>> CheckSymbolExists.c:(.text+0x16): undefined reference to `pthread_create'
>>> collect2: error: ld returned 1 exit status
>>> gmake[1]: *** [cmTC_18658] Error 1
>>> gmake[1]: Leaving directory
>>> `/opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/CMakeTmp'
>>> gmake: *** [cmTC_18658/fast] Error 2
>>>
>>> File
>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/CMakeTmp/CheckSymbolExists.c:
>>> /* */
>>> #include <pthread.h>
>>>
>>> int main(int argc, char** argv)
>>> {
>>> (void)argv;
>>> #ifndef pthread_create
>>> return ((int*)(&pthread_create))[argc];
>>> #else
>>> (void)argc;
>>> return 0;
>>> #endif
>>> }
>>>
>>> Determining if the function pthread_create exists in the pthreads failed
>>> with the following output:
>>> Change Dir:
>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/CMakeTmp
>>>
>>> Run Build Command:"/usr/bin/gmake" "cmTC_81779/fast"
>>> /usr/bin/gmake -f CMakeFiles/cmTC_81779.dir/build.make
>>> CMakeFiles/cmTC_81779.dir/build
>>> gmake[1]: Entering directory
>>> `/opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/CMakeTmp'
>>> Building C object CMakeFiles/cmTC_81779.dir/CheckFunctionExists.c.o
>>>
>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
>>> -DCHECK_FUNCTION_EXISTS=pthread_create -o
>>> CMakeFiles/cmTC_81779.dir/CheckFunctionExists.c.o -c
>>> /opt/apache-s/apache-impala/native-toolchain/build/cmake-3.8.2-p1/share/cmake-3.8/Modules/CheckFunctionExists.c
>>> Linking C executable cmTC_81779
>>>
>>> /opt/apache-s/apache-impala/native-toolchain/build/cmake-3.8.2-p1/bin/cmake
>>> -E cmake_link_script CMakeFiles/cmTC_81779.dir/link.txt --verbose=1
>>>
>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
>>> -DCHECK_FUNCTION_EXISTS=pthread_create -rdynamic
>>> CMakeFiles/cmTC_81779.dir/CheckFunctionExists.c.o -o cmTC_81779 -lpthreads
>>> /usr/bin/ld: cannot find -lpthreads
>>> collect2: error: ld returned 1 exit status
>>> gmake[1]: *** [cmTC_81779] Error 1
>>> gmake[1]: Leaving directory
>>> `/opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/CMakeTmp'
>>> gmake: *** [cmTC_81779/fast] Error 2
>>>
>>>
>>>
>>>
>>>
>>> Regards,
>>> Anup Tiwari
>>>
>>> On Tue, Feb 6, 2018 at 6:07 PM, Vincent Tran <vt...@cloudera.com> wrote:
>>>>
>>>> Hey Anup,
>>>>
>>>> Lools like ypu may be missing some prereq packages. This particular one
>>>> is libsasl2-dev
>>>>
>>>>
>>>>
>>>> On Feb 6, 2018 5:12 AM, "Anup Tiwari" <an...@gmail.com> wrote:
>>>>>
>>>>> I found this link
>>>>> http://mail-archives.apache.org/mod_mbox/impala-dev/201708.mbox/%3CCADZba-Td-wXRqLRxctvFM2mY44cYJhkX-JSs4=uPQCnFDhoDyA@mail.gmail.com%3E
>>>>> which says
>>>>>
>>>>> Do the following:
>>>>>
>>>>> rm -rf ${IMPALA_TOOLCHAIN}/gflags-${IMPALA_GFLAGS_VERSION}
>>>>> . bin/impala-config.sh
>>>>> bin/bootstrap_toolchain.py
>>>>>
>>>>> But post executing bootstrap command ; i am getting below error :-
>>>>>
>>>>> Traceback (most recent call last):
>>>>> File
>>>>> "/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/bootstrap_virtualenv.py",
>>>>> line 367, in <module>
>>>>> if install_compiled_deps_if_possible():
>>>>> File
>>>>> "/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/bootstrap_virtualenv.py",
>>>>> line 216, in install_compiled_deps_if_possible
>>>>> exec_pip_install(["-r", COMPILED_REQS_PATH], cc=cc, env=env)
>>>>> File
>>>>> "/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/bootstrap_virtualenv.py",
>>>>> line 145, in exec_pip_install
>>>>> "file://%s" % urllib.pathname2url(os.path.abspath(DEPS_DIR))] +
>>>>> args, env=env)
>>>>> File
>>>>> "/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/bootstrap_virtualenv.py",
>>>>> line 96, in exec_cmd
>>>>> % (args, output))
>>>>> Exception: Command returned non-zero status
>>>>> Command:
>>>>> ['/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/env/bin/python',
>>>>> '/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/env/bin/pip',
>>>>> 'install', '--no-binary', '--no-index', '--no-cache-dir', '--find-links',
>>>>> 'file:///opt/apache-s/apache-impala/apache-impala-2.11.0/infra/python/deps',
>>>>> '-r',
>>>>> '/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt']
>>>>> Output: You are using pip version 7.1.0, however version 9.0.1 is
>>>>> available.
>>>>> You should consider upgrading via the 'pip install --upgrade pip'
>>>>> command.
>>>>> Requirement already satisfied (use --upgrade to upgrade):
>>>>> argparse==1.4.0 in ./infra/python/env/lib/python2.7/site-packages (from -r
>>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt
>>>>> (line 1))
>>>>> Requirement already satisfied (use --upgrade to upgrade):
>>>>> Fabric==1.10.2 in ./infra/python/env/lib/python2.7/site-packages (from -r
>>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt
>>>>> (line 2))
>>>>> Requirement already satisfied (use --upgrade to upgrade):
>>>>> paramiko==1.15.2 in ./infra/python/env/lib/python2.7/site-packages (from -r
>>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt
>>>>> (line 3))
>>>>> Requirement already satisfied (use --upgrade to upgrade): ecdsa==0.13
>>>>> in ./infra/python/env/lib/python2.7/site-packages (from -r
>>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt
>>>>> (line 4))
>>>>> Requirement already satisfied (use --upgrade to upgrade):
>>>>> pycrypto==2.6.1 in ./infra/python/env/lib/python2.7/site-packages (from -r
>>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt
>>>>> (line 5))
>>>>> Requirement already satisfied (use --upgrade to upgrade):
>>>>> impyla==0.14.0 in ./infra/python/env/lib/python2.7/site-packages (from -r
>>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt
>>>>> (line 6))
>>>>> Requirement already satisfied (use --upgrade to upgrade):
>>>>> bitarray==0.8.1 in ./infra/python/env/lib/python2.7/site-packages (from -r
>>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt
>>>>> (line 7))
>>>>> Collecting sasl==0.1.3 (from -r
>>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt
>>>>> (line 8))
>>>>> Requirement already satisfied (use --upgrade to upgrade): six==1.9.0 in
>>>>> ./infra/python/env/lib/python2.7/site-packages (from -r
>>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt
>>>>> (line 9))
>>>>> Collecting thrift==0.9.0 (from -r
>>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt
>>>>> (line 10))
>>>>> Collecting thrift-sasl==0.1.0 (from -r
>>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt
>>>>> (line 11))
>>>>> Collecting psutil==0.7.1 (from -r
>>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt
>>>>> (line 12))
>>>>> Collecting Cython==0.23.4 (from -r
>>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt
>>>>> (line 13))
>>>>> Collecting numpy==1.10.4 (from -r
>>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt
>>>>> (line 14))
>>>>> Installing collected packages: sasl, thrift, thrift-sasl, psutil,
>>>>> Cython, numpy
>>>>> Running setup.py install for sasl
>>>>> Complete output from command
>>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/env/bin/python
>>>>> -c "import setuptools,
>>>>> tokenize;__file__='/tmp/pip-build-MTPxSs/sasl/setup.py';exec(compile(getattr(tokenize,
>>>>> 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))"
>>>>> install --record /tmp/pip-EHfJDn-record/install-record.txt
>>>>> --single-version-externally-managed --compile --install-headers
>>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/env/include/site/python2.7/sasl:
>>>>> running install
>>>>> running build
>>>>> running build_py
>>>>> creating build
>>>>> creating build/lib.linux-x86_64-2.7
>>>>> creating build/lib.linux-x86_64-2.7/sasl
>>>>> copying sasl/__init__.py -> build/lib.linux-x86_64-2.7/sasl
>>>>> copying sasl/saslwrapper.py -> build/lib.linux-x86_64-2.7/sasl
>>>>> running egg_info
>>>>> writing sasl.egg-info/PKG-INFO
>>>>> writing top-level names to sasl.egg-info/top_level.txt
>>>>> writing dependency_links to sasl.egg-info/dependency_links.txt
>>>>> warning: manifest_maker: standard file '-c' not found
>>>>>
>>>>> reading manifest file 'sasl.egg-info/SOURCES.txt'
>>>>> reading manifest template 'MANIFEST.in'
>>>>> writing manifest file 'sasl.egg-info/SOURCES.txt'
>>>>> running build_ext
>>>>> building '_saslwrapper' extension
>>>>> creating build/temp.linux-x86_64-2.7
>>>>> creating build/temp.linux-x86_64-2.7/sasl
>>>>>
>>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
>>>>> -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions
>>>>> -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches
>>>>> -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall
>>>>> -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong
>>>>> --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic
>>>>> -D_GNU_SOURCE -fPIC -fwrapv -fPIC -Isasl -I/usr/include/python2.7 -c
>>>>> sasl/saslwrapper.cpp -o build/temp.linux-x86_64-2.7/sasl/saslwrapper.o
>>>>> sasl/saslwrapper.cpp:21:23: fatal error: sasl/sasl.h: No such file
>>>>> or directory
>>>>> #include <sasl/sasl.h>
>>>>> ^
>>>>> compilation terminated.
>>>>> error: command
>>>>> '/opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc'
>>>>> failed with exit status 1
>>>>>
>>>>> ----------------------------------------
>>>>> Command
>>>>> "/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/env/bin/python
>>>>> -c "import setuptools,
>>>>> tokenize;__file__='/tmp/pip-build-MTPxSs/sasl/setup.py';exec(compile(getattr(tokenize,
>>>>> 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))"
>>>>> install --record /tmp/pip-EHfJDn-record/install-record.txt
>>>>> --single-version-externally-managed --compile --install-headers
>>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/env/include/site/python2.7/sasl"
>>>>> failed with error code 1 in /tmp/pip-build-MTPxSs/sasl
>>>>>
>>>>> Error in
>>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/impala-python at line
>>>>> 28:
>>>>>
>>>>>
>>>>>
>>>>> Regards,
>>>>> Anup Tiwari
>>>>>
>>>>> On Tue, Feb 6, 2018 at 2:52 PM, Anup Tiwari <an...@gmail.com>
>>>>> wrote:
>>>>>>
>>>>>> Hi Zoltan,
>>>>>>
>>>>>> I followed the steps mention in
>>>>>> https://cwiki.apache.org/confluence/display/IMPALA/Building+Impala and when
>>>>>> i executed below command , i got error after some time :-
>>>>>>
>>>>>> Build Command :
>>>>>> ${IMPALA_HOME}/buildall.sh -noclean -skiptests -build_shared_libs
>>>>>> -format
>>>>>>
>>>>>> Error :
>>>>>> -- ----------> Adding thirdparty library cyrus_sasl. <----------
>>>>>> CMake Error at CMakeLists.txt:136 (message):
>>>>>> Library 'cyrus_sasl' has neither shared nor static library files
>>>>>> Call Stack (most recent call first):
>>>>>> CMakeLists.txt:227 (IMPALA_ADD_THIRDPARTY_LIB)
>>>>>>
>>>>>> -- Configuring incomplete, errors occurred!
>>>>>> See also
>>>>>> "/opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/CMakeOutput.log".
>>>>>> See also
>>>>>> "/opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/CMakeError.log".
>>>>>> Error in
>>>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/make_impala.sh at line
>>>>>> 161: cmake . ${CMAKE_ARGS[@]}
>>>>>>
>>>>>>
>>>>>> Logs in "CMakeOutput.log" :-
>>>>>>
>>>>>> Determining if the function pthread_create exists in the pthread
>>>>>> passed with the following output:
>>>>>> Change Dir:
>>>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/CMakeTmp
>>>>>>
>>>>>> Run Build Command:"/usr/bin/gmake" "cmTC_ee4f0/fast"
>>>>>> /usr/bin/gmake -f CMakeFiles/cmTC_ee4f0.dir/build.make
>>>>>> CMakeFiles/cmTC_ee4f0.dir/build
>>>>>> gmake[1]: Entering directory
>>>>>> `/opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/CMakeTmp'
>>>>>> Building C object CMakeFiles/cmTC_ee4f0.dir/CheckFunctionExists.c.o
>>>>>>
>>>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
>>>>>> -DCHECK_FUNCTION_EXISTS=pthread_create -o
>>>>>> CMakeFiles/cmTC_ee4f0.dir/CheckFunctionExists.c.o -c
>>>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/cmake-3.8.2-p1/share/cmake-3.8/Modules/CheckFunctionExists.c
>>>>>> Linking C executable cmTC_ee4f0
>>>>>>
>>>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/cmake-3.8.2-p1/bin/cmake
>>>>>> -E cmake_link_script CMakeFiles/cmTC_ee4f0.dir/link.txt --verbose=1
>>>>>>
>>>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
>>>>>> -DCHECK_FUNCTION_EXISTS=pthread_create -rdynamic
>>>>>> CMakeFiles/cmTC_ee4f0.dir/CheckFunctionExists.c.o -o cmTC_ee4f0 -lpthread
>>>>>> gmake[1]: Leaving directory
>>>>>> `/opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/CMakeTmp'
>>>>>>
>>>>>> Logs in "CMakeError.log" :-
>>>>>>
>>>>>> Determining if the function pthread_create exists in the pthreads
>>>>>> failed with the following output:
>>>>>> Change Dir:
>>>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/CMakeTmp
>>>>>>
>>>>>> Run Build Command:"/usr/bin/gmake" "cmTC_18bf3/fast"
>>>>>> /usr/bin/gmake -f CMakeFiles/cmTC_18bf3.dir/build.make
>>>>>> CMakeFiles/cmTC_18bf3.dir/build
>>>>>> gmake[1]: Entering directory
>>>>>> `/opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/CMakeTmp'
>>>>>> Building C object CMakeFiles/cmTC_18bf3.dir/CheckFunctionExists.c.o
>>>>>>
>>>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
>>>>>> -DCHECK_FUNCTION_EXISTS=pthread_create -o
>>>>>> CMakeFiles/cmTC_18bf3.dir/CheckFunctionExists.c.o -c
>>>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/cmake-3.8.2-p1/share/cmake-3.8/Modules/CheckFunctionExists.c
>>>>>> Linking C executable cmTC_18bf3
>>>>>>
>>>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/cmake-3.8.2-p1/bin/cmake
>>>>>> -E cmake_link_script CMakeFiles/cmTC_18bf3.dir/link.txt --verbose=1
>>>>>>
>>>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
>>>>>> -DCHECK_FUNCTION_EXISTS=pthread_create -rdynamic
>>>>>> CMakeFiles/cmTC_18bf3.dir/CheckFunctionExists.c.o -o cmTC_18bf3 -lpthreads
>>>>>> /usr/bin/ld: cannot find -lpthreads
>>>>>> collect2: error: ld returned 1 exit status
>>>>>> gmake[1]: *** [cmTC_18bf3] Error 1
>>>>>> gmake[1]: Leaving directory
>>>>>> `/opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/CMakeTmp'
>>>>>> gmake: *** [cmTC_18bf3/fast] Error 2
>>>>>>
>>>>>> Also it seems that this is related to toolchain error so i followed
>>>>>> this link :
>>>>>> https://cwiki.apache.org/confluence/display/IMPALA/Building+native-toolchain+from+scratch+and+using+with+Impala
>>>>>> where i first builded native-toolchain and then followed step mention in
>>>>>> "Using custom toolchain with Impala" but while creating symbolic link i got
>>>>>> error "ln: ‘./gcc-4.9.2’: cannot overwrite directory".
>>>>>>
>>>>>> @xinran, i guess post building impala, i will have to perform steps
>>>>>> mentioned by you.. correct me if i am wrong?
>>>>>>
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>> Anup Tiwari
>>>>>>
>>>>>> On Mon, Feb 5, 2018 at 11:14 PM, Xinran Yu Tinney
>>>>>> <yu...@gmail.com> wrote:
>>>>>>>
>>>>>>> Hi Anup,
>>>>>>> Did you also source the config.sh? Run the following before
>>>>>>> impala-shell.sh:
>>>>>>>
>>>>>>> source ~/Impala/bin/impala-config.sh
>>>>>>>
>>>>>>> ${IMPALA_HOME}/buildall.sh -notests -build_shared_libs
>>>>>>> -start_minicluster
>>>>>>>
>>>>>>> ${IMPALA_HOME}/bin/start-impala-cluster.py
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2018-02-05 10:07 GMT-06:00 Michael Brown <mi...@cloudera.com>:
>>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> Please see:
>>>>>>>>
>>>>>>>>
>>>>>>>> https://cwiki.apache.org/confluence/display/IMPALA/Bootstrapping+an+Impala+Development+Environment+From+Scratch
>>>>>>>> https://cwiki.apache.org/confluence/display/IMPALA/Building+Impala
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Feb 5, 2018 at 12:09 AM, Anup Tiwari
>>>>>>>> <an...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>> Hi All,
>>>>>>>>>
>>>>>>>>> I am new to impala so i am getting errors while i am setting
>>>>>>>>> impala.
>>>>>>>>>
>>>>>>>>> I have downloaded latest impala release from
>>>>>>>>> "http://www.apache.org/dyn/closer.cgi?action=download&filename=impala/2.11.0/apache-impala-2.11.0.tar.gz".
>>>>>>>>> I have extracted this in a folder and followed description
>>>>>>>>> mentioned in below link :-
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> https://impala.apache.org/docs/build/html/topics/impala_processes.html#processes
>>>>>>>>>
>>>>>>>>> As mentioned in above link "To start the Impala state store and
>>>>>>>>> Impala from the command line or a script, you can either use the service
>>>>>>>>> command or you can start the daemons directly through the impalad,
>>>>>>>>> statestored, and catalogd executables." So i searched for executable and
>>>>>>>>> found "start-impalad.sh"(which i think is correct one, let me know if i have
>>>>>>>>> missed this) but when i am executing this , i am getting below errors :-
>>>>>>>>>
>>>>>>>>> bin/start-impalad.sh: line 89: /admin: No such file or directory
>>>>>>>>> bin/start-impalad.sh: line 106:
>>>>>>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/be/build/latest/service/impalad:
>>>>>>>>> No such file or directory
>>>>>>>>>
>>>>>>>>> Also when i executed ./impala-config.sh then i got below error :-
>>>>>>>>>
>>>>>>>>> Unable to find the lsb_release command. Please ensure it is
>>>>>>>>> available in your PATH.
>>>>>>>>> ./impala-config.sh: line 265: return: can only `return' from a
>>>>>>>>> function or sourced script
>>>>>>>>> lsb_release command failed, output was: ./impala-config.sh: line
>>>>>>>>> 267: lsb_release: command not found
>>>>>>>>> ./impala-config.sh: line 270: return: can only `return' from a
>>>>>>>>> function or sourced script
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Questions :-
>>>>>>>>> 1. When we setup Hive/Hadoop/Drill etc then we have certain steps
>>>>>>>>> for setting HIVE_HOME/HADOOP_HOME/DRILL_HOME and also we specify certain
>>>>>>>>> setting in conf folder. Do we have similar documentation which can specify
>>>>>>>>> step by step setup.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Anup Tiwari
>>>>>>>>>
>>>>>>>>> On Tue, Jan 23, 2018 at 8:56 PM, Jim Apple <jb...@cloudera.com>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Ubuntu 14.04 and 16.04 are the OSes most Impala developers use,
>>>>>>>>>> but
>>>>>>>>>> many Impala users use CentOS 7.
>>>>>>>>>>
>>>>>>>>>> Here is part of the docs on tuning:
>>>>>>>>>>
>>>>>>>>>> https://impala.apache.org/docs/build/html/topics/impala_performance.html
>>>>>>>>>>
>>>>>>>>>> Here is some tuning info that was produced by Cloudera:
>>>>>>>>>>
>>>>>>>>>> http://blog.cloudera.com/wp-content/uploads/2017/03/Impala-Cookbook-01-20173.pdf
>>>>>>>>>>
>>>>>>>>>> On Tue, Jan 23, 2018 at 6:53 AM, Anup Tiwari
>>>>>>>>>> <an...@gmail.com> wrote:
>>>>>>>>>> > Hi Vincent,
>>>>>>>>>> >
>>>>>>>>>> > I found this link which says Supported Operating Systems are
>>>>>>>>>> > Ubuntu 14.04,
>>>>>>>>>> > 16.04 so does this mean it won't work on centos 7.
>>>>>>>>>> >
>>>>>>>>>> > Also can you send some link which can help us in tuning as well.
>>>>>>>>>> >
>>>>>>>>>> > Regards,
>>>>>>>>>> > Anup Tiwari
>>>>>>>>>> >
>>>>>>>>>> > On Tue, Jan 23, 2018 at 7:26 PM, Vincent Tran
>>>>>>>>>> > <vt...@cloudera.com> wrote:
>>>>>>>>>> >>
>>>>>>>>>> >> Hi Anup,
>>>>>>>>>> >>
>>>>>>>>>> >> Is this the link you are looking for?
>>>>>>>>>> >>
>>>>>>>>>> >>
>>>>>>>>>> >> https://impala.apache.org/docs/build/html/topics/impala_install.html
>>>>>>>>>> >>
>>>>>>>>>> >> On Tue, Jan 23, 2018 at 7:52 AM, Anup Tiwari
>>>>>>>>>> >> <an...@gmail.com>
>>>>>>>>>> >> wrote:
>>>>>>>>>> >>>
>>>>>>>>>> >>> Hi All
>>>>>>>>>> >>>
>>>>>>>>>> >>> I want to install impala on my production
>>>>>>>>>> >>> environment(distributed) :-
>>>>>>>>>> >>>
>>>>>>>>>> >>> 1. Os Details : CentOS Linux release 7.2.1511 (Core)
>>>>>>>>>> >>> 2. Apache Hadoop 2.8.0
>>>>>>>>>> >>> 3. Apache Hive 2.1.1
>>>>>>>>>> >>>
>>>>>>>>>> >>> I didn't found any link on mail, please suggest if it is
>>>>>>>>>> >>> possible or not
>>>>>>>>>> >>> and if possible then please share some links.
>>>>>>>>>> >>>
>>>>>>>>>> >>> Thanks in advance!!
>>>>>>>>>> >>>
>>>>>>>>>> >>> Regards,
>>>>>>>>>> >>> Anup Tiwari
>>>>>>>>>> >>
>>>>>>>>>> >>
>>>>>>>>>> >>
>>>>>>>>>> >>
>>>>>>>>>> >> --
>>>>>>>>>> >> Vincent T. Tran
>>>>>>>>>> >> Customer Operations Engineer
>>>>>>>>>> >> Cloudera, Inc.
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>
>>
>>
>
Re: Install Impala on Apache Hadoop and Hive
Posted by Anup Tiwari <an...@gmail.com>.
Hi,
Any update on this?
Regards,
Anup Tiwari
On Wed, Feb 7, 2018 at 11:43 PM, Anup Tiwari <an...@gmail.com> wrote:
> Hi,
>
> I have executed below commands
>
> *Build Command :*
> ${IMPALA_HOME}/buildall.sh -noclean -skiptests -build_shared_libs -format
>
> Please refer trail mail for details or steps.
>
> On 07-Feb-2018 10:51 PM, "Tim Armstrong" <ta...@cloudera.com> wrote:
>
> What command did you run to kick off the build? It looks like you're
> missing the HDFS dependency from the expected location, but that should be
> automatically downloaded if you run buildall.sh with the default settings.
>
> On Wed, Feb 7, 2018 at 6:30 AM, Anup Tiwari <an...@gmail.com>
> wrote:
>
>> Hi Vincent,
>>
>> I have installed all packages/modules mentioned in
>> *https://cwiki.apache.org/confluence/display/IMPALA/Impala+Build+Prerequisites*
>> <https://cwiki.apache.org/confluence/display/IMPALA/Impala+Build+Prerequisites>
>> and now getting below error while building and the one which you have
>> mention is for *Ubuntu 14.04* as per docs and i am using centos7 as
>> mention in trail mail :-
>>
>> -- Added shared library dependency zlib: /opt/apache-s/apache-impala/ap
>> ache-impala-2.11.0/toolchain/zlib-1.2.8/lib/libz.so
>> CMake Error at cmake_modules/FindHDFS.cmake:78 (message):
>> HDFS includes and libraries NOT found. HDFS_INCLUDE_DIR-NOTFOUND, )
>> Call Stack (most recent call first):
>> CMakeLists.txt:183 (find_package)
>> -- Configuring incomplete, errors occurred!
>> See also "/opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles
>> /CMakeOutput.log".
>> See also "/opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles
>> /CMakeError.log".
>> Error in /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/make_impala.sh
>> at line 161: cmake . ${CMAKE_ARGS[@]}
>>
>>
>> *OUTPUT of CMakeError.log :-*
>>
>> Determining if the pthread_create exist failed with the following output:
>> Change Dir: /opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/
>> CMakeTmp
>>
>> Run Build Command:"/usr/bin/gmake" "cmTC_18658/fast"
>> /usr/bin/gmake -f CMakeFiles/cmTC_18658.dir/build.make
>> CMakeFiles/cmTC_18658.dir/build
>> gmake[1]: Entering directory `/opt/apache-s/apache-impala/a
>> pache-impala-2.11.0/CMakeFiles/CMakeTmp'
>> Building C object CMakeFiles/cmTC_18658.dir/CheckSymbolExists.c.o
>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
>> -o CMakeFiles/cmTC_18658.dir/CheckSymbolExists.c.o -c
>> /opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/
>> CMakeTmp/CheckSymbolExists.c
>> Linking C executable cmTC_18658
>> /opt/apache-s/apache-impala/native-toolchain/build/cmake-3.8.2-p1/bin/cmake
>> -E cmake_link_script CMakeFiles/cmTC_18658.dir/link.txt --verbose=1
>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
>> -rdynamic CMakeFiles/cmTC_18658.dir/CheckSymbolExists.c.o -o cmTC_18658
>> CMakeFiles/cmTC_18658.dir/CheckSymbolExists.c.o: In function `main':
>> CheckSymbolExists.c:(.text+0x16): undefined reference to `pthread_create'
>> collect2: error: ld returned 1 exit status
>> gmake[1]: *** [cmTC_18658] Error 1
>> gmake[1]: Leaving directory `/opt/apache-s/apache-impala/a
>> pache-impala-2.11.0/CMakeFiles/CMakeTmp'
>> gmake: *** [cmTC_18658/fast] Error 2
>>
>> File /opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/
>> CMakeTmp/CheckSymbolExists.c:
>> /* */
>> #include <pthread.h>
>>
>> int main(int argc, char** argv)
>> {
>> (void)argv;
>> #ifndef pthread_create
>> return ((int*)(&pthread_create))[argc];
>> #else
>> (void)argc;
>> return 0;
>> #endif
>> }
>>
>> Determining if the function pthread_create exists in the pthreads failed
>> with the following output:
>> Change Dir: /opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/
>> CMakeTmp
>>
>> Run Build Command:"/usr/bin/gmake" "cmTC_81779/fast"
>> /usr/bin/gmake -f CMakeFiles/cmTC_81779.dir/build.make
>> CMakeFiles/cmTC_81779.dir/build
>> gmake[1]: Entering directory `/opt/apache-s/apache-impala/a
>> pache-impala-2.11.0/CMakeFiles/CMakeTmp'
>> Building C object CMakeFiles/cmTC_81779.dir/CheckFunctionExists.c.o
>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
>> -DCHECK_FUNCTION_EXISTS=pthread_create -o
>> CMakeFiles/cmTC_81779.dir/CheckFunctionExists.c.o -c
>> /opt/apache-s/apache-impala/native-toolchain/build/cmake-3.8
>> .2-p1/share/cmake-3.8/Modules/CheckFunctionExists.c
>> Linking C executable cmTC_81779
>> /opt/apache-s/apache-impala/native-toolchain/build/cmake-3.8.2-p1/bin/cmake
>> -E cmake_link_script CMakeFiles/cmTC_81779.dir/link.txt --verbose=1
>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
>> -DCHECK_FUNCTION_EXISTS=pthread_create -rdynamic
>> CMakeFiles/cmTC_81779.dir/CheckFunctionExists.c.o -o cmTC_81779
>> -lpthreads
>> /usr/bin/ld: cannot find -lpthreads
>> collect2: error: ld returned 1 exit status
>> gmake[1]: *** [cmTC_81779] Error 1
>> gmake[1]: Leaving directory `/opt/apache-s/apache-impala/a
>> pache-impala-2.11.0/CMakeFiles/CMakeTmp'
>> gmake: *** [cmTC_81779/fast] Error 2
>>
>>
>>
>>
>>
>> Regards,
>> Anup Tiwari
>>
>> On Tue, Feb 6, 2018 at 6:07 PM, Vincent Tran <vt...@cloudera.com> wrote:
>>
>>> Hey Anup,
>>>
>>> Lools like ypu may be missing some prereq packages. This particular one
>>> is libsasl2-dev
>>>
>>>
>>>
>>> On Feb 6, 2018 5:12 AM, "Anup Tiwari" <an...@gmail.com> wrote:
>>>
>>>> I found this link http://mail-archives.apache.or
>>>> g/mod_mbox/impala-dev/201708.mbox/%3CCADZba-Td-wXRqLRxctvFM2
>>>> mY44cYJhkX-JSs4=uPQCnFDhoDyA@mail.gmail.com%3E which says
>>>>
>>>> Do the following:
>>>>
>>>> rm -rf ${IMPALA_TOOLCHAIN}/gflags-${IMPALA_GFLAGS_VERSION}
>>>> . bin/impala-config.sh
>>>> bin/bootstrap_toolchain.py
>>>>
>>>> But post executing bootstrap command ; i am getting below error :-
>>>>
>>>> Traceback (most recent call last):
>>>> File "/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/bootstrap_virtualenv.py", line 367, in <module>
>>>> if install_compiled_deps_if_possible():
>>>> File "/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/bootstrap_virtualenv.py", line 216, in install_compiled_deps_if_possible
>>>> exec_pip_install(["-r", COMPILED_REQS_PATH], cc=cc, env=env)
>>>> File "/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/bootstrap_virtualenv.py", line 145, in exec_pip_install
>>>> "file://%s" % urllib.pathname2url(os.path.abspath(DEPS_DIR))] + args, env=env)
>>>> File "/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/bootstrap_virtualenv.py", line 96, in exec_cmd
>>>> % (args, output))
>>>> Exception: Command returned non-zero status
>>>> Command: ['/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/env/bin/python', '/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/env/bin/pip', 'install', '--no-binary', '--no-index', '--no-cache-dir', '--find-links', 'file:///opt/apache-s/apache-impala/apache-impala-2.11.0/infra/python/deps', '-r', '/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt']
>>>> Output: You are using pip version 7.1.0, however version 9.0.1 is available.
>>>> You should consider upgrading via the 'pip install --upgrade pip' command.
>>>> Requirement already satisfied (use --upgrade to upgrade): argparse==1.4.0 in ./infra/python/env/lib/python2.7/site-packages (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 1))
>>>> Requirement already satisfied (use --upgrade to upgrade): Fabric==1.10.2 in ./infra/python/env/lib/python2.7/site-packages (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 2))
>>>> Requirement already satisfied (use --upgrade to upgrade): paramiko==1.15.2 in ./infra/python/env/lib/python2.7/site-packages (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 3))
>>>> Requirement already satisfied (use --upgrade to upgrade): ecdsa==0.13 in ./infra/python/env/lib/python2.7/site-packages (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 4))
>>>> Requirement already satisfied (use --upgrade to upgrade): pycrypto==2.6.1 in ./infra/python/env/lib/python2.7/site-packages (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 5))
>>>> Requirement already satisfied (use --upgrade to upgrade): impyla==0.14.0 in ./infra/python/env/lib/python2.7/site-packages (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 6))
>>>> Requirement already satisfied (use --upgrade to upgrade): bitarray==0.8.1 in ./infra/python/env/lib/python2.7/site-packages (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 7))
>>>> Collecting sasl==0.1.3 (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 8))
>>>> Requirement already satisfied (use --upgrade to upgrade): six==1.9.0 in ./infra/python/env/lib/python2.7/site-packages (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 9))
>>>> Collecting thrift==0.9.0 (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 10))
>>>> Collecting thrift-sasl==0.1.0 (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 11))
>>>> Collecting psutil==0.7.1 (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 12))
>>>> Collecting Cython==0.23.4 (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 13))
>>>> Collecting numpy==1.10.4 (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 14))
>>>> Installing collected packages: sasl, thrift, thrift-sasl, psutil, Cython, numpy
>>>> Running setup.py install for sasl
>>>> Complete output from command /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/env/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-build-MTPxSs/sasl/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-EHfJDn-record/install-record.txt --single-version-externally-managed --compile --install-headers /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/env/include/site/python2.7/sasl:
>>>> running install
>>>> running build
>>>> running build_py
>>>> creating build
>>>> creating build/lib.linux-x86_64-2.7
>>>> creating build/lib.linux-x86_64-2.7/sasl
>>>> copying sasl/__init__.py -> build/lib.linux-x86_64-2.7/sasl
>>>> copying sasl/saslwrapper.py -> build/lib.linux-x86_64-2.7/sasl
>>>> running egg_info
>>>> writing sasl.egg-info/PKG-INFO
>>>> writing top-level names to sasl.egg-info/top_level.txt
>>>> writing dependency_links to sasl.egg-info/dependency_links.txt
>>>> warning: manifest_maker: standard file '-c' not found
>>>>
>>>> reading manifest file 'sasl.egg-info/SOURCES.txt'
>>>> reading manifest template 'MANIFEST.in'
>>>> writing manifest file 'sasl.egg-info/SOURCES.txt'
>>>> running build_ext
>>>> building '_saslwrapper' extension
>>>> creating build/temp.linux-x86_64-2.7
>>>> creating build/temp.linux-x86_64-2.7/sasl
>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -Isasl -I/usr/include/python2.7 -c sasl/saslwrapper.cpp -o build/temp.linux-x86_64-2.7/sasl/saslwrapper.o
>>>> sasl/saslwrapper.cpp:21:23: fatal error: sasl/sasl.h: No such file or directory
>>>> #include <sasl/sasl.h>
>>>> ^
>>>> compilation terminated.
>>>> error: command '/opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc' failed with exit status 1
>>>>
>>>> ----------------------------------------
>>>> Command "/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/env/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-build-MTPxSs/sasl/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-EHfJDn-record/install-record.txt --single-version-externally-managed --compile --install-headers /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/env/include/site/python2.7/sasl" failed with error code 1 in /tmp/pip-build-MTPxSs/sasl
>>>>
>>>> Error in /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/impala-python at line 28:
>>>>
>>>>
>>>>
>>>> Regards,
>>>> Anup Tiwari
>>>>
>>>> On Tue, Feb 6, 2018 at 2:52 PM, Anup Tiwari <an...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Zoltan,
>>>>>
>>>>> I followed the steps mention in https://cwiki.apache.org/confl
>>>>> uence/display/IMPALA/Building+Impala and when i executed below
>>>>> command , i got error after some time :-
>>>>>
>>>>> *Build Command :*
>>>>> ${IMPALA_HOME}/buildall.sh -noclean -skiptests -build_shared_libs
>>>>> -format
>>>>>
>>>>>
>>>>>
>>>>> *Error :-- ----------> Adding thirdparty library cyrus_sasl.
>>>>> <----------*CMake Error at CMakeLists.txt:136 (message):
>>>>> Library 'cyrus_sasl' has neither shared nor static library files
>>>>> Call Stack (most recent call first):
>>>>> CMakeLists.txt:227 (IMPALA_ADD_THIRDPARTY_LIB)
>>>>>
>>>>> -- Configuring incomplete, errors occurred!
>>>>> See also "/opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles
>>>>> /CMakeOutput.log".
>>>>> See also "/opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles
>>>>> /CMakeError.log".
>>>>> Error in /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/make_impala.sh
>>>>> at line 161: cmake . ${CMAKE_ARGS[@]}
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *Logs in "CMakeOutput.log" :-*Determining if the function
>>>>> pthread_create exists in the pthread passed with the following output:
>>>>> Change Dir: /opt/apache-s/apache-impala/ap
>>>>> ache-impala-2.11.0/CMakeFiles/CMakeTmp
>>>>>
>>>>> Run Build Command:"/usr/bin/gmake" "cmTC_ee4f0/fast"
>>>>> /usr/bin/gmake -f CMakeFiles/cmTC_ee4f0.dir/build.make
>>>>> CMakeFiles/cmTC_ee4f0.dir/build
>>>>> gmake[1]: Entering directory `/opt/apache-s/apache-impala/a
>>>>> pache-impala-2.11.0/CMakeFiles/CMakeTmp'
>>>>> Building C object CMakeFiles/cmTC_ee4f0.dir/CheckFunctionExists.c.o
>>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
>>>>> -DCHECK_FUNCTION_EXISTS=pthread_create -o
>>>>> CMakeFiles/cmTC_ee4f0.dir/CheckFunctionExists.c.o -c
>>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/c
>>>>> make-3.8.2-p1/share/cmake-3.8/Modules/CheckFunctionExists.c
>>>>> Linking C executable cmTC_ee4f0
>>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/cmake-3.8.2-p1/bin/cmake
>>>>> -E cmake_link_script CMakeFiles/cmTC_ee4f0.dir/link.txt --verbose=1
>>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
>>>>> -DCHECK_FUNCTION_EXISTS=pthread_create -rdynamic
>>>>> CMakeFiles/cmTC_ee4f0.dir/CheckFunctionExists.c.o -o cmTC_ee4f0
>>>>> -lpthread
>>>>> gmake[1]: Leaving directory `/opt/apache-s/apache-impala/a
>>>>> pache-impala-2.11.0/CMakeFiles/CMakeTmp'
>>>>>
>>>>>
>>>>>
>>>>> *Logs in "CMakeError.log" :-*Determining if the function
>>>>> pthread_create exists in the pthreads failed with the following output:
>>>>> Change Dir: /opt/apache-s/apache-impala/ap
>>>>> ache-impala-2.11.0/CMakeFiles/CMakeTmp
>>>>>
>>>>> Run Build Command:"/usr/bin/gmake" "cmTC_18bf3/fast"
>>>>> /usr/bin/gmake -f CMakeFiles/cmTC_18bf3.dir/build.make
>>>>> CMakeFiles/cmTC_18bf3.dir/build
>>>>> gmake[1]: Entering directory `/opt/apache-s/apache-impala/a
>>>>> pache-impala-2.11.0/CMakeFiles/CMakeTmp'
>>>>> Building C object CMakeFiles/cmTC_18bf3.dir/CheckFunctionExists.c.o
>>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
>>>>> -DCHECK_FUNCTION_EXISTS=pthread_create -o
>>>>> CMakeFiles/cmTC_18bf3.dir/CheckFunctionExists.c.o -c
>>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/c
>>>>> make-3.8.2-p1/share/cmake-3.8/Modules/CheckFunctionExists.c
>>>>> Linking C executable cmTC_18bf3
>>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/cmake-3.8.2-p1/bin/cmake
>>>>> -E cmake_link_script CMakeFiles/cmTC_18bf3.dir/link.txt --verbose=1
>>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
>>>>> -DCHECK_FUNCTION_EXISTS=pthread_create -rdynamic
>>>>> CMakeFiles/cmTC_18bf3.dir/CheckFunctionExists.c.o -o cmTC_18bf3
>>>>> -lpthreads
>>>>> /usr/bin/ld: cannot find -lpthreads
>>>>> collect2: error: ld returned 1 exit status
>>>>> gmake[1]: *** [cmTC_18bf3] Error 1
>>>>> gmake[1]: Leaving directory `/opt/apache-s/apache-impala/a
>>>>> pache-impala-2.11.0/CMakeFiles/CMakeTmp'
>>>>> gmake: *** [cmTC_18bf3/fast] Error 2
>>>>>
>>>>> Also it seems that this is related to toolchain error so i followed
>>>>> this link : https://cwiki.apache.org/confl
>>>>> uence/display/IMPALA/Building+native-toolchain+from+scratch+
>>>>> and+using+with+Impala where i first builded native-toolchain and then
>>>>> followed step mention in "Using custom toolchain with Impala" but while
>>>>> creating symbolic link i got error "ln: ‘./gcc-4.9.2’: cannot
>>>>> overwrite directory".
>>>>>
>>>>> @xinran, i guess post building impala, i will have to perform steps
>>>>> mentioned by you.. correct me if i am wrong?
>>>>>
>>>>>
>>>>>
>>>>> Regards,
>>>>> Anup Tiwari
>>>>>
>>>>> On Mon, Feb 5, 2018 at 11:14 PM, Xinran Yu Tinney <
>>>>> yuxinran8011@gmail.com> wrote:
>>>>>
>>>>>> Hi Anup,
>>>>>> Did you also source the config.sh? Run the following before
>>>>>> impala-shell.sh:
>>>>>>
>>>>>> source ~/Impala/bin/impala-config.sh
>>>>>>
>>>>>> ${IMPALA_HOME}/buildall.sh -notests -build_shared_libs
>>>>>> -start_minicluster
>>>>>>
>>>>>> ${IMPALA_HOME}/bin/start-impala-cluster.py
>>>>>>
>>>>>>
>>>>>> 2018-02-05 10:07 GMT-06:00 Michael Brown <mi...@cloudera.com>:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> Please see:
>>>>>>>
>>>>>>> https://cwiki.apache.org/confluence/display/IMPALA/Bootstrap
>>>>>>> ping+an+Impala+Development+Environment+From+Scratch
>>>>>>> https://cwiki.apache.org/confluence/display/IMPALA/Building+Impala
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Feb 5, 2018 at 12:09 AM, Anup Tiwari <anupsdtiwari@gmail.com
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> Hi All,
>>>>>>>>
>>>>>>>> I am new to impala so i am getting errors while i am setting impala.
>>>>>>>>
>>>>>>>> I have downloaded latest impala release from "
>>>>>>>> http://www.apache.org/dyn/closer.cgi?action=download&filena
>>>>>>>> me=impala/2.11.0/apache-impala-2.11.0.tar.gz".
>>>>>>>> I have extracted this in a folder and followed description
>>>>>>>> mentioned in below link :-
>>>>>>>>
>>>>>>>> https://impala.apache.org/docs/build/html/topics/impala_proc
>>>>>>>> esses.html#processes
>>>>>>>>
>>>>>>>> As mentioned in above link "*To start the Impala state store and
>>>>>>>> Impala from the command line or a script, you can either use the service
>>>>>>>> command or you can start the daemons directly through the impalad,
>>>>>>>> statestored, and catalogd executables.*" So i searched for
>>>>>>>> executable and found "start-impalad.sh"(which i think is correct one, let
>>>>>>>> me know if i have missed this) but when i am executing this , i am getting
>>>>>>>> below *errors* :-
>>>>>>>>
>>>>>>>> bin/start-impalad.sh: line 89: /admin: No such file or directory
>>>>>>>> bin/start-impalad.sh: line 106: /opt/apache-s/apache-impala/ap
>>>>>>>> ache-impala-2.11.0/be/build/latest/service/impalad: No such file
>>>>>>>> or directory
>>>>>>>>
>>>>>>>> Also when i executed ./impala-config.sh then i got below *error* :-
>>>>>>>>
>>>>>>>> Unable to find the lsb_release command. Please ensure it is
>>>>>>>> available in your PATH.
>>>>>>>> ./impala-config.sh: line 265: return: can only `return' from a
>>>>>>>> function or sourced script
>>>>>>>> lsb_release command failed, output was: ./impala-config.sh: line
>>>>>>>> 267: lsb_release: command not found
>>>>>>>> ./impala-config.sh: line 270: return: can only `return' from a
>>>>>>>> function or sourced script
>>>>>>>>
>>>>>>>>
>>>>>>>> *Questions :-*
>>>>>>>> 1. When we setup Hive/Hadoop/Drill etc then we have certain steps
>>>>>>>> for setting HIVE_HOME/HADOOP_HOME/DRILL_HOME and also we specify
>>>>>>>> certain setting in conf folder. Do we have similar documentation which can
>>>>>>>> specify step by step setup.
>>>>>>>>
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Anup Tiwari
>>>>>>>>
>>>>>>>> On Tue, Jan 23, 2018 at 8:56 PM, Jim Apple <jb...@cloudera.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Ubuntu 14.04 and 16.04 are the OSes most Impala developers use, but
>>>>>>>>> many Impala users use CentOS 7.
>>>>>>>>>
>>>>>>>>> Here is part of the docs on tuning:
>>>>>>>>> https://impala.apache.org/docs/build/html/topics/impala_perf
>>>>>>>>> ormance.html
>>>>>>>>>
>>>>>>>>> Here is some tuning info that was produced by Cloudera:
>>>>>>>>> http://blog.cloudera.com/wp-content/uploads/2017/03/Impala-C
>>>>>>>>> ookbook-01-20173.pdf
>>>>>>>>>
>>>>>>>>> On Tue, Jan 23, 2018 at 6:53 AM, Anup Tiwari <
>>>>>>>>> anupsdtiwari@gmail.com> wrote:
>>>>>>>>> > Hi Vincent,
>>>>>>>>> >
>>>>>>>>> > I found this link which says Supported Operating Systems are
>>>>>>>>> Ubuntu 14.04,
>>>>>>>>> > 16.04 so does this mean it won't work on centos 7.
>>>>>>>>> >
>>>>>>>>> > Also can you send some link which can help us in tuning as well.
>>>>>>>>> >
>>>>>>>>> > Regards,
>>>>>>>>> > Anup Tiwari
>>>>>>>>> >
>>>>>>>>> > On Tue, Jan 23, 2018 at 7:26 PM, Vincent Tran <
>>>>>>>>> vttran@cloudera.com> wrote:
>>>>>>>>> >>
>>>>>>>>> >> Hi Anup,
>>>>>>>>> >>
>>>>>>>>> >> Is this the link you are looking for?
>>>>>>>>> >>
>>>>>>>>> >> https://impala.apache.org/docs/build/html/topics/impala_inst
>>>>>>>>> all.html
>>>>>>>>> >>
>>>>>>>>> >> On Tue, Jan 23, 2018 at 7:52 AM, Anup Tiwari <
>>>>>>>>> anupsdtiwari@gmail.com>
>>>>>>>>> >> wrote:
>>>>>>>>> >>>
>>>>>>>>> >>> Hi All
>>>>>>>>> >>>
>>>>>>>>> >>> I want to install impala on my production
>>>>>>>>> environment(distributed) :-
>>>>>>>>> >>>
>>>>>>>>> >>> 1. Os Details : CentOS Linux release 7.2.1511 (Core)
>>>>>>>>> >>> 2. Apache Hadoop 2.8.0
>>>>>>>>> >>> 3. Apache Hive 2.1.1
>>>>>>>>> >>>
>>>>>>>>> >>> I didn't found any link on mail, please suggest if it is
>>>>>>>>> possible or not
>>>>>>>>> >>> and if possible then please share some links.
>>>>>>>>> >>>
>>>>>>>>> >>> Thanks in advance!!
>>>>>>>>> >>>
>>>>>>>>> >>> Regards,
>>>>>>>>> >>> Anup Tiwari
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >> --
>>>>>>>>> >> Vincent T. Tran
>>>>>>>>> >> Customer Operations Engineer
>>>>>>>>> >> Cloudera, Inc.
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>
>
>
Re: Install Impala on Apache Hadoop and Hive
Posted by Anup Tiwari <an...@gmail.com>.
Hi,
I have executed below commands
*Build Command :*
${IMPALA_HOME}/buildall.sh -noclean -skiptests -build_shared_libs -format
Please refer trail mail for details or steps.
On 07-Feb-2018 10:51 PM, "Tim Armstrong" <ta...@cloudera.com> wrote:
What command did you run to kick off the build? It looks like you're
missing the HDFS dependency from the expected location, but that should be
automatically downloaded if you run buildall.sh with the default settings.
On Wed, Feb 7, 2018 at 6:30 AM, Anup Tiwari <an...@gmail.com> wrote:
> Hi Vincent,
>
> I have installed all packages/modules mentioned in
> *https://cwiki.apache.org/confluence/display/IMPALA/Impala+Build+Prerequisites*
> <https://cwiki.apache.org/confluence/display/IMPALA/Impala+Build+Prerequisites>
> and now getting below error while building and the one which you have
> mention is for *Ubuntu 14.04* as per docs and i am using centos7 as
> mention in trail mail :-
>
> -- Added shared library dependency zlib: /opt/apache-s/apache-impala/ap
> ache-impala-2.11.0/toolchain/zlib-1.2.8/lib/libz.so
> CMake Error at cmake_modules/FindHDFS.cmake:78 (message):
> HDFS includes and libraries NOT found. HDFS_INCLUDE_DIR-NOTFOUND, )
> Call Stack (most recent call first):
> CMakeLists.txt:183 (find_package)
> -- Configuring incomplete, errors occurred!
> See also "/opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles
> /CMakeOutput.log".
> See also "/opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles
> /CMakeError.log".
> Error in /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/make_impala.sh
> at line 161: cmake . ${CMAKE_ARGS[@]}
>
>
> *OUTPUT of CMakeError.log :-*
>
> Determining if the pthread_create exist failed with the following output:
> Change Dir: /opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/
> CMakeTmp
>
> Run Build Command:"/usr/bin/gmake" "cmTC_18658/fast"
> /usr/bin/gmake -f CMakeFiles/cmTC_18658.dir/build.make
> CMakeFiles/cmTC_18658.dir/build
> gmake[1]: Entering directory `/opt/apache-s/apache-impala/a
> pache-impala-2.11.0/CMakeFiles/CMakeTmp'
> Building C object CMakeFiles/cmTC_18658.dir/CheckSymbolExists.c.o
> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
> -o CMakeFiles/cmTC_18658.dir/CheckSymbolExists.c.o -c
> /opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/
> CMakeTmp/CheckSymbolExists.c
> Linking C executable cmTC_18658
> /opt/apache-s/apache-impala/native-toolchain/build/cmake-3.8.2-p1/bin/cmake
> -E cmake_link_script CMakeFiles/cmTC_18658.dir/link.txt --verbose=1
> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
> -rdynamic CMakeFiles/cmTC_18658.dir/CheckSymbolExists.c.o -o cmTC_18658
> CMakeFiles/cmTC_18658.dir/CheckSymbolExists.c.o: In function `main':
> CheckSymbolExists.c:(.text+0x16): undefined reference to `pthread_create'
> collect2: error: ld returned 1 exit status
> gmake[1]: *** [cmTC_18658] Error 1
> gmake[1]: Leaving directory `/opt/apache-s/apache-impala/a
> pache-impala-2.11.0/CMakeFiles/CMakeTmp'
> gmake: *** [cmTC_18658/fast] Error 2
>
> File /opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/
> CMakeTmp/CheckSymbolExists.c:
> /* */
> #include <pthread.h>
>
> int main(int argc, char** argv)
> {
> (void)argv;
> #ifndef pthread_create
> return ((int*)(&pthread_create))[argc];
> #else
> (void)argc;
> return 0;
> #endif
> }
>
> Determining if the function pthread_create exists in the pthreads failed
> with the following output:
> Change Dir: /opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/
> CMakeTmp
>
> Run Build Command:"/usr/bin/gmake" "cmTC_81779/fast"
> /usr/bin/gmake -f CMakeFiles/cmTC_81779.dir/build.make
> CMakeFiles/cmTC_81779.dir/build
> gmake[1]: Entering directory `/opt/apache-s/apache-impala/a
> pache-impala-2.11.0/CMakeFiles/CMakeTmp'
> Building C object CMakeFiles/cmTC_81779.dir/CheckFunctionExists.c.o
> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
> -DCHECK_FUNCTION_EXISTS=pthread_create -o CMakeFiles/cmTC_81779.dir/CheckFunctionExists.c.o
> -c /opt/apache-s/apache-impala/native-toolchain/build/cmake-3.
> 8.2-p1/share/cmake-3.8/Modules/CheckFunctionExists.c
> Linking C executable cmTC_81779
> /opt/apache-s/apache-impala/native-toolchain/build/cmake-3.8.2-p1/bin/cmake
> -E cmake_link_script CMakeFiles/cmTC_81779.dir/link.txt --verbose=1
> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
> -DCHECK_FUNCTION_EXISTS=pthread_create -rdynamic
> CMakeFiles/cmTC_81779.dir/CheckFunctionExists.c.o -o cmTC_81779
> -lpthreads
> /usr/bin/ld: cannot find -lpthreads
> collect2: error: ld returned 1 exit status
> gmake[1]: *** [cmTC_81779] Error 1
> gmake[1]: Leaving directory `/opt/apache-s/apache-impala/a
> pache-impala-2.11.0/CMakeFiles/CMakeTmp'
> gmake: *** [cmTC_81779/fast] Error 2
>
>
>
>
>
> Regards,
> Anup Tiwari
>
> On Tue, Feb 6, 2018 at 6:07 PM, Vincent Tran <vt...@cloudera.com> wrote:
>
>> Hey Anup,
>>
>> Lools like ypu may be missing some prereq packages. This particular one
>> is libsasl2-dev
>>
>>
>>
>> On Feb 6, 2018 5:12 AM, "Anup Tiwari" <an...@gmail.com> wrote:
>>
>>> I found this link http://mail-archives.apache.or
>>> g/mod_mbox/impala-dev/201708.mbox/%3CCADZba-Td-wXRqLRxctvFM2
>>> mY44cYJhkX-JSs4=uPQCnFDhoDyA@mail.gmail.com%3E which says
>>>
>>> Do the following:
>>>
>>> rm -rf ${IMPALA_TOOLCHAIN}/gflags-${IMPALA_GFLAGS_VERSION}
>>> . bin/impala-config.sh
>>> bin/bootstrap_toolchain.py
>>>
>>> But post executing bootstrap command ; i am getting below error :-
>>>
>>> Traceback (most recent call last):
>>> File "/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/bootstrap_virtualenv.py", line 367, in <module>
>>> if install_compiled_deps_if_possible():
>>> File "/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/bootstrap_virtualenv.py", line 216, in install_compiled_deps_if_possible
>>> exec_pip_install(["-r", COMPILED_REQS_PATH], cc=cc, env=env)
>>> File "/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/bootstrap_virtualenv.py", line 145, in exec_pip_install
>>> "file://%s" % urllib.pathname2url(os.path.abspath(DEPS_DIR))] + args, env=env)
>>> File "/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/bootstrap_virtualenv.py", line 96, in exec_cmd
>>> % (args, output))
>>> Exception: Command returned non-zero status
>>> Command: ['/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/env/bin/python', '/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/env/bin/pip', 'install', '--no-binary', '--no-index', '--no-cache-dir', '--find-links', 'file:///opt/apache-s/apache-impala/apache-impala-2.11.0/infra/python/deps', '-r', '/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt']
>>> Output: You are using pip version 7.1.0, however version 9.0.1 is available.
>>> You should consider upgrading via the 'pip install --upgrade pip' command.
>>> Requirement already satisfied (use --upgrade to upgrade): argparse==1.4.0 in ./infra/python/env/lib/python2.7/site-packages (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 1))
>>> Requirement already satisfied (use --upgrade to upgrade): Fabric==1.10.2 in ./infra/python/env/lib/python2.7/site-packages (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 2))
>>> Requirement already satisfied (use --upgrade to upgrade): paramiko==1.15.2 in ./infra/python/env/lib/python2.7/site-packages (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 3))
>>> Requirement already satisfied (use --upgrade to upgrade): ecdsa==0.13 in ./infra/python/env/lib/python2.7/site-packages (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 4))
>>> Requirement already satisfied (use --upgrade to upgrade): pycrypto==2.6.1 in ./infra/python/env/lib/python2.7/site-packages (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 5))
>>> Requirement already satisfied (use --upgrade to upgrade): impyla==0.14.0 in ./infra/python/env/lib/python2.7/site-packages (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 6))
>>> Requirement already satisfied (use --upgrade to upgrade): bitarray==0.8.1 in ./infra/python/env/lib/python2.7/site-packages (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 7))
>>> Collecting sasl==0.1.3 (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 8))
>>> Requirement already satisfied (use --upgrade to upgrade): six==1.9.0 in ./infra/python/env/lib/python2.7/site-packages (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 9))
>>> Collecting thrift==0.9.0 (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 10))
>>> Collecting thrift-sasl==0.1.0 (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 11))
>>> Collecting psutil==0.7.1 (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 12))
>>> Collecting Cython==0.23.4 (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 13))
>>> Collecting numpy==1.10.4 (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 14))
>>> Installing collected packages: sasl, thrift, thrift-sasl, psutil, Cython, numpy
>>> Running setup.py install for sasl
>>> Complete output from command /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/env/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-build-MTPxSs/sasl/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-EHfJDn-record/install-record.txt --single-version-externally-managed --compile --install-headers /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/env/include/site/python2.7/sasl:
>>> running install
>>> running build
>>> running build_py
>>> creating build
>>> creating build/lib.linux-x86_64-2.7
>>> creating build/lib.linux-x86_64-2.7/sasl
>>> copying sasl/__init__.py -> build/lib.linux-x86_64-2.7/sasl
>>> copying sasl/saslwrapper.py -> build/lib.linux-x86_64-2.7/sasl
>>> running egg_info
>>> writing sasl.egg-info/PKG-INFO
>>> writing top-level names to sasl.egg-info/top_level.txt
>>> writing dependency_links to sasl.egg-info/dependency_links.txt
>>> warning: manifest_maker: standard file '-c' not found
>>>
>>> reading manifest file 'sasl.egg-info/SOURCES.txt'
>>> reading manifest template 'MANIFEST.in'
>>> writing manifest file 'sasl.egg-info/SOURCES.txt'
>>> running build_ext
>>> building '_saslwrapper' extension
>>> creating build/temp.linux-x86_64-2.7
>>> creating build/temp.linux-x86_64-2.7/sasl
>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -Isasl -I/usr/include/python2.7 -c sasl/saslwrapper.cpp -o build/temp.linux-x86_64-2.7/sasl/saslwrapper.o
>>> sasl/saslwrapper.cpp:21:23: fatal error: sasl/sasl.h: No such file or directory
>>> #include <sasl/sasl.h>
>>> ^
>>> compilation terminated.
>>> error: command '/opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc' failed with exit status 1
>>>
>>> ----------------------------------------
>>> Command "/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/env/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-build-MTPxSs/sasl/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-EHfJDn-record/install-record.txt --single-version-externally-managed --compile --install-headers /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/env/include/site/python2.7/sasl" failed with error code 1 in /tmp/pip-build-MTPxSs/sasl
>>>
>>> Error in /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/impala-python at line 28:
>>>
>>>
>>>
>>> Regards,
>>> Anup Tiwari
>>>
>>> On Tue, Feb 6, 2018 at 2:52 PM, Anup Tiwari <an...@gmail.com>
>>> wrote:
>>>
>>>> Hi Zoltan,
>>>>
>>>> I followed the steps mention in https://cwiki.apache.org/confl
>>>> uence/display/IMPALA/Building+Impala and when i executed below command
>>>> , i got error after some time :-
>>>>
>>>> *Build Command :*
>>>> ${IMPALA_HOME}/buildall.sh -noclean -skiptests -build_shared_libs
>>>> -format
>>>>
>>>>
>>>>
>>>> *Error :-- ----------> Adding thirdparty library cyrus_sasl.
>>>> <----------*CMake Error at CMakeLists.txt:136 (message):
>>>> Library 'cyrus_sasl' has neither shared nor static library files
>>>> Call Stack (most recent call first):
>>>> CMakeLists.txt:227 (IMPALA_ADD_THIRDPARTY_LIB)
>>>>
>>>> -- Configuring incomplete, errors occurred!
>>>> See also "/opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles
>>>> /CMakeOutput.log".
>>>> See also "/opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles
>>>> /CMakeError.log".
>>>> Error in /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/make_impala.sh
>>>> at line 161: cmake . ${CMAKE_ARGS[@]}
>>>>
>>>>
>>>>
>>>>
>>>> *Logs in "CMakeOutput.log" :-*Determining if the function
>>>> pthread_create exists in the pthread passed with the following output:
>>>> Change Dir: /opt/apache-s/apache-impala/ap
>>>> ache-impala-2.11.0/CMakeFiles/CMakeTmp
>>>>
>>>> Run Build Command:"/usr/bin/gmake" "cmTC_ee4f0/fast"
>>>> /usr/bin/gmake -f CMakeFiles/cmTC_ee4f0.dir/build.make
>>>> CMakeFiles/cmTC_ee4f0.dir/build
>>>> gmake[1]: Entering directory `/opt/apache-s/apache-impala/a
>>>> pache-impala-2.11.0/CMakeFiles/CMakeTmp'
>>>> Building C object CMakeFiles/cmTC_ee4f0.dir/CheckFunctionExists.c.o
>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
>>>> -DCHECK_FUNCTION_EXISTS=pthread_create -o
>>>> CMakeFiles/cmTC_ee4f0.dir/CheckFunctionExists.c.o -c
>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/c
>>>> make-3.8.2-p1/share/cmake-3.8/Modules/CheckFunctionExists.c
>>>> Linking C executable cmTC_ee4f0
>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/cmake-3.8.2-p1/bin/cmake
>>>> -E cmake_link_script CMakeFiles/cmTC_ee4f0.dir/link.txt --verbose=1
>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
>>>> -DCHECK_FUNCTION_EXISTS=pthread_create -rdynamic
>>>> CMakeFiles/cmTC_ee4f0.dir/CheckFunctionExists.c.o -o cmTC_ee4f0
>>>> -lpthread
>>>> gmake[1]: Leaving directory `/opt/apache-s/apache-impala/a
>>>> pache-impala-2.11.0/CMakeFiles/CMakeTmp'
>>>>
>>>>
>>>>
>>>> *Logs in "CMakeError.log" :-*Determining if the function
>>>> pthread_create exists in the pthreads failed with the following output:
>>>> Change Dir: /opt/apache-s/apache-impala/ap
>>>> ache-impala-2.11.0/CMakeFiles/CMakeTmp
>>>>
>>>> Run Build Command:"/usr/bin/gmake" "cmTC_18bf3/fast"
>>>> /usr/bin/gmake -f CMakeFiles/cmTC_18bf3.dir/build.make
>>>> CMakeFiles/cmTC_18bf3.dir/build
>>>> gmake[1]: Entering directory `/opt/apache-s/apache-impala/a
>>>> pache-impala-2.11.0/CMakeFiles/CMakeTmp'
>>>> Building C object CMakeFiles/cmTC_18bf3.dir/CheckFunctionExists.c.o
>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
>>>> -DCHECK_FUNCTION_EXISTS=pthread_create -o
>>>> CMakeFiles/cmTC_18bf3.dir/CheckFunctionExists.c.o -c
>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/c
>>>> make-3.8.2-p1/share/cmake-3.8/Modules/CheckFunctionExists.c
>>>> Linking C executable cmTC_18bf3
>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/cmake-3.8.2-p1/bin/cmake
>>>> -E cmake_link_script CMakeFiles/cmTC_18bf3.dir/link.txt --verbose=1
>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
>>>> -DCHECK_FUNCTION_EXISTS=pthread_create -rdynamic
>>>> CMakeFiles/cmTC_18bf3.dir/CheckFunctionExists.c.o -o cmTC_18bf3
>>>> -lpthreads
>>>> /usr/bin/ld: cannot find -lpthreads
>>>> collect2: error: ld returned 1 exit status
>>>> gmake[1]: *** [cmTC_18bf3] Error 1
>>>> gmake[1]: Leaving directory `/opt/apache-s/apache-impala/a
>>>> pache-impala-2.11.0/CMakeFiles/CMakeTmp'
>>>> gmake: *** [cmTC_18bf3/fast] Error 2
>>>>
>>>> Also it seems that this is related to toolchain error so i followed
>>>> this link : https://cwiki.apache.org/confl
>>>> uence/display/IMPALA/Building+native-toolchain+from+scratch+
>>>> and+using+with+Impala where i first builded native-toolchain and then
>>>> followed step mention in "Using custom toolchain with Impala" but while
>>>> creating symbolic link i got error "ln: ‘./gcc-4.9.2’: cannot
>>>> overwrite directory".
>>>>
>>>> @xinran, i guess post building impala, i will have to perform steps
>>>> mentioned by you.. correct me if i am wrong?
>>>>
>>>>
>>>>
>>>> Regards,
>>>> Anup Tiwari
>>>>
>>>> On Mon, Feb 5, 2018 at 11:14 PM, Xinran Yu Tinney <
>>>> yuxinran8011@gmail.com> wrote:
>>>>
>>>>> Hi Anup,
>>>>> Did you also source the config.sh? Run the following before
>>>>> impala-shell.sh:
>>>>>
>>>>> source ~/Impala/bin/impala-config.sh
>>>>>
>>>>> ${IMPALA_HOME}/buildall.sh -notests -build_shared_libs
>>>>> -start_minicluster
>>>>>
>>>>> ${IMPALA_HOME}/bin/start-impala-cluster.py
>>>>>
>>>>>
>>>>> 2018-02-05 10:07 GMT-06:00 Michael Brown <mi...@cloudera.com>:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> Please see:
>>>>>>
>>>>>> https://cwiki.apache.org/confluence/display/IMPALA/Bootstrap
>>>>>> ping+an+Impala+Development+Environment+From+Scratch
>>>>>> https://cwiki.apache.org/confluence/display/IMPALA/Building+Impala
>>>>>>
>>>>>>
>>>>>> On Mon, Feb 5, 2018 at 12:09 AM, Anup Tiwari <an...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi All,
>>>>>>>
>>>>>>> I am new to impala so i am getting errors while i am setting impala.
>>>>>>>
>>>>>>> I have downloaded latest impala release from "
>>>>>>> http://www.apache.org/dyn/closer.cgi?action=download&filena
>>>>>>> me=impala/2.11.0/apache-impala-2.11.0.tar.gz".
>>>>>>> I have extracted this in a folder and followed description mentioned
>>>>>>> in below link :-
>>>>>>>
>>>>>>> https://impala.apache.org/docs/build/html/topics/impala_proc
>>>>>>> esses.html#processes
>>>>>>>
>>>>>>> As mentioned in above link "*To start the Impala state store and
>>>>>>> Impala from the command line or a script, you can either use the service
>>>>>>> command or you can start the daemons directly through the impalad,
>>>>>>> statestored, and catalogd executables.*" So i searched for
>>>>>>> executable and found "start-impalad.sh"(which i think is correct one, let
>>>>>>> me know if i have missed this) but when i am executing this , i am getting
>>>>>>> below *errors* :-
>>>>>>>
>>>>>>> bin/start-impalad.sh: line 89: /admin: No such file or directory
>>>>>>> bin/start-impalad.sh: line 106: /opt/apache-s/apache-impala/ap
>>>>>>> ache-impala-2.11.0/be/build/latest/service/impalad: No such file or
>>>>>>> directory
>>>>>>>
>>>>>>> Also when i executed ./impala-config.sh then i got below *error* :-
>>>>>>>
>>>>>>> Unable to find the lsb_release command. Please ensure it is
>>>>>>> available in your PATH.
>>>>>>> ./impala-config.sh: line 265: return: can only `return' from a
>>>>>>> function or sourced script
>>>>>>> lsb_release command failed, output was: ./impala-config.sh: line
>>>>>>> 267: lsb_release: command not found
>>>>>>> ./impala-config.sh: line 270: return: can only `return' from a
>>>>>>> function or sourced script
>>>>>>>
>>>>>>>
>>>>>>> *Questions :-*
>>>>>>> 1. When we setup Hive/Hadoop/Drill etc then we have certain steps
>>>>>>> for setting HIVE_HOME/HADOOP_HOME/DRILL_HOME and also we specify
>>>>>>> certain setting in conf folder. Do we have similar documentation which can
>>>>>>> specify step by step setup.
>>>>>>>
>>>>>>>
>>>>>>> Regards,
>>>>>>> Anup Tiwari
>>>>>>>
>>>>>>> On Tue, Jan 23, 2018 at 8:56 PM, Jim Apple <jb...@cloudera.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Ubuntu 14.04 and 16.04 are the OSes most Impala developers use, but
>>>>>>>> many Impala users use CentOS 7.
>>>>>>>>
>>>>>>>> Here is part of the docs on tuning:
>>>>>>>> https://impala.apache.org/docs/build/html/topics/impala_perf
>>>>>>>> ormance.html
>>>>>>>>
>>>>>>>> Here is some tuning info that was produced by Cloudera:
>>>>>>>> http://blog.cloudera.com/wp-content/uploads/2017/03/Impala-C
>>>>>>>> ookbook-01-20173.pdf
>>>>>>>>
>>>>>>>> On Tue, Jan 23, 2018 at 6:53 AM, Anup Tiwari <
>>>>>>>> anupsdtiwari@gmail.com> wrote:
>>>>>>>> > Hi Vincent,
>>>>>>>> >
>>>>>>>> > I found this link which says Supported Operating Systems are
>>>>>>>> Ubuntu 14.04,
>>>>>>>> > 16.04 so does this mean it won't work on centos 7.
>>>>>>>> >
>>>>>>>> > Also can you send some link which can help us in tuning as well.
>>>>>>>> >
>>>>>>>> > Regards,
>>>>>>>> > Anup Tiwari
>>>>>>>> >
>>>>>>>> > On Tue, Jan 23, 2018 at 7:26 PM, Vincent Tran <
>>>>>>>> vttran@cloudera.com> wrote:
>>>>>>>> >>
>>>>>>>> >> Hi Anup,
>>>>>>>> >>
>>>>>>>> >> Is this the link you are looking for?
>>>>>>>> >>
>>>>>>>> >> https://impala.apache.org/docs/build/html/topics/impala_inst
>>>>>>>> all.html
>>>>>>>> >>
>>>>>>>> >> On Tue, Jan 23, 2018 at 7:52 AM, Anup Tiwari <
>>>>>>>> anupsdtiwari@gmail.com>
>>>>>>>> >> wrote:
>>>>>>>> >>>
>>>>>>>> >>> Hi All
>>>>>>>> >>>
>>>>>>>> >>> I want to install impala on my production
>>>>>>>> environment(distributed) :-
>>>>>>>> >>>
>>>>>>>> >>> 1. Os Details : CentOS Linux release 7.2.1511 (Core)
>>>>>>>> >>> 2. Apache Hadoop 2.8.0
>>>>>>>> >>> 3. Apache Hive 2.1.1
>>>>>>>> >>>
>>>>>>>> >>> I didn't found any link on mail, please suggest if it is
>>>>>>>> possible or not
>>>>>>>> >>> and if possible then please share some links.
>>>>>>>> >>>
>>>>>>>> >>> Thanks in advance!!
>>>>>>>> >>>
>>>>>>>> >>> Regards,
>>>>>>>> >>> Anup Tiwari
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >> --
>>>>>>>> >> Vincent T. Tran
>>>>>>>> >> Customer Operations Engineer
>>>>>>>> >> Cloudera, Inc.
>>>>>>>> >
>>>>>>>> >
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>
Re: Install Impala on Apache Hadoop and Hive
Posted by Tim Armstrong <ta...@cloudera.com>.
What command did you run to kick off the build? It looks like you're
missing the HDFS dependency from the expected location, but that should be
automatically downloaded if you run buildall.sh with the default settings.
On Wed, Feb 7, 2018 at 6:30 AM, Anup Tiwari <an...@gmail.com> wrote:
> Hi Vincent,
>
> I have installed all packages/modules mentioned in
> *https://cwiki.apache.org/confluence/display/IMPALA/Impala+Build+Prerequisites*
> <https://cwiki.apache.org/confluence/display/IMPALA/Impala+Build+Prerequisites>
> and now getting below error while building and the one which you have
> mention is for *Ubuntu 14.04* as per docs and i am using centos7 as
> mention in trail mail :-
>
> -- Added shared library dependency zlib: /opt/apache-s/apache-impala/
> apache-impala-2.11.0/toolchain/zlib-1.2.8/lib/libz.so
> CMake Error at cmake_modules/FindHDFS.cmake:78 (message):
> HDFS includes and libraries NOT found. HDFS_INCLUDE_DIR-NOTFOUND, )
> Call Stack (most recent call first):
> CMakeLists.txt:183 (find_package)
> -- Configuring incomplete, errors occurred!
> See also "/opt/apache-s/apache-impala/apache-impala-2.11.0/
> CMakeFiles/CMakeOutput.log".
> See also "/opt/apache-s/apache-impala/apache-impala-2.11.0/
> CMakeFiles/CMakeError.log".
> Error in /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/make_impala.sh
> at line 161: cmake . ${CMAKE_ARGS[@]}
>
>
> *OUTPUT of CMakeError.log :-*
>
> Determining if the pthread_create exist failed with the following output:
> Change Dir: /opt/apache-s/apache-impala/apache-impala-2.11.0/
> CMakeFiles/CMakeTmp
>
> Run Build Command:"/usr/bin/gmake" "cmTC_18658/fast"
> /usr/bin/gmake -f CMakeFiles/cmTC_18658.dir/build.make
> CMakeFiles/cmTC_18658.dir/build
> gmake[1]: Entering directory `/opt/apache-s/apache-impala/
> apache-impala-2.11.0/CMakeFiles/CMakeTmp'
> Building C object CMakeFiles/cmTC_18658.dir/CheckSymbolExists.c.o
> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
> -o CMakeFiles/cmTC_18658.dir/CheckSymbolExists.c.o -c
> /opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/CMakeTmp/
> CheckSymbolExists.c
> Linking C executable cmTC_18658
> /opt/apache-s/apache-impala/native-toolchain/build/cmake-3.8.2-p1/bin/cmake
> -E cmake_link_script CMakeFiles/cmTC_18658.dir/link.txt --verbose=1
> /opt/apache-s/apache-impala/apache-impala-2.11.0/
> toolchain/gcc-4.9.2/bin/gcc -rdynamic CMakeFiles/cmTC_18658.dir/CheckSymbolExists.c.o
> -o cmTC_18658
> CMakeFiles/cmTC_18658.dir/CheckSymbolExists.c.o: In function `main':
> CheckSymbolExists.c:(.text+0x16): undefined reference to `pthread_create'
> collect2: error: ld returned 1 exit status
> gmake[1]: *** [cmTC_18658] Error 1
> gmake[1]: Leaving directory `/opt/apache-s/apache-impala/
> apache-impala-2.11.0/CMakeFiles/CMakeTmp'
> gmake: *** [cmTC_18658/fast] Error 2
>
> File /opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/CMakeTmp/
> CheckSymbolExists.c:
> /* */
> #include <pthread.h>
>
> int main(int argc, char** argv)
> {
> (void)argv;
> #ifndef pthread_create
> return ((int*)(&pthread_create))[argc];
> #else
> (void)argc;
> return 0;
> #endif
> }
>
> Determining if the function pthread_create exists in the pthreads failed
> with the following output:
> Change Dir: /opt/apache-s/apache-impala/apache-impala-2.11.0/
> CMakeFiles/CMakeTmp
>
> Run Build Command:"/usr/bin/gmake" "cmTC_81779/fast"
> /usr/bin/gmake -f CMakeFiles/cmTC_81779.dir/build.make
> CMakeFiles/cmTC_81779.dir/build
> gmake[1]: Entering directory `/opt/apache-s/apache-impala/
> apache-impala-2.11.0/CMakeFiles/CMakeTmp'
> Building C object CMakeFiles/cmTC_81779.dir/CheckFunctionExists.c.o
> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
> -DCHECK_FUNCTION_EXISTS=pthread_create -o CMakeFiles/cmTC_81779.dir/CheckFunctionExists.c.o
> -c /opt/apache-s/apache-impala/native-toolchain/build/cmake-
> 3.8.2-p1/share/cmake-3.8/Modules/CheckFunctionExists.c
> Linking C executable cmTC_81779
> /opt/apache-s/apache-impala/native-toolchain/build/cmake-3.8.2-p1/bin/cmake
> -E cmake_link_script CMakeFiles/cmTC_81779.dir/link.txt --verbose=1
> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
> -DCHECK_FUNCTION_EXISTS=pthread_create -rdynamic
> CMakeFiles/cmTC_81779.dir/CheckFunctionExists.c.o -o cmTC_81779
> -lpthreads
> /usr/bin/ld: cannot find -lpthreads
> collect2: error: ld returned 1 exit status
> gmake[1]: *** [cmTC_81779] Error 1
> gmake[1]: Leaving directory `/opt/apache-s/apache-impala/
> apache-impala-2.11.0/CMakeFiles/CMakeTmp'
> gmake: *** [cmTC_81779/fast] Error 2
>
>
>
>
>
> Regards,
> Anup Tiwari
>
> On Tue, Feb 6, 2018 at 6:07 PM, Vincent Tran <vt...@cloudera.com> wrote:
>
>> Hey Anup,
>>
>> Lools like ypu may be missing some prereq packages. This particular one
>> is libsasl2-dev
>>
>>
>>
>> On Feb 6, 2018 5:12 AM, "Anup Tiwari" <an...@gmail.com> wrote:
>>
>>> I found this link http://mail-archives.apache.or
>>> g/mod_mbox/impala-dev/201708.mbox/%3CCADZba-Td-wXRqLRxctvFM2
>>> mY44cYJhkX-JSs4=uPQCnFDhoDyA@mail.gmail.com%3E which says
>>>
>>> Do the following:
>>>
>>> rm -rf ${IMPALA_TOOLCHAIN}/gflags-${IMPALA_GFLAGS_VERSION}
>>> . bin/impala-config.sh
>>> bin/bootstrap_toolchain.py
>>>
>>> But post executing bootstrap command ; i am getting below error :-
>>>
>>> Traceback (most recent call last):
>>> File "/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/bootstrap_virtualenv.py", line 367, in <module>
>>> if install_compiled_deps_if_possible():
>>> File "/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/bootstrap_virtualenv.py", line 216, in install_compiled_deps_if_possible
>>> exec_pip_install(["-r", COMPILED_REQS_PATH], cc=cc, env=env)
>>> File "/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/bootstrap_virtualenv.py", line 145, in exec_pip_install
>>> "file://%s" % urllib.pathname2url(os.path.abspath(DEPS_DIR))] + args, env=env)
>>> File "/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/bootstrap_virtualenv.py", line 96, in exec_cmd
>>> % (args, output))
>>> Exception: Command returned non-zero status
>>> Command: ['/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/env/bin/python', '/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/env/bin/pip', 'install', '--no-binary', '--no-index', '--no-cache-dir', '--find-links', 'file:///opt/apache-s/apache-impala/apache-impala-2.11.0/infra/python/deps', '-r', '/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt']
>>> Output: You are using pip version 7.1.0, however version 9.0.1 is available.
>>> You should consider upgrading via the 'pip install --upgrade pip' command.
>>> Requirement already satisfied (use --upgrade to upgrade): argparse==1.4.0 in ./infra/python/env/lib/python2.7/site-packages (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 1))
>>> Requirement already satisfied (use --upgrade to upgrade): Fabric==1.10.2 in ./infra/python/env/lib/python2.7/site-packages (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 2))
>>> Requirement already satisfied (use --upgrade to upgrade): paramiko==1.15.2 in ./infra/python/env/lib/python2.7/site-packages (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 3))
>>> Requirement already satisfied (use --upgrade to upgrade): ecdsa==0.13 in ./infra/python/env/lib/python2.7/site-packages (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 4))
>>> Requirement already satisfied (use --upgrade to upgrade): pycrypto==2.6.1 in ./infra/python/env/lib/python2.7/site-packages (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 5))
>>> Requirement already satisfied (use --upgrade to upgrade): impyla==0.14.0 in ./infra/python/env/lib/python2.7/site-packages (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 6))
>>> Requirement already satisfied (use --upgrade to upgrade): bitarray==0.8.1 in ./infra/python/env/lib/python2.7/site-packages (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 7))
>>> Collecting sasl==0.1.3 (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 8))
>>> Requirement already satisfied (use --upgrade to upgrade): six==1.9.0 in ./infra/python/env/lib/python2.7/site-packages (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 9))
>>> Collecting thrift==0.9.0 (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 10))
>>> Collecting thrift-sasl==0.1.0 (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 11))
>>> Collecting psutil==0.7.1 (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 12))
>>> Collecting Cython==0.23.4 (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 13))
>>> Collecting numpy==1.10.4 (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 14))
>>> Installing collected packages: sasl, thrift, thrift-sasl, psutil, Cython, numpy
>>> Running setup.py install for sasl
>>> Complete output from command /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/env/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-build-MTPxSs/sasl/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-EHfJDn-record/install-record.txt --single-version-externally-managed --compile --install-headers /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/env/include/site/python2.7/sasl:
>>> running install
>>> running build
>>> running build_py
>>> creating build
>>> creating build/lib.linux-x86_64-2.7
>>> creating build/lib.linux-x86_64-2.7/sasl
>>> copying sasl/__init__.py -> build/lib.linux-x86_64-2.7/sasl
>>> copying sasl/saslwrapper.py -> build/lib.linux-x86_64-2.7/sasl
>>> running egg_info
>>> writing sasl.egg-info/PKG-INFO
>>> writing top-level names to sasl.egg-info/top_level.txt
>>> writing dependency_links to sasl.egg-info/dependency_links.txt
>>> warning: manifest_maker: standard file '-c' not found
>>>
>>> reading manifest file 'sasl.egg-info/SOURCES.txt'
>>> reading manifest template 'MANIFEST.in'
>>> writing manifest file 'sasl.egg-info/SOURCES.txt'
>>> running build_ext
>>> building '_saslwrapper' extension
>>> creating build/temp.linux-x86_64-2.7
>>> creating build/temp.linux-x86_64-2.7/sasl
>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -Isasl -I/usr/include/python2.7 -c sasl/saslwrapper.cpp -o build/temp.linux-x86_64-2.7/sasl/saslwrapper.o
>>> sasl/saslwrapper.cpp:21:23: fatal error: sasl/sasl.h: No such file or directory
>>> #include <sasl/sasl.h>
>>> ^
>>> compilation terminated.
>>> error: command '/opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc' failed with exit status 1
>>>
>>> ----------------------------------------
>>> Command "/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/env/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-build-MTPxSs/sasl/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-EHfJDn-record/install-record.txt --single-version-externally-managed --compile --install-headers /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/env/include/site/python2.7/sasl" failed with error code 1 in /tmp/pip-build-MTPxSs/sasl
>>>
>>> Error in /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/impala-python at line 28:
>>>
>>>
>>>
>>> Regards,
>>> Anup Tiwari
>>>
>>> On Tue, Feb 6, 2018 at 2:52 PM, Anup Tiwari <an...@gmail.com>
>>> wrote:
>>>
>>>> Hi Zoltan,
>>>>
>>>> I followed the steps mention in https://cwiki.apache.org/confl
>>>> uence/display/IMPALA/Building+Impala and when i executed below command
>>>> , i got error after some time :-
>>>>
>>>> *Build Command :*
>>>> ${IMPALA_HOME}/buildall.sh -noclean -skiptests -build_shared_libs
>>>> -format
>>>>
>>>>
>>>>
>>>> *Error :-- ----------> Adding thirdparty library cyrus_sasl.
>>>> <----------*CMake Error at CMakeLists.txt:136 (message):
>>>> Library 'cyrus_sasl' has neither shared nor static library files
>>>> Call Stack (most recent call first):
>>>> CMakeLists.txt:227 (IMPALA_ADD_THIRDPARTY_LIB)
>>>>
>>>> -- Configuring incomplete, errors occurred!
>>>> See also "/opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles
>>>> /CMakeOutput.log".
>>>> See also "/opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles
>>>> /CMakeError.log".
>>>> Error in /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/make_impala.sh
>>>> at line 161: cmake . ${CMAKE_ARGS[@]}
>>>>
>>>>
>>>>
>>>>
>>>> *Logs in "CMakeOutput.log" :-*Determining if the function
>>>> pthread_create exists in the pthread passed with the following output:
>>>> Change Dir: /opt/apache-s/apache-impala/ap
>>>> ache-impala-2.11.0/CMakeFiles/CMakeTmp
>>>>
>>>> Run Build Command:"/usr/bin/gmake" "cmTC_ee4f0/fast"
>>>> /usr/bin/gmake -f CMakeFiles/cmTC_ee4f0.dir/build.make
>>>> CMakeFiles/cmTC_ee4f0.dir/build
>>>> gmake[1]: Entering directory `/opt/apache-s/apache-impala/a
>>>> pache-impala-2.11.0/CMakeFiles/CMakeTmp'
>>>> Building C object CMakeFiles/cmTC_ee4f0.dir/CheckFunctionExists.c.o
>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
>>>> -DCHECK_FUNCTION_EXISTS=pthread_create -o
>>>> CMakeFiles/cmTC_ee4f0.dir/CheckFunctionExists.c.o -c
>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/c
>>>> make-3.8.2-p1/share/cmake-3.8/Modules/CheckFunctionExists.c
>>>> Linking C executable cmTC_ee4f0
>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/cmake-3.8.2-p1/bin/cmake
>>>> -E cmake_link_script CMakeFiles/cmTC_ee4f0.dir/link.txt --verbose=1
>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
>>>> -DCHECK_FUNCTION_EXISTS=pthread_create -rdynamic
>>>> CMakeFiles/cmTC_ee4f0.dir/CheckFunctionExists.c.o -o cmTC_ee4f0
>>>> -lpthread
>>>> gmake[1]: Leaving directory `/opt/apache-s/apache-impala/a
>>>> pache-impala-2.11.0/CMakeFiles/CMakeTmp'
>>>>
>>>>
>>>>
>>>> *Logs in "CMakeError.log" :-*Determining if the function
>>>> pthread_create exists in the pthreads failed with the following output:
>>>> Change Dir: /opt/apache-s/apache-impala/ap
>>>> ache-impala-2.11.0/CMakeFiles/CMakeTmp
>>>>
>>>> Run Build Command:"/usr/bin/gmake" "cmTC_18bf3/fast"
>>>> /usr/bin/gmake -f CMakeFiles/cmTC_18bf3.dir/build.make
>>>> CMakeFiles/cmTC_18bf3.dir/build
>>>> gmake[1]: Entering directory `/opt/apache-s/apache-impala/a
>>>> pache-impala-2.11.0/CMakeFiles/CMakeTmp'
>>>> Building C object CMakeFiles/cmTC_18bf3.dir/CheckFunctionExists.c.o
>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
>>>> -DCHECK_FUNCTION_EXISTS=pthread_create -o
>>>> CMakeFiles/cmTC_18bf3.dir/CheckFunctionExists.c.o -c
>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/c
>>>> make-3.8.2-p1/share/cmake-3.8/Modules/CheckFunctionExists.c
>>>> Linking C executable cmTC_18bf3
>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/cmake-3.8.2-p1/bin/cmake
>>>> -E cmake_link_script CMakeFiles/cmTC_18bf3.dir/link.txt --verbose=1
>>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
>>>> -DCHECK_FUNCTION_EXISTS=pthread_create -rdynamic
>>>> CMakeFiles/cmTC_18bf3.dir/CheckFunctionExists.c.o -o cmTC_18bf3
>>>> -lpthreads
>>>> /usr/bin/ld: cannot find -lpthreads
>>>> collect2: error: ld returned 1 exit status
>>>> gmake[1]: *** [cmTC_18bf3] Error 1
>>>> gmake[1]: Leaving directory `/opt/apache-s/apache-impala/a
>>>> pache-impala-2.11.0/CMakeFiles/CMakeTmp'
>>>> gmake: *** [cmTC_18bf3/fast] Error 2
>>>>
>>>> Also it seems that this is related to toolchain error so i followed
>>>> this link : https://cwiki.apache.org/confl
>>>> uence/display/IMPALA/Building+native-toolchain+from+scratch+
>>>> and+using+with+Impala where i first builded native-toolchain and then
>>>> followed step mention in "Using custom toolchain with Impala" but while
>>>> creating symbolic link i got error "ln: ‘./gcc-4.9.2’: cannot
>>>> overwrite directory".
>>>>
>>>> @xinran, i guess post building impala, i will have to perform steps
>>>> mentioned by you.. correct me if i am wrong?
>>>>
>>>>
>>>>
>>>> Regards,
>>>> Anup Tiwari
>>>>
>>>> On Mon, Feb 5, 2018 at 11:14 PM, Xinran Yu Tinney <
>>>> yuxinran8011@gmail.com> wrote:
>>>>
>>>>> Hi Anup,
>>>>> Did you also source the config.sh? Run the following before
>>>>> impala-shell.sh:
>>>>>
>>>>> source ~/Impala/bin/impala-config.sh
>>>>>
>>>>> ${IMPALA_HOME}/buildall.sh -notests -build_shared_libs
>>>>> -start_minicluster
>>>>>
>>>>> ${IMPALA_HOME}/bin/start-impala-cluster.py
>>>>>
>>>>>
>>>>> 2018-02-05 10:07 GMT-06:00 Michael Brown <mi...@cloudera.com>:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> Please see:
>>>>>>
>>>>>> https://cwiki.apache.org/confluence/display/IMPALA/Bootstrap
>>>>>> ping+an+Impala+Development+Environment+From+Scratch
>>>>>> https://cwiki.apache.org/confluence/display/IMPALA/Building+Impala
>>>>>>
>>>>>>
>>>>>> On Mon, Feb 5, 2018 at 12:09 AM, Anup Tiwari <an...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi All,
>>>>>>>
>>>>>>> I am new to impala so i am getting errors while i am setting impala.
>>>>>>>
>>>>>>> I have downloaded latest impala release from "
>>>>>>> http://www.apache.org/dyn/closer.cgi?action=download&filena
>>>>>>> me=impala/2.11.0/apache-impala-2.11.0.tar.gz".
>>>>>>> I have extracted this in a folder and followed description mentioned
>>>>>>> in below link :-
>>>>>>>
>>>>>>> https://impala.apache.org/docs/build/html/topics/impala_proc
>>>>>>> esses.html#processes
>>>>>>>
>>>>>>> As mentioned in above link "*To start the Impala state store and
>>>>>>> Impala from the command line or a script, you can either use the service
>>>>>>> command or you can start the daemons directly through the impalad,
>>>>>>> statestored, and catalogd executables.*" So i searched for
>>>>>>> executable and found "start-impalad.sh"(which i think is correct one, let
>>>>>>> me know if i have missed this) but when i am executing this , i am getting
>>>>>>> below *errors* :-
>>>>>>>
>>>>>>> bin/start-impalad.sh: line 89: /admin: No such file or directory
>>>>>>> bin/start-impalad.sh: line 106: /opt/apache-s/apache-impala/ap
>>>>>>> ache-impala-2.11.0/be/build/latest/service/impalad: No such file or
>>>>>>> directory
>>>>>>>
>>>>>>> Also when i executed ./impala-config.sh then i got below *error* :-
>>>>>>>
>>>>>>> Unable to find the lsb_release command. Please ensure it is
>>>>>>> available in your PATH.
>>>>>>> ./impala-config.sh: line 265: return: can only `return' from a
>>>>>>> function or sourced script
>>>>>>> lsb_release command failed, output was: ./impala-config.sh: line
>>>>>>> 267: lsb_release: command not found
>>>>>>> ./impala-config.sh: line 270: return: can only `return' from a
>>>>>>> function or sourced script
>>>>>>>
>>>>>>>
>>>>>>> *Questions :-*
>>>>>>> 1. When we setup Hive/Hadoop/Drill etc then we have certain steps
>>>>>>> for setting HIVE_HOME/HADOOP_HOME/DRILL_HOME and also we specify
>>>>>>> certain setting in conf folder. Do we have similar documentation which can
>>>>>>> specify step by step setup.
>>>>>>>
>>>>>>>
>>>>>>> Regards,
>>>>>>> Anup Tiwari
>>>>>>>
>>>>>>> On Tue, Jan 23, 2018 at 8:56 PM, Jim Apple <jb...@cloudera.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Ubuntu 14.04 and 16.04 are the OSes most Impala developers use, but
>>>>>>>> many Impala users use CentOS 7.
>>>>>>>>
>>>>>>>> Here is part of the docs on tuning:
>>>>>>>> https://impala.apache.org/docs/build/html/topics/impala_perf
>>>>>>>> ormance.html
>>>>>>>>
>>>>>>>> Here is some tuning info that was produced by Cloudera:
>>>>>>>> http://blog.cloudera.com/wp-content/uploads/2017/03/Impala-C
>>>>>>>> ookbook-01-20173.pdf
>>>>>>>>
>>>>>>>> On Tue, Jan 23, 2018 at 6:53 AM, Anup Tiwari <
>>>>>>>> anupsdtiwari@gmail.com> wrote:
>>>>>>>> > Hi Vincent,
>>>>>>>> >
>>>>>>>> > I found this link which says Supported Operating Systems are
>>>>>>>> Ubuntu 14.04,
>>>>>>>> > 16.04 so does this mean it won't work on centos 7.
>>>>>>>> >
>>>>>>>> > Also can you send some link which can help us in tuning as well.
>>>>>>>> >
>>>>>>>> > Regards,
>>>>>>>> > Anup Tiwari
>>>>>>>> >
>>>>>>>> > On Tue, Jan 23, 2018 at 7:26 PM, Vincent Tran <
>>>>>>>> vttran@cloudera.com> wrote:
>>>>>>>> >>
>>>>>>>> >> Hi Anup,
>>>>>>>> >>
>>>>>>>> >> Is this the link you are looking for?
>>>>>>>> >>
>>>>>>>> >> https://impala.apache.org/docs/build/html/topics/impala_inst
>>>>>>>> all.html
>>>>>>>> >>
>>>>>>>> >> On Tue, Jan 23, 2018 at 7:52 AM, Anup Tiwari <
>>>>>>>> anupsdtiwari@gmail.com>
>>>>>>>> >> wrote:
>>>>>>>> >>>
>>>>>>>> >>> Hi All
>>>>>>>> >>>
>>>>>>>> >>> I want to install impala on my production
>>>>>>>> environment(distributed) :-
>>>>>>>> >>>
>>>>>>>> >>> 1. Os Details : CentOS Linux release 7.2.1511 (Core)
>>>>>>>> >>> 2. Apache Hadoop 2.8.0
>>>>>>>> >>> 3. Apache Hive 2.1.1
>>>>>>>> >>>
>>>>>>>> >>> I didn't found any link on mail, please suggest if it is
>>>>>>>> possible or not
>>>>>>>> >>> and if possible then please share some links.
>>>>>>>> >>>
>>>>>>>> >>> Thanks in advance!!
>>>>>>>> >>>
>>>>>>>> >>> Regards,
>>>>>>>> >>> Anup Tiwari
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >> --
>>>>>>>> >> Vincent T. Tran
>>>>>>>> >> Customer Operations Engineer
>>>>>>>> >> Cloudera, Inc.
>>>>>>>> >
>>>>>>>> >
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>
Re: Install Impala on Apache Hadoop and Hive
Posted by Anup Tiwari <an...@gmail.com>.
Hi Vincent,
I have installed all packages/modules mentioned in
*https://cwiki.apache.org/confluence/display/IMPALA/Impala+Build+Prerequisites*
<https://cwiki.apache.org/confluence/display/IMPALA/Impala+Build+Prerequisites>
and now getting below error while building and the one which you have
mention is for *Ubuntu 14.04* as per docs and i am using centos7 as mention
in trail mail :-
-- Added shared library dependency zlib:
/opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/zlib-1.2.8/lib/libz.so
CMake Error at cmake_modules/FindHDFS.cmake:78 (message):
HDFS includes and libraries NOT found. HDFS_INCLUDE_DIR-NOTFOUND, )
Call Stack (most recent call first):
CMakeLists.txt:183 (find_package)
-- Configuring incomplete, errors occurred!
See also
"/opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/CMakeOutput.log".
See also
"/opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/CMakeError.log".
Error in
/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/make_impala.sh at line
161: cmake . ${CMAKE_ARGS[@]}
*OUTPUT of CMakeError.log :-*
Determining if the pthread_create exist failed with the following output:
Change Dir:
/opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/CMakeTmp
Run Build Command:"/usr/bin/gmake" "cmTC_18658/fast"
/usr/bin/gmake -f CMakeFiles/cmTC_18658.dir/build.make
CMakeFiles/cmTC_18658.dir/build
gmake[1]: Entering directory
`/opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/CMakeTmp'
Building C object CMakeFiles/cmTC_18658.dir/CheckSymbolExists.c.o
/opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
-o CMakeFiles/cmTC_18658.dir/CheckSymbolExists.c.o -c
/opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/CMakeTmp/CheckSymbolExists.c
Linking C executable cmTC_18658
/opt/apache-s/apache-impala/native-toolchain/build/cmake-3.8.2-p1/bin/cmake
-E cmake_link_script CMakeFiles/cmTC_18658.dir/link.txt --verbose=1
/opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
-rdynamic CMakeFiles/cmTC_18658.dir/CheckSymbolExists.c.o -o cmTC_18658
CMakeFiles/cmTC_18658.dir/CheckSymbolExists.c.o: In function `main':
CheckSymbolExists.c:(.text+0x16): undefined reference to `pthread_create'
collect2: error: ld returned 1 exit status
gmake[1]: *** [cmTC_18658] Error 1
gmake[1]: Leaving directory
`/opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/CMakeTmp'
gmake: *** [cmTC_18658/fast] Error 2
File
/opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/CMakeTmp/CheckSymbolExists.c:
/* */
#include <pthread.h>
int main(int argc, char** argv)
{
(void)argv;
#ifndef pthread_create
return ((int*)(&pthread_create))[argc];
#else
(void)argc;
return 0;
#endif
}
Determining if the function pthread_create exists in the pthreads failed
with the following output:
Change Dir:
/opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/CMakeTmp
Run Build Command:"/usr/bin/gmake" "cmTC_81779/fast"
/usr/bin/gmake -f CMakeFiles/cmTC_81779.dir/build.make
CMakeFiles/cmTC_81779.dir/build
gmake[1]: Entering directory
`/opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/CMakeTmp'
Building C object CMakeFiles/cmTC_81779.dir/CheckFunctionExists.c.o
/opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
-DCHECK_FUNCTION_EXISTS=pthread_create -o
CMakeFiles/cmTC_81779.dir/CheckFunctionExists.c.o -c
/opt/apache-s/apache-impala/native-toolchain/build/cmake-3.8.2-p1/share/cmake-3.8/Modules/CheckFunctionExists.c
Linking C executable cmTC_81779
/opt/apache-s/apache-impala/native-toolchain/build/cmake-3.8.2-p1/bin/cmake
-E cmake_link_script CMakeFiles/cmTC_81779.dir/link.txt --verbose=1
/opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
-DCHECK_FUNCTION_EXISTS=pthread_create -rdynamic
CMakeFiles/cmTC_81779.dir/CheckFunctionExists.c.o -o cmTC_81779 -lpthreads
/usr/bin/ld: cannot find -lpthreads
collect2: error: ld returned 1 exit status
gmake[1]: *** [cmTC_81779] Error 1
gmake[1]: Leaving directory
`/opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/CMakeTmp'
gmake: *** [cmTC_81779/fast] Error 2
Regards,
Anup Tiwari
On Tue, Feb 6, 2018 at 6:07 PM, Vincent Tran <vt...@cloudera.com> wrote:
> Hey Anup,
>
> Lools like ypu may be missing some prereq packages. This particular one is
> libsasl2-dev
>
>
>
> On Feb 6, 2018 5:12 AM, "Anup Tiwari" <an...@gmail.com> wrote:
>
>> I found this link http://mail-archives.apache.or
>> g/mod_mbox/impala-dev/201708.mbox/%3CCADZba-Td-wXRqLRxctvFM
>> 2mY44cYJhkX-JSs4=uPQCnFDhoDyA@mail.gmail.com%3E which says
>>
>> Do the following:
>>
>> rm -rf ${IMPALA_TOOLCHAIN}/gflags-${IMPALA_GFLAGS_VERSION}
>> . bin/impala-config.sh
>> bin/bootstrap_toolchain.py
>>
>> But post executing bootstrap command ; i am getting below error :-
>>
>> Traceback (most recent call last):
>> File "/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/bootstrap_virtualenv.py", line 367, in <module>
>> if install_compiled_deps_if_possible():
>> File "/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/bootstrap_virtualenv.py", line 216, in install_compiled_deps_if_possible
>> exec_pip_install(["-r", COMPILED_REQS_PATH], cc=cc, env=env)
>> File "/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/bootstrap_virtualenv.py", line 145, in exec_pip_install
>> "file://%s" % urllib.pathname2url(os.path.abspath(DEPS_DIR))] + args, env=env)
>> File "/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/bootstrap_virtualenv.py", line 96, in exec_cmd
>> % (args, output))
>> Exception: Command returned non-zero status
>> Command: ['/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/env/bin/python', '/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/env/bin/pip', 'install', '--no-binary', '--no-index', '--no-cache-dir', '--find-links', 'file:///opt/apache-s/apache-impala/apache-impala-2.11.0/infra/python/deps', '-r', '/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt']
>> Output: You are using pip version 7.1.0, however version 9.0.1 is available.
>> You should consider upgrading via the 'pip install --upgrade pip' command.
>> Requirement already satisfied (use --upgrade to upgrade): argparse==1.4.0 in ./infra/python/env/lib/python2.7/site-packages (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 1))
>> Requirement already satisfied (use --upgrade to upgrade): Fabric==1.10.2 in ./infra/python/env/lib/python2.7/site-packages (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 2))
>> Requirement already satisfied (use --upgrade to upgrade): paramiko==1.15.2 in ./infra/python/env/lib/python2.7/site-packages (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 3))
>> Requirement already satisfied (use --upgrade to upgrade): ecdsa==0.13 in ./infra/python/env/lib/python2.7/site-packages (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 4))
>> Requirement already satisfied (use --upgrade to upgrade): pycrypto==2.6.1 in ./infra/python/env/lib/python2.7/site-packages (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 5))
>> Requirement already satisfied (use --upgrade to upgrade): impyla==0.14.0 in ./infra/python/env/lib/python2.7/site-packages (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 6))
>> Requirement already satisfied (use --upgrade to upgrade): bitarray==0.8.1 in ./infra/python/env/lib/python2.7/site-packages (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 7))
>> Collecting sasl==0.1.3 (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 8))
>> Requirement already satisfied (use --upgrade to upgrade): six==1.9.0 in ./infra/python/env/lib/python2.7/site-packages (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 9))
>> Collecting thrift==0.9.0 (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 10))
>> Collecting thrift-sasl==0.1.0 (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 11))
>> Collecting psutil==0.7.1 (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 12))
>> Collecting Cython==0.23.4 (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 13))
>> Collecting numpy==1.10.4 (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 14))
>> Installing collected packages: sasl, thrift, thrift-sasl, psutil, Cython, numpy
>> Running setup.py install for sasl
>> Complete output from command /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/env/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-build-MTPxSs/sasl/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-EHfJDn-record/install-record.txt --single-version-externally-managed --compile --install-headers /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/env/include/site/python2.7/sasl:
>> running install
>> running build
>> running build_py
>> creating build
>> creating build/lib.linux-x86_64-2.7
>> creating build/lib.linux-x86_64-2.7/sasl
>> copying sasl/__init__.py -> build/lib.linux-x86_64-2.7/sasl
>> copying sasl/saslwrapper.py -> build/lib.linux-x86_64-2.7/sasl
>> running egg_info
>> writing sasl.egg-info/PKG-INFO
>> writing top-level names to sasl.egg-info/top_level.txt
>> writing dependency_links to sasl.egg-info/dependency_links.txt
>> warning: manifest_maker: standard file '-c' not found
>>
>> reading manifest file 'sasl.egg-info/SOURCES.txt'
>> reading manifest template 'MANIFEST.in'
>> writing manifest file 'sasl.egg-info/SOURCES.txt'
>> running build_ext
>> building '_saslwrapper' extension
>> creating build/temp.linux-x86_64-2.7
>> creating build/temp.linux-x86_64-2.7/sasl
>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -Isasl -I/usr/include/python2.7 -c sasl/saslwrapper.cpp -o build/temp.linux-x86_64-2.7/sasl/saslwrapper.o
>> sasl/saslwrapper.cpp:21:23: fatal error: sasl/sasl.h: No such file or directory
>> #include <sasl/sasl.h>
>> ^
>> compilation terminated.
>> error: command '/opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc' failed with exit status 1
>>
>> ----------------------------------------
>> Command "/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/env/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-build-MTPxSs/sasl/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-EHfJDn-record/install-record.txt --single-version-externally-managed --compile --install-headers /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/env/include/site/python2.7/sasl" failed with error code 1 in /tmp/pip-build-MTPxSs/sasl
>>
>> Error in /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/impala-python at line 28:
>>
>>
>>
>> Regards,
>> Anup Tiwari
>>
>> On Tue, Feb 6, 2018 at 2:52 PM, Anup Tiwari <an...@gmail.com>
>> wrote:
>>
>>> Hi Zoltan,
>>>
>>> I followed the steps mention in https://cwiki.apache.org/confl
>>> uence/display/IMPALA/Building+Impala and when i executed below command
>>> , i got error after some time :-
>>>
>>> *Build Command :*
>>> ${IMPALA_HOME}/buildall.sh -noclean -skiptests -build_shared_libs -format
>>>
>>>
>>>
>>> *Error :-- ----------> Adding thirdparty library cyrus_sasl. <----------*CMake
>>> Error at CMakeLists.txt:136 (message):
>>> Library 'cyrus_sasl' has neither shared nor static library files
>>> Call Stack (most recent call first):
>>> CMakeLists.txt:227 (IMPALA_ADD_THIRDPARTY_LIB)
>>>
>>> -- Configuring incomplete, errors occurred!
>>> See also "/opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles
>>> /CMakeOutput.log".
>>> See also "/opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles
>>> /CMakeError.log".
>>> Error in /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/make_impala.sh
>>> at line 161: cmake . ${CMAKE_ARGS[@]}
>>>
>>>
>>>
>>>
>>> *Logs in "CMakeOutput.log" :-*Determining if the function
>>> pthread_create exists in the pthread passed with the following output:
>>> Change Dir: /opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/
>>> CMakeTmp
>>>
>>> Run Build Command:"/usr/bin/gmake" "cmTC_ee4f0/fast"
>>> /usr/bin/gmake -f CMakeFiles/cmTC_ee4f0.dir/build.make
>>> CMakeFiles/cmTC_ee4f0.dir/build
>>> gmake[1]: Entering directory `/opt/apache-s/apache-impala/a
>>> pache-impala-2.11.0/CMakeFiles/CMakeTmp'
>>> Building C object CMakeFiles/cmTC_ee4f0.dir/CheckFunctionExists.c.o
>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
>>> -DCHECK_FUNCTION_EXISTS=pthread_create -o
>>> CMakeFiles/cmTC_ee4f0.dir/CheckFunctionExists.c.o -c
>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/c
>>> make-3.8.2-p1/share/cmake-3.8/Modules/CheckFunctionExists.c
>>> Linking C executable cmTC_ee4f0
>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/cmake-3.8.2-p1/bin/cmake
>>> -E cmake_link_script CMakeFiles/cmTC_ee4f0.dir/link.txt --verbose=1
>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
>>> -DCHECK_FUNCTION_EXISTS=pthread_create -rdynamic
>>> CMakeFiles/cmTC_ee4f0.dir/CheckFunctionExists.c.o -o cmTC_ee4f0
>>> -lpthread
>>> gmake[1]: Leaving directory `/opt/apache-s/apache-impala/a
>>> pache-impala-2.11.0/CMakeFiles/CMakeTmp'
>>>
>>>
>>>
>>> *Logs in "CMakeError.log" :-*Determining if the function pthread_create
>>> exists in the pthreads failed with the following output:
>>> Change Dir: /opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/
>>> CMakeTmp
>>>
>>> Run Build Command:"/usr/bin/gmake" "cmTC_18bf3/fast"
>>> /usr/bin/gmake -f CMakeFiles/cmTC_18bf3.dir/build.make
>>> CMakeFiles/cmTC_18bf3.dir/build
>>> gmake[1]: Entering directory `/opt/apache-s/apache-impala/a
>>> pache-impala-2.11.0/CMakeFiles/CMakeTmp'
>>> Building C object CMakeFiles/cmTC_18bf3.dir/CheckFunctionExists.c.o
>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
>>> -DCHECK_FUNCTION_EXISTS=pthread_create -o
>>> CMakeFiles/cmTC_18bf3.dir/CheckFunctionExists.c.o -c
>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/c
>>> make-3.8.2-p1/share/cmake-3.8/Modules/CheckFunctionExists.c
>>> Linking C executable cmTC_18bf3
>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/cmake-3.8.2-p1/bin/cmake
>>> -E cmake_link_script CMakeFiles/cmTC_18bf3.dir/link.txt --verbose=1
>>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
>>> -DCHECK_FUNCTION_EXISTS=pthread_create -rdynamic
>>> CMakeFiles/cmTC_18bf3.dir/CheckFunctionExists.c.o -o cmTC_18bf3
>>> -lpthreads
>>> /usr/bin/ld: cannot find -lpthreads
>>> collect2: error: ld returned 1 exit status
>>> gmake[1]: *** [cmTC_18bf3] Error 1
>>> gmake[1]: Leaving directory `/opt/apache-s/apache-impala/a
>>> pache-impala-2.11.0/CMakeFiles/CMakeTmp'
>>> gmake: *** [cmTC_18bf3/fast] Error 2
>>>
>>> Also it seems that this is related to toolchain error so i followed this
>>> link : https://cwiki.apache.org/confluence/display/IMPALA/Building+
>>> native-toolchain+from+scratch+and+using+with+Impala where i first
>>> builded native-toolchain and then followed step mention in "Using custom
>>> toolchain with Impala" but while creating symbolic link i got error "ln:
>>> ‘./gcc-4.9.2’: cannot overwrite directory".
>>>
>>> @xinran, i guess post building impala, i will have to perform steps
>>> mentioned by you.. correct me if i am wrong?
>>>
>>>
>>>
>>> Regards,
>>> Anup Tiwari
>>>
>>> On Mon, Feb 5, 2018 at 11:14 PM, Xinran Yu Tinney <
>>> yuxinran8011@gmail.com> wrote:
>>>
>>>> Hi Anup,
>>>> Did you also source the config.sh? Run the following before
>>>> impala-shell.sh:
>>>>
>>>> source ~/Impala/bin/impala-config.sh
>>>>
>>>> ${IMPALA_HOME}/buildall.sh -notests -build_shared_libs
>>>> -start_minicluster
>>>>
>>>> ${IMPALA_HOME}/bin/start-impala-cluster.py
>>>>
>>>>
>>>> 2018-02-05 10:07 GMT-06:00 Michael Brown <mi...@cloudera.com>:
>>>>
>>>>> Hello,
>>>>>
>>>>> Please see:
>>>>>
>>>>> https://cwiki.apache.org/confluence/display/IMPALA/Bootstrap
>>>>> ping+an+Impala+Development+Environment+From+Scratch
>>>>> https://cwiki.apache.org/confluence/display/IMPALA/Building+Impala
>>>>>
>>>>>
>>>>> On Mon, Feb 5, 2018 at 12:09 AM, Anup Tiwari <an...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>> I am new to impala so i am getting errors while i am setting impala.
>>>>>>
>>>>>> I have downloaded latest impala release from "
>>>>>> http://www.apache.org/dyn/closer.cgi?action=download&filena
>>>>>> me=impala/2.11.0/apache-impala-2.11.0.tar.gz".
>>>>>> I have extracted this in a folder and followed description mentioned
>>>>>> in below link :-
>>>>>>
>>>>>> https://impala.apache.org/docs/build/html/topics/impala_proc
>>>>>> esses.html#processes
>>>>>>
>>>>>> As mentioned in above link "*To start the Impala state store and
>>>>>> Impala from the command line or a script, you can either use the service
>>>>>> command or you can start the daemons directly through the impalad,
>>>>>> statestored, and catalogd executables.*" So i searched for
>>>>>> executable and found "start-impalad.sh"(which i think is correct one, let
>>>>>> me know if i have missed this) but when i am executing this , i am getting
>>>>>> below *errors* :-
>>>>>>
>>>>>> bin/start-impalad.sh: line 89: /admin: No such file or directory
>>>>>> bin/start-impalad.sh: line 106: /opt/apache-s/apache-impala/ap
>>>>>> ache-impala-2.11.0/be/build/latest/service/impalad: No such file or
>>>>>> directory
>>>>>>
>>>>>> Also when i executed ./impala-config.sh then i got below *error* :-
>>>>>>
>>>>>> Unable to find the lsb_release command. Please ensure it is available
>>>>>> in your PATH.
>>>>>> ./impala-config.sh: line 265: return: can only `return' from a
>>>>>> function or sourced script
>>>>>> lsb_release command failed, output was: ./impala-config.sh: line 267:
>>>>>> lsb_release: command not found
>>>>>> ./impala-config.sh: line 270: return: can only `return' from a
>>>>>> function or sourced script
>>>>>>
>>>>>>
>>>>>> *Questions :-*
>>>>>> 1. When we setup Hive/Hadoop/Drill etc then we have certain steps for
>>>>>> setting HIVE_HOME/HADOOP_HOME/DRILL_HOME and also we specify certain
>>>>>> setting in conf folder. Do we have similar documentation which can specify
>>>>>> step by step setup.
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>> Anup Tiwari
>>>>>>
>>>>>> On Tue, Jan 23, 2018 at 8:56 PM, Jim Apple <jb...@cloudera.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Ubuntu 14.04 and 16.04 are the OSes most Impala developers use, but
>>>>>>> many Impala users use CentOS 7.
>>>>>>>
>>>>>>> Here is part of the docs on tuning:
>>>>>>> https://impala.apache.org/docs/build/html/topics/impala_perf
>>>>>>> ormance.html
>>>>>>>
>>>>>>> Here is some tuning info that was produced by Cloudera:
>>>>>>> http://blog.cloudera.com/wp-content/uploads/2017/03/Impala-C
>>>>>>> ookbook-01-20173.pdf
>>>>>>>
>>>>>>> On Tue, Jan 23, 2018 at 6:53 AM, Anup Tiwari <an...@gmail.com>
>>>>>>> wrote:
>>>>>>> > Hi Vincent,
>>>>>>> >
>>>>>>> > I found this link which says Supported Operating Systems are
>>>>>>> Ubuntu 14.04,
>>>>>>> > 16.04 so does this mean it won't work on centos 7.
>>>>>>> >
>>>>>>> > Also can you send some link which can help us in tuning as well.
>>>>>>> >
>>>>>>> > Regards,
>>>>>>> > Anup Tiwari
>>>>>>> >
>>>>>>> > On Tue, Jan 23, 2018 at 7:26 PM, Vincent Tran <vt...@cloudera.com>
>>>>>>> wrote:
>>>>>>> >>
>>>>>>> >> Hi Anup,
>>>>>>> >>
>>>>>>> >> Is this the link you are looking for?
>>>>>>> >>
>>>>>>> >> https://impala.apache.org/docs/build/html/topics/impala_inst
>>>>>>> all.html
>>>>>>> >>
>>>>>>> >> On Tue, Jan 23, 2018 at 7:52 AM, Anup Tiwari <
>>>>>>> anupsdtiwari@gmail.com>
>>>>>>> >> wrote:
>>>>>>> >>>
>>>>>>> >>> Hi All
>>>>>>> >>>
>>>>>>> >>> I want to install impala on my production
>>>>>>> environment(distributed) :-
>>>>>>> >>>
>>>>>>> >>> 1. Os Details : CentOS Linux release 7.2.1511 (Core)
>>>>>>> >>> 2. Apache Hadoop 2.8.0
>>>>>>> >>> 3. Apache Hive 2.1.1
>>>>>>> >>>
>>>>>>> >>> I didn't found any link on mail, please suggest if it is
>>>>>>> possible or not
>>>>>>> >>> and if possible then please share some links.
>>>>>>> >>>
>>>>>>> >>> Thanks in advance!!
>>>>>>> >>>
>>>>>>> >>> Regards,
>>>>>>> >>> Anup Tiwari
>>>>>>> >>
>>>>>>> >>
>>>>>>> >>
>>>>>>> >>
>>>>>>> >> --
>>>>>>> >> Vincent T. Tran
>>>>>>> >> Customer Operations Engineer
>>>>>>> >> Cloudera, Inc.
>>>>>>> >
>>>>>>> >
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
Re: Install Impala on Apache Hadoop and Hive
Posted by Vincent Tran <vt...@cloudera.com>.
Hey Anup,
Lools like ypu may be missing some prereq packages. This particular one is
libsasl2-dev
On Feb 6, 2018 5:12 AM, "Anup Tiwari" <an...@gmail.com> wrote:
> I found this link http://mail-archives.apache.org/mod_mbox/impala-dev/
> 201708.mbox/%3CCADZba-Td-wXRqLRxctvFM2mY44cYJhkX-JSs4=
> uPQCnFDhoDyA@mail.gmail.com%3E which says
>
> Do the following:
>
> rm -rf ${IMPALA_TOOLCHAIN}/gflags-${IMPALA_GFLAGS_VERSION}
> . bin/impala-config.sh
> bin/bootstrap_toolchain.py
>
> But post executing bootstrap command ; i am getting below error :-
>
> Traceback (most recent call last):
> File "/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/bootstrap_virtualenv.py", line 367, in <module>
> if install_compiled_deps_if_possible():
> File "/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/bootstrap_virtualenv.py", line 216, in install_compiled_deps_if_possible
> exec_pip_install(["-r", COMPILED_REQS_PATH], cc=cc, env=env)
> File "/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/bootstrap_virtualenv.py", line 145, in exec_pip_install
> "file://%s" % urllib.pathname2url(os.path.abspath(DEPS_DIR))] + args, env=env)
> File "/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/bootstrap_virtualenv.py", line 96, in exec_cmd
> % (args, output))
> Exception: Command returned non-zero status
> Command: ['/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/env/bin/python', '/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/env/bin/pip', 'install', '--no-binary', '--no-index', '--no-cache-dir', '--find-links', 'file:///opt/apache-s/apache-impala/apache-impala-2.11.0/infra/python/deps', '-r', '/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt']
> Output: You are using pip version 7.1.0, however version 9.0.1 is available.
> You should consider upgrading via the 'pip install --upgrade pip' command.
> Requirement already satisfied (use --upgrade to upgrade): argparse==1.4.0 in ./infra/python/env/lib/python2.7/site-packages (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 1))
> Requirement already satisfied (use --upgrade to upgrade): Fabric==1.10.2 in ./infra/python/env/lib/python2.7/site-packages (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 2))
> Requirement already satisfied (use --upgrade to upgrade): paramiko==1.15.2 in ./infra/python/env/lib/python2.7/site-packages (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 3))
> Requirement already satisfied (use --upgrade to upgrade): ecdsa==0.13 in ./infra/python/env/lib/python2.7/site-packages (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 4))
> Requirement already satisfied (use --upgrade to upgrade): pycrypto==2.6.1 in ./infra/python/env/lib/python2.7/site-packages (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 5))
> Requirement already satisfied (use --upgrade to upgrade): impyla==0.14.0 in ./infra/python/env/lib/python2.7/site-packages (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 6))
> Requirement already satisfied (use --upgrade to upgrade): bitarray==0.8.1 in ./infra/python/env/lib/python2.7/site-packages (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 7))
> Collecting sasl==0.1.3 (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 8))
> Requirement already satisfied (use --upgrade to upgrade): six==1.9.0 in ./infra/python/env/lib/python2.7/site-packages (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 9))
> Collecting thrift==0.9.0 (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 10))
> Collecting thrift-sasl==0.1.0 (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 11))
> Collecting psutil==0.7.1 (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 12))
> Collecting Cython==0.23.4 (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 13))
> Collecting numpy==1.10.4 (from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt (line 14))
> Installing collected packages: sasl, thrift, thrift-sasl, psutil, Cython, numpy
> Running setup.py install for sasl
> Complete output from command /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/env/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-build-MTPxSs/sasl/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-EHfJDn-record/install-record.txt --single-version-externally-managed --compile --install-headers /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/env/include/site/python2.7/sasl:
> running install
> running build
> running build_py
> creating build
> creating build/lib.linux-x86_64-2.7
> creating build/lib.linux-x86_64-2.7/sasl
> copying sasl/__init__.py -> build/lib.linux-x86_64-2.7/sasl
> copying sasl/saslwrapper.py -> build/lib.linux-x86_64-2.7/sasl
> running egg_info
> writing sasl.egg-info/PKG-INFO
> writing top-level names to sasl.egg-info/top_level.txt
> writing dependency_links to sasl.egg-info/dependency_links.txt
> warning: manifest_maker: standard file '-c' not found
>
> reading manifest file 'sasl.egg-info/SOURCES.txt'
> reading manifest template 'MANIFEST.in'
> writing manifest file 'sasl.egg-info/SOURCES.txt'
> running build_ext
> building '_saslwrapper' extension
> creating build/temp.linux-x86_64-2.7
> creating build/temp.linux-x86_64-2.7/sasl
> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -Isasl -I/usr/include/python2.7 -c sasl/saslwrapper.cpp -o build/temp.linux-x86_64-2.7/sasl/saslwrapper.o
> sasl/saslwrapper.cpp:21:23: fatal error: sasl/sasl.h: No such file or directory
> #include <sasl/sasl.h>
> ^
> compilation terminated.
> error: command '/opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc' failed with exit status 1
>
> ----------------------------------------
> Command "/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/env/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-build-MTPxSs/sasl/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-EHfJDn-record/install-record.txt --single-version-externally-managed --compile --install-headers /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/env/include/site/python2.7/sasl" failed with error code 1 in /tmp/pip-build-MTPxSs/sasl
>
> Error in /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/impala-python at line 28:
>
>
>
> Regards,
> Anup Tiwari
>
> On Tue, Feb 6, 2018 at 2:52 PM, Anup Tiwari <an...@gmail.com>
> wrote:
>
>> Hi Zoltan,
>>
>> I followed the steps mention in https://cwiki.apache.org/confl
>> uence/display/IMPALA/Building+Impala and when i executed below command ,
>> i got error after some time :-
>>
>> *Build Command :*
>> ${IMPALA_HOME}/buildall.sh -noclean -skiptests -build_shared_libs -format
>>
>>
>>
>> *Error :-- ----------> Adding thirdparty library cyrus_sasl. <----------*CMake
>> Error at CMakeLists.txt:136 (message):
>> Library 'cyrus_sasl' has neither shared nor static library files
>> Call Stack (most recent call first):
>> CMakeLists.txt:227 (IMPALA_ADD_THIRDPARTY_LIB)
>>
>> -- Configuring incomplete, errors occurred!
>> See also "/opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles
>> /CMakeOutput.log".
>> See also "/opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles
>> /CMakeError.log".
>> Error in /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/make_impala.sh
>> at line 161: cmake . ${CMAKE_ARGS[@]}
>>
>>
>>
>>
>> *Logs in "CMakeOutput.log" :-*Determining if the function pthread_create
>> exists in the pthread passed with the following output:
>> Change Dir: /opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/
>> CMakeTmp
>>
>> Run Build Command:"/usr/bin/gmake" "cmTC_ee4f0/fast"
>> /usr/bin/gmake -f CMakeFiles/cmTC_ee4f0.dir/build.make
>> CMakeFiles/cmTC_ee4f0.dir/build
>> gmake[1]: Entering directory `/opt/apache-s/apache-impala/a
>> pache-impala-2.11.0/CMakeFiles/CMakeTmp'
>> Building C object CMakeFiles/cmTC_ee4f0.dir/CheckFunctionExists.c.o
>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
>> -DCHECK_FUNCTION_EXISTS=pthread_create -o
>> CMakeFiles/cmTC_ee4f0.dir/CheckFunctionExists.c.o -c
>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/
>> cmake-3.8.2-p1/share/cmake-3.8/Modules/CheckFunctionExists.c
>> Linking C executable cmTC_ee4f0
>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/cmake-3.8.2-p1/bin/cmake
>> -E cmake_link_script CMakeFiles/cmTC_ee4f0.dir/link.txt --verbose=1
>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
>> -DCHECK_FUNCTION_EXISTS=pthread_create -rdynamic
>> CMakeFiles/cmTC_ee4f0.dir/CheckFunctionExists.c.o -o cmTC_ee4f0
>> -lpthread
>> gmake[1]: Leaving directory `/opt/apache-s/apache-impala/a
>> pache-impala-2.11.0/CMakeFiles/CMakeTmp'
>>
>>
>>
>> *Logs in "CMakeError.log" :-*Determining if the function pthread_create
>> exists in the pthreads failed with the following output:
>> Change Dir: /opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/
>> CMakeTmp
>>
>> Run Build Command:"/usr/bin/gmake" "cmTC_18bf3/fast"
>> /usr/bin/gmake -f CMakeFiles/cmTC_18bf3.dir/build.make
>> CMakeFiles/cmTC_18bf3.dir/build
>> gmake[1]: Entering directory `/opt/apache-s/apache-impala/a
>> pache-impala-2.11.0/CMakeFiles/CMakeTmp'
>> Building C object CMakeFiles/cmTC_18bf3.dir/CheckFunctionExists.c.o
>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
>> -DCHECK_FUNCTION_EXISTS=pthread_create -o
>> CMakeFiles/cmTC_18bf3.dir/CheckFunctionExists.c.o -c
>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/
>> cmake-3.8.2-p1/share/cmake-3.8/Modules/CheckFunctionExists.c
>> Linking C executable cmTC_18bf3
>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/cmake-3.8.2-p1/bin/cmake
>> -E cmake_link_script CMakeFiles/cmTC_18bf3.dir/link.txt --verbose=1
>> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
>> -DCHECK_FUNCTION_EXISTS=pthread_create -rdynamic
>> CMakeFiles/cmTC_18bf3.dir/CheckFunctionExists.c.o -o cmTC_18bf3
>> -lpthreads
>> /usr/bin/ld: cannot find -lpthreads
>> collect2: error: ld returned 1 exit status
>> gmake[1]: *** [cmTC_18bf3] Error 1
>> gmake[1]: Leaving directory `/opt/apache-s/apache-impala/a
>> pache-impala-2.11.0/CMakeFiles/CMakeTmp'
>> gmake: *** [cmTC_18bf3/fast] Error 2
>>
>> Also it seems that this is related to toolchain error so i followed this
>> link : https://cwiki.apache.org/confluence/display/IMPALA/Building+
>> native-toolchain+from+scratch+and+using+with+Impala where i first
>> builded native-toolchain and then followed step mention in "Using custom
>> toolchain with Impala" but while creating symbolic link i got error "ln:
>> ‘./gcc-4.9.2’: cannot overwrite directory".
>>
>> @xinran, i guess post building impala, i will have to perform steps
>> mentioned by you.. correct me if i am wrong?
>>
>>
>>
>> Regards,
>> Anup Tiwari
>>
>> On Mon, Feb 5, 2018 at 11:14 PM, Xinran Yu Tinney <yuxinran8011@gmail.com
>> > wrote:
>>
>>> Hi Anup,
>>> Did you also source the config.sh? Run the following before
>>> impala-shell.sh:
>>>
>>> source ~/Impala/bin/impala-config.sh
>>>
>>> ${IMPALA_HOME}/buildall.sh -notests -build_shared_libs -start_minicluster
>>>
>>> ${IMPALA_HOME}/bin/start-impala-cluster.py
>>>
>>>
>>> 2018-02-05 10:07 GMT-06:00 Michael Brown <mi...@cloudera.com>:
>>>
>>>> Hello,
>>>>
>>>> Please see:
>>>>
>>>> https://cwiki.apache.org/confluence/display/IMPALA/Bootstrap
>>>> ping+an+Impala+Development+Environment+From+Scratch
>>>> https://cwiki.apache.org/confluence/display/IMPALA/Building+Impala
>>>>
>>>>
>>>> On Mon, Feb 5, 2018 at 12:09 AM, Anup Tiwari <an...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi All,
>>>>>
>>>>> I am new to impala so i am getting errors while i am setting impala.
>>>>>
>>>>> I have downloaded latest impala release from "
>>>>> http://www.apache.org/dyn/closer.cgi?action=download&filena
>>>>> me=impala/2.11.0/apache-impala-2.11.0.tar.gz".
>>>>> I have extracted this in a folder and followed description mentioned
>>>>> in below link :-
>>>>>
>>>>> https://impala.apache.org/docs/build/html/topics/impala_proc
>>>>> esses.html#processes
>>>>>
>>>>> As mentioned in above link "*To start the Impala state store and
>>>>> Impala from the command line or a script, you can either use the service
>>>>> command or you can start the daemons directly through the impalad,
>>>>> statestored, and catalogd executables.*" So i searched for executable
>>>>> and found "start-impalad.sh"(which i think is correct one, let me know if i
>>>>> have missed this) but when i am executing this , i am getting below
>>>>> *errors* :-
>>>>>
>>>>> bin/start-impalad.sh: line 89: /admin: No such file or directory
>>>>> bin/start-impalad.sh: line 106: /opt/apache-s/apache-impala/ap
>>>>> ache-impala-2.11.0/be/build/latest/service/impalad: No such file or
>>>>> directory
>>>>>
>>>>> Also when i executed ./impala-config.sh then i got below *error* :-
>>>>>
>>>>> Unable to find the lsb_release command. Please ensure it is available
>>>>> in your PATH.
>>>>> ./impala-config.sh: line 265: return: can only `return' from a
>>>>> function or sourced script
>>>>> lsb_release command failed, output was: ./impala-config.sh: line 267:
>>>>> lsb_release: command not found
>>>>> ./impala-config.sh: line 270: return: can only `return' from a
>>>>> function or sourced script
>>>>>
>>>>>
>>>>> *Questions :-*
>>>>> 1. When we setup Hive/Hadoop/Drill etc then we have certain steps for
>>>>> setting HIVE_HOME/HADOOP_HOME/DRILL_HOME and also we specify certain
>>>>> setting in conf folder. Do we have similar documentation which can specify
>>>>> step by step setup.
>>>>>
>>>>>
>>>>> Regards,
>>>>> Anup Tiwari
>>>>>
>>>>> On Tue, Jan 23, 2018 at 8:56 PM, Jim Apple <jb...@cloudera.com>
>>>>> wrote:
>>>>>
>>>>>> Ubuntu 14.04 and 16.04 are the OSes most Impala developers use, but
>>>>>> many Impala users use CentOS 7.
>>>>>>
>>>>>> Here is part of the docs on tuning:
>>>>>> https://impala.apache.org/docs/build/html/topics/impala_perf
>>>>>> ormance.html
>>>>>>
>>>>>> Here is some tuning info that was produced by Cloudera:
>>>>>> http://blog.cloudera.com/wp-content/uploads/2017/03/Impala-C
>>>>>> ookbook-01-20173.pdf
>>>>>>
>>>>>> On Tue, Jan 23, 2018 at 6:53 AM, Anup Tiwari <an...@gmail.com>
>>>>>> wrote:
>>>>>> > Hi Vincent,
>>>>>> >
>>>>>> > I found this link which says Supported Operating Systems are Ubuntu
>>>>>> 14.04,
>>>>>> > 16.04 so does this mean it won't work on centos 7.
>>>>>> >
>>>>>> > Also can you send some link which can help us in tuning as well.
>>>>>> >
>>>>>> > Regards,
>>>>>> > Anup Tiwari
>>>>>> >
>>>>>> > On Tue, Jan 23, 2018 at 7:26 PM, Vincent Tran <vt...@cloudera.com>
>>>>>> wrote:
>>>>>> >>
>>>>>> >> Hi Anup,
>>>>>> >>
>>>>>> >> Is this the link you are looking for?
>>>>>> >>
>>>>>> >> https://impala.apache.org/docs/build/html/topics/impala_inst
>>>>>> all.html
>>>>>> >>
>>>>>> >> On Tue, Jan 23, 2018 at 7:52 AM, Anup Tiwari <
>>>>>> anupsdtiwari@gmail.com>
>>>>>> >> wrote:
>>>>>> >>>
>>>>>> >>> Hi All
>>>>>> >>>
>>>>>> >>> I want to install impala on my production
>>>>>> environment(distributed) :-
>>>>>> >>>
>>>>>> >>> 1. Os Details : CentOS Linux release 7.2.1511 (Core)
>>>>>> >>> 2. Apache Hadoop 2.8.0
>>>>>> >>> 3. Apache Hive 2.1.1
>>>>>> >>>
>>>>>> >>> I didn't found any link on mail, please suggest if it is possible
>>>>>> or not
>>>>>> >>> and if possible then please share some links.
>>>>>> >>>
>>>>>> >>> Thanks in advance!!
>>>>>> >>>
>>>>>> >>> Regards,
>>>>>> >>> Anup Tiwari
>>>>>> >>
>>>>>> >>
>>>>>> >>
>>>>>> >>
>>>>>> >> --
>>>>>> >> Vincent T. Tran
>>>>>> >> Customer Operations Engineer
>>>>>> >> Cloudera, Inc.
>>>>>> >
>>>>>> >
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Install Impala on Apache Hadoop and Hive
Posted by Anup Tiwari <an...@gmail.com>.
I found this link
http://mail-archives.apache.org/mod_mbox/impala-dev/201708.mbox/%3CCADZba-Td-wXRqLRxctvFM2mY44cYJhkX-JSs4=uPQCnFDhoDyA@mail.gmail.com%3E
which says
Do the following:
rm -rf ${IMPALA_TOOLCHAIN}/gflags-${IMPALA_GFLAGS_VERSION}
. bin/impala-config.sh
bin/bootstrap_toolchain.py
But post executing bootstrap command ; i am getting below error :-
Traceback (most recent call last):
File "/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/bootstrap_virtualenv.py",
line 367, in <module>
if install_compiled_deps_if_possible():
File "/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/bootstrap_virtualenv.py",
line 216, in install_compiled_deps_if_possible
exec_pip_install(["-r", COMPILED_REQS_PATH], cc=cc, env=env)
File "/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/bootstrap_virtualenv.py",
line 145, in exec_pip_install
"file://%s" % urllib.pathname2url(os.path.abspath(DEPS_DIR))] +
args, env=env)
File "/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/bootstrap_virtualenv.py",
line 96, in exec_cmd
% (args, output))
Exception: Command returned non-zero status
Command: ['/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/env/bin/python',
'/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/env/bin/pip',
'install', '--no-binary', '--no-index', '--no-cache-dir',
'--find-links',
'file:///opt/apache-s/apache-impala/apache-impala-2.11.0/infra/python/deps',
'-r', '/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt']
Output: You are using pip version 7.1.0, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Requirement already satisfied (use --upgrade to upgrade):
argparse==1.4.0 in ./infra/python/env/lib/python2.7/site-packages
(from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt
(line 1))
Requirement already satisfied (use --upgrade to upgrade):
Fabric==1.10.2 in ./infra/python/env/lib/python2.7/site-packages (from
-r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt
(line 2))
Requirement already satisfied (use --upgrade to upgrade):
paramiko==1.15.2 in ./infra/python/env/lib/python2.7/site-packages
(from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt
(line 3))
Requirement already satisfied (use --upgrade to upgrade): ecdsa==0.13
in ./infra/python/env/lib/python2.7/site-packages (from -r
/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt
(line 4))
Requirement already satisfied (use --upgrade to upgrade):
pycrypto==2.6.1 in ./infra/python/env/lib/python2.7/site-packages
(from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt
(line 5))
Requirement already satisfied (use --upgrade to upgrade):
impyla==0.14.0 in ./infra/python/env/lib/python2.7/site-packages (from
-r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt
(line 6))
Requirement already satisfied (use --upgrade to upgrade):
bitarray==0.8.1 in ./infra/python/env/lib/python2.7/site-packages
(from -r /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt
(line 7))
Collecting sasl==0.1.3 (from -r
/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt
(line 8))
Requirement already satisfied (use --upgrade to upgrade): six==1.9.0
in ./infra/python/env/lib/python2.7/site-packages (from -r
/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt
(line 9))
Collecting thrift==0.9.0 (from -r
/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt
(line 10))
Collecting thrift-sasl==0.1.0 (from -r
/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt
(line 11))
Collecting psutil==0.7.1 (from -r
/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt
(line 12))
Collecting Cython==0.23.4 (from -r
/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt
(line 13))
Collecting numpy==1.10.4 (from -r
/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/deps/compiled-requirements.txt
(line 14))
Installing collected packages: sasl, thrift, thrift-sasl, psutil, Cython, numpy
Running setup.py install for sasl
Complete output from command
/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/env/bin/python
-c "import setuptools,
tokenize;__file__='/tmp/pip-build-MTPxSs/sasl/setup.py';exec(compile(getattr(tokenize,
'open', open)(__file__).read().replace('\r\n', '\n'), __file__,
'exec'))" install --record /tmp/pip-EHfJDn-record/install-record.txt
--single-version-externally-managed --compile --install-headers
/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/env/include/site/python2.7/sasl:
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/sasl
copying sasl/__init__.py -> build/lib.linux-x86_64-2.7/sasl
copying sasl/saslwrapper.py -> build/lib.linux-x86_64-2.7/sasl
running egg_info
writing sasl.egg-info/PKG-INFO
writing top-level names to sasl.egg-info/top_level.txt
writing dependency_links to sasl.egg-info/dependency_links.txt
warning: manifest_maker: standard file '-c' not found
reading manifest file 'sasl.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'sasl.egg-info/SOURCES.txt'
running build_ext
building '_saslwrapper' extension
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/sasl
/opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
-fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2
-fexceptions -fstack-protector-strong --param=ssp-buffer-size=4
-grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv
-DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions
-fstack-protector-strong --param=ssp-buffer-size=4
-grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv
-fPIC -Isasl -I/usr/include/python2.7 -c sasl/saslwrapper.cpp -o
build/temp.linux-x86_64-2.7/sasl/saslwrapper.o
sasl/saslwrapper.cpp:21:23: fatal error: sasl/sasl.h: No such file
or directory
#include <sasl/sasl.h>
^
compilation terminated.
error: command
'/opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc'
failed with exit status 1
----------------------------------------
Command "/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/env/bin/python
-c "import setuptools,
tokenize;__file__='/tmp/pip-build-MTPxSs/sasl/setup.py';exec(compile(getattr(tokenize,
'open', open)(__file__).read().replace('\r\n', '\n'), __file__,
'exec'))" install --record /tmp/pip-EHfJDn-record/install-record.txt
--single-version-externally-managed --compile --install-headers
/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/../infra/python/env/include/site/python2.7/sasl"
failed with error code 1 in /tmp/pip-build-MTPxSs/sasl
Error in /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/impala-python
at line 28:
Regards,
Anup Tiwari
On Tue, Feb 6, 2018 at 2:52 PM, Anup Tiwari <an...@gmail.com> wrote:
> Hi Zoltan,
>
> I followed the steps mention in https://cwiki.apache.org/confl
> uence/display/IMPALA/Building+Impala and when i executed below command ,
> i got error after some time :-
>
> *Build Command :*
> ${IMPALA_HOME}/buildall.sh -noclean -skiptests -build_shared_libs -format
>
>
>
> *Error :-- ----------> Adding thirdparty library cyrus_sasl. <----------*CMake
> Error at CMakeLists.txt:136 (message):
> Library 'cyrus_sasl' has neither shared nor static library files
> Call Stack (most recent call first):
> CMakeLists.txt:227 (IMPALA_ADD_THIRDPARTY_LIB)
>
> -- Configuring incomplete, errors occurred!
> See also "/opt/apache-s/apache-impala/apache-impala-2.11.0/
> CMakeFiles/CMakeOutput.log".
> See also "/opt/apache-s/apache-impala/apache-impala-2.11.0/
> CMakeFiles/CMakeError.log".
> Error in /opt/apache-s/apache-impala/apache-impala-2.11.0/bin/make_impala.sh
> at line 161: cmake . ${CMAKE_ARGS[@]}
>
>
>
>
> *Logs in "CMakeOutput.log" :-*Determining if the function pthread_create
> exists in the pthread passed with the following output:
> Change Dir: /opt/apache-s/apache-impala/apache-impala-2.11.0/
> CMakeFiles/CMakeTmp
>
> Run Build Command:"/usr/bin/gmake" "cmTC_ee4f0/fast"
> /usr/bin/gmake -f CMakeFiles/cmTC_ee4f0.dir/build.make
> CMakeFiles/cmTC_ee4f0.dir/build
> gmake[1]: Entering directory `/opt/apache-s/apache-impala/
> apache-impala-2.11.0/CMakeFiles/CMakeTmp'
> Building C object CMakeFiles/cmTC_ee4f0.dir/CheckFunctionExists.c.o
> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
> -DCHECK_FUNCTION_EXISTS=pthread_create -o CMakeFiles/cmTC_ee4f0.dir/CheckFunctionExists.c.o
> -c /opt/apache-s/apache-impala/apache-impala-2.11.0/
> toolchain/cmake-3.8.2-p1/share/cmake-3.8/Modules/CheckFunctionExists.c
> Linking C executable cmTC_ee4f0
> /opt/apache-s/apache-impala/apache-impala-2.11.0/
> toolchain/cmake-3.8.2-p1/bin/cmake -E cmake_link_script
> CMakeFiles/cmTC_ee4f0.dir/link.txt --verbose=1
> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
> -DCHECK_FUNCTION_EXISTS=pthread_create -rdynamic
> CMakeFiles/cmTC_ee4f0.dir/CheckFunctionExists.c.o -o cmTC_ee4f0
> -lpthread
> gmake[1]: Leaving directory `/opt/apache-s/apache-impala/
> apache-impala-2.11.0/CMakeFiles/CMakeTmp'
>
>
>
> *Logs in "CMakeError.log" :-*Determining if the function pthread_create
> exists in the pthreads failed with the following output:
> Change Dir: /opt/apache-s/apache-impala/apache-impala-2.11.0/
> CMakeFiles/CMakeTmp
>
> Run Build Command:"/usr/bin/gmake" "cmTC_18bf3/fast"
> /usr/bin/gmake -f CMakeFiles/cmTC_18bf3.dir/build.make
> CMakeFiles/cmTC_18bf3.dir/build
> gmake[1]: Entering directory `/opt/apache-s/apache-impala/
> apache-impala-2.11.0/CMakeFiles/CMakeTmp'
> Building C object CMakeFiles/cmTC_18bf3.dir/CheckFunctionExists.c.o
> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
> -DCHECK_FUNCTION_EXISTS=pthread_create -o CMakeFiles/cmTC_18bf3.dir/CheckFunctionExists.c.o
> -c /opt/apache-s/apache-impala/apache-impala-2.11.0/
> toolchain/cmake-3.8.2-p1/share/cmake-3.8/Modules/CheckFunctionExists.c
> Linking C executable cmTC_18bf3
> /opt/apache-s/apache-impala/apache-impala-2.11.0/
> toolchain/cmake-3.8.2-p1/bin/cmake -E cmake_link_script
> CMakeFiles/cmTC_18bf3.dir/link.txt --verbose=1
> /opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
> -DCHECK_FUNCTION_EXISTS=pthread_create -rdynamic
> CMakeFiles/cmTC_18bf3.dir/CheckFunctionExists.c.o -o cmTC_18bf3
> -lpthreads
> /usr/bin/ld: cannot find -lpthreads
> collect2: error: ld returned 1 exit status
> gmake[1]: *** [cmTC_18bf3] Error 1
> gmake[1]: Leaving directory `/opt/apache-s/apache-impala/
> apache-impala-2.11.0/CMakeFiles/CMakeTmp'
> gmake: *** [cmTC_18bf3/fast] Error 2
>
> Also it seems that this is related to toolchain error so i followed this
> link : https://cwiki.apache.org/confluence/display/IMPALA/
> Building+native-toolchain+from+scratch+and+using+with+Impala where i
> first builded native-toolchain and then followed step mention in "Using
> custom toolchain with Impala" but while creating symbolic link i got error "ln:
> ‘./gcc-4.9.2’: cannot overwrite directory".
>
> @xinran, i guess post building impala, i will have to perform steps
> mentioned by you.. correct me if i am wrong?
>
>
>
> Regards,
> Anup Tiwari
>
> On Mon, Feb 5, 2018 at 11:14 PM, Xinran Yu Tinney <yu...@gmail.com>
> wrote:
>
>> Hi Anup,
>> Did you also source the config.sh? Run the following before
>> impala-shell.sh:
>>
>> source ~/Impala/bin/impala-config.sh
>>
>> ${IMPALA_HOME}/buildall.sh -notests -build_shared_libs -start_minicluster
>>
>> ${IMPALA_HOME}/bin/start-impala-cluster.py
>>
>>
>> 2018-02-05 10:07 GMT-06:00 Michael Brown <mi...@cloudera.com>:
>>
>>> Hello,
>>>
>>> Please see:
>>>
>>> https://cwiki.apache.org/confluence/display/IMPALA/Bootstrap
>>> ping+an+Impala+Development+Environment+From+Scratch
>>> https://cwiki.apache.org/confluence/display/IMPALA/Building+Impala
>>>
>>>
>>> On Mon, Feb 5, 2018 at 12:09 AM, Anup Tiwari <an...@gmail.com>
>>> wrote:
>>>
>>>> Hi All,
>>>>
>>>> I am new to impala so i am getting errors while i am setting impala.
>>>>
>>>> I have downloaded latest impala release from "
>>>> http://www.apache.org/dyn/closer.cgi?action=download&filena
>>>> me=impala/2.11.0/apache-impala-2.11.0.tar.gz".
>>>> I have extracted this in a folder and followed description mentioned in
>>>> below link :-
>>>>
>>>> https://impala.apache.org/docs/build/html/topics/impala_proc
>>>> esses.html#processes
>>>>
>>>> As mentioned in above link "*To start the Impala state store and
>>>> Impala from the command line or a script, you can either use the service
>>>> command or you can start the daemons directly through the impalad,
>>>> statestored, and catalogd executables.*" So i searched for executable
>>>> and found "start-impalad.sh"(which i think is correct one, let me know if i
>>>> have missed this) but when i am executing this , i am getting below
>>>> *errors* :-
>>>>
>>>> bin/start-impalad.sh: line 89: /admin: No such file or directory
>>>> bin/start-impalad.sh: line 106: /opt/apache-s/apache-impala/ap
>>>> ache-impala-2.11.0/be/build/latest/service/impalad: No such file or
>>>> directory
>>>>
>>>> Also when i executed ./impala-config.sh then i got below *error* :-
>>>>
>>>> Unable to find the lsb_release command. Please ensure it is available
>>>> in your PATH.
>>>> ./impala-config.sh: line 265: return: can only `return' from a function
>>>> or sourced script
>>>> lsb_release command failed, output was: ./impala-config.sh: line 267:
>>>> lsb_release: command not found
>>>> ./impala-config.sh: line 270: return: can only `return' from a function
>>>> or sourced script
>>>>
>>>>
>>>> *Questions :-*
>>>> 1. When we setup Hive/Hadoop/Drill etc then we have certain steps for
>>>> setting HIVE_HOME/HADOOP_HOME/DRILL_HOME and also we specify certain
>>>> setting in conf folder. Do we have similar documentation which can specify
>>>> step by step setup.
>>>>
>>>>
>>>> Regards,
>>>> Anup Tiwari
>>>>
>>>> On Tue, Jan 23, 2018 at 8:56 PM, Jim Apple <jb...@cloudera.com>
>>>> wrote:
>>>>
>>>>> Ubuntu 14.04 and 16.04 are the OSes most Impala developers use, but
>>>>> many Impala users use CentOS 7.
>>>>>
>>>>> Here is part of the docs on tuning:
>>>>> https://impala.apache.org/docs/build/html/topics/impala_perf
>>>>> ormance.html
>>>>>
>>>>> Here is some tuning info that was produced by Cloudera:
>>>>> http://blog.cloudera.com/wp-content/uploads/2017/03/Impala-C
>>>>> ookbook-01-20173.pdf
>>>>>
>>>>> On Tue, Jan 23, 2018 at 6:53 AM, Anup Tiwari <an...@gmail.com>
>>>>> wrote:
>>>>> > Hi Vincent,
>>>>> >
>>>>> > I found this link which says Supported Operating Systems are Ubuntu
>>>>> 14.04,
>>>>> > 16.04 so does this mean it won't work on centos 7.
>>>>> >
>>>>> > Also can you send some link which can help us in tuning as well.
>>>>> >
>>>>> > Regards,
>>>>> > Anup Tiwari
>>>>> >
>>>>> > On Tue, Jan 23, 2018 at 7:26 PM, Vincent Tran <vt...@cloudera.com>
>>>>> wrote:
>>>>> >>
>>>>> >> Hi Anup,
>>>>> >>
>>>>> >> Is this the link you are looking for?
>>>>> >>
>>>>> >> https://impala.apache.org/docs/build/html/topics/impala_inst
>>>>> all.html
>>>>> >>
>>>>> >> On Tue, Jan 23, 2018 at 7:52 AM, Anup Tiwari <
>>>>> anupsdtiwari@gmail.com>
>>>>> >> wrote:
>>>>> >>>
>>>>> >>> Hi All
>>>>> >>>
>>>>> >>> I want to install impala on my production environment(distributed)
>>>>> :-
>>>>> >>>
>>>>> >>> 1. Os Details : CentOS Linux release 7.2.1511 (Core)
>>>>> >>> 2. Apache Hadoop 2.8.0
>>>>> >>> 3. Apache Hive 2.1.1
>>>>> >>>
>>>>> >>> I didn't found any link on mail, please suggest if it is possible
>>>>> or not
>>>>> >>> and if possible then please share some links.
>>>>> >>>
>>>>> >>> Thanks in advance!!
>>>>> >>>
>>>>> >>> Regards,
>>>>> >>> Anup Tiwari
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> --
>>>>> >> Vincent T. Tran
>>>>> >> Customer Operations Engineer
>>>>> >> Cloudera, Inc.
>>>>> >
>>>>> >
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Install Impala on Apache Hadoop and Hive
Posted by Anup Tiwari <an...@gmail.com>.
Hi Zoltan,
I followed the steps mention in https://cwiki.apache.org/
confluence/display/IMPALA/Building+Impala and when i executed below command
, i got error after some time :-
*Build Command :*
${IMPALA_HOME}/buildall.sh -noclean -skiptests -build_shared_libs -format
*Error :-- ----------> Adding thirdparty library cyrus_sasl. <----------*CMake
Error at CMakeLists.txt:136 (message):
Library 'cyrus_sasl' has neither shared nor static library files
Call Stack (most recent call first):
CMakeLists.txt:227 (IMPALA_ADD_THIRDPARTY_LIB)
-- Configuring incomplete, errors occurred!
See also
"/opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/CMakeOutput.log".
See also
"/opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/CMakeError.log".
Error in
/opt/apache-s/apache-impala/apache-impala-2.11.0/bin/make_impala.sh at line
161: cmake . ${CMAKE_ARGS[@]}
*Logs in "CMakeOutput.log" :-*Determining if the function pthread_create
exists in the pthread passed with the following output:
Change Dir:
/opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/CMakeTmp
Run Build Command:"/usr/bin/gmake" "cmTC_ee4f0/fast"
/usr/bin/gmake -f CMakeFiles/cmTC_ee4f0.dir/build.make
CMakeFiles/cmTC_ee4f0.dir/build
gmake[1]: Entering directory
`/opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/CMakeTmp'
Building C object CMakeFiles/cmTC_ee4f0.dir/CheckFunctionExists.c.o
/opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
-DCHECK_FUNCTION_EXISTS=pthread_create -o
CMakeFiles/cmTC_ee4f0.dir/CheckFunctionExists.c.o -c
/opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/cmake-3.8.2-p1/share/cmake-3.8/Modules/CheckFunctionExists.c
Linking C executable cmTC_ee4f0
/opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/cmake-3.8.2-p1/bin/cmake
-E cmake_link_script CMakeFiles/cmTC_ee4f0.dir/link.txt --verbose=1
/opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
-DCHECK_FUNCTION_EXISTS=pthread_create -rdynamic
CMakeFiles/cmTC_ee4f0.dir/CheckFunctionExists.c.o -o cmTC_ee4f0 -lpthread
gmake[1]: Leaving directory
`/opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/CMakeTmp'
*Logs in "CMakeError.log" :-*Determining if the function pthread_create
exists in the pthreads failed with the following output:
Change Dir:
/opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/CMakeTmp
Run Build Command:"/usr/bin/gmake" "cmTC_18bf3/fast"
/usr/bin/gmake -f CMakeFiles/cmTC_18bf3.dir/build.make
CMakeFiles/cmTC_18bf3.dir/build
gmake[1]: Entering directory
`/opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/CMakeTmp'
Building C object CMakeFiles/cmTC_18bf3.dir/CheckFunctionExists.c.o
/opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
-DCHECK_FUNCTION_EXISTS=pthread_create -o
CMakeFiles/cmTC_18bf3.dir/CheckFunctionExists.c.o -c
/opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/cmake-3.8.2-p1/share/cmake-3.8/Modules/CheckFunctionExists.c
Linking C executable cmTC_18bf3
/opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/cmake-3.8.2-p1/bin/cmake
-E cmake_link_script CMakeFiles/cmTC_18bf3.dir/link.txt --verbose=1
/opt/apache-s/apache-impala/apache-impala-2.11.0/toolchain/gcc-4.9.2/bin/gcc
-DCHECK_FUNCTION_EXISTS=pthread_create -rdynamic
CMakeFiles/cmTC_18bf3.dir/CheckFunctionExists.c.o -o cmTC_18bf3 -lpthreads
/usr/bin/ld: cannot find -lpthreads
collect2: error: ld returned 1 exit status
gmake[1]: *** [cmTC_18bf3] Error 1
gmake[1]: Leaving directory
`/opt/apache-s/apache-impala/apache-impala-2.11.0/CMakeFiles/CMakeTmp'
gmake: *** [cmTC_18bf3/fast] Error 2
Also it seems that this is related to toolchain error so i followed this
link :
https://cwiki.apache.org/confluence/display/IMPALA/Building+native-toolchain+from+scratch+and+using+with+Impala
where i first builded native-toolchain and then followed step mention in
"Using custom toolchain with Impala" but while creating symbolic link i got
error "ln: ‘./gcc-4.9.2’: cannot overwrite directory".
@xinran, i guess post building impala, i will have to perform steps
mentioned by you.. correct me if i am wrong?
Regards,
Anup Tiwari
On Mon, Feb 5, 2018 at 11:14 PM, Xinran Yu Tinney <yu...@gmail.com>
wrote:
> Hi Anup,
> Did you also source the config.sh? Run the following before
> impala-shell.sh:
>
> source ~/Impala/bin/impala-config.sh
>
> ${IMPALA_HOME}/buildall.sh -notests -build_shared_libs -start_minicluster
>
> ${IMPALA_HOME}/bin/start-impala-cluster.py
>
>
> 2018-02-05 10:07 GMT-06:00 Michael Brown <mi...@cloudera.com>:
>
>> Hello,
>>
>> Please see:
>>
>> https://cwiki.apache.org/confluence/display/IMPALA/Bootstrap
>> ping+an+Impala+Development+Environment+From+Scratch
>> https://cwiki.apache.org/confluence/display/IMPALA/Building+Impala
>>
>>
>> On Mon, Feb 5, 2018 at 12:09 AM, Anup Tiwari <an...@gmail.com>
>> wrote:
>>
>>> Hi All,
>>>
>>> I am new to impala so i am getting errors while i am setting impala.
>>>
>>> I have downloaded latest impala release from "
>>> http://www.apache.org/dyn/closer.cgi?action=download&filena
>>> me=impala/2.11.0/apache-impala-2.11.0.tar.gz".
>>> I have extracted this in a folder and followed description mentioned in
>>> below link :-
>>>
>>> https://impala.apache.org/docs/build/html/topics/impala_proc
>>> esses.html#processes
>>>
>>> As mentioned in above link "*To start the Impala state store and Impala
>>> from the command line or a script, you can either use the service command
>>> or you can start the daemons directly through the impalad, statestored, and
>>> catalogd executables.*" So i searched for executable and found
>>> "start-impalad.sh"(which i think is correct one, let me know if i have
>>> missed this) but when i am executing this , i am getting below *errors*
>>> :-
>>>
>>> bin/start-impalad.sh: line 89: /admin: No such file or directory
>>> bin/start-impalad.sh: line 106: /opt/apache-s/apache-impala/ap
>>> ache-impala-2.11.0/be/build/latest/service/impalad: No such file or
>>> directory
>>>
>>> Also when i executed ./impala-config.sh then i got below *error* :-
>>>
>>> Unable to find the lsb_release command. Please ensure it is available in
>>> your PATH.
>>> ./impala-config.sh: line 265: return: can only `return' from a function
>>> or sourced script
>>> lsb_release command failed, output was: ./impala-config.sh: line 267:
>>> lsb_release: command not found
>>> ./impala-config.sh: line 270: return: can only `return' from a function
>>> or sourced script
>>>
>>>
>>> *Questions :-*
>>> 1. When we setup Hive/Hadoop/Drill etc then we have certain steps for
>>> setting HIVE_HOME/HADOOP_HOME/DRILL_HOME and also we specify certain
>>> setting in conf folder. Do we have similar documentation which can specify
>>> step by step setup.
>>>
>>>
>>> Regards,
>>> Anup Tiwari
>>>
>>> On Tue, Jan 23, 2018 at 8:56 PM, Jim Apple <jb...@cloudera.com> wrote:
>>>
>>>> Ubuntu 14.04 and 16.04 are the OSes most Impala developers use, but
>>>> many Impala users use CentOS 7.
>>>>
>>>> Here is part of the docs on tuning:
>>>> https://impala.apache.org/docs/build/html/topics/impala_perf
>>>> ormance.html
>>>>
>>>> Here is some tuning info that was produced by Cloudera:
>>>> http://blog.cloudera.com/wp-content/uploads/2017/03/Impala-C
>>>> ookbook-01-20173.pdf
>>>>
>>>> On Tue, Jan 23, 2018 at 6:53 AM, Anup Tiwari <an...@gmail.com>
>>>> wrote:
>>>> > Hi Vincent,
>>>> >
>>>> > I found this link which says Supported Operating Systems are Ubuntu
>>>> 14.04,
>>>> > 16.04 so does this mean it won't work on centos 7.
>>>> >
>>>> > Also can you send some link which can help us in tuning as well.
>>>> >
>>>> > Regards,
>>>> > Anup Tiwari
>>>> >
>>>> > On Tue, Jan 23, 2018 at 7:26 PM, Vincent Tran <vt...@cloudera.com>
>>>> wrote:
>>>> >>
>>>> >> Hi Anup,
>>>> >>
>>>> >> Is this the link you are looking for?
>>>> >>
>>>> >> https://impala.apache.org/docs/build/html/topics/impala_install.html
>>>> >>
>>>> >> On Tue, Jan 23, 2018 at 7:52 AM, Anup Tiwari <anupsdtiwari@gmail.com
>>>> >
>>>> >> wrote:
>>>> >>>
>>>> >>> Hi All
>>>> >>>
>>>> >>> I want to install impala on my production environment(distributed)
>>>> :-
>>>> >>>
>>>> >>> 1. Os Details : CentOS Linux release 7.2.1511 (Core)
>>>> >>> 2. Apache Hadoop 2.8.0
>>>> >>> 3. Apache Hive 2.1.1
>>>> >>>
>>>> >>> I didn't found any link on mail, please suggest if it is possible
>>>> or not
>>>> >>> and if possible then please share some links.
>>>> >>>
>>>> >>> Thanks in advance!!
>>>> >>>
>>>> >>> Regards,
>>>> >>> Anup Tiwari
>>>> >>
>>>> >>
>>>> >>
>>>> >>
>>>> >> --
>>>> >> Vincent T. Tran
>>>> >> Customer Operations Engineer
>>>> >> Cloudera, Inc.
>>>> >
>>>> >
>>>>
>>>
>>>
>>
>
Re: Install Impala on Apache Hadoop and Hive
Posted by Xinran Yu Tinney <yu...@gmail.com>.
Hi Anup,
Did you also source the config.sh? Run the following before
impala-shell.sh:
source ~/Impala/bin/impala-config.sh
${IMPALA_HOME}/buildall.sh -notests -build_shared_libs -start_minicluster
${IMPALA_HOME}/bin/start-impala-cluster.py
2018-02-05 10:07 GMT-06:00 Michael Brown <mi...@cloudera.com>:
> Hello,
>
> Please see:
>
> https://cwiki.apache.org/confluence/display/IMPALA/
> Bootstrapping+an+Impala+Development+Environment+From+Scratch
> https://cwiki.apache.org/confluence/display/IMPALA/Building+Impala
>
>
> On Mon, Feb 5, 2018 at 12:09 AM, Anup Tiwari <an...@gmail.com>
> wrote:
>
>> Hi All,
>>
>> I am new to impala so i am getting errors while i am setting impala.
>>
>> I have downloaded latest impala release from "
>> http://www.apache.org/dyn/closer.cgi?action=download&filena
>> me=impala/2.11.0/apache-impala-2.11.0.tar.gz".
>> I have extracted this in a folder and followed description mentioned in
>> below link :-
>>
>> https://impala.apache.org/docs/build/html/topics/impala_proc
>> esses.html#processes
>>
>> As mentioned in above link "*To start the Impala state store and Impala
>> from the command line or a script, you can either use the service command
>> or you can start the daemons directly through the impalad, statestored, and
>> catalogd executables.*" So i searched for executable and found
>> "start-impalad.sh"(which i think is correct one, let me know if i have
>> missed this) but when i am executing this , i am getting below *errors*
>> :-
>>
>> bin/start-impalad.sh: line 89: /admin: No such file or directory
>> bin/start-impalad.sh: line 106: /opt/apache-s/apache-impala/ap
>> ache-impala-2.11.0/be/build/latest/service/impalad: No such file or
>> directory
>>
>> Also when i executed ./impala-config.sh then i got below *error* :-
>>
>> Unable to find the lsb_release command. Please ensure it is available in
>> your PATH.
>> ./impala-config.sh: line 265: return: can only `return' from a function
>> or sourced script
>> lsb_release command failed, output was: ./impala-config.sh: line 267:
>> lsb_release: command not found
>> ./impala-config.sh: line 270: return: can only `return' from a function
>> or sourced script
>>
>>
>> *Questions :-*
>> 1. When we setup Hive/Hadoop/Drill etc then we have certain steps for
>> setting HIVE_HOME/HADOOP_HOME/DRILL_HOME and also we specify certain
>> setting in conf folder. Do we have similar documentation which can specify
>> step by step setup.
>>
>>
>> Regards,
>> Anup Tiwari
>>
>> On Tue, Jan 23, 2018 at 8:56 PM, Jim Apple <jb...@cloudera.com> wrote:
>>
>>> Ubuntu 14.04 and 16.04 are the OSes most Impala developers use, but
>>> many Impala users use CentOS 7.
>>>
>>> Here is part of the docs on tuning:
>>> https://impala.apache.org/docs/build/html/topics/impala_performance.html
>>>
>>> Here is some tuning info that was produced by Cloudera:
>>> http://blog.cloudera.com/wp-content/uploads/2017/03/Impala-C
>>> ookbook-01-20173.pdf
>>>
>>> On Tue, Jan 23, 2018 at 6:53 AM, Anup Tiwari <an...@gmail.com>
>>> wrote:
>>> > Hi Vincent,
>>> >
>>> > I found this link which says Supported Operating Systems are Ubuntu
>>> 14.04,
>>> > 16.04 so does this mean it won't work on centos 7.
>>> >
>>> > Also can you send some link which can help us in tuning as well.
>>> >
>>> > Regards,
>>> > Anup Tiwari
>>> >
>>> > On Tue, Jan 23, 2018 at 7:26 PM, Vincent Tran <vt...@cloudera.com>
>>> wrote:
>>> >>
>>> >> Hi Anup,
>>> >>
>>> >> Is this the link you are looking for?
>>> >>
>>> >> https://impala.apache.org/docs/build/html/topics/impala_install.html
>>> >>
>>> >> On Tue, Jan 23, 2018 at 7:52 AM, Anup Tiwari <an...@gmail.com>
>>> >> wrote:
>>> >>>
>>> >>> Hi All
>>> >>>
>>> >>> I want to install impala on my production environment(distributed) :-
>>> >>>
>>> >>> 1. Os Details : CentOS Linux release 7.2.1511 (Core)
>>> >>> 2. Apache Hadoop 2.8.0
>>> >>> 3. Apache Hive 2.1.1
>>> >>>
>>> >>> I didn't found any link on mail, please suggest if it is possible or
>>> not
>>> >>> and if possible then please share some links.
>>> >>>
>>> >>> Thanks in advance!!
>>> >>>
>>> >>> Regards,
>>> >>> Anup Tiwari
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Vincent T. Tran
>>> >> Customer Operations Engineer
>>> >> Cloudera, Inc.
>>> >
>>> >
>>>
>>
>>
>
Re: Install Impala on Apache Hadoop and Hive
Posted by Michael Brown <mi...@cloudera.com>.
Hello,
Please see:
https://cwiki.apache.org/confluence/display/IMPALA/Bootstrapping+an+Impala+Development+Environment+From+Scratch
https://cwiki.apache.org/confluence/display/IMPALA/Building+Impala
On Mon, Feb 5, 2018 at 12:09 AM, Anup Tiwari <an...@gmail.com> wrote:
> Hi All,
>
> I am new to impala so i am getting errors while i am setting impala.
>
> I have downloaded latest impala release from "http://www.apache.org/dyn/
> closer.cgi?action=download&filename=impala/2.11.0/apache-
> impala-2.11.0.tar.gz".
> I have extracted this in a folder and followed description mentioned in
> below link :-
>
> https://impala.apache.org/docs/build/html/topics/impala_
> processes.html#processes
>
> As mentioned in above link "*To start the Impala state store and Impala
> from the command line or a script, you can either use the service command
> or you can start the daemons directly through the impalad, statestored, and
> catalogd executables.*" So i searched for executable and found
> "start-impalad.sh"(which i think is correct one, let me know if i have
> missed this) but when i am executing this , i am getting below *errors* :-
>
> bin/start-impalad.sh: line 89: /admin: No such file or directory
> bin/start-impalad.sh: line 106: /opt/apache-s/apache-impala/
> apache-impala-2.11.0/be/build/latest/service/impalad: No such file or
> directory
>
> Also when i executed ./impala-config.sh then i got below *error* :-
>
> Unable to find the lsb_release command. Please ensure it is available in
> your PATH.
> ./impala-config.sh: line 265: return: can only `return' from a function or
> sourced script
> lsb_release command failed, output was: ./impala-config.sh: line 267:
> lsb_release: command not found
> ./impala-config.sh: line 270: return: can only `return' from a function or
> sourced script
>
>
> *Questions :-*
> 1. When we setup Hive/Hadoop/Drill etc then we have certain steps for
> setting HIVE_HOME/HADOOP_HOME/DRILL_HOME and also we specify certain
> setting in conf folder. Do we have similar documentation which can specify
> step by step setup.
>
>
> Regards,
> Anup Tiwari
>
> On Tue, Jan 23, 2018 at 8:56 PM, Jim Apple <jb...@cloudera.com> wrote:
>
>> Ubuntu 14.04 and 16.04 are the OSes most Impala developers use, but
>> many Impala users use CentOS 7.
>>
>> Here is part of the docs on tuning:
>> https://impala.apache.org/docs/build/html/topics/impala_performance.html
>>
>> Here is some tuning info that was produced by Cloudera:
>> http://blog.cloudera.com/wp-content/uploads/2017/03/Impala-
>> Cookbook-01-20173.pdf
>>
>> On Tue, Jan 23, 2018 at 6:53 AM, Anup Tiwari <an...@gmail.com>
>> wrote:
>> > Hi Vincent,
>> >
>> > I found this link which says Supported Operating Systems are Ubuntu
>> 14.04,
>> > 16.04 so does this mean it won't work on centos 7.
>> >
>> > Also can you send some link which can help us in tuning as well.
>> >
>> > Regards,
>> > Anup Tiwari
>> >
>> > On Tue, Jan 23, 2018 at 7:26 PM, Vincent Tran <vt...@cloudera.com>
>> wrote:
>> >>
>> >> Hi Anup,
>> >>
>> >> Is this the link you are looking for?
>> >>
>> >> https://impala.apache.org/docs/build/html/topics/impala_install.html
>> >>
>> >> On Tue, Jan 23, 2018 at 7:52 AM, Anup Tiwari <an...@gmail.com>
>> >> wrote:
>> >>>
>> >>> Hi All
>> >>>
>> >>> I want to install impala on my production environment(distributed) :-
>> >>>
>> >>> 1. Os Details : CentOS Linux release 7.2.1511 (Core)
>> >>> 2. Apache Hadoop 2.8.0
>> >>> 3. Apache Hive 2.1.1
>> >>>
>> >>> I didn't found any link on mail, please suggest if it is possible or
>> not
>> >>> and if possible then please share some links.
>> >>>
>> >>> Thanks in advance!!
>> >>>
>> >>> Regards,
>> >>> Anup Tiwari
>> >>
>> >>
>> >>
>> >>
>> >> --
>> >> Vincent T. Tran
>> >> Customer Operations Engineer
>> >> Cloudera, Inc.
>> >
>> >
>>
>
>
Re: Install Impala on Apache Hadoop and Hive
Posted by Anup Tiwari <an...@gmail.com>.
Hi All,
I am new to impala so i am getting errors while i am setting impala.
I have downloaded latest impala release from "
http://www.apache.org/dyn/closer.cgi?action=download&filename=impala/2.11.0/apache-impala-2.11.0.tar.gz
".
I have extracted this in a folder and followed description mentioned in
below link :-
https://impala.apache.org/docs/build/html/topics/impala_processes.html#processes
As mentioned in above link "*To start the Impala state store and Impala
from the command line or a script, you can either use the service command
or you can start the daemons directly through the impalad, statestored, and
catalogd executables.*" So i searched for executable and found
"start-impalad.sh"(which i think is correct one, let me know if i have
missed this) but when i am executing this , i am getting below *errors* :-
bin/start-impalad.sh: line 89: /admin: No such file or directory
bin/start-impalad.sh: line 106:
/opt/apache-s/apache-impala/apache-impala-2.11.0/be/build/latest/service/impalad:
No such file or directory
Also when i executed ./impala-config.sh then i got below *error* :-
Unable to find the lsb_release command. Please ensure it is available in
your PATH.
./impala-config.sh: line 265: return: can only `return' from a function or
sourced script
lsb_release command failed, output was: ./impala-config.sh: line 267:
lsb_release: command not found
./impala-config.sh: line 270: return: can only `return' from a function or
sourced script
*Questions :-*
1. When we setup Hive/Hadoop/Drill etc then we have certain steps for
setting HIVE_HOME/HADOOP_HOME/DRILL_HOME and also we specify certain
setting in conf folder. Do we have similar documentation which can specify
step by step setup.
Regards,
Anup Tiwari
On Tue, Jan 23, 2018 at 8:56 PM, Jim Apple <jb...@cloudera.com> wrote:
> Ubuntu 14.04 and 16.04 are the OSes most Impala developers use, but
> many Impala users use CentOS 7.
>
> Here is part of the docs on tuning:
> https://impala.apache.org/docs/build/html/topics/impala_performance.html
>
> Here is some tuning info that was produced by Cloudera:
> http://blog.cloudera.com/wp-content/uploads/2017/03/
> Impala-Cookbook-01-20173.pdf
>
> On Tue, Jan 23, 2018 at 6:53 AM, Anup Tiwari <an...@gmail.com>
> wrote:
> > Hi Vincent,
> >
> > I found this link which says Supported Operating Systems are Ubuntu
> 14.04,
> > 16.04 so does this mean it won't work on centos 7.
> >
> > Also can you send some link which can help us in tuning as well.
> >
> > Regards,
> > Anup Tiwari
> >
> > On Tue, Jan 23, 2018 at 7:26 PM, Vincent Tran <vt...@cloudera.com>
> wrote:
> >>
> >> Hi Anup,
> >>
> >> Is this the link you are looking for?
> >>
> >> https://impala.apache.org/docs/build/html/topics/impala_install.html
> >>
> >> On Tue, Jan 23, 2018 at 7:52 AM, Anup Tiwari <an...@gmail.com>
> >> wrote:
> >>>
> >>> Hi All
> >>>
> >>> I want to install impala on my production environment(distributed) :-
> >>>
> >>> 1. Os Details : CentOS Linux release 7.2.1511 (Core)
> >>> 2. Apache Hadoop 2.8.0
> >>> 3. Apache Hive 2.1.1
> >>>
> >>> I didn't found any link on mail, please suggest if it is possible or
> not
> >>> and if possible then please share some links.
> >>>
> >>> Thanks in advance!!
> >>>
> >>> Regards,
> >>> Anup Tiwari
> >>
> >>
> >>
> >>
> >> --
> >> Vincent T. Tran
> >> Customer Operations Engineer
> >> Cloudera, Inc.
> >
> >
>
Re: Install Impala on Apache Hadoop and Hive
Posted by Jim Apple <jb...@cloudera.com>.
Ubuntu 14.04 and 16.04 are the OSes most Impala developers use, but
many Impala users use CentOS 7.
Here is part of the docs on tuning:
https://impala.apache.org/docs/build/html/topics/impala_performance.html
Here is some tuning info that was produced by Cloudera:
http://blog.cloudera.com/wp-content/uploads/2017/03/Impala-Cookbook-01-20173.pdf
On Tue, Jan 23, 2018 at 6:53 AM, Anup Tiwari <an...@gmail.com> wrote:
> Hi Vincent,
>
> I found this link which says Supported Operating Systems are Ubuntu 14.04,
> 16.04 so does this mean it won't work on centos 7.
>
> Also can you send some link which can help us in tuning as well.
>
> Regards,
> Anup Tiwari
>
> On Tue, Jan 23, 2018 at 7:26 PM, Vincent Tran <vt...@cloudera.com> wrote:
>>
>> Hi Anup,
>>
>> Is this the link you are looking for?
>>
>> https://impala.apache.org/docs/build/html/topics/impala_install.html
>>
>> On Tue, Jan 23, 2018 at 7:52 AM, Anup Tiwari <an...@gmail.com>
>> wrote:
>>>
>>> Hi All
>>>
>>> I want to install impala on my production environment(distributed) :-
>>>
>>> 1. Os Details : CentOS Linux release 7.2.1511 (Core)
>>> 2. Apache Hadoop 2.8.0
>>> 3. Apache Hive 2.1.1
>>>
>>> I didn't found any link on mail, please suggest if it is possible or not
>>> and if possible then please share some links.
>>>
>>> Thanks in advance!!
>>>
>>> Regards,
>>> Anup Tiwari
>>
>>
>>
>>
>> --
>> Vincent T. Tran
>> Customer Operations Engineer
>> Cloudera, Inc.
>
>
Re: Install Impala on Apache Hadoop and Hive
Posted by Anup Tiwari <an...@gmail.com>.
Hi Vincent,
I found this link which says Supported Operating Systems are Ubuntu 14.04,
16.04 so does this mean it won't work on centos 7.
Also can you send some link which can help us in tuning as well.
Regards,
Anup Tiwari
On Tue, Jan 23, 2018 at 7:26 PM, Vincent Tran <vt...@cloudera.com> wrote:
> Hi Anup,
>
> Is this the link you are looking for?
>
> https://impala.apache.org/docs/build/html/topics/impala_install.html
>
> On Tue, Jan 23, 2018 at 7:52 AM, Anup Tiwari <an...@gmail.com>
> wrote:
>
>> Hi All
>>
>> I want to install impala on my production environment(distributed) :-
>>
>> *1. Os Details :* CentOS Linux release 7.2.1511 (Core)
>> *2.* Apache Hadoop 2.8.0
>> *3.* Apache Hive 2.1.1
>>
>> I didn't found any link on mail, please suggest if it is possible or not
>> and if possible then please share some links.
>>
>> Thanks in advance!!
>>
>> Regards,
>> Anup Tiwari
>>
>
>
>
> --
> Vincent T. Tran
> Customer Operations Engineer
> Cloudera, Inc.
>
Re: Install Impala on Apache Hadoop and Hive
Posted by Vincent Tran <vt...@cloudera.com>.
Hi Anup,
Is this the link you are looking for?
https://impala.apache.org/docs/build/html/topics/impala_install.html
On Tue, Jan 23, 2018 at 7:52 AM, Anup Tiwari <an...@gmail.com> wrote:
> Hi All
>
> I want to install impala on my production environment(distributed) :-
>
> *1. Os Details :* CentOS Linux release 7.2.1511 (Core)
> *2.* Apache Hadoop 2.8.0
> *3.* Apache Hive 2.1.1
>
> I didn't found any link on mail, please suggest if it is possible or not
> and if possible then please share some links.
>
> Thanks in advance!!
>
> Regards,
> Anup Tiwari
>
--
Vincent T. Tran
Customer Operations Engineer
Cloudera, Inc.