You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kudu.apache.org by "veto@myridia.com" <ve...@myridia.com> on 2018/12/08 07:48:29 UTC
trying to install kudu from source
i'm trying to install kudu from source.
i use debian9 within a docker and follow the docs on
https://kudu.apache.org/docs/installation.html
the compiling and installation finished without errors,
but the make install did not install the binaries
this it what the installer gave back:
.....
Install the project...
-- Install configuration: "RELEASE"
-- Installing: /usr/lib/x86_64-linux-gnu/libkudu_client.so.0.1.0
-- Installing: /usr/lib/x86_64-linux-gnu/libkudu_client.so.0
-- Installing: /usr/lib/x86_64-linux-gnu/libkudu_client.so
-- Installing: /usr/include/kudu/client/callbacks.h
-- Installing: /usr/include/kudu/client/client.h
-- Installing: /usr/include/kudu/client/row_result.h
-- Installing: /usr/include/kudu/client/scan_batch.h
-- Installing: /usr/include/kudu/client/scan_predicate.h
-- Installing: /usr/include/kudu/client/schema.h
-- Installing: /usr/include/kudu/client/shared_ptr.h
-- Installing: /usr/include/kudu/client/stubs.h
-- Installing: /usr/include/kudu/client/value.h
-- Installing: /usr/include/kudu/client/write_op.h
-- Installing: /usr/include/kudu/client/resource_metrics.h
-- Installing: /usr/include/kudu/common/partial_row.h
-- Installing: /usr/include/kudu/util/kudu_export.h
-- Installing: /usr/include/kudu/util/int128.h
-- Installing: /usr/include/kudu/util/monotime.h
-- Installing: /usr/include/kudu/util/slice.h
-- Installing: /usr/include/kudu/util/status.h
-- Installing: /usr/share/doc/kuduClient/examples/CMakeLists.txt
-- Installing: /usr/share/doc/kuduClient/examples/example.cc
-- Installing: /usr/share/kuduClient/cmake/kuduClientTargets.cmake
-- Installing: /usr/share/kuduClient/cmake/kuduClientTargets-release.cmake
-- Installing: /usr/share/kuduClient/cmake/kuduClientConfig.cmake
-- Munging kudu client targets in
/usr/share/kuduClient/cmake/kuduClientConfig.cmake
-- Munging kudu client targets in
/usr/share/kuduClient/cmake/kuduClientTargets-release.cmake
-- Munging kudu client targets in
/usr/share/kuduClient/cmake/kuduClientTargets.cmake
.....
this is my install/docker script so far:
ROM debian:latest
# Install dependencies
RUN apt-get update && export DEBIAN_FRONTEND=noninteractive && \
apt-get -y install apt-utils \
aptitude \
autoconf \
automake \
curl \
dstat \
emacs24-nox \
flex \
g++ \
gcc \
gdb \
git \
krb5-admin-server \
krb5-kdc \
krb5-user \
libkrb5-dev \
libsasl2-dev \
libsasl2-modules \
libsasl2-modules-gssapi-mit \
libssl-dev \
libtool \
lsb-release \
make \
ntp \
net-tools \
openjdk-8-jdk \
openssl \
patch \
python-dev \
python-pip \
python3-dev \
python3 \
python3-pip \
pkg-config \
python \
rsync \
unzip \
vim-common \
wget
#Install Kudu
#RUN git clone https://github.com/apache/kudu \
user@kudu.apache.orgWORKDIR /
RUN wget https://www-us.apache.org/dist/kudu/1.8.0/apache-kudu-1.8.0.tar.gz
RUN mkdir -p /kudu && tar -xzf apache-kudu-1.8.0.tar.gz -C /kudu
--strip-components=1
RUN ls /
RUN cd /kudu \
&& thirdparty/build-if-necessary.sh
RUN cd /kudu && mkdir -p build/release \
&& cd /kudu/build/release \
&& ../../thirdparty/installed/common/bin/cmake -DCMAKE_BUILD_TYPE=release
-DCMAKE_INSTALL_PREFIX:PATH=/usr ../.. \
&& make -j4
RUN cd /kudu/build/release \
&& make install
Re: trying to install kudu from source
Posted by Todd Lipcon <to...@cloudera.com>.
If you care about image size you might want to strip the resulting binaries
as well -- should get much smaller than 10G.
-Todd
On Mon, Dec 10, 2018 at 12:36 PM veto <ve...@myridia.com> wrote:
> thanks,
>
> yes I figured this out and copied only the files i wanted(~10GB) instead
> of ~50GB total
> finally it seems to work great:
>
> FROM debian:latest
> RUN useradd -M kudu
> # Install dependencies and tools
> RUN apt-get update && export DEBIAN_FRONTEND=noninteractive && \
> apt-get -y install apt-utils \
> aptitude \
> autoconf \
> automake \
> curl \
> dstat \
> emacs24-nox \
> flex \
> g++ \
> gcc \
> gdb \
> git \
> krb5-admin-server \
> krb5-kdc \
> krb5-user \
> libkrb5-dev \
> libsasl2-dev \
> libsasl2-modules \
> libsasl2-modules-gssapi-mit \
> libssl-dev \
> libtool \
> lsb-release \
> make \
> ntp \
> net-tools \
> openjdk-8-jdk \
> openssl \
> patch \
> python-dev \
> python-pip \
> python3-dev \
> python3 \
> python3-pip \
> pkg-config \
> python \
> rsync \
> unzip \
> vim-common \
> wget \
> ack-grep
> #Install Kudu
> WORKDIR /
> RUN git clone https://github.com/apache/kudu \
> && cd /kudu \
> && thirdparty/build-if-necessary.sh
> RUN cd /kudu && mkdir -p build/release \
> && cd /kudu/build/release \
> && ../../thirdparty/installed/common/bin/cmake -DCMAKE_BUILD_TYPE=release -DCMAKE_INSTALL_PREFIX:PATH=/usr ../.. \
> && make -j4 \
> && cd /kudu/build/release \
> && make install \
> && mv /kudu/build/release/bin/kudu* /usr/bin/ \
> && mv /kudu/build/release/lib/* /usr/lib/ \
> && mkdir -p /usr/lib/kudu/www \
> && mv /kudu/www/* /usr/lib/kudu/www/ \
> && rm /kudu -Rf
> #Setup Python Client
> RUN pip3 install setuptools --upgrade \
> && pip3 install cython \
> && pip3 install kudu-python
> #Setup Kudu
> RUN mkdir -m 700 -p /data && chown kudu:kudu /data
> USER root
> VOLUME /data
> COPY startup.sh /
> ENTRYPOINT ["/startup.sh"]
> EXPOSE 7050 7051 8050 8051
>
>
> https://calantas.org/kudu-docker/files/tip/
>
>
> <https://calantas.org/kudu-docker/files/7e402da80acf734cc535fbabf77b68ef6fd40d0c/latest/Dockerfile>
>
> On 11 Dec BE 2561, at 03:12, Adar Lieber-Dembo <ad...@cloudera.com> wrote:
>
> The 'make install' target only installs client headers/libraries; it
> doesn't not install Kudu binaries. We should really update the
> documentation to clarify that (KUDU-1375 describes the misleading
> docs, but nobody has stepped up to fix it yet).
>
> KUDU-1344 tracks additional work for improving the Kudu installation
> experience.
>
>
>
> On Fri, Dec 7, 2018 at 11:48 PM veto@myridia.com <ve...@myridia.com> wrote:
>
>
> i'm trying to install kudu from source.
> i use debian9 within a docker and follow the docs on
> https://kudu.apache.org/docs/installation.html
>
> the compiling and installation finished without errors,
> but the make install did not install the binaries
>
> this it what the installer gave back:
>
>
> .....
> Install the project...
> -- Install configuration: "RELEASE"
> -- Installing: /usr/lib/x86_64-linux-gnu/libkudu_client.so.0.1.0
> -- Installing: /usr/lib/x86_64-linux-gnu/libkudu_client.so.0
> -- Installing: /usr/lib/x86_64-linux-gnu/libkudu_client.so
> -- Installing: /usr/include/kudu/client/callbacks.h
> -- Installing: /usr/include/kudu/client/client.h
> -- Installing: /usr/include/kudu/client/row_result.h
> -- Installing: /usr/include/kudu/client/scan_batch.h
> -- Installing: /usr/include/kudu/client/scan_predicate.h
> -- Installing: /usr/include/kudu/client/schema.h
> -- Installing: /usr/include/kudu/client/shared_ptr.h
> -- Installing: /usr/include/kudu/client/stubs.h
> -- Installing: /usr/include/kudu/client/value.h
> -- Installing: /usr/include/kudu/client/write_op.h
> -- Installing: /usr/include/kudu/client/resource_metrics.h
> -- Installing: /usr/include/kudu/common/partial_row.h
> -- Installing: /usr/include/kudu/util/kudu_export.h
> -- Installing: /usr/include/kudu/util/int128.h
> -- Installing: /usr/include/kudu/util/monotime.h
> -- Installing: /usr/include/kudu/util/slice.h
> -- Installing: /usr/include/kudu/util/status.h
> -- Installing: /usr/share/doc/kuduClient/examples/CMakeLists.txt
> -- Installing: /usr/share/doc/kuduClient/examples/example.cc
> -- Installing: /usr/share/kuduClient/cmake/kuduClientTargets.cmake
> -- Installing: /usr/share/kuduClient/cmake/kuduClientTargets-release.cmake
> -- Installing: /usr/share/kuduClient/cmake/kuduClientConfig.cmake
> -- Munging kudu client targets in
> /usr/share/kuduClient/cmake/kuduClientConfig.cmake
> -- Munging kudu client targets in
> /usr/share/kuduClient/cmake/kuduClientTargets-release.cmake
> -- Munging kudu client targets in
> /usr/share/kuduClient/cmake/kuduClientTargets.cmake
> .....
>
>
> this is my install/docker script so far:
> ROM debian:latest
> # Install dependencies
> RUN apt-get update && export DEBIAN_FRONTEND=noninteractive && \
> apt-get -y install apt-utils \
> aptitude \
> autoconf \
> automake \
> curl \
> dstat \
> emacs24-nox \
> flex \
> g++ \
> gcc \
> gdb \
> git \
> krb5-admin-server \
> krb5-kdc \
> krb5-user \
> libkrb5-dev \
> libsasl2-dev \
> libsasl2-modules \
> libsasl2-modules-gssapi-mit \
> libssl-dev \
> libtool \
> lsb-release \
> make \
> ntp \
> net-tools \
> openjdk-8-jdk \
> openssl \
> patch \
> python-dev \
> python-pip \
> python3-dev \
> python3 \
> python3-pip \
> pkg-config \
> python \
> rsync \
> unzip \
> vim-common \
> wget
>
> #Install Kudu
> #RUN git clone https://github.com/apache/kudu \
> user@kudu.apache.orgWORKDIR /
> RUN wget
> https://www-us.apache.org/dist/kudu/1.8.0/apache-kudu-1.8.0.tar.gz
> RUN mkdir -p /kudu && tar -xzf apache-kudu-1.8.0.tar.gz -C /kudu
> --strip-components=1
> RUN ls /
>
> RUN cd /kudu \
> && thirdparty/build-if-necessary.sh
> RUN cd /kudu && mkdir -p build/release \
> && cd /kudu/build/release \
> && ../../thirdparty/installed/common/bin/cmake -DCMAKE_BUILD_TYPE=release
> -DCMAKE_INSTALL_PREFIX:PATH=/usr ../.. \
> && make -j4
>
> RUN cd /kudu/build/release \
> && make install
>
>
>
>
>
>
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: trying to install kudu from source
Posted by veto <ve...@myridia.com>.
thanks,
yes I figured this out and copied only the files i wanted(~10GB) instead of ~50GB total
finally it seems to work great:
FROM debian:latest
<>
<>RUN useradd -M kudu
<>
<># Install dependencies and tools
<>RUN apt-get update && export DEBIAN_FRONTEND=noninteractive && \
<>apt-get -y install apt-utils \
<>aptitude \
<>autoconf \
<>automake \
<>curl \
<>dstat \
<>emacs24-nox \
<>flex \
<>g++ \
<>gcc \
<>gdb \
<>git \
<>krb5-admin-server \
<>krb5-kdc \
<>krb5-user \
<>libkrb5-dev \
<>libsasl2-dev \
<>libsasl2-modules \
<>libsasl2-modules-gssapi-mit \
<>libssl-dev \
<>libtool \
<>lsb-release \
<>make \
<>ntp \
<>net-tools \
<>openjdk-8-jdk \
<>openssl \
<>patch \
<>python-dev \
<>python-pip \
<>python3-dev \
<>python3 \
<>python3-pip \
<>pkg-config \
<>python \
<>rsync \
<>unzip \
<>vim-common \
<>wget \
<>ack-grep
<>
<>
<>#Install Kudu
<>WORKDIR /
<>RUN git clone https://github.com/apache/kudu \
<>&& cd /kudu \
<>&& thirdparty/build-if-necessary.sh
<>
<>RUN cd /kudu && mkdir -p build/release \
<>&& cd /kudu/build/release \
<>&& ../../thirdparty/installed/common/bin/cmake -DCMAKE_BUILD_TYPE=release -DCMAKE_INSTALL_PREFIX:PATH=/usr ../.. \
<>&& make -j4 \
<>&& cd /kudu/build/release \
<>&& make install \
<>&& mv /kudu/build/release/bin/kudu* /usr/bin/ \
<>&& mv /kudu/build/release/lib/* /usr/lib/ \
<>&& mkdir -p /usr/lib/kudu/www \
<>&& mv /kudu/www/* /usr/lib/kudu/www/ \
<>&& rm /kudu -Rf
<>
<>
<>#Setup Python Client
<>RUN pip3 install setuptools --upgrade \
<>&& pip3 install cython \
<>&& pip3 install kudu-python
<>
<>
<>#Setup Kudu
<>RUN mkdir -m 700 -p /data && chown kudu:kudu /data
<>USER root
<>VOLUME /data
<>
<>COPY startup.sh /
<>ENTRYPOINT ["/startup.sh"]
<>EXPOSE 7050 7051 8050 8051
https://calantas.org/kudu-docker/files/tip/
<https://calantas.org/kudu-docker/files/7e402da80acf734cc535fbabf77b68ef6fd40d0c/latest/Dockerfile>
<>
> On 11 Dec BE 2561, at 03:12, Adar Lieber-Dembo <ad...@cloudera.com> wrote:
>
> The 'make install' target only installs client headers/libraries; it
> doesn't not install Kudu binaries. We should really update the
> documentation to clarify that (KUDU-1375 describes the misleading
> docs, but nobody has stepped up to fix it yet).
>
> KUDU-1344 tracks additional work for improving the Kudu installation experience.
>
>
>
> On Fri, Dec 7, 2018 at 11:48 PM veto@myridia.com <ve...@myridia.com> wrote:
>>
>> i'm trying to install kudu from source.
>> i use debian9 within a docker and follow the docs on
>> https://kudu.apache.org/docs/installation.html
>>
>> the compiling and installation finished without errors,
>> but the make install did not install the binaries
>>
>> this it what the installer gave back:
>>
>>
>> .....
>> Install the project...
>> -- Install configuration: "RELEASE"
>> -- Installing: /usr/lib/x86_64-linux-gnu/libkudu_client.so.0.1.0
>> -- Installing: /usr/lib/x86_64-linux-gnu/libkudu_client.so.0
>> -- Installing: /usr/lib/x86_64-linux-gnu/libkudu_client.so
>> -- Installing: /usr/include/kudu/client/callbacks.h
>> -- Installing: /usr/include/kudu/client/client.h
>> -- Installing: /usr/include/kudu/client/row_result.h
>> -- Installing: /usr/include/kudu/client/scan_batch.h
>> -- Installing: /usr/include/kudu/client/scan_predicate.h
>> -- Installing: /usr/include/kudu/client/schema.h
>> -- Installing: /usr/include/kudu/client/shared_ptr.h
>> -- Installing: /usr/include/kudu/client/stubs.h
>> -- Installing: /usr/include/kudu/client/value.h
>> -- Installing: /usr/include/kudu/client/write_op.h
>> -- Installing: /usr/include/kudu/client/resource_metrics.h
>> -- Installing: /usr/include/kudu/common/partial_row.h
>> -- Installing: /usr/include/kudu/util/kudu_export.h
>> -- Installing: /usr/include/kudu/util/int128.h
>> -- Installing: /usr/include/kudu/util/monotime.h
>> -- Installing: /usr/include/kudu/util/slice.h
>> -- Installing: /usr/include/kudu/util/status.h
>> -- Installing: /usr/share/doc/kuduClient/examples/CMakeLists.txt
>> -- Installing: /usr/share/doc/kuduClient/examples/example.cc
>> -- Installing: /usr/share/kuduClient/cmake/kuduClientTargets.cmake
>> -- Installing: /usr/share/kuduClient/cmake/kuduClientTargets-release.cmake
>> -- Installing: /usr/share/kuduClient/cmake/kuduClientConfig.cmake
>> -- Munging kudu client targets in
>> /usr/share/kuduClient/cmake/kuduClientConfig.cmake
>> -- Munging kudu client targets in
>> /usr/share/kuduClient/cmake/kuduClientTargets-release.cmake
>> -- Munging kudu client targets in
>> /usr/share/kuduClient/cmake/kuduClientTargets.cmake
>> .....
>>
>>
>> this is my install/docker script so far:
>> ROM debian:latest
>> # Install dependencies
>> RUN apt-get update && export DEBIAN_FRONTEND=noninteractive && \
>> apt-get -y install apt-utils \
>> aptitude \
>> autoconf \
>> automake \
>> curl \
>> dstat \
>> emacs24-nox \
>> flex \
>> g++ \
>> gcc \
>> gdb \
>> git \
>> krb5-admin-server \
>> krb5-kdc \
>> krb5-user \
>> libkrb5-dev \
>> libsasl2-dev \
>> libsasl2-modules \
>> libsasl2-modules-gssapi-mit \
>> libssl-dev \
>> libtool \
>> lsb-release \
>> make \
>> ntp \
>> net-tools \
>> openjdk-8-jdk \
>> openssl \
>> patch \
>> python-dev \
>> python-pip \
>> python3-dev \
>> python3 \
>> python3-pip \
>> pkg-config \
>> python \
>> rsync \
>> unzip \
>> vim-common \
>> wget
>>
>> #Install Kudu
>> #RUN git clone https://github.com/apache/kudu \
>> user@kudu.apache.orgWORKDIR /
>> RUN wget https://www-us.apache.org/dist/kudu/1.8.0/apache-kudu-1.8.0.tar.gz
>> RUN mkdir -p /kudu && tar -xzf apache-kudu-1.8.0.tar.gz -C /kudu
>> --strip-components=1
>> RUN ls /
>>
>> RUN cd /kudu \
>> && thirdparty/build-if-necessary.sh
>> RUN cd /kudu && mkdir -p build/release \
>> && cd /kudu/build/release \
>> && ../../thirdparty/installed/common/bin/cmake -DCMAKE_BUILD_TYPE=release
>> -DCMAKE_INSTALL_PREFIX:PATH=/usr ../.. \
>> && make -j4
>>
>> RUN cd /kudu/build/release \
>> && make install
>>
>>
>>
>>
>>
Re: trying to install kudu from source
Posted by Adar Lieber-Dembo <ad...@cloudera.com>.
The 'make install' target only installs client headers/libraries; it
doesn't not install Kudu binaries. We should really update the
documentation to clarify that (KUDU-1375 describes the misleading
docs, but nobody has stepped up to fix it yet).
KUDU-1344 tracks additional work for improving the Kudu installation experience.
On Fri, Dec 7, 2018 at 11:48 PM veto@myridia.com <ve...@myridia.com> wrote:
>
> i'm trying to install kudu from source.
> i use debian9 within a docker and follow the docs on
> https://kudu.apache.org/docs/installation.html
>
> the compiling and installation finished without errors,
> but the make install did not install the binaries
>
> this it what the installer gave back:
>
>
> .....
> Install the project...
> -- Install configuration: "RELEASE"
> -- Installing: /usr/lib/x86_64-linux-gnu/libkudu_client.so.0.1.0
> -- Installing: /usr/lib/x86_64-linux-gnu/libkudu_client.so.0
> -- Installing: /usr/lib/x86_64-linux-gnu/libkudu_client.so
> -- Installing: /usr/include/kudu/client/callbacks.h
> -- Installing: /usr/include/kudu/client/client.h
> -- Installing: /usr/include/kudu/client/row_result.h
> -- Installing: /usr/include/kudu/client/scan_batch.h
> -- Installing: /usr/include/kudu/client/scan_predicate.h
> -- Installing: /usr/include/kudu/client/schema.h
> -- Installing: /usr/include/kudu/client/shared_ptr.h
> -- Installing: /usr/include/kudu/client/stubs.h
> -- Installing: /usr/include/kudu/client/value.h
> -- Installing: /usr/include/kudu/client/write_op.h
> -- Installing: /usr/include/kudu/client/resource_metrics.h
> -- Installing: /usr/include/kudu/common/partial_row.h
> -- Installing: /usr/include/kudu/util/kudu_export.h
> -- Installing: /usr/include/kudu/util/int128.h
> -- Installing: /usr/include/kudu/util/monotime.h
> -- Installing: /usr/include/kudu/util/slice.h
> -- Installing: /usr/include/kudu/util/status.h
> -- Installing: /usr/share/doc/kuduClient/examples/CMakeLists.txt
> -- Installing: /usr/share/doc/kuduClient/examples/example.cc
> -- Installing: /usr/share/kuduClient/cmake/kuduClientTargets.cmake
> -- Installing: /usr/share/kuduClient/cmake/kuduClientTargets-release.cmake
> -- Installing: /usr/share/kuduClient/cmake/kuduClientConfig.cmake
> -- Munging kudu client targets in
> /usr/share/kuduClient/cmake/kuduClientConfig.cmake
> -- Munging kudu client targets in
> /usr/share/kuduClient/cmake/kuduClientTargets-release.cmake
> -- Munging kudu client targets in
> /usr/share/kuduClient/cmake/kuduClientTargets.cmake
> .....
>
>
> this is my install/docker script so far:
> ROM debian:latest
> # Install dependencies
> RUN apt-get update && export DEBIAN_FRONTEND=noninteractive && \
> apt-get -y install apt-utils \
> aptitude \
> autoconf \
> automake \
> curl \
> dstat \
> emacs24-nox \
> flex \
> g++ \
> gcc \
> gdb \
> git \
> krb5-admin-server \
> krb5-kdc \
> krb5-user \
> libkrb5-dev \
> libsasl2-dev \
> libsasl2-modules \
> libsasl2-modules-gssapi-mit \
> libssl-dev \
> libtool \
> lsb-release \
> make \
> ntp \
> net-tools \
> openjdk-8-jdk \
> openssl \
> patch \
> python-dev \
> python-pip \
> python3-dev \
> python3 \
> python3-pip \
> pkg-config \
> python \
> rsync \
> unzip \
> vim-common \
> wget
>
> #Install Kudu
> #RUN git clone https://github.com/apache/kudu \
> user@kudu.apache.orgWORKDIR /
> RUN wget https://www-us.apache.org/dist/kudu/1.8.0/apache-kudu-1.8.0.tar.gz
> RUN mkdir -p /kudu && tar -xzf apache-kudu-1.8.0.tar.gz -C /kudu
> --strip-components=1
> RUN ls /
>
> RUN cd /kudu \
> && thirdparty/build-if-necessary.sh
> RUN cd /kudu && mkdir -p build/release \
> && cd /kudu/build/release \
> && ../../thirdparty/installed/common/bin/cmake -DCMAKE_BUILD_TYPE=release
> -DCMAKE_INSTALL_PREFIX:PATH=/usr ../.. \
> && make -j4
>
> RUN cd /kudu/build/release \
> && make install
>
>
>
>
>