You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/11/10 23:32:07 UTC
[GitHub] [hudi] rubenssoto opened a new issue #3966: [SUPPORT] Hudi Sync Remote Hive - EKS
rubenssoto opened a new issue #3966:
URL: https://github.com/apache/hudi/issues/3966
Hello Guys,
We are starting to deploy Spark on Kubernetes and we need to Hudi sync with our Hive metastore, but we are facing a lot of errors like you could see on the following logs:
[log_2.txt](https://github.com/apache/hudi/files/7516536/log_2.txt)
Our hive metastore is in another Pod, but in the same cluster.
We create hive-site.xml, but with spark in kubernetes is not possible to use the $SPARK_HOME/conf, so we use the following path
ENV SPARK_CLASSPATH=$SPARK_HOME/cluster-conf
The Docker File of our Spark Image:
```
# Global args
ARG SPARK_VERSION=3.0.3
ARG HADOOP_VERSION=3.2.0-cloud
ARG SCALA_VERSION=2.12
# Build Spark
FROM openjdk:8-jdk-slim as build
ENV DEBIAN_FRONTEND=noninteractive
ENV M2_HOME=/opt/maven
ENV MAVEN_HOME=/opt/maven
ENV PATH=${PATH}:${M2_HOME}/bin
ARG SPARK_VERSION
ARG HADOOP_VERSION
ARG SCALA_VERSION
RUN apt-get -qq update && \
apt-get -qq upgrade -y && \
apt-get -qq install -y git wget curl && \
wget -q https://dlcdn.apache.org/maven/maven-3/3.8.1/binaries/apache-maven-3.8.1-bin.tar.gz -P /tmp && \
tar xf /tmp/apache-maven-3.8.1-bin.tar.gz -C /opt && \
ln -s /opt/apache-maven-3.8.1 /opt/maven && \
apt-get -qq install gnupg && \
echo "deb https://repo.scala-sbt.org/scalasbt/debian all main" | tee -a /etc/apt/sources.list.d/sbt.list && \
echo "deb https://repo.scala-sbt.org/scalasbt/debian /" | tee -a /etc/apt/sources.list.d/sbt_old.list && \
curl -sL "https://keyserver.ubuntu.com/pks/lookup?op=get&search=0x2EE0EA64E40A89B84B2DF73499E82A75642AC823" | apt-key add && \
apt-get -qq update && \
apt-get -qq install sbt && \
apt-get -qq install -y r-base && \
apt install -y python && \
curl https://bootstrap.pypa.io/pip/2.7/get-pip.py -o get-pip.py && \
python get-pip.py && \
apt install -y python3 python3-pip && \
rm -r /usr/lib/python*/ensurepip && \
pip install --upgrade pip setuptools
RUN cd / && \
git clone https://github.com/apache/spark.git --branch v${SPARK_VERSION} --single-branch && \
cd /spark && \
dev/make-distribution.sh \
--name hadoop-${HADOOP_VERSION}-scala-${SCALA_VERSION} --pip --tgz -DskipTests \
-Phadoop-3.2 \
-Phadoop-cloud \
-Pkubernetes \
-Phive && \
cp spark-${SPARK_VERSION}-bin-hadoop-${HADOOP_VERSION}-scala-${SCALA_VERSION}.tgz /
# Spark image
FROM openjdk:8-jre-slim
ENV BASE_IMAGE openjdk:8-jre-slim
RUN set -ex && \
sed -i 's/http:/https:/g' /etc/apt/sources.list && \
apt-get update && \
ln -s /lib /lib64 && \
apt install -y bash tini libc6 libpam-modules krb5-user libnss3 wget bzip2 && \
rm /bin/sh && \
ln -sv /bin/bash /bin/sh && \
echo "auth required pam_wheel.so use_uid" >> /etc/pam.d/su && \
chgrp root /etc/passwd && chmod ug+rw /etc/passwd && \
rm -rf /var/cache/apt/*
RUN apt-get -qq update && \
apt-get -qq upgrade -y && \
apt-get -qq install -y coreutils \
cron \
initscripts \
git \
curl \
unixodbc-dev \
sasl2-bin \
libsasl2-2 \
libsasl2-modules \
libsasl2-dev \
g++ \
gcc \
libspatialindex-dev
ARG SPARK_VERSION
ARG HADOOP_VERSION
ARG SCALA_VERSION
ENV SPARK_HOME=/opt/spark
ENV SPARK_CONF_DIR=$SPARK_HOME/conf
ENV SPARK_CLASSPATH=$SPARK_HOME/cluster-conf
ENV PYTHONHASHSEED=0
ENV CONDA_DIR=/opt/conda
ENV SHELL=/bin/bash
ENV PATH=$PATH:$SPARK_HOME/bin:$CONDA_DIR/bin
ENV M2_HOME=/opt/maven
ENV MAVEN_HOME=/opt/maven
ENV PATH=${PATH}:${M2_HOME}/bin
ARG MINICONDA_VERSION=4.8.3
ARG MINICONDA_MD5=d63adf39f2c220950a063e0529d4ff74
ARG CONDA_VERSION=4.8.3
ARG PYTHON_VERSION=3.7.8
ARG spark_uid=185
# Install Conda (https://github.com/jupyter/docker-stacks/blob/6d42503c684f3de9b17ce92a6b0c952ef2d1ecd8/base-notebook/Dockerfile#L78-L101)
RUN mkdir -p $CONDA_DIR && \
cd /tmp && \
wget --quiet https://repo.continuum.io/miniconda/Miniconda3-py38_${MINICONDA_VERSION}-Linux-x86_64.sh && \
echo "${MINICONDA_MD5} *Miniconda3-py38_${MINICONDA_VERSION}-Linux-x86_64.sh" | md5sum -c - && \
/bin/bash Miniconda3-py38_${MINICONDA_VERSION}-Linux-x86_64.sh -f -b -p $CONDA_DIR && \
rm Miniconda3-py38_${MINICONDA_VERSION}-Linux-x86_64.sh && \
echo "conda ${CONDA_VERSION}" >> $CONDA_DIR/conda-meta/pinned && \
conda config --system --prepend channels conda-forge && \
conda config --system --set auto_update_conda false && \
conda config --system --set show_channel_urls true && \
conda config --system --set channel_priority strict && \
if [ ! $PYTHON_VERSION = 'default' ]; then conda install --yes python=$PYTHON_VERSION; fi && \
conda list python | grep '^python ' | tr -s ' ' | cut -d '.' -f 1,2 | sed 's/$/.*/' >> $CONDA_DIR/conda-meta/pinned && \
conda install --quiet --yes conda && \
conda install --quiet --yes pip && \
conda install --quiet --yes numpy scipy pandas scikit-learn && \
conda install --quiet --yes -c conda-forge pyarrow && \
conda update --all --quiet --yes && \
conda clean --all -f -y
# Install Spark
COPY --from=build /spark-${SPARK_VERSION}-bin-hadoop-${HADOOP_VERSION}-scala-${SCALA_VERSION}.tgz /
RUN tar -xzf /spark-${SPARK_VERSION}-bin-hadoop-${HADOOP_VERSION}-scala-${SCALA_VERSION}.tgz -C /opt/ && \
ln -s /opt/spark-${SPARK_VERSION}-bin-hadoop-${HADOOP_VERSION}-scala-${SCALA_VERSION} $SPARK_HOME && \
rm -f /spark-${SPARK_VERSION}-bin-hadoop-${HADOOP_VERSION}-scala-${SCALA_VERSION}.tgz && \
mkdir -p $SPARK_HOME/work-dir && \
mkdir -p $SPARK_HOME/spark-warehouse && \
mkdir -p $SPARK_HOME/cluster-conf
COPY config/* $SPARK_CONF_DIR/
COPY config/* $SPARK_HOME/cluster-conf/
COPY entrypoint.sh /opt/
RUN chmod +x /opt/entrypoint.sh
WORKDIR $SPARK_HOME/work-dir
RUN chmod g+w /opt/spark/work-dir
##Install Hudi
RUN wget -q https://dlcdn.apache.org/maven/maven-3/3.8.3/binaries/apache-maven-3.8.3-bin.tar.gz -P /tmp && \
tar xf /tmp/apache-maven-3.8.3-bin.tar.gz -C /opt && \
ln -s /opt/apache-maven-3.8.3 /opt/maven
RUN mvn dependency:copy -Dartifact=org.apache.hudi:hudi-spark3-bundle_2.12:0.9.0 -DoutputDirectory=$SPARK_HOME/jars/
RUN mvn dependency:copy -Dartifact=org.apache.spark:spark-avro_2.12:3.1.2 -DoutputDirectory=$SPARK_HOME/jars/
RUN mvn dependency:copy -Dartifact=org.apache.hudi:hudi-hive-sync:0.9.0 -DoutputDirectory=$SPARK_HOME/jars/
ENTRYPOINT [ "/opt/entrypoint.sh" ]
USER ${spark_uid}
```
Could you help me?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] rubenssoto commented on issue #3966: [SUPPORT] Hudi Sync Remote Hive - EKS
Posted by GitBox <gi...@apache.org>.
rubenssoto commented on issue #3966:
URL: https://github.com/apache/hudi/issues/3966#issuecomment-965906492
I have no problems to create and query tables in hive using spark
```
from os.path import expanduser, join, abspath
from pyspark.sql import SparkSession
from pyspark.sql import Row
# warehouse_location points to the default location for managed databases and tables
warehouse_location = abspath('spark-warehouse')
spark = SparkSession \
.builder \
.appName("Python Spark SQL Hive integration example") \
.enableHiveSupport() \
.config("hive.metastore.uris", "thrift://hive-metastore.hive.svc.cluster.local:9083") \
.enableHiveSupport() \
.getOrCreate()
spark.sql("create table test_metastore(a int,b int);")
spark.sql("select count(1) from test_metastore").show()
```
<img width="1583" alt="Captura de Tela 2021-11-10 às 22 18 28" src="https://user-images.githubusercontent.com/36298331/141219711-a02a167b-c657-4087-a59b-a18f175d07ba.png">
<img width="1519" alt="Captura de Tela 2021-11-10 às 22 17 21" src="https://user-images.githubusercontent.com/36298331/141219715-61f484a1-8a7a-46cd-9fe8-678827aa2108.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] rubenssoto commented on issue #3966: [SUPPORT] Hudi Sync Remote Hive - EKS
Posted by GitBox <gi...@apache.org>.
rubenssoto commented on issue #3966:
URL: https://github.com/apache/hudi/issues/3966#issuecomment-967728548
Hello,
The problem was solved, thank you to Rajesh Mahindra.
I was using Spark Kubernetes Operator, Im trying to configure the URI metastore like a Spark Conf, but the right place is Hadoop Conf.
With these two options:
```
hadoopConf:
hive.metastore.uris: thrift://metastore123:9083
hive.metastore.client.socket.timeout: 1500s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] rubenssoto closed issue #3966: [SUPPORT] Hudi Sync Remote Hive - EKS
Posted by GitBox <gi...@apache.org>.
rubenssoto closed issue #3966:
URL: https://github.com/apache/hudi/issues/3966
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] rubenssoto commented on issue #3966: [SUPPORT] Hudi Sync Remote Hive - EKS
Posted by GitBox <gi...@apache.org>.
rubenssoto commented on issue #3966:
URL: https://github.com/apache/hudi/issues/3966#issuecomment-965908176
My hive-site:
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:postgresql://DATA_BASE_URL:5432/metastore</value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>org.postgresql.Driver</value>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>USER</value>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>PASSWORD</value>
</property>
</configuration>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] rubenssoto edited a comment on issue #3966: [SUPPORT] Hudi Sync Remote Hive - EKS
Posted by GitBox <gi...@apache.org>.
rubenssoto edited a comment on issue #3966:
URL: https://github.com/apache/hudi/issues/3966#issuecomment-965908176
My hive-site:
```
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:postgresql://DATA_BASE_URL:5432/metastore</value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>org.postgresql.Driver</value>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>USER</value>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>PASSWORD</value>
</property>
</configuration>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org