You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Luke Adolph <ke...@gmail.com> on 2016/07/13 10:57:03 UTC
Dependencies with runing Spark Streaming on Mesos cluster using Python
Hi all:
My spark runs on mesos.I write a spark streaming app using python, code on
GitHub <https://github.com/adolphlwq/linkerProcessorSample>.
The app has dependency "*org.apache.spark:spark-streaming-kafka_2.10:1.6.1*
".
Spark on mesos has two important concepts: Spark Framework and Spark
exector.
I set my exector run in docker image.The docker image Dockerfile
<https://github.com/adolphlwq/linkerProcessorSample/blob/master/docker/Dockerfile>
is
below:
# refer '
> http://spark.apache.org/docs/latest/running-on-mesos.html#spark-properties'
> on 'spark.mesos.executor.docker.image' section
FROM ubuntu:14.04
> WORKDIR /linker
> RUN ln -f -s /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
> #download mesos
> RUN echo "deb http://repos.mesosphere.io/ubuntu/ trusty main" >
> /etc/apt/sources.list.d/mesosphere.list && \
> apt-key adv --keyserver keyserver.ubuntu.com --recv E56151BF && \
> apt-get update && \
> apt-get -y install mesos=0.28.1-2.0.20.ubuntu1404 openjdk-7-jre
> python-pip git vim curl
> RUN git clone https://github.com/adolphlwq/linkerProcessorSample.git && \
> pip install -r linkerProcessorSample/docker/requirements.txt
> RUN curl -fL
> http://archive.apache.org/dist/spark/spark-1.6.0/spark-1.6.0-bin-hadoop2.6.tgz
> | tar xzf - -C /usr/local && \
> apt-get clean
> ENV MESOS_NATIVE_JAVA_LIBRARY=/usr/lib/libmesos.so \
> JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64 \
> SPARK_HOME=/usr/local/spark-1.6.0-bin-hadoop2.6
> ENV PATH=$JAVA_HOME/bin:$PATH
> WORKDIR $SPARK_HOME
When I use below command to submit my app program:
dcos spark run --submit-args='--packages
> org.apache.spark:spark-streaming-kafka_2.10:1.6.1 \
> spark2cassandra.py zk topic' \
> -docker-image=adolphlwq/mesos-for-spark-exector-image:1.6.0.beta
The exector docker container run successfully, but it has no package for
*org.apache.spark:spark-streaming-kafka_2.10:1.6.1*.
The *stderr* om mesos is:
I0713 09:34:52.715551 18124 logging.cpp:188] INFO level logging started!
> I0713 09:34:52.717797 18124 fetcher.cpp:424] Fetcher Info:
> {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/6097419e-c2d0-4e5f-9a91-e5815de640c4-S4","items":[{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"\/home\/ubuntu\/spark2cassandra.py"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"\/root\/.ivy2\/jars\/org.apache.spark_spark-streaming-kafka_2.10-1.6.0.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"\/root\/.ivy2\/jars\/com.101tec_zkclient-0.3.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"\/root\/.ivy2\/jars\/org.apache.kafka_kafka_2.10-0.8.2.1.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"\/root\/.ivy2\/jars\/org.slf4j_slf4j-api-1.7.10.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"\/root\/.ivy2\/jars\/org.spark-project.spark_unused-1.0.0.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"\/root\/.ivy2\/jars\/net.jpountz.lz4_lz4-1.3.0.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"\/root\/.ivy2\/jars\/log4j_log4j-1.2.17.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"\/root\/.ivy2\/jars\/com.yammer.metrics_metrics-core-2.2.0.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"\/root\/.ivy2\/jars\/org.apache.kafka_kafka-clients-0.8.2.1.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"\/root\/.ivy2\/jars\/org.xerial.snappy_snappy-java-1.1.2.jar"}}],"sandbox_directory":"\/tmp\/mesos\/slaves\/6097419e-c2d0-4e5f-9a91-e5815de640c4-S4\/frameworks\/7399b6f7-5dcd-4a9b-9846-e7948d5ffd11-0024\/executors\/driver-20160713093451-0015\/runs\/84419372-9482-4c58-8f87-4ba528b6885c"}
> I0713 09:34:52.719846 18124 fetcher.cpp:379] Fetching URI
> '/home/ubuntu/spark2cassandra.py'
> I0713 09:34:52.719866 18124 fetcher.cpp:250] Fetching directly into the
> sandbox directory
> I0713 09:34:52.719925 18124 fetcher.cpp:187] Fetching URI
> '/home/ubuntu/spark2cassandra.py'
> I0713 09:34:52.719945 18124 fetcher.cpp:167] Copying resource with
> command:cp '/home/ubuntu/spark2cassandra.py'
> '/tmp/mesos/slaves/6097419e-c2d0-4e5f-9a91-e5815de640c4-S4/frameworks/7399b6f7-5dcd-4a9b-9846-e7948d5ffd11-0024/executors/driver-20160713093451-0015/runs/84419372-9482-4c58-8f87-4ba528b6885c/spark2cassandra.py'
> W0713 09:34:52.722587 18124 fetcher.cpp:272] Copying instead of extracting
> resource from URI with 'extract' flag, because it does not seem to be an
> archive: /home/ubuntu/spark2cassandra.py
> I0713 09:34:52.724138 18124 fetcher.cpp:456] Fetched
> '/home/ubuntu/spark2cassandra.py' to
> '/tmp/mesos/slaves/6097419e-c2d0-4e5f-9a91-e5815de640c4-S4/frameworks/7399b6f7-5dcd-4a9b-9846-e7948d5ffd11-0024/executors/driver-20160713093451-0015/runs/84419372-9482-4c58-8f87-4ba528b6885c/spark2cassandra.py'
> I0713 09:34:52.724148 18124 fetcher.cpp:379] Fetching URI
> '/root/.ivy2/jars/org.apache.spark_spark-streaming-kafka_2.10-1.6.0.jar'
> I0713 09:34:52.724153 18124 fetcher.cpp:250] Fetching directly into the
> sandbox directory
> I0713 09:34:52.724162 18124 fetcher.cpp:187] Fetching URI
> '/root/.ivy2/jars/org.apache.spark_spark-streaming-kafka_2.10-1.6.0.jar'
> I0713 09:34:52.724171 18124 fetcher.cpp:167] Copying resource with
> command:cp
> '/root/.ivy2/jars/org.apache.spark_spark-streaming-kafka_2.10-1.6.0.jar'
> '/tmp/mesos/slaves/6097419e-c2d0-4e5f-9a91-e5815de640c4-S4/frameworks/7399b6f7-5dcd-4a9b-9846-e7948d5ffd11-0024/executors/driver-20160713093451-0015/runs/84419372-9482-4c58-8f87-4ba528b6885c/org.apache.spark_spark-streaming-kafka_2.10-1.6.0.jar'
> cp: cannot stat
> '/root/.ivy2/jars/org.apache.spark_spark-streaming-kafka_2.10-1.6.0.jar':
> No such file or directory
> Failed to fetch
> '/root/.ivy2/jars/org.apache.spark_spark-streaming-kafka_2.10-1.6.0.jar':
> Failed to copy with command 'cp
> '/root/.ivy2/jars/org.apache.spark_spark-streaming-kafka_2.10-1.6.0.jar'
May somebody help me to solve the dependencies problem!
Thanks.
--
Thanks & Best Regards
卢文泉 | Adolph Lu
TEL：+86 15651006559
Linker Networks(http://www.linkernetworks.com/)
Re: Dependencies with runing Spark Streaming on Mesos cluster using Python
Posted by Shuai Lin <li...@gmail.com>.
I think there are two options for you:
First you can set `--conf spark.mesos.executor.docker.image=
adolphlwq/mesos-for-spark-exector-image:1.6.0.beta2` in your spark submit
args, so mesos would launch the executor with your custom image.
Or you can remove the `local:` prefix in the --jars flag, this way the
executors would download the jars from your spark driver.
On Wed, Jul 13, 2016 at 9:08 PM, Luke Adolph <ke...@gmail.com> wrote:
> Update:
> I revuild my mesos-exector-image ,I download
> *spark-streaming-kafka_2.10-1.6.0.jar* on *`/linker/jars`*
>
> I change my submit command:
>
> dcos spark run \ --submit-args='--jars
>> local:/linker/jars/spark-streaming-kafka_2.10-1.6.0.jar spark2cassandra.py
>> 10.140.0.14:2181 wlu_spark2cassandra' --docker-image
>> adolphlwq/mesos-for-spark-exector-image:1.6.0.beta2
>
>
> Where I get new stderr output on mesos:
>
>
>
> I only problem is submit the dependency
> spark-streaming-kafka_2.10-1.6.0.jar to worker.
>
> Thanks.
>
>
> 2016-07-13 18:57 GMT+08:00 Luke Adolph <ke...@gmail.com>:
>
>> Hi all:
>> My spark runs on mesos.I write a spark streaming app using python, code
>> on GitHub <https://github.com/adolphlwq/linkerProcessorSample>.
>>
>> The app has dependency "
>> *org.apache.spark:spark-streaming-kafka_2.10:1.6.1*".
>>
>> Spark on mesos has two important concepts: Spark Framework and Spark
>> exector.
>>
>> I set my exector run in docker image.The docker image Dockerfile
>> <https://github.com/adolphlwq/linkerProcessorSample/blob/master/docker/Dockerfile> is
>> below:
>>
>> # refer '
>>> http://spark.apache.org/docs/latest/running-on-mesos.html#spark-properties'
>>> on 'spark.mesos.executor.docker.image' section
>>
>> FROM ubuntu:14.04
>>> WORKDIR /linker
>>> RUN ln -f -s /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
>>> #download mesos
>>> RUN echo "deb http://repos.mesosphere.io/ubuntu/ trusty main" >
>>> /etc/apt/sources.list.d/mesosphere.list && \
>>> apt-key adv --keyserver keyserver.ubuntu.com --recv E56151BF && \
>>> apt-get update && \
>>> apt-get -y install mesos=0.28.1-2.0.20.ubuntu1404 openjdk-7-jre
>>> python-pip git vim curl
>>> RUN git clone https://github.com/adolphlwq/linkerProcessorSample.git &&
>>> \
>>> pip install -r linkerProcessorSample/docker/requirements.txt
>>> RUN curl -fL
>>> http://archive.apache.org/dist/spark/spark-1.6.0/spark-1.6.0-bin-hadoop2.6.tgz
>>> | tar xzf - -C /usr/local && \
>>> apt-get clean
>>> ENV MESOS_NATIVE_JAVA_LIBRARY=/usr/lib/libmesos.so \
>>> JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64 \
>>> SPARK_HOME=/usr/local/spark-1.6.0-bin-hadoop2.6
>>> ENV PATH=$JAVA_HOME/bin:$PATH
>>> WORKDIR $SPARK_HOME
>>
>>
>> When I use below command to submit my app program:
>>
>> dcos spark run --submit-args='--packages
>>> org.apache.spark:spark-streaming-kafka_2.10:1.6.1 \
>>> spark2cassandra.py zk topic' \
>>> -docker-image=adolphlwq/mesos-for-spark-exector-image:1.6.0.beta
>>
>>
>> The exector docker container run successfully, but it has no package for
>> *org.apache.spark:spark-streaming-kafka_2.10:1.6.1*.
>>
>> The *stderr* om mesos is:
>>
>> I0713 09:34:52.715551 18124 logging.cpp:188] INFO level logging started!
>>> I0713 09:34:52.717797 18124 fetcher.cpp:424] Fetcher Info:
>>> {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/6097419e-c2d0-4e5f-9a91-e5815de640c4-S4","items":[{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"\/home\/ubuntu\/spark2cassandra.py"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"\/root\/.ivy2\/jars\/org.apache.spark_spark-streaming-kafka_2.10-1.6.0.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"\/root\/.ivy2\/jars\/com.101tec_zkclient-0.3.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"\/root\/.ivy2\/jars\/org.apache.kafka_kafka_2.10-0.8.2.1.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"\/root\/.ivy2\/jars\/org.slf4j_slf4j-api-1.7.10.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"\/root\/.ivy2\/jars\/org.spark-project.spark_unused-1.0.0.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"\/root\/.ivy2\/jars\/net.jpountz.lz4_lz4-1.3.0.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"\/root\/.ivy2\/jars\/log4j_log4j-1.2.17.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"\/root\/.ivy2\/jars\/com.yammer.metrics_metrics-core-2.2.0.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"\/root\/.ivy2\/jars\/org.apache.kafka_kafka-clients-0.8.2.1.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"\/root\/.ivy2\/jars\/org.xerial.snappy_snappy-java-1.1.2.jar"}}],"sandbox_directory":"\/tmp\/mesos\/slaves\/6097419e-c2d0-4e5f-9a91-e5815de640c4-S4\/frameworks\/7399b6f7-5dcd-4a9b-9846-e7948d5ffd11-0024\/executors\/driver-20160713093451-0015\/runs\/84419372-9482-4c58-8f87-4ba528b6885c"}
>>> I0713 09:34:52.719846 18124 fetcher.cpp:379] Fetching URI
>>> '/home/ubuntu/spark2cassandra.py'
>>> I0713 09:34:52.719866 18124 fetcher.cpp:250] Fetching directly into the
>>> sandbox directory
>>> I0713 09:34:52.719925 18124 fetcher.cpp:187] Fetching URI
>>> '/home/ubuntu/spark2cassandra.py'
>>> I0713 09:34:52.719945 18124 fetcher.cpp:167] Copying resource with
>>> command:cp '/home/ubuntu/spark2cassandra.py'
>>> '/tmp/mesos/slaves/6097419e-c2d0-4e5f-9a91-e5815de640c4-S4/frameworks/7399b6f7-5dcd-4a9b-9846-e7948d5ffd11-0024/executors/driver-20160713093451-0015/runs/84419372-9482-4c58-8f87-4ba528b6885c/spark2cassandra.py'
>>> W0713 09:34:52.722587 18124 fetcher.cpp:272] Copying instead of
>>> extracting resource from URI with 'extract' flag, because it does not seem
>>> to be an archive: /home/ubuntu/spark2cassandra.py
>>> I0713 09:34:52.724138 18124 fetcher.cpp:456] Fetched
>>> '/home/ubuntu/spark2cassandra.py' to
>>> '/tmp/mesos/slaves/6097419e-c2d0-4e5f-9a91-e5815de640c4-S4/frameworks/7399b6f7-5dcd-4a9b-9846-e7948d5ffd11-0024/executors/driver-20160713093451-0015/runs/84419372-9482-4c58-8f87-4ba528b6885c/spark2cassandra.py'
>>> I0713 09:34:52.724148 18124 fetcher.cpp:379] Fetching URI
>>> '/root/.ivy2/jars/org.apache.spark_spark-streaming-kafka_2.10-1.6.0.jar'
>>> I0713 09:34:52.724153 18124 fetcher.cpp:250] Fetching directly into the
>>> sandbox directory
>>> I0713 09:34:52.724162 18124 fetcher.cpp:187] Fetching URI
>>> '/root/.ivy2/jars/org.apache.spark_spark-streaming-kafka_2.10-1.6.0.jar'
>>> I0713 09:34:52.724171 18124 fetcher.cpp:167] Copying resource with
>>> command:cp
>>> '/root/.ivy2/jars/org.apache.spark_spark-streaming-kafka_2.10-1.6.0.jar'
>>> '/tmp/mesos/slaves/6097419e-c2d0-4e5f-9a91-e5815de640c4-S4/frameworks/7399b6f7-5dcd-4a9b-9846-e7948d5ffd11-0024/executors/driver-20160713093451-0015/runs/84419372-9482-4c58-8f87-4ba528b6885c/org.apache.spark_spark-streaming-kafka_2.10-1.6.0.jar'
>>> cp: cannot stat
>>> '/root/.ivy2/jars/org.apache.spark_spark-streaming-kafka_2.10-1.6.0.jar':
>>> No such file or directory
>>> Failed to fetch
>>> '/root/.ivy2/jars/org.apache.spark_spark-streaming-kafka_2.10-1.6.0.jar':
>>> Failed to copy with command 'cp
>>> '/root/.ivy2/jars/org.apache.spark_spark-streaming-kafka_2.10-1.6.0.jar'
>>
>>
>> May somebody help me to solve the dependencies problem!
>>
>> Thanks.
>> --
>> Thanks & Best Regards
>> 卢文泉 | Adolph Lu
>> TEL：+86 15651006559
>> Linker Networks(http://www.linkernetworks.com/)
>>
>
>
>
> --
> Thanks & Best Regards
> 卢文泉 | Adolph Lu
> TEL：+86 15651006559
> Linker Networks(http://www.linkernetworks.com/)
>
Re: Dependencies with runing Spark Streaming on Mesos cluster using Python
Posted by Luke Adolph <ke...@gmail.com>.
Update:
I revuild my mesos-exector-image ,I download
*spark-streaming-kafka_2.10-1.6.0.jar* on *`/linker/jars`*
I change my submit command:
dcos spark run \ --submit-args='--jars
> local:/linker/jars/spark-streaming-kafka_2.10-1.6.0.jar spark2cassandra.py
> 10.140.0.14:2181 wlu_spark2cassandra' --docker-image
> adolphlwq/mesos-for-spark-exector-image:1.6.0.beta2
Where I get new stderr output on mesos:
I only problem is submit the dependency
spark-streaming-kafka_2.10-1.6.0.jar to worker.
Thanks.
2016-07-13 18:57 GMT+08:00 Luke Adolph <ke...@gmail.com>:
> Hi all:
> My spark runs on mesos.I write a spark streaming app using python, code
> on GitHub <https://github.com/adolphlwq/linkerProcessorSample>.
>
> The app has dependency "
> *org.apache.spark:spark-streaming-kafka_2.10:1.6.1*".
>
> Spark on mesos has two important concepts: Spark Framework and Spark
> exector.
>
> I set my exector run in docker image.The docker image Dockerfile
> <https://github.com/adolphlwq/linkerProcessorSample/blob/master/docker/Dockerfile> is
> below:
>
> # refer '
>> http://spark.apache.org/docs/latest/running-on-mesos.html#spark-properties'
>> on 'spark.mesos.executor.docker.image' section
>
> FROM ubuntu:14.04
>> WORKDIR /linker
>> RUN ln -f -s /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
>> #download mesos
>> RUN echo "deb http://repos.mesosphere.io/ubuntu/ trusty main" >
>> /etc/apt/sources.list.d/mesosphere.list && \
>> apt-key adv --keyserver keyserver.ubuntu.com --recv E56151BF && \
>> apt-get update && \
>> apt-get -y install mesos=0.28.1-2.0.20.ubuntu1404 openjdk-7-jre
>> python-pip git vim curl
>> RUN git clone https://github.com/adolphlwq/linkerProcessorSample.git && \
>> pip install -r linkerProcessorSample/docker/requirements.txt
>> RUN curl -fL
>> http://archive.apache.org/dist/spark/spark-1.6.0/spark-1.6.0-bin-hadoop2.6.tgz
>> | tar xzf - -C /usr/local && \
>> apt-get clean
>> ENV MESOS_NATIVE_JAVA_LIBRARY=/usr/lib/libmesos.so \
>> JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64 \
>> SPARK_HOME=/usr/local/spark-1.6.0-bin-hadoop2.6
>> ENV PATH=$JAVA_HOME/bin:$PATH
>> WORKDIR $SPARK_HOME
>
>
> When I use below command to submit my app program:
>
> dcos spark run --submit-args='--packages
>> org.apache.spark:spark-streaming-kafka_2.10:1.6.1 \
>> spark2cassandra.py zk topic' \
>> -docker-image=adolphlwq/mesos-for-spark-exector-image:1.6.0.beta
>
>
> The exector docker container run successfully, but it has no package for
> *org.apache.spark:spark-streaming-kafka_2.10:1.6.1*.
>
> The *stderr* om mesos is:
>
> I0713 09:34:52.715551 18124 logging.cpp:188] INFO level logging started!
>> I0713 09:34:52.717797 18124 fetcher.cpp:424] Fetcher Info:
>> {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/6097419e-c2d0-4e5f-9a91-e5815de640c4-S4","items":[{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"\/home\/ubuntu\/spark2cassandra.py"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"\/root\/.ivy2\/jars\/org.apache.spark_spark-streaming-kafka_2.10-1.6.0.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"\/root\/.ivy2\/jars\/com.101tec_zkclient-0.3.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"\/root\/.ivy2\/jars\/org.apache.kafka_kafka_2.10-0.8.2.1.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"\/root\/.ivy2\/jars\/org.slf4j_slf4j-api-1.7.10.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"\/root\/.ivy2\/jars\/org.spark-project.spark_unused-1.0.0.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"\/root\/.ivy2\/jars\/net.jpountz.lz4_lz4-1.3.0.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"\/root\/.ivy2\/jars\/log4j_log4j-1.2.17.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"\/root\/.ivy2\/jars\/com.yammer.metrics_metrics-core-2.2.0.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"\/root\/.ivy2\/jars\/org.apache.kafka_kafka-clients-0.8.2.1.jar"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"\/root\/.ivy2\/jars\/org.xerial.snappy_snappy-java-1.1.2.jar"}}],"sandbox_directory":"\/tmp\/mesos\/slaves\/6097419e-c2d0-4e5f-9a91-e5815de640c4-S4\/frameworks\/7399b6f7-5dcd-4a9b-9846-e7948d5ffd11-0024\/executors\/driver-20160713093451-0015\/runs\/84419372-9482-4c58-8f87-4ba528b6885c"}
>> I0713 09:34:52.719846 18124 fetcher.cpp:379] Fetching URI
>> '/home/ubuntu/spark2cassandra.py'
>> I0713 09:34:52.719866 18124 fetcher.cpp:250] Fetching directly into the
>> sandbox directory
>> I0713 09:34:52.719925 18124 fetcher.cpp:187] Fetching URI
>> '/home/ubuntu/spark2cassandra.py'
>> I0713 09:34:52.719945 18124 fetcher.cpp:167] Copying resource with
>> command:cp '/home/ubuntu/spark2cassandra.py'
>> '/tmp/mesos/slaves/6097419e-c2d0-4e5f-9a91-e5815de640c4-S4/frameworks/7399b6f7-5dcd-4a9b-9846-e7948d5ffd11-0024/executors/driver-20160713093451-0015/runs/84419372-9482-4c58-8f87-4ba528b6885c/spark2cassandra.py'
>> W0713 09:34:52.722587 18124 fetcher.cpp:272] Copying instead of
>> extracting resource from URI with 'extract' flag, because it does not seem
>> to be an archive: /home/ubuntu/spark2cassandra.py
>> I0713 09:34:52.724138 18124 fetcher.cpp:456] Fetched
>> '/home/ubuntu/spark2cassandra.py' to
>> '/tmp/mesos/slaves/6097419e-c2d0-4e5f-9a91-e5815de640c4-S4/frameworks/7399b6f7-5dcd-4a9b-9846-e7948d5ffd11-0024/executors/driver-20160713093451-0015/runs/84419372-9482-4c58-8f87-4ba528b6885c/spark2cassandra.py'
>> I0713 09:34:52.724148 18124 fetcher.cpp:379] Fetching URI
>> '/root/.ivy2/jars/org.apache.spark_spark-streaming-kafka_2.10-1.6.0.jar'
>> I0713 09:34:52.724153 18124 fetcher.cpp:250] Fetching directly into the
>> sandbox directory
>> I0713 09:34:52.724162 18124 fetcher.cpp:187] Fetching URI
>> '/root/.ivy2/jars/org.apache.spark_spark-streaming-kafka_2.10-1.6.0.jar'
>> I0713 09:34:52.724171 18124 fetcher.cpp:167] Copying resource with
>> command:cp
>> '/root/.ivy2/jars/org.apache.spark_spark-streaming-kafka_2.10-1.6.0.jar'
>> '/tmp/mesos/slaves/6097419e-c2d0-4e5f-9a91-e5815de640c4-S4/frameworks/7399b6f7-5dcd-4a9b-9846-e7948d5ffd11-0024/executors/driver-20160713093451-0015/runs/84419372-9482-4c58-8f87-4ba528b6885c/org.apache.spark_spark-streaming-kafka_2.10-1.6.0.jar'
>> cp: cannot stat
>> '/root/.ivy2/jars/org.apache.spark_spark-streaming-kafka_2.10-1.6.0.jar':
>> No such file or directory
>> Failed to fetch
>> '/root/.ivy2/jars/org.apache.spark_spark-streaming-kafka_2.10-1.6.0.jar':
>> Failed to copy with command 'cp
>> '/root/.ivy2/jars/org.apache.spark_spark-streaming-kafka_2.10-1.6.0.jar'
>
>
> May somebody help me to solve the dependencies problem!
>
> Thanks.
> --
> Thanks & Best Regards
> 卢文泉 | Adolph Lu
> TEL：+86 15651006559
> Linker Networks(http://www.linkernetworks.com/)
>
--
Thanks & Best Regards
卢文泉 | Adolph Lu
TEL：+86 15651006559
Linker Networks(http://www.linkernetworks.com/)