You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@zeppelin.apache.org by kartik sivadas <ka...@outlook.com> on 2016/04/13 12:00:41 UTC

Create worker Instance in Apache Zeppelin running on Spark Standalone

Hi,

     I am running apache zeppelin on a digital ocean droplet where I have created a Spark standalone cluster also. I am trying to extract data from mongoDB using mongo-spark connector by stratio which is taking around 8 min to extract a single collection of 12 million documents from the mongoDB replica sets. So for testing purposes I will like to increase my worker instance. Please help me Thank You.


I am relatively new to spark and zeppelin.


zeppelin-env.sh


export ZEPPELIN_JAVA_OPTS="-Dspark.executor.memory=6g"

export SPARK_HOME="/home/username/spark"
export SPARK_SUBMIT_OPTIONS="--packages com.stratio.datasource:spark-mongodb_2.10:0.10.1"
export MASTER=spark://master-ip:7077
export ZEPPELIN_PORT=8088
export ZEPPELIN_MEM=-Xmx8g
export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64


Thanks.

Karthik

Re: Create worker Instance in Apache Zeppelin running on Spark Standalone

Posted by moon soo Lee <mo...@apache.org>.

Hi,

To increase the number of worker instance, you'll need change your
configuration of Spark standalone cluster.
Please check
http://spark.apache.org/docs/latest/spark-standalone.html#cluster-launch-scripts
 and SPARK_WORKER_INSTANCES might help.

Thanks,
moon

On Wed, Apr 13, 2016 at 11:07 AM kartik sivadas <ka...@outlook.com>
wrote:

> Hi,
>
>      I am running apache zeppelin on a digital ocean droplet where I have
> created a Spark standalone cluster also. I am trying to extract data from
> mongoDB using mongo-spark connector by stratio which is taking around 8 min
> to extract a single collection of 12 million documents from the mongoDB
> replica sets. So for testing purposes I will like to increase my worker
> instance. Please help me Thank You.
>
>
> I am relatively new to spark and zeppelin.
>
>
> zeppelin-env.sh
>
>
> export ZEPPELIN_JAVA_OPTS="-Dspark.executor.memory=6g"
>
> export SPARK_HOME="/home/username/spark"
> export SPARK_SUBMIT_OPTIONS="--packages
> com.stratio.datasource:spark-mongodb_2.10:0.10.1"
> export MASTER=spark://master-ip:7077
> export ZEPPELIN_PORT=8088
> export ZEPPELIN_MEM=-Xmx8g
> export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64
>
>
> Thanks.
>
> Karthik
>