You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Anahita Talebi <an...@gmail.com> on 2017/01/09 12:43:39 UTC

Fwd: Entering the variables in the Argument part in Submit job section to run a spark code on Google Cloud

Dear friends,

I am trying to run a run a spark code on Google cloud using submit job.
https://cloud.google.com/dataproc/docs/tutorials/spark-scala

My question is about the part "argument".
In my spark code, they are some variables that their values are defined in
a shell file (.sh), as following:

--trainFile=small_train.dat \
--testFile=small_test.dat \
--numFeatures=9947 \
--numRounds=100 \


- I have tried to enter only the values and each value in a separate box as
following but it is not working:

data/small_train.dat
data/small_test.dat
9947
100

I have also tried to give the parameters like in this below, but it is not
working neither:
trainFile=small_train.dat
testFile=small_test.dat
numFeatures=9947
numRounds=100

I added the files small_train.dat and small_test.dat in the same bucket
where I saved the .jar file. Let's say if my bucket is named Anahita, I
added spark.jar, small_train.dat and small_test.dat in the bucket "Anahita".


Does anyone know, how I can enter these values in the argument part?

Thanks in advance,
Anahita

Re: Entering the variables in the Argument part in Submit job section to run a spark code on Google Cloud

Posted by Dinko Srkoč <di...@gmail.com>.
Not knowing how the code that handles those arguments look like, I
would, in the "Arguments" field for submitting a dataproc job, put:

--trainFile=gs://Anahita/small_train.dat
--testFile=gs://Anahita/small_test.dat
--numFeatures=9947
--numRounds=100

... providing you still keep those files in the "Anahita" bucket.

Each line in the "Arguments" field ends up as an element of the `args`
argument (an Array) of the method `main`.

Cheers,
Dinko

On 9 January 2017 at 13:43, Anahita Talebi <an...@gmail.com> wrote:
> Dear friends,
>
> I am trying to run a run a spark code on Google cloud using submit job.
> https://cloud.google.com/dataproc/docs/tutorials/spark-scala
>
> My question is about the part "argument".
> In my spark code, they are some variables that their values are defined in a
> shell file (.sh), as following:
>
> --trainFile=small_train.dat \
> --testFile=small_test.dat \
> --numFeatures=9947 \
> --numRounds=100 \
>
>
> - I have tried to enter only the values and each value in a separate box as
> following but it is not working:
>
> data/small_train.dat
> data/small_test.dat
> 9947
> 100
>
> I have also tried to give the parameters like in this below, but it is not
> working neither:
> trainFile=small_train.dat
> testFile=small_test.dat
> numFeatures=9947
> numRounds=100
>
> I added the files small_train.dat and small_test.dat in the same bucket
> where I saved the .jar file. Let's say if my bucket is named Anahita, I
> added spark.jar, small_train.dat and small_test.dat in the bucket "Anahita".
>
>
> Does anyone know, how I can enter these values in the argument part?
>
> Thanks in advance,
> Anahita
>

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org