You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by hnahak <ha...@gmail.com> on 2015/02/22 23:28:25 UTC

How to send user variables from Spark client to custom InputFormat or RecordReader ?

Hi, 

I have written custom InputFormat and RecordReader for Spark, I need  to use
user variables from spark client program. 

I added them in SparkConf 

 val sparkConf = new
SparkConf().setAppName(args(0)).set("developer","MyName")

*and in InputFormat class*

	protected boolean isSplitable(JobContext context, Path filename) {
		
		System.out.println("######################################### Developer "
+ context.getConfiguration().get("developer") );
		return false;
	}

but its return me *null* , is there any way I can pass user variables to my
custom code? 

Thanks !!    



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-send-user-variables-from-Spark-client-to-custom-InputFormat-or-RecordReader-tp21755.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: How to send user variables from Spark client to custom InputFormat or RecordReader ?

Posted by hnahak <ha...@gmail.com>.

Instead of setting in SparkConf , set it into
SparkContext.hadoopconfiguration.set(key,value) 

and from JobContext extract same key.

--Harihar



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-send-user-variables-from-Spark-client-to-custom-InputFormat-or-RecordReader-tp21755p21759.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: How to send user variables from Spark client to custom InputFormat or RecordReader ?

Posted by haihar nahak <ha...@gmail.com>.

Thanks. I extract hadoop configuration and set a my arbitrary variable and
able to get inside InputFormat from JobContext.configuration

On Mon, Feb 23, 2015 at 12:04 PM, Tom Vacek <mi...@gmail.com> wrote:

> The SparkConf doesn't allow you to set arbitrary variables.  You can use
> SparkContext's HadoopRDD and create a JobConf (with whatever variables you
> want), and then grab them out of the JobConf in your RecordReader.
>
> On Sun, Feb 22, 2015 at 4:28 PM, hnahak <ha...@gmail.com> wrote:
>
>> Hi,
>>
>> I have written custom InputFormat and RecordReader for Spark, I need  to
>> use
>> user variables from spark client program.
>>
>> I added them in SparkConf
>>
>>  val sparkConf = new
>> SparkConf().setAppName(args(0)).set("developer","MyName")
>>
>> *and in InputFormat class*
>>
>>         protected boolean isSplitable(JobContext context, Path filename) {
>>
>>
>> System.out.println("######################################### Developer "
>> + context.getConfiguration().get("developer") );
>>                 return false;
>>         }
>>
>> but its return me *null* , is there any way I can pass user variables to
>> my
>> custom code?
>>
>> Thanks !!
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-send-user-variables-from-Spark-client-to-custom-InputFormat-or-RecordReader-tp21755.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>


-- 
{{{H2N}}}-----(@:

Re: How to send user variables from Spark client to custom InputFormat or RecordReader ?

Posted by Tom Vacek <mi...@gmail.com>.

The SparkConf doesn't allow you to set arbitrary variables.  You can use
SparkContext's HadoopRDD and create a JobConf (with whatever variables you
want), and then grab them out of the JobConf in your RecordReader.

On Sun, Feb 22, 2015 at 4:28 PM, hnahak <ha...@gmail.com> wrote:

> Hi,
>
> I have written custom InputFormat and RecordReader for Spark, I need  to
> use
> user variables from spark client program.
>
> I added them in SparkConf
>
>  val sparkConf = new
> SparkConf().setAppName(args(0)).set("developer","MyName")
>
> *and in InputFormat class*
>
>         protected boolean isSplitable(JobContext context, Path filename) {
>
>
> System.out.println("######################################### Developer "
> + context.getConfiguration().get("developer") );
>                 return false;
>         }
>
> but its return me *null* , is there any way I can pass user variables to my
> custom code?
>
> Thanks !!
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-send-user-variables-from-Spark-client-to-custom-InputFormat-or-RecordReader-tp21755.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>