You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by ayan guha <gu...@gmail.com> on 2015/07/01 01:23:06 UTC
Re: Retrieve hadoop conf object from Python API
There is a sc._jsc_ which you can access to get/set hadoop conf.
On Wed, Jul 1, 2015 at 7:41 AM, Richard Ding <pi...@gmail.com> wrote:
> Hi,
>
> I noticed that, in Scala API, one can call hadoopConfiguration on
> SparkContext to retrieve the hadoop configuration object which is very
> handy in modifying certain hadoop properties at runtime. But there is no
> corresponding method in Python API.
>
> Will this method be added to Python API in a later release? In the mean
> time, what will be the work around (of setting hadoop properties at
> runtime) using Python API?
>
> Thanks,
> Richard
>
--
Best Regards,
Ayan Guha
Re: Retrieve hadoop conf object from Python API
Posted by Ted Yu <yu...@gmail.com>.
To close the loop.
This should work:
sc._jsc.hadoopConfiguration
See this method in JavaSparkContext :
def hadoopConfiguration(): Configuration = {
sc.hadoopConfiguration
On Tue, Jun 30, 2015 at 5:52 PM, Ted Yu <yu...@gmail.com> wrote:
> Minor correction:
> It should be sc._jsc
>
> Cheers
>
> On Tue, Jun 30, 2015 at 4:23 PM, ayan guha <gu...@gmail.com> wrote:
>
>> There is a sc._jsc_ which you can access to get/set hadoop conf.
>>
>> On Wed, Jul 1, 2015 at 7:41 AM, Richard Ding <pi...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I noticed that, in Scala API, one can call hadoopConfiguration on
>>> SparkContext to retrieve the hadoop configuration object which is very
>>> handy in modifying certain hadoop properties at runtime. But there is no
>>> corresponding method in Python API.
>>>
>>> Will this method be added to Python API in a later release? In the mean
>>> time, what will be the work around (of setting hadoop properties at
>>> runtime) using Python API?
>>>
>>> Thanks,
>>> Richard
>>>
>>
>>
>>
>> --
>> Best Regards,
>> Ayan Guha
>>
>
>
Re: Retrieve hadoop conf object from Python API
Posted by Ted Yu <yu...@gmail.com>.
Minor correction:
It should be sc._jsc
Cheers
On Tue, Jun 30, 2015 at 4:23 PM, ayan guha <gu...@gmail.com> wrote:
> There is a sc._jsc_ which you can access to get/set hadoop conf.
>
> On Wed, Jul 1, 2015 at 7:41 AM, Richard Ding <pi...@gmail.com> wrote:
>
>> Hi,
>>
>> I noticed that, in Scala API, one can call hadoopConfiguration on
>> SparkContext to retrieve the hadoop configuration object which is very
>> handy in modifying certain hadoop properties at runtime. But there is no
>> corresponding method in Python API.
>>
>> Will this method be added to Python API in a later release? In the mean
>> time, what will be the work around (of setting hadoop properties at
>> runtime) using Python API?
>>
>> Thanks,
>> Richard
>>
>
>
>
> --
> Best Regards,
> Ayan Guha
>