how to distribute cabon.properties file

[ZhuWilliam <al...@gmail.com>]

As we know, cabon.properties is the configuration file of carbondata. However
,for now, the driver and executors load this file from local disk which
means we should distribute this file to all yarn nodes. At the same time ,we
also should configure some items like follows:

--conf
"spark.driver.extraJavaOptions=-Dcarbon.properties.filepath=/home/carbon/carbon.properties"  
--conf
"spark.executor.extraJavaOptions=-Dcarbon.properties.filepath=/home/carbon/carbon.properties"

I think the best way to handle this is using "--files". Eg.

--files /home/carbon/carbon.properties

Then carbondata should try to load from classpath first. 

Also,we hope we can using --conf to overwrite the configurations in
carbon.properties. Eg.

--conf 'carbon.properties.filepath=HDFSLock'

It's easy to implements this function, we can just modify the the code like
follows:

<http://apache-carbondata-mailing-list-archive.1130556.n5.nabble.com/file/n10687/WX20170411-145049%402x.png> 






--
View this message in context: http://apache-carbondata-mailing-list-archive.1130556.n5.nabble.com/how-to-distribute-cabon-properties-file-tp10687.html
Sent from the Apache CarbonData Mailing List archive mailing list archive at Nabble.com.

Re: how to distribute cabon.properties file

[Venkata Gollamudi <g....@gmail.com>]

Hi ZhuWilliam,

Yes we need to distribute carbon.properties to all the executors. In
current carbon implementation you can achieve the same with below
configurations.

in spark-defaults.conf:(in driver)
   #this places carbon.properties in executor path
   spark.yarn.dist.files=/home/carbondriver/carbon.properties
   spark.executor.extraJavaOptions="-Dcarbon.properties.filepath=
carbon.properties"

Regards,
Ramana

On Wed, Apr 12, 2017 at 5:29 PM, QiangCai <qi...@qq.com> wrote:

> +1
>
>
>
> --
> View this message in context: http://apache-carbondata-
> mailing-list-archive.1130556.n5.nabble.com/how-to-
> distribute-cabon-properties-file-tp10687p10869.html
> Sent from the Apache CarbonData Mailing List archive mailing list archive
> at Nabble.com.
>

Re: how to distribute cabon.properties file

[QiangCai <qi...@qq.com>]

+1



--
View this message in context: http://apache-carbondata-mailing-list-archive.1130556.n5.nabble.com/how-to-distribute-cabon-properties-file-tp10687p10869.html
Sent from the Apache CarbonData Mailing List archive mailing list archive at Nabble.com.