You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@phoenix.apache.org by "Chinmay Kulkarni (JIRA)" <ji...@apache.org> on 2019/03/14 21:08:00 UTC
[jira] [Updated] (PHOENIX-5197) Use extraOptions to set the
configuration for Spark Workers
[ https://issues.apache.org/jira/browse/PHOENIX-5197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chinmay Kulkarni updated PHOENIX-5197:
--------------------------------------
Description:
We need a standardized way to set configurations for Spark driver and workers when reading and writing data.
Quoting from my offline discussion with [~twdsilva@gmail.com]:
_"if we use phoenixTableAsRDD to create the rdd then we can pass in a config_
_if we use the standard df.write.format() we can't pass in a conf_
_maybe the best was to do this is to put config in the option map when we call read/write .format and then set those in the config"_
was:
We need a standardized way to set configurations for Spark driver and workers when reading and writing data.
Quoting from my offline discussion with [~twdsilva@gmail.com]:
"_if we use phoenixTableAsRDD to create the rdd then we can pass in a config
if we use the standard df.write.format() we can't pass in a conf
maybe the best was to do this is to put config in the option map when we call read/write .format and then set those in the config _"
> Use extraOptions to set the configuration for Spark Workers
> -----------------------------------------------------------
>
> Key: PHOENIX-5197
> URL: https://issues.apache.org/jira/browse/PHOENIX-5197
> Project: Phoenix
> Issue Type: Improvement
> Affects Versions: 5.0.0, 4.15.0
> Reporter: Chinmay Kulkarni
> Assignee: Chinmay Kulkarni
> Priority: Major
>
> We need a standardized way to set configurations for Spark driver and workers when reading and writing data.
> Quoting from my offline discussion with [~twdsilva@gmail.com]:
> _"if we use phoenixTableAsRDD to create the rdd then we can pass in a config_
> _if we use the standard df.write.format() we can't pass in a conf_
> _maybe the best was to do this is to put config in the option map when we call read/write .format and then set those in the config"_
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)