[jira] [Commented] (OOZIE-3112) SparkConfigrationService overwrites properties provided via --properties-file option in SparkAction

["Hadoop QA (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/OOZIE-3112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16245813#comment-16245813 ] 

Hadoop QA commented on OOZIE-3112:
----------------------------------

Testing JIRA OOZIE-3112

Cleaning local git workspace

----------------------------

{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.    {color:green}+1{color} the patch does not introduce any @author tags
.    {color:green}+1{color} the patch does not introduce any tabs
.    {color:green}+1{color} the patch does not introduce any trailing spaces
.    {color:green}+1{color} the patch does not introduce any line longer than 132
.    {color:green}+1{color} the patch does adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.    {color:green}+1{color} the patch does not seem to introduce new RAT warnings
{color:green}+1 JAVADOC{color}
.    {color:green}+1{color} the patch does not seem to introduce new Javadoc warnings
.    {color:red}WARNING{color}: the current HEAD has 77 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.    {color:green}+1{color} HEAD compiles
.    {color:green}+1{color} patch compiles
.    {color:green}+1{color} the patch does not seem to introduce new javac warnings
{color:red}-1{color} There are [3] new bugs found below threshold in total that must be fixed.
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:orange}0{color} There are [1] new bugs found in [core] that would be nice to have fixed.
. You can find the FindBugs diff here: core/findbugs-new.html
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:red}-1{color} There are [3] new bugs found below threshold in [sharelib/spark] that must be fixed.
. You can find the FindBugs diff here (look for the red and orange ones): sharelib/spark/findbugs-new.html
. The most important FindBugs errors are:
. At SparkArgsExtractor.java:[line 376]: Found reliance on default encoding in org.apache.oozie.action.hadoop.SparkArgsExtractor.loadProperties(File, Properties): new java.io.FileReader(File)
. At SparkArgsExtractor.java:[line 364]: Found reliance on default encoding in org.apache.oozie.action.hadoop.SparkArgsExtractor.mergeAndAddPropetiesFile(List, String): new java.io.FileWriter(String)
. At SparkArgsExtractor.java:[line 359]: File(...) reads a file whose location might be specified by user input
. At SparkArgsExtractor.java:[line 147]: At SparkArgsExtractor.java:[line 144]
. At SparkArgsExtractor.java:[line 298]: At SparkArgsExtractor.java:[line 190]
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.    {color:green}+1{color} the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations
.    {color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.    Tests run: 2055
.    Tests failed: 2
.    Tests errors: 1

.    The patch failed the following testcases:

.      testSetupMethodsWithSparkConfiguration(org.apache.oozie.action.hadoop.TestSparkActionExecutor)
.      testQuotedDriverAndExecutorExtraJavaOptionsParsing(org.apache.oozie.action.hadoop.TestSparkArgsExtractor)

.    Tests failing with errors:
.      testJMXInstrumentation(org.apache.oozie.util.TestMetricsInstrumentation)

{color:green}+1 DISTRO{color}
.    {color:green}+1{color} distro tarball builds with the patch 

----------------------------
{color:red}*-1 Overall result, please check the reported -1(s)*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/197/

> SparkConfigrationService overwrites properties provided via --properties-file option in SparkAction
> ---------------------------------------------------------------------------------------------------
>
>                 Key: OOZIE-3112
>                 URL: https://issues.apache.org/jira/browse/OOZIE-3112
>             Project: Oozie
>          Issue Type: Bug
>            Reporter: Peter Cseh
>            Assignee: Peter Cseh
>         Attachments: OOZIE-3112.01.patch
>
>
> SparkConfigurationService injects the config values as --conf to SparkSubmit. This will overwrite propties provided in the --properies-file option which is not the expected behavior.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)