You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by rxue <gi...@git.apache.org> on 2016/06/09 11:42:15 UTC
[GitHub] oozie pull request #17: Oozie Spark action configuration limit issue
GitHub user rxue opened a pull request:
https://github.com/apache/oozie/pull/17
Oozie Spark action configuration limit issue
The current Oozie spark action, can only accept conf without double quotations:
For instance --conf "xxx yyy" is not accepted, but the Spark-submit does accept the format --conf "xxx yyy". This limitation is due to the code in SparkMain.java (https://github.com/mapr/oozie/blob/release-4.2.0-mapr/sharelib/spark/src/main/java/org.apache.oozie.action.hadoop/SparkMain.java) (row 67-73):
String sparkOpts = actionConf.get(SparkActionExecutor.SPARK_OPTS);
if (StringUtils.isNotEmpty(sparkOpts)) {
String[] sparkOptions = sparkOpts.split(DELIM);
for (String opt : sparkOptions) {
sparkArgs.add(opt);
}
}
In our project, I changed it to:
String sparkOpts = actionConf.get(SparkActionExecutor.SPARK_OPTS);
if (StringUtils.isNotEmpty(sparkOpts)) {
String[] sparkOptions = sparkOpts.split(DELIM);
boolean quoted = false;
String quotedOpt = null;
for (String opt : sparkOptions) {
//if (!quoted) sparkArgs.add(opt);
if (!quoted && opt.charAt(0) == '"' && opt.charAt(opt.length()-1) == '"') {
quotedOpt = opt.substring(1, opt.length()-1);
sparkArgs.add(quotedOpt);
}
else if (!quoted && opt.charAt(0) == '"' && opt.charAt(opt.length()-1) != '"') {
quotedOpt = opt;
quoted = true;
}
else if (quoted && opt.charAt(opt.length()-1) != '"') {
quotedOpt += " " + opt;
}
else if (quoted && opt.charAt(opt.length()-1) == '"') {
quotedOpt += " " + opt;
quotedOpt = quotedOpt.substring(1, quotedOpt.length() - 1);
sparkArgs.add(quotedOpt);
quoted = false;
}
else if (!quoted) sparkArgs.add(opt);
}
}
As such, it can accept the double quotation marks.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mapr/oozie release-4.2.0-mapr
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/oozie/pull/17.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #17
----
commit b4ad510029679de3503d1b0d8dfb2d975fd08928
Author: Shwetha GS <ss...@hortonworks.com>
Date: 2015-05-18T06:57:14Z
changed version from 4.2.0-SNAPSHOT to 4.2.0
commit ce3cc4b6ce6dc08250ec59612d23414eccb54666
Author: Bowen Zhang <bo...@yahoo.com>
Date: 2015-05-18T21:44:56Z
OOZIE-2236 Need to package hive-hcatalog-server-extensions.jar in the hcatalog sharelib (venkatnrangan via bzhang)
commit c0148e2d38718c2747f4cb3364d6eb51f7f88200
Author: Shwetha GS <ss...@hortonworks.com>
Date: 2015-05-20T06:31:44Z
OOZIE-1993 Rerun fails during join in certain condition (shwethags)
commit 791ec93ae62e68c552663b49461cf635b9898d83
Author: Shwetha GS <ss...@hortonworks.com>
Date: 2015-05-20T06:59:14Z
OOZIE-2027 Disable re-runs using the workflow directly if it has a parent (jaydeepvishwakarma via shwethags) - patch2
commit 545b37c187498a612f3df876ddae714726df82ac
Author: Shwetha GS <ss...@hortonworks.com>
Date: 2015-05-22T05:45:35Z
OOZIE-1963 Create a Hive Server 2 example (qwertymaniac via shwethags)
commit 46402ac45577fcbe946ab8805ce471f83d1479dd
Author: Shwetha GS <ss...@hortonworks.com>
Date: 2015-05-22T06:04:57Z
OOZIE-2240 add configuration to disable email attachment support (egashira via shwethags)
commit 59f1142357b1ec0da1a6f26d939fbab0c933ca04
Author: ssvinarchuk <ss...@cybervisiontech.com>
Date: 2015-06-16T12:48:15Z
MapR patches applied
commit 62745d0a04a67108308ea35e1c3831dc132ce0f2
Author: ssvinarchuk <ss...@cybervisiontech.com>
Date: 2015-06-16T12:57:09Z
MAPR-14319 Update addtowar script to include FCS art
commit dc513f545ddd5128c300d00a7ca4758e7ab8ec56
Author: ssvinarchuk <ss...@cybervisiontech.com>
Date: 2015-06-16T12:58:03Z
Add condition to find hadoopCoreJar only for hadoop 1
commit df28660b13547cc89796a89047359d76bcae73b5
Author: ssvinarchuk <ss...@cybervisiontech.com>
Date: 2015-06-16T12:59:09Z
Fix set hadoop home
commit bb892a642ff0dee3712714fd33ae166d1acfca1f
Author: ssvinarchuk <ss...@cybervisiontech.com>
Date: 2015-06-16T13:01:47Z
MAPR-14035 - Oozie job mapper launch fails due to null
commit ce5ae5b70c3491520a34f5c327d483394e66a7b6
Author: ssvinarchuk <ss...@cybervisiontech.com>
Date: 2015-06-16T13:15:26Z
Make oozie work with hadoop version read from the cluster
commit ad09859d4882651f3a353c5ed2598786f20491e2
Author: ssvinarchuk <ss...@cybervisiontech.com>
Date: 2015-06-16T13:17:18Z
Fixed the check for adding both jars
commit 27a778dd6c581741ed6f70da1e05ed5862fd542a
Author: ssvinarchuk <ss...@cybervisiontech.com>
Date: 2015-06-16T13:19:21Z
Changed the keys stored in hadoop_version file hadoop_version file now stores classic_version and yarn_version variables and the old ones are gone. Updating oozie-setup.sh to reflect that change
commit 00252451195f7d41087a66c2476571e02c7aa94e
Author: ssvinarchuk <ss...@cybervisiontech.com>
Date: 2015-06-16T13:23:26Z
Changed oozied.sh to honor the new environment
commit 589cfcd736f8cc6f779be20ff095444f2b446126
Author: ssvinarchuk <ss...@cybervisiontech.com>
Date: 2015-06-16T13:25:32Z
BUG-14830 Added hadoop path to oozie-setup option
commit 07347cce82022708fb6ceb3df59038b5ecb9ae4a
Author: ssvinarchuk <ss...@cybervisiontech.com>
Date: 2015-06-16T13:26:21Z
MAPR-15065 Added service.env default value to be able to choose which M/R to use : MRv1 or YARN
commit 2578e7cdf32ef593374db077de199ad5240d1621
Author: ssvinarchuk <ss...@cybervisiontech.com>
Date: 2015-06-16T13:28:33Z
BUG-14830 Fix problem with "Unknown hadoop version" message
commit 459533decdf6d927ca75392f13a203624d33e1a0
Author: ssvinarchuk <ss...@cybervisiontech.com>
Date: 2015-06-16T13:30:36Z
Exclude hadoop jars from libs
commit 33d99ff90ca4fdfe521304e5573df92233d1ef00
Author: ssvinarchuk <ss...@cybervisiontech.com>
Date: 2015-06-16T13:31:45Z
Copy yarn-site.xml to oozie/conf/hadoop-conf/ if yarn-site.xml include yarn.resourcemanager.hostname
commit 0a00030e311cfafd0b57897c3189b82af65f4ae8
Author: ssvinarchuk <ss...@cybervisiontech.com>
Date: 2015-06-16T13:34:41Z
Fixed path to hadoop conf dir for catalina
commit 5cf0de6d31115bc3138ef5c8a47ff451ba59833e
Author: ssvinarchuk <ss...@cybervisiontech.com>
Date: 2015-06-16T13:47:18Z
Fix oozie build
commit ac613e37c46a13769120117d1830f770f48825fe
Author: ssvinarchuk <ss...@cybervisiontech.com>
Date: 2015-06-16T13:50:00Z
Added common-configuration dependency
commit cfbdc8917ce9c6fce8df412b3c295a6bbe1acaf9
Author: ssvinarchuk <ss...@cybervisiontech.com>
Date: 2015-06-16T13:52:11Z
Added common-io dependency
commit 9f743f54a4d6162fccb558a7fdc899cd3e52983f
Author: ssvinarchuk <ss...@cybervisiontech.com>
Date: 2015-06-16T13:53:01Z
Bug 16541 - copy share lib for oozie
commit eafd795c550b7cc91f69fbe1ececda949895f58b
Author: ssvinarchuk <ss...@cybervisiontech.com>
Date: 2015-06-16T13:54:48Z
Added execute permission to mapr-oozie files
commit 4d295bd12e144ec135403f8778117fd7fa8fadfb
Author: ssvinarchuk <ss...@cybervisiontech.com>
Date: 2015-06-16T13:55:43Z
Bug 17289 - changed sqoop version to 1.4.4
commit 44d91334f1550359ed8c01a53b237db994924ada
Author: ssvinarchuk <ss...@cybervisiontech.com>
Date: 2015-06-16T13:56:41Z
Bug 17311 - removed old share lib
commit fec5864bb16abc9998fa9892227f3b2382563670
Author: ssvinarchuk <ss...@cybervisiontech.com>
Date: 2015-06-16T13:57:14Z
Bug 17311 - removed old share lib
commit cbcaf0158d5b1b0cef1835884fa291b6a4755d71
Author: ssvinarchuk <ss...@cybervisiontech.com>
Date: 2015-06-16T13:59:15Z
Changed spark version
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] oozie pull request #17: no need to change
Posted by rxue <gi...@git.apache.org>.
Github user rxue closed the pull request at:
https://github.com/apache/oozie/pull/17
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---