You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@falcon.apache.org by ve...@apache.org on 2014/11/08 06:32:35 UTC
[1/5] incubator-falcon git commit: FALCON-815 Rename mapBandwidthKB
to mapBandwidth in DistCp. Contributed by Sowmya Ramesh
Repository: incubator-falcon
Updated Branches:
refs/heads/master 81904565a -> 394055a54
FALCON-815 Rename mapBandwidthKB to mapBandwidth in DistCp. Contributed by Sowmya Ramesh
Project: http://git-wip-us.apache.org/repos/asf/incubator-falcon/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-falcon/commit/112f9009
Tree: http://git-wip-us.apache.org/repos/asf/incubator-falcon/tree/112f9009
Diff: http://git-wip-us.apache.org/repos/asf/incubator-falcon/diff/112f9009
Branch: refs/heads/master
Commit: 112f90094df224e47760300bf768d264cbea11ff
Parents: d2a2508
Author: Venkatesh Seetharam <ve...@apache.org>
Authored: Fri Nov 7 20:21:13 2014 -0800
Committer: Venkatesh Seetharam <ve...@apache.org>
Committed: Fri Nov 7 21:33:25 2014 -0800
----------------------------------------------------------------------
CHANGES.txt | 3 +++
.../src/main/resources/hdfs-replication-workflow.xml | 4 ++--
.../src/main/resources/hdfs-replication.properties | 4 ++--
common/src/main/resources/runtime.properties | 2 +-
docs/src/site/twiki/EntitySpecification.twiki | 4 ++--
.../falcon/oozie/feed/FeedReplicationCoordinatorBuilder.java | 4 ++--
oozie/src/main/resources/action/feed/replication-action.xml | 4 ++--
.../apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java | 4 ++--
oozie/src/test/resources/feed/fs-replication-feed.xml | 2 +-
.../java/org/apache/falcon/replication/FeedReplicator.java | 6 +++---
.../java/org/apache/falcon/replication/FeedReplicatorTest.java | 6 +++---
11 files changed, 23 insertions(+), 20 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/112f9009/CHANGES.txt
----------------------------------------------------------------------
diff --git a/CHANGES.txt b/CHANGES.txt
index aaea457..7abc820 100755
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -129,6 +129,9 @@ Trunk (Unreleased)
OPTIMIZATIONS
BUG FIXES
+ FALCON-815 Rename mapBandwidthKB to mapBandwidth in DistCp
+ (Sowmya Ramesh via Venkatesh Seetharam)
+
FALCON-595 Improvements to DefaultAuthorizationProvider code
(Raghav Kumar Gautam via Venkatesh Seetharam)
http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/112f9009/addons/recipes/hdfs-replication/src/main/resources/hdfs-replication-workflow.xml
----------------------------------------------------------------------
diff --git a/addons/recipes/hdfs-replication/src/main/resources/hdfs-replication-workflow.xml b/addons/recipes/hdfs-replication/src/main/resources/hdfs-replication-workflow.xml
index 841b9df..4ef8fc7 100644
--- a/addons/recipes/hdfs-replication/src/main/resources/hdfs-replication-workflow.xml
+++ b/addons/recipes/hdfs-replication/src/main/resources/hdfs-replication-workflow.xml
@@ -49,8 +49,8 @@
<arg>-Dmapred.job.priority=${jobPriority}</arg>
<arg>-maxMaps</arg>
<arg>${maxMaps}</arg>
- <arg>-mapBandwidthKB</arg>
- <arg>${mapBandwidthKB}</arg>
+ <arg>-mapBandwidth</arg>
+ <arg>${mapBandwidth}</arg>
<arg>-sourcePaths</arg>
<arg>${nameNode}${drSourceDir}</arg>
<arg>-targetPath</arg>
http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/112f9009/addons/recipes/hdfs-replication/src/main/resources/hdfs-replication.properties
----------------------------------------------------------------------
diff --git a/addons/recipes/hdfs-replication/src/main/resources/hdfs-replication.properties b/addons/recipes/hdfs-replication/src/main/resources/hdfs-replication.properties
index 29d6cab..19b8459 100644
--- a/addons/recipes/hdfs-replication/src/main/resources/hdfs-replication.properties
+++ b/addons/recipes/hdfs-replication/src/main/resources/hdfs-replication.properties
@@ -61,5 +61,5 @@ falcon.recipe.process.property5.name=drTargetCluster
falcon.recipe.process.property5.value=backupCluster
falcon.recipe.process.property6.name=maxMaps
falcon.recipe.process.property6.value=5
-falcon.recipe.process.property7.name=mapBandwidthKB
-falcon.recipe.process.property7.value=102400
+falcon.recipe.process.property7.name=mapBandwidth
+falcon.recipe.process.property7.value=100
http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/112f9009/common/src/main/resources/runtime.properties
----------------------------------------------------------------------
diff --git a/common/src/main/resources/runtime.properties b/common/src/main/resources/runtime.properties
index 185602c..1d35667 100644
--- a/common/src/main/resources/runtime.properties
+++ b/common/src/main/resources/runtime.properties
@@ -27,4 +27,4 @@
*.falcon.parentworkflow.retry.interval.secs=1
*.falcon.replication.workflow.maxmaps=5
-*.falcon.replication.workflow.mapbandwidthKB=102400
+*.falcon.replication.workflow.mapbandwidth=100
http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/112f9009/docs/src/site/twiki/EntitySpecification.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/EntitySpecification.twiki b/docs/src/site/twiki/EntitySpecification.twiki
index 1e603ec..a81a626 100644
--- a/docs/src/site/twiki/EntitySpecification.twiki
+++ b/docs/src/site/twiki/EntitySpecification.twiki
@@ -278,13 +278,13 @@ permission indicates the permission.
<property name="timeout" value="hours(1)"/>
<property name="parallel" value="3"/>
<property name="maxMaps" value="8"/>
- <property name="mapBandwidthKB" value="1024"/>
+ <property name="mapBandwidth" value="1"/>
</properties>
</verbatim>
A key-value pair, which are propagated to the workflow engine. "queueName" and "jobPriority" are special properties
available to user to specify the Hadoop job queue and priority, the same value is used by Falcons launcher job.
"timeout" and "parallel" are other special properties which decides replication instance's timeout value while
-waiting for the feed instance and parallel decides the concurrent replication instances that can run at any given time. "maxMaps" represents the maximum number of maps used during replication. "mapBandwidthKB" represents the bandwidth in KB/s used by each mapper during replication.
+waiting for the feed instance and parallel decides the concurrent replication instances that can run at any given time. "maxMaps" represents the maximum number of maps used during replication. "mapBandwidth" represents the bandwidth in MB/s used by each mapper during replication.
---++ Process Specification
A process defines configuration for a workflow. A workflow is a directed acyclic graph(DAG) which defines the job for the workflow engine. A process definition defines the configurations required to run the workflow job. For example, process defines the frequency at which the workflow should run, the clusters on which the workflow should run, the inputs and outputs for the workflow, how the workflow failures should be handled, how the late inputs should be handled and so on.
http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/112f9009/oozie/src/main/java/org/apache/falcon/oozie/feed/FeedReplicationCoordinatorBuilder.java
----------------------------------------------------------------------
diff --git a/oozie/src/main/java/org/apache/falcon/oozie/feed/FeedReplicationCoordinatorBuilder.java b/oozie/src/main/java/org/apache/falcon/oozie/feed/FeedReplicationCoordinatorBuilder.java
index c5366dc..8f7f01a 100644
--- a/oozie/src/main/java/org/apache/falcon/oozie/feed/FeedReplicationCoordinatorBuilder.java
+++ b/oozie/src/main/java/org/apache/falcon/oozie/feed/FeedReplicationCoordinatorBuilder.java
@@ -73,7 +73,7 @@ public class FeedReplicationCoordinatorBuilder extends OozieCoordinatorBuilder<F
private static final String PARALLEL = "parallel";
private static final String TIMEOUT = "timeout";
private static final String MR_MAX_MAPS = "maxMaps";
- private static final String MR_MAP_BANDWIDTH = "mapBandwidthKB";
+ private static final String MR_MAP_BANDWIDTH = "mapBandwidth";
public FeedReplicationCoordinatorBuilder(Feed entity) {
super(entity, LifeCycle.REPLICATION);
@@ -205,7 +205,7 @@ public class FeedReplicationCoordinatorBuilder extends OozieCoordinatorBuilder<F
}
private String getDefaultMapBandwidth() {
- return RuntimeProperties.get().getProperty("falcon.replication.workflow.mapbandwidthKB", "102400");
+ return RuntimeProperties.get().getProperty("falcon.replication.workflow.mapbandwidth", "100");
}
private String getPathsWithPartitions(Cluster srcCluster, Cluster trgCluster) throws FalconException {
http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/112f9009/oozie/src/main/resources/action/feed/replication-action.xml
----------------------------------------------------------------------
diff --git a/oozie/src/main/resources/action/feed/replication-action.xml b/oozie/src/main/resources/action/feed/replication-action.xml
index 5d154b4..da40b74 100644
--- a/oozie/src/main/resources/action/feed/replication-action.xml
+++ b/oozie/src/main/resources/action/feed/replication-action.xml
@@ -44,8 +44,8 @@
<arg>-Dmapred.job.priority=${jobPriority}</arg>
<arg>-maxMaps</arg>
<arg>${maxMaps}</arg>
- <arg>-mapBandwidthKB</arg>
- <arg>${mapBandwidthKB}</arg>
+ <arg>-mapBandwidth</arg>
+ <arg>${mapBandwidth}</arg>
<arg>-sourcePaths</arg>
<arg>${distcpSourcePaths}</arg>
<arg>-targetPath</arg>
http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/112f9009/oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java
----------------------------------------------------------------------
diff --git a/oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java b/oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java
index 74cc509..42c231f 100644
--- a/oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java
+++ b/oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java
@@ -229,7 +229,7 @@ public class OozieFeedWorkflowBuilderTest extends AbstractTestBase {
Assert.assertEquals(props.get("queueName"), "default");
Assert.assertEquals(props.get("jobPriority"), "NORMAL");
Assert.assertEquals(props.get("maxMaps"), "5");
- Assert.assertEquals(props.get("mapBandwidthKB"), "102400");
+ Assert.assertEquals(props.get("mapBandwidth"), "100");
assertLibExtensions(coord, "replication");
WORKFLOWAPP wf = getWorkflowapp(trgMiniDFS.getFileSystem(), coord);
@@ -336,7 +336,7 @@ public class OozieFeedWorkflowBuilderTest extends AbstractTestBase {
Assert.assertEquals(props.get("distcpTargetPaths"), "${coord:dataOut('output')}");
Assert.assertEquals(props.get("falconFeedStorageType"), Storage.TYPE.FILESYSTEM.name());
Assert.assertEquals(props.get("maxMaps"), "33");
- Assert.assertEquals(props.get("mapBandwidthKB"), "2048");
+ Assert.assertEquals(props.get("mapBandwidth"), "2");
verifyEntityProperties(aFeed, aCluster, srcCluster,
WorkflowExecutionContext.EntityOperations.REPLICATE, props);
http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/112f9009/oozie/src/test/resources/feed/fs-replication-feed.xml
----------------------------------------------------------------------
diff --git a/oozie/src/test/resources/feed/fs-replication-feed.xml b/oozie/src/test/resources/feed/fs-replication-feed.xml
index bada507..0e9065c 100644
--- a/oozie/src/test/resources/feed/fs-replication-feed.xml
+++ b/oozie/src/test/resources/feed/fs-replication-feed.xml
@@ -63,6 +63,6 @@
<schema provider="protobuf" location="/databus/streams_local/click_rr/schema/"/>
<properties>
<property name="maxMaps" value="33" />
- <property name="mapBandwidthKB" value="2048" />
+ <property name="mapBandwidth" value="2" />
</properties>
</feed>
http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/112f9009/replication/src/main/java/org/apache/falcon/replication/FeedReplicator.java
----------------------------------------------------------------------
diff --git a/replication/src/main/java/org/apache/falcon/replication/FeedReplicator.java b/replication/src/main/java/org/apache/falcon/replication/FeedReplicator.java
index 2bf2a98..90ac753 100644
--- a/replication/src/main/java/org/apache/falcon/replication/FeedReplicator.java
+++ b/replication/src/main/java/org/apache/falcon/replication/FeedReplicator.java
@@ -89,8 +89,8 @@ public class FeedReplicator extends Configured implements Tool {
opt.setRequired(true);
options.addOption(opt);
- opt = new Option("mapBandwidthKB", true,
- "bandwidth per map (in KB) to use for this copy");
+ opt = new Option("mapBandwidth", true,
+ "bandwidth per map (in MB) to use for this copy");
opt.setRequired(true);
options.addOption(opt);
@@ -119,7 +119,7 @@ public class FeedReplicator extends Configured implements Tool {
distcpOptions.setSyncFolder(true);
distcpOptions.setBlocking(true);
distcpOptions.setMaxMaps(Integer.valueOf(cmd.getOptionValue("maxMaps")));
- distcpOptions.setMapBandwidth(Integer.valueOf(cmd.getOptionValue("mapBandwidthKB")));
+ distcpOptions.setMapBandwidth(Integer.valueOf(cmd.getOptionValue("mapBandwidth")));
return distcpOptions;
}
http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/112f9009/replication/src/test/java/org/apache/falcon/replication/FeedReplicatorTest.java
----------------------------------------------------------------------
diff --git a/replication/src/test/java/org/apache/falcon/replication/FeedReplicatorTest.java b/replication/src/test/java/org/apache/falcon/replication/FeedReplicatorTest.java
index d64eb61..539d00d 100644
--- a/replication/src/test/java/org/apache/falcon/replication/FeedReplicatorTest.java
+++ b/replication/src/test/java/org/apache/falcon/replication/FeedReplicatorTest.java
@@ -38,14 +38,14 @@ public class FeedReplicatorTest {
* <arg>-update</arg>
* <arg>-blocking</arg><arg>true</arg>
* <arg>-maxMaps</arg><arg>3</arg>
- * <arg>-mapBandwidthKB</arg><arg>4</arg>
+ * <arg>-mapBandwidth</arg><arg>4</arg>
* <arg>-sourcePaths</arg><arg>${distcpSourcePaths}</arg>
* <arg>-targetPath</arg><arg>${distcpTargetPaths}</arg>
*/
final String[] args = {
"true",
"-maxMaps", "3",
- "-mapBandwidthKB", "4096",
+ "-mapBandwidth", "4",
"-sourcePaths", "hdfs://localhost:8020/tmp/",
"-targetPath", "hdfs://localhost1:8020/tmp/",
"-falconFeedStorageType", Storage.TYPE.FILESYSTEM.name(),
@@ -58,7 +58,7 @@ public class FeedReplicatorTest {
List<Path> srcPaths = new ArrayList<Path>();
srcPaths.add(new Path("hdfs://localhost:8020/tmp/"));
Assert.assertEquals(options.getMaxMaps(), 3);
- Assert.assertEquals(options.getMapBandwidth(), 4096);
+ Assert.assertEquals(options.getMapBandwidth(), 4);
Assert.assertEquals(options.getSourcePaths(), srcPaths);
Assert.assertEquals(options.getTargetPath(), new Path("hdfs://localhost1:8020/tmp/"));
}
[3/5] incubator-falcon git commit: FALCON-820 Fix minor nits with
HadoopClientFactory. Contributed by Sowmya Ramesh
Posted by ve...@apache.org.
FALCON-820 Fix minor nits with HadoopClientFactory. Contributed by Sowmya Ramesh
Project: http://git-wip-us.apache.org/repos/asf/incubator-falcon/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-falcon/commit/80b0a16d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-falcon/tree/80b0a16d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-falcon/diff/80b0a16d
Branch: refs/heads/master
Commit: 80b0a16dc5a918f99f74071e7c7ca6a48243cba4
Parents: 112f900
Author: Venkatesh Seetharam <ve...@apache.org>
Authored: Fri Nov 7 20:21:59 2014 -0800
Committer: Venkatesh Seetharam <ve...@apache.org>
Committed: Fri Nov 7 21:33:26 2014 -0800
----------------------------------------------------------------------
CHANGES.txt | 3 +++
.../java/org/apache/falcon/entity/parser/ClusterEntityParser.java | 2 +-
2 files changed, 4 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/80b0a16d/CHANGES.txt
----------------------------------------------------------------------
diff --git a/CHANGES.txt b/CHANGES.txt
index 7abc820..d2635b7 100755
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -129,6 +129,9 @@ Trunk (Unreleased)
OPTIMIZATIONS
BUG FIXES
+ FALCON-820 Fix minor nits with HadoopClientFactory
+ (Sowmya Ramesh via Venkatesh Seetharam)
+
FALCON-815 Rename mapBandwidthKB to mapBandwidth in DistCp
(Sowmya Ramesh via Venkatesh Seetharam)
http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/80b0a16d/common/src/main/java/org/apache/falcon/entity/parser/ClusterEntityParser.java
----------------------------------------------------------------------
diff --git a/common/src/main/java/org/apache/falcon/entity/parser/ClusterEntityParser.java b/common/src/main/java/org/apache/falcon/entity/parser/ClusterEntityParser.java
index b801dec..b3496c4 100644
--- a/common/src/main/java/org/apache/falcon/entity/parser/ClusterEntityParser.java
+++ b/common/src/main/java/org/apache/falcon/entity/parser/ClusterEntityParser.java
@@ -271,7 +271,6 @@ public class ClusterEntityParser extends EntityParser<Cluster> {
throws IOException, ValidationException {
Path locationPath = new Path(location);
- FileStatus fileStatus = fs.getFileStatus(locationPath);
if (!fs.exists(locationPath)) {
throw new ValidationException("Location " + location
+ " for cluster " + clusterName + " must exist.");
@@ -279,6 +278,7 @@ public class ClusterEntityParser extends EntityParser<Cluster> {
// falcon owns this path on each cluster
final String loginUser = UserGroupInformation.getLoginUser().getShortUserName();
+ FileStatus fileStatus = fs.getFileStatus(locationPath);
final String locationOwner = fileStatus.getOwner();
if (!locationOwner.equals(loginUser)) {
LOG.error("Location {} has owner {}, should be the process user {}",
[5/5] incubator-falcon git commit: FALCON-875 Enitiy Summary endpoint
filterBy does not filter entities without pipelines. Contributed by Balu
Vellanki
Posted by ve...@apache.org.
FALCON-875 Enitiy Summary endpoint filterBy does not filter entities without pipelines. Contributed by Balu Vellanki
Project: http://git-wip-us.apache.org/repos/asf/incubator-falcon/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-falcon/commit/394055a5
Tree: http://git-wip-us.apache.org/repos/asf/incubator-falcon/tree/394055a5
Diff: http://git-wip-us.apache.org/repos/asf/incubator-falcon/diff/394055a5
Branch: refs/heads/master
Commit: 394055a544dc3dddf20469f3f9b2c1ee53a208bb
Parents: 2186a36
Author: Venkatesh Seetharam <ve...@apache.org>
Authored: Fri Nov 7 21:32:33 2014 -0800
Committer: Venkatesh Seetharam <ve...@apache.org>
Committed: Fri Nov 7 21:33:26 2014 -0800
----------------------------------------------------------------------
CHANGES.txt | 3 +
.../falcon/resource/AbstractEntityManager.java | 68 ++++++++++++--------
.../falcon/resource/EntityManagerTest.java | 27 +++++---
3 files changed, 61 insertions(+), 37 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/394055a5/CHANGES.txt
----------------------------------------------------------------------
diff --git a/CHANGES.txt b/CHANGES.txt
index d2ca59f..5de597e 100755
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -129,6 +129,9 @@ Trunk (Unreleased)
OPTIMIZATIONS
BUG FIXES
+ FALCON-875 Enitiy Summary endpoint filterBy does not filter entities
+ without pipelines (Balu Vellanki via Venkatesh Seetharam)
+
FALCON-874 Rest API - Instance and Entity orderBy param values should be
case insensitive (Balu Vellanki via Venkatesh Seetharam)
http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/394055a5/prism/src/main/java/org/apache/falcon/resource/AbstractEntityManager.java
----------------------------------------------------------------------
diff --git a/prism/src/main/java/org/apache/falcon/resource/AbstractEntityManager.java b/prism/src/main/java/org/apache/falcon/resource/AbstractEntityManager.java
index ceacb4e..7fce88c 100644
--- a/prism/src/main/java/org/apache/falcon/resource/AbstractEntityManager.java
+++ b/prism/src/main/java/org/apache/falcon/resource/AbstractEntityManager.java
@@ -672,10 +672,8 @@ public abstract class AbstractEntityManager {
private boolean filterEntity(Entity entity, String entityStatus,
Map<String, String> filterByFieldsValues, List<String> filterByTags,
List<String> tags, List<String> pipelines) {
- return !((filterByTags.isEmpty() || !filterEntityByTags(filterByTags, tags))
- && (filterByFieldsValues.isEmpty()
- || !filterEntityByFields(entity, filterByFieldsValues, entityStatus, pipelines)));
-
+ return filterEntityByTags(filterByTags, tags)
+ || filterEntityByFields(entity, filterByFieldsValues, entityStatus, pipelines);
}
protected boolean isEntityAuthorized(Entity entity) {
@@ -693,51 +691,65 @@ public abstract class AbstractEntityManager {
}
private boolean filterEntityByTags(List<String> filterTagsList, List<String> tags) {
- boolean filterEntity = false;
- if (!filterTagsList.isEmpty() && tags.isEmpty()) {
+ if (filterTagsList.isEmpty()) {
+ return false;
+ } else if (tags.isEmpty()) {
return true;
}
+
for (String tag : filterTagsList) {
if (!tags.contains(tag)) {
- filterEntity = true;
- break;
+ return true;
}
}
- return filterEntity;
+ return false;
}
private boolean filterEntityByFields(Entity entity, Map<String, String> filterKeyVals,
String status, List<String> pipelines) {
+ if (filterKeyVals.isEmpty()) {
+ return false;
+ }
+
for (Map.Entry<String, String> pair : filterKeyVals.entrySet()) {
- String filterValue = pair.getValue();
- if (StringUtils.isEmpty(filterValue)) {
- continue; // nothing to filter
- }
EntityList.EntityFilterByFields filter =
EntityList.EntityFilterByFields.valueOf(pair.getKey().toUpperCase());
- switch (filter) {
-
- case TYPE:
- return !entity.getEntityType().toString().equalsIgnoreCase(filterValue);
+ if (isEntityFiltered(entity, filter, pair, status, pipelines)) {
+ return true;
+ }
+ }
- case NAME:
- return !entity.getName().equalsIgnoreCase(filterValue);
+ return false;
+ }
- case STATUS:
- return !status.equalsIgnoreCase(filterValue);
+ private boolean isEntityFiltered(Entity entity, EntityList.EntityFilterByFields filter,
+ Map.Entry<String, String> pair,
+ String status, List<String> pipelines) {
+ switch (filter) {
+ case TYPE:
+ return !entity.getEntityType().toString().equalsIgnoreCase(pair.getValue());
- case PIPELINES:
- return entity.getEntityType().equals(EntityType.PROCESS) && !pipelines.contains(filterValue);
+ case NAME:
+ return !entity.getName().equalsIgnoreCase(pair.getValue());
- case CLUSTER:
- return !EntityUtil.getClustersDefined(entity).contains(filterValue);
+ case STATUS:
+ return !status.equalsIgnoreCase(pair.getValue());
- default:
- break;
+ case PIPELINES:
+ if (!entity.getEntityType().equals(EntityType.PROCESS)) {
+ throw FalconWebException.newException(
+ "Invalid filterBy key for non process entities " + pair.getKey(),
+ Response.Status.BAD_REQUEST);
}
+ return !pipelines.contains(pair.getValue());
+
+ case CLUSTER:
+ return !EntityUtil.getClustersDefined(entity).contains(pair.getValue());
+
+ default:
+ return false;
}
- return false;
}
private List<Entity> sortEntities(List<Entity> entities, String orderBy, String sortOrder) {
http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/394055a5/prism/src/test/java/org/apache/falcon/resource/EntityManagerTest.java
----------------------------------------------------------------------
diff --git a/prism/src/test/java/org/apache/falcon/resource/EntityManagerTest.java b/prism/src/test/java/org/apache/falcon/resource/EntityManagerTest.java
index 2244e5f..e39947a 100644
--- a/prism/src/test/java/org/apache/falcon/resource/EntityManagerTest.java
+++ b/prism/src/test/java/org/apache/falcon/resource/EntityManagerTest.java
@@ -17,6 +17,7 @@
*/
package org.apache.falcon.resource;
+import org.apache.commons.lang.StringUtils;
import org.apache.falcon.FalconWebException;
import org.apache.falcon.entity.v0.Entity;
import org.apache.falcon.entity.v0.EntityType;
@@ -173,19 +174,19 @@ public class EntityManagerTest extends AbstractEntityManager {
"testPipeline,dataReplicationPipeline");
configStore.publish(EntityType.PROCESS, process2);
- Entity process3 = buildProcess("process3", user, "consumer=consumer@xyz.com", "testPipeline");
+ Entity process3 = buildProcess("process3", user, "", "testPipeline");
configStore.publish(EntityType.PROCESS, process3);
- Entity process4 = buildProcess("process4", user, "owner=producer@xyz.com", "dataReplicationPipeline");
+ Entity process4 = buildProcess("process4", user, "owner=producer@xyz.com", "");
configStore.publish(EntityType.PROCESS, process4);
EntityList entityList = this.getEntityList("process", "tags", "PIPELINES:dataReplicationPipeline",
- "", "name", "desc", 1, 2);
+ "", "name", "desc", 1, 1);
Assert.assertNotNull(entityList.getElements());
- Assert.assertEquals(entityList.getElements().length, 2);
- Assert.assertEquals(entityList.getElements()[1].name, "process1");
- Assert.assertEquals(entityList.getElements()[1].tag.size(), 2);
- Assert.assertEquals(entityList.getElements()[1].tag.get(0), "consumer=consumer@xyz.com");
+ Assert.assertEquals(entityList.getElements().length, 1);
+ Assert.assertEquals(entityList.getElements()[0].name, "process1");
+ Assert.assertEquals(entityList.getElements()[0].tag.size(), 2);
+ Assert.assertEquals(entityList.getElements()[0].tag.get(0), "consumer=consumer@xyz.com");
Assert.assertEquals(entityList.getElements()[0].status, null);
@@ -202,6 +203,10 @@ public class EntityManagerTest extends AbstractEntityManager {
"consumer=consumer@xyz.com, owner=producer@xyz.com", "name", "", 10, 2);
Assert.assertEquals(entityList.getElements().length, 0);
+ entityList = this.getEntityList("process", "pipelines", "",
+ "owner=producer@xyz.com", "name", "", 1, 2);
+ Assert.assertEquals(entityList.getElements().length, 2);
+
// Test negative value for numResults, should throw an exception.
try {
this.getEntityList("process", "pipelines", "",
@@ -230,8 +235,12 @@ public class EntityManagerTest extends AbstractEntityManager {
Process process = new Process();
process.setName(name);
process.setACL(acl);
- process.setPipelines(pipelines);
- process.setTags(tags);
+ if (!StringUtils.isEmpty(pipelines)) {
+ process.setPipelines(pipelines);
+ }
+ if (!StringUtils.isEmpty(tags)) {
+ process.setTags(tags);
+ }
process.setClusters(buildClusters("cluster" + name));
return process;
}
[2/5] incubator-falcon git commit: FALCON-595 Improvements to
DefaultAuthorizationProvider code. Contributed by Raghav Kumar Gautam
Posted by ve...@apache.org.
FALCON-595 Improvements to DefaultAuthorizationProvider code. Contributed by Raghav Kumar Gautam
Project: http://git-wip-us.apache.org/repos/asf/incubator-falcon/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-falcon/commit/d2a25082
Tree: http://git-wip-us.apache.org/repos/asf/incubator-falcon/tree/d2a25082
Diff: http://git-wip-us.apache.org/repos/asf/incubator-falcon/diff/d2a25082
Branch: refs/heads/master
Commit: d2a25082b85f5075eef4cd50e18efe2fce67e46b
Parents: 8190456
Author: Venkatesh Seetharam <ve...@apache.org>
Authored: Fri Nov 7 20:18:12 2014 -0800
Committer: Venkatesh Seetharam <ve...@apache.org>
Committed: Fri Nov 7 21:33:25 2014 -0800
----------------------------------------------------------------------
CHANGES.txt | 3 +++
.../security/DefaultAuthorizationProvider.java | 28 +++++++-------------
2 files changed, 13 insertions(+), 18 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/d2a25082/CHANGES.txt
----------------------------------------------------------------------
diff --git a/CHANGES.txt b/CHANGES.txt
index d6d8747..aaea457 100755
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -129,6 +129,9 @@ Trunk (Unreleased)
OPTIMIZATIONS
BUG FIXES
+ FALCON-595 Improvements to DefaultAuthorizationProvider code
+ (Raghav Kumar Gautam via Venkatesh Seetharam)
+
FALCON-868 Rerun command incorrect in falcon CLI documentation
(Karishma Gulati via Venkatesh Seetharam)
http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/d2a25082/common/src/main/java/org/apache/falcon/security/DefaultAuthorizationProvider.java
----------------------------------------------------------------------
diff --git a/common/src/main/java/org/apache/falcon/security/DefaultAuthorizationProvider.java b/common/src/main/java/org/apache/falcon/security/DefaultAuthorizationProvider.java
index b59718c..d2d48c7 100644
--- a/common/src/main/java/org/apache/falcon/security/DefaultAuthorizationProvider.java
+++ b/common/src/main/java/org/apache/falcon/security/DefaultAuthorizationProvider.java
@@ -85,9 +85,9 @@ public class DefaultAuthorizationProvider implements AuthorizationProvider {
/**
* Super user group.
*/
- private String superUserGroup;
- private Set<String> adminUsers;
- private Set<String> adminGroups;
+ private final String superUserGroup;
+ private final Set<String> adminUsers;
+ private final Set<String> adminGroups;
public DefaultAuthorizationProvider() {
superUserGroup = StartupProperties.get().getProperty(SUPER_USER_GROUP_KEY);
@@ -95,14 +95,14 @@ public class DefaultAuthorizationProvider implements AuthorizationProvider {
adminGroups = getAdminNamesFromConfig(ADMIN_GROUPS_KEY);
}
- private HashSet<String> getAdminNamesFromConfig(String key) {
- HashSet<String> adminNames = new HashSet<String>();
+ private Set<String> getAdminNamesFromConfig(String key) {
+ Set<String> adminNames = new HashSet<String>();
String adminNamesConfig = StartupProperties.get().getProperty(key);
if (!StringUtils.isEmpty(adminNamesConfig)) {
adminNames.addAll(Arrays.asList(adminNamesConfig.split(",")));
}
- return adminNames;
+ return Collections.unmodifiableSet(adminNames);
}
/**
@@ -180,8 +180,7 @@ public class DefaultAuthorizationProvider implements AuthorizationProvider {
}
protected Set<String> getGroupNames(UserGroupInformation proxyUgi) {
- HashSet<String> s = new HashSet<String>(Arrays.asList(proxyUgi.getGroupNames()));
- return Collections.unmodifiableSet(s);
+ return new HashSet<String>(Arrays.asList(proxyUgi.getGroupNames()));
}
/**
@@ -288,16 +287,9 @@ public class DefaultAuthorizationProvider implements AuthorizationProvider {
}
protected boolean isUserInAdminGroups(UserGroupInformation proxyUgi) {
- Set<String> groups = getGroupNames(proxyUgi);
- boolean isUserGroupInAdmin = false;
- for (String group : groups) {
- if (adminGroups.contains(group)) {
- isUserGroupInAdmin = true;
- break;
- }
- }
-
- return isUserGroupInAdmin;
+ final Set<String> groups = getGroupNames(proxyUgi);
+ groups.retainAll(adminGroups);
+ return !groups.isEmpty();
}
protected void authorizeEntityResource(UserGroupInformation authenticatedUGI,
[4/5] incubator-falcon git commit: FALCON-874 Rest API - Instance and
Entity orderBy param values should be case insensitive. Contributed by Balu
Vellanki
Posted by ve...@apache.org.
FALCON-874 Rest API - Instance and Entity orderBy param values should be case insensitive. Contributed by Balu Vellanki
Project: http://git-wip-us.apache.org/repos/asf/incubator-falcon/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-falcon/commit/2186a360
Tree: http://git-wip-us.apache.org/repos/asf/incubator-falcon/tree/2186a360
Diff: http://git-wip-us.apache.org/repos/asf/incubator-falcon/diff/2186a360
Branch: refs/heads/master
Commit: 2186a360c177a549cc2b0179069b10213f4052e9
Parents: 80b0a16
Author: Venkatesh Seetharam <ve...@apache.org>
Authored: Fri Nov 7 21:05:37 2014 -0800
Committer: Venkatesh Seetharam <ve...@apache.org>
Committed: Fri Nov 7 21:33:26 2014 -0800
----------------------------------------------------------------------
CHANGES.txt | 3 +++
.../org/apache/falcon/resource/AbstractInstanceManager.java | 8 ++++----
2 files changed, 7 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/2186a360/CHANGES.txt
----------------------------------------------------------------------
diff --git a/CHANGES.txt b/CHANGES.txt
index d2635b7..d2ca59f 100755
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -129,6 +129,9 @@ Trunk (Unreleased)
OPTIMIZATIONS
BUG FIXES
+ FALCON-874 Rest API - Instance and Entity orderBy param values should be
+ case insensitive (Balu Vellanki via Venkatesh Seetharam)
+
FALCON-820 Fix minor nits with HadoopClientFactory
(Sowmya Ramesh via Venkatesh Seetharam)
http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/2186a360/prism/src/main/java/org/apache/falcon/resource/AbstractInstanceManager.java
----------------------------------------------------------------------
diff --git a/prism/src/main/java/org/apache/falcon/resource/AbstractInstanceManager.java b/prism/src/main/java/org/apache/falcon/resource/AbstractInstanceManager.java
index 5174dbd..26a4289 100644
--- a/prism/src/main/java/org/apache/falcon/resource/AbstractInstanceManager.java
+++ b/prism/src/main/java/org/apache/falcon/resource/AbstractInstanceManager.java
@@ -221,7 +221,7 @@ public abstract class AbstractInstanceManager extends AbstractEntityManager {
return result;
}
// Sort the ArrayList using orderBy
- instanceSet = sortInstances(instanceSet, orderBy, sortOrder);
+ instanceSet = sortInstances(instanceSet, orderBy.toLowerCase(), sortOrder);
result.setCollection(instanceSet.subList(
offset, (offset+pageCount)).toArray(new Instance[pageCount]));
return result;
@@ -281,7 +281,7 @@ public abstract class AbstractInstanceManager extends AbstractEntityManager {
}
private List<Instance> sortInstances(List<Instance> instanceSet,
- String orderBy, String sortOrder) {
+ String orderBy, String sortOrder) {
final String order = getValidSortOrder(sortOrder, orderBy);
if (orderBy.equals("status")) {
Collections.sort(instanceSet, new Comparator<Instance>() {
@@ -305,7 +305,7 @@ public abstract class AbstractInstanceManager extends AbstractEntityManager {
: i2.getCluster().compareTo(i1.getCluster());
}
});
- } else if (orderBy.equals("startTime")){
+ } else if (orderBy.equals("starttime")){
Collections.sort(instanceSet, new Comparator<Instance>() {
@Override
public int compare(Instance i1, Instance i2) {
@@ -315,7 +315,7 @@ public abstract class AbstractInstanceManager extends AbstractEntityManager {
: start2.compareTo(start1);
}
});
- } else if (orderBy.equals("endTime")) {
+ } else if (orderBy.equals("endtime")) {
Collections.sort(instanceSet, new Comparator<Instance>() {
@Override
public int compare(Instance i1, Instance i2) {