You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@falcon.apache.org by ve...@apache.org on 2014/04/24 06:16:18 UTC
[1/3] git commit: FALCON-393 Display error messages when the web UI
fails to get the data from the server. Contributed by Haohui Mai
Repository: incubator-falcon
Updated Branches:
refs/heads/master d28ad7e04 -> 946182a09
FALCON-393 Display error messages when the web UI fails to get the data from the server. Contributed by Haohui Mai
Project: http://git-wip-us.apache.org/repos/asf/incubator-falcon/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-falcon/commit/9dcbdf54
Tree: http://git-wip-us.apache.org/repos/asf/incubator-falcon/tree/9dcbdf54
Diff: http://git-wip-us.apache.org/repos/asf/incubator-falcon/diff/9dcbdf54
Branch: refs/heads/master
Commit: 9dcbdf5471191ad75200f1941e86d3d2375b5239
Parents: d28ad7e
Author: Venkatesh Seetharam <ve...@hortonworks.com>
Authored: Wed Apr 23 12:40:16 2014 -0700
Committer: Venkatesh Seetharam <ve...@hortonworks.com>
Committed: Wed Apr 23 12:40:16 2014 -0700
----------------------------------------------------------------------
CHANGES.txt | 3 +++
html5-ui/css/falcon.css | 23 +++++++++++++++++++++--
html5-ui/entity.html | 4 ++--
html5-ui/index.html | 11 +++++++++--
html5-ui/js/falcon-entity.js | 6 +++---
html5-ui/js/falcon-index.js | 2 +-
html5-ui/js/falcon-lineage.js | 18 ++++++++++--------
html5-ui/js/falcon.js | 11 +++++++++--
8 files changed, 58 insertions(+), 20 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/9dcbdf54/CHANGES.txt
----------------------------------------------------------------------
diff --git a/CHANGES.txt b/CHANGES.txt
index 6f7589e..625de9e 100755
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -98,6 +98,9 @@ Trunk (Unreleased)
FALCON-373 Display lineage link only for jobs that are succeeded in the web UI
(Haohui Mai via Venkatesh Seetharam)
+ FALCON-393 Display error messages when the web UI fails to get the data from the server
+ (Haohui Mai via Venkatesh Seetharam)
+
OPTIMIZATIONS
FALCON-123 Improve build speeds in falcon. (Srikanth Sundarrajan via Shwetha GS)
http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/9dcbdf54/html5-ui/css/falcon.css
----------------------------------------------------------------------
diff --git a/html5-ui/css/falcon.css b/html5-ui/css/falcon.css
index 990d00c..7b0a929 100644
--- a/html5-ui/css/falcon.css
+++ b/html5-ui/css/falcon.css
@@ -15,14 +15,33 @@
* limitations under the License.
*/
+.falcon-docs-nav, .navbar-header .navbar-brand {
+ border-radius: 0px;
+ background-color: #5fa33e;
+ color: #fff;
+}
+
+.navbar-nav > li > a.btn-entity-list {
+ color: #dcf0d3;
+}
+
+.navbar-nav > .active > a {
+ color: #dcf0d3;
+}
+
.falcon-brand {
- padding-left: 40px;
+ padding-left: 50px;
background-image: url('../img/falcon-64.png');
background-size: 40px 40px;
background-repeat: no-repeat;
background-position: left center;
}
+#alert-panel {
+ margin-top:20px;
+ display: none;
+}
+
.btn-entity-list {
cursor: pointer;
}
@@ -219,4 +238,4 @@ color: #456ab5;
.lineage-legend-terminal:before {
-webkit-text-stroke-width: 2px;
-webkit-text-stroke-color: #ccc;
-}
\ No newline at end of file
+}
http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/9dcbdf54/html5-ui/entity.html
----------------------------------------------------------------------
diff --git a/html5-ui/entity.html b/html5-ui/entity.html
index d08512e..d4536fb 100644
--- a/html5-ui/entity.html
+++ b/html5-ui/entity.html
@@ -32,9 +32,9 @@
<link rel="shortcut icon" href="img/favicon.png" />
</head>
<body>
-<div class="navbar navbar-default">
+<div class="navbar navbar-inverse falcon-docs-nav">
<div class="navbar-header"><button type="button" class="navbar-toggle" data-toggle="collapse" data-target=
-".navbar-collapse"></button> <a class="navbar-brand falcon-brand">Falcon Server</a></div>
+".navbar-collapse"></button> <a class="navbar-brand falcon-brand">Apache Falcon</a></div>
<div class="navbar-collapse navbar-right collapse">
<ul class="nav navbar-nav">
<li><a class="btn-entity-list" href="index.html?type=feed">Data sets</a></li>
http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/9dcbdf54/html5-ui/index.html
----------------------------------------------------------------------
diff --git a/html5-ui/index.html b/html5-ui/index.html
index 5719f00..6763cfe 100644
--- a/html5-ui/index.html
+++ b/html5-ui/index.html
@@ -32,9 +32,10 @@
<link rel="shortcut icon" href="img/favicon.png" />
</head>
<body>
-<div class="navbar navbar-default">
+
+<div class="navbar navbar-inverse falcon-docs-nav">
<div class="navbar-header"><button type="button" class="navbar-toggle" data-toggle="collapse" data-target=
-".navbar-collapse"></button> <a class="navbar-brand falcon-brand">Falcon Server</a></div>
+".navbar-collapse"></button> <a class="navbar-brand falcon-brand">Apache Falcon</a></div>
<div class="navbar-collapse navbar-right collapse">
<ul class="nav navbar-nav">
<li><a class="btn-entity-list" data-entity-type="feed" href="index.html?type=feed">Data sets</a></li>
@@ -44,6 +45,12 @@
</div>
</div>
<div class="container">
+<div id="alert-panel">
+ <div class="alert alert-danger">
+ <button type="button" class="close" onclick="$('#alert-panel').hide();">×</button>
+ <div class="alert-body" id="alert-panel-body"></div>
+ </div>
+</div>
<div id="entity-list-container"></div>
<div id="entity-paginator"></div>
<hr />
http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/9dcbdf54/html5-ui/js/falcon-entity.js
----------------------------------------------------------------------
diff --git a/html5-ui/js/falcon-entity.js b/html5-ui/js/falcon-entity.js
index f0407c4..f5563a2 100644
--- a/html5-ui/js/falcon-entity.js
+++ b/html5-ui/js/falcon-entity.js
@@ -60,7 +60,7 @@
$('.instance-hdfs-log').tooltip();
$('#panel-instance').show();
});
- });
+ }).fail(falcon.ajaxFailureHandler);
}
function loadDependency() {
@@ -81,7 +81,7 @@
switchDependencyView('list');
$('#panel-dependency').show();
});
- });
+ }).fail(falcon.ajaxFailureHandler);
}
function load() {
@@ -99,7 +99,7 @@
loadInstance(e.attr('start'), e.attr('end'));
}
}
- });
+ }).fail(falcon.ajaxFailureHandler);
if (!isCluster) {
loadDependency();
http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/9dcbdf54/html5-ui/js/falcon-index.js
----------------------------------------------------------------------
diff --git a/html5-ui/js/falcon-index.js b/html5-ui/js/falcon-index.js
index af5a396..8abdbf1 100644
--- a/html5-ui/js/falcon-index.js
+++ b/html5-ui/js/falcon-index.js
@@ -91,7 +91,7 @@
$('#entity-paginator').data('activePage', 1);
falcon.entities = data.entity;
gotoPage(1);
- });
+ }).fail(falcon.ajaxFailureHandler);
}
function initialize() {
http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/9dcbdf54/html5-ui/js/falcon-lineage.js
----------------------------------------------------------------------
diff --git a/html5-ui/js/falcon-lineage.js b/html5-ui/js/falcon-lineage.js
index b7e0bda..b554c80 100644
--- a/html5-ui/js/falcon-lineage.js
+++ b/html5-ui/js/falcon-lineage.js
@@ -60,9 +60,10 @@
data.nodes[n._id] = n;
q.push({'id': n._id, 'depth': depth - 1});
}
- }).always(function() {
- process_queue(done_cb);
- });
+ }).fail(falcon.ajaxFailureHandler)
+ .always(function() {
+ process_queue(done_cb);
+ });
}
var v = data.queue.pop();
@@ -81,9 +82,10 @@
data.edges[e._id] = e;
}
node.is_terminal = !(terminal_has_in_edge && terminal_has_out_edge);
- }).always(function () {
- visit_neighbor(v);
- });
+ }).fail(falcon.ajaxFailureHandler)
+ .always(function () {
+ visit_neighbor(v);
+ });
}
function draw_graph() {
@@ -111,7 +113,7 @@
dust.render('info', resp, function(err, out) {
$('#lineage-info-panel').html(out);
});
- });
+ }).fail(falcon.ajaxFailureHandler);
};
}
@@ -233,6 +235,6 @@
data.nodes[n._id] = n;
update_graph();
$('#lineage-modal').modal('show');
- });
+ }).fail(falcon.ajaxFailureHandler);
};
})(falcon);
http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/9dcbdf54/html5-ui/js/falcon.js
----------------------------------------------------------------------
diff --git a/html5-ui/js/falcon.js b/html5-ui/js/falcon.js
index 0a75f6a..b4faf1c 100644
--- a/html5-ui/js/falcon.js
+++ b/html5-ui/js/falcon.js
@@ -19,8 +19,10 @@
var USER_ID = 'dashboard';
- function onError (msg) {
- alert(msg);
+ function onError(msg) {
+ $('#alert-panel-body').html(msg);
+ $('#alert-panel').alert();
+ $('#alert-panel').show();
}
function ajax_impl(options) {
@@ -60,6 +62,11 @@
ajax: ajax_impl,
getJson: getJson_impl,
getText: getText_impl,
+ ajaxFailureHandler: function(jqXHR, status, err) {
+ if (jqXHR.status !== 0) {
+ onError('Failed to load data. Error: ' + jqXHR.status + ' ' + err);
+ }
+ },
/**
* Calling the REST API recursively to get the dependency graph
[2/3] git commit: FALCON-395 Falcon service does not start when _HOST
is used in the spnego principal. Contributed by Venkatesh Seetharam
Posted by ve...@apache.org.
FALCON-395 Falcon service does not start when _HOST is used in the spnego principal. Contributed by Venkatesh Seetharam
Project: http://git-wip-us.apache.org/repos/asf/incubator-falcon/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-falcon/commit/ae5d29b0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-falcon/tree/ae5d29b0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-falcon/diff/ae5d29b0
Branch: refs/heads/master
Commit: ae5d29b065e507bd2e13e6d04d6cf2c6c58e140a
Parents: 9dcbdf5
Author: Venkatesh Seetharam <ve...@hortonworks.com>
Authored: Wed Apr 23 12:41:14 2014 -0700
Committer: Venkatesh Seetharam <ve...@hortonworks.com>
Committed: Wed Apr 23 12:41:14 2014 -0700
----------------------------------------------------------------------
CHANGES.txt | 3 ++
.../apache/falcon/security/BasicAuthFilter.java | 29 ++++++++++++-
.../falcon/security/BasicAuthFilterTest.java | 45 ++++++++++++++++++++
3 files changed, 76 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/ae5d29b0/CHANGES.txt
----------------------------------------------------------------------
diff --git a/CHANGES.txt b/CHANGES.txt
index 625de9e..57e8beb 100755
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -101,6 +101,9 @@ Trunk (Unreleased)
FALCON-393 Display error messages when the web UI fails to get the data from the server
(Haohui Mai via Venkatesh Seetharam)
+ FALCON-395 Falcon service does not start when _HOST is used in the spnego principal
+ (Venkatesh Seetharam)
+
OPTIMIZATIONS
FALCON-123 Improve build speeds in falcon. (Srikanth Sundarrajan via Shwetha GS)
http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/ae5d29b0/prism/src/main/java/org/apache/falcon/security/BasicAuthFilter.java
----------------------------------------------------------------------
diff --git a/prism/src/main/java/org/apache/falcon/security/BasicAuthFilter.java b/prism/src/main/java/org/apache/falcon/security/BasicAuthFilter.java
index b4b544c..52ede1d 100644
--- a/prism/src/main/java/org/apache/falcon/security/BasicAuthFilter.java
+++ b/prism/src/main/java/org/apache/falcon/security/BasicAuthFilter.java
@@ -20,7 +20,9 @@ package org.apache.falcon.security;
import org.apache.commons.lang.StringUtils;
import org.apache.falcon.util.StartupProperties;
+import org.apache.hadoop.security.UserGroupInformation;
import org.apache.hadoop.security.authentication.server.AuthenticationFilter;
+import org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler;
import org.apache.log4j.Logger;
import org.apache.log4j.NDC;
@@ -52,7 +54,8 @@ public class BasicAuthFilter extends AuthenticationFilter {
/**
* Constant for the configuration property that indicates the prefix.
*/
- private static final String FALCON_PREFIX = "falcon.http.authentication.";
+ protected static final String FALCON_PREFIX = "falcon.http.authentication.";
+ protected static final String KERBEROS_PRINCIPAL = FALCON_PREFIX + KerberosAuthenticationHandler.PRINCIPAL;
/**
* Constant for the configuration property that indicates the blacklisted super users for falcon.
@@ -121,9 +124,33 @@ public class BasicAuthFilter extends AuthenticationFilter {
}
}
+ if (UserGroupInformation.isSecurityEnabled()) { // replace _HOST in principal
+ String principal = getKerberosPrincipalWithSubstitutedHost(configProperties);
+ // principal cannot be null in secure mode, is validated in submission
+ authProperties.setProperty(KerberosAuthenticationHandler.PRINCIPAL, principal);
+ }
+
return authProperties;
}
+ /**
+ * Replaces _HOST in the principal with the actual hostname.
+ *
+ * @param configProperties Falcon config properties
+ * @return principal with _HOST substituted
+ */
+ private String getKerberosPrincipalWithSubstitutedHost(Properties configProperties) {
+ String principal = configProperties.getProperty(KERBEROS_PRINCIPAL);
+ try {
+ principal = org.apache.hadoop.security.SecurityUtil.getServerPrincipal(
+ principal, SecurityUtil.getLocalHostName());
+ } catch (IOException ignored) {
+ // do nothing
+ }
+
+ return principal;
+ }
+
@Override
public void doFilter(final ServletRequest request, final ServletResponse response,
final FilterChain filterChain) throws IOException, ServletException {
http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/ae5d29b0/webapp/src/test/java/org/apache/falcon/security/BasicAuthFilterTest.java
----------------------------------------------------------------------
diff --git a/webapp/src/test/java/org/apache/falcon/security/BasicAuthFilterTest.java b/webapp/src/test/java/org/apache/falcon/security/BasicAuthFilterTest.java
index 1caf914..74073d0 100644
--- a/webapp/src/test/java/org/apache/falcon/security/BasicAuthFilterTest.java
+++ b/webapp/src/test/java/org/apache/falcon/security/BasicAuthFilterTest.java
@@ -19,6 +19,10 @@
package org.apache.falcon.security;
import org.apache.falcon.util.StartupProperties;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.CommonConfigurationKeysPublic;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler;
import org.mockito.Mock;
import org.mockito.Mockito;
import org.mockito.MockitoAnnotations;
@@ -33,6 +37,7 @@ import javax.servlet.FilterConfig;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.util.Map;
+import java.util.Properties;
import java.util.concurrent.ConcurrentHashMap;
@@ -53,6 +58,9 @@ public class BasicAuthFilterTest {
@Mock
private FilterConfig mockConfig;
+ @Mock
+ private UserGroupInformation mockUgi;
+
@BeforeClass
public void init() {
MockitoAnnotations.initMocks(this);
@@ -169,4 +177,41 @@ public class BasicAuthFilterTest {
StartupProperties.get().setProperty("falcon.http.authentication.type", httpAuthType);
}
}
+
+ @Test
+ public void testGetKerberosPrincipalWithSubstitutedHostSecure() throws Exception {
+ String principal = StartupProperties.get().getProperty(BasicAuthFilter.KERBEROS_PRINCIPAL);
+
+ String expectedPrincipal = "falcon/" + SecurityUtil.getLocalHostName() + "@Example.com";
+ try {
+ Configuration conf = new Configuration(false);
+ conf.set(CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTHENTICATION,
+ UserGroupInformation.AuthenticationMethod.KERBEROS.name());
+ UserGroupInformation.setConfiguration(conf);
+ Assert.assertTrue(UserGroupInformation.isSecurityEnabled());
+
+ StartupProperties.get().setProperty(
+ BasicAuthFilter.KERBEROS_PRINCIPAL, "falcon/_HOST@Example.com");
+ BasicAuthFilter filter = new BasicAuthFilter();
+ Properties properties = filter.getConfiguration(BasicAuthFilter.FALCON_PREFIX, null);
+ Assert.assertEquals(
+ properties.get(KerberosAuthenticationHandler.PRINCIPAL), expectedPrincipal);
+ } finally {
+ StartupProperties.get().setProperty(BasicAuthFilter.KERBEROS_PRINCIPAL, principal);
+ }
+ }
+
+ @Test
+ public void testGetKerberosPrincipalWithSubstitutedHostNonSecure() throws Exception {
+ String principal = StartupProperties.get().getProperty(BasicAuthFilter.KERBEROS_PRINCIPAL);
+ Configuration conf = new Configuration(false);
+ conf.set(CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTHENTICATION,
+ UserGroupInformation.AuthenticationMethod.SIMPLE.name());
+ UserGroupInformation.setConfiguration(conf);
+ Assert.assertFalse(UserGroupInformation.isSecurityEnabled());
+
+ BasicAuthFilter filter = new BasicAuthFilter();
+ Properties properties = filter.getConfiguration(BasicAuthFilter.FALCON_PREFIX, null);
+ Assert.assertEquals(properties.get(KerberosAuthenticationHandler.PRINCIPAL), principal);
+ }
}
[3/3] git commit: FALCON-397 DistCp uses JobSubmissionFiles API which
is incompatible between hadoop versions. Contributed by Venkatesh Seetharam
Posted by ve...@apache.org.
FALCON-397 DistCp uses JobSubmissionFiles API which is incompatible between hadoop versions. Contributed by Venkatesh Seetharam
Project: http://git-wip-us.apache.org/repos/asf/incubator-falcon/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-falcon/commit/946182a0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-falcon/tree/946182a0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-falcon/diff/946182a0
Branch: refs/heads/master
Commit: 946182a093afb5d5fbc15486c868ae349164a3b6
Parents: ae5d29b
Author: Venkatesh Seetharam <ve...@hortonworks.com>
Authored: Wed Apr 23 12:41:59 2014 -0700
Committer: Venkatesh Seetharam <ve...@hortonworks.com>
Committed: Wed Apr 23 12:41:59 2014 -0700
----------------------------------------------------------------------
CHANGES.txt | 3 +++
pom.xml | 2 +-
2 files changed, 4 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/946182a0/CHANGES.txt
----------------------------------------------------------------------
diff --git a/CHANGES.txt b/CHANGES.txt
index 57e8beb..d123bc7 100755
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -104,6 +104,9 @@ Trunk (Unreleased)
FALCON-395 Falcon service does not start when _HOST is used in the spnego principal
(Venkatesh Seetharam)
+ FALCON-397 DistCp uses JobSubmissionFiles API which is incompatible between hadoop versions
+ (Venkatesh Seetharam)
+
OPTIMIZATIONS
FALCON-123 Improve build speeds in falcon. (Srikanth Sundarrajan via Shwetha GS)
http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/946182a0/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index 8e31626..e15ec16 100644
--- a/pom.xml
+++ b/pom.xml
@@ -109,7 +109,7 @@
<activemq.version>5.4.3</activemq.version>
<hive.version>0.11.0</hive.version>
<hcatalog.version>0.11.0</hcatalog.version>
- <hadoop-distcp.version>0.9</hadoop-distcp.version>
+ <hadoop-distcp.version>0.11</hadoop-distcp.version>
<jetty.version>6.1.26</jetty.version>
<jersey.version>1.9</jersey.version>
<internal.maven.repo>file:///tmp/falcontemprepo</internal.maven.repo>