You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-commits@hadoop.apache.org by ro...@apache.org on 2019/03/20 15:59:21 UTC
[hadoop] branch trunk updated (072750c -> b3b0e33)
This is an automated email from the ASF dual-hosted git repository.
rohithsharmaks pushed a change to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git.
from 072750c HDDS-1306. TestContainerStateManagerIntegration fails in Ratis shutdown. Contributed by Lokesh Jain.
new c1a4eeb YARN-9389. FlowActivity and FlowRun table prefix is wrong. Contributed by Prabhu Joseph.
new f6e60e9 YARN-9387. Update document for ATS HBase Custom tablenames (-entityTableName). Contributed by Prabhu Joseph.
new 0d24684 YARN-9357. Modify HBase Liveness monitor log to debug. Contributed by Prabhu Joseph.
new b3b0e33 YARN-9299. TestTimelineReaderWhitelistAuthorizationFilter ignores Http Errors. Contributed by Prabhu Joseph.
The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../storage/HBaseTimelineReaderImpl.java | 2 +-
.../storage/flow/FlowActivityTableRW.java | 2 +-
.../storage/flow/FlowRunTableRW.java | 2 +-
...TimelineReaderWhitelistAuthorizationFilter.java | 58 +++++++++++++++++-----
.../src/site/markdown/TimelineServiceV2.md | 19 +++++--
5 files changed, 64 insertions(+), 19 deletions(-)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[hadoop] 01/04: YARN-9389. FlowActivity and FlowRun table prefix is
wrong. Contributed by Prabhu Joseph.
Posted by ro...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rohithsharmaks pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit c1a4eeb7c877c78311044c2df4f810ae7c21489f
Author: Rohith Sharma K S <ro...@apache.org>
AuthorDate: Wed Mar 20 21:18:19 2019 +0530
YARN-9389. FlowActivity and FlowRun table prefix is wrong. Contributed by Prabhu Joseph.
---
.../yarn/server/timelineservice/storage/flow/FlowActivityTableRW.java | 2 +-
.../hadoop/yarn/server/timelineservice/storage/flow/FlowRunTableRW.java | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/flow/FlowActivityTableRW.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/flow/FlowActivityTableRW.java
index 5b9fe13..ead6a79 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/flow/FlowActivityTableRW.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/flow/FlowActivityTableRW.java
@@ -36,7 +36,7 @@ import org.slf4j.LoggerFactory;
public class FlowActivityTableRW extends BaseTableRW<FlowActivityTable> {
/** flow activity table prefix. */
private static final String PREFIX =
- YarnConfiguration.TIMELINE_SERVICE_PREFIX + ".flowactivity";
+ YarnConfiguration.TIMELINE_SERVICE_PREFIX + "flowactivity";
/** config param name that specifies the flowactivity table name. */
public static final String TABLE_NAME_CONF_NAME = PREFIX + ".table.name";
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/flow/FlowRunTableRW.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/flow/FlowRunTableRW.java
index 61c0734..b3e81a1 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/flow/FlowRunTableRW.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/flow/FlowRunTableRW.java
@@ -38,7 +38,7 @@ import org.apache.hadoop.hbase.Coprocessor;
public class FlowRunTableRW extends BaseTableRW<FlowRunTable> {
/** entity prefix. */
private static final String PREFIX =
- YarnConfiguration.TIMELINE_SERVICE_PREFIX + ".flowrun";
+ YarnConfiguration.TIMELINE_SERVICE_PREFIX + "flowrun";
/** config param name that specifies the flowrun table name. */
public static final String TABLE_NAME_CONF_NAME = PREFIX + ".table.name";
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[hadoop] 04/04: YARN-9299.
TestTimelineReaderWhitelistAuthorizationFilter ignores Http Errors.
Contributed by Prabhu Joseph.
Posted by ro...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rohithsharmaks pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit b3b0e332e6d023052d7bf96c7c1f82abab939e03
Author: Rohith Sharma K S <ro...@apache.org>
AuthorDate: Wed Mar 20 21:24:31 2019 +0530
YARN-9299. TestTimelineReaderWhitelistAuthorizationFilter ignores Http Errors. Contributed by Prabhu Joseph.
---
...TimelineReaderWhitelistAuthorizationFilter.java | 58 +++++++++++++++++-----
1 file changed, 46 insertions(+), 12 deletions(-)
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/test/java/org/apache/hadoop/yarn/server/timelineservice/reader/TestTimelineReaderWhitelistAuthorizationFilter.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/test/java/org/apache/hadoop/yarn/server/timelineservice/reader/TestTimelineReaderWhitelistAuthorizationFilter.java
index 9ab9b27..576699d 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/test/java/org/apache/hadoop/yarn/server/timelineservice/reader/TestTimelineReaderWhitelistAuthorizationFilter.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/test/java/org/apache/hadoop/yarn/server/timelineservice/reader/TestTimelineReaderWhitelistAuthorizationFilter.java
@@ -21,6 +21,8 @@ package org.apache.hadoop.yarn.server.timelineservice.reader;
import static org.mockito.ArgumentMatchers.eq;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
import java.io.IOException;
import java.security.Principal;
@@ -93,15 +95,19 @@ public class TestTimelineReaderWhitelistAuthorizationFilter {
FilterConfig fc = new DummyFilterConfig(map);
f.init(fc);
HttpServletRequest mockHsr = Mockito.mock(HttpServletRequest.class);
+ String userName = "user1";
Mockito.when(mockHsr.getUserPrincipal()).thenReturn(new Principal() {
@Override
public String getName() {
- return "user1";
+ return userName;
}
});
HttpServletResponse r = Mockito.mock(HttpServletResponse.class);
f.doFilter(mockHsr, r, null);
+ String msg = "User " + userName
+ + " is not allowed to read TimelineService V2 data.";
+ verify(r, times(0)).sendError(HttpServletResponse.SC_FORBIDDEN, msg);
}
@Test
@@ -143,15 +149,16 @@ public class TestTimelineReaderWhitelistAuthorizationFilter {
FilterConfig fc = new DummyFilterConfig(map);
f.init(fc);
HttpServletRequest mockHsr = Mockito.mock(HttpServletRequest.class);
+ String userName = "user1";
Mockito.when(mockHsr.getUserPrincipal()).thenReturn(new Principal() {
@Override
public String getName() {
- return "user1";
+ return userName;
}
});
HttpServletResponse r = Mockito.mock(HttpServletResponse.class);
UserGroupInformation user1 =
- UserGroupInformation.createUserForTesting("user1", GROUP_NAMES);
+ UserGroupInformation.createUserForTesting(userName, GROUP_NAMES);
user1.doAs(new PrivilegedExceptionAction<Object>() {
@Override
public Object run() throws Exception {
@@ -159,6 +166,9 @@ public class TestTimelineReaderWhitelistAuthorizationFilter {
return null;
}
});
+ String msg = "User " + userName
+ + " is not allowed to read TimelineService V2 data.";
+ verify(r, times(0)).sendError(HttpServletResponse.SC_FORBIDDEN, msg);
}
@Test
@@ -210,15 +220,16 @@ public class TestTimelineReaderWhitelistAuthorizationFilter {
FilterConfig fc = new DummyFilterConfig(map);
f.init(fc);
HttpServletRequest mockHsr = Mockito.mock(HttpServletRequest.class);
+ String userName = "user90";
Mockito.when(mockHsr.getUserPrincipal()).thenReturn(new Principal() {
@Override
public String getName() {
- return "user90";
+ return userName;
}
});
HttpServletResponse r = Mockito.mock(HttpServletResponse.class);
UserGroupInformation user1 =
- UserGroupInformation.createUserForTesting("user90", GROUP_NAMES);
+ UserGroupInformation.createUserForTesting(userName, GROUP_NAMES);
user1.doAs(new PrivilegedExceptionAction<Object>() {
@Override
public Object run() throws Exception {
@@ -226,6 +237,9 @@ public class TestTimelineReaderWhitelistAuthorizationFilter {
return null;
}
});
+ String msg = "User " + userName
+ + " is not allowed to read TimelineService V2 data.";
+ verify(r, times(0)).sendError(HttpServletResponse.SC_FORBIDDEN, msg);
}
@Test
@@ -240,15 +254,16 @@ public class TestTimelineReaderWhitelistAuthorizationFilter {
FilterConfig fc = new DummyFilterConfig(map);
f.init(fc);
HttpServletRequest mockHsr = Mockito.mock(HttpServletRequest.class);
+ String userName = "user90";
Mockito.when(mockHsr.getUserPrincipal()).thenReturn(new Principal() {
@Override
public String getName() {
- return "user90";
+ return userName;
}
});
HttpServletResponse r = Mockito.mock(HttpServletResponse.class);
UserGroupInformation user1 =
- UserGroupInformation.createUserForTesting("user90", GROUP_NAMES);
+ UserGroupInformation.createUserForTesting(userName, GROUP_NAMES);
user1.doAs(new PrivilegedExceptionAction<Object>() {
@Override
public Object run() throws Exception {
@@ -256,6 +271,9 @@ public class TestTimelineReaderWhitelistAuthorizationFilter {
return null;
}
});
+ String msg = "User " + userName
+ + " is not allowed to read TimelineService V2 data.";
+ verify(r, times(0)).sendError(HttpServletResponse.SC_FORBIDDEN, msg);
}
@Test
@@ -303,15 +321,16 @@ public class TestTimelineReaderWhitelistAuthorizationFilter {
FilterConfig fc = new DummyFilterConfig(map);
f.init(fc);
HttpServletRequest mockHsr = Mockito.mock(HttpServletRequest.class);
+ String userName = "user437";
Mockito.when(mockHsr.getUserPrincipal()).thenReturn(new Principal() {
@Override
public String getName() {
- return "user437";
+ return userName;
}
});
HttpServletResponse r = Mockito.mock(HttpServletResponse.class);
UserGroupInformation user1 =
- UserGroupInformation.createUserForTesting("user437", GROUP_NAMES);
+ UserGroupInformation.createUserForTesting(userName, GROUP_NAMES);
user1.doAs(new PrivilegedExceptionAction<Object>() {
@Override
public Object run() throws Exception {
@@ -319,6 +338,9 @@ public class TestTimelineReaderWhitelistAuthorizationFilter {
return null;
}
});
+ String msg = "User " + userName
+ + " is not allowed to read TimelineService V2 data.";
+ verify(r, times(0)).sendError(HttpServletResponse.SC_FORBIDDEN, msg);
}
@Test
@@ -336,6 +358,7 @@ public class TestTimelineReaderWhitelistAuthorizationFilter {
f.init(fc);
HttpServletRequest mockHsr = mock(HttpServletRequest.class);
+ String userName = "user37";
when(mockHsr.getUserPrincipal()).thenReturn(new Principal() {
@Override
public String getName() {
@@ -348,7 +371,7 @@ public class TestTimelineReaderWhitelistAuthorizationFilter {
// both username and group name are not part of admin and
// read allowed users
// but read auth is turned off
- UserGroupInformation.createUserForTesting("user37", GROUP_NAMES);
+ UserGroupInformation.createUserForTesting(userName, GROUP_NAMES);
user1.doAs(new PrivilegedExceptionAction<Object>() {
@Override
public Object run() throws Exception {
@@ -356,8 +379,12 @@ public class TestTimelineReaderWhitelistAuthorizationFilter {
return null;
}
});
+ String msg = "User " + userName
+ + " is not allowed to read TimelineService V2 data.";
+ verify(r, times(0)).sendError(HttpServletResponse.SC_FORBIDDEN, msg);
// test with username in read allowed users
+ userName = "user27";
Mockito.when(mockHsr.getUserPrincipal()).thenReturn(new Principal() {
@Override
public String getName() {
@@ -366,7 +393,7 @@ public class TestTimelineReaderWhitelistAuthorizationFilter {
});
HttpServletResponse r2 = Mockito.mock(HttpServletResponse.class);
UserGroupInformation user2 =
- UserGroupInformation.createUserForTesting("user27", GROUP_NAMES);
+ UserGroupInformation.createUserForTesting(userName, GROUP_NAMES);
user2.doAs(new PrivilegedExceptionAction<Object>() {
@Override
public Object run() throws Exception {
@@ -374,8 +401,12 @@ public class TestTimelineReaderWhitelistAuthorizationFilter {
return null;
}
});
+ msg = "User " + userName
+ + " is not allowed to read TimelineService V2 data.";
+ verify(r, times(0)).sendError(HttpServletResponse.SC_FORBIDDEN, msg);
// test with username in admin users
+ userName = "user2";
Mockito.when(mockHsr.getUserPrincipal()).thenReturn(new Principal() {
@Override
public String getName() {
@@ -384,7 +415,7 @@ public class TestTimelineReaderWhitelistAuthorizationFilter {
});
HttpServletResponse r3 = Mockito.mock(HttpServletResponse.class);
UserGroupInformation user3 =
- UserGroupInformation.createUserForTesting("user2", GROUP_NAMES);
+ UserGroupInformation.createUserForTesting(userName, GROUP_NAMES);
user3.doAs(new PrivilegedExceptionAction<Object>() {
@Override
public Object run() throws Exception {
@@ -392,5 +423,8 @@ public class TestTimelineReaderWhitelistAuthorizationFilter {
return null;
}
});
+ msg = "User " + userName
+ + " is not allowed to read TimelineService V2 data.";
+ verify(r, times(0)).sendError(HttpServletResponse.SC_FORBIDDEN, msg);
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[hadoop] 03/04: YARN-9357. Modify HBase Liveness monitor log to
debug. Contributed by Prabhu Joseph.
Posted by ro...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rohithsharmaks pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit 0d24684eee77dce7fd328d5e9479345e30b699ee
Author: Rohith Sharma K S <ro...@apache.org>
AuthorDate: Wed Mar 20 21:22:54 2019 +0530
YARN-9357. Modify HBase Liveness monitor log to debug. Contributed by Prabhu Joseph.
---
.../yarn/server/timelineservice/storage/HBaseTimelineReaderImpl.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/HBaseTimelineReaderImpl.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/HBaseTimelineReaderImpl.java
index f50621e..d00ae4b 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/HBaseTimelineReaderImpl.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/HBaseTimelineReaderImpl.java
@@ -167,7 +167,7 @@ public class HBaseTimelineReaderImpl
@Override
public void run() {
try {
- LOG.info("Running HBase liveness monitor");
+ LOG.debug("Running HBase liveness monitor");
TimelineEntityReader reader =
TimelineEntityReaderFactory.createMultipleEntitiesReader(
monitorContext, MONITOR_FILTERS, DATA_TO_RETRIEVE);
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[hadoop] 02/04: YARN-9387. Update document for ATS HBase Custom
tablenames (-entityTableName). Contributed by Prabhu Joseph.
Posted by ro...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rohithsharmaks pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit f6e60e966e3c843ae5e6777b8daf1ae411f4297f
Author: Rohith Sharma K S <ro...@apache.org>
AuthorDate: Wed Mar 20 21:20:50 2019 +0530
YARN-9387. Update document for ATS HBase Custom tablenames (-entityTableName). Contributed by Prabhu Joseph.
---
.../src/site/markdown/TimelineServiceV2.md | 19 +++++++++++++++----
1 file changed, 15 insertions(+), 4 deletions(-)
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md
index eb84377..cf98c57 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md
@@ -284,10 +284,21 @@ The `TimelineSchemaCreator` tool supports a few options that may come handy espe
are testing. For example, you can use `-skipExistingTable` (`-s` for short) to skip existing tables
and continue to create other tables rather than failing the schema creation. By default, the tables
will have a schema prefix of "prod.". When no option or '-help' ('-h' for short) is provided, the
-command usage is printed.
-and continue to create other tables rather than failing the schema creation. When no option or '-help'
-('-h' for short) is provided, the command usage is printed. By default, the tables
-will have a schema prefix of "prod."
+command usage is printed. The options (-entityTableName, -appToflowTableName, -applicationTableName,
+-subApplicationTableName) will help to override the default table names. On using custom table names,
+The below corresponding configs with custom table name has to be set in hbase-site.xml configured
+at yarn.timeline-service.hbase.configuration.file.
+
+```
+yarn.timeline-service.app-flow.table.name
+yarn.timeline-service.entity.table.name
+yarn.timeline-service.application.table.name
+yarn.timeline-service.subapplication.table.name
+yarn.timeline-service.flowactivity.table.name
+yarn.timeline-service.flowrun.table.name
+yarn.timeline-service.domain.table.name
+
+```
#### Enabling Timeline Service v.2
Following are the basic configurations to start Timeline service v.2:
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org