You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by HeartSaVioR <gi...@git.apache.org> on 2017/07/12 08:41:44 UTC
[GitHub] storm pull request #2204: (WIP) STORM-1280 port backtype.storm.daemon.logvie...
GitHub user HeartSaVioR opened a pull request:
https://github.com/apache/storm/pull/2204
(WIP) STORM-1280 port backtype.storm.daemon.logviewer to java
* port logviewer to Java using Dropwizard
* follow implementation of DRPC Server
* add '/api/v1/' to the logviewer API endpoints
* seems like it is not possible (or not easy) to serve both API endpoints and static resources in same servlet
* reflect the change to the static resources
Please note that logviewer test should be ported to Java since ported version of Logviewer is placed to storm-webapp module. Below is TODO list:
TODO
* [ ] - port logviewer-test.clj to Java and delete logviewer-test.clj
* [ ] - delete logviewer.clj and rename package to remove 'wip'
I've added J2HTML and jOOL as dependencies for storm-webapp: J2HTML as replacement for hiccup, and jOOL as Java 8 lambda helper.
Manually tested. Unit test will be covered with porting logviewer-test.clj.
Please let me know deleting logviewer.clj is more convenient to review.
Please review and comment. Thanks in advance!
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/HeartSaVioR/storm STORM-1280
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/2204.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2204
----
commit 73b84d71f9ac3e9b718dbc8794cfaee2ab1faead
Author: Jungtaek Lim <ka...@gmail.com>
Date: 2017-07-11T04:49:56Z
(WIP) STORM-1280 port backtype.storm.daemon.logviewer to java
* port logviewer to Java using Dropwizard
* follow implementation of DRPC Server
* add '/api/v1/' to the logviewer API endpoints
* seems like it is not possible (or not easy) to serve both API endpoints and static resources in same servlet
* reflect the change to the static resources
* TODO: logviewer test should be ported to Java since ported version of Logviewer is placed to storm-webapp module
* can't remove logviewer.clj for now, hence adding 'wip' to package to avoid clash
* ported version of Logviewer is manually tested, not testing with logviewer-test
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #2204: STORM-1280 port backtype.storm.daemon.logviewer to java
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/storm/pull/2204
@Ethanlm
For now you should include `ethan` to `logs.users` in `topology configuration`. Global configuration doesn't take effect. This behavior seems same as current logviewer.
If we need to merge global configuration of `logs.users` and `logs.groups` into topology configuration, we can file a new issue and handle it accordingly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #2204: STORM-1280 port backtype.storm.daemon.logviewer to...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/storm/pull/2204#discussion_r128402244
--- Diff: storm-webapp/src/main/java/org/apache/storm/daemon/logviewer/webapp/LogviewerResource.java ---
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.daemon.logviewer.webapp;
+
+import com.codahale.metrics.Meter;
+
+import java.io.IOException;
+import java.net.URLDecoder;
+import java.util.Map;
+
+import javax.servlet.http.HttpServletRequest;
+import javax.ws.rs.GET;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.core.Context;
+import javax.ws.rs.core.Response;
+
+import org.apache.commons.lang.BooleanUtils;
+import org.apache.commons.lang.StringUtils;
+import org.apache.storm.daemon.common.JsonResponseBuilder;
+import org.apache.storm.daemon.logviewer.handler.LogviewerLogDownloadHandler;
+import org.apache.storm.daemon.logviewer.handler.LogviewerLogPageHandler;
+import org.apache.storm.daemon.logviewer.handler.LogviewerLogSearchHandler;
+import org.apache.storm.daemon.logviewer.handler.LogviewerProfileHandler;
+import org.apache.storm.metric.StormMetricsRegistry;
+import org.apache.storm.security.auth.IHttpCredentialsPlugin;
+import org.apache.storm.ui.InvalidRequestException;
+import org.apache.storm.ui.UIHelpers;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Handles HTTP requests for Logviewer.
+ */
+@Path("/")
+public class LogviewerResource {
+ private static final Logger LOG = LoggerFactory.getLogger(LogviewerResource.class);
+
+ private static final Meter meterLogPageHttpRequests = StormMetricsRegistry.registerMeter("logviewer:num-log-page-http-requests");
+ private static final Meter meterDaemonLogPageHttpRequests = StormMetricsRegistry.registerMeter(
+ "logviewer:num-daemonlog-page-http-requests");
+ private static final Meter meterDownloadLogFileHttpRequests = StormMetricsRegistry.registerMeter(
+ "logviewer:num-download-log-file-http-requests");
+ private static final Meter meterDownloadLogDaemonFileHttpRequests = StormMetricsRegistry.registerMeter(
+ "logviewer:num-download-log-daemon-file-http-requests");
+ private static final Meter meterListLogsHttpRequests = StormMetricsRegistry.registerMeter("logviewer:num-list-logs-http-requests");
+
+ private final LogviewerLogPageHandler logviewer;
+ private final LogviewerProfileHandler profileHandler;
+ private final LogviewerLogDownloadHandler logDownloadHandler;
+ private final LogviewerLogSearchHandler logSearchHandler;
+ private final IHttpCredentialsPlugin httpCredsHandler;
+
+ /**
+ * Constructor.
+ *
+ * @param logviewerParam {@link LogviewerLogPageHandler}
+ * @param profileHandler {@link LogviewerProfileHandler}
+ * @param logDownloadHandler {@link LogviewerLogDownloadHandler}
+ * @param logSearchHandler {@link LogviewerLogSearchHandler}
+ * @param httpCredsHandler {@link IHttpCredentialsPlugin}
+ */
+ public LogviewerResource(LogviewerLogPageHandler logviewerParam, LogviewerProfileHandler profileHandler,
+ LogviewerLogDownloadHandler logDownloadHandler, LogviewerLogSearchHandler logSearchHandler,
+ IHttpCredentialsPlugin httpCredsHandler) {
+ this.logviewer = logviewerParam;
+ this.profileHandler = profileHandler;
+ this.logDownloadHandler = logDownloadHandler;
+ this.logSearchHandler = logSearchHandler;
+ this.httpCredsHandler = httpCredsHandler;
+ }
+
+ /**
+ * Handles '/log' request.
+ */
+ @GET
+ @Path("/log")
+ public Response log(@Context HttpServletRequest request) throws IOException {
+ meterLogPageHttpRequests.mark();
+
+ try {
+ String user = httpCredsHandler.getUserName(request);
+ Integer start = request.getParameter("start") != null ? parseIntegerFromMap(request.getParameterMap(), "start") : null;
+ Integer length = request.getParameter("length") != null ? parseIntegerFromMap(request.getParameterMap(), "length") : null;
+ String decodedFileName = URLDecoder.decode(request.getParameter("file"));
+ String grep = request.getParameter("grep");
+ return logviewer.logPage(decodedFileName, start, length, grep, user);
+ } catch (InvalidRequestException e) {
+ LOG.error(e.getMessage(), e);
+ return Response.status(400).entity(e.getMessage()).build();
+ }
+ }
+
+ /**
+ * Handles '/daemonlog' request.
+ */
+ @GET
+ @Path("/daemonlog")
+ public Response daemonLog(@Context HttpServletRequest request) throws IOException {
+ meterDaemonLogPageHttpRequests.mark();
+
+ try {
+ String user = httpCredsHandler.getUserName(request);
+ Integer start = request.getParameter("start") != null ? parseIntegerFromMap(request.getParameterMap(), "start") : null;
+ Integer length = request.getParameter("length") != null ? parseIntegerFromMap(request.getParameterMap(), "length") : null;
+ String decodedFileName = URLDecoder.decode(request.getParameter("file"));
+ String grep = request.getParameter("grep");
+ return logviewer.daemonLogPage(decodedFileName, start, length, grep, user);
+ } catch (InvalidRequestException e) {
+ LOG.error(e.getMessage(), e);
+ return Response.status(400).entity(e.getMessage()).build();
+ }
+ }
+
+ /**
+ * Handles '/searchLogs' request.
+ */
+ @GET
+ @Path("/searchLogs")
+ public Response searchLogs(@Context HttpServletRequest request) throws IOException {
+ String user = httpCredsHandler.getUserName(request);
+ String topologyId = request.getParameter("topoId");
+ String portStr = request.getParameter("port");
+ String callback = request.getParameter("callback");
+ String origin = request.getHeader("Origin");
+
+ return logviewer.listLogFiles(user, portStr != null ? Integer.parseInt(portStr) : null, topologyId, callback, origin);
--- End diff --
InvalidRequestException is a checked exception. Clojure seems not check whether the exception class is checked or runtime so placing InvalidRequestException is possible, but from here it doesn't throw InvalidRequestException so can't place it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #2204: STORM-1280 port backtype.storm.daemon.logviewer to java
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/storm/pull/2204
@revans2 Just to check: did you play around secured cluster as well? I still didn't test this on secured cluster, just now set up secured cluster and plan to set up Storm and test.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #2204: STORM-1280 port backtype.storm.daemon.logviewer to java
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/storm/pull/2204
@Ethanlm
> 1. "Prev" "First" etc buttons seems to mix together. (On non-secure and secure cluster)
Looks like failing to load css and also js file. We don't include static resource files to the class path, so you need to build binary distribution and use it to test correctly.
(Please let me know if you tested it with binary distribution.)
Btw, it seems to be by intention to avoid adding static resources to the client side. Maybe we can move static files to storm-webapp after porting UI and Logviewer is completed, since storm-webapp module is not intended to be used (as a dependency) for Storm client. It also helps to run and test Logviewer and UI with IDE.
> 2. Got "page not found" while querying worker.log on secure cluster. The file actually exists.
If you could spend some time to help me debugging, could you check the file's permission?
> 3. Got "Request failed" which querying dumps on secure cluster
Is it occurred only from secure cluster? Looks like actual error is not relevant to secure, so just curious.
To be honest, I didn't have a chance to deploy the patch to secured cluster, since I feel there should be no difference between non-secure and secure cluster. I might be wrong. I'll also spin secured cluster and test it again.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #2204: STORM-1280 port backtype.storm.daemon.logviewer to java
Posted by Ethanlm <gi...@git.apache.org>.
Github user Ethanlm commented on the issue:
https://github.com/apache/storm/pull/2204
1. I believe I tested it from binary distribution. I used "mvn clean install" to compile and "cd storm-dist/binary && mvn package" to created the binary distribution. And then I deployed it under /tmp directory on my openstack instance. I do it every time when I need to do some tests
2. Will try to do something about it.
3. It happens only on secure cluster. Non-secure cluster works fine in my tests. I could have done something wrong. It would be great if you could test it on your secure cluster.
Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #2204: STORM-1280 port backtype.storm.daemon.logviewer to java
Posted by Ethanlm <gi...@git.apache.org>.
Github user Ethanlm commented on the issue:
https://github.com/apache/storm/pull/2204
Yea~ It works now.
Use nimbus admin account:
Use "ethan" account (it's not authorized. Probably that why I got "page-not-found" while querying worker.log):
A quick question.. "ethan" is nimbus.user, ui.user, and logs.user. But it's not able to view worker.log. Which configuration should I set "ethan" to..? Thanks very much.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #2204: STORM-1280 port backtype.storm.daemon.logviewer to java
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/storm/pull/2204
@Ethanlm
Yeah I think I addressed (2) and forgot to debug (3). I'll test it quickly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #2204: STORM-1280 port backtype.storm.daemon.logviewer to...
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2204#discussion_r128368272
--- Diff: storm-webapp/src/main/java/org/apache/storm/daemon/logviewer/handler/LogviewerLogPageHandler.java ---
@@ -0,0 +1,467 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.daemon.logviewer.handler;
+
+import static j2html.TagCreator.a;
+import static j2html.TagCreator.body;
+import static j2html.TagCreator.div;
+import static j2html.TagCreator.form;
+import static j2html.TagCreator.h3;
+import static j2html.TagCreator.head;
+import static j2html.TagCreator.html;
+import static j2html.TagCreator.input;
+import static j2html.TagCreator.link;
+import static j2html.TagCreator.p;
+import static j2html.TagCreator.pre;
+import static j2html.TagCreator.select;
+import static j2html.TagCreator.text;
+import static j2html.TagCreator.title;
+import static java.util.stream.Collectors.toCollection;
+import static java.util.stream.Collectors.toList;
+import static org.apache.commons.lang.StringEscapeUtils.escapeHtml;
+
+import j2html.TagCreator;
+import j2html.tags.DomContent;
+
+import java.io.ByteArrayOutputStream;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.UncheckedIOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.zip.GZIPInputStream;
+
+import javax.ws.rs.core.Response;
+
+import org.apache.commons.lang.StringUtils;
+import org.apache.storm.daemon.logviewer.LogviewerConstant;
+import org.apache.storm.daemon.logviewer.utils.DirectoryCleaner;
+import org.apache.storm.daemon.logviewer.utils.LogviewerResponseBuilder;
+import org.apache.storm.daemon.logviewer.utils.ResourceAuthorizer;
+import org.apache.storm.daemon.logviewer.utils.WorkerLogs;
+import org.apache.storm.daemon.utils.StreamUtil;
+import org.apache.storm.daemon.utils.UrlBuilder;
+import org.apache.storm.ui.InvalidRequestException;
+import org.apache.storm.ui.UIHelpers;
+import org.apache.storm.utils.ConfigUtils;
+import org.apache.storm.utils.ServerUtils;
+import org.apache.storm.utils.Utils;
+import org.jooq.lambda.Unchecked;
+
+public class LogviewerLogPageHandler {
+ private final String logRoot;
+ private final String daemonLogRoot;
+ private final WorkerLogs workerLogs;
+ private final ResourceAuthorizer resourceAuthorizer;
+
+ /**
+ * Constructor.
+ *
+ * @param logRoot root worker log directory
+ * @param daemonLogRoot root daemon log directory
+ * @param workerLogs {@link WorkerLogs}
+ * @param resourceAuthorizer {@link ResourceAuthorizer}
+ */
+ public LogviewerLogPageHandler(String logRoot, String daemonLogRoot,
+ WorkerLogs workerLogs,
+ ResourceAuthorizer resourceAuthorizer) {
+ this.logRoot = logRoot;
+ this.daemonLogRoot = daemonLogRoot;
+ this.workerLogs = workerLogs;
+ this.resourceAuthorizer = resourceAuthorizer;
+ }
+
+ /**
+ * Enumerate worker log files for given criteria.
+ *
+ * @param user username
+ * @param port worker's port, null for all workers
+ * @param topologyId topology ID, null for all topologies
+ * @param callback callback for JSONP
+ * @param origin origin
+ * @return list of worker logs for given criteria
+ */
+ public Response listLogFiles(String user, Integer port, String topologyId, String callback, String origin) throws IOException {
+ List<File> fileResults = null;
+ if (topologyId == null) {
+ if (port == null) {
+ fileResults = workerLogs.getAllLogsForRootDir();
+ } else {
+ fileResults = new ArrayList<>();
+
+ File[] logRootFiles = new File(logRoot).listFiles();
+ if (logRootFiles != null) {
+ for (File topoDir : logRootFiles) {
+ File[] topoDirFiles = topoDir.listFiles();
+ if (topoDirFiles != null) {
+ for (File portDir : topoDirFiles) {
+ if (portDir.getName().equals(port.toString())) {
+ fileResults.addAll(DirectoryCleaner.getFilesForDir(portDir));
+ }
+ }
+ }
+ }
+ }
+ }
+ } else {
+ if (port == null) {
+ fileResults = new ArrayList<>();
+
+ File topoDir = new File(logRoot + Utils.FILE_PATH_SEPARATOR + topologyId);
--- End diff --
nit: I know this is an exact translation, but we could just use `new File(logRoot, topologyId)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #2204: STORM-1280 port backtype.storm.daemon.logviewer to...
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2204#discussion_r128599601
--- Diff: storm-webapp/src/main/java/org/apache/storm/daemon/logviewer/LogviewerServer.java ---
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.daemon.logviewer;
+
+import static org.apache.storm.DaemonConfig.UI_HEADER_BUFFER_BYTES;
+
+import com.codahale.metrics.Meter;
+import com.google.common.annotations.VisibleForTesting;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.storm.DaemonConfig;
+import org.apache.storm.daemon.logviewer.utils.DirectoryCleaner;
+import org.apache.storm.daemon.logviewer.utils.LogCleaner;
+import org.apache.storm.daemon.logviewer.utils.WorkerLogs;
+import org.apache.storm.daemon.logviewer.webapp.LogviewerApplication;
+import org.apache.storm.metric.StormMetricsRegistry;
+import org.apache.storm.ui.FilterConfiguration;
+import org.apache.storm.ui.UIHelpers;
+import org.apache.storm.utils.ConfigUtils;
+import org.apache.storm.utils.ObjectReader;
+import org.apache.storm.utils.Utils;
+import org.eclipse.jetty.server.Server;
+import org.eclipse.jetty.servlet.DefaultServlet;
+import org.eclipse.jetty.servlet.ServletContextHandler;
+import org.eclipse.jetty.servlet.ServletHolder;
+import org.eclipse.jetty.util.resource.Resource;
+import org.glassfish.jersey.servlet.ServletContainer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The main entry of Logviewer.
+ */
+public class LogviewerServer implements AutoCloseable {
+ private static final Logger LOG = LoggerFactory.getLogger(LogviewerServer.class);
+ private static final Meter meterShutdownCalls = StormMetricsRegistry.registerMeter("drpc:num-shutdown-calls");
+ public static final String STATIC_RESOURCE_DIRECTORY_PATH = "./public";
+
+ private static Server mkHttpServer(Map<String, Object> conf) {
+ Integer logviewerHttpPort = (Integer) conf.get(DaemonConfig.LOGVIEWER_PORT);
+ Server ret = null;
+ if (logviewerHttpPort != null && logviewerHttpPort >= 0) {
+ LOG.info("Starting Logviewer HTTP servers...");
+ Integer headerBufferSize = ObjectReader.getInt(conf.get(UI_HEADER_BUFFER_BYTES));
+ String filterClass = (String) (conf.get(DaemonConfig.UI_FILTER));
+ @SuppressWarnings("unchecked")
+ Map<String, String> filterParams = (Map<String, String>) (conf.get(DaemonConfig.UI_FILTER_PARAMS));
+ FilterConfiguration filterConfiguration = new FilterConfiguration(filterClass, filterParams);
+ final List<FilterConfiguration> filterConfigurations = Arrays.asList(filterConfiguration);
+
+ final Integer httpsPort = ObjectReader.getInt(conf.get(DaemonConfig.LOGVIEWER_HTTPS_PORT), 0);
+ final String httpsKsPath = (String) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_KEYSTORE_PATH));
+ final String httpsKsPassword = (String) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_KEYSTORE_PASSWORD));
+ final String httpsKsType = (String) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_KEYSTORE_TYPE));
+ final String httpsKeyPassword = (String) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_KEY_PASSWORD));
+ final String httpsTsPath = (String) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_TRUSTSTORE_PATH));
+ final String httpsTsPassword = (String) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_TRUSTSTORE_PASSWORD));
+ final String httpsTsType = (String) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_TRUSTSTORE_TYPE));
+ final Boolean httpsWantClientAuth = (Boolean) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_WANT_CLIENT_AUTH));
+ final Boolean httpsNeedClientAuth = (Boolean) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_NEED_CLIENT_AUTH));
+
+ //TODO a better way to do this would be great.
+ LogviewerApplication.setup(conf);
+ ret = UIHelpers.jettyCreateServer(logviewerHttpPort, null, httpsPort);
+
+ UIHelpers.configSsl(ret, httpsPort, httpsKsPath, httpsKsPassword, httpsKsType, httpsKeyPassword,
+ httpsTsPath, httpsTsPassword, httpsTsType, httpsNeedClientAuth, httpsWantClientAuth);
+
+ ServletContextHandler context = new ServletContextHandler(ServletContextHandler.NO_SESSIONS);
+ try {
+ context.setBaseResource(Resource.newResource(STATIC_RESOURCE_DIRECTORY_PATH));
+ } catch (IOException e) {
+ throw new RuntimeException("Can't locate static resource directory " + STATIC_RESOURCE_DIRECTORY_PATH);
+ }
+
+ context.setContextPath("/");
+ ret.setHandler(context);
+
+ ServletHolder holderPwd = new ServletHolder("default", DefaultServlet.class);
+ holderPwd.setInitOrder(1);
+ context.addServlet(holderPwd,"/");
+
+ ServletHolder jerseyServlet = context.addServlet(ServletContainer.class, "/api/v1/*");
+ jerseyServlet.setInitOrder(2);
+ jerseyServlet.setInitParameter("javax.ws.rs.Application", LogviewerApplication.class.getName());
+
+ UIHelpers.configFilters(context, filterConfigurations);
+ }
+ return ret;
+ }
+
+ private final Server httpServer;
+ private boolean closed = false;
+
+ /**
+ * Constructor.
+ * @param conf Logviewer conf for the servers
+ */
+ public LogviewerServer(Map<String, Object> conf) {
+ httpServer = mkHttpServer(conf);
+ }
+
+ @VisibleForTesting
+ void start() throws Exception {
+ LOG.info("Starting Logviewer...");
+ if (httpServer != null) {
+ httpServer.start();
+ }
+ }
+
+ @VisibleForTesting
+ void awaitTermination() throws InterruptedException {
+ httpServer.join();
+ }
+
+ @Override
+ public synchronized void close() {
+ if (!closed) {
+ //This is kind of useless...
+ meterShutdownCalls.mark();
+
+ //TODO this is causing issues...
--- End diff --
I thought it looked familiar. I'll file a follow on JIRA to try and clean it up once we have everything ported over to java.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #2204: STORM-1280 port backtype.storm.daemon.logviewer to...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/storm/pull/2204#discussion_r128403293
--- Diff: storm-webapp/src/main/java/org/apache/storm/daemon/logviewer/LogviewerServer.java ---
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.daemon.logviewer;
+
+import static org.apache.storm.DaemonConfig.UI_HEADER_BUFFER_BYTES;
+
+import com.codahale.metrics.Meter;
+import com.google.common.annotations.VisibleForTesting;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.storm.DaemonConfig;
+import org.apache.storm.daemon.logviewer.utils.DirectoryCleaner;
+import org.apache.storm.daemon.logviewer.utils.LogCleaner;
+import org.apache.storm.daemon.logviewer.utils.WorkerLogs;
+import org.apache.storm.daemon.logviewer.webapp.LogviewerApplication;
+import org.apache.storm.metric.StormMetricsRegistry;
+import org.apache.storm.ui.FilterConfiguration;
+import org.apache.storm.ui.UIHelpers;
+import org.apache.storm.utils.ConfigUtils;
+import org.apache.storm.utils.ObjectReader;
+import org.apache.storm.utils.Utils;
+import org.eclipse.jetty.server.Server;
+import org.eclipse.jetty.servlet.DefaultServlet;
+import org.eclipse.jetty.servlet.ServletContextHandler;
+import org.eclipse.jetty.servlet.ServletHolder;
+import org.eclipse.jetty.util.resource.Resource;
+import org.glassfish.jersey.servlet.ServletContainer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The main entry of Logviewer.
+ */
+public class LogviewerServer implements AutoCloseable {
+ private static final Logger LOG = LoggerFactory.getLogger(LogviewerServer.class);
+ private static final Meter meterShutdownCalls = StormMetricsRegistry.registerMeter("drpc:num-shutdown-calls");
+ public static final String STATIC_RESOURCE_DIRECTORY_PATH = "./public";
+
+ private static Server mkHttpServer(Map<String, Object> conf) {
+ Integer logviewerHttpPort = (Integer) conf.get(DaemonConfig.LOGVIEWER_PORT);
+ Server ret = null;
+ if (logviewerHttpPort != null && logviewerHttpPort >= 0) {
+ LOG.info("Starting Logviewer HTTP servers...");
+ Integer headerBufferSize = ObjectReader.getInt(conf.get(UI_HEADER_BUFFER_BYTES));
+ String filterClass = (String) (conf.get(DaemonConfig.UI_FILTER));
+ @SuppressWarnings("unchecked")
+ Map<String, String> filterParams = (Map<String, String>) (conf.get(DaemonConfig.UI_FILTER_PARAMS));
+ FilterConfiguration filterConfiguration = new FilterConfiguration(filterClass, filterParams);
+ final List<FilterConfiguration> filterConfigurations = Arrays.asList(filterConfiguration);
+
+ final Integer httpsPort = ObjectReader.getInt(conf.get(DaemonConfig.LOGVIEWER_HTTPS_PORT), 0);
+ final String httpsKsPath = (String) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_KEYSTORE_PATH));
+ final String httpsKsPassword = (String) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_KEYSTORE_PASSWORD));
+ final String httpsKsType = (String) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_KEYSTORE_TYPE));
+ final String httpsKeyPassword = (String) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_KEY_PASSWORD));
+ final String httpsTsPath = (String) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_TRUSTSTORE_PATH));
+ final String httpsTsPassword = (String) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_TRUSTSTORE_PASSWORD));
+ final String httpsTsType = (String) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_TRUSTSTORE_TYPE));
+ final Boolean httpsWantClientAuth = (Boolean) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_WANT_CLIENT_AUTH));
+ final Boolean httpsNeedClientAuth = (Boolean) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_NEED_CLIENT_AUTH));
+
+ //TODO a better way to do this would be great.
+ LogviewerApplication.setup(conf);
+ ret = UIHelpers.jettyCreateServer(logviewerHttpPort, null, httpsPort);
+
+ UIHelpers.configSsl(ret, httpsPort, httpsKsPath, httpsKsPassword, httpsKsType, httpsKeyPassword,
+ httpsTsPath, httpsTsPassword, httpsTsType, httpsNeedClientAuth, httpsWantClientAuth);
+
+ ServletContextHandler context = new ServletContextHandler(ServletContextHandler.NO_SESSIONS);
+ try {
+ context.setBaseResource(Resource.newResource(STATIC_RESOURCE_DIRECTORY_PATH));
+ } catch (IOException e) {
+ throw new RuntimeException("Can't locate static resource directory " + STATIC_RESOURCE_DIRECTORY_PATH);
+ }
+
+ context.setContextPath("/");
+ ret.setHandler(context);
+
+ ServletHolder holderPwd = new ServletHolder("default", DefaultServlet.class);
+ holderPwd.setInitOrder(1);
+ context.addServlet(holderPwd,"/");
+
+ ServletHolder jerseyServlet = context.addServlet(ServletContainer.class, "/api/v1/*");
+ jerseyServlet.setInitOrder(2);
+ jerseyServlet.setInitParameter("javax.ws.rs.Application", LogviewerApplication.class.getName());
+
+ UIHelpers.configFilters(context, filterConfigurations);
+ }
+ return ret;
+ }
+
+ private final Server httpServer;
+ private boolean closed = false;
+
+ /**
+ * Constructor.
+ * @param conf Logviewer conf for the servers
+ */
+ public LogviewerServer(Map<String, Object> conf) {
+ httpServer = mkHttpServer(conf);
+ }
+
+ @VisibleForTesting
+ void start() throws Exception {
+ LOG.info("Starting Logviewer...");
+ if (httpServer != null) {
+ httpServer.start();
+ }
+ }
+
+ @VisibleForTesting
+ void awaitTermination() throws InterruptedException {
+ httpServer.join();
+ }
+
+ @Override
+ public synchronized void close() {
+ if (!closed) {
+ //This is kind of useless...
+ meterShutdownCalls.mark();
+
+ //TODO this is causing issues...
+ //if (httpServer != null) {
+ // httpServer.destroy();
+ //}
+
+ closed = true;
+ }
+ }
+
+ /**
+ * @return The port the HTTP server is listening on. Not available until {@link #start() } has run.
+ */
+ public int getHttpServerPort() {
+ assert httpServer.getConnectors().length == 1;
--- End diff --
Same here: this is from copy and paste, and even DRPCServer it seems to be only used for unit test. We could just remove the method and re-add if needed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #2204: STORM-1280 port backtype.storm.daemon.logviewer to...
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2204#discussion_r128366709
--- Diff: storm-webapp/src/main/java/org/apache/storm/daemon/logviewer/webapp/LogviewerResource.java ---
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.daemon.logviewer.webapp;
+
+import com.codahale.metrics.Meter;
+
+import java.io.IOException;
+import java.net.URLDecoder;
+import java.util.Map;
+
+import javax.servlet.http.HttpServletRequest;
+import javax.ws.rs.GET;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.core.Context;
+import javax.ws.rs.core.Response;
+
+import org.apache.commons.lang.BooleanUtils;
+import org.apache.commons.lang.StringUtils;
+import org.apache.storm.daemon.common.JsonResponseBuilder;
+import org.apache.storm.daemon.logviewer.handler.LogviewerLogDownloadHandler;
+import org.apache.storm.daemon.logviewer.handler.LogviewerLogPageHandler;
+import org.apache.storm.daemon.logviewer.handler.LogviewerLogSearchHandler;
+import org.apache.storm.daemon.logviewer.handler.LogviewerProfileHandler;
+import org.apache.storm.metric.StormMetricsRegistry;
+import org.apache.storm.security.auth.IHttpCredentialsPlugin;
+import org.apache.storm.ui.InvalidRequestException;
+import org.apache.storm.ui.UIHelpers;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Handles HTTP requests for Logviewer.
+ */
+@Path("/")
+public class LogviewerResource {
+ private static final Logger LOG = LoggerFactory.getLogger(LogviewerResource.class);
+
+ private static final Meter meterLogPageHttpRequests = StormMetricsRegistry.registerMeter("logviewer:num-log-page-http-requests");
+ private static final Meter meterDaemonLogPageHttpRequests = StormMetricsRegistry.registerMeter(
+ "logviewer:num-daemonlog-page-http-requests");
+ private static final Meter meterDownloadLogFileHttpRequests = StormMetricsRegistry.registerMeter(
+ "logviewer:num-download-log-file-http-requests");
+ private static final Meter meterDownloadLogDaemonFileHttpRequests = StormMetricsRegistry.registerMeter(
+ "logviewer:num-download-log-daemon-file-http-requests");
+ private static final Meter meterListLogsHttpRequests = StormMetricsRegistry.registerMeter("logviewer:num-list-logs-http-requests");
+
+ private final LogviewerLogPageHandler logviewer;
+ private final LogviewerProfileHandler profileHandler;
+ private final LogviewerLogDownloadHandler logDownloadHandler;
+ private final LogviewerLogSearchHandler logSearchHandler;
+ private final IHttpCredentialsPlugin httpCredsHandler;
+
+ /**
+ * Constructor.
+ *
+ * @param logviewerParam {@link LogviewerLogPageHandler}
+ * @param profileHandler {@link LogviewerProfileHandler}
+ * @param logDownloadHandler {@link LogviewerLogDownloadHandler}
+ * @param logSearchHandler {@link LogviewerLogSearchHandler}
+ * @param httpCredsHandler {@link IHttpCredentialsPlugin}
+ */
+ public LogviewerResource(LogviewerLogPageHandler logviewerParam, LogviewerProfileHandler profileHandler,
+ LogviewerLogDownloadHandler logDownloadHandler, LogviewerLogSearchHandler logSearchHandler,
+ IHttpCredentialsPlugin httpCredsHandler) {
+ this.logviewer = logviewerParam;
+ this.profileHandler = profileHandler;
+ this.logDownloadHandler = logDownloadHandler;
+ this.logSearchHandler = logSearchHandler;
+ this.httpCredsHandler = httpCredsHandler;
+ }
+
+ /**
+ * Handles '/log' request.
+ */
+ @GET
+ @Path("/log")
+ public Response log(@Context HttpServletRequest request) throws IOException {
+ meterLogPageHttpRequests.mark();
+
+ try {
+ String user = httpCredsHandler.getUserName(request);
+ Integer start = request.getParameter("start") != null ? parseIntegerFromMap(request.getParameterMap(), "start") : null;
+ Integer length = request.getParameter("length") != null ? parseIntegerFromMap(request.getParameterMap(), "length") : null;
+ String decodedFileName = URLDecoder.decode(request.getParameter("file"));
+ String grep = request.getParameter("grep");
+ return logviewer.logPage(decodedFileName, start, length, grep, user);
+ } catch (InvalidRequestException e) {
+ LOG.error(e.getMessage(), e);
+ return Response.status(400).entity(e.getMessage()).build();
+ }
+ }
+
+ /**
+ * Handles '/daemonlog' request.
+ */
+ @GET
+ @Path("/daemonlog")
+ public Response daemonLog(@Context HttpServletRequest request) throws IOException {
+ meterDaemonLogPageHttpRequests.mark();
+
+ try {
+ String user = httpCredsHandler.getUserName(request);
+ Integer start = request.getParameter("start") != null ? parseIntegerFromMap(request.getParameterMap(), "start") : null;
+ Integer length = request.getParameter("length") != null ? parseIntegerFromMap(request.getParameterMap(), "length") : null;
+ String decodedFileName = URLDecoder.decode(request.getParameter("file"));
+ String grep = request.getParameter("grep");
+ return logviewer.daemonLogPage(decodedFileName, start, length, grep, user);
+ } catch (InvalidRequestException e) {
+ LOG.error(e.getMessage(), e);
+ return Response.status(400).entity(e.getMessage()).build();
+ }
+ }
+
+ /**
+ * Handles '/searchLogs' request.
+ */
+ @GET
+ @Path("/searchLogs")
+ public Response searchLogs(@Context HttpServletRequest request) throws IOException {
+ String user = httpCredsHandler.getUserName(request);
+ String topologyId = request.getParameter("topoId");
+ String portStr = request.getParameter("port");
+ String callback = request.getParameter("callback");
+ String origin = request.getHeader("Origin");
+
+ return logviewer.listLogFiles(user, portStr != null ? Integer.parseInt(portStr) : null, topologyId, callback, origin);
+ }
+
+ /**
+ * Handles '/listLogs' request.
+ */
+ @GET
+ @Path("/listLogs")
+ public Response listLogs(@Context HttpServletRequest request) throws IOException {
+ meterListLogsHttpRequests.mark();
+
+ String user = httpCredsHandler.getUserName(request);
+ String topologyId = request.getParameter("topoId");
+ String portStr = request.getParameter("port");
+ String callback = request.getParameter("callback");
+ String origin = request.getHeader("Origin");
+
+ return logviewer.listLogFiles(user, portStr != null ? Integer.parseInt(portStr) : null, topologyId, callback, origin);
+ }
+
+ /**
+ * Handles '/dumps' (listing dump files) request.
+ */
+ @GET
+ @Path("/dumps/{topo-id}/{host-port}")
+ public Response listDumpFiles(@PathParam("topo-id") String topologyId, @PathParam("host-port") String hostPort,
+ @Context HttpServletRequest request) throws IOException {
+ String user = httpCredsHandler.getUserName(request);
+ return profileHandler.listDumpFiles(topologyId, hostPort, user);
+ }
+
+ /**
+ * Handles '/dumps' (downloading specific dump file) request.
+ */
+ @GET
+ @Path("/dumps/{topo-id}/{host-port}/{filename}")
+ public Response downloadDumpFile(@PathParam("topo-id") String topologyId, @PathParam("host-port") String hostPort,
+ @PathParam("filename") String fileName, @Context HttpServletRequest request) throws IOException {
+ String user = httpCredsHandler.getUserName(request);
+ return profileHandler.downloadDumpFile(topologyId, hostPort, fileName, user);
+ }
+
+ /**
+ * Handles '/download' (downloading specific log file) request.
+ */
+ @GET
+ @Path("/download")
+ public Response downloadLogFile(@Context HttpServletRequest request) throws IOException {
+ meterDownloadLogFileHttpRequests.mark();
+
+ String user = httpCredsHandler.getUserName(request);
+ String file = request.getParameter("file");
+ String decodedFileName = URLDecoder.decode(file);
+ return logDownloadHandler.downloadLogFile(decodedFileName, user);
--- End diff --
Do we need to handle InvalidRequestException?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #2204: STORM-1280 port backtype.storm.daemon.logviewer to...
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2204#discussion_r128353794
--- Diff: storm-core/src/clj/org/apache/storm/ui/core.clj ---
@@ -134,12 +134,12 @@
(defn logviewer-link [host fname secure?]
(if (and secure? (*STORM-CONF* LOGVIEWER-HTTPS-PORT))
- (UIHelpers/urlFormat "https://%s:%s/log?file=%s"
+ (UIHelpers/urlFormat "https://%s:%s/api/v1/log?file=%s"
--- End diff --
I just realized that we don't have the UI REST API documented anywhere. I think that is OK, but we might want to think about doing it at some point.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #2204: STORM-1280 port backtype.storm.daemon.logviewer to java
Posted by Ethanlm <gi...@git.apache.org>.
Github user Ethanlm commented on the issue:
https://github.com/apache/storm/pull/2204
@HeartSaVioR I will test it quickly
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #2204: STORM-1280 port backtype.storm.daemon.logviewer to...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/storm/pull/2204#discussion_r128404337
--- Diff: storm-core/src/clj/org/apache/storm/ui/core.clj ---
@@ -134,12 +134,12 @@
(defn logviewer-link [host fname secure?]
(if (and secure? (*STORM-CONF* LOGVIEWER-HTTPS-PORT))
- (UIHelpers/urlFormat "https://%s:%s/log?file=%s"
+ (UIHelpers/urlFormat "https://%s:%s/api/v1/log?file=%s"
--- End diff --
Yes agreed. There're some responses returning HTML page instead of JSON, so we may need to change it first and try to document it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #2204: STORM-1280 port backtype.storm.daemon.logviewer to java
Posted by Ethanlm <gi...@git.apache.org>.
Github user Ethanlm commented on the issue:
https://github.com/apache/storm/pull/2204
Still not authorized.. But I guess it's my problem.. Will try to figure it out. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #2204: STORM-1280 port backtype.storm.daemon.logviewer to java
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/storm/pull/2204
@revans2
I fixed the drop down issue, and also fixed issue regarding authorization check. I think it was the reason of issues what @Ethanlm observed.
@Ethanlm
Could you test it again if you can spend time on this? If you don't have time to test, I'll just merge this after reviewing the latest commit from @revans2 .
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #2204: STORM-1280 port backtype.storm.daemon.logviewer to...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/storm/pull/2204#discussion_r127633153
--- Diff: storm-buildtools/storm_checkstyle.xml ---
@@ -46,6 +46,9 @@
-->
<module name = "Checker">
+ <!-- Filter out Checkstyle warnings that have been suppressed with the @SuppressWarnings annotation -->
--- End diff --
I just add allowing SuppressWarnings annotation, since the test code contains long string constant. I didn't recognize that maven checkstyle doesn't check test codes, but it might be useful for specific situations anyway.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #2204: STORM-1280 port backtype.storm.daemon.logviewer to java
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on the issue:
https://github.com/apache/storm/pull/2204
@Ethanlm since you found this could you file a follow on JIRA to use the PrincipalToLocal plugin in the logviewer when authorizing users?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #2204: (WIP) STORM-1280 port backtype.storm.daemon.logviewer to ...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/storm/pull/2204
Also fixed the tests.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #2204: STORM-1280 port backtype.storm.daemon.logviewer to java
Posted by Ethanlm <gi...@git.apache.org>.
Github user Ethanlm commented on the issue:
https://github.com/apache/storm/pull/2204
@HeartSaVioR An update about _"(2)Got "page not found" while querying worker.log on secure cluster. The file actually exists."_
I probably have setup something wrong.
I used "mapredqa" (the nimbus admin account and also the topology submitter in my test) to login and I was able to view the log files on logviewer, i.e. no "page not found" issue.
<img width="1436" alt="screen shot 2017-07-20 at 11 37 14 pm" src="https://user-images.githubusercontent.com/14900612/28449318-7429fa80-6da4-11e7-9d44-28091e19ff8f.png">
But if I use "ethan" account to login, I am able to see topology summary, component summary, except log files. "ethan" is a nimbus.user, ui.user, logs.user and topology user of this topology. I got an warning in logviewer.log while querying for the worker.log
<img width="1440" alt="screen shot 2017-07-20 at 11 26 07 pm" src="https://user-images.githubusercontent.com/14900612/28449229-b9f8d69a-6da3-11e7-8141-5f95593d8b91.png">
I don't know the reason yet. I guess there could be something wrong in my cluster settings. It would be great if someone can test it on secure cluster about it.
Note that I was still testing the version before @revans2 's comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #2204: STORM-1280 port backtype.storm.daemon.logviewer to...
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2204#discussion_r128356199
--- Diff: storm-webapp/src/main/java/org/apache/storm/daemon/logviewer/LogviewerServer.java ---
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.daemon.logviewer;
+
+import static org.apache.storm.DaemonConfig.UI_HEADER_BUFFER_BYTES;
+
+import com.codahale.metrics.Meter;
+import com.google.common.annotations.VisibleForTesting;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.storm.DaemonConfig;
+import org.apache.storm.daemon.logviewer.utils.DirectoryCleaner;
+import org.apache.storm.daemon.logviewer.utils.LogCleaner;
+import org.apache.storm.daemon.logviewer.utils.WorkerLogs;
+import org.apache.storm.daemon.logviewer.webapp.LogviewerApplication;
+import org.apache.storm.metric.StormMetricsRegistry;
+import org.apache.storm.ui.FilterConfiguration;
+import org.apache.storm.ui.UIHelpers;
+import org.apache.storm.utils.ConfigUtils;
+import org.apache.storm.utils.ObjectReader;
+import org.apache.storm.utils.Utils;
+import org.eclipse.jetty.server.Server;
+import org.eclipse.jetty.servlet.DefaultServlet;
+import org.eclipse.jetty.servlet.ServletContextHandler;
+import org.eclipse.jetty.servlet.ServletHolder;
+import org.eclipse.jetty.util.resource.Resource;
+import org.glassfish.jersey.servlet.ServletContainer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The main entry of Logviewer.
+ */
+public class LogviewerServer implements AutoCloseable {
+ private static final Logger LOG = LoggerFactory.getLogger(LogviewerServer.class);
+ private static final Meter meterShutdownCalls = StormMetricsRegistry.registerMeter("drpc:num-shutdown-calls");
+ public static final String STATIC_RESOURCE_DIRECTORY_PATH = "./public";
+
+ private static Server mkHttpServer(Map<String, Object> conf) {
+ Integer logviewerHttpPort = (Integer) conf.get(DaemonConfig.LOGVIEWER_PORT);
+ Server ret = null;
+ if (logviewerHttpPort != null && logviewerHttpPort >= 0) {
+ LOG.info("Starting Logviewer HTTP servers...");
+ Integer headerBufferSize = ObjectReader.getInt(conf.get(UI_HEADER_BUFFER_BYTES));
+ String filterClass = (String) (conf.get(DaemonConfig.UI_FILTER));
+ @SuppressWarnings("unchecked")
+ Map<String, String> filterParams = (Map<String, String>) (conf.get(DaemonConfig.UI_FILTER_PARAMS));
+ FilterConfiguration filterConfiguration = new FilterConfiguration(filterClass, filterParams);
+ final List<FilterConfiguration> filterConfigurations = Arrays.asList(filterConfiguration);
+
+ final Integer httpsPort = ObjectReader.getInt(conf.get(DaemonConfig.LOGVIEWER_HTTPS_PORT), 0);
+ final String httpsKsPath = (String) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_KEYSTORE_PATH));
+ final String httpsKsPassword = (String) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_KEYSTORE_PASSWORD));
+ final String httpsKsType = (String) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_KEYSTORE_TYPE));
+ final String httpsKeyPassword = (String) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_KEY_PASSWORD));
+ final String httpsTsPath = (String) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_TRUSTSTORE_PATH));
+ final String httpsTsPassword = (String) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_TRUSTSTORE_PASSWORD));
+ final String httpsTsType = (String) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_TRUSTSTORE_TYPE));
+ final Boolean httpsWantClientAuth = (Boolean) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_WANT_CLIENT_AUTH));
+ final Boolean httpsNeedClientAuth = (Boolean) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_NEED_CLIENT_AUTH));
+
+ //TODO a better way to do this would be great.
--- End diff --
Can we either find a better way to do this or file a follow on JIRA and delete the TODO.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #2204: STORM-1280 port backtype.storm.daemon.logviewer to...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/storm/pull/2204#discussion_r128401972
--- Diff: storm-webapp/src/main/java/org/apache/storm/daemon/logviewer/webapp/LogviewerResource.java ---
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.daemon.logviewer.webapp;
+
+import com.codahale.metrics.Meter;
+
+import java.io.IOException;
+import java.net.URLDecoder;
+import java.util.Map;
+
+import javax.servlet.http.HttpServletRequest;
+import javax.ws.rs.GET;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.core.Context;
+import javax.ws.rs.core.Response;
+
+import org.apache.commons.lang.BooleanUtils;
+import org.apache.commons.lang.StringUtils;
+import org.apache.storm.daemon.common.JsonResponseBuilder;
+import org.apache.storm.daemon.logviewer.handler.LogviewerLogDownloadHandler;
+import org.apache.storm.daemon.logviewer.handler.LogviewerLogPageHandler;
+import org.apache.storm.daemon.logviewer.handler.LogviewerLogSearchHandler;
+import org.apache.storm.daemon.logviewer.handler.LogviewerProfileHandler;
+import org.apache.storm.metric.StormMetricsRegistry;
+import org.apache.storm.security.auth.IHttpCredentialsPlugin;
+import org.apache.storm.ui.InvalidRequestException;
+import org.apache.storm.ui.UIHelpers;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Handles HTTP requests for Logviewer.
+ */
+@Path("/")
+public class LogviewerResource {
+ private static final Logger LOG = LoggerFactory.getLogger(LogviewerResource.class);
+
+ private static final Meter meterLogPageHttpRequests = StormMetricsRegistry.registerMeter("logviewer:num-log-page-http-requests");
+ private static final Meter meterDaemonLogPageHttpRequests = StormMetricsRegistry.registerMeter(
+ "logviewer:num-daemonlog-page-http-requests");
+ private static final Meter meterDownloadLogFileHttpRequests = StormMetricsRegistry.registerMeter(
+ "logviewer:num-download-log-file-http-requests");
+ private static final Meter meterDownloadLogDaemonFileHttpRequests = StormMetricsRegistry.registerMeter(
+ "logviewer:num-download-log-daemon-file-http-requests");
+ private static final Meter meterListLogsHttpRequests = StormMetricsRegistry.registerMeter("logviewer:num-list-logs-http-requests");
+
+ private final LogviewerLogPageHandler logviewer;
+ private final LogviewerProfileHandler profileHandler;
+ private final LogviewerLogDownloadHandler logDownloadHandler;
+ private final LogviewerLogSearchHandler logSearchHandler;
+ private final IHttpCredentialsPlugin httpCredsHandler;
+
+ /**
+ * Constructor.
+ *
+ * @param logviewerParam {@link LogviewerLogPageHandler}
+ * @param profileHandler {@link LogviewerProfileHandler}
+ * @param logDownloadHandler {@link LogviewerLogDownloadHandler}
+ * @param logSearchHandler {@link LogviewerLogSearchHandler}
+ * @param httpCredsHandler {@link IHttpCredentialsPlugin}
+ */
+ public LogviewerResource(LogviewerLogPageHandler logviewerParam, LogviewerProfileHandler profileHandler,
+ LogviewerLogDownloadHandler logDownloadHandler, LogviewerLogSearchHandler logSearchHandler,
+ IHttpCredentialsPlugin httpCredsHandler) {
+ this.logviewer = logviewerParam;
+ this.profileHandler = profileHandler;
+ this.logDownloadHandler = logDownloadHandler;
+ this.logSearchHandler = logSearchHandler;
+ this.httpCredsHandler = httpCredsHandler;
+ }
+
+ /**
+ * Handles '/log' request.
+ */
+ @GET
+ @Path("/log")
+ public Response log(@Context HttpServletRequest request) throws IOException {
+ meterLogPageHttpRequests.mark();
+
+ try {
+ String user = httpCredsHandler.getUserName(request);
+ Integer start = request.getParameter("start") != null ? parseIntegerFromMap(request.getParameterMap(), "start") : null;
+ Integer length = request.getParameter("length") != null ? parseIntegerFromMap(request.getParameterMap(), "length") : null;
+ String decodedFileName = URLDecoder.decode(request.getParameter("file"));
+ String grep = request.getParameter("grep");
+ return logviewer.logPage(decodedFileName, start, length, grep, user);
+ } catch (InvalidRequestException e) {
+ LOG.error(e.getMessage(), e);
+ return Response.status(400).entity(e.getMessage()).build();
+ }
+ }
+
+ /**
+ * Handles '/daemonlog' request.
+ */
+ @GET
+ @Path("/daemonlog")
+ public Response daemonLog(@Context HttpServletRequest request) throws IOException {
+ meterDaemonLogPageHttpRequests.mark();
+
+ try {
+ String user = httpCredsHandler.getUserName(request);
+ Integer start = request.getParameter("start") != null ? parseIntegerFromMap(request.getParameterMap(), "start") : null;
+ Integer length = request.getParameter("length") != null ? parseIntegerFromMap(request.getParameterMap(), "length") : null;
+ String decodedFileName = URLDecoder.decode(request.getParameter("file"));
+ String grep = request.getParameter("grep");
+ return logviewer.daemonLogPage(decodedFileName, start, length, grep, user);
+ } catch (InvalidRequestException e) {
+ LOG.error(e.getMessage(), e);
+ return Response.status(400).entity(e.getMessage()).build();
+ }
+ }
+
+ /**
+ * Handles '/searchLogs' request.
+ */
+ @GET
+ @Path("/searchLogs")
+ public Response searchLogs(@Context HttpServletRequest request) throws IOException {
+ String user = httpCredsHandler.getUserName(request);
+ String topologyId = request.getParameter("topoId");
+ String portStr = request.getParameter("port");
+ String callback = request.getParameter("callback");
+ String origin = request.getHeader("Origin");
+
+ return logviewer.listLogFiles(user, portStr != null ? Integer.parseInt(portStr) : null, topologyId, callback, origin);
+ }
+
+ /**
+ * Handles '/listLogs' request.
+ */
+ @GET
+ @Path("/listLogs")
+ public Response listLogs(@Context HttpServletRequest request) throws IOException {
+ meterListLogsHttpRequests.mark();
+
+ String user = httpCredsHandler.getUserName(request);
+ String topologyId = request.getParameter("topoId");
+ String portStr = request.getParameter("port");
+ String callback = request.getParameter("callback");
+ String origin = request.getHeader("Origin");
+
+ return logviewer.listLogFiles(user, portStr != null ? Integer.parseInt(portStr) : null, topologyId, callback, origin);
+ }
+
+ /**
+ * Handles '/dumps' (listing dump files) request.
+ */
+ @GET
+ @Path("/dumps/{topo-id}/{host-port}")
+ public Response listDumpFiles(@PathParam("topo-id") String topologyId, @PathParam("host-port") String hostPort,
+ @Context HttpServletRequest request) throws IOException {
+ String user = httpCredsHandler.getUserName(request);
+ return profileHandler.listDumpFiles(topologyId, hostPort, user);
+ }
+
+ /**
+ * Handles '/dumps' (downloading specific dump file) request.
+ */
+ @GET
+ @Path("/dumps/{topo-id}/{host-port}/{filename}")
+ public Response downloadDumpFile(@PathParam("topo-id") String topologyId, @PathParam("host-port") String hostPort,
+ @PathParam("filename") String fileName, @Context HttpServletRequest request) throws IOException {
+ String user = httpCredsHandler.getUserName(request);
+ return profileHandler.downloadDumpFile(topologyId, hostPort, fileName, user);
+ }
+
+ /**
+ * Handles '/download' (downloading specific log file) request.
+ */
+ @GET
+ @Path("/download")
+ public Response downloadLogFile(@Context HttpServletRequest request) throws IOException {
+ meterDownloadLogFileHttpRequests.mark();
+
+ String user = httpCredsHandler.getUserName(request);
+ String file = request.getParameter("file");
+ String decodedFileName = URLDecoder.decode(file);
+ return logDownloadHandler.downloadLogFile(decodedFileName, user);
+ }
+
+ /**
+ * Handles '/daemondownload' (downloading specific daemon log file) request.
+ */
+ @GET
+ @Path("/daemondownload")
+ public Response downloadDaemonLogFile(@Context HttpServletRequest request) throws IOException {
+ meterDownloadLogDaemonFileHttpRequests.mark();
+
+ String user = httpCredsHandler.getUserName(request);
+ String file = request.getParameter("file");
+ String decodedFileName = URLDecoder.decode(file);
+ return logDownloadHandler.downloadDaemonLogFile(decodedFileName, user);
+ }
+
+ /**
+ * Handles '/search' (searching from specific worker or daemon log file) request.
+ */
+ @GET
+ @Path("/search/{file}")
+ public Response search(@PathParam("file") String file, @Context HttpServletRequest request) throws IOException {
+ String user = httpCredsHandler.getUserName(request);
+ boolean isDaemon = StringUtils.equals(request.getParameter("is-daemon"), "yes");
+ String decodedFileName = URLDecoder.decode(file);
+ String searchString = request.getParameter("search-string");
+ String numMatchesStr = request.getParameter("num-matches");
+ String startByteOffset = request.getParameter("start-byte-offset");
+ String callback = request.getParameter("callback");
+ String origin = request.getHeader("Origin");
+
+ try {
+ return logSearchHandler.searchLogFile(decodedFileName, user, isDaemon, searchString, numMatchesStr,
+ startByteOffset, callback, origin);
+ } catch (InvalidRequestException e) {
+ LOG.error(e.getMessage(), e);
+ return new JsonResponseBuilder().setData(UIHelpers.exceptionToJson(e)).setCallback(callback)
+ .setStatus(400).build();
+ }
+ }
+
+ /**
+ * Handles '/deepSearch' request.
+ */
+ @GET
+ @Path("/deepSearch/{topoId}")
+ public Response deepSearch(@PathParam("topoId") String topologyId,
+ @Context HttpServletRequest request) throws IOException {
+ String user = httpCredsHandler.getUserName(request);
+ String searchString = request.getParameter("search-string");
+ String numMatchesStr = request.getParameter("num-matches");
+ String portStr = request.getParameter("port");
+ String startFileOffset = request.getParameter("start-file-offset");
+ String startByteOffset = request.getParameter("start-byte-offset");
+ String searchArchived = request.getParameter("search-archived");
+ String callback = request.getParameter("callback");
+ String origin = request.getHeader("Origin");
--- End diff --
Yes agreed. I'm also not sure which is clearer, so I just applied what we did.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #2204: STORM-1280 port backtype.storm.daemon.logviewer to java
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on the issue:
https://github.com/apache/storm/pull/2204
@HeartSaVioR I have not tested on a secure cluster at all. It was all on a single node cluster.
I verified that as many links as I could worked correctly. I verified I could switch between files, search through files. I verified that log files were cleaned up as expected as the size grew.
The only other thing that I just noticed is that the pull down for selecting files does not have the current file selected as the default. (again this can probably be done in a follow on JIRA)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #2204: STORM-1280 port backtype.storm.daemon.logviewer to java
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on the issue:
https://github.com/apache/storm/pull/2204
I did a bunch of testing and I am +1 on merging this in. There is still the issue with double escaping the contents returned for a file, but I think we can fix that after if we want.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #2204: STORM-1280 port backtype.storm.daemon.logviewer to...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/storm/pull/2204#discussion_r128402268
--- Diff: storm-webapp/src/main/java/org/apache/storm/daemon/logviewer/webapp/LogviewerResource.java ---
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.daemon.logviewer.webapp;
+
+import com.codahale.metrics.Meter;
+
+import java.io.IOException;
+import java.net.URLDecoder;
+import java.util.Map;
+
+import javax.servlet.http.HttpServletRequest;
+import javax.ws.rs.GET;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.core.Context;
+import javax.ws.rs.core.Response;
+
+import org.apache.commons.lang.BooleanUtils;
+import org.apache.commons.lang.StringUtils;
+import org.apache.storm.daemon.common.JsonResponseBuilder;
+import org.apache.storm.daemon.logviewer.handler.LogviewerLogDownloadHandler;
+import org.apache.storm.daemon.logviewer.handler.LogviewerLogPageHandler;
+import org.apache.storm.daemon.logviewer.handler.LogviewerLogSearchHandler;
+import org.apache.storm.daemon.logviewer.handler.LogviewerProfileHandler;
+import org.apache.storm.metric.StormMetricsRegistry;
+import org.apache.storm.security.auth.IHttpCredentialsPlugin;
+import org.apache.storm.ui.InvalidRequestException;
+import org.apache.storm.ui.UIHelpers;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Handles HTTP requests for Logviewer.
+ */
+@Path("/")
+public class LogviewerResource {
+ private static final Logger LOG = LoggerFactory.getLogger(LogviewerResource.class);
+
+ private static final Meter meterLogPageHttpRequests = StormMetricsRegistry.registerMeter("logviewer:num-log-page-http-requests");
+ private static final Meter meterDaemonLogPageHttpRequests = StormMetricsRegistry.registerMeter(
+ "logviewer:num-daemonlog-page-http-requests");
+ private static final Meter meterDownloadLogFileHttpRequests = StormMetricsRegistry.registerMeter(
+ "logviewer:num-download-log-file-http-requests");
+ private static final Meter meterDownloadLogDaemonFileHttpRequests = StormMetricsRegistry.registerMeter(
+ "logviewer:num-download-log-daemon-file-http-requests");
+ private static final Meter meterListLogsHttpRequests = StormMetricsRegistry.registerMeter("logviewer:num-list-logs-http-requests");
+
+ private final LogviewerLogPageHandler logviewer;
+ private final LogviewerProfileHandler profileHandler;
+ private final LogviewerLogDownloadHandler logDownloadHandler;
+ private final LogviewerLogSearchHandler logSearchHandler;
+ private final IHttpCredentialsPlugin httpCredsHandler;
+
+ /**
+ * Constructor.
+ *
+ * @param logviewerParam {@link LogviewerLogPageHandler}
+ * @param profileHandler {@link LogviewerProfileHandler}
+ * @param logDownloadHandler {@link LogviewerLogDownloadHandler}
+ * @param logSearchHandler {@link LogviewerLogSearchHandler}
+ * @param httpCredsHandler {@link IHttpCredentialsPlugin}
+ */
+ public LogviewerResource(LogviewerLogPageHandler logviewerParam, LogviewerProfileHandler profileHandler,
+ LogviewerLogDownloadHandler logDownloadHandler, LogviewerLogSearchHandler logSearchHandler,
+ IHttpCredentialsPlugin httpCredsHandler) {
+ this.logviewer = logviewerParam;
+ this.profileHandler = profileHandler;
+ this.logDownloadHandler = logDownloadHandler;
+ this.logSearchHandler = logSearchHandler;
+ this.httpCredsHandler = httpCredsHandler;
+ }
+
+ /**
+ * Handles '/log' request.
+ */
+ @GET
+ @Path("/log")
+ public Response log(@Context HttpServletRequest request) throws IOException {
+ meterLogPageHttpRequests.mark();
+
+ try {
+ String user = httpCredsHandler.getUserName(request);
+ Integer start = request.getParameter("start") != null ? parseIntegerFromMap(request.getParameterMap(), "start") : null;
+ Integer length = request.getParameter("length") != null ? parseIntegerFromMap(request.getParameterMap(), "length") : null;
+ String decodedFileName = URLDecoder.decode(request.getParameter("file"));
+ String grep = request.getParameter("grep");
+ return logviewer.logPage(decodedFileName, start, length, grep, user);
+ } catch (InvalidRequestException e) {
+ LOG.error(e.getMessage(), e);
+ return Response.status(400).entity(e.getMessage()).build();
+ }
+ }
+
+ /**
+ * Handles '/daemonlog' request.
+ */
+ @GET
+ @Path("/daemonlog")
+ public Response daemonLog(@Context HttpServletRequest request) throws IOException {
+ meterDaemonLogPageHttpRequests.mark();
+
+ try {
+ String user = httpCredsHandler.getUserName(request);
+ Integer start = request.getParameter("start") != null ? parseIntegerFromMap(request.getParameterMap(), "start") : null;
+ Integer length = request.getParameter("length") != null ? parseIntegerFromMap(request.getParameterMap(), "length") : null;
+ String decodedFileName = URLDecoder.decode(request.getParameter("file"));
+ String grep = request.getParameter("grep");
+ return logviewer.daemonLogPage(decodedFileName, start, length, grep, user);
+ } catch (InvalidRequestException e) {
+ LOG.error(e.getMessage(), e);
+ return Response.status(400).entity(e.getMessage()).build();
+ }
+ }
+
+ /**
+ * Handles '/searchLogs' request.
+ */
+ @GET
+ @Path("/searchLogs")
+ public Response searchLogs(@Context HttpServletRequest request) throws IOException {
+ String user = httpCredsHandler.getUserName(request);
+ String topologyId = request.getParameter("topoId");
+ String portStr = request.getParameter("port");
+ String callback = request.getParameter("callback");
+ String origin = request.getHeader("Origin");
+
+ return logviewer.listLogFiles(user, portStr != null ? Integer.parseInt(portStr) : null, topologyId, callback, origin);
+ }
+
+ /**
+ * Handles '/listLogs' request.
+ */
+ @GET
+ @Path("/listLogs")
+ public Response listLogs(@Context HttpServletRequest request) throws IOException {
+ meterListLogsHttpRequests.mark();
+
+ String user = httpCredsHandler.getUserName(request);
+ String topologyId = request.getParameter("topoId");
+ String portStr = request.getParameter("port");
+ String callback = request.getParameter("callback");
+ String origin = request.getHeader("Origin");
+
+ return logviewer.listLogFiles(user, portStr != null ? Integer.parseInt(portStr) : null, topologyId, callback, origin);
--- End diff --
Same as above.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #2204: STORM-1280 port backtype.storm.daemon.logviewer to...
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2204#discussion_r128367236
--- Diff: storm-webapp/src/main/java/org/apache/storm/daemon/logviewer/webapp/LogviewerResource.java ---
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.daemon.logviewer.webapp;
+
+import com.codahale.metrics.Meter;
+
+import java.io.IOException;
+import java.net.URLDecoder;
+import java.util.Map;
+
+import javax.servlet.http.HttpServletRequest;
+import javax.ws.rs.GET;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.core.Context;
+import javax.ws.rs.core.Response;
+
+import org.apache.commons.lang.BooleanUtils;
+import org.apache.commons.lang.StringUtils;
+import org.apache.storm.daemon.common.JsonResponseBuilder;
+import org.apache.storm.daemon.logviewer.handler.LogviewerLogDownloadHandler;
+import org.apache.storm.daemon.logviewer.handler.LogviewerLogPageHandler;
+import org.apache.storm.daemon.logviewer.handler.LogviewerLogSearchHandler;
+import org.apache.storm.daemon.logviewer.handler.LogviewerProfileHandler;
+import org.apache.storm.metric.StormMetricsRegistry;
+import org.apache.storm.security.auth.IHttpCredentialsPlugin;
+import org.apache.storm.ui.InvalidRequestException;
+import org.apache.storm.ui.UIHelpers;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Handles HTTP requests for Logviewer.
+ */
+@Path("/")
+public class LogviewerResource {
+ private static final Logger LOG = LoggerFactory.getLogger(LogviewerResource.class);
+
+ private static final Meter meterLogPageHttpRequests = StormMetricsRegistry.registerMeter("logviewer:num-log-page-http-requests");
+ private static final Meter meterDaemonLogPageHttpRequests = StormMetricsRegistry.registerMeter(
+ "logviewer:num-daemonlog-page-http-requests");
+ private static final Meter meterDownloadLogFileHttpRequests = StormMetricsRegistry.registerMeter(
+ "logviewer:num-download-log-file-http-requests");
+ private static final Meter meterDownloadLogDaemonFileHttpRequests = StormMetricsRegistry.registerMeter(
+ "logviewer:num-download-log-daemon-file-http-requests");
+ private static final Meter meterListLogsHttpRequests = StormMetricsRegistry.registerMeter("logviewer:num-list-logs-http-requests");
+
+ private final LogviewerLogPageHandler logviewer;
+ private final LogviewerProfileHandler profileHandler;
+ private final LogviewerLogDownloadHandler logDownloadHandler;
+ private final LogviewerLogSearchHandler logSearchHandler;
+ private final IHttpCredentialsPlugin httpCredsHandler;
+
+ /**
+ * Constructor.
+ *
+ * @param logviewerParam {@link LogviewerLogPageHandler}
+ * @param profileHandler {@link LogviewerProfileHandler}
+ * @param logDownloadHandler {@link LogviewerLogDownloadHandler}
+ * @param logSearchHandler {@link LogviewerLogSearchHandler}
+ * @param httpCredsHandler {@link IHttpCredentialsPlugin}
+ */
+ public LogviewerResource(LogviewerLogPageHandler logviewerParam, LogviewerProfileHandler profileHandler,
+ LogviewerLogDownloadHandler logDownloadHandler, LogviewerLogSearchHandler logSearchHandler,
+ IHttpCredentialsPlugin httpCredsHandler) {
+ this.logviewer = logviewerParam;
+ this.profileHandler = profileHandler;
+ this.logDownloadHandler = logDownloadHandler;
+ this.logSearchHandler = logSearchHandler;
+ this.httpCredsHandler = httpCredsHandler;
+ }
+
+ /**
+ * Handles '/log' request.
+ */
+ @GET
+ @Path("/log")
+ public Response log(@Context HttpServletRequest request) throws IOException {
+ meterLogPageHttpRequests.mark();
+
+ try {
+ String user = httpCredsHandler.getUserName(request);
+ Integer start = request.getParameter("start") != null ? parseIntegerFromMap(request.getParameterMap(), "start") : null;
+ Integer length = request.getParameter("length") != null ? parseIntegerFromMap(request.getParameterMap(), "length") : null;
+ String decodedFileName = URLDecoder.decode(request.getParameter("file"));
+ String grep = request.getParameter("grep");
+ return logviewer.logPage(decodedFileName, start, length, grep, user);
+ } catch (InvalidRequestException e) {
+ LOG.error(e.getMessage(), e);
+ return Response.status(400).entity(e.getMessage()).build();
+ }
+ }
+
+ /**
+ * Handles '/daemonlog' request.
+ */
+ @GET
+ @Path("/daemonlog")
+ public Response daemonLog(@Context HttpServletRequest request) throws IOException {
+ meterDaemonLogPageHttpRequests.mark();
+
+ try {
+ String user = httpCredsHandler.getUserName(request);
+ Integer start = request.getParameter("start") != null ? parseIntegerFromMap(request.getParameterMap(), "start") : null;
+ Integer length = request.getParameter("length") != null ? parseIntegerFromMap(request.getParameterMap(), "length") : null;
+ String decodedFileName = URLDecoder.decode(request.getParameter("file"));
+ String grep = request.getParameter("grep");
+ return logviewer.daemonLogPage(decodedFileName, start, length, grep, user);
+ } catch (InvalidRequestException e) {
+ LOG.error(e.getMessage(), e);
+ return Response.status(400).entity(e.getMessage()).build();
+ }
+ }
+
+ /**
+ * Handles '/searchLogs' request.
+ */
+ @GET
+ @Path("/searchLogs")
+ public Response searchLogs(@Context HttpServletRequest request) throws IOException {
+ String user = httpCredsHandler.getUserName(request);
+ String topologyId = request.getParameter("topoId");
+ String portStr = request.getParameter("port");
+ String callback = request.getParameter("callback");
+ String origin = request.getHeader("Origin");
+
+ return logviewer.listLogFiles(user, portStr != null ? Integer.parseInt(portStr) : null, topologyId, callback, origin);
+ }
+
+ /**
+ * Handles '/listLogs' request.
+ */
+ @GET
+ @Path("/listLogs")
+ public Response listLogs(@Context HttpServletRequest request) throws IOException {
+ meterListLogsHttpRequests.mark();
+
+ String user = httpCredsHandler.getUserName(request);
+ String topologyId = request.getParameter("topoId");
+ String portStr = request.getParameter("port");
+ String callback = request.getParameter("callback");
+ String origin = request.getHeader("Origin");
+
+ return logviewer.listLogFiles(user, portStr != null ? Integer.parseInt(portStr) : null, topologyId, callback, origin);
+ }
+
+ /**
+ * Handles '/dumps' (listing dump files) request.
+ */
+ @GET
+ @Path("/dumps/{topo-id}/{host-port}")
+ public Response listDumpFiles(@PathParam("topo-id") String topologyId, @PathParam("host-port") String hostPort,
+ @Context HttpServletRequest request) throws IOException {
+ String user = httpCredsHandler.getUserName(request);
+ return profileHandler.listDumpFiles(topologyId, hostPort, user);
+ }
+
+ /**
+ * Handles '/dumps' (downloading specific dump file) request.
+ */
+ @GET
+ @Path("/dumps/{topo-id}/{host-port}/{filename}")
+ public Response downloadDumpFile(@PathParam("topo-id") String topologyId, @PathParam("host-port") String hostPort,
+ @PathParam("filename") String fileName, @Context HttpServletRequest request) throws IOException {
+ String user = httpCredsHandler.getUserName(request);
+ return profileHandler.downloadDumpFile(topologyId, hostPort, fileName, user);
+ }
+
+ /**
+ * Handles '/download' (downloading specific log file) request.
+ */
+ @GET
+ @Path("/download")
+ public Response downloadLogFile(@Context HttpServletRequest request) throws IOException {
+ meterDownloadLogFileHttpRequests.mark();
+
+ String user = httpCredsHandler.getUserName(request);
+ String file = request.getParameter("file");
+ String decodedFileName = URLDecoder.decode(file);
+ return logDownloadHandler.downloadLogFile(decodedFileName, user);
+ }
+
+ /**
+ * Handles '/daemondownload' (downloading specific daemon log file) request.
+ */
+ @GET
+ @Path("/daemondownload")
+ public Response downloadDaemonLogFile(@Context HttpServletRequest request) throws IOException {
+ meterDownloadLogDaemonFileHttpRequests.mark();
+
+ String user = httpCredsHandler.getUserName(request);
+ String file = request.getParameter("file");
+ String decodedFileName = URLDecoder.decode(file);
+ return logDownloadHandler.downloadDaemonLogFile(decodedFileName, user);
+ }
+
+ /**
+ * Handles '/search' (searching from specific worker or daemon log file) request.
+ */
+ @GET
+ @Path("/search/{file}")
+ public Response search(@PathParam("file") String file, @Context HttpServletRequest request) throws IOException {
+ String user = httpCredsHandler.getUserName(request);
+ boolean isDaemon = StringUtils.equals(request.getParameter("is-daemon"), "yes");
+ String decodedFileName = URLDecoder.decode(file);
+ String searchString = request.getParameter("search-string");
+ String numMatchesStr = request.getParameter("num-matches");
+ String startByteOffset = request.getParameter("start-byte-offset");
+ String callback = request.getParameter("callback");
+ String origin = request.getHeader("Origin");
+
+ try {
+ return logSearchHandler.searchLogFile(decodedFileName, user, isDaemon, searchString, numMatchesStr,
+ startByteOffset, callback, origin);
+ } catch (InvalidRequestException e) {
+ LOG.error(e.getMessage(), e);
+ return new JsonResponseBuilder().setData(UIHelpers.exceptionToJson(e)).setCallback(callback)
+ .setStatus(400).build();
+ }
+ }
+
+ /**
+ * Handles '/deepSearch' request.
+ */
+ @GET
+ @Path("/deepSearch/{topoId}")
+ public Response deepSearch(@PathParam("topoId") String topologyId,
+ @Context HttpServletRequest request) throws IOException {
+ String user = httpCredsHandler.getUserName(request);
+ String searchString = request.getParameter("search-string");
+ String numMatchesStr = request.getParameter("num-matches");
+ String portStr = request.getParameter("port");
+ String startFileOffset = request.getParameter("start-file-offset");
+ String startByteOffset = request.getParameter("start-byte-offset");
+ String searchArchived = request.getParameter("search-archived");
+ String callback = request.getParameter("callback");
+ String origin = request.getHeader("Origin");
--- End diff --
Just an FYI Headers and parameters can be marked with annotations that are pulled out as parameters. I am not sure if this is cleaner or if that would be. Bother are rather long.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #2204: STORM-1280 port backtype.storm.daemon.logviewer to java
Posted by Ethanlm <gi...@git.apache.org>.
Github user Ethanlm commented on the issue:
https://github.com/apache/storm/pull/2204
@HeartSaVioR Thanks! It's good to learn
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #2204: STORM-1280 port backtype.storm.daemon.logviewer to java
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/storm/pull/2204
@Ethanlm
I realized I missed addressing other places for parameter swapped. Just addressed and tested as well.
You can just build `storm-webapp` module and copy `storm-webapp jar` into `lib-webapp` directory.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #2204: STORM-1280 port backtype.storm.daemon.logviewer to...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/storm/pull/2204#discussion_r128401244
--- Diff: storm-webapp/src/main/java/org/apache/storm/daemon/logviewer/handler/LogviewerLogPageHandler.java ---
@@ -0,0 +1,467 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.daemon.logviewer.handler;
+
+import static j2html.TagCreator.a;
+import static j2html.TagCreator.body;
+import static j2html.TagCreator.div;
+import static j2html.TagCreator.form;
+import static j2html.TagCreator.h3;
+import static j2html.TagCreator.head;
+import static j2html.TagCreator.html;
+import static j2html.TagCreator.input;
+import static j2html.TagCreator.link;
+import static j2html.TagCreator.p;
+import static j2html.TagCreator.pre;
+import static j2html.TagCreator.select;
+import static j2html.TagCreator.text;
+import static j2html.TagCreator.title;
+import static java.util.stream.Collectors.toCollection;
+import static java.util.stream.Collectors.toList;
+import static org.apache.commons.lang.StringEscapeUtils.escapeHtml;
+
+import j2html.TagCreator;
+import j2html.tags.DomContent;
+
+import java.io.ByteArrayOutputStream;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.UncheckedIOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.zip.GZIPInputStream;
+
+import javax.ws.rs.core.Response;
+
+import org.apache.commons.lang.StringUtils;
+import org.apache.storm.daemon.logviewer.LogviewerConstant;
+import org.apache.storm.daemon.logviewer.utils.DirectoryCleaner;
+import org.apache.storm.daemon.logviewer.utils.LogviewerResponseBuilder;
+import org.apache.storm.daemon.logviewer.utils.ResourceAuthorizer;
+import org.apache.storm.daemon.logviewer.utils.WorkerLogs;
+import org.apache.storm.daemon.utils.StreamUtil;
+import org.apache.storm.daemon.utils.UrlBuilder;
+import org.apache.storm.ui.InvalidRequestException;
+import org.apache.storm.ui.UIHelpers;
+import org.apache.storm.utils.ConfigUtils;
+import org.apache.storm.utils.ServerUtils;
+import org.apache.storm.utils.Utils;
+import org.jooq.lambda.Unchecked;
+
+public class LogviewerLogPageHandler {
+ private final String logRoot;
+ private final String daemonLogRoot;
+ private final WorkerLogs workerLogs;
+ private final ResourceAuthorizer resourceAuthorizer;
+
+ /**
+ * Constructor.
+ *
+ * @param logRoot root worker log directory
+ * @param daemonLogRoot root daemon log directory
+ * @param workerLogs {@link WorkerLogs}
+ * @param resourceAuthorizer {@link ResourceAuthorizer}
+ */
+ public LogviewerLogPageHandler(String logRoot, String daemonLogRoot,
+ WorkerLogs workerLogs,
+ ResourceAuthorizer resourceAuthorizer) {
+ this.logRoot = logRoot;
+ this.daemonLogRoot = daemonLogRoot;
+ this.workerLogs = workerLogs;
+ this.resourceAuthorizer = resourceAuthorizer;
+ }
+
+ /**
+ * Enumerate worker log files for given criteria.
+ *
+ * @param user username
+ * @param port worker's port, null for all workers
+ * @param topologyId topology ID, null for all topologies
+ * @param callback callback for JSONP
+ * @param origin origin
+ * @return list of worker logs for given criteria
+ */
+ public Response listLogFiles(String user, Integer port, String topologyId, String callback, String origin) throws IOException {
+ List<File> fileResults = null;
+ if (topologyId == null) {
+ if (port == null) {
+ fileResults = workerLogs.getAllLogsForRootDir();
+ } else {
+ fileResults = new ArrayList<>();
+
+ File[] logRootFiles = new File(logRoot).listFiles();
+ if (logRootFiles != null) {
+ for (File topoDir : logRootFiles) {
+ File[] topoDirFiles = topoDir.listFiles();
+ if (topoDirFiles != null) {
+ for (File portDir : topoDirFiles) {
+ if (portDir.getName().equals(port.toString())) {
+ fileResults.addAll(DirectoryCleaner.getFilesForDir(portDir));
+ }
+ }
+ }
+ }
+ }
+ }
+ } else {
+ if (port == null) {
+ fileResults = new ArrayList<>();
+
+ File topoDir = new File(logRoot + Utils.FILE_PATH_SEPARATOR + topologyId);
--- End diff --
Yes it would be better to fix that. I'll address it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #2204: STORM-1280 port backtype.storm.daemon.logviewer to java
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/storm/pull/2204
Addressed RAT and checkstyle violation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #2204: STORM-1280 port backtype.storm.daemon.logviewer to java
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/storm/pull/2204
@Ethanlm Great news for me to figure it out. I just added pair of `username` & `username@domain` while testing so didn't recognize the issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #2204: STORM-1280 port backtype.storm.daemon.logviewer to java
Posted by Ethanlm <gi...@git.apache.org>.
Github user Ethanlm commented on the issue:
https://github.com/apache/storm/pull/2204
@HeartSaVioR Yes. Sure. Should I wait until this PR being merged in?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #2204: STORM-1280 port backtype.storm.daemon.logviewer to java
Posted by Ethanlm <gi...@git.apache.org>.
Github user Ethanlm commented on the issue:
https://github.com/apache/storm/pull/2204
@HeartSaVioR That's great. I filed a PR addressing this bug. https://github.com/apache/storm/pull/2239 Could you please review it if you get a chance? Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #2204: STORM-1280 port backtype.storm.daemon.logviewer to java
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/storm/pull/2204
@Ethanlm
Thanks for the update. I'll try to get it today and put my observation.
Btw, my additional commits didn't address anything about bug fix so it doesn't matter which commit you're testing with.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #2204: STORM-1280 port backtype.storm.daemon.logviewer to java
Posted by Ethanlm <gi...@git.apache.org>.
Github user Ethanlm commented on the issue:
https://github.com/apache/storm/pull/2204
Just tested (3). Don't know why (3) is still not working (even with nimbus admin account).
(querying worker.log as nimbus admin works.)
But I think it's fine to fix it later if it's a real problem (if it doesn't only happen on my cluster) .
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #2204: STORM-1280 port backtype.storm.daemon.logviewer to...
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2204#discussion_r128366771
--- Diff: storm-webapp/src/main/java/org/apache/storm/daemon/logviewer/webapp/LogviewerResource.java ---
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.daemon.logviewer.webapp;
+
+import com.codahale.metrics.Meter;
+
+import java.io.IOException;
+import java.net.URLDecoder;
+import java.util.Map;
+
+import javax.servlet.http.HttpServletRequest;
+import javax.ws.rs.GET;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.core.Context;
+import javax.ws.rs.core.Response;
+
+import org.apache.commons.lang.BooleanUtils;
+import org.apache.commons.lang.StringUtils;
+import org.apache.storm.daemon.common.JsonResponseBuilder;
+import org.apache.storm.daemon.logviewer.handler.LogviewerLogDownloadHandler;
+import org.apache.storm.daemon.logviewer.handler.LogviewerLogPageHandler;
+import org.apache.storm.daemon.logviewer.handler.LogviewerLogSearchHandler;
+import org.apache.storm.daemon.logviewer.handler.LogviewerProfileHandler;
+import org.apache.storm.metric.StormMetricsRegistry;
+import org.apache.storm.security.auth.IHttpCredentialsPlugin;
+import org.apache.storm.ui.InvalidRequestException;
+import org.apache.storm.ui.UIHelpers;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Handles HTTP requests for Logviewer.
+ */
+@Path("/")
+public class LogviewerResource {
+ private static final Logger LOG = LoggerFactory.getLogger(LogviewerResource.class);
+
+ private static final Meter meterLogPageHttpRequests = StormMetricsRegistry.registerMeter("logviewer:num-log-page-http-requests");
+ private static final Meter meterDaemonLogPageHttpRequests = StormMetricsRegistry.registerMeter(
+ "logviewer:num-daemonlog-page-http-requests");
+ private static final Meter meterDownloadLogFileHttpRequests = StormMetricsRegistry.registerMeter(
+ "logviewer:num-download-log-file-http-requests");
+ private static final Meter meterDownloadLogDaemonFileHttpRequests = StormMetricsRegistry.registerMeter(
+ "logviewer:num-download-log-daemon-file-http-requests");
+ private static final Meter meterListLogsHttpRequests = StormMetricsRegistry.registerMeter("logviewer:num-list-logs-http-requests");
+
+ private final LogviewerLogPageHandler logviewer;
+ private final LogviewerProfileHandler profileHandler;
+ private final LogviewerLogDownloadHandler logDownloadHandler;
+ private final LogviewerLogSearchHandler logSearchHandler;
+ private final IHttpCredentialsPlugin httpCredsHandler;
+
+ /**
+ * Constructor.
+ *
+ * @param logviewerParam {@link LogviewerLogPageHandler}
+ * @param profileHandler {@link LogviewerProfileHandler}
+ * @param logDownloadHandler {@link LogviewerLogDownloadHandler}
+ * @param logSearchHandler {@link LogviewerLogSearchHandler}
+ * @param httpCredsHandler {@link IHttpCredentialsPlugin}
+ */
+ public LogviewerResource(LogviewerLogPageHandler logviewerParam, LogviewerProfileHandler profileHandler,
+ LogviewerLogDownloadHandler logDownloadHandler, LogviewerLogSearchHandler logSearchHandler,
+ IHttpCredentialsPlugin httpCredsHandler) {
+ this.logviewer = logviewerParam;
+ this.profileHandler = profileHandler;
+ this.logDownloadHandler = logDownloadHandler;
+ this.logSearchHandler = logSearchHandler;
+ this.httpCredsHandler = httpCredsHandler;
+ }
+
+ /**
+ * Handles '/log' request.
+ */
+ @GET
+ @Path("/log")
+ public Response log(@Context HttpServletRequest request) throws IOException {
+ meterLogPageHttpRequests.mark();
+
+ try {
+ String user = httpCredsHandler.getUserName(request);
+ Integer start = request.getParameter("start") != null ? parseIntegerFromMap(request.getParameterMap(), "start") : null;
+ Integer length = request.getParameter("length") != null ? parseIntegerFromMap(request.getParameterMap(), "length") : null;
+ String decodedFileName = URLDecoder.decode(request.getParameter("file"));
+ String grep = request.getParameter("grep");
+ return logviewer.logPage(decodedFileName, start, length, grep, user);
+ } catch (InvalidRequestException e) {
+ LOG.error(e.getMessage(), e);
+ return Response.status(400).entity(e.getMessage()).build();
+ }
+ }
+
+ /**
+ * Handles '/daemonlog' request.
+ */
+ @GET
+ @Path("/daemonlog")
+ public Response daemonLog(@Context HttpServletRequest request) throws IOException {
+ meterDaemonLogPageHttpRequests.mark();
+
+ try {
+ String user = httpCredsHandler.getUserName(request);
+ Integer start = request.getParameter("start") != null ? parseIntegerFromMap(request.getParameterMap(), "start") : null;
+ Integer length = request.getParameter("length") != null ? parseIntegerFromMap(request.getParameterMap(), "length") : null;
+ String decodedFileName = URLDecoder.decode(request.getParameter("file"));
+ String grep = request.getParameter("grep");
+ return logviewer.daemonLogPage(decodedFileName, start, length, grep, user);
+ } catch (InvalidRequestException e) {
+ LOG.error(e.getMessage(), e);
+ return Response.status(400).entity(e.getMessage()).build();
+ }
+ }
+
+ /**
+ * Handles '/searchLogs' request.
+ */
+ @GET
+ @Path("/searchLogs")
+ public Response searchLogs(@Context HttpServletRequest request) throws IOException {
+ String user = httpCredsHandler.getUserName(request);
+ String topologyId = request.getParameter("topoId");
+ String portStr = request.getParameter("port");
+ String callback = request.getParameter("callback");
+ String origin = request.getHeader("Origin");
+
+ return logviewer.listLogFiles(user, portStr != null ? Integer.parseInt(portStr) : null, topologyId, callback, origin);
+ }
+
+ /**
+ * Handles '/listLogs' request.
+ */
+ @GET
+ @Path("/listLogs")
+ public Response listLogs(@Context HttpServletRequest request) throws IOException {
+ meterListLogsHttpRequests.mark();
+
+ String user = httpCredsHandler.getUserName(request);
+ String topologyId = request.getParameter("topoId");
+ String portStr = request.getParameter("port");
+ String callback = request.getParameter("callback");
+ String origin = request.getHeader("Origin");
+
+ return logviewer.listLogFiles(user, portStr != null ? Integer.parseInt(portStr) : null, topologyId, callback, origin);
+ }
+
+ /**
+ * Handles '/dumps' (listing dump files) request.
+ */
+ @GET
+ @Path("/dumps/{topo-id}/{host-port}")
+ public Response listDumpFiles(@PathParam("topo-id") String topologyId, @PathParam("host-port") String hostPort,
+ @Context HttpServletRequest request) throws IOException {
+ String user = httpCredsHandler.getUserName(request);
+ return profileHandler.listDumpFiles(topologyId, hostPort, user);
+ }
+
+ /**
+ * Handles '/dumps' (downloading specific dump file) request.
+ */
+ @GET
+ @Path("/dumps/{topo-id}/{host-port}/{filename}")
+ public Response downloadDumpFile(@PathParam("topo-id") String topologyId, @PathParam("host-port") String hostPort,
+ @PathParam("filename") String fileName, @Context HttpServletRequest request) throws IOException {
+ String user = httpCredsHandler.getUserName(request);
+ return profileHandler.downloadDumpFile(topologyId, hostPort, fileName, user);
+ }
+
+ /**
+ * Handles '/download' (downloading specific log file) request.
+ */
+ @GET
+ @Path("/download")
+ public Response downloadLogFile(@Context HttpServletRequest request) throws IOException {
+ meterDownloadLogFileHttpRequests.mark();
+
+ String user = httpCredsHandler.getUserName(request);
+ String file = request.getParameter("file");
+ String decodedFileName = URLDecoder.decode(file);
+ return logDownloadHandler.downloadLogFile(decodedFileName, user);
+ }
+
+ /**
+ * Handles '/daemondownload' (downloading specific daemon log file) request.
+ */
+ @GET
+ @Path("/daemondownload")
+ public Response downloadDaemonLogFile(@Context HttpServletRequest request) throws IOException {
+ meterDownloadLogDaemonFileHttpRequests.mark();
+
+ String user = httpCredsHandler.getUserName(request);
+ String file = request.getParameter("file");
+ String decodedFileName = URLDecoder.decode(file);
+ return logDownloadHandler.downloadDaemonLogFile(decodedFileName, user);
--- End diff --
Again InvalidRequestException?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #2204: STORM-1280 port backtype.storm.daemon.logviewer to java
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/storm/pull/2204
@Ethanlm
No problem. Thanks for testing my patch thoughtfully. Please let me know if (2) becomes fixed with setting `logs.users` in the topology configuration.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #2204: STORM-1280 port backtype.storm.daemon.logviewer to java
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on the issue:
https://github.com/apache/storm/pull/2204
The change looks good to me and I am +1. Will wait for @Ethanlm to work through his issues first though.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #2204: (WIP) STORM-1280 port backtype.storm.daemon.logviewer to ...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/storm/pull/2204
DirectoryCleaner also can be moved to storm-webapp after porting logviewer-test.clj
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #2204: STORM-1280 port backtype.storm.daemon.logviewer to...
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2204#discussion_r128366130
--- Diff: storm-webapp/src/main/java/org/apache/storm/daemon/logviewer/webapp/LogviewerResource.java ---
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.daemon.logviewer.webapp;
+
+import com.codahale.metrics.Meter;
+
+import java.io.IOException;
+import java.net.URLDecoder;
+import java.util.Map;
+
+import javax.servlet.http.HttpServletRequest;
+import javax.ws.rs.GET;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.core.Context;
+import javax.ws.rs.core.Response;
+
+import org.apache.commons.lang.BooleanUtils;
+import org.apache.commons.lang.StringUtils;
+import org.apache.storm.daemon.common.JsonResponseBuilder;
+import org.apache.storm.daemon.logviewer.handler.LogviewerLogDownloadHandler;
+import org.apache.storm.daemon.logviewer.handler.LogviewerLogPageHandler;
+import org.apache.storm.daemon.logviewer.handler.LogviewerLogSearchHandler;
+import org.apache.storm.daemon.logviewer.handler.LogviewerProfileHandler;
+import org.apache.storm.metric.StormMetricsRegistry;
+import org.apache.storm.security.auth.IHttpCredentialsPlugin;
+import org.apache.storm.ui.InvalidRequestException;
+import org.apache.storm.ui.UIHelpers;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Handles HTTP requests for Logviewer.
+ */
+@Path("/")
+public class LogviewerResource {
+ private static final Logger LOG = LoggerFactory.getLogger(LogviewerResource.class);
+
+ private static final Meter meterLogPageHttpRequests = StormMetricsRegistry.registerMeter("logviewer:num-log-page-http-requests");
+ private static final Meter meterDaemonLogPageHttpRequests = StormMetricsRegistry.registerMeter(
+ "logviewer:num-daemonlog-page-http-requests");
+ private static final Meter meterDownloadLogFileHttpRequests = StormMetricsRegistry.registerMeter(
+ "logviewer:num-download-log-file-http-requests");
+ private static final Meter meterDownloadLogDaemonFileHttpRequests = StormMetricsRegistry.registerMeter(
+ "logviewer:num-download-log-daemon-file-http-requests");
+ private static final Meter meterListLogsHttpRequests = StormMetricsRegistry.registerMeter("logviewer:num-list-logs-http-requests");
+
+ private final LogviewerLogPageHandler logviewer;
+ private final LogviewerProfileHandler profileHandler;
+ private final LogviewerLogDownloadHandler logDownloadHandler;
+ private final LogviewerLogSearchHandler logSearchHandler;
+ private final IHttpCredentialsPlugin httpCredsHandler;
+
+ /**
+ * Constructor.
+ *
+ * @param logviewerParam {@link LogviewerLogPageHandler}
+ * @param profileHandler {@link LogviewerProfileHandler}
+ * @param logDownloadHandler {@link LogviewerLogDownloadHandler}
+ * @param logSearchHandler {@link LogviewerLogSearchHandler}
+ * @param httpCredsHandler {@link IHttpCredentialsPlugin}
+ */
+ public LogviewerResource(LogviewerLogPageHandler logviewerParam, LogviewerProfileHandler profileHandler,
+ LogviewerLogDownloadHandler logDownloadHandler, LogviewerLogSearchHandler logSearchHandler,
+ IHttpCredentialsPlugin httpCredsHandler) {
+ this.logviewer = logviewerParam;
+ this.profileHandler = profileHandler;
+ this.logDownloadHandler = logDownloadHandler;
+ this.logSearchHandler = logSearchHandler;
+ this.httpCredsHandler = httpCredsHandler;
+ }
+
+ /**
+ * Handles '/log' request.
+ */
+ @GET
+ @Path("/log")
+ public Response log(@Context HttpServletRequest request) throws IOException {
+ meterLogPageHttpRequests.mark();
+
+ try {
+ String user = httpCredsHandler.getUserName(request);
+ Integer start = request.getParameter("start") != null ? parseIntegerFromMap(request.getParameterMap(), "start") : null;
+ Integer length = request.getParameter("length") != null ? parseIntegerFromMap(request.getParameterMap(), "length") : null;
+ String decodedFileName = URLDecoder.decode(request.getParameter("file"));
+ String grep = request.getParameter("grep");
+ return logviewer.logPage(decodedFileName, start, length, grep, user);
+ } catch (InvalidRequestException e) {
+ LOG.error(e.getMessage(), e);
+ return Response.status(400).entity(e.getMessage()).build();
+ }
+ }
+
+ /**
+ * Handles '/daemonlog' request.
+ */
+ @GET
+ @Path("/daemonlog")
+ public Response daemonLog(@Context HttpServletRequest request) throws IOException {
+ meterDaemonLogPageHttpRequests.mark();
+
+ try {
+ String user = httpCredsHandler.getUserName(request);
+ Integer start = request.getParameter("start") != null ? parseIntegerFromMap(request.getParameterMap(), "start") : null;
+ Integer length = request.getParameter("length") != null ? parseIntegerFromMap(request.getParameterMap(), "length") : null;
+ String decodedFileName = URLDecoder.decode(request.getParameter("file"));
+ String grep = request.getParameter("grep");
+ return logviewer.daemonLogPage(decodedFileName, start, length, grep, user);
+ } catch (InvalidRequestException e) {
+ LOG.error(e.getMessage(), e);
+ return Response.status(400).entity(e.getMessage()).build();
+ }
+ }
+
+ /**
+ * Handles '/searchLogs' request.
+ */
+ @GET
+ @Path("/searchLogs")
+ public Response searchLogs(@Context HttpServletRequest request) throws IOException {
+ String user = httpCredsHandler.getUserName(request);
+ String topologyId = request.getParameter("topoId");
+ String portStr = request.getParameter("port");
+ String callback = request.getParameter("callback");
+ String origin = request.getHeader("Origin");
+
+ return logviewer.listLogFiles(user, portStr != null ? Integer.parseInt(portStr) : null, topologyId, callback, origin);
--- End diff --
In the original code we handled an InvalidRequestException. Do we need to do that here too?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #2204: STORM-1280 port backtype.storm.daemon.logviewer to java
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/storm/pull/2204
Removed WIP though I still need to address checkstyle issues. I think check style violation doesn't hurt review process.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #2204: STORM-1280 port backtype.storm.daemon.logviewer to java
Posted by Ethanlm <gi...@git.apache.org>.
Github user Ethanlm commented on the issue:
https://github.com/apache/storm/pull/2204
@HeartSaVioR @revans2 I think I figured it out.
I added "ethan@DEREKD.YSTORM.NET" to logs.users and then it works. I traced down to the [isAuthorizedLogUser](https://github.com/apache/storm/pull/2204/files#diff-aad25f432acc108d98b3126bbd914873R74) function and found that it doesn't translate "principal" to "user" so that "ethan@DEREKD.YSTORM.NET" doesn't match with logs.user "ethan" in the configuration. @revans2 and I think the code is identical to the old code. So this should an existing bug.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #2204: (WIP) STORM-1280 port backtype.storm.daemon.logviewer to ...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/storm/pull/2204
Applied recent change of logviewer and rebased.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #2204: STORM-1280 port backtype.storm.daemon.logviewer to...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/storm/pull/2204#discussion_r128402280
--- Diff: storm-webapp/src/main/java/org/apache/storm/daemon/logviewer/webapp/LogviewerResource.java ---
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.daemon.logviewer.webapp;
+
+import com.codahale.metrics.Meter;
+
+import java.io.IOException;
+import java.net.URLDecoder;
+import java.util.Map;
+
+import javax.servlet.http.HttpServletRequest;
+import javax.ws.rs.GET;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.core.Context;
+import javax.ws.rs.core.Response;
+
+import org.apache.commons.lang.BooleanUtils;
+import org.apache.commons.lang.StringUtils;
+import org.apache.storm.daemon.common.JsonResponseBuilder;
+import org.apache.storm.daemon.logviewer.handler.LogviewerLogDownloadHandler;
+import org.apache.storm.daemon.logviewer.handler.LogviewerLogPageHandler;
+import org.apache.storm.daemon.logviewer.handler.LogviewerLogSearchHandler;
+import org.apache.storm.daemon.logviewer.handler.LogviewerProfileHandler;
+import org.apache.storm.metric.StormMetricsRegistry;
+import org.apache.storm.security.auth.IHttpCredentialsPlugin;
+import org.apache.storm.ui.InvalidRequestException;
+import org.apache.storm.ui.UIHelpers;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Handles HTTP requests for Logviewer.
+ */
+@Path("/")
+public class LogviewerResource {
+ private static final Logger LOG = LoggerFactory.getLogger(LogviewerResource.class);
+
+ private static final Meter meterLogPageHttpRequests = StormMetricsRegistry.registerMeter("logviewer:num-log-page-http-requests");
+ private static final Meter meterDaemonLogPageHttpRequests = StormMetricsRegistry.registerMeter(
+ "logviewer:num-daemonlog-page-http-requests");
+ private static final Meter meterDownloadLogFileHttpRequests = StormMetricsRegistry.registerMeter(
+ "logviewer:num-download-log-file-http-requests");
+ private static final Meter meterDownloadLogDaemonFileHttpRequests = StormMetricsRegistry.registerMeter(
+ "logviewer:num-download-log-daemon-file-http-requests");
+ private static final Meter meterListLogsHttpRequests = StormMetricsRegistry.registerMeter("logviewer:num-list-logs-http-requests");
+
+ private final LogviewerLogPageHandler logviewer;
+ private final LogviewerProfileHandler profileHandler;
+ private final LogviewerLogDownloadHandler logDownloadHandler;
+ private final LogviewerLogSearchHandler logSearchHandler;
+ private final IHttpCredentialsPlugin httpCredsHandler;
+
+ /**
+ * Constructor.
+ *
+ * @param logviewerParam {@link LogviewerLogPageHandler}
+ * @param profileHandler {@link LogviewerProfileHandler}
+ * @param logDownloadHandler {@link LogviewerLogDownloadHandler}
+ * @param logSearchHandler {@link LogviewerLogSearchHandler}
+ * @param httpCredsHandler {@link IHttpCredentialsPlugin}
+ */
+ public LogviewerResource(LogviewerLogPageHandler logviewerParam, LogviewerProfileHandler profileHandler,
+ LogviewerLogDownloadHandler logDownloadHandler, LogviewerLogSearchHandler logSearchHandler,
+ IHttpCredentialsPlugin httpCredsHandler) {
+ this.logviewer = logviewerParam;
+ this.profileHandler = profileHandler;
+ this.logDownloadHandler = logDownloadHandler;
+ this.logSearchHandler = logSearchHandler;
+ this.httpCredsHandler = httpCredsHandler;
+ }
+
+ /**
+ * Handles '/log' request.
+ */
+ @GET
+ @Path("/log")
+ public Response log(@Context HttpServletRequest request) throws IOException {
+ meterLogPageHttpRequests.mark();
+
+ try {
+ String user = httpCredsHandler.getUserName(request);
+ Integer start = request.getParameter("start") != null ? parseIntegerFromMap(request.getParameterMap(), "start") : null;
+ Integer length = request.getParameter("length") != null ? parseIntegerFromMap(request.getParameterMap(), "length") : null;
+ String decodedFileName = URLDecoder.decode(request.getParameter("file"));
+ String grep = request.getParameter("grep");
+ return logviewer.logPage(decodedFileName, start, length, grep, user);
+ } catch (InvalidRequestException e) {
+ LOG.error(e.getMessage(), e);
+ return Response.status(400).entity(e.getMessage()).build();
+ }
+ }
+
+ /**
+ * Handles '/daemonlog' request.
+ */
+ @GET
+ @Path("/daemonlog")
+ public Response daemonLog(@Context HttpServletRequest request) throws IOException {
+ meterDaemonLogPageHttpRequests.mark();
+
+ try {
+ String user = httpCredsHandler.getUserName(request);
+ Integer start = request.getParameter("start") != null ? parseIntegerFromMap(request.getParameterMap(), "start") : null;
+ Integer length = request.getParameter("length") != null ? parseIntegerFromMap(request.getParameterMap(), "length") : null;
+ String decodedFileName = URLDecoder.decode(request.getParameter("file"));
+ String grep = request.getParameter("grep");
+ return logviewer.daemonLogPage(decodedFileName, start, length, grep, user);
+ } catch (InvalidRequestException e) {
+ LOG.error(e.getMessage(), e);
+ return Response.status(400).entity(e.getMessage()).build();
+ }
+ }
+
+ /**
+ * Handles '/searchLogs' request.
+ */
+ @GET
+ @Path("/searchLogs")
+ public Response searchLogs(@Context HttpServletRequest request) throws IOException {
+ String user = httpCredsHandler.getUserName(request);
+ String topologyId = request.getParameter("topoId");
+ String portStr = request.getParameter("port");
+ String callback = request.getParameter("callback");
+ String origin = request.getHeader("Origin");
+
+ return logviewer.listLogFiles(user, portStr != null ? Integer.parseInt(portStr) : null, topologyId, callback, origin);
+ }
+
+ /**
+ * Handles '/listLogs' request.
+ */
+ @GET
+ @Path("/listLogs")
+ public Response listLogs(@Context HttpServletRequest request) throws IOException {
+ meterListLogsHttpRequests.mark();
+
+ String user = httpCredsHandler.getUserName(request);
+ String topologyId = request.getParameter("topoId");
+ String portStr = request.getParameter("port");
+ String callback = request.getParameter("callback");
+ String origin = request.getHeader("Origin");
+
+ return logviewer.listLogFiles(user, portStr != null ? Integer.parseInt(portStr) : null, topologyId, callback, origin);
+ }
+
+ /**
+ * Handles '/dumps' (listing dump files) request.
+ */
+ @GET
+ @Path("/dumps/{topo-id}/{host-port}")
+ public Response listDumpFiles(@PathParam("topo-id") String topologyId, @PathParam("host-port") String hostPort,
+ @Context HttpServletRequest request) throws IOException {
+ String user = httpCredsHandler.getUserName(request);
+ return profileHandler.listDumpFiles(topologyId, hostPort, user);
+ }
+
+ /**
+ * Handles '/dumps' (downloading specific dump file) request.
+ */
+ @GET
+ @Path("/dumps/{topo-id}/{host-port}/{filename}")
+ public Response downloadDumpFile(@PathParam("topo-id") String topologyId, @PathParam("host-port") String hostPort,
+ @PathParam("filename") String fileName, @Context HttpServletRequest request) throws IOException {
+ String user = httpCredsHandler.getUserName(request);
+ return profileHandler.downloadDumpFile(topologyId, hostPort, fileName, user);
+ }
+
+ /**
+ * Handles '/download' (downloading specific log file) request.
+ */
+ @GET
+ @Path("/download")
+ public Response downloadLogFile(@Context HttpServletRequest request) throws IOException {
+ meterDownloadLogFileHttpRequests.mark();
+
+ String user = httpCredsHandler.getUserName(request);
+ String file = request.getParameter("file");
+ String decodedFileName = URLDecoder.decode(file);
+ return logDownloadHandler.downloadLogFile(decodedFileName, user);
--- End diff --
Same as above.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #2204: STORM-1280 port backtype.storm.daemon.logviewer to java
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/storm/pull/2204
Test is failing with RAT issue.
```
[WARNING] Files with unapproved licenses:
storm-webapp/src/test/resources/test-3072.log.test
storm-webapp/src/test/resources/small-worker.log.test
storm-webapp/src/test/resources/test-worker.log.test
storm-webapp/src/test/resources/logviewer-search-context-tests.log.test
```
I'll address it after addressing checkstyle.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #2204: STORM-1280 port backtype.storm.daemon.logviewer to...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/storm/pull/2204#discussion_r128404026
--- Diff: storm-webapp/src/main/java/org/apache/storm/daemon/logviewer/LogviewerServer.java ---
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.daemon.logviewer;
+
+import static org.apache.storm.DaemonConfig.UI_HEADER_BUFFER_BYTES;
+
+import com.codahale.metrics.Meter;
+import com.google.common.annotations.VisibleForTesting;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.storm.DaemonConfig;
+import org.apache.storm.daemon.logviewer.utils.DirectoryCleaner;
+import org.apache.storm.daemon.logviewer.utils.LogCleaner;
+import org.apache.storm.daemon.logviewer.utils.WorkerLogs;
+import org.apache.storm.daemon.logviewer.webapp.LogviewerApplication;
+import org.apache.storm.metric.StormMetricsRegistry;
+import org.apache.storm.ui.FilterConfiguration;
+import org.apache.storm.ui.UIHelpers;
+import org.apache.storm.utils.ConfigUtils;
+import org.apache.storm.utils.ObjectReader;
+import org.apache.storm.utils.Utils;
+import org.eclipse.jetty.server.Server;
+import org.eclipse.jetty.servlet.DefaultServlet;
+import org.eclipse.jetty.servlet.ServletContextHandler;
+import org.eclipse.jetty.servlet.ServletHolder;
+import org.eclipse.jetty.util.resource.Resource;
+import org.glassfish.jersey.servlet.ServletContainer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The main entry of Logviewer.
+ */
+public class LogviewerServer implements AutoCloseable {
+ private static final Logger LOG = LoggerFactory.getLogger(LogviewerServer.class);
+ private static final Meter meterShutdownCalls = StormMetricsRegistry.registerMeter("drpc:num-shutdown-calls");
+ public static final String STATIC_RESOURCE_DIRECTORY_PATH = "./public";
+
+ private static Server mkHttpServer(Map<String, Object> conf) {
+ Integer logviewerHttpPort = (Integer) conf.get(DaemonConfig.LOGVIEWER_PORT);
+ Server ret = null;
+ if (logviewerHttpPort != null && logviewerHttpPort >= 0) {
+ LOG.info("Starting Logviewer HTTP servers...");
+ Integer headerBufferSize = ObjectReader.getInt(conf.get(UI_HEADER_BUFFER_BYTES));
+ String filterClass = (String) (conf.get(DaemonConfig.UI_FILTER));
+ @SuppressWarnings("unchecked")
+ Map<String, String> filterParams = (Map<String, String>) (conf.get(DaemonConfig.UI_FILTER_PARAMS));
+ FilterConfiguration filterConfiguration = new FilterConfiguration(filterClass, filterParams);
+ final List<FilterConfiguration> filterConfigurations = Arrays.asList(filterConfiguration);
+
+ final Integer httpsPort = ObjectReader.getInt(conf.get(DaemonConfig.LOGVIEWER_HTTPS_PORT), 0);
+ final String httpsKsPath = (String) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_KEYSTORE_PATH));
+ final String httpsKsPassword = (String) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_KEYSTORE_PASSWORD));
+ final String httpsKsType = (String) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_KEYSTORE_TYPE));
+ final String httpsKeyPassword = (String) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_KEY_PASSWORD));
+ final String httpsTsPath = (String) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_TRUSTSTORE_PATH));
+ final String httpsTsPassword = (String) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_TRUSTSTORE_PASSWORD));
+ final String httpsTsType = (String) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_TRUSTSTORE_TYPE));
+ final Boolean httpsWantClientAuth = (Boolean) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_WANT_CLIENT_AUTH));
+ final Boolean httpsNeedClientAuth = (Boolean) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_NEED_CLIENT_AUTH));
+
+ //TODO a better way to do this would be great.
--- End diff --
The comment was copied and pasted from DRPCServer.
While porting I thought about how to do it better, but no idea given that LogviewerApplication instance is generated dynamically. It may be possible to not passing config map and re-read storm configuration in LogviewerApplication, but it is duplicated effort and we will not be able to inject other config map in unit test.
Maybe we could just remove the TODO comment.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #2204: STORM-1280 port backtype.storm.daemon.logviewer to...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/storm/pull/2204#discussion_r128402564
--- Diff: storm-webapp/src/main/java/org/apache/storm/daemon/logviewer/LogviewerServer.java ---
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.daemon.logviewer;
+
+import static org.apache.storm.DaemonConfig.UI_HEADER_BUFFER_BYTES;
+
+import com.codahale.metrics.Meter;
+import com.google.common.annotations.VisibleForTesting;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.storm.DaemonConfig;
+import org.apache.storm.daemon.logviewer.utils.DirectoryCleaner;
+import org.apache.storm.daemon.logviewer.utils.LogCleaner;
+import org.apache.storm.daemon.logviewer.utils.WorkerLogs;
+import org.apache.storm.daemon.logviewer.webapp.LogviewerApplication;
+import org.apache.storm.metric.StormMetricsRegistry;
+import org.apache.storm.ui.FilterConfiguration;
+import org.apache.storm.ui.UIHelpers;
+import org.apache.storm.utils.ConfigUtils;
+import org.apache.storm.utils.ObjectReader;
+import org.apache.storm.utils.Utils;
+import org.eclipse.jetty.server.Server;
+import org.eclipse.jetty.servlet.DefaultServlet;
+import org.eclipse.jetty.servlet.ServletContextHandler;
+import org.eclipse.jetty.servlet.ServletHolder;
+import org.eclipse.jetty.util.resource.Resource;
+import org.glassfish.jersey.servlet.ServletContainer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The main entry of Logviewer.
+ */
+public class LogviewerServer implements AutoCloseable {
+ private static final Logger LOG = LoggerFactory.getLogger(LogviewerServer.class);
+ private static final Meter meterShutdownCalls = StormMetricsRegistry.registerMeter("drpc:num-shutdown-calls");
+ public static final String STATIC_RESOURCE_DIRECTORY_PATH = "./public";
+
+ private static Server mkHttpServer(Map<String, Object> conf) {
+ Integer logviewerHttpPort = (Integer) conf.get(DaemonConfig.LOGVIEWER_PORT);
+ Server ret = null;
+ if (logviewerHttpPort != null && logviewerHttpPort >= 0) {
+ LOG.info("Starting Logviewer HTTP servers...");
+ Integer headerBufferSize = ObjectReader.getInt(conf.get(UI_HEADER_BUFFER_BYTES));
+ String filterClass = (String) (conf.get(DaemonConfig.UI_FILTER));
+ @SuppressWarnings("unchecked")
+ Map<String, String> filterParams = (Map<String, String>) (conf.get(DaemonConfig.UI_FILTER_PARAMS));
+ FilterConfiguration filterConfiguration = new FilterConfiguration(filterClass, filterParams);
+ final List<FilterConfiguration> filterConfigurations = Arrays.asList(filterConfiguration);
+
+ final Integer httpsPort = ObjectReader.getInt(conf.get(DaemonConfig.LOGVIEWER_HTTPS_PORT), 0);
+ final String httpsKsPath = (String) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_KEYSTORE_PATH));
+ final String httpsKsPassword = (String) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_KEYSTORE_PASSWORD));
+ final String httpsKsType = (String) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_KEYSTORE_TYPE));
+ final String httpsKeyPassword = (String) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_KEY_PASSWORD));
+ final String httpsTsPath = (String) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_TRUSTSTORE_PATH));
+ final String httpsTsPassword = (String) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_TRUSTSTORE_PASSWORD));
+ final String httpsTsType = (String) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_TRUSTSTORE_TYPE));
+ final Boolean httpsWantClientAuth = (Boolean) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_WANT_CLIENT_AUTH));
+ final Boolean httpsNeedClientAuth = (Boolean) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_NEED_CLIENT_AUTH));
+
+ //TODO a better way to do this would be great.
+ LogviewerApplication.setup(conf);
+ ret = UIHelpers.jettyCreateServer(logviewerHttpPort, null, httpsPort);
+
+ UIHelpers.configSsl(ret, httpsPort, httpsKsPath, httpsKsPassword, httpsKsType, httpsKeyPassword,
+ httpsTsPath, httpsTsPassword, httpsTsType, httpsNeedClientAuth, httpsWantClientAuth);
+
+ ServletContextHandler context = new ServletContextHandler(ServletContextHandler.NO_SESSIONS);
+ try {
+ context.setBaseResource(Resource.newResource(STATIC_RESOURCE_DIRECTORY_PATH));
+ } catch (IOException e) {
+ throw new RuntimeException("Can't locate static resource directory " + STATIC_RESOURCE_DIRECTORY_PATH);
+ }
+
+ context.setContextPath("/");
+ ret.setHandler(context);
+
+ ServletHolder holderPwd = new ServletHolder("default", DefaultServlet.class);
+ holderPwd.setInitOrder(1);
+ context.addServlet(holderPwd,"/");
+
+ ServletHolder jerseyServlet = context.addServlet(ServletContainer.class, "/api/v1/*");
+ jerseyServlet.setInitOrder(2);
+ jerseyServlet.setInitParameter("javax.ws.rs.Application", LogviewerApplication.class.getName());
+
+ UIHelpers.configFilters(context, filterConfigurations);
+ }
+ return ret;
+ }
+
+ private final Server httpServer;
+ private boolean closed = false;
+
+ /**
+ * Constructor.
+ * @param conf Logviewer conf for the servers
+ */
+ public LogviewerServer(Map<String, Object> conf) {
+ httpServer = mkHttpServer(conf);
+ }
+
+ @VisibleForTesting
+ void start() throws Exception {
+ LOG.info("Starting Logviewer...");
+ if (httpServer != null) {
+ httpServer.start();
+ }
+ }
+
+ @VisibleForTesting
+ void awaitTermination() throws InterruptedException {
+ httpServer.join();
+ }
+
+ @Override
+ public synchronized void close() {
+ if (!closed) {
+ //This is kind of useless...
+ meterShutdownCalls.mark();
+
+ //TODO this is causing issues...
--- End diff --
Actually most of part for LogviewerServer is copy and paste of DRPCServer, and that comment was from there. Could you revisit DRPCServer and address comments?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #2204: STORM-1280 port backtype.storm.daemon.logviewer to java
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on the issue:
https://github.com/apache/storm/pull/2204
I was playing around with it and I am seeing `>` in the output log. i.e.
```
2017-07-19 15:00:49.413 o.a.s.d.s.Slot SLOT_6701 [INFO] STATE EMPTY msInState: 38110 -> WAITING_FOR_BASIC_LOCALIZATION msInState: 1
```
It looks like we might be double escaping some things.
I'll play around with it some more, and then start looking through the code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #2204: STORM-1280 port backtype.storm.daemon.logviewer to java
Posted by Ethanlm <gi...@git.apache.org>.
Github user Ethanlm commented on the issue:
https://github.com/apache/storm/pull/2204
I did some simple tests manually in non-secure and secure storm cluster. It seems that it works well on non-secure storm cluster except some display issues. I found some issues on secure storm cluster. I am not sure if these only happen in my cluster. Thanks.
1. "Prev" "First" etc buttons seems to mix together. (On non-secure and secure cluster)
2. Got "page not found" while querying worker.log on **secure** cluster. The file actually exists.
3. Got "Request failed" which querying dumps.
The error message from logviewer.log:
<img width="728" alt="screen shot 2017-07-18 at 1 42 39 pm" src="https://user-images.githubusercontent.com/14900612/28334447-d2182134-6bc0-11e7-93b7-a02ee68ea421.png">
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #2204: (WIP) STORM-1280 port backtype.storm.daemon.logviewer to ...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/storm/pull/2204
Also removed unused imports from other spots. I hope this is OK.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #2204: STORM-1280 port backtype.storm.daemon.logviewer to java
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on the issue:
https://github.com/apache/storm/pull/2204
Overall it looks good. At least as an accurate translation of the clojure code. I want to play around with it some more, and also try to understand what is happening with some of the escaped charters (just to see if it was happening before too).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #2204: STORM-1280 port backtype.storm.daemon.logviewer to...
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2204#discussion_r128366249
--- Diff: storm-webapp/src/main/java/org/apache/storm/daemon/logviewer/webapp/LogviewerResource.java ---
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.daemon.logviewer.webapp;
+
+import com.codahale.metrics.Meter;
+
+import java.io.IOException;
+import java.net.URLDecoder;
+import java.util.Map;
+
+import javax.servlet.http.HttpServletRequest;
+import javax.ws.rs.GET;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.core.Context;
+import javax.ws.rs.core.Response;
+
+import org.apache.commons.lang.BooleanUtils;
+import org.apache.commons.lang.StringUtils;
+import org.apache.storm.daemon.common.JsonResponseBuilder;
+import org.apache.storm.daemon.logviewer.handler.LogviewerLogDownloadHandler;
+import org.apache.storm.daemon.logviewer.handler.LogviewerLogPageHandler;
+import org.apache.storm.daemon.logviewer.handler.LogviewerLogSearchHandler;
+import org.apache.storm.daemon.logviewer.handler.LogviewerProfileHandler;
+import org.apache.storm.metric.StormMetricsRegistry;
+import org.apache.storm.security.auth.IHttpCredentialsPlugin;
+import org.apache.storm.ui.InvalidRequestException;
+import org.apache.storm.ui.UIHelpers;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Handles HTTP requests for Logviewer.
+ */
+@Path("/")
+public class LogviewerResource {
+ private static final Logger LOG = LoggerFactory.getLogger(LogviewerResource.class);
+
+ private static final Meter meterLogPageHttpRequests = StormMetricsRegistry.registerMeter("logviewer:num-log-page-http-requests");
+ private static final Meter meterDaemonLogPageHttpRequests = StormMetricsRegistry.registerMeter(
+ "logviewer:num-daemonlog-page-http-requests");
+ private static final Meter meterDownloadLogFileHttpRequests = StormMetricsRegistry.registerMeter(
+ "logviewer:num-download-log-file-http-requests");
+ private static final Meter meterDownloadLogDaemonFileHttpRequests = StormMetricsRegistry.registerMeter(
+ "logviewer:num-download-log-daemon-file-http-requests");
+ private static final Meter meterListLogsHttpRequests = StormMetricsRegistry.registerMeter("logviewer:num-list-logs-http-requests");
+
+ private final LogviewerLogPageHandler logviewer;
+ private final LogviewerProfileHandler profileHandler;
+ private final LogviewerLogDownloadHandler logDownloadHandler;
+ private final LogviewerLogSearchHandler logSearchHandler;
+ private final IHttpCredentialsPlugin httpCredsHandler;
+
+ /**
+ * Constructor.
+ *
+ * @param logviewerParam {@link LogviewerLogPageHandler}
+ * @param profileHandler {@link LogviewerProfileHandler}
+ * @param logDownloadHandler {@link LogviewerLogDownloadHandler}
+ * @param logSearchHandler {@link LogviewerLogSearchHandler}
+ * @param httpCredsHandler {@link IHttpCredentialsPlugin}
+ */
+ public LogviewerResource(LogviewerLogPageHandler logviewerParam, LogviewerProfileHandler profileHandler,
+ LogviewerLogDownloadHandler logDownloadHandler, LogviewerLogSearchHandler logSearchHandler,
+ IHttpCredentialsPlugin httpCredsHandler) {
+ this.logviewer = logviewerParam;
+ this.profileHandler = profileHandler;
+ this.logDownloadHandler = logDownloadHandler;
+ this.logSearchHandler = logSearchHandler;
+ this.httpCredsHandler = httpCredsHandler;
+ }
+
+ /**
+ * Handles '/log' request.
+ */
+ @GET
+ @Path("/log")
+ public Response log(@Context HttpServletRequest request) throws IOException {
+ meterLogPageHttpRequests.mark();
+
+ try {
+ String user = httpCredsHandler.getUserName(request);
+ Integer start = request.getParameter("start") != null ? parseIntegerFromMap(request.getParameterMap(), "start") : null;
+ Integer length = request.getParameter("length") != null ? parseIntegerFromMap(request.getParameterMap(), "length") : null;
+ String decodedFileName = URLDecoder.decode(request.getParameter("file"));
+ String grep = request.getParameter("grep");
+ return logviewer.logPage(decodedFileName, start, length, grep, user);
+ } catch (InvalidRequestException e) {
+ LOG.error(e.getMessage(), e);
+ return Response.status(400).entity(e.getMessage()).build();
+ }
+ }
+
+ /**
+ * Handles '/daemonlog' request.
+ */
+ @GET
+ @Path("/daemonlog")
+ public Response daemonLog(@Context HttpServletRequest request) throws IOException {
+ meterDaemonLogPageHttpRequests.mark();
+
+ try {
+ String user = httpCredsHandler.getUserName(request);
+ Integer start = request.getParameter("start") != null ? parseIntegerFromMap(request.getParameterMap(), "start") : null;
+ Integer length = request.getParameter("length") != null ? parseIntegerFromMap(request.getParameterMap(), "length") : null;
+ String decodedFileName = URLDecoder.decode(request.getParameter("file"));
+ String grep = request.getParameter("grep");
+ return logviewer.daemonLogPage(decodedFileName, start, length, grep, user);
+ } catch (InvalidRequestException e) {
+ LOG.error(e.getMessage(), e);
+ return Response.status(400).entity(e.getMessage()).build();
+ }
+ }
+
+ /**
+ * Handles '/searchLogs' request.
+ */
+ @GET
+ @Path("/searchLogs")
+ public Response searchLogs(@Context HttpServletRequest request) throws IOException {
+ String user = httpCredsHandler.getUserName(request);
+ String topologyId = request.getParameter("topoId");
+ String portStr = request.getParameter("port");
+ String callback = request.getParameter("callback");
+ String origin = request.getHeader("Origin");
+
+ return logviewer.listLogFiles(user, portStr != null ? Integer.parseInt(portStr) : null, topologyId, callback, origin);
+ }
+
+ /**
+ * Handles '/listLogs' request.
+ */
+ @GET
+ @Path("/listLogs")
+ public Response listLogs(@Context HttpServletRequest request) throws IOException {
+ meterListLogsHttpRequests.mark();
+
+ String user = httpCredsHandler.getUserName(request);
+ String topologyId = request.getParameter("topoId");
+ String portStr = request.getParameter("port");
+ String callback = request.getParameter("callback");
+ String origin = request.getHeader("Origin");
+
+ return logviewer.listLogFiles(user, portStr != null ? Integer.parseInt(portStr) : null, topologyId, callback, origin);
--- End diff --
Do we need to handle an InvalidRequestException?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #2204: STORM-1280 port backtype.storm.daemon.logviewer to...
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2204#discussion_r128356585
--- Diff: storm-webapp/src/main/java/org/apache/storm/daemon/logviewer/LogviewerServer.java ---
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.daemon.logviewer;
+
+import static org.apache.storm.DaemonConfig.UI_HEADER_BUFFER_BYTES;
+
+import com.codahale.metrics.Meter;
+import com.google.common.annotations.VisibleForTesting;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.storm.DaemonConfig;
+import org.apache.storm.daemon.logviewer.utils.DirectoryCleaner;
+import org.apache.storm.daemon.logviewer.utils.LogCleaner;
+import org.apache.storm.daemon.logviewer.utils.WorkerLogs;
+import org.apache.storm.daemon.logviewer.webapp.LogviewerApplication;
+import org.apache.storm.metric.StormMetricsRegistry;
+import org.apache.storm.ui.FilterConfiguration;
+import org.apache.storm.ui.UIHelpers;
+import org.apache.storm.utils.ConfigUtils;
+import org.apache.storm.utils.ObjectReader;
+import org.apache.storm.utils.Utils;
+import org.eclipse.jetty.server.Server;
+import org.eclipse.jetty.servlet.DefaultServlet;
+import org.eclipse.jetty.servlet.ServletContextHandler;
+import org.eclipse.jetty.servlet.ServletHolder;
+import org.eclipse.jetty.util.resource.Resource;
+import org.glassfish.jersey.servlet.ServletContainer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The main entry of Logviewer.
+ */
+public class LogviewerServer implements AutoCloseable {
+ private static final Logger LOG = LoggerFactory.getLogger(LogviewerServer.class);
+ private static final Meter meterShutdownCalls = StormMetricsRegistry.registerMeter("drpc:num-shutdown-calls");
+ public static final String STATIC_RESOURCE_DIRECTORY_PATH = "./public";
+
+ private static Server mkHttpServer(Map<String, Object> conf) {
+ Integer logviewerHttpPort = (Integer) conf.get(DaemonConfig.LOGVIEWER_PORT);
+ Server ret = null;
+ if (logviewerHttpPort != null && logviewerHttpPort >= 0) {
+ LOG.info("Starting Logviewer HTTP servers...");
+ Integer headerBufferSize = ObjectReader.getInt(conf.get(UI_HEADER_BUFFER_BYTES));
+ String filterClass = (String) (conf.get(DaemonConfig.UI_FILTER));
+ @SuppressWarnings("unchecked")
+ Map<String, String> filterParams = (Map<String, String>) (conf.get(DaemonConfig.UI_FILTER_PARAMS));
+ FilterConfiguration filterConfiguration = new FilterConfiguration(filterClass, filterParams);
+ final List<FilterConfiguration> filterConfigurations = Arrays.asList(filterConfiguration);
+
+ final Integer httpsPort = ObjectReader.getInt(conf.get(DaemonConfig.LOGVIEWER_HTTPS_PORT), 0);
+ final String httpsKsPath = (String) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_KEYSTORE_PATH));
+ final String httpsKsPassword = (String) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_KEYSTORE_PASSWORD));
+ final String httpsKsType = (String) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_KEYSTORE_TYPE));
+ final String httpsKeyPassword = (String) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_KEY_PASSWORD));
+ final String httpsTsPath = (String) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_TRUSTSTORE_PATH));
+ final String httpsTsPassword = (String) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_TRUSTSTORE_PASSWORD));
+ final String httpsTsType = (String) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_TRUSTSTORE_TYPE));
+ final Boolean httpsWantClientAuth = (Boolean) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_WANT_CLIENT_AUTH));
+ final Boolean httpsNeedClientAuth = (Boolean) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_NEED_CLIENT_AUTH));
+
+ //TODO a better way to do this would be great.
+ LogviewerApplication.setup(conf);
+ ret = UIHelpers.jettyCreateServer(logviewerHttpPort, null, httpsPort);
+
+ UIHelpers.configSsl(ret, httpsPort, httpsKsPath, httpsKsPassword, httpsKsType, httpsKeyPassword,
+ httpsTsPath, httpsTsPassword, httpsTsType, httpsNeedClientAuth, httpsWantClientAuth);
+
+ ServletContextHandler context = new ServletContextHandler(ServletContextHandler.NO_SESSIONS);
+ try {
+ context.setBaseResource(Resource.newResource(STATIC_RESOURCE_DIRECTORY_PATH));
+ } catch (IOException e) {
+ throw new RuntimeException("Can't locate static resource directory " + STATIC_RESOURCE_DIRECTORY_PATH);
+ }
+
+ context.setContextPath("/");
+ ret.setHandler(context);
+
+ ServletHolder holderPwd = new ServletHolder("default", DefaultServlet.class);
+ holderPwd.setInitOrder(1);
+ context.addServlet(holderPwd,"/");
+
+ ServletHolder jerseyServlet = context.addServlet(ServletContainer.class, "/api/v1/*");
+ jerseyServlet.setInitOrder(2);
+ jerseyServlet.setInitParameter("javax.ws.rs.Application", LogviewerApplication.class.getName());
+
+ UIHelpers.configFilters(context, filterConfigurations);
+ }
+ return ret;
+ }
+
+ private final Server httpServer;
+ private boolean closed = false;
+
+ /**
+ * Constructor.
+ * @param conf Logviewer conf for the servers
+ */
+ public LogviewerServer(Map<String, Object> conf) {
+ httpServer = mkHttpServer(conf);
+ }
+
+ @VisibleForTesting
+ void start() throws Exception {
+ LOG.info("Starting Logviewer...");
+ if (httpServer != null) {
+ httpServer.start();
+ }
+ }
+
+ @VisibleForTesting
+ void awaitTermination() throws InterruptedException {
+ httpServer.join();
+ }
+
+ @Override
+ public synchronized void close() {
+ if (!closed) {
+ //This is kind of useless...
+ meterShutdownCalls.mark();
+
+ //TODO this is causing issues...
--- End diff --
What issues is this causing?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #2204: (WIP) STORM-1280 port backtype.storm.daemon.logviewer to ...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/storm/pull/2204
UPDATE:
I also ported logviewer-test.clj to Java. Some tests are failing and I'm looking into it.
I still need to apply recent change of logviewer:
https://github.com/apache/storm/commit/0201ae79a5f4fc7ba7ee1c524818a29a92f140f1#diff-81a8fa5a47d77bd6cca8d98d9916f172.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #2204: STORM-1280 port backtype.storm.daemon.logviewer to...
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2204#discussion_r128599677
--- Diff: storm-webapp/src/main/java/org/apache/storm/daemon/logviewer/webapp/LogviewerResource.java ---
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.daemon.logviewer.webapp;
+
+import com.codahale.metrics.Meter;
+
+import java.io.IOException;
+import java.net.URLDecoder;
+import java.util.Map;
+
+import javax.servlet.http.HttpServletRequest;
+import javax.ws.rs.GET;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.core.Context;
+import javax.ws.rs.core.Response;
+
+import org.apache.commons.lang.BooleanUtils;
+import org.apache.commons.lang.StringUtils;
+import org.apache.storm.daemon.common.JsonResponseBuilder;
+import org.apache.storm.daemon.logviewer.handler.LogviewerLogDownloadHandler;
+import org.apache.storm.daemon.logviewer.handler.LogviewerLogPageHandler;
+import org.apache.storm.daemon.logviewer.handler.LogviewerLogSearchHandler;
+import org.apache.storm.daemon.logviewer.handler.LogviewerProfileHandler;
+import org.apache.storm.metric.StormMetricsRegistry;
+import org.apache.storm.security.auth.IHttpCredentialsPlugin;
+import org.apache.storm.ui.InvalidRequestException;
+import org.apache.storm.ui.UIHelpers;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Handles HTTP requests for Logviewer.
+ */
+@Path("/")
+public class LogviewerResource {
+ private static final Logger LOG = LoggerFactory.getLogger(LogviewerResource.class);
+
+ private static final Meter meterLogPageHttpRequests = StormMetricsRegistry.registerMeter("logviewer:num-log-page-http-requests");
+ private static final Meter meterDaemonLogPageHttpRequests = StormMetricsRegistry.registerMeter(
+ "logviewer:num-daemonlog-page-http-requests");
+ private static final Meter meterDownloadLogFileHttpRequests = StormMetricsRegistry.registerMeter(
+ "logviewer:num-download-log-file-http-requests");
+ private static final Meter meterDownloadLogDaemonFileHttpRequests = StormMetricsRegistry.registerMeter(
+ "logviewer:num-download-log-daemon-file-http-requests");
+ private static final Meter meterListLogsHttpRequests = StormMetricsRegistry.registerMeter("logviewer:num-list-logs-http-requests");
+
+ private final LogviewerLogPageHandler logviewer;
+ private final LogviewerProfileHandler profileHandler;
+ private final LogviewerLogDownloadHandler logDownloadHandler;
+ private final LogviewerLogSearchHandler logSearchHandler;
+ private final IHttpCredentialsPlugin httpCredsHandler;
+
+ /**
+ * Constructor.
+ *
+ * @param logviewerParam {@link LogviewerLogPageHandler}
+ * @param profileHandler {@link LogviewerProfileHandler}
+ * @param logDownloadHandler {@link LogviewerLogDownloadHandler}
+ * @param logSearchHandler {@link LogviewerLogSearchHandler}
+ * @param httpCredsHandler {@link IHttpCredentialsPlugin}
+ */
+ public LogviewerResource(LogviewerLogPageHandler logviewerParam, LogviewerProfileHandler profileHandler,
+ LogviewerLogDownloadHandler logDownloadHandler, LogviewerLogSearchHandler logSearchHandler,
+ IHttpCredentialsPlugin httpCredsHandler) {
+ this.logviewer = logviewerParam;
+ this.profileHandler = profileHandler;
+ this.logDownloadHandler = logDownloadHandler;
+ this.logSearchHandler = logSearchHandler;
+ this.httpCredsHandler = httpCredsHandler;
+ }
+
+ /**
+ * Handles '/log' request.
+ */
+ @GET
+ @Path("/log")
+ public Response log(@Context HttpServletRequest request) throws IOException {
+ meterLogPageHttpRequests.mark();
+
+ try {
+ String user = httpCredsHandler.getUserName(request);
+ Integer start = request.getParameter("start") != null ? parseIntegerFromMap(request.getParameterMap(), "start") : null;
+ Integer length = request.getParameter("length") != null ? parseIntegerFromMap(request.getParameterMap(), "length") : null;
+ String decodedFileName = URLDecoder.decode(request.getParameter("file"));
+ String grep = request.getParameter("grep");
+ return logviewer.logPage(decodedFileName, start, length, grep, user);
+ } catch (InvalidRequestException e) {
+ LOG.error(e.getMessage(), e);
+ return Response.status(400).entity(e.getMessage()).build();
+ }
+ }
+
+ /**
+ * Handles '/daemonlog' request.
+ */
+ @GET
+ @Path("/daemonlog")
+ public Response daemonLog(@Context HttpServletRequest request) throws IOException {
+ meterDaemonLogPageHttpRequests.mark();
+
+ try {
+ String user = httpCredsHandler.getUserName(request);
+ Integer start = request.getParameter("start") != null ? parseIntegerFromMap(request.getParameterMap(), "start") : null;
+ Integer length = request.getParameter("length") != null ? parseIntegerFromMap(request.getParameterMap(), "length") : null;
+ String decodedFileName = URLDecoder.decode(request.getParameter("file"));
+ String grep = request.getParameter("grep");
+ return logviewer.daemonLogPage(decodedFileName, start, length, grep, user);
+ } catch (InvalidRequestException e) {
+ LOG.error(e.getMessage(), e);
+ return Response.status(400).entity(e.getMessage()).build();
+ }
+ }
+
+ /**
+ * Handles '/searchLogs' request.
+ */
+ @GET
+ @Path("/searchLogs")
+ public Response searchLogs(@Context HttpServletRequest request) throws IOException {
+ String user = httpCredsHandler.getUserName(request);
+ String topologyId = request.getParameter("topoId");
+ String portStr = request.getParameter("port");
+ String callback = request.getParameter("callback");
+ String origin = request.getHeader("Origin");
+
+ return logviewer.listLogFiles(user, portStr != null ? Integer.parseInt(portStr) : null, topologyId, callback, origin);
--- End diff --
OK Sounds good.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #2204: STORM-1280 port backtype.storm.daemon.logviewer to...
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2204#discussion_r128364464
--- Diff: storm-webapp/src/main/java/org/apache/storm/daemon/logviewer/LogviewerServer.java ---
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.daemon.logviewer;
+
+import static org.apache.storm.DaemonConfig.UI_HEADER_BUFFER_BYTES;
+
+import com.codahale.metrics.Meter;
+import com.google.common.annotations.VisibleForTesting;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.storm.DaemonConfig;
+import org.apache.storm.daemon.logviewer.utils.DirectoryCleaner;
+import org.apache.storm.daemon.logviewer.utils.LogCleaner;
+import org.apache.storm.daemon.logviewer.utils.WorkerLogs;
+import org.apache.storm.daemon.logviewer.webapp.LogviewerApplication;
+import org.apache.storm.metric.StormMetricsRegistry;
+import org.apache.storm.ui.FilterConfiguration;
+import org.apache.storm.ui.UIHelpers;
+import org.apache.storm.utils.ConfigUtils;
+import org.apache.storm.utils.ObjectReader;
+import org.apache.storm.utils.Utils;
+import org.eclipse.jetty.server.Server;
+import org.eclipse.jetty.servlet.DefaultServlet;
+import org.eclipse.jetty.servlet.ServletContextHandler;
+import org.eclipse.jetty.servlet.ServletHolder;
+import org.eclipse.jetty.util.resource.Resource;
+import org.glassfish.jersey.servlet.ServletContainer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The main entry of Logviewer.
+ */
+public class LogviewerServer implements AutoCloseable {
+ private static final Logger LOG = LoggerFactory.getLogger(LogviewerServer.class);
+ private static final Meter meterShutdownCalls = StormMetricsRegistry.registerMeter("drpc:num-shutdown-calls");
+ public static final String STATIC_RESOURCE_DIRECTORY_PATH = "./public";
+
+ private static Server mkHttpServer(Map<String, Object> conf) {
+ Integer logviewerHttpPort = (Integer) conf.get(DaemonConfig.LOGVIEWER_PORT);
+ Server ret = null;
+ if (logviewerHttpPort != null && logviewerHttpPort >= 0) {
+ LOG.info("Starting Logviewer HTTP servers...");
+ Integer headerBufferSize = ObjectReader.getInt(conf.get(UI_HEADER_BUFFER_BYTES));
+ String filterClass = (String) (conf.get(DaemonConfig.UI_FILTER));
+ @SuppressWarnings("unchecked")
+ Map<String, String> filterParams = (Map<String, String>) (conf.get(DaemonConfig.UI_FILTER_PARAMS));
+ FilterConfiguration filterConfiguration = new FilterConfiguration(filterClass, filterParams);
+ final List<FilterConfiguration> filterConfigurations = Arrays.asList(filterConfiguration);
+
+ final Integer httpsPort = ObjectReader.getInt(conf.get(DaemonConfig.LOGVIEWER_HTTPS_PORT), 0);
+ final String httpsKsPath = (String) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_KEYSTORE_PATH));
+ final String httpsKsPassword = (String) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_KEYSTORE_PASSWORD));
+ final String httpsKsType = (String) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_KEYSTORE_TYPE));
+ final String httpsKeyPassword = (String) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_KEY_PASSWORD));
+ final String httpsTsPath = (String) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_TRUSTSTORE_PATH));
+ final String httpsTsPassword = (String) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_TRUSTSTORE_PASSWORD));
+ final String httpsTsType = (String) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_TRUSTSTORE_TYPE));
+ final Boolean httpsWantClientAuth = (Boolean) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_WANT_CLIENT_AUTH));
+ final Boolean httpsNeedClientAuth = (Boolean) (conf.get(DaemonConfig.LOGVIEWER_HTTPS_NEED_CLIENT_AUTH));
+
+ //TODO a better way to do this would be great.
+ LogviewerApplication.setup(conf);
+ ret = UIHelpers.jettyCreateServer(logviewerHttpPort, null, httpsPort);
+
+ UIHelpers.configSsl(ret, httpsPort, httpsKsPath, httpsKsPassword, httpsKsType, httpsKeyPassword,
+ httpsTsPath, httpsTsPassword, httpsTsType, httpsNeedClientAuth, httpsWantClientAuth);
+
+ ServletContextHandler context = new ServletContextHandler(ServletContextHandler.NO_SESSIONS);
+ try {
+ context.setBaseResource(Resource.newResource(STATIC_RESOURCE_DIRECTORY_PATH));
+ } catch (IOException e) {
+ throw new RuntimeException("Can't locate static resource directory " + STATIC_RESOURCE_DIRECTORY_PATH);
+ }
+
+ context.setContextPath("/");
+ ret.setHandler(context);
+
+ ServletHolder holderPwd = new ServletHolder("default", DefaultServlet.class);
+ holderPwd.setInitOrder(1);
+ context.addServlet(holderPwd,"/");
+
+ ServletHolder jerseyServlet = context.addServlet(ServletContainer.class, "/api/v1/*");
+ jerseyServlet.setInitOrder(2);
+ jerseyServlet.setInitParameter("javax.ws.rs.Application", LogviewerApplication.class.getName());
+
+ UIHelpers.configFilters(context, filterConfigurations);
+ }
+ return ret;
+ }
+
+ private final Server httpServer;
+ private boolean closed = false;
+
+ /**
+ * Constructor.
+ * @param conf Logviewer conf for the servers
+ */
+ public LogviewerServer(Map<String, Object> conf) {
+ httpServer = mkHttpServer(conf);
+ }
+
+ @VisibleForTesting
+ void start() throws Exception {
+ LOG.info("Starting Logviewer...");
+ if (httpServer != null) {
+ httpServer.start();
+ }
+ }
+
+ @VisibleForTesting
+ void awaitTermination() throws InterruptedException {
+ httpServer.join();
+ }
+
+ @Override
+ public synchronized void close() {
+ if (!closed) {
+ //This is kind of useless...
+ meterShutdownCalls.mark();
+
+ //TODO this is causing issues...
+ //if (httpServer != null) {
+ // httpServer.destroy();
+ //}
+
+ closed = true;
+ }
+ }
+
+ /**
+ * @return The port the HTTP server is listening on. Not available until {@link #start() } has run.
+ */
+ public int getHttpServerPort() {
+ assert httpServer.getConnectors().length == 1;
--- End diff --
What about if we want to support https and http in the future? the reason I ask is because I do and I plan on pushing some changes back at some point soon.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #2204: STORM-1280 port backtype.storm.daemon.logviewer to...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/storm/pull/2204#discussion_r128402277
--- Diff: storm-webapp/src/main/java/org/apache/storm/daemon/logviewer/webapp/LogviewerResource.java ---
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.daemon.logviewer.webapp;
+
+import com.codahale.metrics.Meter;
+
+import java.io.IOException;
+import java.net.URLDecoder;
+import java.util.Map;
+
+import javax.servlet.http.HttpServletRequest;
+import javax.ws.rs.GET;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.core.Context;
+import javax.ws.rs.core.Response;
+
+import org.apache.commons.lang.BooleanUtils;
+import org.apache.commons.lang.StringUtils;
+import org.apache.storm.daemon.common.JsonResponseBuilder;
+import org.apache.storm.daemon.logviewer.handler.LogviewerLogDownloadHandler;
+import org.apache.storm.daemon.logviewer.handler.LogviewerLogPageHandler;
+import org.apache.storm.daemon.logviewer.handler.LogviewerLogSearchHandler;
+import org.apache.storm.daemon.logviewer.handler.LogviewerProfileHandler;
+import org.apache.storm.metric.StormMetricsRegistry;
+import org.apache.storm.security.auth.IHttpCredentialsPlugin;
+import org.apache.storm.ui.InvalidRequestException;
+import org.apache.storm.ui.UIHelpers;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Handles HTTP requests for Logviewer.
+ */
+@Path("/")
+public class LogviewerResource {
+ private static final Logger LOG = LoggerFactory.getLogger(LogviewerResource.class);
+
+ private static final Meter meterLogPageHttpRequests = StormMetricsRegistry.registerMeter("logviewer:num-log-page-http-requests");
+ private static final Meter meterDaemonLogPageHttpRequests = StormMetricsRegistry.registerMeter(
+ "logviewer:num-daemonlog-page-http-requests");
+ private static final Meter meterDownloadLogFileHttpRequests = StormMetricsRegistry.registerMeter(
+ "logviewer:num-download-log-file-http-requests");
+ private static final Meter meterDownloadLogDaemonFileHttpRequests = StormMetricsRegistry.registerMeter(
+ "logviewer:num-download-log-daemon-file-http-requests");
+ private static final Meter meterListLogsHttpRequests = StormMetricsRegistry.registerMeter("logviewer:num-list-logs-http-requests");
+
+ private final LogviewerLogPageHandler logviewer;
+ private final LogviewerProfileHandler profileHandler;
+ private final LogviewerLogDownloadHandler logDownloadHandler;
+ private final LogviewerLogSearchHandler logSearchHandler;
+ private final IHttpCredentialsPlugin httpCredsHandler;
+
+ /**
+ * Constructor.
+ *
+ * @param logviewerParam {@link LogviewerLogPageHandler}
+ * @param profileHandler {@link LogviewerProfileHandler}
+ * @param logDownloadHandler {@link LogviewerLogDownloadHandler}
+ * @param logSearchHandler {@link LogviewerLogSearchHandler}
+ * @param httpCredsHandler {@link IHttpCredentialsPlugin}
+ */
+ public LogviewerResource(LogviewerLogPageHandler logviewerParam, LogviewerProfileHandler profileHandler,
+ LogviewerLogDownloadHandler logDownloadHandler, LogviewerLogSearchHandler logSearchHandler,
+ IHttpCredentialsPlugin httpCredsHandler) {
+ this.logviewer = logviewerParam;
+ this.profileHandler = profileHandler;
+ this.logDownloadHandler = logDownloadHandler;
+ this.logSearchHandler = logSearchHandler;
+ this.httpCredsHandler = httpCredsHandler;
+ }
+
+ /**
+ * Handles '/log' request.
+ */
+ @GET
+ @Path("/log")
+ public Response log(@Context HttpServletRequest request) throws IOException {
+ meterLogPageHttpRequests.mark();
+
+ try {
+ String user = httpCredsHandler.getUserName(request);
+ Integer start = request.getParameter("start") != null ? parseIntegerFromMap(request.getParameterMap(), "start") : null;
+ Integer length = request.getParameter("length") != null ? parseIntegerFromMap(request.getParameterMap(), "length") : null;
+ String decodedFileName = URLDecoder.decode(request.getParameter("file"));
+ String grep = request.getParameter("grep");
+ return logviewer.logPage(decodedFileName, start, length, grep, user);
+ } catch (InvalidRequestException e) {
+ LOG.error(e.getMessage(), e);
+ return Response.status(400).entity(e.getMessage()).build();
+ }
+ }
+
+ /**
+ * Handles '/daemonlog' request.
+ */
+ @GET
+ @Path("/daemonlog")
+ public Response daemonLog(@Context HttpServletRequest request) throws IOException {
+ meterDaemonLogPageHttpRequests.mark();
+
+ try {
+ String user = httpCredsHandler.getUserName(request);
+ Integer start = request.getParameter("start") != null ? parseIntegerFromMap(request.getParameterMap(), "start") : null;
+ Integer length = request.getParameter("length") != null ? parseIntegerFromMap(request.getParameterMap(), "length") : null;
+ String decodedFileName = URLDecoder.decode(request.getParameter("file"));
+ String grep = request.getParameter("grep");
+ return logviewer.daemonLogPage(decodedFileName, start, length, grep, user);
+ } catch (InvalidRequestException e) {
+ LOG.error(e.getMessage(), e);
+ return Response.status(400).entity(e.getMessage()).build();
+ }
+ }
+
+ /**
+ * Handles '/searchLogs' request.
+ */
+ @GET
+ @Path("/searchLogs")
+ public Response searchLogs(@Context HttpServletRequest request) throws IOException {
+ String user = httpCredsHandler.getUserName(request);
+ String topologyId = request.getParameter("topoId");
+ String portStr = request.getParameter("port");
+ String callback = request.getParameter("callback");
+ String origin = request.getHeader("Origin");
+
+ return logviewer.listLogFiles(user, portStr != null ? Integer.parseInt(portStr) : null, topologyId, callback, origin);
+ }
+
+ /**
+ * Handles '/listLogs' request.
+ */
+ @GET
+ @Path("/listLogs")
+ public Response listLogs(@Context HttpServletRequest request) throws IOException {
+ meterListLogsHttpRequests.mark();
+
+ String user = httpCredsHandler.getUserName(request);
+ String topologyId = request.getParameter("topoId");
+ String portStr = request.getParameter("port");
+ String callback = request.getParameter("callback");
+ String origin = request.getHeader("Origin");
+
+ return logviewer.listLogFiles(user, portStr != null ? Integer.parseInt(portStr) : null, topologyId, callback, origin);
+ }
+
+ /**
+ * Handles '/dumps' (listing dump files) request.
+ */
+ @GET
+ @Path("/dumps/{topo-id}/{host-port}")
+ public Response listDumpFiles(@PathParam("topo-id") String topologyId, @PathParam("host-port") String hostPort,
+ @Context HttpServletRequest request) throws IOException {
+ String user = httpCredsHandler.getUserName(request);
+ return profileHandler.listDumpFiles(topologyId, hostPort, user);
+ }
+
+ /**
+ * Handles '/dumps' (downloading specific dump file) request.
+ */
+ @GET
+ @Path("/dumps/{topo-id}/{host-port}/{filename}")
+ public Response downloadDumpFile(@PathParam("topo-id") String topologyId, @PathParam("host-port") String hostPort,
+ @PathParam("filename") String fileName, @Context HttpServletRequest request) throws IOException {
+ String user = httpCredsHandler.getUserName(request);
+ return profileHandler.downloadDumpFile(topologyId, hostPort, fileName, user);
+ }
+
+ /**
+ * Handles '/download' (downloading specific log file) request.
+ */
+ @GET
+ @Path("/download")
+ public Response downloadLogFile(@Context HttpServletRequest request) throws IOException {
+ meterDownloadLogFileHttpRequests.mark();
+
+ String user = httpCredsHandler.getUserName(request);
+ String file = request.getParameter("file");
+ String decodedFileName = URLDecoder.decode(file);
+ return logDownloadHandler.downloadLogFile(decodedFileName, user);
+ }
+
+ /**
+ * Handles '/daemondownload' (downloading specific daemon log file) request.
+ */
+ @GET
+ @Path("/daemondownload")
+ public Response downloadDaemonLogFile(@Context HttpServletRequest request) throws IOException {
+ meterDownloadLogDaemonFileHttpRequests.mark();
+
+ String user = httpCredsHandler.getUserName(request);
+ String file = request.getParameter("file");
+ String decodedFileName = URLDecoder.decode(file);
+ return logDownloadHandler.downloadDaemonLogFile(decodedFileName, user);
--- End diff --
Same as above.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---