You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by Rohini Palaniswamy <ro...@gmail.com> on 2014/04/22 00:04:16 UTC
Re: Review Request 19520: OOZIE-1737 : Oozie log streaming is slow
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19520/#review38818
-----------------------------------------------------------
core/src/main/java/org/apache/oozie/CoordinatorEngine.java
<https://reviews.apache.org/r/19520/#comment71146>
You can have it as true by default and set to false only when no scope is specified and in bundle
client/src/main/java/org/apache/oozie/cli/OozieCLI.java
<https://reviews.apache.org/r/19520/#comment71145>
job log filtering
Command line help should be more informative. List out the options of the -logfilter and usage. Refer to FILTER_OPTION in OozieCLI
core/src/main/java/org/apache/oozie/CoordinatorEngine.java
<https://reviews.apache.org/r/19520/#comment74152>
XLogFilter filter = new XLogFilter(new XLogUserFilterParam(params));
Other places too
core/src/main/java/org/apache/oozie/DagEngine.java
<https://reviews.apache.org/r/19520/#comment71314>
Remove this
core/src/main/java/org/apache/oozie/DagEngine.java
<https://reviews.apache.org/r/19520/#comment74148>
Date lastTime = job.getEndTime();
core/src/main/java/org/apache/oozie/service/XLogService.java
<https://reviews.apache.org/r/19520/#comment74155>
Can we rename the class to XLogFilter to have consistent naming with other XLog classes?
core/src/main/java/org/apache/oozie/util/LogFilter.java
<https://reviews.apache.org/r/19520/#comment74165>
formatting
core/src/main/java/org/apache/oozie/util/LogFilter.java
<https://reviews.apache.org/r/19520/#comment74164>
formatting.
Add import instead of full package name
core/src/main/java/org/apache/oozie/util/LogFilter.java
<https://reviews.apache.org/r/19520/#comment74167>
isDebugMode()
core/src/main/java/org/apache/oozie/util/LogFilter.java
<https://reviews.apache.org/r/19520/#comment74177>
getDebugMessage()
core/src/main/java/org/apache/oozie/util/LogFilter.java
<https://reviews.apache.org/r/19520/#comment74159>
Use : or = instead of \ as the separator
core/src/main/java/org/apache/oozie/util/LogFilter.java
<https://reviews.apache.org/r/19520/#comment74168>
\n should be fine
core/src/main/java/org/apache/oozie/util/LogFilter.java
<https://reviews.apache.org/r/19520/#comment74160>
Follow standard getter/setter convention
core/src/main/java/org/apache/oozie/util/LogFilter.java
<https://reviews.apache.org/r/19520/#comment74162>
Camel case.
calculateScanStartEndDate
core/src/main/java/org/apache/oozie/util/LogFilter.java
<https://reviews.apache.org/r/19520/#comment74209>
Simplify and make the logic clean. Avoid handling parse exceptions here.
core/src/main/java/org/apache/oozie/util/LogFilter.java
<https://reviews.apache.org/r/19520/#comment74181>
Camelcase
core/src/main/java/org/apache/oozie/util/LogFilter.java
<https://reviews.apache.org/r/19520/#comment74184>
less than
core/src/main/java/org/apache/oozie/util/LogFilter.java
<https://reviews.apache.org/r/19520/#comment74185>
actionLogDuration
core/src/main/java/org/apache/oozie/util/LogFilter.java
<https://reviews.apache.org/r/19520/#comment74212>
Please reduce the scan time range.
core/src/main/java/org/apache/oozie/util/LogFilter.java
<https://reviews.apache.org/r/19520/#comment74192>
returned
core/src/main/java/org/apache/oozie/util/LogFilter.java
<https://reviews.apache.org/r/19520/#comment74215>
simplify and clean up code without ParseException
core/src/main/java/org/apache/oozie/util/TimestampedMessageParser.java
<https://reviews.apache.org/r/19520/#comment74208>
formatting
core/src/main/java/org/apache/oozie/util/UserLogFilter.java
<https://reviews.apache.org/r/19520/#comment74195>
RECENT_OFFSET
core/src/main/java/org/apache/oozie/util/UserLogFilter.java
<https://reviews.apache.org/r/19520/#comment74198>
dateFormat
core/src/main/java/org/apache/oozie/util/UserLogFilter.java
<https://reviews.apache.org/r/19520/#comment74201>
private static final LOG_LEVELS
core/src/main/java/org/apache/oozie/util/UserLogFilter.java
<https://reviews.apache.org/r/19520/#comment74204>
private
have local variables for
startDate, endDate, startOffset, endOffset, recentOffset and getters for them.
core/src/main/java/org/apache/oozie/util/UserLogFilter.java
<https://reviews.apache.org/r/19520/#comment74193>
populate
core/src/main/java/org/apache/oozie/util/UserLogFilter.java
<https://reviews.apache.org/r/19520/#comment74194>
getRecentOffset
This is being already validated. So assign it to a local variable and return integer offset in minutes
core/src/main/java/org/apache/oozie/util/UserLogFilter.java
<https://reviews.apache.org/r/19520/#comment74196>
getSearchPattern
core/src/main/java/org/apache/oozie/util/UserLogFilter.java
<https://reviews.apache.org/r/19520/#comment74202>
Invalid start time : " + getStartDate() , e
core/src/main/java/org/apache/oozie/util/UserLogFilter.java
<https://reviews.apache.org/r/19520/#comment74203>
Invalid end time : " + getEndDate() , e
core/src/main/java/org/apache/oozie/util/UserLogFilter.java
<https://reviews.apache.org/r/19520/#comment74206>
Start time is + getStartDate() + ", End time is " + getEndDate())
core/src/main/java/org/apache/oozie/util/UserLogFilter.java
<https://reviews.apache.org/r/19520/#comment74200>
Supported log levels are
core/src/main/java/org/apache/oozie/util/XLogStreamer.java
<https://reviews.apache.org/r/19520/#comment71318>
Move this to streamLog() method and keep the getReader and makeReader methods as is
core/src/main/resources/oozie-default.xml
<https://reviews.apache.org/r/19520/#comment71320>
Max log scan duration in hours
then exception is thrown to user to reduce the scan duration.
-1 indicates
core/src/main/resources/oozie-default.xml
<https://reviews.apache.org/r/19520/#comment74171>
Max log scan duration in hours for coordinator job when list of actions are specified.
then exception is thrown to user to reduce the scan duration.
-1 indicates
core/src/main/resources/oozie-default.xml
<https://reviews.apache.org/r/19520/#comment71325>
This setting is separate from max.log.scan.duration as we want to allow higher durations when actions are specified.
docs/src/site/twiki/DG_CommandLineTool.twiki
<https://reviews.apache.org/r/19520/#comment74172>
Remove purushah in all places and just start with $
webapp/src/main/webapp/oozie-console.js
<https://reviews.apache.org/r/19520/#comment74174>
Please add text here to refer to the documentation link in same server on how to specify search filter.
- Rohini Palaniswamy
On March 25, 2014, 9:23 p.m., Purshotam Shah wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/19520/
> -----------------------------------------------------------
>
> (Updated March 25, 2014, 9:23 p.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-1737
> https://issues.apache.org/jira/browse/OOZIE-1737
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> User can provide multiple option as -param <>;<>;<> to make streaming faster and precise.
> Option can be.
> * Recent: Define offset of recent hours/Min of log to stream. Server will search user query only in recent log duration.
> * Start: Start time of log parsing. Oozie server will start parsing logs from start date. User can provide a valid date or
> offset.
> * End: Server will streaming logs till specified end date. User can provide a valid date or offset.
> * Loglevel : Multiple log levels separated by "|" can be specified.
> * Text: String to search in logs.
> * Limit : Limit number of line to be searched. Log search will end when when n lines(excluding exception) of patten has
> matched.
> * Debug : Will print log start and end time. This will be useful, if user are not sure what his/her offset will be
> calculated to.
>
>
> In addition to this
> 1. System setting to control log streaming duration.
> 2. Added a option to show log streaming status.
> A label on UI will show if log streaming is in progress, completed or erred.
> 3. Many user has complied that get button doesn't look like button. Changes the look of "get log" button.
> 4. End the execution when data is date is reached.
> earlier started date and end date is only used to load logs files.
> ......
>
>
> Diffs
> -----
>
> client/src/main/java/org/apache/oozie/cli/OozieCLI.java 3c780dd
> client/src/main/java/org/apache/oozie/client/OozieClient.java b0a85fd
> client/src/main/java/org/apache/oozie/client/rest/RestConstants.java 0466ffe
> core/src/main/java/org/apache/oozie/BundleEngine.java ce7c9c4
> core/src/main/java/org/apache/oozie/BundleJobBean.java 2c06035
> core/src/main/java/org/apache/oozie/CoordinatorEngine.java 315a330
> core/src/main/java/org/apache/oozie/DagEngine.java 300d6eb
> core/src/main/java/org/apache/oozie/service/DagXLogInfoService.java 1b18140
> core/src/main/java/org/apache/oozie/service/XLogService.java f9383d0
> core/src/main/java/org/apache/oozie/service/XLogStreamingService.java f77794e
> core/src/main/java/org/apache/oozie/service/ZKXLogStreamingService.java 8dc8b4b
> core/src/main/java/org/apache/oozie/servlet/V1JobServlet.java 2ef351e
> core/src/main/java/org/apache/oozie/util/LogFilter.java e69de29
> core/src/main/java/org/apache/oozie/util/LogUserParam.java e69de29
> core/src/main/java/org/apache/oozie/util/SimpleTimestampedMessageParser.java 0a5e8d1
> core/src/main/java/org/apache/oozie/util/TimestampedMessageParser.java 8a11780
> core/src/main/java/org/apache/oozie/util/UserLogFilter.java e69de29
> core/src/main/java/org/apache/oozie/util/XLogStreamer.java bc6b009
> core/src/main/resources/oozie-default.xml 34362aa
> core/src/test/java/org/apache/oozie/TestCoordinatorEngineStreamLog.java b4f161a
> core/src/test/java/org/apache/oozie/service/TestLogStreamingParam.java e69de29
> core/src/test/java/org/apache/oozie/service/TestUserLogFilter.java e69de29
> core/src/test/java/org/apache/oozie/service/TestXLogStreamingService.java c03385d
> core/src/test/java/org/apache/oozie/service/TestZKXLogStreamingService.java 29bca41
> core/src/test/java/org/apache/oozie/util/TestLogStreamer.java 659949f
> core/src/test/java/org/apache/oozie/util/TestSimplifiedTimestampedMessageParser.java 75341aa
> core/src/test/java/org/apache/oozie/util/TestTimestampedMessageParser.java c2630e7
> core/src/test/java/org/apache/oozie/util/TestXLogFilter.java 8d0fd71
> core/src/test/resources/userLogFilterTestlog.log e69de29
> core/src/test/resources/userParamTestlog.log e69de29
> docs/src/site/twiki/DG_CommandLineTool.twiki 0748ff8
> docs/src/site/twiki/WebServicesAPI.twiki 4b20075
> webapp/src/main/webapp/oozie-console.js 0b6cedf
>
> Diff: https://reviews.apache.org/r/19520/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Purshotam Shah
>
>