You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by joemeszaros <gi...@git.apache.org> on 2015/09/22 15:18:50 UTC
[GitHub] nifi pull request: NIFI-985: Custom log prefix for LogAttribute pr...
GitHub user joemeszaros opened a pull request:
https://github.com/apache/nifi/pull/90
NIFI-985: Custom log prefix for LogAttribute processor
When you have a data flow with a bunch of LogAttribute processors it is really hard to identify the right LogAttribute processor in the log file and distinguish the output of different LogAttribute processors.
There is a new property, called log prefix, which helps the user to set a custom log prefix for each Log Attribute processor, which will appear in the log output of the processor. Log prefix appears in the first and the last log line, followed by the original 50 dashes. If you configure log prefix 'STEP 1: ' the log output looks like this:
```
STEP 1 : --------------------------------------------------
Standard FlowFile Attributes
Key: 'entryDate'
Value: 'Tue Sep 22 15:13:02 CEST 2015'
Key: 'lineageStartDate'
Value: 'Tue Sep 22 15:13:02 CEST 2015'
Key: 'fileSize'
Value: '9'
FlowFile Attribute Map Content
Key: 'customAttribute'
Value: 'custom value'
STEP 1 : --------------------------------------------------
flow file content...
```
The following screen shot shows the configuration and the output of the processor:
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ImpressTV/nifi NIFI-985
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/90.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #90
----
commit 90741868a8b7d42f5949fd741e2276c024a5866a
Author: Joe <jo...@impresstv.com>
Date: 2015-09-22T13:04:37Z
NIFI-985: Custom log prefix for LogAttribute processor
Log prefix helps to distinguish the log output of multiple LogAttribute processors and identify the right processor. Log prefix appears in the first and the last log line, followed by the original 50 dashes. If you configure log prefix 'STEP 1: ' the log output looks like this:
STEP 1 : --------------------------------------------------
Standard FlowFile Attributes
Key: 'entryDate'
Value: 'Tue Sep 22 15:13:02 CEST 2015'
Key: 'lineageStartDate'
Value: 'Tue Sep 22 15:13:02 CEST 2015'
Key: 'fileSize'
Value: '9'
FlowFile Attribute Map Content
Key: 'customAttribute'
Value: 'custom value'
STEP 1 : --------------------------------------------------
flow file content...
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-985: Custom log prefix for LogAttribute pr...
Posted by apiri <gi...@git.apache.org>.
Github user apiri commented on a diff in the pull request:
https://github.com/apache/nifi/pull/90#discussion_r40323765
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/LogAttribute.java ---
@@ -123,12 +133,26 @@ protected void init(final ProcessorInitializationContext context) {
protected String processFlowFile(final ProcessorLog logger, final DebugLevels logLevel, final FlowFile flowFile, final ProcessSession session, final ProcessContext context) {
final Set<String> attributeKeys = getAttributesToLog(flowFile.getAttributes().keySet(), context);
final ProcessorLog LOG = getLogger();
+ final String dashedLine;
+
+ String logPrefix = context.getProperty(LOG_PREFIX).evaluateAttributeExpressions().getValue();
--- End diff --
Indeed! :) Thanks for fixing that up. Everything looks good to go from this side and I shall merge it in shortly. Thanks again for your contribution and efforts!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-985: Custom log prefix for LogAttribute pr...
Posted by joemeszaros <gi...@git.apache.org>.
Github user joemeszaros commented on a diff in the pull request:
https://github.com/apache/nifi/pull/90#discussion_r40297509
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/LogAttribute.java ---
@@ -123,12 +133,26 @@ protected void init(final ProcessorInitializationContext context) {
protected String processFlowFile(final ProcessorLog logger, final DebugLevels logLevel, final FlowFile flowFile, final ProcessSession session, final ProcessContext context) {
final Set<String> attributeKeys = getAttributesToLog(flowFile.getAttributes().keySet(), context);
final ProcessorLog LOG = getLogger();
+ final String dashedLine;
+
+ String logPrefix = context.getProperty(LOG_PREFIX).evaluateAttributeExpressions().getValue();
--- End diff --
it was a really quick fix :-)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-985: Custom log prefix for LogAttribute pr...
Posted by apiri <gi...@git.apache.org>.
Github user apiri commented on a diff in the pull request:
https://github.com/apache/nifi/pull/90#discussion_r40088113
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/LogAttribute.java ---
@@ -123,11 +131,15 @@ protected void init(final ProcessorInitializationContext context) {
protected String processFlowFile(final ProcessorLog logger, final DebugLevels logLevel, final FlowFile flowFile, final ProcessSession session, final ProcessContext context) {
final Set<String> attributeKeys = getAttributesToLog(flowFile.getAttributes().keySet(), context);
final ProcessorLog LOG = getLogger();
-
+ String logPrefix = context.getProperty(LOG_PREFIX).getValue();
// Pretty print metadata
final StringBuilder message = new StringBuilder();
message.append("logging for flow file ").append(flowFile);
message.append("\n");
+ if (logPrefix != null) {
+ message.append(logPrefix);
+ message.append(" ");
+ }
message.append(FIFTY_DASHES);
--- End diff --
Since we are changing the output format would like to see a String.format or something similar to give a consistent width with number of dashes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-985: Custom log prefix for LogAttribute pr...
Posted by apiri <gi...@git.apache.org>.
Github user apiri commented on a diff in the pull request:
https://github.com/apache/nifi/pull/90#discussion_r40137346
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/LogAttribute.java ---
@@ -123,12 +134,26 @@ protected void init(final ProcessorInitializationContext context) {
protected String processFlowFile(final ProcessorLog logger, final DebugLevels logLevel, final FlowFile flowFile, final ProcessSession session, final ProcessContext context) {
final Set<String> attributeKeys = getAttributesToLog(flowFile.getAttributes().keySet(), context);
final ProcessorLog LOG = getLogger();
+ final String dashedLine;
+
+ String logPrefix = context.getProperty(LOG_PREFIX).getValue();
--- End diff --
An explicit call to evaluateAttributeExpressions is needed so that EL values are evaluated
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-985: Custom log prefix for LogAttribute pr...
Posted by apiri <gi...@git.apache.org>.
Github user apiri commented on a diff in the pull request:
https://github.com/apache/nifi/pull/90#discussion_r40258768
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/LogAttribute.java ---
@@ -123,12 +133,26 @@ protected void init(final ProcessorInitializationContext context) {
protected String processFlowFile(final ProcessorLog logger, final DebugLevels logLevel, final FlowFile flowFile, final ProcessSession session, final ProcessContext context) {
final Set<String> attributeKeys = getAttributesToLog(flowFile.getAttributes().keySet(), context);
final ProcessorLog LOG = getLogger();
+ final String dashedLine;
+
+ String logPrefix = context.getProperty(LOG_PREFIX).evaluateAttributeExpressions().getValue();
--- End diff --
My apologies, I led you slightly astray in method mentioned before. The evaluateAttributeExpressions seems like it would benefit from making use of the current flowfile, #evaluateAttributeExpressions(FlowFile flowfile). Otherwise, everything looks great. If you would like me to make this change on merge, let me know. Otherwise, that would be the last adjustment needed to call this complete.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-985: Custom log prefix for LogAttribute pr...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/nifi/pull/90
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-985: Custom log prefix for LogAttribute pr...
Posted by apiri <gi...@git.apache.org>.
Github user apiri commented on a diff in the pull request:
https://github.com/apache/nifi/pull/90#discussion_r40137272
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/LogAttribute.java ---
@@ -79,6 +81,14 @@
.allowableValues("true", "false")
.build();
+ public static final PropertyDescriptor LOG_PREFIX = new PropertyDescriptor.Builder()
+ .name("Log prefix")
+ .required(false)
+ .description("Log prefix appended to the log lines. It helps to distinguish the output of multiple LogAttribute processors.")
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(true)
--- End diff --
This is good to note that the property supports EL (this is the indicator via the UI)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-985: Custom log prefix for LogAttribute pr...
Posted by apiri <gi...@git.apache.org>.
Github user apiri commented on a diff in the pull request:
https://github.com/apache/nifi/pull/90#discussion_r40137422
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/LogAttribute.java ---
@@ -45,6 +45,8 @@
import org.apache.commons.io.IOUtils;
import org.apache.commons.lang3.StringUtils;
+import org.bouncycastle.util.Strings;
--- End diff --
This is unused and is causing errors in the checkstyle. (Checkstyle can be evaluated via Maven Profile contrib-check, e.g. mvn <goals> -P contrib-check)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-985: Custom log prefix for LogAttribute pr...
Posted by joemeszaros <gi...@git.apache.org>.
Github user joemeszaros commented on a diff in the pull request:
https://github.com/apache/nifi/pull/90#discussion_r40128133
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/LogAttribute.java ---
@@ -79,6 +79,13 @@
.allowableValues("true", "false")
.build();
+ public static final PropertyDescriptor LOG_PREFIX = new PropertyDescriptor.Builder()
--- End diff --
committed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-985: Custom log prefix for LogAttribute pr...
Posted by joemeszaros <gi...@git.apache.org>.
Github user joemeszaros commented on a diff in the pull request:
https://github.com/apache/nifi/pull/90#discussion_r40088634
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/LogAttribute.java ---
@@ -79,6 +79,13 @@
.allowableValues("true", "false")
.build();
+ public static final PropertyDescriptor LOG_PREFIX = new PropertyDescriptor.Builder()
--- End diff --
agreed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-985: Custom log prefix for LogAttribute pr...
Posted by apiri <gi...@git.apache.org>.
Github user apiri commented on a diff in the pull request:
https://github.com/apache/nifi/pull/90#discussion_r40087962
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/LogAttribute.java ---
@@ -79,6 +79,13 @@
.allowableValues("true", "false")
.build();
+ public static final PropertyDescriptor LOG_PREFIX = new PropertyDescriptor.Builder()
--- End diff --
Would be good to have this property support NiFi's Expression Language.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---