You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by ottobackwards <gi...@git.apache.org> on 2018/06/20 18:32:02 UTC
[GitHub] nifi pull request #2805: NIFI-5325 A Syslog Parser that fully supports RFC 5...
GitHub user ottobackwards opened a pull request:
https://github.com/apache/nifi/pull/2805
NIFI-5325 A Syslog Parser that fully supports RFC 5424 Structured Data
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [x] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
- [x] Is your initial contribution a single, squashed commit?
### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
- [x] Have you written or updated unit tests to verify your changes?
- [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [-] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
- [x] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
- [x] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?
### For documentation related changes:
- [-] Have you ensured that format looks appropriate for the output in which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ottobackwards/nifi syslog5424
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/2805.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2805
----
commit 145582a9d3d7b6a00d14039bbb9e1f8d1a9ba5a1
Author: Otto Fowler <ot...@...>
Date: 2018-06-19T22:22:26Z
NIFI-5325 A Syslog Parser that fully supports RFC 5424 Structured Data
----
---
[GitHub] nifi pull request #2805: NIFI-5325 A Syslog Parser that fully supports RFC 5...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2805#discussion_r197144105
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ParseSyslog5424.java ---
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.standard.syslog.StrictSyslog5424Parser;
+import org.apache.nifi.processors.standard.syslog.Syslog5424Event;
+import org.apache.nifi.stream.io.StreamUtils;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"logs", "syslog", "syslog5424", "attributes", "system", "event", "message"})
+@CapabilityDescription("Attempts to parse the contents of a well formed Syslog message in accordance to RFC5424 " +
+ "format and adds attributes to the FlowFile for each of the parts of the Syslog message, including Structured Data." +
+ "Structured Data will be written to attributes as on attribute per item id + parameter "+
+ "see https://tools.ietf.org/html/rfc5424." +
+ "Note: ParseSyslog5424 follows the specification more closely than ParseSyslog. If your Syslog producer " +
+ "does not follow the spec closely, with regards to using '-' for missing header entries for example, those logs " +
+ "will fail with this parser, where they would not fail with ParseSyslog.")
+@WritesAttributes({@WritesAttribute(attribute = "syslog.priority", description = "The priority of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.severity", description = "The severity of the Syslog message derived from the priority."),
+ @WritesAttribute(attribute = "syslog.facility", description = "The facility of the Syslog message derived from the priority."),
+ @WritesAttribute(attribute = "syslog.version", description = "The optional version from the Syslog message."),
+ @WritesAttribute(attribute = "syslog.timestamp", description = "The timestamp of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.hostname", description = "The hostname or IP address of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.appname", description = "The appname of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.procid", description = "The procid of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.messageid", description = "The messageid the Syslog message."),
+ @WritesAttribute(attribute = "syslog.structuredData", description = "Multiple entries per structuredData of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.sender", description = "The hostname of the Syslog server that sent the message."),
+ @WritesAttribute(attribute = "syslog.body", description = "The body of the Syslog message, everything after the hostname.")})
--- End diff --
Ok, I'll add that.
---
[GitHub] nifi pull request #2805: NIFI-5325 A Syslog Parser that fully supports RFC 5...
Posted by bbende <gi...@git.apache.org>.
Github user bbende commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2805#discussion_r196908047
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ParseSyslog5424.java ---
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.standard.syslog.StrictSyslog5424Parser;
+import org.apache.nifi.processors.standard.syslog.Syslog5424Event;
+import org.apache.nifi.stream.io.StreamUtils;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"logs", "syslog", "syslog5424", "attributes", "system", "event", "message"})
+@CapabilityDescription("Attempts to parse the contents of a well formed Syslog message in accordance to RFC5424 " +
+ "format and adds attributes to the FlowFile for each of the parts of the Syslog message, including Structured Data." +
+ "Structured Data will be written to attributes as on attribute per item id + parameter "+
+ "see https://tools.ietf.org/html/rfc5424." +
+ "Note: ParseSyslog5424 follows the specification more closely than ParseSyslog. If your Syslog producer " +
+ "does not follow the spec closely, with regards to using '-' for missing header entries for example, those logs " +
+ "will fail with this parser, where they would not fail with ParseSyslog.")
+@WritesAttributes({@WritesAttribute(attribute = "syslog.priority", description = "The priority of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.severity", description = "The severity of the Syslog message derived from the priority."),
+ @WritesAttribute(attribute = "syslog.facility", description = "The facility of the Syslog message derived from the priority."),
+ @WritesAttribute(attribute = "syslog.version", description = "The optional version from the Syslog message."),
+ @WritesAttribute(attribute = "syslog.timestamp", description = "The timestamp of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.hostname", description = "The hostname or IP address of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.appname", description = "The appname of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.procid", description = "The procid of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.messageid", description = "The messageid the Syslog message."),
+ @WritesAttribute(attribute = "syslog.structuredData", description = "Multiple entries per structuredData of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.sender", description = "The hostname of the Syslog server that sent the message."),
+ @WritesAttribute(attribute = "syslog.body", description = "The body of the Syslog message, everything after the hostname.")})
+@SeeAlso({ListenSyslog.class, ParseSyslog.class, PutSyslog.class})
+public class ParseSyslog5424 extends AbstractProcessor {
+
+ public static final AllowableValue OMIT = new AllowableValue(NilPolicy.OMIT.name(),NilPolicy.OMIT.name(),"The missing field will not have an attribute added.");
+ public static final AllowableValue NULL = new AllowableValue(NilPolicy.NULL.name(),NilPolicy.NULL.name(),"The missing field will have an empty attribute added.");
+ public static final AllowableValue DASH = new AllowableValue(NilPolicy.DASH.name(),NilPolicy.DASH.name(),"The missing field will have an attribute added with the value of '-'.");
+
+ public static final PropertyDescriptor CHARSET = new PropertyDescriptor.Builder()
+ .name("Character Set")
+ .description("Specifies which character set of the Syslog messages")
+ .required(true)
+ .defaultValue("UTF-8")
+ .addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor NIL_POLICY = new PropertyDescriptor.Builder()
+ .name("nil_policy")
+ .displayName("NIL Policy")
+ .description("Defines how NIL values are handled for header fields.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .allowableValues(OMIT,NULL,DASH)
+ .required(true)
+ .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+ .defaultValue(NULL.getValue())
+ .build();
+
+ static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Any FlowFile that could not be parsed as a Syslog message will be transferred to this Relationship without any attributes being added")
+ .build();
+ static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Any FlowFile that is successfully parsed as a Syslog message will be to this Relationship.")
+ .build();
+
+ private StrictSyslog5424Parser parser;
--- End diff --
Since this gets initialized in onTrigger which can be called by multiple different threads, it is generally a good idea to mark it as volatile
---
[GitHub] nifi issue #2805: NIFI-5325 A Syslog Parser that fully supports RFC 5424 Str...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/nifi/pull/2805
@bbende thanks for the review
---
[GitHub] nifi pull request #2805: NIFI-5325 A Syslog Parser that fully supports RFC 5...
Posted by bbende <gi...@git.apache.org>.
Github user bbende commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2805#discussion_r196907444
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ParseSyslog5424.java ---
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.standard.syslog.StrictSyslog5424Parser;
+import org.apache.nifi.processors.standard.syslog.Syslog5424Event;
+import org.apache.nifi.stream.io.StreamUtils;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"logs", "syslog", "syslog5424", "attributes", "system", "event", "message"})
+@CapabilityDescription("Attempts to parse the contents of a well formed Syslog message in accordance to RFC5424 " +
+ "format and adds attributes to the FlowFile for each of the parts of the Syslog message, including Structured Data." +
+ "Structured Data will be written to attributes as on attribute per item id + parameter "+
+ "see https://tools.ietf.org/html/rfc5424." +
+ "Note: ParseSyslog5424 follows the specification more closely than ParseSyslog. If your Syslog producer " +
+ "does not follow the spec closely, with regards to using '-' for missing header entries for example, those logs " +
+ "will fail with this parser, where they would not fail with ParseSyslog.")
+@WritesAttributes({@WritesAttribute(attribute = "syslog.priority", description = "The priority of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.severity", description = "The severity of the Syslog message derived from the priority."),
+ @WritesAttribute(attribute = "syslog.facility", description = "The facility of the Syslog message derived from the priority."),
+ @WritesAttribute(attribute = "syslog.version", description = "The optional version from the Syslog message."),
+ @WritesAttribute(attribute = "syslog.timestamp", description = "The timestamp of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.hostname", description = "The hostname or IP address of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.appname", description = "The appname of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.procid", description = "The procid of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.messageid", description = "The messageid the Syslog message."),
+ @WritesAttribute(attribute = "syslog.structuredData", description = "Multiple entries per structuredData of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.sender", description = "The hostname of the Syslog server that sent the message."),
+ @WritesAttribute(attribute = "syslog.body", description = "The body of the Syslog message, everything after the hostname.")})
--- End diff --
I remember a while ago someone suggested that the ParseSyslog processor should make it optional to put the body into attributes since it could be a lot larger than the rest of the attributes. Wondering if it makes sense to have a property like "Include Body" with true or false?
---
[GitHub] nifi issue #2805: NIFI-5325 A Syslog Parser that fully supports RFC 5424 Str...
Posted by bbende <gi...@git.apache.org>.
Github user bbende commented on the issue:
https://github.com/apache/nifi/pull/2805
Looks good, merging, thanks!
---
[GitHub] nifi pull request #2805: NIFI-5325 A Syslog Parser that fully supports RFC 5...
Posted by bbende <gi...@git.apache.org>.
Github user bbende commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2805#discussion_r196910966
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/syslog/StrictSyslog5424Parser.java ---
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard.syslog;
+
+import com.github.palindromicity.syslog.DefaultKeyProvider;
+import com.github.palindromicity.syslog.KeyProvider;
+import com.github.palindromicity.syslog.NilPolicy;
+import com.github.palindromicity.syslog.SyslogParserBuilder;
+
+import java.nio.ByteBuffer;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.regex.Pattern;
+
+/**
+ * Parses a Syslog message from a ByteBuffer into a SyslogEvent instance.
+ *
+ * The Syslog regular expressions below were adapted from the Apache Flume project for RFC3164 logs.
--- End diff --
Can probably remove the part about Flume and RFC3164 since its not part of this class right?
---
[GitHub] nifi pull request #2805: NIFI-5325 A Syslog Parser that fully supports RFC 5...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2805#discussion_r196931673
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestParseSyslog5424.java ---
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.Test;
+
+public class TestParseSyslog5424 {
+ private static final String SYSLOG_LINE_ALL = "<14>1 2014-06-20T09:14:07+00:00 loggregator"
+ + " d0602076-b14a-4c55-852a-981e7afeed38 DEA MSG-01"
+ + " [exampleSDID@32473 iut=\"3\" eventSource=\"Application\" eventID=\"1011\"]"
+ + " [exampleSDID@32480 iut=\"4\" eventSource=\"Other Application\" eventID=\"2022\"] Removing instance";
+ private static final String SYSLOG_LINE_NILS= "<14>1 2014-06-20T09:14:07+00:00 -"
+ + " d0602076-b14a-4c55-852a-981e7afeed38 - -"
+ + " [exampleSDID@32473 iut=\"3\" eventSource=\"Application\" eventID=\"1011\"]"
+ + " [exampleSDID@32480 iut=\"4\" eventSource=\"Other Application\" eventID=\"2022\"] Removing instance";
+
+ @Test
+ public void testValidMessage() {
+ final TestRunner runner = TestRunners.newTestRunner(new ParseSyslog5424());
+ runner.setProperty(ParseSyslog5424.NIL_POLICY,NilPolicy.DASH.name());
+ runner.enqueue(SYSLOG_LINE_ALL.getBytes());
+ runner.run();
+ runner.assertAllFlowFilesTransferred(ParseSyslog5424.REL_SUCCESS,1);
--- End diff --
Only version and priority must be present ( IE> cannot be the NIL '-' character ), and therefore based on the NilPolicy by be missing etc.
I *could* check for those, but their absence would actually throw an exception. I have a test for that specifically wrt priority
---
[GitHub] nifi pull request #2805: NIFI-5325 A Syslog Parser that fully supports RFC 5...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2805#discussion_r197151315
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ParseSyslog5424.java ---
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.standard.syslog.StrictSyslog5424Parser;
+import org.apache.nifi.processors.standard.syslog.Syslog5424Event;
+import org.apache.nifi.stream.io.StreamUtils;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"logs", "syslog", "syslog5424", "attributes", "system", "event", "message"})
+@CapabilityDescription("Attempts to parse the contents of a well formed Syslog message in accordance to RFC5424 " +
+ "format and adds attributes to the FlowFile for each of the parts of the Syslog message, including Structured Data." +
+ "Structured Data will be written to attributes as on attribute per item id + parameter "+
+ "see https://tools.ietf.org/html/rfc5424." +
+ "Note: ParseSyslog5424 follows the specification more closely than ParseSyslog. If your Syslog producer " +
+ "does not follow the spec closely, with regards to using '-' for missing header entries for example, those logs " +
+ "will fail with this parser, where they would not fail with ParseSyslog.")
+@WritesAttributes({@WritesAttribute(attribute = "syslog.priority", description = "The priority of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.severity", description = "The severity of the Syslog message derived from the priority."),
+ @WritesAttribute(attribute = "syslog.facility", description = "The facility of the Syslog message derived from the priority."),
+ @WritesAttribute(attribute = "syslog.version", description = "The optional version from the Syslog message."),
+ @WritesAttribute(attribute = "syslog.timestamp", description = "The timestamp of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.hostname", description = "The hostname or IP address of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.appname", description = "The appname of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.procid", description = "The procid of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.messageid", description = "The messageid the Syslog message."),
+ @WritesAttribute(attribute = "syslog.structuredData", description = "Multiple entries per structuredData of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.sender", description = "The hostname of the Syslog server that sent the message."),
+ @WritesAttribute(attribute = "syslog.body", description = "The body of the Syslog message, everything after the hostname.")})
--- End diff --
Done
---
[GitHub] nifi pull request #2805: NIFI-5325 A Syslog Parser that fully supports RFC 5...
Posted by bbende <gi...@git.apache.org>.
Github user bbende commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2805#discussion_r196905162
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-nar/src/main/resources/META-INF/NOTICE ---
@@ -218,6 +218,14 @@ The following binary components are provided under the Apache Software License v
Copyright 2006 - 2013 Pentaho Corporation. All rights reserved.
Copyright 2000-2005, 2014-2016 Julian Hyde
+ (ASLv2) simple-syslog-5424
+ The following NOTICE information applies:
+
+ simple-syslog-5424
+ https://github.com/palindromicity/simple-syslog-5424
+
+ Copyright 2018 simple-syslog-5424 authors.
+
--- End diff --
Lets also add this to the NOTICE in the nifi-assembly
---
[GitHub] nifi pull request #2805: NIFI-5325 A Syslog Parser that fully supports RFC 5...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2805#discussion_r196932374
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/pom.xml ---
@@ -327,6 +327,10 @@
<version>1.7.0-SNAPSHOT</version>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>com.github.palindromicity</groupId>
+ <artifactId>simple-syslog-5424</artifactId>
+ </dependency>
--- End diff --
I could only find references to this in the LICENSE files, not the NOTICE, so I made sure it was there where it was absent before ( assembly had it )
---
[GitHub] nifi pull request #2805: NIFI-5325 A Syslog Parser that fully supports RFC 5...
Posted by bbende <gi...@git.apache.org>.
Github user bbende commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2805#discussion_r196913302
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestParseSyslog5424.java ---
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.Test;
+
+public class TestParseSyslog5424 {
+ private static final String SYSLOG_LINE_ALL = "<14>1 2014-06-20T09:14:07+00:00 loggregator"
+ + " d0602076-b14a-4c55-852a-981e7afeed38 DEA MSG-01"
+ + " [exampleSDID@32473 iut=\"3\" eventSource=\"Application\" eventID=\"1011\"]"
+ + " [exampleSDID@32480 iut=\"4\" eventSource=\"Other Application\" eventID=\"2022\"] Removing instance";
+ private static final String SYSLOG_LINE_NILS= "<14>1 2014-06-20T09:14:07+00:00 -"
+ + " d0602076-b14a-4c55-852a-981e7afeed38 - -"
+ + " [exampleSDID@32473 iut=\"3\" eventSource=\"Application\" eventID=\"1011\"]"
+ + " [exampleSDID@32480 iut=\"4\" eventSource=\"Other Application\" eventID=\"2022\"] Removing instance";
+
+ @Test
+ public void testValidMessage() {
+ final TestRunner runner = TestRunners.newTestRunner(new ParseSyslog5424());
+ runner.setProperty(ParseSyslog5424.NIL_POLICY,NilPolicy.DASH.name());
+ runner.enqueue(SYSLOG_LINE_ALL.getBytes());
+ runner.run();
+ runner.assertAllFlowFilesTransferred(ParseSyslog5424.REL_SUCCESS,1);
--- End diff --
Not totally necessary, but may want to verify that the flow file transferred to success has some of the attributes you would expect from the parsed message
---
[GitHub] nifi pull request #2805: NIFI-5325 A Syslog Parser that fully supports RFC 5...
Posted by bbende <gi...@git.apache.org>.
Github user bbende commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2805#discussion_r197144571
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ParseSyslog5424.java ---
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.standard.syslog.StrictSyslog5424Parser;
+import org.apache.nifi.processors.standard.syslog.Syslog5424Event;
+import org.apache.nifi.stream.io.StreamUtils;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"logs", "syslog", "syslog5424", "attributes", "system", "event", "message"})
+@CapabilityDescription("Attempts to parse the contents of a well formed Syslog message in accordance to RFC5424 " +
+ "format and adds attributes to the FlowFile for each of the parts of the Syslog message, including Structured Data." +
+ "Structured Data will be written to attributes as one attribute per item id + parameter "+
+ "see https://tools.ietf.org/html/rfc5424." +
+ "Note: ParseSyslog5424 follows the specification more closely than ParseSyslog. If your Syslog producer " +
+ "does not follow the spec closely, with regards to using '-' for missing header entries for example, those logs " +
+ "will fail with this parser, where they would not fail with ParseSyslog.")
+@WritesAttributes({@WritesAttribute(attribute = "syslog.priority", description = "The priority of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.severity", description = "The severity of the Syslog message derived from the priority."),
+ @WritesAttribute(attribute = "syslog.facility", description = "The facility of the Syslog message derived from the priority."),
+ @WritesAttribute(attribute = "syslog.version", description = "The optional version from the Syslog message."),
+ @WritesAttribute(attribute = "syslog.timestamp", description = "The timestamp of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.hostname", description = "The hostname or IP address of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.appname", description = "The appname of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.procid", description = "The procid of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.messageid", description = "The messageid the Syslog message."),
+ @WritesAttribute(attribute = "syslog.structuredData", description = "Multiple entries per structuredData of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.sender", description = "The hostname of the Syslog server that sent the message."),
+ @WritesAttribute(attribute = "syslog.body", description = "The body of the Syslog message, everything after the hostname.")})
+@SeeAlso({ListenSyslog.class, ParseSyslog.class, PutSyslog.class})
+public class ParseSyslog5424 extends AbstractProcessor {
+
+ public static final AllowableValue OMIT = new AllowableValue(NilPolicy.OMIT.name(),NilPolicy.OMIT.name(),"The missing field will not have an attribute added.");
+ public static final AllowableValue NULL = new AllowableValue(NilPolicy.NULL.name(),NilPolicy.NULL.name(),"The missing field will have an empty attribute added.");
+ public static final AllowableValue DASH = new AllowableValue(NilPolicy.DASH.name(),NilPolicy.DASH.name(),"The missing field will have an attribute added with the value of '-'.");
+
+ public static final PropertyDescriptor CHARSET = new PropertyDescriptor.Builder()
+ .name("Character Set")
+ .description("Specifies which character set of the Syslog messages")
+ .required(true)
+ .defaultValue("UTF-8")
+ .addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor NIL_POLICY = new PropertyDescriptor.Builder()
+ .name("nil_policy")
+ .displayName("NIL Policy")
+ .description("Defines how NIL values are handled for header fields.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .allowableValues(OMIT,NULL,DASH)
+ .required(true)
+ .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+ .defaultValue(NULL.getValue())
+ .build();
+
+ static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Any FlowFile that could not be parsed as a Syslog message will be transferred to this Relationship without any attributes being added")
+ .build();
+ static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Any FlowFile that is successfully parsed as a Syslog message will be to this Relationship.")
+ .build();
+
+ private volatile StrictSyslog5424Parser parser;
+
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ final List<PropertyDescriptor> properties = new ArrayList<>(2);
+ properties.add(CHARSET);
+ properties.add(NIL_POLICY);
+ return properties;
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ final Set<Relationship> relationships = new HashSet<>();
+ relationships.add(REL_FAILURE);
+ relationships.add(REL_SUCCESS);
+ return relationships;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ FlowFile flowFile = session.get();
+ if (flowFile == null) {
+ return;
+ }
+
+ final String charsetName = context.getProperty(CHARSET).getValue();
+ final String nilPolicyString = context.getProperty(NIL_POLICY).getValue();
+
+ // If the parser already exists and uses the same charset, it does not need to be re-initialized
+ if (parser == null || !parser.getCharsetName().equals(charsetName)) {
--- End diff --
I thought I wrote a comment here yesterday, but it seems to have not made it through..
If the processor is configured with concurrent tasks > 1 then multiple threads may enter this logic concurrently and be in a race condition to initialize the parser. It may not be a big problem, but usually we try to protect against this.
Since the parser here is not making any external calls when being constructed, it would safe to construct it in an @OnSchedule method every time the processor starts.
If you prefer the lazy init in onTrigger then usually a check-synchronize-recheck pattern works...
```
if (parser == null || !parser.getCharsetName().equals(charsetName)) {
synchronized(this) {
if (parser == null || !parser.getCharsetName().equals(charsetName)) {
parser = new StrictSyslog5424Parser(Charset.forName(charsetName),NilPolicy.valueOf(nilPolicyString));
}
}
}
```
---
[GitHub] nifi issue #2805: NIFI-5325 A Syslog Parser that fully supports RFC 5424 Str...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/nifi/pull/2805
@bbende I think I addressed all the feedback except for the new msgbody idea
---
[GitHub] nifi pull request #2805: NIFI-5325 A Syslog Parser that fully supports RFC 5...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2805#discussion_r196932588
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/syslog/StrictSyslog5424Parser.java ---
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard.syslog;
+
+import com.github.palindromicity.syslog.DefaultKeyProvider;
+import com.github.palindromicity.syslog.KeyProvider;
+import com.github.palindromicity.syslog.NilPolicy;
+import com.github.palindromicity.syslog.SyslogParserBuilder;
+
+import java.nio.ByteBuffer;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.regex.Pattern;
+
+/**
+ * Parses a Syslog message from a ByteBuffer into a SyslogEvent instance.
+ *
+ * The Syslog regular expressions below were adapted from the Apache Flume project for RFC3164 logs.
--- End diff --
sorry, leftover from my initial SyslogParser modifications
---
[GitHub] nifi pull request #2805: NIFI-5325 A Syslog Parser that fully supports RFC 5...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2805#discussion_r196932176
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-nar/src/main/resources/META-INF/NOTICE ---
@@ -218,6 +218,14 @@ The following binary components are provided under the Apache Software License v
Copyright 2006 - 2013 Pentaho Corporation. All rights reserved.
Copyright 2000-2005, 2014-2016 Julian Hyde
+ (ASLv2) simple-syslog-5424
+ The following NOTICE information applies:
+
+ simple-syslog-5424
+ https://github.com/palindromicity/simple-syslog-5424
+
+ Copyright 2018 simple-syslog-5424 authors.
+
--- End diff --
done
---
[GitHub] nifi pull request #2805: NIFI-5325 A Syslog Parser that fully supports RFC 5...
Posted by bbende <gi...@git.apache.org>.
Github user bbende commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2805#discussion_r197142251
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ParseSyslog5424.java ---
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.standard.syslog.StrictSyslog5424Parser;
+import org.apache.nifi.processors.standard.syslog.Syslog5424Event;
+import org.apache.nifi.stream.io.StreamUtils;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"logs", "syslog", "syslog5424", "attributes", "system", "event", "message"})
+@CapabilityDescription("Attempts to parse the contents of a well formed Syslog message in accordance to RFC5424 " +
+ "format and adds attributes to the FlowFile for each of the parts of the Syslog message, including Structured Data." +
+ "Structured Data will be written to attributes as on attribute per item id + parameter "+
+ "see https://tools.ietf.org/html/rfc5424." +
+ "Note: ParseSyslog5424 follows the specification more closely than ParseSyslog. If your Syslog producer " +
+ "does not follow the spec closely, with regards to using '-' for missing header entries for example, those logs " +
+ "will fail with this parser, where they would not fail with ParseSyslog.")
+@WritesAttributes({@WritesAttribute(attribute = "syslog.priority", description = "The priority of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.severity", description = "The severity of the Syslog message derived from the priority."),
+ @WritesAttribute(attribute = "syslog.facility", description = "The facility of the Syslog message derived from the priority."),
+ @WritesAttribute(attribute = "syslog.version", description = "The optional version from the Syslog message."),
+ @WritesAttribute(attribute = "syslog.timestamp", description = "The timestamp of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.hostname", description = "The hostname or IP address of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.appname", description = "The appname of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.procid", description = "The procid of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.messageid", description = "The messageid the Syslog message."),
+ @WritesAttribute(attribute = "syslog.structuredData", description = "Multiple entries per structuredData of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.sender", description = "The hostname of the Syslog server that sent the message."),
+ @WritesAttribute(attribute = "syslog.body", description = "The body of the Syslog message, everything after the hostname.")})
--- End diff --
I guess it depends what is most likely to be done with syslog messages after this processor... basically will it be more common that a user wants the full message in the flow file content because they are going to send this data to an external system that accepts the full message? or will it be more common that they just want the message body?
I lean towards leaving the flow file content as is, since that is what ParseSyslog does, and just letting them control whether or not the message body is added as an attribute when parsing. Most cases won't need the message body in attribute, they just want to use one of the other fields to make a routing/filtering decisions and then send the whole message somewhere.
---
[GitHub] nifi pull request #2805: NIFI-5325 A Syslog Parser that fully supports RFC 5...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2805#discussion_r197154264
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ParseSyslog5424.java ---
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.standard.syslog.StrictSyslog5424Parser;
+import org.apache.nifi.processors.standard.syslog.Syslog5424Event;
+import org.apache.nifi.stream.io.StreamUtils;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"logs", "syslog", "syslog5424", "attributes", "system", "event", "message"})
+@CapabilityDescription("Attempts to parse the contents of a well formed Syslog message in accordance to RFC5424 " +
+ "format and adds attributes to the FlowFile for each of the parts of the Syslog message, including Structured Data." +
+ "Structured Data will be written to attributes as one attribute per item id + parameter "+
+ "see https://tools.ietf.org/html/rfc5424." +
+ "Note: ParseSyslog5424 follows the specification more closely than ParseSyslog. If your Syslog producer " +
+ "does not follow the spec closely, with regards to using '-' for missing header entries for example, those logs " +
+ "will fail with this parser, where they would not fail with ParseSyslog.")
+@WritesAttributes({@WritesAttribute(attribute = "syslog.priority", description = "The priority of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.severity", description = "The severity of the Syslog message derived from the priority."),
+ @WritesAttribute(attribute = "syslog.facility", description = "The facility of the Syslog message derived from the priority."),
+ @WritesAttribute(attribute = "syslog.version", description = "The optional version from the Syslog message."),
+ @WritesAttribute(attribute = "syslog.timestamp", description = "The timestamp of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.hostname", description = "The hostname or IP address of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.appname", description = "The appname of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.procid", description = "The procid of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.messageid", description = "The messageid the Syslog message."),
+ @WritesAttribute(attribute = "syslog.structuredData", description = "Multiple entries per structuredData of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.sender", description = "The hostname of the Syslog server that sent the message."),
+ @WritesAttribute(attribute = "syslog.body", description = "The body of the Syslog message, everything after the hostname.")})
+@SeeAlso({ListenSyslog.class, ParseSyslog.class, PutSyslog.class})
+public class ParseSyslog5424 extends AbstractProcessor {
+
+ public static final AllowableValue OMIT = new AllowableValue(NilPolicy.OMIT.name(),NilPolicy.OMIT.name(),"The missing field will not have an attribute added.");
+ public static final AllowableValue NULL = new AllowableValue(NilPolicy.NULL.name(),NilPolicy.NULL.name(),"The missing field will have an empty attribute added.");
+ public static final AllowableValue DASH = new AllowableValue(NilPolicy.DASH.name(),NilPolicy.DASH.name(),"The missing field will have an attribute added with the value of '-'.");
+
+ public static final PropertyDescriptor CHARSET = new PropertyDescriptor.Builder()
+ .name("Character Set")
+ .description("Specifies which character set of the Syslog messages")
+ .required(true)
+ .defaultValue("UTF-8")
+ .addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor NIL_POLICY = new PropertyDescriptor.Builder()
+ .name("nil_policy")
+ .displayName("NIL Policy")
+ .description("Defines how NIL values are handled for header fields.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .allowableValues(OMIT,NULL,DASH)
+ .required(true)
+ .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+ .defaultValue(NULL.getValue())
+ .build();
+
+ static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Any FlowFile that could not be parsed as a Syslog message will be transferred to this Relationship without any attributes being added")
+ .build();
+ static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Any FlowFile that is successfully parsed as a Syslog message will be to this Relationship.")
+ .build();
+
+ private volatile StrictSyslog5424Parser parser;
+
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ final List<PropertyDescriptor> properties = new ArrayList<>(2);
+ properties.add(CHARSET);
+ properties.add(NIL_POLICY);
+ return properties;
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ final Set<Relationship> relationships = new HashSet<>();
+ relationships.add(REL_FAILURE);
+ relationships.add(REL_SUCCESS);
+ return relationships;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ FlowFile flowFile = session.get();
+ if (flowFile == null) {
+ return;
+ }
+
+ final String charsetName = context.getProperty(CHARSET).getValue();
+ final String nilPolicyString = context.getProperty(NIL_POLICY).getValue();
+
+ // If the parser already exists and uses the same charset, it does not need to be re-initialized
+ if (parser == null || !parser.getCharsetName().equals(charsetName)) {
--- End diff --
done
---
[GitHub] nifi pull request #2805: NIFI-5325 A Syslog Parser that fully supports RFC 5...
Posted by bbende <gi...@git.apache.org>.
Github user bbende commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2805#discussion_r196906833
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/pom.xml ---
@@ -327,6 +327,10 @@
<version>1.7.0-SNAPSHOT</version>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>com.github.palindromicity</groupId>
+ <artifactId>simple-syslog-5424</artifactId>
+ </dependency>
--- End diff --
Since this transitively brings in antlr-runtime (I think) then we probably need to account for that in the NAR and assembly LICENSE/NOTICE.
From a quick glance it looks like antlr is BSD 3-Clause so it probably goes in the LICENSE files with the verbiage like: "The binary distribution of this product bundles 'antlr-runtime' which is available under a "3-clause BSD" license."
---
[GitHub] nifi pull request #2805: NIFI-5325 A Syslog Parser that fully supports RFC 5...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2805#discussion_r196932469
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ParseSyslog5424.java ---
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.standard.syslog.StrictSyslog5424Parser;
+import org.apache.nifi.processors.standard.syslog.Syslog5424Event;
+import org.apache.nifi.stream.io.StreamUtils;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"logs", "syslog", "syslog5424", "attributes", "system", "event", "message"})
+@CapabilityDescription("Attempts to parse the contents of a well formed Syslog message in accordance to RFC5424 " +
+ "format and adds attributes to the FlowFile for each of the parts of the Syslog message, including Structured Data." +
+ "Structured Data will be written to attributes as on attribute per item id + parameter "+
+ "see https://tools.ietf.org/html/rfc5424." +
+ "Note: ParseSyslog5424 follows the specification more closely than ParseSyslog. If your Syslog producer " +
+ "does not follow the spec closely, with regards to using '-' for missing header entries for example, those logs " +
+ "will fail with this parser, where they would not fail with ParseSyslog.")
+@WritesAttributes({@WritesAttribute(attribute = "syslog.priority", description = "The priority of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.severity", description = "The severity of the Syslog message derived from the priority."),
+ @WritesAttribute(attribute = "syslog.facility", description = "The facility of the Syslog message derived from the priority."),
+ @WritesAttribute(attribute = "syslog.version", description = "The optional version from the Syslog message."),
+ @WritesAttribute(attribute = "syslog.timestamp", description = "The timestamp of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.hostname", description = "The hostname or IP address of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.appname", description = "The appname of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.procid", description = "The procid of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.messageid", description = "The messageid the Syslog message."),
+ @WritesAttribute(attribute = "syslog.structuredData", description = "Multiple entries per structuredData of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.sender", description = "The hostname of the Syslog server that sent the message."),
+ @WritesAttribute(attribute = "syslog.body", description = "The body of the Syslog message, everything after the hostname.")})
+@SeeAlso({ListenSyslog.class, ParseSyslog.class, PutSyslog.class})
+public class ParseSyslog5424 extends AbstractProcessor {
+
+ public static final AllowableValue OMIT = new AllowableValue(NilPolicy.OMIT.name(),NilPolicy.OMIT.name(),"The missing field will not have an attribute added.");
+ public static final AllowableValue NULL = new AllowableValue(NilPolicy.NULL.name(),NilPolicy.NULL.name(),"The missing field will have an empty attribute added.");
+ public static final AllowableValue DASH = new AllowableValue(NilPolicy.DASH.name(),NilPolicy.DASH.name(),"The missing field will have an attribute added with the value of '-'.");
+
+ public static final PropertyDescriptor CHARSET = new PropertyDescriptor.Builder()
+ .name("Character Set")
+ .description("Specifies which character set of the Syslog messages")
+ .required(true)
+ .defaultValue("UTF-8")
+ .addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor NIL_POLICY = new PropertyDescriptor.Builder()
+ .name("nil_policy")
+ .displayName("NIL Policy")
+ .description("Defines how NIL values are handled for header fields.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .allowableValues(OMIT,NULL,DASH)
+ .required(true)
+ .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+ .defaultValue(NULL.getValue())
+ .build();
+
+ static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Any FlowFile that could not be parsed as a Syslog message will be transferred to this Relationship without any attributes being added")
+ .build();
+ static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Any FlowFile that is successfully parsed as a Syslog message will be to this Relationship.")
+ .build();
+
+ private StrictSyslog5424Parser parser;
--- End diff --
done
---
[GitHub] nifi pull request #2805: NIFI-5325 A Syslog Parser that fully supports RFC 5...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2805#discussion_r196932621
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/syslog/StrictSyslog5424Parser.java ---
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard.syslog;
+
+import com.github.palindromicity.syslog.DefaultKeyProvider;
+import com.github.palindromicity.syslog.KeyProvider;
+import com.github.palindromicity.syslog.NilPolicy;
+import com.github.palindromicity.syslog.SyslogParserBuilder;
+
+import java.nio.ByteBuffer;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.regex.Pattern;
+
+/**
+ * Parses a Syslog message from a ByteBuffer into a SyslogEvent instance.
+ *
+ * The Syslog regular expressions below were adapted from the Apache Flume project for RFC3164 logs.
+ * For 5424 we use simple-syslog-5424 since it parsers out structured data.
+ */
+public class StrictSyslog5424Parser {
+ private Charset charset;
+ private com.github.palindromicity.syslog.SyslogParser parser;
+ KeyProvider keyProvider = new DefaultKeyProvider();
--- End diff --
done
---
[GitHub] nifi pull request #2805: NIFI-5325 A Syslog Parser that fully supports RFC 5...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/nifi/pull/2805
---
[GitHub] nifi pull request #2805: NIFI-5325 A Syslog Parser that fully supports RFC 5...
Posted by bbende <gi...@git.apache.org>.
Github user bbende commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2805#discussion_r196912736
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/syslog/StrictSyslog5424Parser.java ---
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard.syslog;
+
+import com.github.palindromicity.syslog.DefaultKeyProvider;
+import com.github.palindromicity.syslog.KeyProvider;
+import com.github.palindromicity.syslog.NilPolicy;
+import com.github.palindromicity.syslog.SyslogParserBuilder;
+
+import java.nio.ByteBuffer;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.regex.Pattern;
+
+/**
+ * Parses a Syslog message from a ByteBuffer into a SyslogEvent instance.
+ *
+ * The Syslog regular expressions below were adapted from the Apache Flume project for RFC3164 logs.
+ * For 5424 we use simple-syslog-5424 since it parsers out structured data.
+ */
+public class StrictSyslog5424Parser {
+ private Charset charset;
+ private com.github.palindromicity.syslog.SyslogParser parser;
+ KeyProvider keyProvider = new DefaultKeyProvider();
--- End diff --
Seems like this is unused
---
[GitHub] nifi pull request #2805: NIFI-5325 A Syslog Parser that fully supports RFC 5...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2805#discussion_r196929192
--- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ParseSyslog5424.java ---
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.standard.syslog.StrictSyslog5424Parser;
+import org.apache.nifi.processors.standard.syslog.Syslog5424Event;
+import org.apache.nifi.stream.io.StreamUtils;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@Tags({"logs", "syslog", "syslog5424", "attributes", "system", "event", "message"})
+@CapabilityDescription("Attempts to parse the contents of a well formed Syslog message in accordance to RFC5424 " +
+ "format and adds attributes to the FlowFile for each of the parts of the Syslog message, including Structured Data." +
+ "Structured Data will be written to attributes as on attribute per item id + parameter "+
+ "see https://tools.ietf.org/html/rfc5424." +
+ "Note: ParseSyslog5424 follows the specification more closely than ParseSyslog. If your Syslog producer " +
+ "does not follow the spec closely, with regards to using '-' for missing header entries for example, those logs " +
+ "will fail with this parser, where they would not fail with ParseSyslog.")
+@WritesAttributes({@WritesAttribute(attribute = "syslog.priority", description = "The priority of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.severity", description = "The severity of the Syslog message derived from the priority."),
+ @WritesAttribute(attribute = "syslog.facility", description = "The facility of the Syslog message derived from the priority."),
+ @WritesAttribute(attribute = "syslog.version", description = "The optional version from the Syslog message."),
+ @WritesAttribute(attribute = "syslog.timestamp", description = "The timestamp of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.hostname", description = "The hostname or IP address of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.appname", description = "The appname of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.procid", description = "The procid of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.messageid", description = "The messageid the Syslog message."),
+ @WritesAttribute(attribute = "syslog.structuredData", description = "Multiple entries per structuredData of the Syslog message."),
+ @WritesAttribute(attribute = "syslog.sender", description = "The hostname of the Syslog server that sent the message."),
+ @WritesAttribute(attribute = "syslog.body", description = "The body of the Syslog message, everything after the hostname.")})
--- End diff --
Another option would be to put the attributes in, and write the body AS the flowfile
Kind of separating syslog the transport from the message fmt.
I think this is a good idea, anyone else we can bounce this off of?
---