You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by JohannesDaniel <gi...@git.apache.org> on 2018/05/04 14:48:00 UTC
[GitHub] nifi pull request #2675: NIFI-5113 Add XMLRecordSetWriter
GitHub user JohannesDaniel opened a pull request:
https://github.com/apache/nifi/pull/2675
NIFI-5113 Add XMLRecordSetWriter
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [ ] Has your PR been rebased against the latest commit within the target branch (typically master)?
- [ ] Is your initial contribution a single, squashed commit?
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JohannesDaniel/nifi NIFI-5113
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/2675.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2675
----
commit 58600a6655a8b1ce11cbc329c9feaa308c240f08
Author: JohannesDaniel <jo...@...>
Date: 2018-04-23T19:35:40Z
Add XMLRecordSetWriter
----
---
[GitHub] nifi pull request #2675: NIFI-5113 Add XMLRecordSetWriter
Posted by markap14 <gi...@git.apache.org>.
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2675#discussion_r187359257
--- Diff: nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/NullSuppression.java ---
@@ -15,7 +15,7 @@
* limitations under the License.
*/
-package org.apache.nifi.json;
+package org.apache.nifi;
--- End diff --
I'm fine moving this out of the json package into something more generic, but I don't think org.apache.nifi is the appropriate package. Perhaps org.apache.nifi.record or something like that...
---
[GitHub] nifi pull request #2675: NIFI-5113 Add XMLRecordSetWriter
Posted by JohannesDaniel <gi...@git.apache.org>.
Github user JohannesDaniel commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2675#discussion_r187592686
--- Diff: nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/xml/WriteXMLResult.java ---
@@ -0,0 +1,602 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.xml;
+
+import javanet.staxutils.IndentingXMLStreamWriter;
+import org.apache.nifi.NullSuppression;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.schema.access.SchemaAccessWriter;
+import org.apache.nifi.serialization.AbstractRecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.WriteResult;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RawRecordWriter;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordField;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.serialization.record.type.ArrayDataType;
+import org.apache.nifi.serialization.record.type.ChoiceDataType;
+import org.apache.nifi.serialization.record.type.MapDataType;
+import org.apache.nifi.serialization.record.type.RecordDataType;
+import org.apache.nifi.serialization.record.util.DataTypeUtils;
+
+import javax.xml.stream.XMLOutputFactory;
+import javax.xml.stream.XMLStreamException;
+import javax.xml.stream.XMLStreamWriter;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.text.DateFormat;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Supplier;
+
+
+public class WriteXMLResult extends AbstractRecordSetWriter implements RecordSetWriter, RawRecordWriter {
+
+ final ComponentLog logger;
+ final RecordSchema recordSchema;
+ final SchemaAccessWriter schemaAccess;
+ final XMLStreamWriter writer;
+ final NullSuppression nullSuppression;
+ final ArrayWrapping arrayWrapping;
+ final String arrayTagName;
+ final String recordTagName;
+ final String rootTagName;
+
+ private final Supplier<DateFormat> LAZY_DATE_FORMAT;
+ private final Supplier<DateFormat> LAZY_TIME_FORMAT;
+ private final Supplier<DateFormat> LAZY_TIMESTAMP_FORMAT;
+
+ public WriteXMLResult(final ComponentLog logger, final RecordSchema recordSchema, final SchemaAccessWriter schemaAccess, final OutputStream out, final boolean prettyPrint,
+ final NullSuppression nullSuppression, final ArrayWrapping arrayWrapping, final String arrayTagName, final String rootTagName, final String recordTagName,
+ final String dateFormat, final String timeFormat, final String timestampFormat) throws IOException {
+
+ super(out);
+
+ this.logger = logger;
+ this.recordSchema = recordSchema;
+ this.schemaAccess = schemaAccess;
+ this.nullSuppression = nullSuppression;
+
+ this.arrayWrapping = arrayWrapping;
+ this.arrayTagName = arrayTagName;
+
+ this.rootTagName = rootTagName;
+ this.recordTagName = recordTagName;
+
+ final DateFormat df = dateFormat == null ? null : DataTypeUtils.getDateFormat(dateFormat);
+ final DateFormat tf = timeFormat == null ? null : DataTypeUtils.getDateFormat(timeFormat);
+ final DateFormat tsf = timestampFormat == null ? null : DataTypeUtils.getDateFormat(timestampFormat);
+
+ LAZY_DATE_FORMAT = () -> df;
+ LAZY_TIME_FORMAT = () -> tf;
+ LAZY_TIMESTAMP_FORMAT = () -> tsf;
+
+ try {
+ XMLOutputFactory factory = XMLOutputFactory.newInstance();
+
+ if (prettyPrint) {
+ writer = new IndentingXMLStreamWriter(factory.createXMLStreamWriter(out));
+ } else {
+ writer = factory.createXMLStreamWriter(out);
+ }
+
+ } catch (XMLStreamException e) {
+ throw new IOException(e.getMessage());
+ }
+ }
+
+ @Override
+ protected void onBeginRecordSet() throws IOException {
+
+ final OutputStream out = getOutputStream();
+ schemaAccess.writeHeader(recordSchema, out);
+
+ try {
+ writer.writeStartDocument();
+
+ writer.writeStartElement(rootTagName);
+
+ } catch (XMLStreamException e) {
+ throw new IOException(e.getMessage());
+ }
+ }
+
+ @Override
+ protected Map<String, String> onFinishRecordSet() throws IOException {
+
+ try {
+ writer.writeEndElement();
+ writer.writeEndDocument();
+ } catch (XMLStreamException e) {
+ throw new IOException(e.getMessage());
+ }
+ return schemaAccess.getAttributes(recordSchema);
+ }
+
+ @Override
+ public void close() throws IOException {
+
+ try {
+ writer.close();
+
+ } catch (XMLStreamException e) {
+ throw new IOException(e.getMessage());
+ }
+
+ super.close();
+ }
+
+ @Override
+ public void flush() throws IOException {
+
+ try {
+ writer.flush();
+
+ } catch (XMLStreamException e) {
+ throw new IOException(e.getMessage());
+ }
+ }
+
+ @Override
+ protected Map<String, String> writeRecord(Record record) throws IOException {
+
+ if (!isActiveRecordSet()) {
+ schemaAccess.writeHeader(recordSchema, getOutputStream());
+ }
+
+ List<String> tagsToOpen = new ArrayList<>();
--- End diff --
cool, thx!
---
[GitHub] nifi pull request #2675: NIFI-5113 Add XMLRecordSetWriter
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2675#discussion_r187591727
--- Diff: nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/xml/WriteXMLResult.java ---
@@ -0,0 +1,602 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.xml;
+
+import javanet.staxutils.IndentingXMLStreamWriter;
+import org.apache.nifi.NullSuppression;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.schema.access.SchemaAccessWriter;
+import org.apache.nifi.serialization.AbstractRecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.WriteResult;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RawRecordWriter;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordField;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.serialization.record.type.ArrayDataType;
+import org.apache.nifi.serialization.record.type.ChoiceDataType;
+import org.apache.nifi.serialization.record.type.MapDataType;
+import org.apache.nifi.serialization.record.type.RecordDataType;
+import org.apache.nifi.serialization.record.util.DataTypeUtils;
+
+import javax.xml.stream.XMLOutputFactory;
+import javax.xml.stream.XMLStreamException;
+import javax.xml.stream.XMLStreamWriter;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.text.DateFormat;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Supplier;
+
+
+public class WriteXMLResult extends AbstractRecordSetWriter implements RecordSetWriter, RawRecordWriter {
+
+ final ComponentLog logger;
+ final RecordSchema recordSchema;
+ final SchemaAccessWriter schemaAccess;
+ final XMLStreamWriter writer;
+ final NullSuppression nullSuppression;
+ final ArrayWrapping arrayWrapping;
+ final String arrayTagName;
+ final String recordTagName;
+ final String rootTagName;
+
+ private final Supplier<DateFormat> LAZY_DATE_FORMAT;
+ private final Supplier<DateFormat> LAZY_TIME_FORMAT;
+ private final Supplier<DateFormat> LAZY_TIMESTAMP_FORMAT;
+
+ public WriteXMLResult(final ComponentLog logger, final RecordSchema recordSchema, final SchemaAccessWriter schemaAccess, final OutputStream out, final boolean prettyPrint,
+ final NullSuppression nullSuppression, final ArrayWrapping arrayWrapping, final String arrayTagName, final String rootTagName, final String recordTagName,
+ final String dateFormat, final String timeFormat, final String timestampFormat) throws IOException {
+
+ super(out);
+
+ this.logger = logger;
+ this.recordSchema = recordSchema;
+ this.schemaAccess = schemaAccess;
+ this.nullSuppression = nullSuppression;
+
+ this.arrayWrapping = arrayWrapping;
+ this.arrayTagName = arrayTagName;
+
+ this.rootTagName = rootTagName;
+ this.recordTagName = recordTagName;
+
+ final DateFormat df = dateFormat == null ? null : DataTypeUtils.getDateFormat(dateFormat);
+ final DateFormat tf = timeFormat == null ? null : DataTypeUtils.getDateFormat(timeFormat);
+ final DateFormat tsf = timestampFormat == null ? null : DataTypeUtils.getDateFormat(timestampFormat);
+
+ LAZY_DATE_FORMAT = () -> df;
+ LAZY_TIME_FORMAT = () -> tf;
+ LAZY_TIMESTAMP_FORMAT = () -> tsf;
+
+ try {
+ XMLOutputFactory factory = XMLOutputFactory.newInstance();
+
+ if (prettyPrint) {
+ writer = new IndentingXMLStreamWriter(factory.createXMLStreamWriter(out));
+ } else {
+ writer = factory.createXMLStreamWriter(out);
+ }
+
+ } catch (XMLStreamException e) {
+ throw new IOException(e.getMessage());
+ }
+ }
+
+ @Override
+ protected void onBeginRecordSet() throws IOException {
+
+ final OutputStream out = getOutputStream();
+ schemaAccess.writeHeader(recordSchema, out);
+
+ try {
+ writer.writeStartDocument();
+
+ writer.writeStartElement(rootTagName);
+
+ } catch (XMLStreamException e) {
+ throw new IOException(e.getMessage());
+ }
+ }
+
+ @Override
+ protected Map<String, String> onFinishRecordSet() throws IOException {
+
+ try {
+ writer.writeEndElement();
+ writer.writeEndDocument();
+ } catch (XMLStreamException e) {
+ throw new IOException(e.getMessage());
+ }
+ return schemaAccess.getAttributes(recordSchema);
+ }
+
+ @Override
+ public void close() throws IOException {
+
+ try {
+ writer.close();
+
+ } catch (XMLStreamException e) {
+ throw new IOException(e.getMessage());
+ }
+
+ super.close();
+ }
+
+ @Override
+ public void flush() throws IOException {
+
+ try {
+ writer.flush();
+
+ } catch (XMLStreamException e) {
+ throw new IOException(e.getMessage());
+ }
+ }
+
+ @Override
+ protected Map<String, String> writeRecord(Record record) throws IOException {
+
+ if (!isActiveRecordSet()) {
+ schemaAccess.writeHeader(recordSchema, getOutputStream());
+ }
+
+ List<String> tagsToOpen = new ArrayList<>();
--- End diff --
https://docs.oracle.com/javase/7/docs/api/java/util/Deque.html
---
[GitHub] nifi pull request #2675: NIFI-5113 Add XMLRecordSetWriter
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/nifi/pull/2675
---
[GitHub] nifi pull request #2675: NIFI-5113 Add XMLRecordSetWriter
Posted by markap14 <gi...@git.apache.org>.
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2675#discussion_r187364738
--- Diff: nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/xml/XMLRecordSetWriter.java ---
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.xml;
+
+import org.apache.nifi.NullSuppression;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.serialization.DateTimeTextRecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+
+@Tags({"xml", "resultset", "writer", "serialize", "record", "recordset", "row"})
+@CapabilityDescription("Writes a RecordSet to XML. The records are wrapped by a root tag.")
+public class XMLRecordSetWriter extends DateTimeTextRecordSetWriter implements RecordSetWriterFactory {
+
+ public static final AllowableValue ALWAYS_SUPPRESS = new AllowableValue("always-suppress", "Always Suppress",
+ "Fields that are missing (present in the schema but not in the record), or that have a value of null, will not be written out");
+ public static final AllowableValue NEVER_SUPPRESS = new AllowableValue("never-suppress", "Never Suppress",
+ "Fields that are missing (present in the schema but not in the record), or that have a value of null, will be written out as a null value");
+ public static final AllowableValue SUPPRESS_MISSING = new AllowableValue("suppress-missing", "Suppress Missing Values",
+ "When a field has a value of null, it will be written out. However, if a field is defined in the schema and not present in the record, the field will not be written out.");
+
+ public static final AllowableValue USE_PROPERTY_AS_WRAPPER = new AllowableValue("use-property-as-wrapper", "Use Property as Wrapper",
+ "The value of the property \"Array Tag Name\" will be used as the tag name to wrap elements of an array. The field name of the array field will be used for the tag name " +
+ "of the elements.");
+ public static final AllowableValue USE_PROPERTY_FOR_ELEMENTS = new AllowableValue("use-property-for-elements", "Use Property for Elements",
+ "The value of the property \"Array Tag Name\" will be used for the tag name of the elements of an array. The field name of the array field will be used as the tag name " +
+ "to wrap elements.");
+ public static final AllowableValue NO_WRAPPING = new AllowableValue("no-wrapping", "No Wrapping",
+ "The elements of an array will not be wrapped");
+
+ public static final PropertyDescriptor SUPPRESS_NULLS = new PropertyDescriptor.Builder()
+ .name("suppress_nulls")
+ .displayName("Suppress Null Values")
+ .description("Specifies how the writer should handle a null field")
+ .allowableValues(NEVER_SUPPRESS, ALWAYS_SUPPRESS, SUPPRESS_MISSING)
+ .defaultValue(NEVER_SUPPRESS.getValue())
+ .required(true)
+ .build();
+
+ public static final PropertyDescriptor PRETTY_PRINT_XML = new PropertyDescriptor.Builder()
+ .name("pretty_print_xml")
+ .displayName("Pretty Print XML")
+ .description("Specifies whether or not the XML should be pretty printed")
+ .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+ .allowableValues("true", "false")
+ .defaultValue("false")
+ .required(true)
+ .build();
+
+ public static final PropertyDescriptor ROOT_TAG_NAME = new PropertyDescriptor.Builder()
+ .name("root_tag_name")
+ .displayName("Name of Root Tag")
+ .description("Specifies the name of the XML root tag wrapping the record set")
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+ .defaultValue("root")
+ .required(true)
--- End diff --
I think this should be optional. If not specified, then just throw an Exception if attempting to write more than 1 record.
---
[GitHub] nifi issue #2675: NIFI-5113 Add XMLRecordSetWriter
Posted by JohannesDaniel <gi...@git.apache.org>.
Github user JohannesDaniel commented on the issue:
https://github.com/apache/nifi/pull/2675
Hi @markap14
here we go!
Initially, I planned to enable this writer also to write XML attributes, but that would have required a complex workaround. Furthermore, it would have raised several additional questions, e. g. how to treat fields of complex types that are flagged as attributes, how to validate the schema, and so on.
@pvillard31 How about a performance test ;)
---
[GitHub] nifi pull request #2675: NIFI-5113 Add XMLRecordSetWriter
Posted by markap14 <gi...@git.apache.org>.
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2675#discussion_r187395204
--- Diff: nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/pom.xml ---
@@ -86,12 +86,23 @@
<artifactId>avro</artifactId>
<version>1.8.1</version>
</dependency>
+ <dependency>
--- End diff --
I think the more appropriate dependency to use here is
```
<dependency>
<groupId>net.java.dev.stax-utils</groupId>
<artifactId>stax-utils</artifactId>
<version>20070216</version>
</dependency>
```
No?
Either way, both have a dependency on BEA's JSR173, which has the following license: https://www.calculate-linux.org/packages/licenses/bea.ri.jsr173 - I'm not entirely sure if this is permissible in Apache land and if it is, how to properly document it. That said, it appears that jsr 173 is now part of java and so that dependency I think can be excluded. I changed the pom as follows:
```
<dependency>
<groupId>net.java.dev.stax-utils</groupId>
<artifactId>stax-utils</artifactId>
<version>20070216</version>
<exclusions>
<exclusion>
<groupId>com.bea.xml</groupId>
<artifactId>jsr173-ri</artifactId>
</exclusion>
</exclusions>
</dependency>
```
And all appears to work just fine. Thoughts?
---
[GitHub] nifi issue #2675: NIFI-5113 Add XMLRecordSetWriter
Posted by JohannesDaniel <gi...@git.apache.org>.
Github user JohannesDaniel commented on the issue:
https://github.com/apache/nifi/pull/2675
@markap14 implemented changes as discussed
---
[GitHub] nifi issue #2675: NIFI-5113 Add XMLRecordSetWriter
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on the issue:
https://github.com/apache/nifi/pull/2675
@markap14 are you good with this or do you want some additional eyes on it?
---
[GitHub] nifi pull request #2675: NIFI-5113 Add XMLRecordSetWriter
Posted by markap14 <gi...@git.apache.org>.
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2675#discussion_r187359929
--- Diff: nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/xml/WriteXMLResult.java ---
@@ -0,0 +1,602 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.xml;
+
+import javanet.staxutils.IndentingXMLStreamWriter;
+import org.apache.nifi.NullSuppression;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.schema.access.SchemaAccessWriter;
+import org.apache.nifi.serialization.AbstractRecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.WriteResult;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RawRecordWriter;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordField;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.serialization.record.type.ArrayDataType;
+import org.apache.nifi.serialization.record.type.ChoiceDataType;
+import org.apache.nifi.serialization.record.type.MapDataType;
+import org.apache.nifi.serialization.record.type.RecordDataType;
+import org.apache.nifi.serialization.record.util.DataTypeUtils;
+
+import javax.xml.stream.XMLOutputFactory;
+import javax.xml.stream.XMLStreamException;
+import javax.xml.stream.XMLStreamWriter;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.text.DateFormat;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Supplier;
+
+
+public class WriteXMLResult extends AbstractRecordSetWriter implements RecordSetWriter, RawRecordWriter {
+
+ final ComponentLog logger;
+ final RecordSchema recordSchema;
+ final SchemaAccessWriter schemaAccess;
+ final XMLStreamWriter writer;
+ final NullSuppression nullSuppression;
+ final ArrayWrapping arrayWrapping;
+ final String arrayTagName;
+ final String recordTagName;
+ final String rootTagName;
+
+ private final Supplier<DateFormat> LAZY_DATE_FORMAT;
+ private final Supplier<DateFormat> LAZY_TIME_FORMAT;
+ private final Supplier<DateFormat> LAZY_TIMESTAMP_FORMAT;
+
+ public WriteXMLResult(final ComponentLog logger, final RecordSchema recordSchema, final SchemaAccessWriter schemaAccess, final OutputStream out, final boolean prettyPrint,
+ final NullSuppression nullSuppression, final ArrayWrapping arrayWrapping, final String arrayTagName, final String rootTagName, final String recordTagName,
+ final String dateFormat, final String timeFormat, final String timestampFormat) throws IOException {
+
+ super(out);
+
+ this.logger = logger;
+ this.recordSchema = recordSchema;
+ this.schemaAccess = schemaAccess;
+ this.nullSuppression = nullSuppression;
+
+ this.arrayWrapping = arrayWrapping;
+ this.arrayTagName = arrayTagName;
+
+ this.rootTagName = rootTagName;
+ this.recordTagName = recordTagName;
+
+ final DateFormat df = dateFormat == null ? null : DataTypeUtils.getDateFormat(dateFormat);
+ final DateFormat tf = timeFormat == null ? null : DataTypeUtils.getDateFormat(timeFormat);
+ final DateFormat tsf = timestampFormat == null ? null : DataTypeUtils.getDateFormat(timestampFormat);
+
+ LAZY_DATE_FORMAT = () -> df;
+ LAZY_TIME_FORMAT = () -> tf;
+ LAZY_TIMESTAMP_FORMAT = () -> tsf;
+
+ try {
+ XMLOutputFactory factory = XMLOutputFactory.newInstance();
--- End diff --
Yeah, any time that we write text we want to make the character set configurable.
---
[GitHub] nifi pull request #2675: NIFI-5113 Add XMLRecordSetWriter
Posted by markap14 <gi...@git.apache.org>.
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2675#discussion_r187366185
--- Diff: nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/xml/XMLRecordSetWriter.java ---
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.xml;
+
+import org.apache.nifi.NullSuppression;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.serialization.DateTimeTextRecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+
+@Tags({"xml", "resultset", "writer", "serialize", "record", "recordset", "row"})
+@CapabilityDescription("Writes a RecordSet to XML. The records are wrapped by a root tag.")
+public class XMLRecordSetWriter extends DateTimeTextRecordSetWriter implements RecordSetWriterFactory {
+
+ public static final AllowableValue ALWAYS_SUPPRESS = new AllowableValue("always-suppress", "Always Suppress",
+ "Fields that are missing (present in the schema but not in the record), or that have a value of null, will not be written out");
+ public static final AllowableValue NEVER_SUPPRESS = new AllowableValue("never-suppress", "Never Suppress",
+ "Fields that are missing (present in the schema but not in the record), or that have a value of null, will be written out as a null value");
+ public static final AllowableValue SUPPRESS_MISSING = new AllowableValue("suppress-missing", "Suppress Missing Values",
+ "When a field has a value of null, it will be written out. However, if a field is defined in the schema and not present in the record, the field will not be written out.");
+
+ public static final AllowableValue USE_PROPERTY_AS_WRAPPER = new AllowableValue("use-property-as-wrapper", "Use Property as Wrapper",
+ "The value of the property \"Array Tag Name\" will be used as the tag name to wrap elements of an array. The field name of the array field will be used for the tag name " +
+ "of the elements.");
+ public static final AllowableValue USE_PROPERTY_FOR_ELEMENTS = new AllowableValue("use-property-for-elements", "Use Property for Elements",
+ "The value of the property \"Array Tag Name\" will be used for the tag name of the elements of an array. The field name of the array field will be used as the tag name " +
+ "to wrap elements.");
+ public static final AllowableValue NO_WRAPPING = new AllowableValue("no-wrapping", "No Wrapping",
+ "The elements of an array will not be wrapped");
+
+ public static final PropertyDescriptor SUPPRESS_NULLS = new PropertyDescriptor.Builder()
+ .name("suppress_nulls")
+ .displayName("Suppress Null Values")
+ .description("Specifies how the writer should handle a null field")
+ .allowableValues(NEVER_SUPPRESS, ALWAYS_SUPPRESS, SUPPRESS_MISSING)
+ .defaultValue(NEVER_SUPPRESS.getValue())
+ .required(true)
+ .build();
+
+ public static final PropertyDescriptor PRETTY_PRINT_XML = new PropertyDescriptor.Builder()
+ .name("pretty_print_xml")
+ .displayName("Pretty Print XML")
+ .description("Specifies whether or not the XML should be pretty printed")
+ .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+ .allowableValues("true", "false")
+ .defaultValue("false")
+ .required(true)
+ .build();
+
+ public static final PropertyDescriptor ROOT_TAG_NAME = new PropertyDescriptor.Builder()
+ .name("root_tag_name")
+ .displayName("Name of Root Tag")
+ .description("Specifies the name of the XML root tag wrapping the record set")
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+ .defaultValue("root")
+ .required(true)
+ .build();
+
--- End diff --
I think that's a reasonable thing to do. I also think that it's a bug that the there's no schema name in that case. Would recommend leaving this property and setting it as optional. Then, if not specified, use the name of the schema. Also in AvroTypeUtil.createSchema(Schema avroSchema) it looks like we are defaulting to SchemaIdentifier.EMPTY but probably should probably build a Schema identifier using avroSchema.getName()
---
[GitHub] nifi pull request #2675: NIFI-5113 Add XMLRecordSetWriter
Posted by JohannesDaniel <gi...@git.apache.org>.
Github user JohannesDaniel commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2675#discussion_r187560805
--- Diff: nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/xml/XMLRecordSetWriter.java ---
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.xml;
+
+import org.apache.nifi.NullSuppression;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.serialization.DateTimeTextRecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+
+@Tags({"xml", "resultset", "writer", "serialize", "record", "recordset", "row"})
+@CapabilityDescription("Writes a RecordSet to XML. The records are wrapped by a root tag.")
+public class XMLRecordSetWriter extends DateTimeTextRecordSetWriter implements RecordSetWriterFactory {
+
+ public static final AllowableValue ALWAYS_SUPPRESS = new AllowableValue("always-suppress", "Always Suppress",
+ "Fields that are missing (present in the schema but not in the record), or that have a value of null, will not be written out");
+ public static final AllowableValue NEVER_SUPPRESS = new AllowableValue("never-suppress", "Never Suppress",
+ "Fields that are missing (present in the schema but not in the record), or that have a value of null, will be written out as a null value");
+ public static final AllowableValue SUPPRESS_MISSING = new AllowableValue("suppress-missing", "Suppress Missing Values",
+ "When a field has a value of null, it will be written out. However, if a field is defined in the schema and not present in the record, the field will not be written out.");
+
+ public static final AllowableValue USE_PROPERTY_AS_WRAPPER = new AllowableValue("use-property-as-wrapper", "Use Property as Wrapper",
+ "The value of the property \"Array Tag Name\" will be used as the tag name to wrap elements of an array. The field name of the array field will be used for the tag name " +
+ "of the elements.");
+ public static final AllowableValue USE_PROPERTY_FOR_ELEMENTS = new AllowableValue("use-property-for-elements", "Use Property for Elements",
+ "The value of the property \"Array Tag Name\" will be used for the tag name of the elements of an array. The field name of the array field will be used as the tag name " +
+ "to wrap elements.");
+ public static final AllowableValue NO_WRAPPING = new AllowableValue("no-wrapping", "No Wrapping",
+ "The elements of an array will not be wrapped");
+
+ public static final PropertyDescriptor SUPPRESS_NULLS = new PropertyDescriptor.Builder()
+ .name("suppress_nulls")
+ .displayName("Suppress Null Values")
+ .description("Specifies how the writer should handle a null field")
+ .allowableValues(NEVER_SUPPRESS, ALWAYS_SUPPRESS, SUPPRESS_MISSING)
+ .defaultValue(NEVER_SUPPRESS.getValue())
+ .required(true)
+ .build();
+
+ public static final PropertyDescriptor PRETTY_PRINT_XML = new PropertyDescriptor.Builder()
+ .name("pretty_print_xml")
+ .displayName("Pretty Print XML")
+ .description("Specifies whether or not the XML should be pretty printed")
+ .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+ .allowableValues("true", "false")
+ .defaultValue("false")
+ .required(true)
+ .build();
+
+ public static final PropertyDescriptor ROOT_TAG_NAME = new PropertyDescriptor.Builder()
+ .name("root_tag_name")
+ .displayName("Name of Root Tag")
+ .description("Specifies the name of the XML root tag wrapping the record set")
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+ .defaultValue("root")
+ .required(true)
+ .build();
+
--- End diff --
ok. shall I create a ticket for this?
---
[GitHub] nifi pull request #2675: NIFI-5113 Add XMLRecordSetWriter
Posted by JohannesDaniel <gi...@git.apache.org>.
Github user JohannesDaniel commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2675#discussion_r187549020
--- Diff: nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/xml/WriteXMLResult.java ---
@@ -0,0 +1,602 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.xml;
+
+import javanet.staxutils.IndentingXMLStreamWriter;
+import org.apache.nifi.NullSuppression;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.schema.access.SchemaAccessWriter;
+import org.apache.nifi.serialization.AbstractRecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.WriteResult;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RawRecordWriter;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordField;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.serialization.record.type.ArrayDataType;
+import org.apache.nifi.serialization.record.type.ChoiceDataType;
+import org.apache.nifi.serialization.record.type.MapDataType;
+import org.apache.nifi.serialization.record.type.RecordDataType;
+import org.apache.nifi.serialization.record.util.DataTypeUtils;
+
+import javax.xml.stream.XMLOutputFactory;
+import javax.xml.stream.XMLStreamException;
+import javax.xml.stream.XMLStreamWriter;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.text.DateFormat;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Supplier;
+
+
+public class WriteXMLResult extends AbstractRecordSetWriter implements RecordSetWriter, RawRecordWriter {
+
+ final ComponentLog logger;
+ final RecordSchema recordSchema;
+ final SchemaAccessWriter schemaAccess;
+ final XMLStreamWriter writer;
+ final NullSuppression nullSuppression;
+ final ArrayWrapping arrayWrapping;
+ final String arrayTagName;
+ final String recordTagName;
+ final String rootTagName;
+
+ private final Supplier<DateFormat> LAZY_DATE_FORMAT;
+ private final Supplier<DateFormat> LAZY_TIME_FORMAT;
+ private final Supplier<DateFormat> LAZY_TIMESTAMP_FORMAT;
+
+ public WriteXMLResult(final ComponentLog logger, final RecordSchema recordSchema, final SchemaAccessWriter schemaAccess, final OutputStream out, final boolean prettyPrint,
+ final NullSuppression nullSuppression, final ArrayWrapping arrayWrapping, final String arrayTagName, final String rootTagName, final String recordTagName,
+ final String dateFormat, final String timeFormat, final String timestampFormat) throws IOException {
+
+ super(out);
+
+ this.logger = logger;
+ this.recordSchema = recordSchema;
+ this.schemaAccess = schemaAccess;
+ this.nullSuppression = nullSuppression;
+
+ this.arrayWrapping = arrayWrapping;
+ this.arrayTagName = arrayTagName;
+
+ this.rootTagName = rootTagName;
+ this.recordTagName = recordTagName;
+
+ final DateFormat df = dateFormat == null ? null : DataTypeUtils.getDateFormat(dateFormat);
+ final DateFormat tf = timeFormat == null ? null : DataTypeUtils.getDateFormat(timeFormat);
+ final DateFormat tsf = timestampFormat == null ? null : DataTypeUtils.getDateFormat(timestampFormat);
+
+ LAZY_DATE_FORMAT = () -> df;
+ LAZY_TIME_FORMAT = () -> tf;
+ LAZY_TIMESTAMP_FORMAT = () -> tsf;
+
+ try {
+ XMLOutputFactory factory = XMLOutputFactory.newInstance();
+
+ if (prettyPrint) {
+ writer = new IndentingXMLStreamWriter(factory.createXMLStreamWriter(out));
+ } else {
+ writer = factory.createXMLStreamWriter(out);
+ }
+
+ } catch (XMLStreamException e) {
+ throw new IOException(e.getMessage());
+ }
+ }
+
+ @Override
+ protected void onBeginRecordSet() throws IOException {
+
+ final OutputStream out = getOutputStream();
+ schemaAccess.writeHeader(recordSchema, out);
+
+ try {
+ writer.writeStartDocument();
+
+ writer.writeStartElement(rootTagName);
+
+ } catch (XMLStreamException e) {
+ throw new IOException(e.getMessage());
+ }
+ }
+
+ @Override
+ protected Map<String, String> onFinishRecordSet() throws IOException {
+
+ try {
+ writer.writeEndElement();
+ writer.writeEndDocument();
+ } catch (XMLStreamException e) {
+ throw new IOException(e.getMessage());
+ }
+ return schemaAccess.getAttributes(recordSchema);
+ }
+
+ @Override
+ public void close() throws IOException {
+
+ try {
+ writer.close();
+
+ } catch (XMLStreamException e) {
+ throw new IOException(e.getMessage());
+ }
+
+ super.close();
+ }
+
+ @Override
+ public void flush() throws IOException {
+
+ try {
+ writer.flush();
+
+ } catch (XMLStreamException e) {
+ throw new IOException(e.getMessage());
+ }
+ }
+
+ @Override
+ protected Map<String, String> writeRecord(Record record) throws IOException {
+
+ if (!isActiveRecordSet()) {
+ schemaAccess.writeHeader(recordSchema, getOutputStream());
+ }
+
+ List<String> tagsToOpen = new ArrayList<>();
--- End diff --
I need LIFO for removing items, but FIFO for flushing the list. Is there a collection supporting both?
---
[GitHub] nifi issue #2675: NIFI-5113 Add XMLRecordSetWriter
Posted by markap14 <gi...@git.apache.org>.
Github user markap14 commented on the issue:
https://github.com/apache/nifi/pull/2675
Thanks for updating @JohannesDaniel! All looks good to me at this point, so I've merged to master. Many thanks for this contrib as well as the XML Reader!
---
[GitHub] nifi pull request #2675: NIFI-5113 Add XMLRecordSetWriter
Posted by markap14 <gi...@git.apache.org>.
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2675#discussion_r187377388
--- Diff: nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/test/java/org/apache/nifi/xml/TestWriteXMLResultUtils.java ---
@@ -0,0 +1,472 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.xml;
+
+import org.apache.nifi.serialization.SimpleRecordSchema;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.ListRecordSet;
+import org.apache.nifi.serialization.record.MapRecord;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordField;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.serialization.record.RecordSet;
+import org.apache.nifi.serialization.record.SchemaIdentifier;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+public class TestWriteXMLResultUtils {
+
+ protected static final String FIELD_NAME = "NAME";
+ protected static final String FIELD_AGE = "AGE";
+ protected static final String FIELD_COUNTRY = "COUNTRY";
+ protected static final String FIELD_ADDRESS = "ADDRESS";
+ protected static final String FIELD_STREET = "STREET";
+ protected static final String FIELD_CITY = "CITY";
+ protected static final String FIELD_CHILDREN = "CHILDREN";
+
+ protected static Map<String,Object> RECORD_FIELDS_PERSON_1 = new HashMap<>();
+ protected static Map<String,Object> RECORD_FIELDS_PERSON_2 = new HashMap<>();
+ protected static Map<String,Object> RECORD_FIELDS_ADDRESS_1 = new HashMap<>();
+ protected static Map<String,Object> RECORD_FIELDS_ADDRESS_2 = new HashMap<>();
+
+ protected static Object[] ARRAY_CHILDREN = {"Tom", "Anna", "Ben"};
+ protected static Object[] ARRAY_CHILDREN_WITH_NULL_VALUE = {"Tom", null, "Ben"};
+ protected static Object[] ARRAY_CHILDREN_ONLY_NULL_VALUES = {null, null, null};
+
+ static {
+ RECORD_FIELDS_PERSON_1.put(FIELD_NAME, "Cleve Butler");
+ RECORD_FIELDS_PERSON_1.put(FIELD_AGE, 42);
+ RECORD_FIELDS_PERSON_1.put(FIELD_COUNTRY, "USA");
+ RECORD_FIELDS_PERSON_2.put(FIELD_NAME, "Ainslie Fletcher");
+ RECORD_FIELDS_PERSON_2.put(FIELD_AGE, 33);
+ RECORD_FIELDS_PERSON_2.put(FIELD_COUNTRY, "UK");
+ RECORD_FIELDS_ADDRESS_1.put(FIELD_STREET, "292 West Street");
+ RECORD_FIELDS_ADDRESS_1.put(FIELD_CITY, "Jersey City");
+ RECORD_FIELDS_ADDRESS_2.put(FIELD_STREET, "123 6th St.");
+ RECORD_FIELDS_ADDRESS_2.put(FIELD_CITY, "Seattle");
+
+ RECORD_FIELDS_PERSON_1 = Collections.unmodifiableMap(RECORD_FIELDS_PERSON_1);
+ RECORD_FIELDS_PERSON_2 = Collections.unmodifiableMap(RECORD_FIELDS_PERSON_2);
+ RECORD_FIELDS_ADDRESS_1 = Collections.unmodifiableMap(RECORD_FIELDS_ADDRESS_1);
+ RECORD_FIELDS_ADDRESS_2 = Collections.unmodifiableMap(RECORD_FIELDS_ADDRESS_2);
+ }
+
+ protected static final SchemaIdentifier SCHEMA_IDENTIFIER_PERSON = SchemaIdentifier.builder().name("PERSON").id(0L).version(0).build();
+ protected static final SchemaIdentifier SCHEMA_IDENTIFIER_RECORD = SchemaIdentifier.builder().name("RECORD").id(0L).version(0).build();
+ // protected static final SchemaIdentifier SCHEMA_IDENTIFIER_ADDRESS = SchemaIdentifier.builder().name("ADDRESS").id(0L).version(0).build();
--- End diff --
Should remove this
---
[GitHub] nifi pull request #2675: NIFI-5113 Add XMLRecordSetWriter
Posted by JohannesDaniel <gi...@git.apache.org>.
Github user JohannesDaniel commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2675#discussion_r186106598
--- Diff: nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/xml/WriteXMLResult.java ---
@@ -0,0 +1,602 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.xml;
+
+import javanet.staxutils.IndentingXMLStreamWriter;
+import org.apache.nifi.NullSuppression;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.schema.access.SchemaAccessWriter;
+import org.apache.nifi.serialization.AbstractRecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.WriteResult;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RawRecordWriter;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordField;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.serialization.record.type.ArrayDataType;
+import org.apache.nifi.serialization.record.type.ChoiceDataType;
+import org.apache.nifi.serialization.record.type.MapDataType;
+import org.apache.nifi.serialization.record.type.RecordDataType;
+import org.apache.nifi.serialization.record.util.DataTypeUtils;
+
+import javax.xml.stream.XMLOutputFactory;
+import javax.xml.stream.XMLStreamException;
+import javax.xml.stream.XMLStreamWriter;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.text.DateFormat;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Supplier;
+
+
+public class WriteXMLResult extends AbstractRecordSetWriter implements RecordSetWriter, RawRecordWriter {
+
+ final ComponentLog logger;
+ final RecordSchema recordSchema;
+ final SchemaAccessWriter schemaAccess;
+ final XMLStreamWriter writer;
+ final NullSuppression nullSuppression;
+ final ArrayWrapping arrayWrapping;
+ final String arrayTagName;
+ final String recordTagName;
+ final String rootTagName;
+
+ private final Supplier<DateFormat> LAZY_DATE_FORMAT;
+ private final Supplier<DateFormat> LAZY_TIME_FORMAT;
+ private final Supplier<DateFormat> LAZY_TIMESTAMP_FORMAT;
+
+ public WriteXMLResult(final ComponentLog logger, final RecordSchema recordSchema, final SchemaAccessWriter schemaAccess, final OutputStream out, final boolean prettyPrint,
+ final NullSuppression nullSuppression, final ArrayWrapping arrayWrapping, final String arrayTagName, final String rootTagName, final String recordTagName,
+ final String dateFormat, final String timeFormat, final String timestampFormat) throws IOException {
+
+ super(out);
+
+ this.logger = logger;
+ this.recordSchema = recordSchema;
+ this.schemaAccess = schemaAccess;
+ this.nullSuppression = nullSuppression;
+
+ this.arrayWrapping = arrayWrapping;
+ this.arrayTagName = arrayTagName;
+
+ this.rootTagName = rootTagName;
+ this.recordTagName = recordTagName;
+
+ final DateFormat df = dateFormat == null ? null : DataTypeUtils.getDateFormat(dateFormat);
+ final DateFormat tf = timeFormat == null ? null : DataTypeUtils.getDateFormat(timeFormat);
+ final DateFormat tsf = timestampFormat == null ? null : DataTypeUtils.getDateFormat(timestampFormat);
+
+ LAZY_DATE_FORMAT = () -> df;
+ LAZY_TIME_FORMAT = () -> tf;
+ LAZY_TIMESTAMP_FORMAT = () -> tsf;
+
+ try {
+ XMLOutputFactory factory = XMLOutputFactory.newInstance();
--- End diff --
should I consider the encoding somehow (e. g. via property)?
e. g. writer = factory.createXMLStreamWriter(out, StandardCharsets.UTF_8.name());
---
[GitHub] nifi pull request #2675: NIFI-5113 Add XMLRecordSetWriter
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2675#discussion_r187770744
--- Diff: nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/xml/XMLRecordSetWriter.java ---
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.xml;
+
+import org.apache.nifi.NullSuppression;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.serialization.DateTimeTextRecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+
+@Tags({"xml", "resultset", "writer", "serialize", "record", "recordset", "row"})
+@CapabilityDescription("Writes a RecordSet to XML. The records are wrapped by a root tag.")
+public class XMLRecordSetWriter extends DateTimeTextRecordSetWriter implements RecordSetWriterFactory {
+
+ public static final AllowableValue ALWAYS_SUPPRESS = new AllowableValue("always-suppress", "Always Suppress",
+ "Fields that are missing (present in the schema but not in the record), or that have a value of null, will not be written out");
+ public static final AllowableValue NEVER_SUPPRESS = new AllowableValue("never-suppress", "Never Suppress",
+ "Fields that are missing (present in the schema but not in the record), or that have a value of null, will be written out as a null value");
+ public static final AllowableValue SUPPRESS_MISSING = new AllowableValue("suppress-missing", "Suppress Missing Values",
+ "When a field has a value of null, it will be written out. However, if a field is defined in the schema and not present in the record, the field will not be written out.");
+
+ public static final AllowableValue USE_PROPERTY_AS_WRAPPER = new AllowableValue("use-property-as-wrapper", "Use Property as Wrapper",
+ "The value of the property \"Array Tag Name\" will be used as the tag name to wrap elements of an array. The field name of the array field will be used for the tag name " +
+ "of the elements.");
+ public static final AllowableValue USE_PROPERTY_FOR_ELEMENTS = new AllowableValue("use-property-for-elements", "Use Property for Elements",
+ "The value of the property \"Array Tag Name\" will be used for the tag name of the elements of an array. The field name of the array field will be used as the tag name " +
+ "to wrap elements.");
+ public static final AllowableValue NO_WRAPPING = new AllowableValue("no-wrapping", "No Wrapping",
+ "The elements of an array will not be wrapped");
+
+ public static final PropertyDescriptor SUPPRESS_NULLS = new PropertyDescriptor.Builder()
+ .name("suppress_nulls")
+ .displayName("Suppress Null Values")
+ .description("Specifies how the writer should handle a null field")
+ .allowableValues(NEVER_SUPPRESS, ALWAYS_SUPPRESS, SUPPRESS_MISSING)
+ .defaultValue(NEVER_SUPPRESS.getValue())
+ .required(true)
+ .build();
+
+ public static final PropertyDescriptor PRETTY_PRINT_XML = new PropertyDescriptor.Builder()
+ .name("pretty_print_xml")
+ .displayName("Pretty Print XML")
+ .description("Specifies whether or not the XML should be pretty printed")
+ .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+ .allowableValues("true", "false")
+ .defaultValue("false")
+ .required(true)
+ .build();
+
--- End diff --
If you want to add it, @mattyb149 did an API update for the DBCPService that [shows how](https://github.com/apache/nifi/commit/099bfcdf3a5873a311312eb7e9e85b7b22ef1b98). I would consider it optional, though, because AFAIK the issue hasn't been given much discussion.
---
[GitHub] nifi pull request #2675: NIFI-5113 Add XMLRecordSetWriter
Posted by JohannesDaniel <gi...@git.apache.org>.
Github user JohannesDaniel commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2675#discussion_r186106245
--- Diff: nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/xml/XMLRecordSetWriter.java ---
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.xml;
+
+import org.apache.nifi.NullSuppression;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.serialization.DateTimeTextRecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+
+@Tags({"xml", "resultset", "writer", "serialize", "record", "recordset", "row"})
+@CapabilityDescription("Writes a RecordSet to XML. The records are wrapped by a root tag.")
+public class XMLRecordSetWriter extends DateTimeTextRecordSetWriter implements RecordSetWriterFactory {
+
+ public static final AllowableValue ALWAYS_SUPPRESS = new AllowableValue("always-suppress", "Always Suppress",
+ "Fields that are missing (present in the schema but not in the record), or that have a value of null, will not be written out");
+ public static final AllowableValue NEVER_SUPPRESS = new AllowableValue("never-suppress", "Never Suppress",
+ "Fields that are missing (present in the schema but not in the record), or that have a value of null, will be written out as a null value");
+ public static final AllowableValue SUPPRESS_MISSING = new AllowableValue("suppress-missing", "Suppress Missing Values",
+ "When a field has a value of null, it will be written out. However, if a field is defined in the schema and not present in the record, the field will not be written out.");
+
+ public static final AllowableValue USE_PROPERTY_AS_WRAPPER = new AllowableValue("use-property-as-wrapper", "Use Property as Wrapper",
+ "The value of the property \"Array Tag Name\" will be used as the tag name to wrap elements of an array. The field name of the array field will be used for the tag name " +
+ "of the elements.");
+ public static final AllowableValue USE_PROPERTY_FOR_ELEMENTS = new AllowableValue("use-property-for-elements", "Use Property for Elements",
+ "The value of the property \"Array Tag Name\" will be used for the tag name of the elements of an array. The field name of the array field will be used as the tag name " +
+ "to wrap elements.");
+ public static final AllowableValue NO_WRAPPING = new AllowableValue("no-wrapping", "No Wrapping",
+ "The elements of an array will not be wrapped");
+
+ public static final PropertyDescriptor SUPPRESS_NULLS = new PropertyDescriptor.Builder()
+ .name("suppress_nulls")
+ .displayName("Suppress Null Values")
+ .description("Specifies how the writer should handle a null field")
+ .allowableValues(NEVER_SUPPRESS, ALWAYS_SUPPRESS, SUPPRESS_MISSING)
+ .defaultValue(NEVER_SUPPRESS.getValue())
+ .required(true)
+ .build();
+
+ public static final PropertyDescriptor PRETTY_PRINT_XML = new PropertyDescriptor.Builder()
+ .name("pretty_print_xml")
+ .displayName("Pretty Print XML")
+ .description("Specifies whether or not the XML should be pretty printed")
+ .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+ .allowableValues("true", "false")
+ .defaultValue("false")
+ .required(true)
+ .build();
+
+ public static final PropertyDescriptor ROOT_TAG_NAME = new PropertyDescriptor.Builder()
+ .name("root_tag_name")
+ .displayName("Name of Root Tag")
+ .description("Specifies the name of the XML root tag wrapping the record set")
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+ .defaultValue("root")
+ .required(true)
+ .build();
+
--- End diff --
Initially, I planned to use the record name of the schema for this using
{code}
recordSchema.getIdentifier().getName().get()
{code}
but for my test cases (add schema as text / attribute) this was null.
---
[GitHub] nifi pull request #2675: NIFI-5113 Add XMLRecordSetWriter
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2675#discussion_r188055672
--- Diff: nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/xml/XMLRecordSetWriter.java ---
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.xml;
+
+import org.apache.nifi.NullSuppression;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.serialization.DateTimeTextRecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+
+@Tags({"xml", "resultset", "writer", "serialize", "record", "recordset", "row"})
+@CapabilityDescription("Writes a RecordSet to XML. The records are wrapped by a root tag.")
+public class XMLRecordSetWriter extends DateTimeTextRecordSetWriter implements RecordSetWriterFactory {
+
+ public static final AllowableValue ALWAYS_SUPPRESS = new AllowableValue("always-suppress", "Always Suppress",
+ "Fields that are missing (present in the schema but not in the record), or that have a value of null, will not be written out");
+ public static final AllowableValue NEVER_SUPPRESS = new AllowableValue("never-suppress", "Never Suppress",
+ "Fields that are missing (present in the schema but not in the record), or that have a value of null, will be written out as a null value");
+ public static final AllowableValue SUPPRESS_MISSING = new AllowableValue("suppress-missing", "Suppress Missing Values",
+ "When a field has a value of null, it will be written out. However, if a field is defined in the schema and not present in the record, the field will not be written out.");
+
+ public static final AllowableValue USE_PROPERTY_AS_WRAPPER = new AllowableValue("use-property-as-wrapper", "Use Property as Wrapper",
+ "The value of the property \"Array Tag Name\" will be used as the tag name to wrap elements of an array. The field name of the array field will be used for the tag name " +
+ "of the elements.");
+ public static final AllowableValue USE_PROPERTY_FOR_ELEMENTS = new AllowableValue("use-property-for-elements", "Use Property for Elements",
+ "The value of the property \"Array Tag Name\" will be used for the tag name of the elements of an array. The field name of the array field will be used as the tag name " +
+ "to wrap elements.");
+ public static final AllowableValue NO_WRAPPING = new AllowableValue("no-wrapping", "No Wrapping",
+ "The elements of an array will not be wrapped");
+
+ public static final PropertyDescriptor SUPPRESS_NULLS = new PropertyDescriptor.Builder()
+ .name("suppress_nulls")
+ .displayName("Suppress Null Values")
+ .description("Specifies how the writer should handle a null field")
+ .allowableValues(NEVER_SUPPRESS, ALWAYS_SUPPRESS, SUPPRESS_MISSING)
+ .defaultValue(NEVER_SUPPRESS.getValue())
+ .required(true)
+ .build();
+
+ public static final PropertyDescriptor PRETTY_PRINT_XML = new PropertyDescriptor.Builder()
+ .name("pretty_print_xml")
+ .displayName("Pretty Print XML")
+ .description("Specifies whether or not the XML should be pretty printed")
+ .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+ .allowableValues("true", "false")
+ .defaultValue("false")
+ .required(true)
+ .build();
+
+ public static final PropertyDescriptor ROOT_TAG_NAME = new PropertyDescriptor.Builder()
+ .name("root_tag_name")
+ .displayName("Name of Root Tag")
+ .description("Specifies the name of the XML root tag wrapping the record set")
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+ .defaultValue("root")
+ .required(true)
+ .build();
+
--- End diff --
@JohannesDaniel @markap14 I just merged the linked ticket. You can rebase now.
---
[GitHub] nifi pull request #2675: NIFI-5113 Add XMLRecordSetWriter
Posted by markap14 <gi...@git.apache.org>.
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2675#discussion_r187361310
--- Diff: nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/xml/WriteXMLResult.java ---
@@ -0,0 +1,602 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.xml;
+
+import javanet.staxutils.IndentingXMLStreamWriter;
+import org.apache.nifi.NullSuppression;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.schema.access.SchemaAccessWriter;
+import org.apache.nifi.serialization.AbstractRecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.WriteResult;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RawRecordWriter;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordField;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.serialization.record.type.ArrayDataType;
+import org.apache.nifi.serialization.record.type.ChoiceDataType;
+import org.apache.nifi.serialization.record.type.MapDataType;
+import org.apache.nifi.serialization.record.type.RecordDataType;
+import org.apache.nifi.serialization.record.util.DataTypeUtils;
+
+import javax.xml.stream.XMLOutputFactory;
+import javax.xml.stream.XMLStreamException;
+import javax.xml.stream.XMLStreamWriter;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.text.DateFormat;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Supplier;
+
+
+public class WriteXMLResult extends AbstractRecordSetWriter implements RecordSetWriter, RawRecordWriter {
+
+ final ComponentLog logger;
+ final RecordSchema recordSchema;
+ final SchemaAccessWriter schemaAccess;
+ final XMLStreamWriter writer;
+ final NullSuppression nullSuppression;
+ final ArrayWrapping arrayWrapping;
+ final String arrayTagName;
+ final String recordTagName;
+ final String rootTagName;
+
+ private final Supplier<DateFormat> LAZY_DATE_FORMAT;
+ private final Supplier<DateFormat> LAZY_TIME_FORMAT;
+ private final Supplier<DateFormat> LAZY_TIMESTAMP_FORMAT;
+
+ public WriteXMLResult(final ComponentLog logger, final RecordSchema recordSchema, final SchemaAccessWriter schemaAccess, final OutputStream out, final boolean prettyPrint,
+ final NullSuppression nullSuppression, final ArrayWrapping arrayWrapping, final String arrayTagName, final String rootTagName, final String recordTagName,
+ final String dateFormat, final String timeFormat, final String timestampFormat) throws IOException {
+
+ super(out);
+
+ this.logger = logger;
+ this.recordSchema = recordSchema;
+ this.schemaAccess = schemaAccess;
+ this.nullSuppression = nullSuppression;
+
+ this.arrayWrapping = arrayWrapping;
+ this.arrayTagName = arrayTagName;
+
+ this.rootTagName = rootTagName;
+ this.recordTagName = recordTagName;
+
+ final DateFormat df = dateFormat == null ? null : DataTypeUtils.getDateFormat(dateFormat);
+ final DateFormat tf = timeFormat == null ? null : DataTypeUtils.getDateFormat(timeFormat);
+ final DateFormat tsf = timestampFormat == null ? null : DataTypeUtils.getDateFormat(timestampFormat);
+
+ LAZY_DATE_FORMAT = () -> df;
+ LAZY_TIME_FORMAT = () -> tf;
+ LAZY_TIMESTAMP_FORMAT = () -> tsf;
+
+ try {
+ XMLOutputFactory factory = XMLOutputFactory.newInstance();
+
+ if (prettyPrint) {
+ writer = new IndentingXMLStreamWriter(factory.createXMLStreamWriter(out));
+ } else {
+ writer = factory.createXMLStreamWriter(out);
+ }
+
+ } catch (XMLStreamException e) {
+ throw new IOException(e.getMessage());
+ }
+ }
+
+ @Override
+ protected void onBeginRecordSet() throws IOException {
+
+ final OutputStream out = getOutputStream();
+ schemaAccess.writeHeader(recordSchema, out);
+
+ try {
+ writer.writeStartDocument();
+
+ writer.writeStartElement(rootTagName);
+
+ } catch (XMLStreamException e) {
+ throw new IOException(e.getMessage());
+ }
+ }
+
+ @Override
+ protected Map<String, String> onFinishRecordSet() throws IOException {
+
+ try {
+ writer.writeEndElement();
+ writer.writeEndDocument();
+ } catch (XMLStreamException e) {
+ throw new IOException(e.getMessage());
+ }
+ return schemaAccess.getAttributes(recordSchema);
+ }
+
+ @Override
+ public void close() throws IOException {
+
+ try {
+ writer.close();
+
+ } catch (XMLStreamException e) {
+ throw new IOException(e.getMessage());
+ }
+
+ super.close();
+ }
+
+ @Override
+ public void flush() throws IOException {
+
+ try {
+ writer.flush();
+
+ } catch (XMLStreamException e) {
+ throw new IOException(e.getMessage());
+ }
+ }
+
+ @Override
+ protected Map<String, String> writeRecord(Record record) throws IOException {
+
+ if (!isActiveRecordSet()) {
+ schemaAccess.writeHeader(recordSchema, getOutputStream());
+ }
+
+ List<String> tagsToOpen = new ArrayList<>();
--- End diff --
Given how this is used, I wonder if it shouldn't instead be a Stack<String> ?
---