[GitHub] [nifi] exceptionfactory commented on a diff in pull request #7194: NIFI-11167 - Add Excel Record Reader

["exceptionfactory (via GitHub)" <gi...@apache.org>]

exceptionfactory commented on code in PR #7194:
URL: https://github.com/apache/nifi/pull/7194#discussion_r1218310754


##########
nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/excel/ExcelSchemaInference.java:
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.RecordSource;
+import org.apache.nifi.schema.inference.SchemaInferenceEngine;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.SimpleRecordSchema;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordField;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.util.SchemaInferenceUtil;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.DataFormatter;
+import org.apache.poi.ss.usermodel.Row;
+
+import java.io.IOException;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+public class ExcelSchemaInference implements SchemaInferenceEngine<Row> {
+    private final TimeValueInference timeValueInference;
+    private final DataFormatter dataFormatter;
+
+    public ExcelSchemaInference(TimeValueInference timeValueInference) {
+        this(timeValueInference, null);
+    }
+
+    public ExcelSchemaInference(TimeValueInference timeValueInference, Locale locale) {
+        this.timeValueInference = timeValueInference;
+        this.dataFormatter = locale == null ? new DataFormatter() : new DataFormatter(locale);
+    }
+
+    @Override
+    public RecordSchema inferSchema(RecordSource<Row> recordSource) throws IOException {
+        final Map<String, FieldTypeInference> typeMap = new LinkedHashMap<>();
+        Row row;
+        while((row = recordSource.next()) != null) {
+            inferSchema(row, typeMap);
+        }
+        return createSchema(typeMap);
+    }
+
+    private void inferSchema(final Row row, final Map<String, FieldTypeInference> typeMap) {
+        if (ExcelUtils.hasCells(row)) {
+            IntStream.range(0, row.getLastCellNum())
+                    .forEach(index -> {
+                        final Cell cell = row.getCell(index);
+                        final String fieldName = Integer.toString(index);

Review Comment:
   What do you think of prefixing this field name with `column_`? So the first field name would be `column_0`? That would be easier to work with in absence of a defined schema.



##########
nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/excel/RowIterator.java:
##########
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import com.github.pjfanning.xlsx.StreamingReader;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.poi.ss.usermodel.Row;
+import org.apache.poi.ss.usermodel.Sheet;
+import org.apache.poi.ss.usermodel.Workbook;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+public class RowIterator implements Iterator<Row>, Closeable {
+    private final Workbook workbook;
+    private final Iterator<Sheet> sheets;
+    private final Map<String, Boolean> requiredSheets;
+    private final int firstRow;
+    private final ComponentLog logger;
+    private Sheet currentSheet;
+    private Iterator<Row> currentRows;
+    private Row currentRow;
+
+    public RowIterator(InputStream in, List<String> requiredSheets, int firstRow, ComponentLog logger) {
+        this.workbook = StreamingReader.builder()
+                .rowCacheSize(100)
+                .bufferSize(4096)
+                .open(in);
+        this.sheets = this.workbook.iterator();
+        this.requiredSheets = requiredSheets != null ? requiredSheets.stream()
+                .collect(Collectors.toMap(key -> key, value -> Boolean.FALSE)) : new HashMap<>();
+        this.firstRow = firstRow;
+        this.logger = logger;
+    }
+
+    @Override
+    public boolean hasNext() {
+        setCurrent();

Review Comment:
   This approach requires calling `hasNext()` before calling `next()`, which may be conventional, but should not be required. Instead, the next row should be set after calling `next()` because that method should move the iterator forward.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org