You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "exceptionfactory (via GitHub)" <gi...@apache.org> on 2024/10/11 15:49:13 UTC
[PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
exceptionfactory opened a new pull request, #9379:
URL: https://github.com/apache/nifi/pull/9379
# Summary
[NIFI-13744](https://issues.apache.org/jira/browse/NIFI-13744) Corrects Excel Reader type inferencing with Record Data Type detection based on spreadsheet cell properties.
The `CellType` provides the general shape of the value, but requires additional evaluation of `NUMERIC` and `STRING` types to determine whether the field could representation a timestamp, based on Cell Style.
The new `CellFieldTypeReader` replaces existing use of the `SchemaInferenceUtil`, which is limited to `String` input values, requiring value formatting. The standard implementation avoids cell value formatting in most cases, instead relying type detection. The POI library can return timestamp values in several forms, including `java.util.Date` and `java.time.LocalDateTime`, but it does not differentiate between values that contain only dates or only times. For these scenarios, using a defined schema for the Excel Reader provides better resolution.
# Tracking
Please complete the following tracking steps prior to pull request creation.
### Issue Tracking
- [X] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created
### Pull Request Tracking
- [X] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-00000`
- [X] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-00000`
### Pull Request Formatting
- [X] Pull Request based on current revision of the `main` branch
- [X] Pull Request refers to a feature branch with one commit containing changes
# Verification
Please indicate the verification steps performed prior to pull request creation.
### Build
- [X] Build completed using `mvn clean install -P contrib-check`
- [X] JDK 21
### Licensing
- [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html)
- [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files
### Documentation
- [ ] Documentation formatting appears as expected in rendered files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on PR #9379:
URL: https://github.com/apache/nifi/pull/9379#issuecomment-2417276756
Thanks @dan-s1! The automated builds for the backported pull request #9397 are also close to completion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1799738254
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getStringCellValue();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
Review Comment:
I just stored a date in an Excel spreadsheet as a text e.g. 5/21/2024 so I see what you mean.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1799954561
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/ExcelHeaderSchemaStrategy.java:
##########
@@ -55,18 +52,18 @@ public class ExcelHeaderSchemaStrategy implements SchemaAccessStrategy {
private final PropertyContext context;
private final ComponentLog logger;
- private final TimeValueInference timeValueInference;
+ private final CellFieldTypeReader cellFieldTypeReader;
private final DataFormatter dataFormatter;
public ExcelHeaderSchemaStrategy(PropertyContext context, ComponentLog logger, TimeValueInference timeValueInference, Locale locale) {
this.context = context;
this.logger = logger;
- this.timeValueInference = timeValueInference;
+ this.cellFieldTypeReader = new StandardCellFieldTypeReader(timeValueInference);
this.dataFormatter = locale == null ? new DataFormatter() : new DataFormatter(locale);
Review Comment:
I do not think the Locale argument is needed any longer as we can instantiate `DataFormatter` as follows:
```suggestion
this.dataFormatter = new DataFormatter(Locale.getDefault());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1801472367
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ /**
+ * Standard Cell Field Type Reader constructor with Time Value Inference for handling STRING Cell Types that may contain values with Timestamps
+ *
+ * @param timeValueInference Time Value Inference required for STRING Cell Type evaluation
+ */
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell can be null
+ * @param fieldName Cell field name for tracking in Field Types required
+ * @param fieldTypes Map of Field Name to Field Type Inference information required
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell can be null
+ * @return Record Data Type or null
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ if (cell == null) {
+ return null;
+ }
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getStringCellValue();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
+ dataType = timeDataType.orElse(RecordFieldType.STRING.getDataType());
+ } else if (CellType.FORMULA == cellType) {
+ dataType = getFormulaResultDataType(cell);
+ } else {
+ // Default to null for known and unknown Cell Types such as ERROR
Review Comment:
```suggestion
// Default to null for known and unknown Cell Types such as BLANK, ERROR
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1801634966
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getStringCellValue();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
Review Comment:
Reviewing the current `additionalDetails.md` documentation, the description actually seems to reflect the behavior accurately:
> If the "Time Format," "Timestamp Format," or "Date Format" properties are configured, any value that would otherwise be considered a STRING type is first checked against the configured formats to see if it matches any of them. If the value matches the Timestamp Format, the value is considered a Timestamp field. If it matches the Date Format, it is considered a Date field. If it matches the Time Format, it is considered a Time field. In the unlikely event that the value matches more than one of the configured formats, they will be matched in the order: Timestamp, Date, Time. I.e., if a value matched both the Timestamp Format and the Date Format, the type that is inferred will be Timestamp. Because parsing dates and times can be expensive, it is advisable not to configure these formats if dates, times, and timestamps are not expected, or if processing the data as a STRING is acceptable. For use cases when this is important, though, the inference engine is intelligent enough to opt
imize the parsing by first checking several very cheap conditions. For example, the string's length is examined to see if it is too long or too short to match the pattern. This results in far more efficient processing than would result if attempting to parse each string value as a timestamp.
I'm not sure how valuable it is to go into some additional implementation details at the moment, if future revisions are necessary. However, if you have a suggestion for an additional bullet point, glad to include.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on PR #9379:
URL: https://github.com/apache/nifi/pull/9379#issuecomment-2414617124
@exceptionfactory I just found this helpful [tutorial](https://www.myonlinetraininghub.com/excel-date-and-time#:~:text=When%20you%20look%20at%20a,decimal%20portion%20is%20the%20time.) which explains how dates and times are stored in Excel. Based on this information I believe we can provide better inferences when the cell type is numeric and is date formatted. If its a whole number we know its date, if its a fraction we know its a time and if its a whole number with a fraction then its a timestamp. Hence lines 81-84 in `StandardCellFieldTypeReader` can be expanded to incorporate this logic to identify the most precise type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1800085624
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
Review Comment:
I do not think defaulting to `TIMESTAMP` is right for every case. In the `TestExcelReader` I have in my PR I have three columns
1. `Date_Custom` which has a date value 07/25/2024 (displayed as `07/25/24`)
2. `Time_Custom` which has a time value of 04:00:00 PM (displayed as `16:00`)
3. `Timestamp` which has a timestamp 07/26/2024 16:00:00
When running through your code they are parsed and look like the following when calling `toString` on the `java.sql.Timestamp` returned from the `Record` object.
1. 2024-07-05 00:00:00.0
2. 1899-12-31 00:00:00.0
3. 2024-07-26 16:00:00.0
For dates and timestamp 1 and 3 are okay but for those fields which only represent time, I do not think users are interested in the default date Excel has for time 1899-12-31 (and it would seem the actual time is lost) . I think there has to be a more fine grained way of determining whether the date we have is representing a timestamp or only time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1800085624
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
Review Comment:
I do not think defaulting to `TIMESTAMP` is right for every case. In the `TestExcelReader` I have in my PR I have three columns
1. `Date_Custom` which has a date value 07/25/2024 (displayed as `07/25/24`)
2. `Time_Custom` which has a time value of 04:00:00 PM (displayed as `16:00`)
3. `Timestamp` which has a timestamp 07/26/2024 16:00:00
When running through your code they are parsed and look like the following when calling `toString` on the `java.sql.Timestamp` returned from the `Record` object read.
1. 2024-07-05 00:00:00.0
2. 1899-12-31 00:00:00.0
3. 2024-07-26 16:00:00.0
For dates and timestamp 1 and 3 are okay but for those fields which only represent time, I do not think users are interested in the default date Excel has for time 1899-12-31. I think there has to a more fine grained way of determining whether the date we have is representing a date or only time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1802057764
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ /**
+ * Standard Cell Field Type Reader constructor with Time Value Inference for handling STRING Cell Types that may contain values with Timestamps
+ *
+ * @param timeValueInference Time Value Inference required for STRING Cell Type evaluation
+ */
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell can be null
+ * @param fieldName Cell field name for tracking in Field Types required
+ * @param fieldTypes Map of Field Name to Field Type Inference information required
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell can be null
+ * @return Record Data Type or null
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ if (cell == null) {
+ return null;
+ }
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ dataType = getDateTimeDataType(cell);
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getStringCellValue();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
+ dataType = timeDataType.orElse(RecordFieldType.STRING.getDataType());
+ } else if (CellType.FORMULA == cellType) {
+ dataType = getFormulaResultDataType(cell);
+ } else {
+ // Default to null for known and unknown Cell Types
+ dataType = null;
+ }
+
+ return dataType;
+ }
+
+ private DataType getDateTimeDataType(final Cell cell) {
+ final DataType dataType;
+
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ // Numbers without decimal fractions indicate Dates without Times
+ dataType = RecordFieldType.DATE.getDataType();
+ } else if (numericCellValue < 1) {
+ // Decimal fractions indicate Times without Dates
+ dataType = RecordFieldType.TIME.getDataType();
+ } else {
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ }
+
+ return dataType;
+ }
+
+ private DataType getFormulaResultDataType(final Cell cell) {
+ final DataType dataType;
+
+ final CellType formulaResultType = cell.getCachedFormulaResultType();
+ if (CellType.BOOLEAN == formulaResultType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == formulaResultType) {
+ dataType = RecordFieldType.STRING.getDataType();
+ } else if (CellType.NUMERIC == formulaResultType) {
+ // Date Formatting check limited to NUMERIC Cell Types without Conditional Formatting Evaluator
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
Review Comment:
I am not seeing anything directly, its probably best to leave as is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1799582387
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getStringCellValue();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
+ dataType = timeDataType.orElse(RecordFieldType.STRING.getDataType());
+ } else if (CellType.FORMULA == cellType) {
+ dataType = getFormulaResultDataType(cell);
+ } else {
+ // Default to null for known and unknown Cell Types: BLANK, ERROR
+ dataType = null;
+ }
+
+ return dataType;
+ }
+
+ private DataType getFormulaResultDataType(final Cell cell) {
+ final DataType dataType;
+
+ final CellType formulaResultType = cell.getCachedFormulaResultType();
+ if (CellType.BOOLEAN == formulaResultType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == formulaResultType) {
+ dataType = RecordFieldType.STRING.getDataType();
+ } else if (CellType.NUMERIC == formulaResultType) {
+ // Date Formatting check limited to NUMERIC Cell Types without Conditional Formatting Evaluator
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ } else {
+ // Default to null for known and unknown Formula Result Cell Types: BLANK, ERROR
+ dataType = null;
+ }
Review Comment:
I didn't realize null type is an option. Doesn't everything need to have an associated type?
Furthermore it is possible to to have a string value of the error which end users may or may not be interested in. I had in my PR two examples in file named `all_types.xlsx` in columns named `Variable_Missing` and `Syntax_Error` with displayed values `Err:511` and `#NAME?` . Hence perhaps ERROR should return `RecordFieldType.STRING.getDataType()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1799748937
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getStringCellValue();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
Review Comment:
I think though this piece should be documented that any time or date format properties specified by the user (stored in `TimeValueInference`) will **_only_** be used to parse those cells whose cell type is string and not whose already is date (for the end user this may be understood is if the Excel format type is text and not date).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1799986120
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/ExcelRecordReader.java:
##########
@@ -133,15 +134,32 @@ private Map<String, Object> getCurrentRowValues(Row currentRow, boolean coerceTy
return currentRowValues;
}
- private static Object getCellValue(Cell cell) {
- if (cell != null) {
- return switch (cell.getCellType()) {
- case _NONE, BLANK, ERROR, FORMULA, STRING -> cell.getStringCellValue();
+ private static Object getCellValue(final Cell cell) {
+ final Object cellValue;
+
+ if (cell == null) {
+ cellValue = null;
+ } else {
+ final CellType cellType = cell.getCellType();
+ cellValue = switch (cellType) {
+ case _NONE, BLANK, ERROR, STRING -> cell.getStringCellValue();
case NUMERIC -> DateUtil.isCellDateFormatted(cell) ? cell.getDateCellValue() : cell.getNumericCellValue();
case BOOLEAN -> cell.getBooleanCellValue();
+ case FORMULA -> getFormulaCellValue(cell);
};
}
- return null;
+
+ return cellValue;
+ }
+
+ private static Object getFormulaCellValue(final Cell cell) {
+ final CellType formulaResultType = cell.getCachedFormulaResultType();
+ return switch (formulaResultType) {
+ case BOOLEAN -> cell.getBooleanCellValue();
+ case STRING -> cell.getStringCellValue();
+ case NUMERIC -> DateUtil.isCellDateFormatted(cell) ? cell.getDateCellValue() : cell.getNumericCellValue();
+ default -> null;
Review Comment:
Good catch, that is worth retaining.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1799888718
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getStringCellValue();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
+ dataType = timeDataType.orElse(RecordFieldType.STRING.getDataType());
+ } else if (CellType.FORMULA == cellType) {
+ dataType = getFormulaResultDataType(cell);
+ } else {
+ // Default to null for known and unknown Cell Types: BLANK, ERROR
+ dataType = null;
+ }
+
+ return dataType;
+ }
+
+ private DataType getFormulaResultDataType(final Cell cell) {
+ final DataType dataType;
+
+ final CellType formulaResultType = cell.getCachedFormulaResultType();
+ if (CellType.BOOLEAN == formulaResultType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == formulaResultType) {
+ dataType = RecordFieldType.STRING.getDataType();
+ } else if (CellType.NUMERIC == formulaResultType) {
+ // Date Formatting check limited to NUMERIC Cell Types without Conditional Formatting Evaluator
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ } else {
+ // Default to null for known and unknown Formula Result Cell Types: BLANK, ERROR
Review Comment:
Per `getCachedFormulaResultType` [javadocs](https://poi.apache.org/apidocs/5.0/org/apache/poi/ss/usermodel/Cell.html#getCachedFormulaResultType--) there is no `BLANK` cell type for cached formula cell type
```suggestion
// Default to null for known and unknown Formula Result Cell Type: ERROR
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1799886701
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getStringCellValue();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
+ dataType = timeDataType.orElse(RecordFieldType.STRING.getDataType());
+ } else if (CellType.FORMULA == cellType) {
+ dataType = getFormulaResultDataType(cell);
+ } else {
+ // Default to null for known and unknown Cell Types: BLANK, ERROR
Review Comment:
Per `getCachedFormulaResultType` [javadocs](https://poi.apache.org/apidocs/5.0/org/apache/poi/ss/usermodel/Cell.html#getCachedFormulaResultType--) there is no BLANK cell type for formulas
```suggestion
// Default to null for known and unknown Cell Type: ERROR
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on PR #9379:
URL: https://github.com/apache/nifi/pull/9379#issuecomment-2414788368
I assume I need your latest changes to have the TIME type in order for the CSVWriter to pick that up with the time pattern.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on PR #9379:
URL: https://github.com/apache/nifi/pull/9379#issuecomment-2415234003
> @exceptionfactory Thank you for making all the requested changes. Also thank you for including me as a coauthor. Everything looks good now and seems to be producing the correct results. LGTM+1.
>
> I plan to merge as soon as the CI build completes.
Thanks @dan-s1, great collaborative effort on refining the behavior on the Excel Reader!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1801603367
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getStringCellValue();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
Review Comment:
Also it might be worth documenting that when inferring a schema, the schema will be limited to the following specific types BOOLEAN, CHOICE, DOUBLE, LONG, STRING, TIMESTAMP and perhaps DATE and TIME.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1802089085
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ /**
+ * Standard Cell Field Type Reader constructor with Time Value Inference for handling STRING Cell Types that may contain values with Timestamps
+ *
+ * @param timeValueInference Time Value Inference required for STRING Cell Type evaluation
+ */
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell can be null
+ * @param fieldName Cell field name for tracking in Field Types required
+ * @param fieldTypes Map of Field Name to Field Type Inference information required
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell can be null
+ * @return Record Data Type or null
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ if (cell == null) {
+ return null;
+ }
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ dataType = getDateTimeDataType(cell);
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getStringCellValue();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
+ dataType = timeDataType.orElse(RecordFieldType.STRING.getDataType());
+ } else if (CellType.FORMULA == cellType) {
+ dataType = getFormulaResultDataType(cell);
+ } else {
+ // Default to null for known and unknown Cell Types
+ dataType = null;
+ }
+
+ return dataType;
+ }
+
+ private DataType getDateTimeDataType(final Cell cell) {
+ final DataType dataType;
+
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ // Numbers without decimal fractions indicate Dates without Times
+ dataType = RecordFieldType.DATE.getDataType();
+ } else if (numericCellValue < 1) {
+ // Decimal fractions indicate Times without Dates
+ dataType = RecordFieldType.TIME.getDataType();
+ } else {
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ }
+
+ return dataType;
+ }
+
+ private DataType getFormulaResultDataType(final Cell cell) {
+ final DataType dataType;
+
+ final CellType formulaResultType = cell.getCachedFormulaResultType();
+ if (CellType.BOOLEAN == formulaResultType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == formulaResultType) {
+ dataType = RecordFieldType.STRING.getDataType();
+ } else if (CellType.NUMERIC == formulaResultType) {
+ // Date Formatting check limited to NUMERIC Cell Types without Conditional Formatting Evaluator
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
Review Comment:
Looking at the DataFormatter code, specifically lines 513-522
```
synchronized(dateFormat) {
if (dateFormat instanceof ExcelStyleDateFormatter) {
((ExcelStyleDateFormatter)dateFormat).setDateToBeFormatted(cell.getNumericCellValue());
}
Date d = cell.getDateCellValue();
return this.performDateFormatting(d, (Format)dateFormat);
}
```
it seems it gets the cell's numeric or date value. If that is the case then I assume you can reuse the `getDateTimeDataType` above to determine the precise date type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1797219608
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getRichStringCellValue().getString();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
+ dataType = timeDataType.orElse(RecordFieldType.STRING.getDataType());
+ } else if (CellType.FORMULA == cellType) {
+ dataType = RecordFieldType.STRING.getDataType();
+ } else {
Review Comment:
In my PR I was trying to correct that logic as I had not realized `Cell.getStringCellValue()` gets the text of the formula. That is why I had added to `ExcelUtils`
```
private static final DataFormatter DATA_FORMATTER = new DataFormatter(Locale.getDefault());
static {
DATA_FORMATTER.setUseCachedValuesForFormulaCells(true);
}
```
as DataFormatter can obtain the formula values after calling `setUseCachedValuesForFormulaCells(true)`
I found out how to do this from this [post](https://stackoverflow.com/questions/7608511/java-poi-how-to-read-excel-cell-value-and-not-the-formula-computing-it) answer from Axel Richter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1797266016
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getRichStringCellValue().getString();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
+ dataType = timeDataType.orElse(RecordFieldType.STRING.getDataType());
+ } else if (CellType.FORMULA == cellType) {
+ dataType = RecordFieldType.STRING.getDataType();
+ } else {
Review Comment:
I pushed another update to improve the handling of Formula Cells, based on the Cached Formula Result Type. This follows the same general behavior of the `DataFormatter` when it is configured to use cached values from formula cells.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1799701807
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getStringCellValue();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
Review Comment:
That's a good question. It depends how flexible we should be for timestamp handling. In other words, it is possible for a `STRING` cell to contain something that looks like a timestamp. This would effectively remove the need for any time or date format properties in the `ExcelReader`, but it would require times and dates to be stored as `NUMERIC` types. I'm inclined to leave it in, given the existence of those format properties, but it might be helpful to keep it for now to enable wider time and date detection from some spreadsheets.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1799834038
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/test/java/org/apache/nifi/excel/TestExcelSchemaInference.java:
##########
@@ -50,69 +45,54 @@ public class TestExcelSchemaInference {
private static final String EXPECTED_SECOND_FIELD_NAME = ExcelUtils.FIELD_NAME_PREFIX + "1";
private static final String EXPECTED_THIRD_FIELD_NAME = ExcelUtils.FIELD_NAME_PREFIX + "2";
private static final String EXPECTED_FOURTH_FIELD_NAME = ExcelUtils.FIELD_NAME_PREFIX + "3";
+
+ private static final String SIMPLE_FORMATTING_PATH = "/excel/simpleDataFormatting.xlsx";
+
private final TimeValueInference timestampInference = new TimeValueInference("MM/dd/yyyy", "HH:mm:ss.SSS", "yyyy/MM/dd/ HH:mm");
@Mock
ComponentLog logger;
- @ParameterizedTest
- @MethodSource("getLocales")
- public void testInferenceAgainstDifferentLocales(Locale locale) throws IOException {
- final Map<PropertyDescriptor, String> properties = new HashMap<>();
- new ExcelReader().getSupportedPropertyDescriptors().forEach(prop -> properties.put(prop, prop.getDefaultValue()));
- final PropertyContext context = new MockConfigurationContext(properties, null, null);
-
- try (final InputStream inputStream = getResourceStream("/excel/numbers.xlsx")) {
- final InferSchemaAccessStrategy<?> accessStrategy = new InferSchemaAccessStrategy<>(
- (variables, content) -> new ExcelRecordSource(content, context, variables, logger),
- new ExcelSchemaInference(timestampInference, locale), logger);
- final RecordSchema schema = accessStrategy.getSchema(null, inputStream, null);
- final List<String> fieldNames = schema.getFieldNames();
- assertEquals(Collections.singletonList(EXPECTED_FIRST_FIELD_NAME), fieldNames);
-
- if (Locale.FRENCH.equals(locale)) {
- assertEquals(RecordFieldType.STRING, schema.getDataType(EXPECTED_FIRST_FIELD_NAME).get().getFieldType());
- } else {
- assertEquals(RecordFieldType.CHOICE.getChoiceDataType(RecordFieldType.FLOAT.getDataType(), RecordFieldType.STRING.getDataType()),
- schema.getDataType(EXPECTED_FIRST_FIELD_NAME).get());
- }
- }
- }
-
- private static Stream<Arguments> getLocales() {
- Locale hindi = Locale.of("hin");
- return Stream.of(
- Arguments.of(Locale.ENGLISH),
- Arguments.of(hindi),
- Arguments.of(Locale.JAPANESE),
- Arguments.of(Locale.FRENCH)
- );
- }
-
@Test
public void testInferenceIncludesAllRecords() throws IOException {
final Map<PropertyDescriptor, String> properties = new HashMap<>();
new ExcelReader().getSupportedPropertyDescriptors().forEach(prop -> properties.put(prop, prop.getDefaultValue()));
final PropertyContext context = new MockConfigurationContext(properties, null, null);
final RecordSchema schema;
- try (final InputStream inputStream = getResourceStream("/excel/simpleDataFormatting.xlsx")) {
+ try (final InputStream inputStream = getResourceStream()) {
final InferSchemaAccessStrategy<?> accessStrategy = new InferSchemaAccessStrategy<>(
(variables, content) -> new ExcelRecordSource(content, context, variables, logger),
new ExcelSchemaInference(timestampInference), Mockito.mock(ComponentLog.class));
schema = accessStrategy.getSchema(null, inputStream, null);
}
- final List<String> fieldNames = schema.getFieldNames();
- assertEquals(Arrays.asList(EXPECTED_FIRST_FIELD_NAME, EXPECTED_SECOND_FIELD_NAME,
- EXPECTED_THIRD_FIELD_NAME, EXPECTED_FOURTH_FIELD_NAME), fieldNames);
- assertEquals(RecordFieldType.CHOICE.getChoiceDataType(RecordFieldType.INT.getDataType(),
- RecordFieldType.STRING.getDataType()), schema.getDataType(EXPECTED_FIRST_FIELD_NAME).get());
- assertEquals(RecordFieldType.CHOICE.getChoiceDataType(RecordFieldType.TIMESTAMP.getDataType("yyyy/MM/dd/ HH:mm"), RecordFieldType.STRING.getDataType()),
- schema.getDataType(EXPECTED_SECOND_FIELD_NAME).get());
- assertEquals(RecordFieldType.STRING, schema.getDataType(EXPECTED_THIRD_FIELD_NAME).get().getFieldType());
- assertEquals(RecordFieldType.CHOICE.getChoiceDataType(RecordFieldType.BOOLEAN.getDataType(),
- RecordFieldType.STRING.getDataType()), schema.getDataType(EXPECTED_FOURTH_FIELD_NAME).get());
+ assertFieldNamesFound(schema);
+ assertFieldDataTypeEquals(schema, EXPECTED_FIRST_FIELD_NAME,
+ RecordFieldType.CHOICE.getChoiceDataType(
+ RecordFieldType.LONG.getDataType(),
+ RecordFieldType.STRING.getDataType()
+ )
+ );
+ assertFieldDataTypeEquals(schema, EXPECTED_SECOND_FIELD_NAME,
+ RecordFieldType.CHOICE.getChoiceDataType(
+ // Assert Timestamp Data Type with standard Date and Time Pattern
Review Comment:
As I mentioned previously, this unit test is not exercising this as we could mock the `TimeValueInference` and not have any patterns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1800148592
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
Review Comment:
I thought though the user entered properties can/should assist the inference logic in help determining whether a field is a date, time or timestamp as I believe they are used in other readers for inferring. They can be used if dates are converted to a string with the `Dataformatter`as they are displayed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1800148592
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
Review Comment:
I thought though the user entered properties can/should assist the inference logic in help determining whether a field is a date, time or timestamp.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1801418393
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getStringCellValue();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
Review Comment:
I actually meant end user documentation as this is a change in functionality from what has been up until now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1802031090
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ /**
+ * Standard Cell Field Type Reader constructor with Time Value Inference for handling STRING Cell Types that may contain values with Timestamps
+ *
+ * @param timeValueInference Time Value Inference required for STRING Cell Type evaluation
+ */
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell can be null
+ * @param fieldName Cell field name for tracking in Field Types required
+ * @param fieldTypes Map of Field Name to Field Type Inference information required
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell can be null
+ * @return Record Data Type or null
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ if (cell == null) {
+ return null;
+ }
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ dataType = getDateTimeDataType(cell);
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getStringCellValue();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
+ dataType = timeDataType.orElse(RecordFieldType.STRING.getDataType());
+ } else if (CellType.FORMULA == cellType) {
+ dataType = getFormulaResultDataType(cell);
+ } else {
+ // Default to null for known and unknown Cell Types
+ dataType = null;
+ }
+
+ return dataType;
+ }
+
+ private DataType getDateTimeDataType(final Cell cell) {
+ final DataType dataType;
+
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ // Numbers without decimal fractions indicate Dates without Times
+ dataType = RecordFieldType.DATE.getDataType();
+ } else if (numericCellValue < 1) {
+ // Decimal fractions indicate Times without Dates
+ dataType = RecordFieldType.TIME.getDataType();
+ } else {
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ }
+
+ return dataType;
+ }
+
+ private DataType getFormulaResultDataType(final Cell cell) {
+ final DataType dataType;
+
+ final CellType formulaResultType = cell.getCachedFormulaResultType();
+ if (CellType.BOOLEAN == formulaResultType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == formulaResultType) {
+ dataType = RecordFieldType.STRING.getDataType();
+ } else if (CellType.NUMERIC == formulaResultType) {
+ // Date Formatting check limited to NUMERIC Cell Types without Conditional Formatting Evaluator
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
Review Comment:
I do not see any methods which can get the cached formula result but do we leave this as `TIMESTAMP` or somehow try to get the result and determine whether its a `DATE` or `TIME` like we did above?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1801655177
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getStringCellValue();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
Review Comment:
> any value that would otherwise be considered a STRING type
That part is bothering me as until now that was everything but now any column whose Excel format is date is not included in this. I am not sure users will understand that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1799816503
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/test/java/org/apache/nifi/excel/TestExcelSchemaInference.java:
##########
@@ -50,69 +45,54 @@ public class TestExcelSchemaInference {
private static final String EXPECTED_SECOND_FIELD_NAME = ExcelUtils.FIELD_NAME_PREFIX + "1";
private static final String EXPECTED_THIRD_FIELD_NAME = ExcelUtils.FIELD_NAME_PREFIX + "2";
private static final String EXPECTED_FOURTH_FIELD_NAME = ExcelUtils.FIELD_NAME_PREFIX + "3";
+
+ private static final String SIMPLE_FORMATTING_PATH = "/excel/simpleDataFormatting.xlsx";
+
private final TimeValueInference timestampInference = new TimeValueInference("MM/dd/yyyy", "HH:mm:ss.SSS", "yyyy/MM/dd/ HH:mm");
Review Comment:
BTW I noticed I could make this a mock using the `@Mock` annotation without any value and the unit tests would still work i.e.
```
@Mock
TimeValueInference timestampInference ;
```
This indicates to me that for the the excel spreadsheet referenced in the unit test the `TimeValueInference` variable is never used since all the cells in the `Timestamps` column have a date format and not a string format (except for the first cell which has the string `Timestamps`. It would be worthwhile to change at least one cell to a string format and have one of the date, time or timestamp formats match it so the code in `StandardCellFieldTypeReader` line 93 is exercised.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1800017219
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
Review Comment:
I do not think there should be this check as there can be cells which have no data in it and `Cell `will be null. I uncovered this running the `TestExcelReader` which I had created for my PR.
```suggestion
Objects.requireNonNull(fieldName, "Field Name required");
Objects.requireNonNull(fieldTypes, "Field Types required");
final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
final DataType dataType = cell != null ? getCellDataType(cell) : null;
fieldTypeInference.addPossibleDataType(dataType);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on PR #9379:
URL: https://github.com/apache/nifi/pull/9379#issuecomment-2414275432
> Also I noticed even with an explicit schema if we do not get the formatted form, an exception can be thrown when reading with `ExcelRecordReader`. The case in point I see is when the schema defines a boolean and the column returns a boolean from a formula. Without the formatted form, `1.0` is returned which will not be interpreted as a boolean in Java. With the formatted form, `TRUE` is returned which can interpreted as a boolean in Java.
This sounds like a case where the cached formula result type is not returning `BOOLEAN` and is instead returning `NUMERIC` or `STRING`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1801391101
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/test/java/org/apache/nifi/excel/TestExcelSchemaInference.java:
##########
@@ -50,69 +45,54 @@ public class TestExcelSchemaInference {
private static final String EXPECTED_SECOND_FIELD_NAME = ExcelUtils.FIELD_NAME_PREFIX + "1";
private static final String EXPECTED_THIRD_FIELD_NAME = ExcelUtils.FIELD_NAME_PREFIX + "2";
private static final String EXPECTED_FOURTH_FIELD_NAME = ExcelUtils.FIELD_NAME_PREFIX + "3";
+
+ private static final String SIMPLE_FORMATTING_PATH = "/excel/simpleDataFormatting.xlsx";
+
private final TimeValueInference timestampInference = new TimeValueInference("MM/dd/yyyy", "HH:mm:ss.SSS", "yyyy/MM/dd/ HH:mm");
Review Comment:
Thanks, I changed the existing `TimeValueInference` to a mock, but added a new test method with a `TimeValueInference` that handles a `STRING` field as a timestamp.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1800013904
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
Review Comment:
I do not think there should be this check as there can be cells which have no data in it and `Cell `will be null. I uncovered this running the `TestExcelReader` which I had created for my PR.
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1797219608
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getRichStringCellValue().getString();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
+ dataType = timeDataType.orElse(RecordFieldType.STRING.getDataType());
+ } else if (CellType.FORMULA == cellType) {
+ dataType = RecordFieldType.STRING.getDataType();
+ } else {
Review Comment:
In my PR I was trying to correct that logic as I had not realized `Cell.getStringCellValue()` gets the text of the formula. That is why I had added to `ExcelUtils`
```
private static final DataFormatter DATA_FORMATTER = new DataFormatter(Locale.getDefault());
static {
DATA_FORMATTER.setUseCachedValuesForFormulaCells(true);
}
```
as DataFormatter can obtain the formula values after calling `setUseCachedValuesForFormulaCells(true)`
I found out how to do this from this [post](https://stackoverflow.com/questions/7608511/java-poi-how-to-read-excel-cell-value-and-not-the-formula-computing-it)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1797219608
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getRichStringCellValue().getString();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
+ dataType = timeDataType.orElse(RecordFieldType.STRING.getDataType());
+ } else if (CellType.FORMULA == cellType) {
+ dataType = RecordFieldType.STRING.getDataType();
+ } else {
Review Comment:
In my PR I was trying to correct that logic as I had not realized `Cell.getStringCellValue()` gets the text of the formula. That is why I had added to `ExcelUtils`
```
private static final DataFormatter DATA_FORMATTER = new DataFormatter(Locale.getDefault());
static {
DATA_FORMATTER.setUseCachedValuesForFormulaCells(true);
}
```
as DataFormatter can obtain the formula values after calling `setUseCachedValuesForFormulaCells(true)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1799954561
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/ExcelHeaderSchemaStrategy.java:
##########
@@ -55,18 +52,18 @@ public class ExcelHeaderSchemaStrategy implements SchemaAccessStrategy {
private final PropertyContext context;
private final ComponentLog logger;
- private final TimeValueInference timeValueInference;
+ private final CellFieldTypeReader cellFieldTypeReader;
private final DataFormatter dataFormatter;
public ExcelHeaderSchemaStrategy(PropertyContext context, ComponentLog logger, TimeValueInference timeValueInference, Locale locale) {
this.context = context;
this.logger = logger;
- this.timeValueInference = timeValueInference;
+ this.cellFieldTypeReader = new StandardCellFieldTypeReader(timeValueInference);
this.dataFormatter = locale == null ? new DataFormatter() : new DataFormatter(locale);
Review Comment:
I do not think the Locale argument is needed any longer as we can instantiate DataFormatter as follows:
```suggestion
this.dataFormatter = new DataFormatter(Locale.getDefault());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1803020591
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ /**
+ * Standard Cell Field Type Reader constructor with Time Value Inference for handling STRING Cell Types that may contain values with Timestamps
+ *
+ * @param timeValueInference Time Value Inference required for STRING Cell Type evaluation
+ */
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell can be null
+ * @param fieldName Cell field name for tracking in Field Types required
+ * @param fieldTypes Map of Field Name to Field Type Inference information required
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell can be null
+ * @return Record Data Type or null
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ if (cell == null) {
+ return null;
+ }
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ dataType = getDateTimeDataType(cell);
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getStringCellValue();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
+ dataType = timeDataType.orElse(RecordFieldType.STRING.getDataType());
+ } else if (CellType.FORMULA == cellType) {
+ dataType = getFormulaResultDataType(cell);
+ } else {
+ // Default to null for known and unknown Cell Types
+ dataType = null;
+ }
+
+ return dataType;
+ }
+
+ private DataType getDateTimeDataType(final Cell cell) {
+ final DataType dataType;
+
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ // Numbers without decimal fractions indicate Dates without Times
+ dataType = RecordFieldType.DATE.getDataType();
+ } else if (numericCellValue < 1) {
+ // Decimal fractions indicate Times without Dates
+ dataType = RecordFieldType.TIME.getDataType();
+ } else {
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ }
+
+ return dataType;
+ }
+
+ private DataType getFormulaResultDataType(final Cell cell) {
+ final DataType dataType;
+
+ final CellType formulaResultType = cell.getCachedFormulaResultType();
+ if (CellType.BOOLEAN == formulaResultType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == formulaResultType) {
+ dataType = RecordFieldType.STRING.getDataType();
+ } else if (CellType.NUMERIC == formulaResultType) {
+ // Date Formatting check limited to NUMERIC Cell Types without Conditional Formatting Evaluator
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
Review Comment:
Thanks for pointing out that this aligns to `getFormulaCellValue()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1799816503
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/test/java/org/apache/nifi/excel/TestExcelSchemaInference.java:
##########
@@ -50,69 +45,54 @@ public class TestExcelSchemaInference {
private static final String EXPECTED_SECOND_FIELD_NAME = ExcelUtils.FIELD_NAME_PREFIX + "1";
private static final String EXPECTED_THIRD_FIELD_NAME = ExcelUtils.FIELD_NAME_PREFIX + "2";
private static final String EXPECTED_FOURTH_FIELD_NAME = ExcelUtils.FIELD_NAME_PREFIX + "3";
+
+ private static final String SIMPLE_FORMATTING_PATH = "/excel/simpleDataFormatting.xlsx";
+
private final TimeValueInference timestampInference = new TimeValueInference("MM/dd/yyyy", "HH:mm:ss.SSS", "yyyy/MM/dd/ HH:mm");
Review Comment:
BTW I noticed I could make this a mock using the `@Mock` annotation without any value and the unit tests would still work i.e.
```
@Mock
TimeValueInference timestampInference ;
```
This indicates to me that for the the excel spreadsheet referenced in the unit test the `TimeValueInference` variable is never used since all the cells in the `Timestamps` column have a date format and not a string format (except for the first cell which has the string `Timestamps`). It would be worthwhile to change at least one cell to a string format and have one of the date, time or timestamp formats match it so the code in `StandardCellFieldTypeReader` line 93 is exercised.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1799969080
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/ExcelRecordReader.java:
##########
@@ -133,15 +134,32 @@ private Map<String, Object> getCurrentRowValues(Row currentRow, boolean coerceTy
return currentRowValues;
}
- private static Object getCellValue(Cell cell) {
- if (cell != null) {
- return switch (cell.getCellType()) {
- case _NONE, BLANK, ERROR, FORMULA, STRING -> cell.getStringCellValue();
+ private static Object getCellValue(final Cell cell) {
+ final Object cellValue;
+
+ if (cell == null) {
+ cellValue = null;
+ } else {
+ final CellType cellType = cell.getCellType();
+ cellValue = switch (cellType) {
+ case _NONE, BLANK, ERROR, STRING -> cell.getStringCellValue();
case NUMERIC -> DateUtil.isCellDateFormatted(cell) ? cell.getDateCellValue() : cell.getNumericCellValue();
case BOOLEAN -> cell.getBooleanCellValue();
+ case FORMULA -> getFormulaCellValue(cell);
};
}
- return null;
+
+ return cellValue;
+ }
+
+ private static Object getFormulaCellValue(final Cell cell) {
+ final CellType formulaResultType = cell.getCachedFormulaResultType();
+ return switch (formulaResultType) {
+ case BOOLEAN -> cell.getBooleanCellValue();
+ case STRING -> cell.getStringCellValue();
+ case NUMERIC -> DateUtil.isCellDateFormatted(cell) ? cell.getDateCellValue() : cell.getNumericCellValue();
+ default -> null;
Review Comment:
This will return null for those formulas which have errors such as `Err:511` and `#NAME?`. End users may want to see these in order to correct their Excel worksheet.
```suggestion
return switch (formulaResultType) {
case BOOLEAN -> cell.getBooleanCellValue();
case ERROR, STRING -> cell.getStringCellValue();
case NUMERIC -> DateUtil.isCellDateFormatted(cell) ? cell.getDateCellValue() : cell.getNumericCellValue();
default -> null;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1800172877
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/ExcelRecordReader.java:
##########
@@ -133,15 +134,32 @@ private Map<String, Object> getCurrentRowValues(Row currentRow, boolean coerceTy
return currentRowValues;
}
- private static Object getCellValue(Cell cell) {
- if (cell != null) {
- return switch (cell.getCellType()) {
- case _NONE, BLANK, ERROR, FORMULA, STRING -> cell.getStringCellValue();
+ private static Object getCellValue(final Cell cell) {
+ final Object cellValue;
+
+ if (cell == null) {
+ cellValue = null;
+ } else {
+ final CellType cellType = cell.getCellType();
+ cellValue = switch (cellType) {
+ case _NONE, BLANK, ERROR, STRING -> cell.getStringCellValue();
case NUMERIC -> DateUtil.isCellDateFormatted(cell) ? cell.getDateCellValue() : cell.getNumericCellValue();
case BOOLEAN -> cell.getBooleanCellValue();
+ case FORMULA -> getFormulaCellValue(cell);
};
}
- return null;
+
+ return cellValue;
+ }
+
+ private static Object getFormulaCellValue(final Cell cell) {
+ final CellType formulaResultType = cell.getCachedFormulaResultType();
+ return switch (formulaResultType) {
+ case BOOLEAN -> cell.getBooleanCellValue();
+ case STRING -> cell.getStringCellValue();
+ case NUMERIC -> DateUtil.isCellDateFormatted(cell) ? cell.getDateCellValue() : cell.getNumericCellValue();
+ default -> null;
Review Comment:
Seems like the `default -> null;` needs to still be there otherwise at least in Intellij there is a warning that the `switch` statement does not cover all cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1799748937
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getStringCellValue();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
Review Comment:
I think though this piece should be documented that any time or date format properties specified by the user (stored in `TimeValueInference`) will **_only_** be used to parse those cells whose cell type is string and not whose already is date (for the end user this may be understood is if the Excel format type is text and not date or time).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1799954561
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/ExcelHeaderSchemaStrategy.java:
##########
@@ -55,18 +52,18 @@ public class ExcelHeaderSchemaStrategy implements SchemaAccessStrategy {
private final PropertyContext context;
private final ComponentLog logger;
- private final TimeValueInference timeValueInference;
+ private final CellFieldTypeReader cellFieldTypeReader;
private final DataFormatter dataFormatter;
public ExcelHeaderSchemaStrategy(PropertyContext context, ComponentLog logger, TimeValueInference timeValueInference, Locale locale) {
this.context = context;
this.logger = logger;
- this.timeValueInference = timeValueInference;
+ this.cellFieldTypeReader = new StandardCellFieldTypeReader(timeValueInference);
this.dataFormatter = locale == null ? new DataFormatter() : new DataFormatter(locale);
Review Comment:
I do not think the `Locale` argument is needed any longer as we can instantiate `DataFormatter` as follows:
```suggestion
this.dataFormatter = new DataFormatter(Locale.getDefault());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1800085624
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
Review Comment:
I do not think defaulting to `TIMESTAMP` is right for every instance. In the `TestExcelReader` I have in my PR I have three columns
1. ` Date_Custom` which has a date value 07/25/2024 (displayed as `07/25/24`)
2. `Time_Custom` which has a time value of 04:00:00 PM (displayed as `16:00`)
3. `Timestamp` which has a timestamp 07/26/2024 16:00:00
When running through your code they are parsed and look like the following when calling `toString` on the `java.sql.Timestamp`
1. 2024-07-05 00:00:00.0
2. 1899-12-31 00:00:00.0
3. 2024-07-26 16:00:00.0
For dates and timestamp 1 and 3 are okay but for those fields which only represent time, I do not think users are interested in the default date Excel has for time 1899-12-31. I think there has to a more fine grained way of determining whether the date we have is representing a date or only time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1801397344
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
Review Comment:
The `TimeValueInference` does provide a narrowed Date or Time DataType with a matching format, but that does not apply to Timestamp fields where Excel stores the value in a `NUMERIC` field. Converting everything to a string using the `DataFormatter` would be less than optimal from a performance point of view, so for type inference handling, this seems like the best option for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on PR #9379:
URL: https://github.com/apache/nifi/pull/9379#issuecomment-2414764301
@exceptionfactory I built the code without the most recent change but I am not sure that will make a difference. Part of the user's issue in this ticket was for his time custom column the final format is a negative number `-2209017600000` which is still occurring with the current changes. I assume this needs to be corrected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1802089085
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ /**
+ * Standard Cell Field Type Reader constructor with Time Value Inference for handling STRING Cell Types that may contain values with Timestamps
+ *
+ * @param timeValueInference Time Value Inference required for STRING Cell Type evaluation
+ */
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell can be null
+ * @param fieldName Cell field name for tracking in Field Types required
+ * @param fieldTypes Map of Field Name to Field Type Inference information required
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell can be null
+ * @return Record Data Type or null
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ if (cell == null) {
+ return null;
+ }
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ dataType = getDateTimeDataType(cell);
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getStringCellValue();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
+ dataType = timeDataType.orElse(RecordFieldType.STRING.getDataType());
+ } else if (CellType.FORMULA == cellType) {
+ dataType = getFormulaResultDataType(cell);
+ } else {
+ // Default to null for known and unknown Cell Types
+ dataType = null;
+ }
+
+ return dataType;
+ }
+
+ private DataType getDateTimeDataType(final Cell cell) {
+ final DataType dataType;
+
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ // Numbers without decimal fractions indicate Dates without Times
+ dataType = RecordFieldType.DATE.getDataType();
+ } else if (numericCellValue < 1) {
+ // Decimal fractions indicate Times without Dates
+ dataType = RecordFieldType.TIME.getDataType();
+ } else {
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ }
+
+ return dataType;
+ }
+
+ private DataType getFormulaResultDataType(final Cell cell) {
+ final DataType dataType;
+
+ final CellType formulaResultType = cell.getCachedFormulaResultType();
+ if (CellType.BOOLEAN == formulaResultType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == formulaResultType) {
+ dataType = RecordFieldType.STRING.getDataType();
+ } else if (CellType.NUMERIC == formulaResultType) {
+ // Date Formatting check limited to NUMERIC Cell Types without Conditional Formatting Evaluator
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
Review Comment:
With a closer look at at the POI's `DataFormatter` code, specifically lines 513-522
```
synchronized(dateFormat) {
if (dateFormat instanceof ExcelStyleDateFormatter) {
((ExcelStyleDateFormatter)dateFormat).setDateToBeFormatted(cell.getNumericCellValue());
}
Date d = cell.getDateCellValue();
return this.performDateFormatting(d, (Format)dateFormat);
}
```
it seems it gets the cell's numeric or date value. If that is the case then I assume you can reuse the `getDateTimeDataType` method above to determine the precise date type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1802222701
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ /**
+ * Standard Cell Field Type Reader constructor with Time Value Inference for handling STRING Cell Types that may contain values with Timestamps
+ *
+ * @param timeValueInference Time Value Inference required for STRING Cell Type evaluation
+ */
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell can be null
+ * @param fieldName Cell field name for tracking in Field Types required
+ * @param fieldTypes Map of Field Name to Field Type Inference information required
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell can be null
+ * @return Record Data Type or null
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ if (cell == null) {
+ return null;
+ }
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ dataType = getDateTimeDataType(cell);
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getStringCellValue();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
+ dataType = timeDataType.orElse(RecordFieldType.STRING.getDataType());
+ } else if (CellType.FORMULA == cellType) {
+ dataType = getFormulaResultDataType(cell);
+ } else {
+ // Default to null for known and unknown Cell Types
+ dataType = null;
+ }
+
+ return dataType;
+ }
+
+ private DataType getDateTimeDataType(final Cell cell) {
+ final DataType dataType;
+
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ // Numbers without decimal fractions indicate Dates without Times
+ dataType = RecordFieldType.DATE.getDataType();
+ } else if (numericCellValue < 1) {
+ // Decimal fractions indicate Times without Dates
+ dataType = RecordFieldType.TIME.getDataType();
+ } else {
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ }
+
+ return dataType;
+ }
+
+ private DataType getFormulaResultDataType(final Cell cell) {
+ final DataType dataType;
+
+ final CellType formulaResultType = cell.getCachedFormulaResultType();
+ if (CellType.BOOLEAN == formulaResultType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == formulaResultType) {
+ dataType = RecordFieldType.STRING.getDataType();
+ } else if (CellType.NUMERIC == formulaResultType) {
+ // Date Formatting check limited to NUMERIC Cell Types without Conditional Formatting Evaluator
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
Review Comment:
Thanks for the additional pointer. I ran a test with a formula cell in XLSX, confirmed the numeric cell value was available, and I was able to reuse the same `getDateTimeDataType()` method to narrow the Data Type detection, as already used with numeric types. This also aligns with the behavior of `getFormulaCellValue()` in `ExcelRecordReader`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1797216419
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
Review Comment:
I considered that as an option, however, inferring `LONG` or `DOUBLE` provides greater type safety. For example, if the maximum numeric value for the evaluated rows fits into an `int`, but subsequent rows exceed the maximum value of an `int`, then processing would fail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1797187964
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getRichStringCellValue().getString();
Review Comment:
Are you sure you want to use `getRichStringCellValue` ? From the following [post](https://stackoverflow.com/questions/16170275/what-is-the-difference-between-getrichstringcellvalue-and-getstringcellvalue) it seems its better to use `getStringCellValue`?
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
Review Comment:
Do we want to determine whether the data types are `INTEGER` or `FLOAT` or just keep at `LONG`/`DOUBLE`?
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getRichStringCellValue().getString();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
+ dataType = timeDataType.orElse(RecordFieldType.STRING.getDataType());
+ } else if (CellType.FORMULA == cellType) {
+ dataType = RecordFieldType.STRING.getDataType();
+ } else {
Review Comment:
I am not sure this is correct because the formula has to be evaluated and can evaluated to a number or a boolean. The text of the formula is a string but I assume most end user's are not interested in the formula text but rather what the formula evaluates to.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1797191879
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getRichStringCellValue().getString();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
+ dataType = timeDataType.orElse(RecordFieldType.STRING.getDataType());
+ } else if (CellType.FORMULA == cellType) {
+ dataType = RecordFieldType.STRING.getDataType();
+ } else {
Review Comment:
I am not sure this is correct because the formula has to be evaluated and can evaluated to a string, number or a boolean. The text of the formula is a string but I assume most end user's are not interested in the formula text but rather what the formula evaluates to.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1799969080
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/ExcelRecordReader.java:
##########
@@ -133,15 +134,32 @@ private Map<String, Object> getCurrentRowValues(Row currentRow, boolean coerceTy
return currentRowValues;
}
- private static Object getCellValue(Cell cell) {
- if (cell != null) {
- return switch (cell.getCellType()) {
- case _NONE, BLANK, ERROR, FORMULA, STRING -> cell.getStringCellValue();
+ private static Object getCellValue(final Cell cell) {
+ final Object cellValue;
+
+ if (cell == null) {
+ cellValue = null;
+ } else {
+ final CellType cellType = cell.getCellType();
+ cellValue = switch (cellType) {
+ case _NONE, BLANK, ERROR, STRING -> cell.getStringCellValue();
case NUMERIC -> DateUtil.isCellDateFormatted(cell) ? cell.getDateCellValue() : cell.getNumericCellValue();
case BOOLEAN -> cell.getBooleanCellValue();
+ case FORMULA -> getFormulaCellValue(cell);
};
}
- return null;
+
+ return cellValue;
+ }
+
+ private static Object getFormulaCellValue(final Cell cell) {
+ final CellType formulaResultType = cell.getCachedFormulaResultType();
+ return switch (formulaResultType) {
+ case BOOLEAN -> cell.getBooleanCellValue();
+ case STRING -> cell.getStringCellValue();
+ case NUMERIC -> DateUtil.isCellDateFormatted(cell) ? cell.getDateCellValue() : cell.getNumericCellValue();
+ default -> null;
Review Comment:
This will return null for those formulas which have errors such as `Err:511` and `#NAME?`. End users may want to see these in order to correct their Excel worksheet.
```suggestion
return switch (formulaResultType) {
case BOOLEAN -> cell.getBooleanCellValue();
case ERROR, STRING -> cell.getStringCellValue();
case NUMERIC -> DateUtil.isCellDateFormatted(cell) ? cell.getDateCellValue() : cell.getNumericCellValue();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1799748937
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getStringCellValue();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
Review Comment:
I think though this piece should be documented that any time or date format properties specified by the user (stored in `TimeValueInference`) will **_only_** be used to parse those cells whose cell type is string and not whose already is date (for the end user this may be understood if the Excel format type is text and not date or time).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1800085624
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
Review Comment:
I do not think defaulting to `TIMESTAMP` is right for every case. In the `TestExcelReader` I have in my PR I have three columns
1. ` Date_Custom` which has a date value 07/25/2024 (displayed as `07/25/24`)
2. `Time_Custom` which has a time value of 04:00:00 PM (displayed as `16:00`)
3. `Timestamp` which has a timestamp 07/26/2024 16:00:00
When running through your code they are parsed and look like the following when calling `toString` on the `java.sql.Timestamp`
1. 2024-07-05 00:00:00.0
2. 1899-12-31 00:00:00.0
3. 2024-07-26 16:00:00.0
For dates and timestamp 1 and 3 are okay but for those fields which only represent time, I do not think users are interested in the default date Excel has for time 1899-12-31. I think there has to a more fine grained way of determining whether the date we have is representing a date or only time.
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
Review Comment:
I do not think defaulting to `TIMESTAMP` is right for every case. In the `TestExcelReader` I have in my PR I have three columns
1. `Date_Custom` which has a date value 07/25/2024 (displayed as `07/25/24`)
2. `Time_Custom` which has a time value of 04:00:00 PM (displayed as `16:00`)
3. `Timestamp` which has a timestamp 07/26/2024 16:00:00
When running through your code they are parsed and look like the following when calling `toString` on the `java.sql.Timestamp`
1. 2024-07-05 00:00:00.0
2. 1899-12-31 00:00:00.0
3. 2024-07-26 16:00:00.0
For dates and timestamp 1 and 3 are okay but for those fields which only represent time, I do not think users are interested in the default date Excel has for time 1899-12-31. I think there has to a more fine grained way of determining whether the date we have is representing a date or only time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1800137398
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
Review Comment:
Thanks for highlighting these details @dan-s1. This is an important issue of the way that Excel stores dates, and the way the POI library handles dates.
For all date and time values, the POI library always returns a `java.sql.Timestamp` or ` java.time.LocalDateTime`. The library interprets date values with zero for hours, minutes, and seconds. For time values, the POI library uses a default starting date, that is the reason for these inferred values.
When using a defined schema, the time and date values will be handled as expected. When using an inferred schema, the Schema Inference logic is unable to differentiate between Timestamp, Date, and Time, so the inferred Schema handles everything as a Timestamp. It might be possible to evaluate some Timestamp values and infer that they contain only a Date or only Time, but such inference would be prone to error. For this scenario, the inference process is necessarily general, and is unable to narrow down values to either or Date or Time. It might be possible to try something based on the date format from Excel, if specified, but this is also prone to error based on the variety of values, which are different than Java date formatting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1800148592
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
Review Comment:
I thought though the user entered properties can/should assist the inference logic in help determining whether a field is a date, time or timestamp as I believe they are used in other readers for inferring. They can be used if dates are converted to a string with the `Dataformatter`as they are displayed. Granted this may not be efficient but it is possible to determine whether its a date, time or timestamp.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1801392235
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getStringCellValue();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
Review Comment:
Thanks, I added some comments around the constructor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1801640705
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ /**
+ * Standard Cell Field Type Reader constructor with Time Value Inference for handling STRING Cell Types that may contain values with Timestamps
+ *
+ * @param timeValueInference Time Value Inference required for STRING Cell Type evaluation
+ */
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell can be null
+ * @param fieldName Cell field name for tracking in Field Types required
+ * @param fieldTypes Map of Field Name to Field Type Inference information required
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell can be null
+ * @return Record Data Type or null
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ if (cell == null) {
+ return null;
+ }
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getStringCellValue();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
+ dataType = timeDataType.orElse(RecordFieldType.STRING.getDataType());
+ } else if (CellType.FORMULA == cellType) {
+ dataType = getFormulaResultDataType(cell);
+ } else {
+ // Default to null for known and unknown Cell Types such as ERROR
Review Comment:
Thanks, I changed the wrong comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1801646053
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ /**
+ * Standard Cell Field Type Reader constructor with Time Value Inference for handling STRING Cell Types that may contain values with Timestamps
+ *
+ * @param timeValueInference Time Value Inference required for STRING Cell Type evaluation
+ */
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell can be null
+ * @param fieldName Cell field name for tracking in Field Types required
+ * @param fieldTypes Map of Field Name to Field Type Inference information required
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell can be null
+ * @return Record Data Type or null
+ */
Review Comment:
It isn't absolutely necessary, but if there is a comment, it should include the parameters. I kept them for completeness, since the method comment is different.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on PR #9379:
URL: https://github.com/apache/nifi/pull/9379#issuecomment-2414783441
> @exceptionfactory I built the code without the most recent change but I am not sure that will make a difference. Part of the user's issue in this ticket was for his time custom column the final format is a negative number `-2209017600000` which is still occurring with the current changes. I assume this needs to be corrected. As for his issue of epoch strings I do not think that is an issue if the writer is configured with a timestamp pattern. Let me take that back when I configure a timestamp pattern my resulting CSV is
>
> ```
> Date_Standard,Date_Custom,Time_Standard,Time_Custom,Timestamp
> 2024-07-24 00:00:00,2024-07-25 00:00:00,2024-07-24 16:00:00,1899-12-31 16:00:00,2024-07-26 16:00:00
> ```
>
> I am not sure though users are going to expect `1899-12-31 16:00:00`
>
> Is there a way we can only have the time?
Have you configured a schema for the CSV Writer that indicates the Time field?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1799704636
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getStringCellValue();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
+ dataType = timeDataType.orElse(RecordFieldType.STRING.getDataType());
+ } else if (CellType.FORMULA == cellType) {
+ dataType = getFormulaResultDataType(cell);
+ } else {
+ // Default to null for known and unknown Cell Types: BLANK, ERROR
+ dataType = null;
+ }
+
+ return dataType;
+ }
+
+ private DataType getFormulaResultDataType(final Cell cell) {
+ final DataType dataType;
+
+ final CellType formulaResultType = cell.getCachedFormulaResultType();
+ if (CellType.BOOLEAN == formulaResultType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == formulaResultType) {
+ dataType = RecordFieldType.STRING.getDataType();
+ } else if (CellType.NUMERIC == formulaResultType) {
+ // Date Formatting check limited to NUMERIC Cell Types without Conditional Formatting Evaluator
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
Review Comment:
I considered that, but it would require evaluating the formula result value, in addition to the cached type. Since formula results could vary more widely, I think it is safer to avoid narrowing the potential type detection and leave it open-ended as a `DOUBLE` so that any integer, long, float, or double, would work as a formula result value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1799816503
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/test/java/org/apache/nifi/excel/TestExcelSchemaInference.java:
##########
@@ -50,69 +45,54 @@ public class TestExcelSchemaInference {
private static final String EXPECTED_SECOND_FIELD_NAME = ExcelUtils.FIELD_NAME_PREFIX + "1";
private static final String EXPECTED_THIRD_FIELD_NAME = ExcelUtils.FIELD_NAME_PREFIX + "2";
private static final String EXPECTED_FOURTH_FIELD_NAME = ExcelUtils.FIELD_NAME_PREFIX + "3";
+
+ private static final String SIMPLE_FORMATTING_PATH = "/excel/simpleDataFormatting.xlsx";
+
private final TimeValueInference timestampInference = new TimeValueInference("MM/dd/yyyy", "HH:mm:ss.SSS", "yyyy/MM/dd/ HH:mm");
Review Comment:
BTW I noticed I could make this a mock using the `@Mock` annotation without any value and the unit tests would still work i.e.
```
@Mock
TimeValueInference timestampInference ;
```
This indicates to me that for the the excel spreadsheet referenced in the unit test the `TimeValueInference` variable is never used since all the cells in the `Timestamps` column have a date format and not a string format (except for the first cell which has the string `Timestamps`). It would be worthwhile to change at least one cell to a string format and have one of the date, time or timestamp formats match it so the code in `StandardCellFieldTypeReader` line 93 is exercised.
Also it would be useful to add columns where there are the different type of formula results so the code of determining its type and value is exercised.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1799888718
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getStringCellValue();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
+ dataType = timeDataType.orElse(RecordFieldType.STRING.getDataType());
+ } else if (CellType.FORMULA == cellType) {
+ dataType = getFormulaResultDataType(cell);
+ } else {
+ // Default to null for known and unknown Cell Types: BLANK, ERROR
+ dataType = null;
+ }
+
+ return dataType;
+ }
+
+ private DataType getFormulaResultDataType(final Cell cell) {
+ final DataType dataType;
+
+ final CellType formulaResultType = cell.getCachedFormulaResultType();
+ if (CellType.BOOLEAN == formulaResultType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == formulaResultType) {
+ dataType = RecordFieldType.STRING.getDataType();
+ } else if (CellType.NUMERIC == formulaResultType) {
+ // Date Formatting check limited to NUMERIC Cell Types without Conditional Formatting Evaluator
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ } else {
+ // Default to null for known and unknown Formula Result Cell Types: BLANK, ERROR
Review Comment:
Per `getCachedFormulaResultType` [javadocs](https://poi.apache.org/apidocs/5.0/org/apache/poi/ss/usermodel/Cell.html#getCachedFormulaResultType--) there is no BLANK cell type for cached formula cell type
```suggestion
// Default to null for known and unknown Formula Result Cell Type: ERROR
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on PR #9379:
URL: https://github.com/apache/nifi/pull/9379#issuecomment-2414736837
> @exceptionfactory With that change should we should also then improve what is returned by ExcelRecordReader on line 146? Currently we return a `java.util.Date` with `cell.getDateCellValue()` but I think we should employ similar logic to determine if its a DATE/TIME then call `cell.getLocalDateTimeCellValue()` to return a `java.time.LocalDateTime`.
The `java.util.Date` and `java.time.LocalDateTime` values are functionally equivalent in terms of representing years, months, days, hours, minutes, and seconds. The potential difference could come in time zones, which `java.util.Date` has, but `LocalDateTime` does not. With the internal NiFi Record handling of timestamps as `java.sql.Timestamp`, I think this can stay as it is for now unless there is some other compelling reason to adjust the return value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1802061668
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ /**
+ * Standard Cell Field Type Reader constructor with Time Value Inference for handling STRING Cell Types that may contain values with Timestamps
+ *
+ * @param timeValueInference Time Value Inference required for STRING Cell Type evaluation
+ */
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell can be null
+ * @param fieldName Cell field name for tracking in Field Types required
+ * @param fieldTypes Map of Field Name to Field Type Inference information required
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell can be null
+ * @return Record Data Type or null
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ if (cell == null) {
+ return null;
+ }
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ dataType = getDateTimeDataType(cell);
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getStringCellValue();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
+ dataType = timeDataType.orElse(RecordFieldType.STRING.getDataType());
+ } else if (CellType.FORMULA == cellType) {
+ dataType = getFormulaResultDataType(cell);
+ } else {
+ // Default to null for known and unknown Cell Types
+ dataType = null;
+ }
+
+ return dataType;
+ }
+
+ private DataType getDateTimeDataType(final Cell cell) {
+ final DataType dataType;
+
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ // Numbers without decimal fractions indicate Dates without Times
+ dataType = RecordFieldType.DATE.getDataType();
+ } else if (numericCellValue < 1) {
+ // Decimal fractions indicate Times without Dates
+ dataType = RecordFieldType.TIME.getDataType();
+ } else {
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ }
+
+ return dataType;
+ }
+
+ private DataType getFormulaResultDataType(final Cell cell) {
+ final DataType dataType;
+
+ final CellType formulaResultType = cell.getCachedFormulaResultType();
+ if (CellType.BOOLEAN == formulaResultType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == formulaResultType) {
+ dataType = RecordFieldType.STRING.getDataType();
+ } else if (CellType.NUMERIC == formulaResultType) {
+ // Date Formatting check limited to NUMERIC Cell Types without Conditional Formatting Evaluator
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
Review Comment:
Sounds good, we can revisit in the future based on potential issues if needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1801603367
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getStringCellValue();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
Review Comment:
Also it might be worth documenting that when inferring a schema, the schema will be limited to the following specific types BOOLEAN, CHOICE, DOUBLE, LONG, STRING, TIMESTAMP and perhaps DATE and TIME (depending if a date/time are parsed from a text format).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on PR #9379:
URL: https://github.com/apache/nifi/pull/9379#issuecomment-2414867204
Now its working as expected after I set on date, time and timestamp patterns on the on the `CSVRecordSetWriter`
```
Date_Standard,Date_Custom,Time_Standard,Time_Custom,Timestamp
2024-07-24,2024-07-25,2024-07-24 16:00:00,16:00:00,2024-07-26 16:00:00
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1799748937
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getStringCellValue();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
Review Comment:
I think though this piece should be documented that any time or date format properties specified by the user (stored in `TimeValueInference`) will **_only_** be used to parse those cells whose cell type is string and not whose already is date (for the end user this may be understood if the Excel format type is text and not date or time) since the current behavior uses any time or date format properties specified by the user.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1800085624
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
Review Comment:
I do not think defaulting to `TIMESTAMP` is right for every case. In the `TestExcelReader` I have in my PR I have three columns
1. `Date_Custom` which has a date value 07/25/2024 (displayed as `07/25/24`)
2. `Time_Custom` which has a time value of 04:00:00 PM (displayed as `16:00`)
3. `Timestamp` which has a timestamp 07/26/2024 16:00:00
When running through your code they are parsed and look like the following when calling `toString` on the `java.sql.Timestamp` returned from the `Record` object read.
1. 2024-07-05 00:00:00.0
2. 1899-12-31 00:00:00.0
3. 2024-07-26 16:00:00.0
For dates and timestamp 1 and 3 are okay but for those fields which only represent time, I do not think users are interested in the default date Excel has for time 1899-12-31. I think there has to be a more fine grained way of determining whether the date we have is representing a date or only time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1800085624
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
Review Comment:
I do not think defaulting to `TIMESTAMP` is right for every case. In the `TestExcelReader` I have in my PR I have three columns
1. `Date_Custom` which has a date value 07/25/2024 (displayed as `07/25/24`)
2. `Time_Custom` which has a time value of 04:00:00 PM (displayed as `16:00`)
3. `Timestamp` which has a timestamp 07/26/2024 16:00:00
When running through your code they are parsed and look like the following when calling `toString` on the `java.sql.Timestamp` returned from the `Record` object.
1. 2024-07-05 00:00:00.0
2. 1899-12-31 00:00:00.0
3. 2024-07-26 16:00:00.0
For dates and timestamp 1 and 3 are okay but for those fields which only represent time, I do not think users are interested in the default date Excel has for time 1899-12-31. I think there has to be a more fine grained way of determining whether the date we have is representing a timestamp or only time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1801689842
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getStringCellValue();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
Review Comment:
I think part of the challenge is that Excel provides a narrow set of stored type values, and a slightly wider set of supported rendered type values. When combined with type inferencing, there is a lot of potential for confusion. That being said, I think the above paragraph is still accurate. At least the difference between a number of string should be clear enough, so that distinction remains, and is now more strongly followed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on PR #9379:
URL: https://github.com/apache/nifi/pull/9379#issuecomment-2414693743
> @exceptionfactory I just found this helpful [tutorial](https://www.myonlinetraininghub.com/excel-date-and-time#:~:text=When%20you%20look%20at%20a,decimal%20portion%20is%20the%20time.) which explains how dates and times are stored in Excel. Based on this information I believe we can provide better inferences when the cell type is numeric and is date formatted. If its a whole number we know its date, if its a fraction we know its a time and if its a whole number with a fraction then its a timestamp. Hence lines 81-84 in `StandardCellFieldTypeReader` can be expanded to incorporate this logic to identify the most precise type (DATE, TIME or TIMESTAMP).
>
> e.g. (Which probably would look better incorporated in its own method)
>
> ```
> if (DateUtil.isCellDateFormatted(cell)) {
> final double numericCellValue = cell.getNumericCellValue();
> if (numericCellValue > 0 && numericCellValue < 1;) {
> dataType = RecordFieldType.TIME.getDataType();
> } else {
> final long roundedCellValue = (long) numericCellValue;
> if (roundedCellValue == numericCellValue) {
> dataType = RecordFieldType.DATE.getDataType();
> } else {
> dataType = RecordFieldType.TIMESTAMP.getDataType();
> }
> }
> }
> ```
Thanks for the background and example @dan-s1, that is very helpful. I will incorporate that approach and include you as a co-author on the commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on PR #9379:
URL: https://github.com/apache/nifi/pull/9379#issuecomment-2417253091
@exceptionfactory Thank you again for making all the requested changes. Also thank you again for including me as a coauthor.
LGTM+1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1797210179
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getRichStringCellValue().getString();
Review Comment:
Good recommendation, thanks for the reference. I started with `getRichStringCellValue()` based on the behavior of the POI `DataFormatter`, but for Cell Type inference purposes, just using `getStringCellValue()` makes more sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1799748937
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getStringCellValue();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
Review Comment:
I think though this piece should be documented that any time or date format properties specified by the user (stored in `TimeValueInference`) will **_only_** be used to parse those cells whose cell type is string and not whose already is date (for the end user this may be understood if the Excel format type is text and not date or time) since the current behavior uses any time or date format properties specified by the user no matter the cell type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1799618989
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/CellFieldTypeReader.java:
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.poi.ss.usermodel.Cell;
+
+import java.util.Map;
+
+/**
+ * Shared abstraction for determining Record Field Data Type from Spreadsheet Cell Types
+ */
+interface CellFieldTypeReader {
+ /**
+ * Infer Cell Field Type and update Map of Field Type information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
Review Comment:
Probably should add in the javadocs here that these other two arguments are also required as the first one and as you have required checks for 3 in the `StandardCellFieldTypeReader` implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1801472949
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ /**
+ * Standard Cell Field Type Reader constructor with Time Value Inference for handling STRING Cell Types that may contain values with Timestamps
+ *
+ * @param timeValueInference Time Value Inference required for STRING Cell Type evaluation
+ */
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell can be null
+ * @param fieldName Cell field name for tracking in Field Types required
+ * @param fieldTypes Map of Field Name to Field Type Inference information required
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell can be null
+ * @return Record Data Type or null
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ if (cell == null) {
+ return null;
+ }
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getStringCellValue();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
+ dataType = timeDataType.orElse(RecordFieldType.STRING.getDataType());
+ } else if (CellType.FORMULA == cellType) {
+ dataType = getFormulaResultDataType(cell);
+ } else {
+ // Default to null for known and unknown Cell Types such as ERROR
+ dataType = null;
+ }
+
+ return dataType;
+ }
+
+ private DataType getFormulaResultDataType(final Cell cell) {
+ final DataType dataType;
+
+ final CellType formulaResultType = cell.getCachedFormulaResultType();
+ if (CellType.BOOLEAN == formulaResultType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == formulaResultType) {
+ dataType = RecordFieldType.STRING.getDataType();
+ } else if (CellType.NUMERIC == formulaResultType) {
+ // Date Formatting check limited to NUMERIC Cell Types without Conditional Formatting Evaluator
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ } else {
+ // Default to null for known and unknown Formula Result Cell Types: BLANK, ERROR
Review Comment:
```suggestion
// Default to null for known and unknown Formula Result Cell Types: ERROR
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1801464276
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ /**
+ * Standard Cell Field Type Reader constructor with Time Value Inference for handling STRING Cell Types that may contain values with Timestamps
+ *
+ * @param timeValueInference Time Value Inference required for STRING Cell Type evaluation
+ */
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell can be null
+ * @param fieldName Cell field name for tracking in Field Types required
+ * @param fieldTypes Map of Field Name to Field Type Inference information required
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell can be null
+ * @return Record Data Type or null
+ */
Review Comment:
Is it necessary to duplicate the parameter javadoc information from the interface? I see the descriptions are different as they add information which I can understand why they are there (unless it would make sense to pull this information to the interface also).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on PR #9379:
URL: https://github.com/apache/nifi/pull/9379#issuecomment-2414744976
> @exceptionfactory With that change should we should also then improve what is returned by ExcelRecordReader on line 146. Currently we return a `java.util.Date` with `cell.getDateCellValue()` but I think we should employ similar logic to determine if its a DATE/TIME then call `cell.getLocalDateTimeCellValue()` to return a `java.time.LocalDateTime`.
>
> e.g.
>
> ```
> case NUMERIC -> {
> if (DateUtil.isCellDateFormatted(cell)) {
> final double numericCellValue = cell.getNumericCellValue();
> final long roundedCellValue = (long) numericCellValue;
> if ((numericCellValue > 0 && numericCellValue < 1) || (roundedCellValue == numericCellValue)) {
> yield cell.getLocalDateTimeCellValue();
> } else {
> yield cell.getDateCellValue();
> }
> } else {
> yield cell.getNumericCellValue();
> }
> }
> ```
The `java.util.Date` includes both Date and Time, so that is why it is functionally equivalent to the return value of `getLocalDateTimeCellValue()` in this scenario.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on PR #9379:
URL: https://github.com/apache/nifi/pull/9379#issuecomment-2414943821
@exceptionfactory Thank you for making all the requested changes. Also thank you for including me as a coauthor Everything looks good now and seems to be producing the correct results. LGTM+1. I plan to merge as soon as the CI build completes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1801655177
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getStringCellValue();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
Review Comment:
> any value that would otherwise be considered a STRING type
That part is bothering me as until now that was everything but now any column whose Excel format is date is not included in this. I am not sure users will understand that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1799837218
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/test/java/org/apache/nifi/excel/TestExcelSchemaInference.java:
##########
@@ -50,69 +45,54 @@ public class TestExcelSchemaInference {
private static final String EXPECTED_SECOND_FIELD_NAME = ExcelUtils.FIELD_NAME_PREFIX + "1";
private static final String EXPECTED_THIRD_FIELD_NAME = ExcelUtils.FIELD_NAME_PREFIX + "2";
private static final String EXPECTED_FOURTH_FIELD_NAME = ExcelUtils.FIELD_NAME_PREFIX + "3";
+
+ private static final String SIMPLE_FORMATTING_PATH = "/excel/simpleDataFormatting.xlsx";
+
private final TimeValueInference timestampInference = new TimeValueInference("MM/dd/yyyy", "HH:mm:ss.SSS", "yyyy/MM/dd/ HH:mm");
@Mock
ComponentLog logger;
- @ParameterizedTest
- @MethodSource("getLocales")
- public void testInferenceAgainstDifferentLocales(Locale locale) throws IOException {
- final Map<PropertyDescriptor, String> properties = new HashMap<>();
- new ExcelReader().getSupportedPropertyDescriptors().forEach(prop -> properties.put(prop, prop.getDefaultValue()));
- final PropertyContext context = new MockConfigurationContext(properties, null, null);
-
- try (final InputStream inputStream = getResourceStream("/excel/numbers.xlsx")) {
- final InferSchemaAccessStrategy<?> accessStrategy = new InferSchemaAccessStrategy<>(
- (variables, content) -> new ExcelRecordSource(content, context, variables, logger),
- new ExcelSchemaInference(timestampInference, locale), logger);
- final RecordSchema schema = accessStrategy.getSchema(null, inputStream, null);
- final List<String> fieldNames = schema.getFieldNames();
- assertEquals(Collections.singletonList(EXPECTED_FIRST_FIELD_NAME), fieldNames);
-
- if (Locale.FRENCH.equals(locale)) {
- assertEquals(RecordFieldType.STRING, schema.getDataType(EXPECTED_FIRST_FIELD_NAME).get().getFieldType());
- } else {
- assertEquals(RecordFieldType.CHOICE.getChoiceDataType(RecordFieldType.FLOAT.getDataType(), RecordFieldType.STRING.getDataType()),
- schema.getDataType(EXPECTED_FIRST_FIELD_NAME).get());
- }
- }
- }
-
- private static Stream<Arguments> getLocales() {
- Locale hindi = Locale.of("hin");
- return Stream.of(
- Arguments.of(Locale.ENGLISH),
- Arguments.of(hindi),
- Arguments.of(Locale.JAPANESE),
- Arguments.of(Locale.FRENCH)
- );
- }
-
@Test
public void testInferenceIncludesAllRecords() throws IOException {
final Map<PropertyDescriptor, String> properties = new HashMap<>();
new ExcelReader().getSupportedPropertyDescriptors().forEach(prop -> properties.put(prop, prop.getDefaultValue()));
final PropertyContext context = new MockConfigurationContext(properties, null, null);
final RecordSchema schema;
- try (final InputStream inputStream = getResourceStream("/excel/simpleDataFormatting.xlsx")) {
+ try (final InputStream inputStream = getResourceStream()) {
final InferSchemaAccessStrategy<?> accessStrategy = new InferSchemaAccessStrategy<>(
(variables, content) -> new ExcelRecordSource(content, context, variables, logger),
new ExcelSchemaInference(timestampInference), Mockito.mock(ComponentLog.class));
Review Comment:
No need to call `Mockito.mock(ComponentLog.class)` as we already have an instance of ComponentLog annotated with the `@Mock` annotation.
```suggestion
new ExcelSchemaInference(timestampInference), logger);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1801603367
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getStringCellValue();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
Review Comment:
Also it might be worth documenting that when inferring a schema, the schema will be limited to the following specific types BOOLEAN, DOUBLE, LONG, STRING, TIMESTAMP and perhaps DATE and TIME.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1799709041
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getStringCellValue();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
+ dataType = timeDataType.orElse(RecordFieldType.STRING.getDataType());
+ } else if (CellType.FORMULA == cellType) {
+ dataType = getFormulaResultDataType(cell);
+ } else {
+ // Default to null for known and unknown Cell Types: BLANK, ERROR
+ dataType = null;
+ }
+
+ return dataType;
+ }
+
+ private DataType getFormulaResultDataType(final Cell cell) {
+ final DataType dataType;
+
+ final CellType formulaResultType = cell.getCachedFormulaResultType();
+ if (CellType.BOOLEAN == formulaResultType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == formulaResultType) {
+ dataType = RecordFieldType.STRING.getDataType();
+ } else if (CellType.NUMERIC == formulaResultType) {
+ // Date Formatting check limited to NUMERIC Cell Types without Conditional Formatting Evaluator
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ } else {
+ // Default to null for known and unknown Formula Result Cell Types: BLANK, ERROR
+ dataType = null;
+ }
Review Comment:
The `null` return is valid, and the Field Type Inference class simply ignores it. In that, case, the inferred type is effectively "unknown", however, the standard behavior of record type inferencing falls back to `STRING`. Since `null` is valid as a return type, this effectively indicates that the type is unknown or unable to be determined by the inferencing process.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on PR #9379:
URL: https://github.com/apache/nifi/pull/9379#issuecomment-2414917424
> > Also I noticed even with an explicit schema if we do not get the formatted form, an exception can be thrown when reading with `ExcelRecordReader`. The case in point I see is when the schema defines a boolean and the column returns a boolean from a formula. Without the formatted form, `1.0` is returned which will not be interpreted as a boolean in Java. With the formatted form, `TRUE` is returned which can interpreted as a boolean in Java.
>
> This sounds like a case where the cached formula result type is not returning `BOOLEAN` and is instead returning `NUMERIC` or `STRING`.
Its actually returning a NUMERIC a double to be precise. From your latest changes the corresponding field in the schema is producing
`{"name":"Boolean_Function","type":["double","null"]}`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on PR #9379:
URL: https://github.com/apache/nifi/pull/9379#issuecomment-2416892429
> I plan to merge this as soon as the CI build finishes (for real this time :) ). In regards to to 1.x would you like me to make a separate PR to have these changes merged into the 1.x branch?
Thanks! I'll put together a separate PR for the support branch today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "asfgit (via GitHub)" <gi...@apache.org>.
asfgit closed pull request #9379: NIFI-13744 Correct Excel Reader Cell Type Inferencing
URL: https://github.com/apache/nifi/pull/9379
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on PR #9379:
URL: https://github.com/apache/nifi/pull/9379#issuecomment-2417292231
Yeah I saw. One failed so I restarted it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1797226109
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getRichStringCellValue().getString();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
+ dataType = timeDataType.orElse(RecordFieldType.STRING.getDataType());
+ } else if (CellType.FORMULA == cellType) {
+ dataType = RecordFieldType.STRING.getDataType();
+ } else {
Review Comment:
That makes sense, thanks for clarifying those changes. In that case, it does sound like changing both the Reader itself, and the inference approach, is warranted. I'll take a closer look at that and follow up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1800085624
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
Review Comment:
I do not think defaulting to `TIMESTAMP` is right for every case. In the `TestExcelReader` I have in my PR I have three columns
1. `Date_Custom` which has a date value 07/25/2024 (displayed as `07/25/24`)
2. `Time_Custom` which has a time value of 04:00:00 PM (displayed as `16:00`)
3. `Timestamp` which has a timestamp 07/26/2024 16:00:00
When running through your code they are parsed and look like the following when calling `toString` on the `java.sql.Timestamp` returned from the `Record` object.
1. 2024-07-05 00:00:00.0
2. 1899-12-31 00:00:00.0
3. 2024-07-26 16:00:00.0
For dates and timestamp 1 and 3 are okay but for those fields which only represent time, I do not think users are interested in the default date Excel has for time 1899-12-31. I think there has to be a more fine grained way of determining whether the date we have is representing a date or only time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1799582387
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getStringCellValue();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
+ dataType = timeDataType.orElse(RecordFieldType.STRING.getDataType());
+ } else if (CellType.FORMULA == cellType) {
+ dataType = getFormulaResultDataType(cell);
+ } else {
+ // Default to null for known and unknown Cell Types: BLANK, ERROR
+ dataType = null;
+ }
+
+ return dataType;
+ }
+
+ private DataType getFormulaResultDataType(final Cell cell) {
+ final DataType dataType;
+
+ final CellType formulaResultType = cell.getCachedFormulaResultType();
+ if (CellType.BOOLEAN == formulaResultType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == formulaResultType) {
+ dataType = RecordFieldType.STRING.getDataType();
+ } else if (CellType.NUMERIC == formulaResultType) {
+ // Date Formatting check limited to NUMERIC Cell Types without Conditional Formatting Evaluator
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ } else {
+ // Default to null for known and unknown Formula Result Cell Types: BLANK, ERROR
+ dataType = null;
+ }
Review Comment:
I didn't realize null type is an option. Doesn't everything need to have an associated type?
Furthermore it is possible to to have a string value of the error which end users may or may not be interested in. I had in my PR two examples in file named `all_types.xlsx` in columns named `Variable_Missing` and `Syntax_Error` with values `Err:511` and `#NAME?` . Hence perhaps ERROR should return `RecordFieldType.STRING.getDataType()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1799582387
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getStringCellValue();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
+ dataType = timeDataType.orElse(RecordFieldType.STRING.getDataType());
+ } else if (CellType.FORMULA == cellType) {
+ dataType = getFormulaResultDataType(cell);
+ } else {
+ // Default to null for known and unknown Cell Types: BLANK, ERROR
+ dataType = null;
+ }
+
+ return dataType;
+ }
+
+ private DataType getFormulaResultDataType(final Cell cell) {
+ final DataType dataType;
+
+ final CellType formulaResultType = cell.getCachedFormulaResultType();
+ if (CellType.BOOLEAN == formulaResultType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == formulaResultType) {
+ dataType = RecordFieldType.STRING.getDataType();
+ } else if (CellType.NUMERIC == formulaResultType) {
+ // Date Formatting check limited to NUMERIC Cell Types without Conditional Formatting Evaluator
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ } else {
+ // Default to null for known and unknown Formula Result Cell Types: BLANK, ERROR
+ dataType = null;
+ }
Review Comment:
I didn't realize null type is an option. Doesn't everything need to have an associated type?
Furthermore it is possible to to have a string value of the error which end users may or may not be interested in. I had in my PR two examples in file named `all_types.xlsx` in columns named `Variable_Missing` and `Syntax_Error` with values `Err:511` and `#NAME?` .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1799506390
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getStringCellValue();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
Review Comment:
Is it necessary to do this? I thought all dates in Excel are stored as a number as detailed in the code above lines 73-77 and in the [javadocs](https://poi.apache.org/apidocs/5.0/org/apache/poi/ss/usermodel/DateUtil.html#isCellDateFormatted-org.apache.poi.ss.usermodel.Cell-). I would think the `TimeValueInference` is no longer necessary as we should always know whether a cell is a date for not.
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getStringCellValue();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
+ dataType = timeDataType.orElse(RecordFieldType.STRING.getDataType());
+ } else if (CellType.FORMULA == cellType) {
+ dataType = getFormulaResultDataType(cell);
+ } else {
+ // Default to null for known and unknown Cell Types: BLANK, ERROR
+ dataType = null;
+ }
+
+ return dataType;
+ }
+
+ private DataType getFormulaResultDataType(final Cell cell) {
+ final DataType dataType;
+
+ final CellType formulaResultType = cell.getCachedFormulaResultType();
+ if (CellType.BOOLEAN == formulaResultType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == formulaResultType) {
+ dataType = RecordFieldType.STRING.getDataType();
+ } else if (CellType.NUMERIC == formulaResultType) {
+ // Date Formatting check limited to NUMERIC Cell Types without Conditional Formatting Evaluator
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ } else {
+ // Default to null for known and unknown Formula Result Cell Types: BLANK, ERROR
+ dataType = null;
+ }
Review Comment:
I didn't realize null type is an option. Doesn't everything need to have an associated type?
Furthermore it is possible to to have a string value of the error which end users may or may not be interested in.
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getStringCellValue();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
+ dataType = timeDataType.orElse(RecordFieldType.STRING.getDataType());
+ } else if (CellType.FORMULA == cellType) {
+ dataType = getFormulaResultDataType(cell);
+ } else {
+ // Default to null for known and unknown Cell Types: BLANK, ERROR
+ dataType = null;
+ }
+
+ return dataType;
+ }
+
+ private DataType getFormulaResultDataType(final Cell cell) {
+ final DataType dataType;
+
+ final CellType formulaResultType = cell.getCachedFormulaResultType();
+ if (CellType.BOOLEAN == formulaResultType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == formulaResultType) {
+ dataType = RecordFieldType.STRING.getDataType();
+ } else if (CellType.NUMERIC == formulaResultType) {
+ // Date Formatting check limited to NUMERIC Cell Types without Conditional Formatting Evaluator
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
Review Comment:
Shouldn't there be an attempt to determine if the resulting number is a long as done earlier on lines 80-82?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1800013904
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
Review Comment:
I do not think there should be this check as there can be cells which have no data in it and `Cell `will be null. I uncovered this running the TestEsxcelReader which I had created for my PR.
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on PR #9379:
URL: https://github.com/apache/nifi/pull/9379#issuecomment-2414325626
> > Also I noticed even with an explicit schema if we do not get the formatted form, an exception can be thrown when reading with `ExcelRecordReader`. The case in point I see is when the schema defines a boolean and the column returns a boolean from a formula. Without the formatted form, `1.0` is returned which will not be interpreted as a boolean in Java. With the formatted form, `TRUE` is returned which can interpreted as a boolean in Java.
>
> This sounds like a case where the cached formula result type is not returning `BOOLEAN` and is instead returning `NUMERIC` or `STRING`.
Is there anything to do for this case or leave it as is?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1799748937
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getStringCellValue();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
Review Comment:
I think though this piece should be documented that any time or date format properties specified by the user (stored in `TimeValueInference`) will **_only_** be used to parse those cells whose cell type is string and not whose already is date (for the end user this may be understood if the Excel format type is text and not date or time) since the current behavior utilizes the any time or date format properties specified by the user to determine the data type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1800148592
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
Review Comment:
I thought though the user entered properties can/should assist the inference logic in help determining whether a field is a date, time or timestamp as I believe they are used in other readers for inferring.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1799888718
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getStringCellValue();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
+ dataType = timeDataType.orElse(RecordFieldType.STRING.getDataType());
+ } else if (CellType.FORMULA == cellType) {
+ dataType = getFormulaResultDataType(cell);
+ } else {
+ // Default to null for known and unknown Cell Types: BLANK, ERROR
+ dataType = null;
+ }
+
+ return dataType;
+ }
+
+ private DataType getFormulaResultDataType(final Cell cell) {
+ final DataType dataType;
+
+ final CellType formulaResultType = cell.getCachedFormulaResultType();
+ if (CellType.BOOLEAN == formulaResultType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == formulaResultType) {
+ dataType = RecordFieldType.STRING.getDataType();
+ } else if (CellType.NUMERIC == formulaResultType) {
+ // Date Formatting check limited to NUMERIC Cell Types without Conditional Formatting Evaluator
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ } else {
+ // Default to null for known and unknown Formula Result Cell Types: BLANK, ERROR
Review Comment:
Per `getCachedFormulaResultType` [javadocs](https://poi.apache.org/apidocs/5.0/org/apache/poi/ss/usermodel/Cell.html#getCachedFormulaResultType--) there is no BLANK cell type for cached formula cell type
```suggestion
// Default to null for known and unknown Formula Result Cell Types: ERROR
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1799886701
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getStringCellValue();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
+ dataType = timeDataType.orElse(RecordFieldType.STRING.getDataType());
+ } else if (CellType.FORMULA == cellType) {
+ dataType = getFormulaResultDataType(cell);
+ } else {
+ // Default to null for known and unknown Cell Types: BLANK, ERROR
Review Comment:
Per `getCachedFormulaResultType` [javadocs](https://poi.apache.org/apidocs/5.0/org/apache/poi/ss/usermodel/Cell.html#getCachedFormulaResultType--) there is no BLANK cell type for formulas
```suggestion
// Default to null for known and unknown Cell Type: ERROR
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on PR #9379:
URL: https://github.com/apache/nifi/pull/9379#issuecomment-2416867705
I plan to merge this as soon as the CI build finishes (for real this time :) ). In regards to to 1.x would you like me to make a separate PR to have these changes merged into the 1.x branch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on PR #9379:
URL: https://github.com/apache/nifi/pull/9379#issuecomment-2414746095
@dan-s1 I pushed a change with the additional `Date` and `Time` inference handling:
https://github.com/apache/nifi/pull/9379/commits/bab137b35f3dbdf249397d24bf75ec824226b292
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1802089085
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ /**
+ * Standard Cell Field Type Reader constructor with Time Value Inference for handling STRING Cell Types that may contain values with Timestamps
+ *
+ * @param timeValueInference Time Value Inference required for STRING Cell Type evaluation
+ */
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell can be null
+ * @param fieldName Cell field name for tracking in Field Types required
+ * @param fieldTypes Map of Field Name to Field Type Inference information required
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell can be null
+ * @return Record Data Type or null
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ if (cell == null) {
+ return null;
+ }
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ dataType = getDateTimeDataType(cell);
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getStringCellValue();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
+ dataType = timeDataType.orElse(RecordFieldType.STRING.getDataType());
+ } else if (CellType.FORMULA == cellType) {
+ dataType = getFormulaResultDataType(cell);
+ } else {
+ // Default to null for known and unknown Cell Types
+ dataType = null;
+ }
+
+ return dataType;
+ }
+
+ private DataType getDateTimeDataType(final Cell cell) {
+ final DataType dataType;
+
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ // Numbers without decimal fractions indicate Dates without Times
+ dataType = RecordFieldType.DATE.getDataType();
+ } else if (numericCellValue < 1) {
+ // Decimal fractions indicate Times without Dates
+ dataType = RecordFieldType.TIME.getDataType();
+ } else {
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ }
+
+ return dataType;
+ }
+
+ private DataType getFormulaResultDataType(final Cell cell) {
+ final DataType dataType;
+
+ final CellType formulaResultType = cell.getCachedFormulaResultType();
+ if (CellType.BOOLEAN == formulaResultType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == formulaResultType) {
+ dataType = RecordFieldType.STRING.getDataType();
+ } else if (CellType.NUMERIC == formulaResultType) {
+ // Date Formatting check limited to NUMERIC Cell Types without Conditional Formatting Evaluator
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
Review Comment:
With a closer look at at the POI's `DataFormatter` code, specifically lines 513-522
```
synchronized(dateFormat) {
if (dateFormat instanceof ExcelStyleDateFormatter) {
((ExcelStyleDateFormatter)dateFormat).setDateToBeFormatted(cell.getNumericCellValue());
}
Date d = cell.getDateCellValue();
return this.performDateFormatting(d, (Format)dateFormat);
}
```
it seems it gets the cell's numeric or date value. If that is the case then I assume you can reuse the `getDateTimeDataType` above to determine the precise date type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on PR #9379:
URL: https://github.com/apache/nifi/pull/9379#issuecomment-2416855822
Thanks @dan-s1, I pushed an update with those recommended changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1797219608
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getRichStringCellValue().getString();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
+ dataType = timeDataType.orElse(RecordFieldType.STRING.getDataType());
+ } else if (CellType.FORMULA == cellType) {
+ dataType = RecordFieldType.STRING.getDataType();
+ } else {
Review Comment:
In my PR I was trying to correct that logic as I had not realized `Cell.getStringCellValue()` get the text of the formula. That is why I had added to ExcelUtils
```
private static final DataFormatter DATA_FORMATTER = new DataFormatter(Locale.getDefault());
static {
DATA_FORMATTER.setUseCachedValuesForFormulaCells(true);
}
```
as DataFormatter can obtain the formula values after calling `setUseCachedValuesForFormulaCells(true)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1797219608
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getRichStringCellValue().getString();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
+ dataType = timeDataType.orElse(RecordFieldType.STRING.getDataType());
+ } else if (CellType.FORMULA == cellType) {
+ dataType = RecordFieldType.STRING.getDataType();
+ } else {
Review Comment:
In my PR I was trying to correct that logic as I had not realized `Cell.getStringCellValue()` gets the text of the formula. That is why I had added to ExcelUtils
```
private static final DataFormatter DATA_FORMATTER = new DataFormatter(Locale.getDefault());
static {
DATA_FORMATTER.setUseCachedValuesForFormulaCells(true);
}
```
as DataFormatter can obtain the formula values after calling `setUseCachedValuesForFormulaCells(true)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1797214036
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getRichStringCellValue().getString();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
+ dataType = timeDataType.orElse(RecordFieldType.STRING.getDataType());
+ } else if (CellType.FORMULA == cellType) {
+ dataType = RecordFieldType.STRING.getDataType();
+ } else {
Review Comment:
That's a good point about the evaluated value of the cell based on the formula. The current `ExcelRecordReader.getCellValue()` method calls `Cell.getStringCellValue()` for the `FORMULA` Cell Type, so this behavior aligns with that approach. This could be changed and improved, but that might make sense as a separate change. What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1800013904
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
Review Comment:
I do not think there should be this check as there can be cells which have no data in it and `Cell `will be null. I uncovered this running the `TestExcelReader` which I had created for my PR.
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on PR #9379:
URL: https://github.com/apache/nifi/pull/9379#issuecomment-2414619856
> > > Also I noticed even with an explicit schema if we do not get the formatted form, an exception can be thrown when reading with `ExcelRecordReader`. The case in point I see is when the schema defines a boolean and the column returns a boolean from a formula. Without the formatted form, `1.0` is returned which will not be interpreted as a boolean in Java. With the formatted form, `TRUE` is returned which can interpreted as a boolean in Java.
> >
> >
> > This sounds like a case where the cached formula result type is not returning `BOOLEAN` and is instead returning `NUMERIC` or `STRING`.
>
> Is there anything to do for this case or leave it as is?
In this case, it sounds like an explicit schema should be configured with a `STRING` type instead of a `BOOLEAN`. The behavior of converting number `1.0` to a string `TRUE` seems a bit difficult to generalize unless there is some other fact we can use to handle the value as a boolean.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1799582387
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getStringCellValue();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
+ dataType = timeDataType.orElse(RecordFieldType.STRING.getDataType());
+ } else if (CellType.FORMULA == cellType) {
+ dataType = getFormulaResultDataType(cell);
+ } else {
+ // Default to null for known and unknown Cell Types: BLANK, ERROR
+ dataType = null;
+ }
+
+ return dataType;
+ }
+
+ private DataType getFormulaResultDataType(final Cell cell) {
+ final DataType dataType;
+
+ final CellType formulaResultType = cell.getCachedFormulaResultType();
+ if (CellType.BOOLEAN == formulaResultType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == formulaResultType) {
+ dataType = RecordFieldType.STRING.getDataType();
+ } else if (CellType.NUMERIC == formulaResultType) {
+ // Date Formatting check limited to NUMERIC Cell Types without Conditional Formatting Evaluator
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ } else {
+ // Default to null for known and unknown Formula Result Cell Types: BLANK, ERROR
+ dataType = null;
+ }
Review Comment:
I didn't realize null type is an option. Doesn't everything need to have an associated type?
Furthermore it is possible to to have a string value of the error which end users may or may not be interested in. I had in my PR two examples in file named `all_types.xlsx` in columns `Variable_Missing` and `Syntax_Error` with values `Err:511` and `#NAME?`
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getStringCellValue();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
+ dataType = timeDataType.orElse(RecordFieldType.STRING.getDataType());
+ } else if (CellType.FORMULA == cellType) {
+ dataType = getFormulaResultDataType(cell);
+ } else {
+ // Default to null for known and unknown Cell Types: BLANK, ERROR
+ dataType = null;
+ }
+
+ return dataType;
+ }
+
+ private DataType getFormulaResultDataType(final Cell cell) {
+ final DataType dataType;
+
+ final CellType formulaResultType = cell.getCachedFormulaResultType();
+ if (CellType.BOOLEAN == formulaResultType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == formulaResultType) {
+ dataType = RecordFieldType.STRING.getDataType();
+ } else if (CellType.NUMERIC == formulaResultType) {
+ // Date Formatting check limited to NUMERIC Cell Types without Conditional Formatting Evaluator
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ } else {
+ // Default to null for known and unknown Formula Result Cell Types: BLANK, ERROR
+ dataType = null;
+ }
Review Comment:
I didn't realize null type is an option. Doesn't everything need to have an associated type?
Furthermore it is possible to to have a string value of the error which end users may or may not be interested in. I had in my PR two examples in file named `all_types.xlsx` in columns `Variable_Missing` and `Syntax_Error` with values `Err:511` and `#NAME?` .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1799748937
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getStringCellValue();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
Review Comment:
I think though this piece should be documented that any time or date format properties specified by the user (stored in `TimeValueInference`) will **_only_** be used to parse those cells whose cell type is string and not whose already is date.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on PR #9379:
URL: https://github.com/apache/nifi/pull/9379#issuecomment-2412419006
Also I noticed even with an explicit schema if we do not get the formatted form there is an exception. The case in point I see is when the schema defines a boolean and the column returns a boolean from a formula. Without the formatted form 1.0 is returned which will not be interpreted as a boolean in Java With the formatted form TRUE is returned which can interpreted as a boolean in Java.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1799748937
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getStringCellValue();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
Review Comment:
I think though this piece should be documented that any time or date format properties specified by the user (stored in `TimeValueInference`) will **_only_** be used to parse those cells whose cell type is string and not whose already is date (for the end user this may be understood is if the Excel format type is text and not date)..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on PR #9379:
URL: https://github.com/apache/nifi/pull/9379#issuecomment-2414288784
Thanks for the thorough review @dan-s1.
I pushed an update to address the comments. I updated comments in several places, adding handling for `null` Cells, and added a new unit test method to exercise the Timestamp inference behavior for `STRING` Cell Types.
I marked several threads as resolved, but if there are any where you think additional feedback is needed, feel free to unresolve and continue the conversation. Otherwise, hopefully we can wrap this up soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1803056675
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ /**
+ * Standard Cell Field Type Reader constructor with Time Value Inference for handling STRING Cell Types that may contain values with Timestamps
+ *
+ * @param timeValueInference Time Value Inference required for STRING Cell Type evaluation
+ */
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell can be null
+ * @param fieldName Cell field name for tracking in Field Types required
+ * @param fieldTypes Map of Field Name to Field Type Inference information required
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell can be null
+ * @return Record Data Type or null
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ if (cell == null) {
+ return null;
+ }
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ final double numericCellValue = cell.getNumericCellValue();
+ dataType = getDateTimeDataType(numericCellValue);
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
Review Comment:
This code is duplicated on 114-115. Perhaps make a reusable method and use it in both places e.g.
```
private boolean isWholeNumber(final double numericCellValue) {
return numericCellValue == (long)numericCellValue;
}
```
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ /**
+ * Standard Cell Field Type Reader constructor with Time Value Inference for handling STRING Cell Types that may contain values with Timestamps
+ *
+ * @param timeValueInference Time Value Inference required for STRING Cell Type evaluation
+ */
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell can be null
+ * @param fieldName Cell field name for tracking in Field Types required
+ * @param fieldTypes Map of Field Name to Field Type Inference information required
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell can be null
+ * @return Record Data Type or null
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ if (cell == null) {
+ return null;
+ }
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ final double numericCellValue = cell.getNumericCellValue();
+ dataType = getDateTimeDataType(numericCellValue);
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
Review Comment:
No need to define `numericCellValue` twice since its used in both branches.
```suggestion
final double numericCellValue = cell.getNumericCellValue();
// Date Formatting check limited to NUMERIC Cell Types
if (DateUtil.isCellDateFormatted(cell)) {
dataType = getDateTimeDataType(numericCellValue);
} else {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1800148592
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
Review Comment:
I thought though the user entered properties can/should assist the inference logic in help determining whether a field is a date, time or timestamp as I believe they are used in other readers for inferring. They can be used if dates are converted to a string as they are displayed with the `Dataformatter`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on PR #9379:
URL: https://github.com/apache/nifi/pull/9379#issuecomment-2414724298
@exceptionfactory With that change should we also then improve what is returned by ExcelRecordReader on line 146?
Currently we a a `java.util.Date` with `cell.getDateCellValue()` but I think we should employ similar logic to determine if its a DATE/TIME then call `cell.getLocalDateTimeCellValue()` to return a `java.time.LocalDateTime`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1802031090
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ /**
+ * Standard Cell Field Type Reader constructor with Time Value Inference for handling STRING Cell Types that may contain values with Timestamps
+ *
+ * @param timeValueInference Time Value Inference required for STRING Cell Type evaluation
+ */
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell can be null
+ * @param fieldName Cell field name for tracking in Field Types required
+ * @param fieldTypes Map of Field Name to Field Type Inference information required
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell can be null
+ * @return Record Data Type or null
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ if (cell == null) {
+ return null;
+ }
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ dataType = getDateTimeDataType(cell);
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getStringCellValue();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
+ dataType = timeDataType.orElse(RecordFieldType.STRING.getDataType());
+ } else if (CellType.FORMULA == cellType) {
+ dataType = getFormulaResultDataType(cell);
+ } else {
+ // Default to null for known and unknown Cell Types
+ dataType = null;
+ }
+
+ return dataType;
+ }
+
+ private DataType getDateTimeDataType(final Cell cell) {
+ final DataType dataType;
+
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ // Numbers without decimal fractions indicate Dates without Times
+ dataType = RecordFieldType.DATE.getDataType();
+ } else if (numericCellValue < 1) {
+ // Decimal fractions indicate Times without Dates
+ dataType = RecordFieldType.TIME.getDataType();
+ } else {
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ }
+
+ return dataType;
+ }
+
+ private DataType getFormulaResultDataType(final Cell cell) {
+ final DataType dataType;
+
+ final CellType formulaResultType = cell.getCachedFormulaResultType();
+ if (CellType.BOOLEAN == formulaResultType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == formulaResultType) {
+ dataType = RecordFieldType.STRING.getDataType();
+ } else if (CellType.NUMERIC == formulaResultType) {
+ // Date Formatting check limited to NUMERIC Cell Types without Conditional Formatting Evaluator
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
Review Comment:
I do not see any methods in `org.apache.poi.ss.usermodel.Cell` which can get the cached formula result but do we leave this as `TIMESTAMP` or somehow try to get the result and determine whether its a `DATE` or `TIME` like we did above? I think we would need to use a `DateFormatter` to obtain those results.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1799748937
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell required
+ * @param fieldName Cell field name for tracking in Field Types
+ * @param fieldTypes Map of Field Name to Field Type Inference information
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(cell, "Cell required");
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell required
+ * @return Record Data Type
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ Objects.requireNonNull(cell, "Cell required");
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getStringCellValue();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
Review Comment:
I think though this piece should be documented that any time or date format properties specified by the user (stored in `TimeValueInference`) will **_only_** be used to parse those cells whose cell type is string and not whose already is date (for the end user this may be understood if the Excel format type is text and not date or time).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1802031090
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ /**
+ * Standard Cell Field Type Reader constructor with Time Value Inference for handling STRING Cell Types that may contain values with Timestamps
+ *
+ * @param timeValueInference Time Value Inference required for STRING Cell Type evaluation
+ */
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell can be null
+ * @param fieldName Cell field name for tracking in Field Types required
+ * @param fieldTypes Map of Field Name to Field Type Inference information required
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell can be null
+ * @return Record Data Type or null
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ if (cell == null) {
+ return null;
+ }
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ dataType = getDateTimeDataType(cell);
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getStringCellValue();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
+ dataType = timeDataType.orElse(RecordFieldType.STRING.getDataType());
+ } else if (CellType.FORMULA == cellType) {
+ dataType = getFormulaResultDataType(cell);
+ } else {
+ // Default to null for known and unknown Cell Types
+ dataType = null;
+ }
+
+ return dataType;
+ }
+
+ private DataType getDateTimeDataType(final Cell cell) {
+ final DataType dataType;
+
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ // Numbers without decimal fractions indicate Dates without Times
+ dataType = RecordFieldType.DATE.getDataType();
+ } else if (numericCellValue < 1) {
+ // Decimal fractions indicate Times without Dates
+ dataType = RecordFieldType.TIME.getDataType();
+ } else {
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ }
+
+ return dataType;
+ }
+
+ private DataType getFormulaResultDataType(final Cell cell) {
+ final DataType dataType;
+
+ final CellType formulaResultType = cell.getCachedFormulaResultType();
+ if (CellType.BOOLEAN == formulaResultType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == formulaResultType) {
+ dataType = RecordFieldType.STRING.getDataType();
+ } else if (CellType.NUMERIC == formulaResultType) {
+ // Date Formatting check limited to NUMERIC Cell Types without Conditional Formatting Evaluator
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
Review Comment:
I do not see any methods in `org.apache.poi.ss.usermodel.Cell` which can get the cached formula result but do we leave this as `TIMESTAMP` or somehow try to get the result and determine whether its a `DATE` or `TIME` like we did above? I think we might need to use a `DateFormatter` to obtain those results.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-13744 Correct Excel Reader Cell Type Inferencing [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #9379:
URL: https://github.com/apache/nifi/pull/9379#discussion_r1802057764
##########
nifi-extension-bundles/nifi-poi-bundle/nifi-poi-services/src/main/java/org/apache/nifi/excel/StandardCellFieldTypeReader.java:
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.excel;
+
+import org.apache.nifi.schema.inference.FieldTypeInference;
+import org.apache.nifi.schema.inference.TimeValueInference;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellType;
+import org.apache.poi.ss.usermodel.DateUtil;
+
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Standard implementation of Cell Field Type Reader uses Cell Type and Cell Value information with inference based on Timestamp formats
+ */
+class StandardCellFieldTypeReader implements CellFieldTypeReader {
+ private final TimeValueInference timeValueInference;
+
+ /**
+ * Standard Cell Field Type Reader constructor with Time Value Inference for handling STRING Cell Types that may contain values with Timestamps
+ *
+ * @param timeValueInference Time Value Inference required for STRING Cell Type evaluation
+ */
+ StandardCellFieldTypeReader(final TimeValueInference timeValueInference) {
+ this.timeValueInference = Objects.requireNonNull(timeValueInference, "Time Value Inference required");
+ }
+
+ /**
+ * Infer Cell Field Type and append possible Data Types to mapped Field Type Inference information
+ *
+ * @param cell Spreadsheet Cell can be null
+ * @param fieldName Cell field name for tracking in Field Types required
+ * @param fieldTypes Map of Field Name to Field Type Inference information required
+ */
+ @Override
+ public void inferCellFieldType(final Cell cell, final String fieldName, final Map<String, FieldTypeInference> fieldTypes) {
+ Objects.requireNonNull(fieldName, "Field Name required");
+ Objects.requireNonNull(fieldTypes, "Field Types required");
+
+ final FieldTypeInference fieldTypeInference = fieldTypes.computeIfAbsent(fieldName, key -> new FieldTypeInference());
+ final DataType dataType = getCellDataType(cell);
+ fieldTypeInference.addPossibleDataType(dataType);
+ }
+
+ /**
+ * Get Record Data Type from Spreadsheet Cell Type and additional resolution of NUMERIC and STRING types
+ *
+ * @param cell Spreadsheet Cell can be null
+ * @return Record Data Type or null
+ */
+ @Override
+ public DataType getCellDataType(final Cell cell) {
+ if (cell == null) {
+ return null;
+ }
+
+ final CellType cellType = cell.getCellType();
+
+ final DataType dataType;
+
+ if (CellType.NUMERIC == cellType) {
+ // Date Formatting check limited to NUMERIC Cell Types
+ if (DateUtil.isCellDateFormatted(cell)) {
+ dataType = getDateTimeDataType(cell);
+ } else {
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ dataType = RecordFieldType.LONG.getDataType();
+ } else {
+ // Default to DOUBLE for NUMERIC values following cell.getNumericCellValue()
+ dataType = RecordFieldType.DOUBLE.getDataType();
+ }
+ }
+ } else if (CellType.BOOLEAN == cellType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == cellType) {
+ final String cellValue = cell.getStringCellValue();
+ // Attempt Time Value inference for STRING cell values
+ final Optional<DataType> timeDataType = timeValueInference.getDataType(cellValue);
+ dataType = timeDataType.orElse(RecordFieldType.STRING.getDataType());
+ } else if (CellType.FORMULA == cellType) {
+ dataType = getFormulaResultDataType(cell);
+ } else {
+ // Default to null for known and unknown Cell Types
+ dataType = null;
+ }
+
+ return dataType;
+ }
+
+ private DataType getDateTimeDataType(final Cell cell) {
+ final DataType dataType;
+
+ final double numericCellValue = cell.getNumericCellValue();
+ final long roundedCellValue = (long) numericCellValue;
+ if (roundedCellValue == numericCellValue) {
+ // Numbers without decimal fractions indicate Dates without Times
+ dataType = RecordFieldType.DATE.getDataType();
+ } else if (numericCellValue < 1) {
+ // Decimal fractions indicate Times without Dates
+ dataType = RecordFieldType.TIME.getDataType();
+ } else {
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
+ }
+
+ return dataType;
+ }
+
+ private DataType getFormulaResultDataType(final Cell cell) {
+ final DataType dataType;
+
+ final CellType formulaResultType = cell.getCachedFormulaResultType();
+ if (CellType.BOOLEAN == formulaResultType) {
+ dataType = RecordFieldType.BOOLEAN.getDataType();
+ } else if (CellType.STRING == formulaResultType) {
+ dataType = RecordFieldType.STRING.getDataType();
+ } else if (CellType.NUMERIC == formulaResultType) {
+ // Date Formatting check limited to NUMERIC Cell Types without Conditional Formatting Evaluator
+ if (DateUtil.isCellDateFormatted(cell)) {
+ // Default to TIMESTAMP for date formatted values using standard Date Time Pattern
+ dataType = RecordFieldType.TIMESTAMP.getDataType();
Review Comment:
I am not seeing anything directly, its probably best to leave as is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org