You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "ravinarayansingh (via GitHub)" <gi...@apache.org> on 2023/07/31 03:28:38 UTC
[GitHub] [nifi] ravinarayansingh opened a new pull request, #7544: NiFi 11858 Improve column name normalization in PutDatabaseRecord and UpdateDatabaseTable processor
ravinarayansingh opened a new pull request, #7544:
URL: https://github.com/apache/nifi/pull/7544
<!-- Licensed to the Apache Software Foundation (ASF) under one or more -->
<!-- contributor license agreements. See the NOTICE file distributed with -->
<!-- this work for additional information regarding copyright ownership. -->
<!-- The ASF licenses this file to You under the Apache License, Version 2.0 -->
<!-- (the "License"); you may not use this file except in compliance with -->
<!-- the License. You may obtain a copy of the License at -->
<!-- http://www.apache.org/licenses/LICENSE-2.0 -->
<!-- Unless required by applicable law or agreed to in writing, software -->
<!-- distributed under the License is distributed on an "AS IS" BASIS, -->
<!-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -->
<!-- See the License for the specific language governing permissions and -->
<!-- limitations under the License. -->
# Summary
[NiFi 11858](https://issues.apache.org/jira/browse/NIFI-11858)
# Tracking
Please complete the following tracking steps prior to pull request creation.
### Issue Tracking
- [ ] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created
### Pull Request Tracking
- [ ] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-00000`
- [ ] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-00000`
### Pull Request Formatting
- [ ] Pull Request based on current revision of the `main` branch
- [ ] Pull Request refers to a feature branch with one commit containing changes
# Verification
Please indicate the verification steps performed prior to pull request creation.
### Build
- [ ] Build completed using `mvn clean install -P contrib-check`
- [ ] JDK 17
### Licensing
- [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html)
- [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files
### Documentation
- [ ] Documentation formatting appears as expected in rendered files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11858 Configurable Column Name Normalization in PutDatabaseRecord and UpdateDatabaseTable [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on PR #7544:
URL: https://github.com/apache/nifi/pull/7544#issuecomment-1873402173
@exceptionfactory Can you please restart the 'ci-workflow / Windows Zulu JDK 21 FR` build again? The error does not seem to relate to the changes in the PR. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11858 Configurable Column Name Normalization in PutDatabaseRecord and UpdateDatabaseTable [nifi]
Posted by "mattyb149 (via GitHub)" <gi...@apache.org>.
mattyb149 commented on code in PR #7544:
URL: https://github.com/apache/nifi/pull/7544#discussion_r1636607648
##########
nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/ColumnDescription.java:
##########
@@ -84,9 +84,6 @@ public static ColumnDescription from(final ResultSet resultSet) throws SQLExcept
return new ColumnDescription(columnName, dataType, required, colSize == 0 ? null : colSize, isNullable);
}
- public static String normalizeColumnName(final String colName, final boolean translateColumnNames) {
- return colName == null ? null : (translateColumnNames ? colName.toUpperCase().replace("_", "") : colName);
Review Comment:
Nah it's not necessary, if there's a driver that returns null for a column name then the driver is plain wrong :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11858 Configurable Column Name Normalization in PutDatabaseRecord and UpdateDatabaseTable [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #7544:
URL: https://github.com/apache/nifi/pull/7544#discussion_r1438276400
##########
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/db/ColumnNameNormalizerUtilityTest.java:
##########
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard.db;
+
+import org.junit.jupiter.api.Test;
+
+import java.util.regex.Pattern;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+
+public class ColumnNameNormalizerUtilityTest {
+
+
+
+ @Test
+ void testNormalizingColumnName_RemoveUnderscore() {
+ String inputColumnName = "example_column_name";
+ String expectedNormalized = "EXAMPLECOLUMNNAME";
+ String normalized = ColumnNameNormalizerUtility.getNormalizedName(inputColumnName,true, TranslationStrategy.REMOVE_UNDERSCORE, null);
+
+ assertEquals(expectedNormalized, normalized);
+ }
+
+ @Test
+ void testNormalizingColumnName_RemoveSpace() {
+ String inputColumnName = "Column Name With Spaces";
+ String expectedNormalized = "COLUMNNAMEWITHSPACES";
+ String normalized = ColumnNameNormalizerUtility.getNormalizedName(inputColumnName,true, TranslationStrategy.REMOVE_SPACE, null);
+
+ assertEquals(expectedNormalized, normalized);
+ }
+
+ @Test
+ void testNormalizingColumnName_RemoveAllSpecialCharacters() {
+ String inputColumnName = "Special!Characters@Here$";
+ String expectedNormalized = "SPECIALCHARACTERSHERE";
+ String normalized = ColumnNameNormalizerUtility.getNormalizedName(inputColumnName,true, TranslationStrategy.REMOVE_ALL_SPECIAL_CHAR, null);
+
+ assertEquals(expectedNormalized, normalized);
+ }
+
+ @Test
+ void testNormalizingColumnName_Regex() {
+ String inputColumnName = "Your @Input -String Here";
+ Pattern translationPattern = Pattern.compile("[@-]");
+ String expectedNormalized = translationPattern.matcher(inputColumnName.toUpperCase()).replaceAll( "");
Review Comment:
Be consistent in this unit test as you did with the other unit tests by just specifying the actual value of the expected string (w/o it be being calculated).
```suggestion
String expectedNormalized = "YOUR INPUT STRING HERE";
```
##########
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/TranslationStrategy.java:
##########
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard.db;
+
+import org.apache.nifi.components.DescribedValue;
+
+/**
+ * Enumeration of supported Database column name Translation Strategy
+ */
+public enum TranslationStrategy implements DescribedValue {
+ REMOVE_UNDERSCORE {
+ @Override
+ public String getDisplayName() {
+ return "Remove Underscore";
+ }
+
+ @Override
+ public String getDescription() {
+ return "Underscore(_) will be removed from column name with empty string Ex. Pics_1_11 become PICS111";
+ }
+ },
+ REMOVE_SPACE {
+ @Override
+ public String getDisplayName() {
+ return "Remove Space";
+ }
+
+ @Override
+ public String getDescription() {
+ return "Spaces will be removed from column name with empty string Ex. 'User Name' become 'USERNAME'";
+ }
+ },
+
+ // REMOVE_ALL_SPECIAL_CHAR("REMOVE_ALL_SPECIAL_CHAR","Remove All Special Character","Remove All Special Character"),
Review Comment:
Remove commented out code
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11858 Configurable Column Name Normalization in PutDatabaseRecord and UpdateDatabaseTable [nifi]
Posted by "mattyb149 (via GitHub)" <gi...@apache.org>.
mattyb149 commented on PR #7544:
URL: https://github.com/apache/nifi/pull/7544#issuecomment-2130235297
The checkstyle rules were recently made more stringent, looks like this needs another rebase and please run your Maven build with the `contrib-check` profile activated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] ravinarayansingh commented on pull request #7544: NIFI-11858 Improve column name normalization in PutDatabaseRecord and UpdateDatabaseTable processor
Posted by "ravinarayansingh (via GitHub)" <gi...@apache.org>.
ravinarayansingh commented on PR #7544:
URL: https://github.com/apache/nifi/pull/7544#issuecomment-1675974454
> @ravinarayansingh you have a merge conflict.
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] ravinarayansingh commented on pull request #7544: NIFI-11858 Improve column name normalization in PutDatabaseRecord and UpdateDatabaseTable processor
Posted by "ravinarayansingh (via GitHub)" <gi...@apache.org>.
ravinarayansingh commented on PR #7544:
URL: https://github.com/apache/nifi/pull/7544#issuecomment-1718450737
Hi @exceptionfactory ,
when this feature will be available in NiFi
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11858 Configurable Column Name Normalization in PutDatabaseRecord and UpdateDatabaseTable [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on code in PR #7544:
URL: https://github.com/apache/nifi/pull/7544#discussion_r1412711140
##########
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutDatabaseRecord.java:
##########
@@ -1276,8 +1303,10 @@ SqlAndIncludedColumns generateUpdate(final RecordSchema recordSchema, final Stri
String fieldName = field.getFieldName();
boolean firstUpdateKey = true;
- final String normalizedColName = ColumnDescription.normalizeColumnName(fieldName, settings.translateFieldNames);
- final ColumnDescription desc = tableSchema.getColumns().get(ColumnDescription.normalizeColumnName(fieldName, settings.translateFieldNames));
+ final String normalizedColName = getNormalizedName(fieldName,
+ settings.translateFieldNames, settings.translationStrategy, settings.translationRegex);
+ final ColumnDescription desc = tableSchema.getColumns().get(getNormalizedName(fieldName,
+ settings.translateFieldNames, settings.translationStrategy, settings.translationRegex));
Review Comment:
```suggestion
final ColumnDescription desc = tableSchema.getColumns().get(normalizedColName);
```
##########
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutDatabaseRecord.java:
##########
@@ -1577,6 +1610,8 @@ static class DMLSettings {
DMLSettings(ProcessContext context) {
translateFieldNames = context.getProperty(TRANSLATE_FIELD_NAMES).asBoolean();
+ translationStrategy = TranslationStrategy.valueOf(context.getProperty(TRANSLATION_STRATEGY).getValue());
+ translationRegex = context.getProperty(TRANSLATION_REGEX).getValue();
Review Comment:
This may need to be null-checked, but should be declared with `Pattern.compile()`
##########
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/TranslationStrategy.java:
##########
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard.db;
+
+import org.apache.nifi.components.DescribedValue;
+
+/**
+ * Enumeration of supported Database column name Translation Strategy
+ */
+public enum TranslationStrategy implements DescribedValue {
+ REMOVE_UNDERSCORE("REMOVE_UNDERSCORE","Remove Underscore","Underscore(_) will be removed from column name with empty string Ex. Pics_1_11 become PICS111"),
+ REMOVE_SPACE("REMOVE_SPACE","Remove Space","Spaces will be removed from column name with empty string Ex. 'User Name' become 'USERNAME'"),
+ REMOVE_ALL_SPECIAL_CHAR("REMOVE_ALL_SPECIAL_CHAR","Remove All Special Character","Remove All Special Character"),
+ REGEX("REGEX","Regular Expression","Remove character matched Regular Expression from column name");
+ private final String value;
+
+ private final String displayName;
+
+ private final String description;
+ TranslationStrategy(final String value, final String displayName, final String description) {
+ this.value = value;
+ this.displayName = displayName;
+ this.description = description;
+ }
+
+ @Override
+ public String getValue() {
+ return value;
Review Comment:
```suggestion
return name();
```
##########
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutDatabaseRecord.java:
##########
@@ -246,6 +249,25 @@ public class PutDatabaseRecord extends AbstractProcessor {
.defaultValue("true")
.build();
+ public static final PropertyDescriptor TRANSLATION_STRATEGY = new PropertyDescriptor.Builder()
+ .required(true)
+ .name("translation-strategy")
+ .description("The strategy used to normalize table column name")
+ .allowableValues(TranslationStrategy.class)
+ .defaultValue(TranslationStrategy.REMOVE_UNDERSCORE.getValue())
+ .dependsOn(TRANSLATE_FIELD_NAMES, TRANSLATE_FIELD_NAMES.getDefaultValue())
+ .build();
+
+ public static final PropertyDescriptor TRANSLATION_REGEX = new PropertyDescriptor.Builder()
+ .required(true)
+ .name("translation-regex")
+ .displayName("Translation Regular Expression")
Review Comment:
```suggestion
.name("Column Name Translation Pattern")
```
##########
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutDatabaseRecord.java:
##########
@@ -246,6 +249,25 @@ public class PutDatabaseRecord extends AbstractProcessor {
.defaultValue("true")
.build();
+ public static final PropertyDescriptor TRANSLATION_STRATEGY = new PropertyDescriptor.Builder()
+ .required(true)
+ .name("translation-strategy")
+ .description("The strategy used to normalize table column name")
+ .allowableValues(TranslationStrategy.class)
+ .defaultValue(TranslationStrategy.REMOVE_UNDERSCORE.getValue())
+ .dependsOn(TRANSLATE_FIELD_NAMES, TRANSLATE_FIELD_NAMES.getDefaultValue())
+ .build();
+
+ public static final PropertyDescriptor TRANSLATION_REGEX = new PropertyDescriptor.Builder()
Review Comment:
```suggestion
public static final PropertyDescriptor TRANSLATION_PATTERN = new PropertyDescriptor.Builder()
```
##########
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutDatabaseRecord.java:
##########
@@ -1231,13 +1257,14 @@ SqlAndIncludedColumns generateUpdate(final RecordSchema recordSchema, final Stri
int fieldCount = fieldNames.size();
AtomicInteger fieldsFound = new AtomicInteger(0);
-
for (int i = 0; i < fieldCount; i++) {
RecordField field = recordSchema.getField(i);
String fieldName = field.getFieldName();
- final String normalizedColName = ColumnDescription.normalizeColumnName(fieldName, settings.translateFieldNames);
- final ColumnDescription desc = tableSchema.getColumns().get(ColumnDescription.normalizeColumnName(fieldName, settings.translateFieldNames));
+ final String normalizedColName = getNormalizedName(fieldName, settings.translateFieldNames,
+ settings.translationStrategy, settings.translationRegex);
+ final ColumnDescription desc = tableSchema.getColumns().get(getNormalizedName(fieldName,
+ settings.translateFieldNames, settings.translationStrategy, settings.translationRegex));
Review Comment:
This appears to be an unnecessary call to `getNormalizedName()` when `normalizedColName` can be used.
```suggestion
final ColumnDescription desc = tableSchema.getColumns().get(normalizedColName);
```
##########
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/TranslationStrategy.java:
##########
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard.db;
+
+import org.apache.nifi.components.DescribedValue;
+
+/**
+ * Enumeration of supported Database column name Translation Strategy
+ */
+public enum TranslationStrategy implements DescribedValue {
+ REMOVE_UNDERSCORE("REMOVE_UNDERSCORE","Remove Underscore","Underscore(_) will be removed from column name with empty string Ex. Pics_1_11 become PICS111"),
+ REMOVE_SPACE("REMOVE_SPACE","Remove Space","Spaces will be removed from column name with empty string Ex. 'User Name' become 'USERNAME'"),
+ REMOVE_ALL_SPECIAL_CHAR("REMOVE_ALL_SPECIAL_CHAR","Remove All Special Character","Remove All Special Character"),
+ REGEX("REGEX","Regular Expression","Remove character matched Regular Expression from column name");
+ private final String value;
+
+ private final String displayName;
+
+ private final String description;
+ TranslationStrategy(final String value, final String displayName, final String description) {
+ this.value = value;
Review Comment:
The `value` can be removed and the `name()` of the enum can be used in `getValue()`.
```suggestion
TranslationStrategy(final String displayName, final String description) {
```
##########
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/ColumnNameNormalizer.java:
##########
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard.db;
+public interface ColumnNameNormalizer {
+
+
+ /**
+ * Normalizes the given column name based on the specified strategy.
+ *
+ * @param colName The column name to be normalized.
+ * @param isTranslationEnabled Boolean value to denote normalization is enabled
+ * @param strategy The TranslationStrategy for normalizing column name
+ * @param translationRegex Regex For translation
+ * @return The normalized column name as a String.
+ */
+ default String getNormalizedName(final String colName,boolean isTranslationEnabled, TranslationStrategy strategy, String translationRegex) {
Review Comment:
```suggestion
default String getNormalizedName(String colName, boolean translationEnabled, TranslationStrategy strategy, Pattern translationPattern) {
```
##########
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/UpdateDatabaseTable.java:
##########
@@ -177,7 +179,25 @@ public class UpdateDatabaseTable extends AbstractProcessor {
.allowableValues("true", "false")
.defaultValue("true")
.build();
+ public static final PropertyDescriptor TRANSLATION_STRATEGY = new PropertyDescriptor.Builder()
+ .required(true)
+ .name("Translation Strategy")
+ .description("The strategy used to normalize table column name")
+ .allowableValues(TranslationStrategy.class)
+ .defaultValue(TranslationStrategy.REMOVE_UNDERSCORE.getValue())
+ .dependsOn(TRANSLATE_FIELD_NAMES, TRANSLATE_FIELD_NAMES.getDefaultValue())
+ .build();
+ public static final PropertyDescriptor TRANSLATION_REGEX = new PropertyDescriptor.Builder()
+ .name("translation-regex")
+ .displayName("Translation Regular Expression")
Review Comment:
```suggestion
.name("Column Name Translation Pattern")
.displayName("Column Name Translation Pattern")
```
##########
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutDatabaseRecord.java:
##########
@@ -246,6 +249,25 @@ public class PutDatabaseRecord extends AbstractProcessor {
.defaultValue("true")
.build();
+ public static final PropertyDescriptor TRANSLATION_STRATEGY = new PropertyDescriptor.Builder()
+ .required(true)
+ .name("translation-strategy")
Review Comment:
```suggestion
.name("Column Name Translation Strategy")
```
##########
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/ColumnNameNormalizer.java:
##########
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard.db;
+public interface ColumnNameNormalizer {
Review Comment:
This should be spaced.
```suggestion
public interface ColumnNameNormalizer {
```
##########
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/UpdateDatabaseTable.java:
##########
@@ -177,7 +179,25 @@ public class UpdateDatabaseTable extends AbstractProcessor {
.allowableValues("true", "false")
.defaultValue("true")
.build();
+ public static final PropertyDescriptor TRANSLATION_STRATEGY = new PropertyDescriptor.Builder()
+ .required(true)
+ .name("Translation Strategy")
Review Comment:
```suggestion
.name("Column Name Translation Strategy")
```
##########
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/TranslationStrategy.java:
##########
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard.db;
+
+import org.apache.nifi.components.DescribedValue;
+
+/**
+ * Enumeration of supported Database column name Translation Strategy
+ */
+public enum TranslationStrategy implements DescribedValue {
+ REMOVE_UNDERSCORE("REMOVE_UNDERSCORE","Remove Underscore","Underscore(_) will be removed from column name with empty string Ex. Pics_1_11 become PICS111"),
+ REMOVE_SPACE("REMOVE_SPACE","Remove Space","Spaces will be removed from column name with empty string Ex. 'User Name' become 'USERNAME'"),
+ REMOVE_ALL_SPECIAL_CHAR("REMOVE_ALL_SPECIAL_CHAR","Remove All Special Character","Remove All Special Character"),
+ REGEX("REGEX","Regular Expression","Remove character matched Regular Expression from column name");
Review Comment:
```suggestion
PATTERN("Regular Expression", "Remove character matched Regular Expression from column name");
```
##########
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/UpdateDatabaseTable.java:
##########
@@ -374,6 +396,9 @@ public void onTrigger(final ProcessContext context, final ProcessSession session
final boolean createIfNotExists = context.getProperty(CREATE_TABLE).getValue().equals(CREATE_IF_NOT_EXISTS.getValue());
final boolean updateFieldNames = context.getProperty(UPDATE_FIELD_NAMES).asBoolean();
final boolean translateFieldNames = context.getProperty(TRANSLATE_FIELD_NAMES).asBoolean();
+ final TranslationStrategy translationStrategy = TranslationStrategy.valueOf(context.getProperty(TRANSLATION_STRATEGY).getValue());
+ final String translationRegex = context.getProperty(TRANSLATION_REGEX).getValue();
Review Comment:
This should be null-checked and parsed with `Pattern.compile()`
##########
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutDatabaseRecord.java:
##########
@@ -1563,6 +1594,8 @@ public String apply(final Record record) {
static class DMLSettings {
private final boolean translateFieldNames;
+ private final TranslationStrategy translationStrategy;
+ private final String translationRegex;
Review Comment:
This should be declared as a `Pattern` for more efficient repeated processing.
##########
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/ColumnNameNormalizerUtility.java:
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard.db;
+
+
+import java.util.Objects;
+
+/**
+ * ColumnNameNormalizerUtility is a utility class that helps to normalize column names. It provides various strategies to
+ * modify column names based on the TranslationStrategy enum. Column names can be normalized by removing underscores,
+ * spaces, all special characters, or by using a custom regular expression defined by the user.
+ */
+public class ColumnNameNormalizerUtility {
+ // Regular expression to remove all special characters from a string.
+ private static final String REMOVE_ALL_SPECIAL_CHAR_REGEX = "[^a-zA-Z0-9]";
Review Comment:
`Pattern.compile()` should be used to preset the parsed pattern.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11858 Configurable Column Name Normalization in PutDatabaseRecord and UpdateDatabaseTable [nifi]
Posted by "ravinarayansingh (via GitHub)" <gi...@apache.org>.
ravinarayansingh commented on code in PR #7544:
URL: https://github.com/apache/nifi/pull/7544#discussion_r1437177360
##########
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/UpdateDatabaseTable.java:
##########
@@ -460,15 +488,18 @@ public void onTrigger(final ProcessContext context, final ProcessSession session
private synchronized OutputMetadataHolder checkAndUpdateTableSchema(final Connection conn, final DatabaseAdapter databaseAdapter, final RecordSchema schema,
final String catalogName, final String schemaName, final String tableName,
- final boolean createIfNotExists, final boolean translateFieldNames, final boolean updateFieldNames,
+ final boolean createIfNotExists, final boolean translateFieldNames,
+ final TranslationStrategy translationStrategy, Pattern translationRegex, final boolean updateFieldNames,
final Set<String> primaryKeyColumnNames, final boolean quoteTableName, final boolean quoteColumnNames) throws IOException {
// Read in the current table metadata, compare it to the reader's schema, and
// add any columns from the schema that are missing in the table
+// final ColumnNameNormalizer normalizer = new ColumnNameNormalizer(translateFieldNames, translationStrategy, translationRegex);
Review Comment:
i have removed the commented code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11858 Improve column name normalization in PutDatabaseRecord and UpdateDatabaseTable processor [nifi]
Posted by "ravinarayansingh (via GitHub)" <gi...@apache.org>.
ravinarayansingh commented on PR #7544:
URL: https://github.com/apache/nifi/pull/7544#issuecomment-1806990586
> Thanks for your patience on this pull request review @ravinarayansingh. Reviewing the implementation details, the general concept of a configurable column normalization strategy is helpful.
>
> As a general recommendation, it looks like it would be helpful to make `ColumnNameNormalizer` an interface, with `getNormalizedName(String columnName)` as the interface method. Although the current implementation is simple enough, providing separate implementations would remove the need for the conditional logic. Given that this method will be called for every column, there seems to be value in having narrowly defined implementation classes.
Hi @exceptionfactory
thanks for suggestion i will make required changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11858 Configurable Column Name Normalization in PutDatabaseRecord and UpdateDatabaseTable [nifi]
Posted by "mattyb149 (via GitHub)" <gi...@apache.org>.
mattyb149 commented on PR #7544:
URL: https://github.com/apache/nifi/pull/7544#issuecomment-2061707094
This needs a rebase against the latest main, then I will take a look also, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] ravinarayansingh commented on pull request #7544: NIFI-11858 Improve column name normalization in PutDatabaseRecord and UpdateDatabaseTable processor
Posted by "ravinarayansingh (via GitHub)" <gi...@apache.org>.
ravinarayansingh commented on PR #7544:
URL: https://github.com/apache/nifi/pull/7544#issuecomment-1660973702
Hi @exceptionfactory
I have made the require changes
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11858 Configurable Column Name Normalization in PutDatabaseRecord and UpdateDatabaseTable [nifi]
Posted by "mattyb149 (via GitHub)" <gi...@apache.org>.
mattyb149 commented on PR #7544:
URL: https://github.com/apache/nifi/pull/7544#issuecomment-2185613576
Closing this in favor of the new PR, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11858 Configurable Column Name Normalization in PutDatabaseRecord and UpdateDatabaseTable [nifi]
Posted by "jrsteinebrey (via GitHub)" <gi...@apache.org>.
jrsteinebrey commented on code in PR #7544:
URL: https://github.com/apache/nifi/pull/7544#discussion_r1631317574
##########
nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutDatabaseRecord.java:
##########
@@ -278,6 +283,24 @@ public class PutDatabaseRecord extends AbstractProcessor {
.defaultValue("true")
.build();
+ public static final PropertyDescriptor TRANSLATION_STRATEGY = new PropertyDescriptor.Builder()
+ .required(true)
+ .name("Column Name Translation Strategy")
+ .description("The strategy used to normalize table column name")
Review Comment:
@ravinarayansingh The code uses all three of the these Translate properties (Translate Field Names, Column Name Translation Strategy, and Column Name Translation Pattern) and uses them to translate BOTH field names AND column names. I recommend that the displayable names and descriptions be changed to reflect the fact that they apply to both field and column names.
This comment also equally applies to UpdateDatabaseTable class.
##########
nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/UpdateDatabaseTable.java:
##########
@@ -177,7 +181,25 @@ public class UpdateDatabaseTable extends AbstractProcessor {
.allowableValues("true", "false")
.defaultValue("true")
.build();
+ public static final PropertyDescriptor TRANSLATION_STRATEGY = new PropertyDescriptor.Builder()
+ .required(true)
+ .name("Column Name Translation Strategy")
Review Comment:
@ravinarayansingh Same comment as above that these apply to both field and column names.
##########
nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutDatabaseRecord.java:
##########
@@ -270,6 +274,7 @@ public class PutDatabaseRecord extends AbstractProcessor {
.build();
static final PropertyDescriptor TRANSLATE_FIELD_NAMES = new Builder()
+ .required(true)
.name("put-db-record-translate-field-names")
.displayName("Translate Field Names")
Review Comment:
I feel it is valuable to make these property names as clear as possible because this is a sophisticated feature.
I think another word besides "Translate" could be found to communicate what this feature does.
Here are some property display names I have thought of. I think Normalize is clearer. What to people think?
"Translate Field and Column Names for Comparison"
"Normalize Field and Column Names for Comparison"
"Adjust Field and Column Names for Comparison"
"Filter Field and Column Names for Comparison"
##########
nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/ColumnNameNormalizer.java:
##########
@@ -0,0 +1,24 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard.db;
+
+public interface ColumnNameNormalizer {
Review Comment:
ColumnNameNormalizer is not ideal since the class is used on json field names and column names, but I am not sure if that is worth changing it to something else like FieldColumnNameNormalizer or maybe NameNormalizer (maybe it will be used in other cases besides fields or columns in the future). Thoughts?
If changed, the assoc factory class name should change also.
##########
nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/TableSchema.java:
##########
@@ -136,14 +140,23 @@ public static TableSchema from(final Connection conn, final String catalog, fina
} else {
// Parse the Update Keys field and normalize the column names
for (final String updateKey : updateKeys.split(",")) {
- primaryKeyColumns.add(ColumnDescription.normalizeColumnName(updateKey.trim(), translateColumnNames));
+ final String colName = normalizedName(updateKey, translateColumnNames, normalizer);
+ primaryKeyColumns.add(colName);
+
}
}
- return new TableSchema(catalog, schema, tableName, cols, translateColumnNames, primaryKeyColumns, dmd.getIdentifierQuoteString());
+ return new TableSchema(catalog, schema, tableName, cols, translateColumnNames, normalizer, primaryKeyColumns, dmd.getIdentifierQuoteString());
}
}
+ public static String normalizedName(final String name, final boolean translateColumnNames, final ColumnNameNormalizer normalizer) {
+ final String colName = name.trim();
+ if (translateColumnNames && normalizer != null)
+ return normalizer.getNormalizedName(colName);
Review Comment:
I feel it would be better to put the toUpperCase() here instead of repeating it in every normalizer.
trim() is not repeated in the normalizers and toUpperCase feels similar to trim.
Also move trim() inside the IF check. translateNames == false should make zero modifications to the names.
return normalizer.getNormalizedName(colName).trim().toUpperCase;
##########
nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/TableSchema.java:
##########
@@ -136,14 +140,23 @@ public static TableSchema from(final Connection conn, final String catalog, fina
} else {
// Parse the Update Keys field and normalize the column names
for (final String updateKey : updateKeys.split(",")) {
- primaryKeyColumns.add(ColumnDescription.normalizeColumnName(updateKey.trim(), translateColumnNames));
+ final String colName = normalizedName(updateKey, translateColumnNames, normalizer);
+ primaryKeyColumns.add(colName);
+
}
}
- return new TableSchema(catalog, schema, tableName, cols, translateColumnNames, primaryKeyColumns, dmd.getIdentifierQuoteString());
+ return new TableSchema(catalog, schema, tableName, cols, translateColumnNames, normalizer, primaryKeyColumns, dmd.getIdentifierQuoteString());
}
}
+ public static String normalizedName(final String name, final boolean translateColumnNames, final ColumnNameNormalizer normalizer) {
Review Comment:
I would remove col and column from all names in this method because field names are also passed to this method..
translateColumnNames could be shortened to "translateNames"
colName could be "normalizedName" or "result".
##########
nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/TableSchema.java:
##########
@@ -136,14 +140,23 @@ public static TableSchema from(final Connection conn, final String catalog, fina
} else {
// Parse the Update Keys field and normalize the column names
for (final String updateKey : updateKeys.split(",")) {
- primaryKeyColumns.add(ColumnDescription.normalizeColumnName(updateKey.trim(), translateColumnNames));
+ final String colName = normalizedName(updateKey, translateColumnNames, normalizer);
+ primaryKeyColumns.add(colName);
+
}
}
- return new TableSchema(catalog, schema, tableName, cols, translateColumnNames, primaryKeyColumns, dmd.getIdentifierQuoteString());
+ return new TableSchema(catalog, schema, tableName, cols, translateColumnNames, normalizer, primaryKeyColumns, dmd.getIdentifierQuoteString());
}
}
+ public static String normalizedName(final String name, final boolean translateColumnNames, final ColumnNameNormalizer normalizer) {
+ final String colName = name.trim();
+ if (translateColumnNames && normalizer != null)
Review Comment:
The old normalizeName() checked for null. Old method returned a null when null name was passed in.
Also the old method did NOT trim() the string.
Is everyone OK with those two functional code changes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on pull request #7544: NIFI-11858 Improve column name normalization in PutDatabaseRecord and UpdateDatabaseTable processor
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on PR #7544:
URL: https://github.com/apache/nifi/pull/7544#issuecomment-1660758721
@ravinarayansingh The Static Analysis build lists the particular problem:
https://github.com/apache/nifi/actions/runs/5719909353/job/15509139654?pr=7544#step:5:5763
It looks like those `nifi-h2` directories are leftover on your local system from the support branch, so they could be removed, but that will not impact the pull request build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] MikeThomsen commented on pull request #7544: NIFI-11858 Improve column name normalization in PutDatabaseRecord and UpdateDatabaseTable processor
Posted by "MikeThomsen (via GitHub)" <gi...@apache.org>.
MikeThomsen commented on PR #7544:
URL: https://github.com/apache/nifi/pull/7544#issuecomment-1675860479
@ravinarayansingh you have a merge conflict.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11858 Configurable Column Name Normalization in PutDatabaseRecord and UpdateDatabaseTable [nifi]
Posted by "ravinarayansingh (via GitHub)" <gi...@apache.org>.
ravinarayansingh commented on PR #7544:
URL: https://github.com/apache/nifi/pull/7544#issuecomment-1825274326
Hi @exceptionfactory ,
I have made the required changes, please have a look
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11858 Configurable Column Name Normalization in PutDatabaseRecord and UpdateDatabaseTable [nifi]
Posted by "jrsteinebrey (via GitHub)" <gi...@apache.org>.
jrsteinebrey commented on code in PR #7544:
URL: https://github.com/apache/nifi/pull/7544#discussion_r1603296706
##########
nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/TranslationStrategy.java:
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard.db;
+
+import org.apache.nifi.components.DescribedValue;
+
+/**
+ * Enumeration of supported Database column name Translation Strategies
+ */
+public enum TranslationStrategy implements DescribedValue {
+ REMOVE_UNDERSCORE("Remove Underscore",
+ "Underscore(_) will be removed from column name with empty string Ex. Pics_1_11 become PICS111"),
Review Comment:
I recommend this wording:
"Underscores '_' will be removed from column names Ex: 'Pics_1_23' becomes 'PICS123'"
##########
nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/TranslationStrategy.java:
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard.db;
+
+import org.apache.nifi.components.DescribedValue;
+
+/**
+ * Enumeration of supported Database column name Translation Strategies
+ */
+public enum TranslationStrategy implements DescribedValue {
+ REMOVE_UNDERSCORE("Remove Underscore",
+ "Underscore(_) will be removed from column name with empty string Ex. Pics_1_11 become PICS111"),
+ REMOVE_SPACE("Remove Space",
+ "Spaces will be removed from column name with empty string Ex. 'User Name' become 'USERNAME'"),
Review Comment:
I recommend this wording:
"Spaces will be removed from column names Ex. 'User Name' becomes 'USERNAME'"
##########
nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/TranslationStrategy.java:
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard.db;
+
+import org.apache.nifi.components.DescribedValue;
+
+/**
+ * Enumeration of supported Database column name Translation Strategies
+ */
+public enum TranslationStrategy implements DescribedValue {
+ REMOVE_UNDERSCORE("Remove Underscore",
+ "Underscore(_) will be removed from column name with empty string Ex. Pics_1_11 become PICS111"),
+ REMOVE_SPACE("Remove Space",
+ "Spaces will be removed from column name with empty string Ex. 'User Name' become 'USERNAME'"),
+ REMOVE_ALL_SPECIAL_CHAR("Remove All Special Character", "Remove All Special Character"),
+ PATTERN("Regular Expression", "Remove character matched Regular Expression from column name");
Review Comment:
"Remove characters matching this Regular Expression from the column names"
##########
nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/TranslationStrategy.java:
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard.db;
+
+import org.apache.nifi.components.DescribedValue;
+
+/**
+ * Enumeration of supported Database column name Translation Strategies
+ */
+public enum TranslationStrategy implements DescribedValue {
+ REMOVE_UNDERSCORE("Remove Underscore",
+ "Underscore(_) will be removed from column name with empty string Ex. Pics_1_11 become PICS111"),
+ REMOVE_SPACE("Remove Space",
+ "Spaces will be removed from column name with empty string Ex. 'User Name' become 'USERNAME'"),
+ REMOVE_ALL_SPECIAL_CHAR("Remove All Special Character", "Remove All Special Character"),
Review Comment:
Remove All Special Character implies it will remove everything character which is and a-z or 0-9.
I think this is more descriptive:
"Remove Regular Expression Characters", "Remove Regular Expression Characters"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11858 Configurable Column Name Normalization in PutDatabaseRecord and UpdateDatabaseTable [nifi]
Posted by "ravinarayansingh (via GitHub)" <gi...@apache.org>.
ravinarayansingh commented on PR #7544:
URL: https://github.com/apache/nifi/pull/7544#issuecomment-2125166378
Hi @jrsteinebrey
I have updated the code as per you suggestion, have a look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11858 Configurable Column Name Normalization in PutDatabaseRecord and UpdateDatabaseTable [nifi]
Posted by "mattyb149 (via GitHub)" <gi...@apache.org>.
mattyb149 closed pull request #7544: NIFI-11858 Configurable Column Name Normalization in PutDatabaseRecord and UpdateDatabaseTable
URL: https://github.com/apache/nifi/pull/7544
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11858 Configurable Column Name Normalization in PutDatabaseRecord and UpdateDatabaseTable [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #7544:
URL: https://github.com/apache/nifi/pull/7544#discussion_r1476145113
##########
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/ColumnNameNormalizerUtility.java:
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard.db;
+
+
+import java.util.Objects;
+import java.util.regex.Pattern;
+
+/**
+ * ColumnNameNormalizerUtility is a utility class that helps to normalize column names. It provides various strategies to
+ * modify column names based on the TranslationStrategy enum. Column names can be normalized by removing underscores,
+ * spaces, all special characters, or by using a custom regular expression defined by the user.
+ */
+public class ColumnNameNormalizerUtility {
+ // Regular expression to remove all special characters from a string.
+ private static final Pattern REMOVE_ALL_SPECIAL_CHAR_REGEX = Pattern.compile("[^a-zA-Z0-9]");
+
+ /**
+ * Normalizes the given column name based on the specified strategy.
+ *
+ * @param colName The column name to be normalized.
+ * @param isTranslationEnabled Boolean value to denote normalization is enabled
+ * @param strategy The TranslationStrategy for normalizing column name
+ * @param translationRegex Regex For translation
+ * @return The normalized column name as a String.
+ */
+ public static String getNormalizedName(final String colName, boolean isTranslationEnabled, TranslationStrategy strategy, Pattern translationRegex) {
+ // If the column name is null or translation is not enabled, return the original column name.
+ if (colName == null || !isTranslationEnabled) {
+ return colName;
+ }
+
+ return switch (Objects.requireNonNull(strategy)) {
+ case REMOVE_UNDERSCORE -> colName.toUpperCase().replace("_", "");
+ case REMOVE_SPACE -> colName.toUpperCase().replace(" ", "");
+ case REMOVE_ALL_SPECIAL_CHAR -> REMOVE_ALL_SPECIAL_CHAR_REGEX .matcher(colName.toUpperCase()).replaceAll("");
Review Comment:
There is a space between `REMOVE_ALL_SPECIAL_CHAR_REGEX` and `.matcher`
```suggestion
case REMOVE_ALL_SPECIAL_CHAR -> REMOVE_ALL_SPECIAL_CHAR_REGEX.matcher(colName.toUpperCase()).replaceAll("");
```
##########
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/TranslationStrategy.java:
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard.db;
+
+import org.apache.nifi.components.DescribedValue;
+
+/**
+ * Enumeration of supported Database column name Translation Strategy
Review Comment:
```suggestion
* Enumeration of supported Database column name Translation Strategies
```
##########
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutDatabaseRecord.java:
##########
@@ -1577,6 +1608,9 @@ static class DMLSettings {
DMLSettings(ProcessContext context) {
translateFieldNames = context.getProperty(TRANSLATE_FIELD_NAMES).asBoolean();
+ translationStrategy = TranslationStrategy.valueOf(context.getProperty(TRANSLATION_STRATEGY).getValue());
+ final String translationRegex = context.getProperty(TRANSLATION_PATTERN).getValue();
+ translationPattern = translationRegex==null?null:Pattern.compile(translationRegex);
Review Comment:
```suggestion
translationPattern = translationRegex == null ? null : Pattern.compile(translationRegex);
```
##########
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/TranslationStrategy.java:
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard.db;
+
+import org.apache.nifi.components.DescribedValue;
+
+/**
+ * Enumeration of supported Database column name Translation Strategy
+ */
+public enum TranslationStrategy implements DescribedValue {
+ REMOVE_UNDERSCORE {
+ @Override
+ public String getDisplayName() {
+ return "Remove Underscore";
+ }
+
+ @Override
+ public String getDescription() {
+ return "Underscore(_) will be removed from column name with empty string Ex. Pics_1_11 become PICS111";
+ }
+ },
+ REMOVE_SPACE {
+ @Override
+ public String getDisplayName() {
+ return "Remove Space";
+ }
+
+ @Override
+ public String getDescription() {
+ return "Spaces will be removed from column name with empty string Ex. 'User Name' become 'USERNAME'";
+ }
+ },
+ REMOVE_ALL_SPECIAL_CHAR {
+ @Override
+ public String getDisplayName() {
+ return "Remove All Special Character";
+ }
+
+ @Override
+ public String getDescription() {
+ return "Remove All Special Character";
+ }
+ },
+ PATTERN{
+ @Override
+ public String getDisplayName() {
+ return "Regular Expression";
+ }
+
+ @Override
+ public String getDescription() {
+ return "Remove character matched Regular Expression from column name";
+ }
+};
+
+
+ @Override
+ public String getValue() {
+ return name();
+ }
+
+ }
Review Comment:
Although your implementation of this enum which extends `DescribedValue` works, if you browse the Apache NIFI code base you will notice that this not the standard way of implementing it. I have included what the standard way is.
```suggestion
public enum TranslationStrategy implements DescribedValue {
REMOVE_UNDERSCORE("Remove Underscore",
"Underscore(_) will be removed from column name with empty string Ex. Pics_1_11 become PICS111"),
REMOVE_SPACE("Remove Space", "Spaces will be removed from column name with empty string Ex. 'User Name' become 'USERNAME'"),
REMOVE_ALL_SPECIAL_CHAR("Remove All Special Character", "Remove All Special Character"),
PATTERN("Regular Expression", "Remove character matched Regular Expression from column name");
private final String displayName;
private final String description;
TranslationStrategy(String displayName, String description) {
this.displayName = displayName;
this.description = description;
}
@Override
public String getValue() {
return name();
}
@Override
public String getDisplayName() {
return displayName;
}
@Override
public String getDescription() {
return description;
}
}
```
##########
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/db/ColumnNameNormalizerUtilityTest.java:
##########
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard.db;
+
+import org.junit.jupiter.api.Test;
+
+import java.util.regex.Pattern;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+
+public class ColumnNameNormalizerUtilityTest {
+
+
+
+ @Test
+ void testNormalizingColumnName_RemoveUnderscore() {
+ String inputColumnName = "example_column_name";
+ String expectedNormalized = "EXAMPLECOLUMNNAME";
+ String normalized = ColumnNameNormalizerUtility.getNormalizedName(inputColumnName,true, TranslationStrategy.REMOVE_UNDERSCORE, null);
+
+ assertEquals(expectedNormalized, normalized);
+ }
+
+ @Test
+ void testNormalizingColumnName_RemoveSpace() {
+ String inputColumnName = "Column Name With Spaces";
+ String expectedNormalized = "COLUMNNAMEWITHSPACES";
+ String normalized = ColumnNameNormalizerUtility.getNormalizedName(inputColumnName,true, TranslationStrategy.REMOVE_SPACE, null);
+
+ assertEquals(expectedNormalized, normalized);
+ }
+
+ @Test
+ void testNormalizingColumnName_RemoveAllSpecialCharacters() {
+ String inputColumnName = "Special!Characters@Here$";
+ String expectedNormalized = "SPECIALCHARACTERSHERE";
+ String normalized = ColumnNameNormalizerUtility.getNormalizedName(inputColumnName,true, TranslationStrategy.REMOVE_ALL_SPECIAL_CHAR, null);
+
+ assertEquals(expectedNormalized, normalized);
+ }
+
+ @Test
+ void testNormalizingColumnName_Regex() {
+ String inputColumnName = "Your @Input -String Here";
+ Pattern translationPattern = Pattern.compile("[@-]");
+ String expectedNormalized = "YOUR INPUT STRING HERE";
+ String normalized = ColumnNameNormalizerUtility.getNormalizedName(inputColumnName,true, TranslationStrategy.PATTERN, translationPattern);
+
+ assertEquals(expectedNormalized, normalized);
+ }
+
+
+ @Test
+ void testNormalizingColumnName_NotEnabled() {
+ String inputColumnName = "example_column_name";
+
+ String normalized = ColumnNameNormalizerUtility.getNormalizedName(inputColumnName,false, TranslationStrategy.REMOVE_UNDERSCORE, null);
Review Comment:
```suggestion
String normalized = ColumnNameNormalizerUtility.getNormalizedName(inputColumnName, false, TranslationStrategy.REMOVE_UNDERSCORE, null);
```
##########
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/PutDatabaseRecordTest.java:
##########
@@ -273,7 +273,7 @@ void testGeneratePreparedStatements() throws SQLException, MalformedRecordExcept
new ColumnDescription("name", 12, true, 255, true),
new ColumnDescription("code", 4, true, 10, true)
),
- false,
+ false,null,null,
Review Comment:
```suggestion
false, null, null,
```
##########
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/db/ColumnNameNormalizerUtilityTest.java:
##########
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard.db;
+
+import org.junit.jupiter.api.Test;
+
+import java.util.regex.Pattern;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+
+public class ColumnNameNormalizerUtilityTest {
+
+
+
+ @Test
+ void testNormalizingColumnName_RemoveUnderscore() {
+ String inputColumnName = "example_column_name";
+ String expectedNormalized = "EXAMPLECOLUMNNAME";
+ String normalized = ColumnNameNormalizerUtility.getNormalizedName(inputColumnName,true, TranslationStrategy.REMOVE_UNDERSCORE, null);
+
+ assertEquals(expectedNormalized, normalized);
+ }
+
+ @Test
+ void testNormalizingColumnName_RemoveSpace() {
+ String inputColumnName = "Column Name With Spaces";
+ String expectedNormalized = "COLUMNNAMEWITHSPACES";
+ String normalized = ColumnNameNormalizerUtility.getNormalizedName(inputColumnName,true, TranslationStrategy.REMOVE_SPACE, null);
Review Comment:
```suggestion
String normalized = ColumnNameNormalizerUtility.getNormalizedName(inputColumnName, true, TranslationStrategy.REMOVE_SPACE, null);
```
##########
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/PutDatabaseRecordTest.java:
##########
@@ -311,7 +311,7 @@ void testGeneratePreparedStatementsFailUnmatchedField() {
new ColumnDescription("name", 12, true, 255, true),
new ColumnDescription("code", 4, true, 10, true)
),
- false,
+ false,null,null,
Review Comment:
```suggestion
false, null, null,
```
##########
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/db/ColumnNameNormalizerUtilityTest.java:
##########
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard.db;
+
+import org.junit.jupiter.api.Test;
+
+import java.util.regex.Pattern;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+
+public class ColumnNameNormalizerUtilityTest {
+
+
+
+ @Test
+ void testNormalizingColumnName_RemoveUnderscore() {
+ String inputColumnName = "example_column_name";
+ String expectedNormalized = "EXAMPLECOLUMNNAME";
+ String normalized = ColumnNameNormalizerUtility.getNormalizedName(inputColumnName,true, TranslationStrategy.REMOVE_UNDERSCORE, null);
+
+ assertEquals(expectedNormalized, normalized);
+ }
+
+ @Test
+ void testNormalizingColumnName_RemoveSpace() {
+ String inputColumnName = "Column Name With Spaces";
+ String expectedNormalized = "COLUMNNAMEWITHSPACES";
+ String normalized = ColumnNameNormalizerUtility.getNormalizedName(inputColumnName,true, TranslationStrategy.REMOVE_SPACE, null);
+
+ assertEquals(expectedNormalized, normalized);
+ }
+
+ @Test
+ void testNormalizingColumnName_RemoveAllSpecialCharacters() {
+ String inputColumnName = "Special!Characters@Here$";
+ String expectedNormalized = "SPECIALCHARACTERSHERE";
+ String normalized = ColumnNameNormalizerUtility.getNormalizedName(inputColumnName,true, TranslationStrategy.REMOVE_ALL_SPECIAL_CHAR, null);
+
+ assertEquals(expectedNormalized, normalized);
+ }
+
+ @Test
+ void testNormalizingColumnName_Regex() {
+ String inputColumnName = "Your @Input -String Here";
+ Pattern translationPattern = Pattern.compile("[@-]");
+ String expectedNormalized = "YOUR INPUT STRING HERE";
+ String normalized = ColumnNameNormalizerUtility.getNormalizedName(inputColumnName,true, TranslationStrategy.PATTERN, translationPattern);
Review Comment:
```suggestion
String normalized = ColumnNameNormalizerUtility.getNormalizedName(inputColumnName, true, TranslationStrategy.PATTERN, translationPattern);
```
##########
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/db/ColumnNameNormalizerUtilityTest.java:
##########
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard.db;
+
+import org.junit.jupiter.api.Test;
+
+import java.util.regex.Pattern;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+
+public class ColumnNameNormalizerUtilityTest {
+
+
+
+ @Test
+ void testNormalizingColumnName_RemoveUnderscore() {
+ String inputColumnName = "example_column_name";
+ String expectedNormalized = "EXAMPLECOLUMNNAME";
+ String normalized = ColumnNameNormalizerUtility.getNormalizedName(inputColumnName,true, TranslationStrategy.REMOVE_UNDERSCORE, null);
+
+ assertEquals(expectedNormalized, normalized);
+ }
+
+ @Test
+ void testNormalizingColumnName_RemoveSpace() {
+ String inputColumnName = "Column Name With Spaces";
+ String expectedNormalized = "COLUMNNAMEWITHSPACES";
+ String normalized = ColumnNameNormalizerUtility.getNormalizedName(inputColumnName,true, TranslationStrategy.REMOVE_SPACE, null);
+
+ assertEquals(expectedNormalized, normalized);
+ }
+
+ @Test
+ void testNormalizingColumnName_RemoveAllSpecialCharacters() {
+ String inputColumnName = "Special!Characters@Here$";
+ String expectedNormalized = "SPECIALCHARACTERSHERE";
+ String normalized = ColumnNameNormalizerUtility.getNormalizedName(inputColumnName,true, TranslationStrategy.REMOVE_ALL_SPECIAL_CHAR, null);
Review Comment:
```suggestion
String normalized = ColumnNameNormalizerUtility.getNormalizedName(inputColumnName, true, TranslationStrategy.REMOVE_ALL_SPECIAL_CHAR, null);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11858 Configurable Column Name Normalization in PutDatabaseRecord and UpdateDatabaseTable [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #7544:
URL: https://github.com/apache/nifi/pull/7544#discussion_r1437164753
##########
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/UpdateDatabaseTable.java:
##########
@@ -460,15 +488,18 @@ public void onTrigger(final ProcessContext context, final ProcessSession session
private synchronized OutputMetadataHolder checkAndUpdateTableSchema(final Connection conn, final DatabaseAdapter databaseAdapter, final RecordSchema schema,
final String catalogName, final String schemaName, final String tableName,
- final boolean createIfNotExists, final boolean translateFieldNames, final boolean updateFieldNames,
+ final boolean createIfNotExists, final boolean translateFieldNames,
+ final TranslationStrategy translationStrategy, Pattern translationRegex, final boolean updateFieldNames,
final Set<String> primaryKeyColumnNames, final boolean quoteTableName, final boolean quoteColumnNames) throws IOException {
// Read in the current table metadata, compare it to the reader's schema, and
// add any columns from the schema that are missing in the table
+// final ColumnNameNormalizer normalizer = new ColumnNameNormalizer(translateFieldNames, translationStrategy, translationRegex);
Review Comment:
Remove commented out code
##########
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/ColumnNameNormalizerUtility.java:
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard.db;
+
+
+import java.util.Objects;
+import java.util.regex.Pattern;
+
+/**
+ * ColumnNameNormalizerUtility is a utility class that helps to normalize column names. It provides various strategies to
+ * modify column names based on the TranslationStrategy enum. Column names can be normalized by removing underscores,
+ * spaces, all special characters, or by using a custom regular expression defined by the user.
+ */
+public class ColumnNameNormalizerUtility {
+ // Regular expression to remove all special characters from a string.
+ private static final Pattern REMOVE_ALL_SPECIAL_CHAR_REGEX = Pattern.compile("[^a-zA-Z0-9]");
+
+ /**
+ * Normalizes the given column name based on the specified strategy.
+ *
+ * @param colName The column name to be normalized.
+ * @param isTranslationEnabled Boolean value to denote normalization is enabled
+ * @param strategy The TranslationStrategy for normalizing column name
+ * @param translationRegex Regex For translation
+ * @return The normalized column name as a String.
+ */
+ public static String getNormalizedName(final String colName, boolean isTranslationEnabled, TranslationStrategy strategy, Pattern translationRegex) {
+ // If the column name is null or translation is not enabled, return the original column name.
+ if (colName == null || !isTranslationEnabled) {
+ return colName;
+ }
+
+ return switch (Objects.requireNonNull(strategy)) {
+ case REMOVE_UNDERSCORE -> colName.toUpperCase().replace("_", "");
+ case REMOVE_SPACE -> colName.toUpperCase().replace(" ", "");
+ case REMOVE_ALL_SPECIAL_CHAR -> REMOVE_ALL_SPECIAL_CHAR_REGEX .matcher(colName.toUpperCase()).replaceAll( "");
Review Comment:
```suggestion
case REMOVE_ALL_SPECIAL_CHAR -> REMOVE_ALL_SPECIAL_CHAR_REGEX .matcher(colName.toUpperCase()).replaceAll("");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11858 Configurable Column Name Normalization in PutDatabaseRecord and UpdateDatabaseTable [nifi]
Posted by "ravinarayansingh (via GitHub)" <gi...@apache.org>.
ravinarayansingh commented on code in PR #7544:
URL: https://github.com/apache/nifi/pull/7544#discussion_r1438454454
##########
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/db/ColumnNameNormalizerUtilityTest.java:
##########
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.standard.db;
+
+import org.junit.jupiter.api.Test;
+
+import java.util.regex.Pattern;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+
+public class ColumnNameNormalizerUtilityTest {
+
+
+
+ @Test
+ void testNormalizingColumnName_RemoveUnderscore() {
+ String inputColumnName = "example_column_name";
+ String expectedNormalized = "EXAMPLECOLUMNNAME";
+ String normalized = ColumnNameNormalizerUtility.getNormalizedName(inputColumnName,true, TranslationStrategy.REMOVE_UNDERSCORE, null);
+
+ assertEquals(expectedNormalized, normalized);
+ }
+
+ @Test
+ void testNormalizingColumnName_RemoveSpace() {
+ String inputColumnName = "Column Name With Spaces";
+ String expectedNormalized = "COLUMNNAMEWITHSPACES";
+ String normalized = ColumnNameNormalizerUtility.getNormalizedName(inputColumnName,true, TranslationStrategy.REMOVE_SPACE, null);
+
+ assertEquals(expectedNormalized, normalized);
+ }
+
+ @Test
+ void testNormalizingColumnName_RemoveAllSpecialCharacters() {
+ String inputColumnName = "Special!Characters@Here$";
+ String expectedNormalized = "SPECIALCHARACTERSHERE";
+ String normalized = ColumnNameNormalizerUtility.getNormalizedName(inputColumnName,true, TranslationStrategy.REMOVE_ALL_SPECIAL_CHAR, null);
+
+ assertEquals(expectedNormalized, normalized);
+ }
+
+ @Test
+ void testNormalizingColumnName_Regex() {
+ String inputColumnName = "Your @Input -String Here";
+ Pattern translationPattern = Pattern.compile("[@-]");
+ String expectedNormalized = translationPattern.matcher(inputColumnName.toUpperCase()).replaceAll( "");
Review Comment:
hi @dan-s1
I have made the changes please have a look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11858 Configurable Column Name Normalization in PutDatabaseRecord and UpdateDatabaseTable [nifi]
Posted by "ravinarayansingh (via GitHub)" <gi...@apache.org>.
ravinarayansingh commented on PR #7544:
URL: https://github.com/apache/nifi/pull/7544#issuecomment-2038164793
> @ravinarayansingh Thanks for your work and patience on this pull request.
>
> Unfortunately I may not have been clear in previous comments regarding the interface-based approached for the `ColumnNameNormalizer`. I did not intend for the `ColumnNameNormalizer` to be implemented by each Processor, but instead, the `ColumnNameNormalizer` should have separate implementations for each strategy.
>
> If it would be helpful, I could follow up with a commit that implements the approach I described.
Hi @exceptionfactory ,
could you please provide more details or a code snippet ?
I'm having trouble understanding the issue and would appreciate your clarification on current implementation
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11858 Configurable Column Name Normalization in PutDatabaseRecord and UpdateDatabaseTable [nifi]
Posted by "jrsteinebrey (via GitHub)" <gi...@apache.org>.
jrsteinebrey commented on PR #7544:
URL: https://github.com/apache/nifi/pull/7544#issuecomment-2127792570
Thanks for the changes you made. Some other files in this PR now have checkstyle violations:
Warning: src/main/java/org/apache/nifi/processors/standard/db/TableSchema.java:[155,47] (whitespace) WhitespaceAround: '!=' is not followed by whitespace.
Warning: src/main/java/org/apache/nifi/processors/standard/db/TableSchema.java:[155,47] (whitespace) WhitespaceAround: '!=' is not preceded with whitespace.
Warning: src/test/java/org/apache/nifi/processors/standard/db/impl/TestOracleDatabaseAdapter.java:[124,122] (whitespace) WhitespaceAfter: ',' is not followed by whitespace.
Warning: src/test/java/org/apache/nifi/processors/standard/db/impl/TestOracle12DatabaseAdapter.java:[166,122] (whitespace) WhitespaceAfter: ',' is not followed by whitespace.
Warning: src/test/java/org/apache/nifi/processors/standard/PutDatabaseRecordTest.java:[1832,11] (whitespace) WhitespaceAfter: 'catch' is not followed by whitespace.
Warning: src/test/java/org/apache/nifi/processors/standard/PutDatabaseRecordTest.java:[1832,11] (whitespace) WhitespaceAround: 'catch' is not followed by whitespace.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11858 Configurable Column Name Normalization in PutDatabaseRecord and UpdateDatabaseTable [nifi]
Posted by "jrsteinebrey (via GitHub)" <gi...@apache.org>.
jrsteinebrey commented on code in PR #7544:
URL: https://github.com/apache/nifi/pull/7544#discussion_r1635355735
##########
nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/ColumnDescription.java:
##########
@@ -84,9 +84,6 @@ public static ColumnDescription from(final ResultSet resultSet) throws SQLExcept
return new ColumnDescription(columnName, dataType, required, colSize == 0 ? null : colSize, isNullable);
}
- public static String normalizeColumnName(final String colName, final boolean translateColumnNames) {
- return colName == null ? null : (translateColumnNames ? colName.toUpperCase().replace("_", "") : colName);
Review Comment:
@mattyb149 GitHub shows you as the author of this line 88 back in 2021 when you created the ColumnDescription class. Do you feel that the null check is required here?
Can you think of a use case where colName parameter would be null?
I ask because the replacement code in TableSchema.normalizeName() does NOT have the null check and I want to see if you or others feel the null check is worth having in the new code?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] ravinarayansingh commented on pull request #7544: NiFi 11858 Improve column name normalization in PutDatabaseRecord and UpdateDatabaseTable processor
Posted by "ravinarayansingh (via GitHub)" <gi...@apache.org>.
ravinarayansingh commented on PR #7544:
URL: https://github.com/apache/nifi/pull/7544#issuecomment-1660749574
Hi @exceptionfactory ,
following is the rat report:
*****************************************************
Summary
-------
Generated at: 2023-08-01T08:57:42-07:00
Notes: 24
Binaries: 5
Archives: 3
Standards: 27
Apache Licensed: 13
Generated Documents: 0
JavaDocs are generated, thus a license header is optional.
Generated files do not require license headers.
14 Unknown Licenses
*****************************************************
Files with unapproved licenses:
nifi-h2/nifi-h2-database/nifi-h2-database.iml
nifi-h2/nifi-h2-database/target/.plxarc
nifi-h2/nifi-h2-database/target/maven-archiver/pom.properties
nifi-h2/nifi-h2-database-migrator/nifi-h2-database-migrator.iml
nifi-h2/nifi-h2-database-migrator/target/.plxarc
nifi-h2/nifi-h2-database-migrator/target/maven-archiver/pom.properties
nifi-h2/nifi-h2-database-migrator/target/maven-status/maven-compiler-plugin/compile/default-compile/createdFiles.lst
nifi-h2/nifi-h2-database-migrator/target/maven-status/maven-compiler-plugin/compile/default-compile/inputFiles.lst
nifi-h2/nifi-h2-database-migrator/target/maven-status/maven-compiler-plugin/testCompile/default-testCompile/createdFiles.lst
nifi-h2/nifi-h2-database-migrator/target/maven-status/maven-compiler-plugin/testCompile/default-testCompile/inputFiles.lst
nifi-h2/nifi-h2-database-migrator/target/maven-status/maven-compiler-plugin/testCompile/groovy-tests/createdFiles.lst
nifi-h2/nifi-h2-database-migrator/target/maven-status/maven-compiler-plugin/testCompile/groovy-tests/inputFiles.lst
nifi-h2/nifi-h2.iml
nifi-h2/target/.plxarc
*****************************************************
Archives:
+ nifi-h2/nifi-h2-database/target/nifi-h2-database-1.18.0-SNAPSHOT.jar
+ nifi-h2/nifi-h2-database/target/original-nifi-h2-database-1.18.0-SNAPSHOT.jar
+ nifi-h2/nifi-h2-database-migrator/target/nifi-h2-database-migrator-1.18.0-SNAPSHOT.jar
*****************************************************
Files with Apache License headers will be marked AL
Binary files (which do not require any license headers) will be marked B
Compressed archives will be marked A
Notices, licenses etc. will be marked N
AL .asf.yaml
AL .github/PULL_REQUEST_TEMPLATE.md
AL .github/workflows/ci-workflow.yml
AL .github/workflows/stale.yml
AL .github/workflows/system-tests.yml
AL .mvn/wrapper/maven-wrapper.properties
AL checkstyle.xml
B jffi10666034361742842335.dll
N KEYS
N LICENSE
AL mvnw
AL mvnw.cmd
AL nifi-dependency-check-maven/suppressions.xml
!????? nifi-h2/nifi-h2-database/nifi-h2-database.iml
!????? nifi-h2/nifi-h2-database/target/.plxarc
N nifi-h2/nifi-h2-database/target/classes/META-INF/DEPENDENCIES
N nifi-h2/nifi-h2-database/target/classes/META-INF/LICENSE
N nifi-h2/nifi-h2-database/target/classes/META-INF/NOTICE
!????? nifi-h2/nifi-h2-database/target/maven-archiver/pom.properties
N nifi-h2/nifi-h2-database/target/maven-shared-archive-resources/META-INF/DEPENDENCIES
N nifi-h2/nifi-h2-database/target/maven-shared-archive-resources/META-INF/LICENSE
N nifi-h2/nifi-h2-database/target/maven-shared-archive-resources/META-INF/NOTICE
A nifi-h2/nifi-h2-database/target/nifi-h2-database-1.18.0-SNAPSHOT.jar
A nifi-h2/nifi-h2-database/target/original-nifi-h2-database-1.18.0-SNAPSHOT.jar
N nifi-h2/nifi-h2-database/target/test-classes/META-INF/DEPENDENCIES
N nifi-h2/nifi-h2-database/target/test-classes/META-INF/LICENSE
N nifi-h2/nifi-h2-database/target/test-classes/META-INF/NOTICE
!????? nifi-h2/nifi-h2-database-migrator/nifi-h2-database-migrator.iml
!????? nifi-h2/nifi-h2-database-migrator/target/.plxarc
N nifi-h2/nifi-h2-database-migrator/target/classes/META-INF/DEPENDENCIES
N nifi-h2/nifi-h2-database-migrator/target/classes/META-INF/LICENSE
N nifi-h2/nifi-h2-database-migrator/target/classes/META-INF/NOTICE
B nifi-h2/nifi-h2-database-migrator/target/classes/org/apache/nifi/h2/database/migration/H2DatabaseMigrator.class
B nifi-h2/nifi-h2-database-migrator/target/classes/org/apache/nifi/h2/database/migration/H2DatabaseUpdater.class
!????? nifi-h2/nifi-h2-database-migrator/target/maven-archiver/pom.properties
N nifi-h2/nifi-h2-database-migrator/target/maven-shared-archive-resources/META-INF/DEPENDENCIES
N nifi-h2/nifi-h2-database-migrator/target/maven-shared-archive-resources/META-INF/LICENSE
N nifi-h2/nifi-h2-database-migrator/target/maven-shared-archive-resources/META-INF/NOTICE
!????? nifi-h2/nifi-h2-database-migrator/target/maven-status/maven-compiler-plugin/compile/default-compile/createdFiles.lst
!????? nifi-h2/nifi-h2-database-migrator/target/maven-status/maven-compiler-plugin/compile/default-compile/inputFiles.lst
!????? nifi-h2/nifi-h2-database-migrator/target/maven-status/maven-compiler-plugin/testCompile/default-testCompile/createdFiles.lst
!????? nifi-h2/nifi-h2-database-migrator/target/maven-status/maven-compiler-plugin/testCompile/default-testCompile/inputFiles.lst
!????? nifi-h2/nifi-h2-database-migrator/target/maven-status/maven-compiler-plugin/testCompile/groovy-tests/createdFiles.lst
!????? nifi-h2/nifi-h2-database-migrator/target/maven-status/maven-compiler-plugin/testCompile/groovy-tests/inputFiles.lst
A nifi-h2/nifi-h2-database-migrator/target/nifi-h2-database-migrator-1.18.0-SNAPSHOT.jar
N nifi-h2/nifi-h2-database-migrator/target/test-classes/META-INF/DEPENDENCIES
N nifi-h2/nifi-h2-database-migrator/target/test-classes/META-INF/LICENSE
N nifi-h2/nifi-h2-database-migrator/target/test-classes/META-INF/NOTICE
B nifi-h2/nifi-h2-database-migrator/target/test-classes/nifi-flow-audit.mv.db
B nifi-h2/nifi-h2-database-migrator/target/test-classes/org/apache/nifi/h2/database/migration/TestH2DatabaseUpdater.class
!????? nifi-h2/nifi-h2.iml
!????? nifi-h2/target/.plxarc
N nifi-h2/target/maven-shared-archive-resources/META-INF/DEPENDENCIES
N nifi-h2/target/maven-shared-archive-resources/META-INF/LICENSE
N nifi-h2/target/maven-shared-archive-resources/META-INF/NOTICE
N NOTICE
AL pom.xml
AL README.md
AL SECURITY.md
*****************************************************
let me know if i have to change anything
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11858 Configurable Column Name Normalization in PutDatabaseRecord and UpdateDatabaseTable [nifi]
Posted by "jrsteinebrey (via GitHub)" <gi...@apache.org>.
jrsteinebrey commented on code in PR #7544:
URL: https://github.com/apache/nifi/pull/7544#discussion_r1635355735
##########
nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/ColumnDescription.java:
##########
@@ -84,9 +84,6 @@ public static ColumnDescription from(final ResultSet resultSet) throws SQLExcept
return new ColumnDescription(columnName, dataType, required, colSize == 0 ? null : colSize, isNullable);
}
- public static String normalizeColumnName(final String colName, final boolean translateColumnNames) {
- return colName == null ? null : (translateColumnNames ? colName.toUpperCase().replace("_", "") : colName);
Review Comment:
@mattyb149 GitHub shows you as the author of this line 88 back in 2021 when you created the ColumnDescription class. Do you feel that the null check is required here?
Can you think of a use case where colName parameter would be null?
I ask because the replacement code does NOT have the null check and I want to see if you or others feel the null check is worth having in the new code?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11858 Configurable Column Name Normalization in PutDatabaseRecord and UpdateDatabaseTable [nifi]
Posted by "ravinarayansingh (via GitHub)" <gi...@apache.org>.
ravinarayansingh commented on PR #7544:
URL: https://github.com/apache/nifi/pull/7544#issuecomment-2039271408
> @ravinarayansingh Thanks for your work and patience on this pull request.
>
> Unfortunately I may not have been clear in previous comments regarding the interface-based approached for the `ColumnNameNormalizer`. I did not intend for the `ColumnNameNormalizer` to be implemented by each Processor, but instead, the `ColumnNameNormalizer` should have separate implementations for each strategy.
>
> If it would be helpful, I could follow up with a commit that implements the approach I described.
Hi @exceptionfactory
I have made the required changes please have a look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11858 Configurable Column Name Normalization in PutDatabaseRecord and UpdateDatabaseTable [nifi]
Posted by "ravinarayansingh (via GitHub)" <gi...@apache.org>.
ravinarayansingh commented on PR #7544:
URL: https://github.com/apache/nifi/pull/7544#issuecomment-2007277327
Hi @dan-s1 and @exceptionfactory
please have look and let me know if any change required
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11858 Configurable Column Name Normalization in PutDatabaseRecord and UpdateDatabaseTable [nifi]
Posted by "jrsteinebrey (via GitHub)" <gi...@apache.org>.
jrsteinebrey commented on PR #7544:
URL: https://github.com/apache/nifi/pull/7544#issuecomment-2163395200
Thanks, @ravinarayansingh. My comments that are automatically marked as Outdated are all resolved.
I still have some open existing comments that I recommend changing property names. I understand if you are waiting on those until you see if other reviews have feedback on my property renaming comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on pull request #7544: NIFI-11858 Improve column name normalization in PutDatabaseRecord and UpdateDatabaseTable processor
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on PR #7544:
URL: https://github.com/apache/nifi/pull/7544#issuecomment-1722249456
> Hi @exceptionfactory , when this feature will be available in NiFi Thanks
@ravinarayansingh These changes still need substantive review, along with a number of other pull requests, but thanks for addressing the initial feedback so far.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11858 Configurable Column Name Normalization in PutDatabaseRecord and UpdateDatabaseTable [nifi]
Posted by "ravinarayansingh (via GitHub)" <gi...@apache.org>.
ravinarayansingh commented on PR #7544:
URL: https://github.com/apache/nifi/pull/7544#issuecomment-2076490348
Hi @mattyb149
Build is failing but i think that is not related to this PR
please let me know if i have change anything form myside
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11858 Configurable Column Name Normalization in PutDatabaseRecord and UpdateDatabaseTable [nifi]
Posted by "ravinarayansingh (via GitHub)" <gi...@apache.org>.
ravinarayansingh commented on PR #7544:
URL: https://github.com/apache/nifi/pull/7544#issuecomment-1839974560
Hi @exceptionfactory
i have made the required changes please have a look
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11858 Configurable Column Name Normalization in PutDatabaseRecord and UpdateDatabaseTable [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on PR #7544:
URL: https://github.com/apache/nifi/pull/7544#issuecomment-2041215127
Thanks for the updates @ravinarayansingh, the `ColumnNameNormalizer` with individual implementations matches what I had in mind. I appreciate your efforts, I will take a closer look soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11858 Configurable Column Name Normalization in PutDatabaseRecord and UpdateDatabaseTable [nifi]
Posted by "ravinarayansingh (via GitHub)" <gi...@apache.org>.
ravinarayansingh commented on PR #7544:
URL: https://github.com/apache/nifi/pull/7544#issuecomment-2184174000
> Looks like there are some merge and/or rebase issues here, might be worth a new PR applying the desired commits to a feature branch based off the latest main branch.
Hi @mattyb149
I have created new PR please have a look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11858 Configurable Column Name Normalization in PutDatabaseRecord and UpdateDatabaseTable [nifi]
Posted by "ravinarayansingh (via GitHub)" <gi...@apache.org>.
ravinarayansingh commented on code in PR #7544:
URL: https://github.com/apache/nifi/pull/7544#discussion_r1644904710
##########
nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutDatabaseRecord.java:
##########
@@ -270,6 +274,7 @@ public class PutDatabaseRecord extends AbstractProcessor {
.build();
static final PropertyDescriptor TRANSLATE_FIELD_NAMES = new Builder()
+ .required(true)
.name("put-db-record-translate-field-names")
.displayName("Translate Field Names")
Review Comment:
Hi @exceptionfactory and @exceptionfactory
if you have any thought or suggestion regarding @jrsteinebrey above comment please let me know
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org