You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by GitBox <gi...@apache.org> on 2021/10/24 20:14:59 UTC
[GitHub] [nifi] mikayla-yang opened a new pull request #5476: NIFI-9259: Adding GeohashRecord Processor
mikayla-yang opened a new pull request #5476:
URL: https://github.com/apache/nifi/pull/5476
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
#### Description of PR
_Enables X functionality; fixes bug NIFI-YYYY._
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [x] Does your PR title start with **NIFI-XXXX** where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target branch (typically `main`)?
- [x] Is your initial contribution a single, squashed commit? _Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not `squash` or use `--force` when pushing to allow for clean monitoring of changes._
### For code changes:
- [x] Have you ensured that the full suite of tests is executed via `mvn -Pcontrib-check clean install` at the root `nifi` folder?
- [x] Have you written or updated unit tests to verify your changes?
- [x] Have you verified that the full build is successful on JDK 8?
- [ ] Have you verified that the full build is successful on JDK 11?
- [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main `LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main `NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to .name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for build issues and submit an update to your PR as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] mikayla-yang commented on a change in pull request #5476: NIFI-9259: Adding GeohashRecord Processor
Posted by GitBox <gi...@apache.org>.
mikayla-yang commented on a change in pull request #5476:
URL: https://github.com/apache/nifi/pull/5476#discussion_r749781535
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,449 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.record.path.validation.RecordPathValidator;
+
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.WriteResult;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+
+ public enum ProcessingMode {
+ ENCODE, DECODE
+ }
+
+ public enum GeohashFormat {
+ BASE_32, BINARY_STRING, LONG
+ }
+
+ public enum RoutingStrategy {
+ SKIP_UNENRICHED,
+ SPLIT,
+ REQUIRE_ALL_ENRICHED
+ }
+
+ public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
+ .name("mode")
+ .displayName("Mode")
+ .description("Specifies whether to encode latitude/longitude to geohash or decode geohash to latitude/longitude")
+ .required(true)
+ .allowableValues(ProcessingMode.values())
+ .defaultValue(ProcessingMode.ENCODE.name())
+ .build();
+
+ public static final PropertyDescriptor ROUTING_STRATEGY = new PropertyDescriptor.Builder()
+ .name("routing-strategy")
+ .displayName("Routing Strategy")
+ .description("Specifies how to route records after encoding or decoding has been performed. "
+ + "Skip_Unenriched will route a flowfile to success if any of its records is enriched, otherwise it will be sent to failure. "
+ + "Split_Enriched_Unenriched will separate the records that have been enriched from those have not and send them to success, while unenriched records will be sent to failure; "
+ + "and the original flowfile will be sent to the original relationship. "
+ + "Require_All_Enriched will route a flowfile to success only if all of its records are enriched, otherwise it will be sent to failure")
+ .required(true)
+ .allowableValues(RoutingStrategy.values())
+ .defaultValue(RoutingStrategy.SKIP_UNENRICHED.name())
+ .build();
+
+ public static final PropertyDescriptor RECORD_READER = new PropertyDescriptor.Builder()
+ .name("record-reader")
+ .displayName("Record Reader")
+ .description("Specifies the record reader service to use for reading incoming data")
+ .required(true)
+ .identifiesControllerService(RecordReaderFactory.class)
+ .build();
+
+ public static final PropertyDescriptor RECORD_WRITER = new PropertyDescriptor.Builder()
+ .name("record-writer")
+ .displayName("Record Writer")
+ .description("Specifies the record writer service to use for writing data")
+ .required(true)
+ .identifiesControllerService(RecordSetWriterFactory.class)
+ .build();
+
+ public static final PropertyDescriptor LATITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("latitude-record-path")
+ .displayName("Latitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the latitude value; "
+ + "in the decode mode, this property specifies the record path to put the latitude value")
+ .required(true)
+ .addValidator(new RecordPathValidator())
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor LONGITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("longitude-record-path")
+ .displayName("Longitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the longitude value; "
+ + "in the decode mode, this property specifies the record path to put the longitude value")
+ .required(true)
+ .addValidator(new RecordPathValidator())
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_FORMAT = new PropertyDescriptor.Builder()
+ .name("geohash-format")
+ .displayName("Geohash Format")
+ .description("In the encode mode, this property specifies the desired format for encoding geohash; "
+ + "in the decode mode, this property specifies the format of geohash provided")
+ .required(true)
+ .allowableValues(GeohashFormat.values())
+ .defaultValue(GeohashFormat.BASE_32.name())
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_LEVEL = new PropertyDescriptor.Builder()
+ .name("geohash-level")
+ .displayName("Geohash Level")
+ .description("The integer precision level(1-12) desired for encoding geohash")
+ .required(true)
+ .addValidator(StandardValidators.createLongValidator(1, 12, true))
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .dependsOn(MODE, ProcessingMode.ENCODE.name())
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("geohash-record-path")
+ .displayName("Geohash Record Path")
+ .description("In the encode mode, this property specifies the record path to put the geohash value; "
+ + "in the decode mode, this property specifies the record path to retrieve the geohash value")
+ .required(true)
+ .addValidator(new RecordPathValidator())
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Records that are successfully encoded or decoded will be routed to success")
+ .build();
+
+ public static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Records that cannot be encoded or decoded will be routed to failure")
+ .build();
+
+ public static final Relationship REL_ORIGINAL = new Relationship.Builder()
+ .name("original")
+ .description("With the Split_Enriched_Unenriched strategy, the original input flowfile will be sent to this relationship regardless of whether it was enriched or not.")
+ .build();
+
+ private static final List<PropertyDescriptor> RECORD_PATH_PROPERTIES = Collections.unmodifiableList(Arrays.asList(
+ LATITUDE_RECORD_PATH, LONGITUDE_RECORD_PATH, GEOHASH_RECORD_PATH
+ ));
+
+ private final RecordPathCache cache = new RecordPathCache(100);
+
+ private List<PropertyDescriptor> descriptors;
+
+ private Set<Relationship> relationships;
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ descriptors = new ArrayList<>();
+ descriptors.add(MODE);
+ descriptors.add(RECORD_READER);
+ descriptors.add(RECORD_WRITER);
+ descriptors.add(ROUTING_STRATEGY);
+ descriptors.add(LATITUDE_RECORD_PATH);
+ descriptors.add(LONGITUDE_RECORD_PATH);
+ descriptors.add(GEOHASH_RECORD_PATH);
+ descriptors.add(GEOHASH_FORMAT);
+ descriptors.add(GEOHASH_LEVEL);
+ descriptors = Collections.unmodifiableList(descriptors);
+
+ relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ relationships.add(REL_FAILURE);
+ relationships.add(REL_ORIGINAL);
+ relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) {
+ FlowFile input = session.get();
+ if (input == null) {
+ return;
+ }
+
+ final RecordReaderFactory readerFactory = context.getProperty(RECORD_READER).asControllerService(RecordReaderFactory.class);
+ final RecordSetWriterFactory writerFactory = context.getProperty(RECORD_WRITER).asControllerService(RecordSetWriterFactory.class);
+ final boolean encode = context.getProperty(MODE).getValue().equalsIgnoreCase(ProcessingMode.ENCODE.toString());
+ final RoutingStrategy routingStrategy = RoutingStrategy.valueOf(context.getProperty(ROUTING_STRATEGY).getValue());
+ final String format = context.getProperty(GEOHASH_FORMAT).getValue();
+
+ FlowFile output = session.create(input);
+ FlowFile notFound = routingStrategy == RoutingStrategy.SPLIT ? session.create(input) : null;
+
+ try (InputStream is = session.read(input);
+ RecordReader reader = readerFactory.createRecordReader(input, is, getLogger());
+ OutputStream os = session.write(output);
+ RecordSetWriter writer = writerFactory.createWriter(getLogger(), writerFactory.getSchema(input.getAttributes(), reader.getSchema()), os, output);
+ OutputStream osNotFound = routingStrategy == RoutingStrategy.SPLIT ? session.write(notFound) : null;
+ RecordSetWriter notFoundWriter = routingStrategy == RoutingStrategy.SPLIT ? writerFactory.createWriter(getLogger(), reader.getSchema(), osNotFound, notFound) : null) {
+
+ Map<PropertyDescriptor, RecordPath> paths = new HashMap<>();
+ for (PropertyDescriptor descriptor : RECORD_PATH_PROPERTIES) {
+ String rawRecordPath = context.getProperty(descriptor).evaluateAttributeExpressions(input).getValue();
+ RecordPath compiled = cache.getCompiled(rawRecordPath);
+ paths.put(descriptor, compiled);
+ }
+
+ Record record;
+
+ //The overall relationship used by the Require_All_Enriched and Skip_Unenriched routing strategies to transfer Flowfiles.
+ //For the Split_Enriched_Unenriched strategy, the transfer of Flowfiles does not rely on this overall relationship. Instead, each individual record will be sent to success or failure.
+ Relationship targetRelationship = routingStrategy == RoutingStrategy.REQUIRE_ALL_ENRICHED ? REL_SUCCESS : REL_FAILURE;
+
+ writer.beginRecordSet();
+
+ if (notFoundWriter != null) {
+ notFoundWriter.beginRecordSet();
+ }
+
+ int foundCount = 0;
+ int notFoundCount = 0;
+
+ int level = context.getProperty(GEOHASH_LEVEL).asInteger();
+ final String rawLatitudePath = context.getProperty(LATITUDE_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath latitudePath = cache.getCompiled(rawLatitudePath);
+ final String rawLongitudePath = context.getProperty(LONGITUDE_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath longitudePath = cache.getCompiled(rawLongitudePath);
+ final String rawgeohashPath = context.getProperty(GEOHASH_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath geohashPath = cache.getCompiled(rawgeohashPath);
+
+ while ((record = reader.nextRecord()) != null) {
+ boolean updated = false;
+ if (encode) {
+ Object encodedGeohash = getEncodedGeohash(latitudePath, longitudePath, record, format, level);
+ updated = updateRecord(GEOHASH_RECORD_PATH, encodedGeohash, record, paths);
+ } else {
+ WGS84Point decodedPoint = getDecodedPointFromGeohash(geohashPath, record, format);
+ if (decodedPoint != null) {
+ updated = updateRecord(LATITUDE_RECORD_PATH, String.valueOf(decodedPoint.getLatitude()), record, paths)
+ && updateRecord(LONGITUDE_RECORD_PATH, String.valueOf(decodedPoint.getLongitude()), record, paths);
+ }
+ }
+
+ if (routingStrategy == RoutingStrategy.REQUIRE_ALL_ENRICHED) {
+ if (!updated) { //If the routing strategy is Require_All_Enriched and there exists a record that is not updated, the entire flowfile should be route to REL_FAILURE
+ targetRelationship = REL_FAILURE;
+ }
+ } else {
+ if (updated) {
+ //If the routing strategy is Skip_Unenriched and there exists a record that is updated, the entire flowfile should be route to REL_SUCCESS
+ //If the routing strategy is Split_Enriched_Unenriched and there exists a record that is updated, this record should be route to REL_SUCCESS
+ targetRelationship = REL_SUCCESS;
+ }
+ }
+
+ if (routingStrategy != RoutingStrategy.SPLIT || updated) {
+ writer.write(record);
+ foundCount++;
+ } else { //if the routing strategy is Split_Enriched_Unenriched and the record is not updated
+ notFoundWriter.write(record);
+ notFoundCount++;
+ }
+ }
+
+ final WriteResult writeResult = writer.finishRecordSet();
+ writer.close();
+
+ WriteResult notFoundWriterResult = null;
+
+ if (notFoundWriter != null) {
+ notFoundWriterResult = notFoundWriter.finishRecordSet();
+ notFoundWriter.close();
+ }
+
+ is.close();
+ os.close();
+
+ if (osNotFound != null) {
+ osNotFound.close();
+ }
Review comment:
Thanks for the comment! I have removed all the close() except the ones for `writer` and `notFoundWriter` because they need to be closed manually.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] markap14 commented on a change in pull request #5476: NIFI-9259: Adding GeohashRecord Processor
Posted by GitBox <gi...@apache.org>.
markap14 commented on a change in pull request #5476:
URL: https://github.com/apache/nifi/pull/5476#discussion_r766160936
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-nar/src/main/resources/META-INF/NOTICE
##########
@@ -0,0 +1,16 @@
+nifi-geohash-nar
+Copyright 2021-2021 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+******************
+Apache Software License v2
+******************
+
+The following binary components are provided under the Apache Software License v2
+
+ (ASLv2) Geohash Java
+ The following NOTICE information applies:
+ Geohash Java
+ Copyright 2016 Silvio Heuberger and contributors
Review comment:
We should not have any Apache licensed software going into the NOTICE file. We can remove this NOTICE file all together.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a change in pull request #5476: NIFI-9259: Adding GeohashRecord Processor
Posted by GitBox <gi...@apache.org>.
exceptionfactory commented on a change in pull request #5476:
URL: https://github.com/apache/nifi/pull/5476#discussion_r737870081
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,367 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+ public static final String ENCODE_MODE = "Encode";
+ public static final String DECODE_MODE = "Decode";
+
+ public static final String BINARY_STRING = "BinaryString";
+ public static final String BASE32 = "Base32";
+ public static final String LONG_INT = "LongInt";
+
+ public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
+ .name("mode")
+ .displayName("Mode")
+ .description("Specifies whether to encode latitude/longitude to geohash or decode geohash to latitude/longitude")
+ .required(true)
+ .allowableValues(ENCODE_MODE, DECODE_MODE)
+ .defaultValue(ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor RECORD_READER = new PropertyDescriptor.Builder()
+ .name("record-reader")
+ .displayName("Record Reader")
+ .description("Specifies the record reader service to use for reading incoming data")
+ .required(true)
+ .identifiesControllerService(RecordReaderFactory.class)
+ .build();
+
+ public static final PropertyDescriptor RECORD_WRITER = new PropertyDescriptor.Builder()
+ .name("record-writer")
+ .displayName("Record Writer")
+ .description("Specifies the record writer service to use for writing data")
+ .required(true)
+ .identifiesControllerService(RecordSetWriterFactory.class)
+ .build();
+
+ public static final PropertyDescriptor LATITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("latitude-record-path")
+ .displayName("Latitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the latitude value;"
+ + "in the decode mode, this property specifies the record path to put the latitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor LONGITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("longitude-record-path")
+ .displayName("Longitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the longitude value;"
+ + "in the decode mode, this property specifies the record path to put the longitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_FORMAT = new PropertyDescriptor.Builder()
+ .name("geohash-format")
+ .displayName("Geohash Format")
+ .description("In the encode mode, this property specifies the desired format for encoding geohash;"
+ + "in the decode mode, this property specifies the format of geohash provided")
+ .required(true)
+ .allowableValues(BASE32, BINARY_STRING, LONG_INT)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_LEVEL = new PropertyDescriptor.Builder()
+ .name("geohash-level")
+ .displayName("Geohash Level")
+ .description("The integer precision level(1-12) desired for encoding geohash")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(MODE, ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("geohash-record-path")
+ .displayName("Geohash Record Path")
+ .description("In the encode mode, this property specifies the record path to put the geohash value;"
+ + "in the decode mode, this property specifies the record path to retrieve the geohash value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Records that are successfully encoded or decoded will be routed to success")
+ .build();
+ public static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Records that cannot be encoded or decoded will be routed to failure")
+ .build();
+
+ public static final List<PropertyDescriptor> RECORD_PATH_PROPERTIES = Collections.unmodifiableList(Arrays.asList(
Review comment:
It looks like this could be changed to `private`.
```suggestion
private static final List<PropertyDescriptor> RECORD_PATH_PROPERTIES = Collections.unmodifiableList(Arrays.asList(
```
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,367 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+ public static final String ENCODE_MODE = "Encode";
+ public static final String DECODE_MODE = "Decode";
+
+ public static final String BINARY_STRING = "BinaryString";
+ public static final String BASE32 = "Base32";
+ public static final String LONG_INT = "LongInt";
Review comment:
Recommend encapsulated these values in a separate `enum` named something like `GeohashFormat`, which could be used directly in `allowableValues()`.
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,367 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+ public static final String ENCODE_MODE = "Encode";
+ public static final String DECODE_MODE = "Decode";
+
+ public static final String BINARY_STRING = "BinaryString";
+ public static final String BASE32 = "Base32";
+ public static final String LONG_INT = "LongInt";
+
+ public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
+ .name("mode")
+ .displayName("Mode")
+ .description("Specifies whether to encode latitude/longitude to geohash or decode geohash to latitude/longitude")
+ .required(true)
+ .allowableValues(ENCODE_MODE, DECODE_MODE)
+ .defaultValue(ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor RECORD_READER = new PropertyDescriptor.Builder()
+ .name("record-reader")
+ .displayName("Record Reader")
+ .description("Specifies the record reader service to use for reading incoming data")
+ .required(true)
+ .identifiesControllerService(RecordReaderFactory.class)
+ .build();
+
+ public static final PropertyDescriptor RECORD_WRITER = new PropertyDescriptor.Builder()
+ .name("record-writer")
+ .displayName("Record Writer")
+ .description("Specifies the record writer service to use for writing data")
+ .required(true)
+ .identifiesControllerService(RecordSetWriterFactory.class)
+ .build();
+
+ public static final PropertyDescriptor LATITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("latitude-record-path")
+ .displayName("Latitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the latitude value;"
+ + "in the decode mode, this property specifies the record path to put the latitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor LONGITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("longitude-record-path")
+ .displayName("Longitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the longitude value;"
+ + "in the decode mode, this property specifies the record path to put the longitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_FORMAT = new PropertyDescriptor.Builder()
+ .name("geohash-format")
+ .displayName("Geohash Format")
+ .description("In the encode mode, this property specifies the desired format for encoding geohash;"
+ + "in the decode mode, this property specifies the format of geohash provided")
+ .required(true)
+ .allowableValues(BASE32, BINARY_STRING, LONG_INT)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_LEVEL = new PropertyDescriptor.Builder()
+ .name("geohash-level")
+ .displayName("Geohash Level")
+ .description("The integer precision level(1-12) desired for encoding geohash")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(MODE, ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("geohash-record-path")
+ .displayName("Geohash Record Path")
+ .description("In the encode mode, this property specifies the record path to put the geohash value;"
+ + "in the decode mode, this property specifies the record path to retrieve the geohash value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Records that are successfully encoded or decoded will be routed to success")
+ .build();
+ public static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Records that cannot be encoded or decoded will be routed to failure")
+ .build();
+
+ public static final List<PropertyDescriptor> RECORD_PATH_PROPERTIES = Collections.unmodifiableList(Arrays.asList(
+ LATITUDE_RECORD_PATH,LONGITUDE_RECORD_PATH,GEOHASH_RECORD_PATH
+ ));
+
+ private List<PropertyDescriptor> descriptors;
+
+ private Set<Relationship> relationships;
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ descriptors = new ArrayList<>();
+ descriptors.add(MODE);
+ descriptors.add(RECORD_READER);
+ descriptors.add(RECORD_WRITER);
+ descriptors.add(LATITUDE_RECORD_PATH);
+ descriptors.add(LONGITUDE_RECORD_PATH);
+ descriptors.add(GEOHASH_FORMAT);
+ descriptors.add(GEOHASH_LEVEL);
+ descriptors.add(GEOHASH_RECORD_PATH);
+ descriptors = Collections.unmodifiableList(descriptors);
+
+ relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ relationships.add(REL_FAILURE);
+ relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) {
+ FlowFile input = session.get();
+ if ( input == null ) {
+ return;
+ }
+
+ final RecordReaderFactory readerFactory = context.getProperty(RECORD_READER).asControllerService(RecordReaderFactory.class);
+ final RecordSetWriterFactory writerFactory = context.getProperty(RECORD_WRITER).asControllerService(RecordSetWriterFactory.class);
+ final boolean encode = context.getProperty(MODE).getValue().equalsIgnoreCase(ENCODE_MODE);
+ final String format = context.getProperty(GEOHASH_FORMAT).getValue();
+
+ FlowFile output = session.create(input);
Review comment:
Is there a reason for creating a new output FlowFile as opposed to changing the content of the input FlowFile? It looks like the input FlowFile is removed, so it seems like the approach could be refactored using a `StreamCallback` to process the InputStream and OuputStream together.
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,367 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+ public static final String ENCODE_MODE = "Encode";
+ public static final String DECODE_MODE = "Decode";
Review comment:
These values could be encapsulated in an `enum` named something like `ProcessingMode`.
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,367 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+ public static final String ENCODE_MODE = "Encode";
+ public static final String DECODE_MODE = "Decode";
+
+ public static final String BINARY_STRING = "BinaryString";
+ public static final String BASE32 = "Base32";
+ public static final String LONG_INT = "LongInt";
+
+ public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
+ .name("mode")
+ .displayName("Mode")
+ .description("Specifies whether to encode latitude/longitude to geohash or decode geohash to latitude/longitude")
+ .required(true)
+ .allowableValues(ENCODE_MODE, DECODE_MODE)
+ .defaultValue(ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor RECORD_READER = new PropertyDescriptor.Builder()
+ .name("record-reader")
+ .displayName("Record Reader")
+ .description("Specifies the record reader service to use for reading incoming data")
+ .required(true)
+ .identifiesControllerService(RecordReaderFactory.class)
+ .build();
+
+ public static final PropertyDescriptor RECORD_WRITER = new PropertyDescriptor.Builder()
+ .name("record-writer")
+ .displayName("Record Writer")
+ .description("Specifies the record writer service to use for writing data")
+ .required(true)
+ .identifiesControllerService(RecordSetWriterFactory.class)
+ .build();
+
+ public static final PropertyDescriptor LATITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("latitude-record-path")
+ .displayName("Latitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the latitude value;"
+ + "in the decode mode, this property specifies the record path to put the latitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor LONGITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("longitude-record-path")
+ .displayName("Longitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the longitude value;"
+ + "in the decode mode, this property specifies the record path to put the longitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_FORMAT = new PropertyDescriptor.Builder()
+ .name("geohash-format")
+ .displayName("Geohash Format")
+ .description("In the encode mode, this property specifies the desired format for encoding geohash;"
+ + "in the decode mode, this property specifies the format of geohash provided")
+ .required(true)
+ .allowableValues(BASE32, BINARY_STRING, LONG_INT)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_LEVEL = new PropertyDescriptor.Builder()
+ .name("geohash-level")
+ .displayName("Geohash Level")
+ .description("The integer precision level(1-12) desired for encoding geohash")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(MODE, ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("geohash-record-path")
+ .displayName("Geohash Record Path")
+ .description("In the encode mode, this property specifies the record path to put the geohash value;"
+ + "in the decode mode, this property specifies the record path to retrieve the geohash value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Records that are successfully encoded or decoded will be routed to success")
+ .build();
+ public static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Records that cannot be encoded or decoded will be routed to failure")
+ .build();
+
+ public static final List<PropertyDescriptor> RECORD_PATH_PROPERTIES = Collections.unmodifiableList(Arrays.asList(
+ LATITUDE_RECORD_PATH,LONGITUDE_RECORD_PATH,GEOHASH_RECORD_PATH
+ ));
+
+ private List<PropertyDescriptor> descriptors;
+
+ private Set<Relationship> relationships;
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ descriptors = new ArrayList<>();
+ descriptors.add(MODE);
+ descriptors.add(RECORD_READER);
+ descriptors.add(RECORD_WRITER);
+ descriptors.add(LATITUDE_RECORD_PATH);
+ descriptors.add(LONGITUDE_RECORD_PATH);
+ descriptors.add(GEOHASH_FORMAT);
+ descriptors.add(GEOHASH_LEVEL);
+ descriptors.add(GEOHASH_RECORD_PATH);
+ descriptors = Collections.unmodifiableList(descriptors);
+
+ relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ relationships.add(REL_FAILURE);
+ relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) {
+ FlowFile input = session.get();
+ if ( input == null ) {
Review comment:
Recommend adjusting the spacing to remove extra spaces:
```suggestion
if (input == null) {
```
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/test/java/org/apache/nifi/processors/geohash/GeohashRecordTest.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.avro.Schema;
+import org.apache.commons.io.IOUtils;
+import org.apache.nifi.avro.AvroTypeUtil;
+import org.apache.nifi.controller.ControllerService;
+import org.apache.nifi.json.JsonRecordSetWriter;
+import org.apache.nifi.json.JsonTreeReader;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.schema.access.SchemaAccessUtils;
+import org.apache.nifi.serialization.record.MockSchemaRegistry;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.codehaus.jackson.map.ObjectMapper;
+
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
Review comment:
Since this is a new test class, recommend adjusting to use JUnit 5 instead of JUnit 4. The dependency is already available as part of standard dependency declarations.
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/pom.xml
##########
@@ -0,0 +1,108 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-geohash-bundle</artifactId>
+ <version>1.15.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-geohash-processors</artifactId>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-utils</artifactId>
+ <version>1.15.0-SNAPSHOT</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-mock</artifactId>
+ <version>1.15.0-SNAPSHOT</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-simple</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>junit</groupId>
+ <artifactId>junit</artifactId>
+ <scope>test</scope>
+ </dependency>
Review comment:
These dependencies are included as part of the default module declarations, so they can be removed.
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,367 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+ public static final String ENCODE_MODE = "Encode";
+ public static final String DECODE_MODE = "Decode";
+
+ public static final String BINARY_STRING = "BinaryString";
+ public static final String BASE32 = "Base32";
+ public static final String LONG_INT = "LongInt";
+
+ public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
+ .name("mode")
+ .displayName("Mode")
+ .description("Specifies whether to encode latitude/longitude to geohash or decode geohash to latitude/longitude")
+ .required(true)
+ .allowableValues(ENCODE_MODE, DECODE_MODE)
+ .defaultValue(ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor RECORD_READER = new PropertyDescriptor.Builder()
+ .name("record-reader")
+ .displayName("Record Reader")
+ .description("Specifies the record reader service to use for reading incoming data")
+ .required(true)
+ .identifiesControllerService(RecordReaderFactory.class)
+ .build();
+
+ public static final PropertyDescriptor RECORD_WRITER = new PropertyDescriptor.Builder()
+ .name("record-writer")
+ .displayName("Record Writer")
+ .description("Specifies the record writer service to use for writing data")
+ .required(true)
+ .identifiesControllerService(RecordSetWriterFactory.class)
+ .build();
+
+ public static final PropertyDescriptor LATITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("latitude-record-path")
+ .displayName("Latitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the latitude value;"
+ + "in the decode mode, this property specifies the record path to put the latitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor LONGITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("longitude-record-path")
+ .displayName("Longitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the longitude value;"
+ + "in the decode mode, this property specifies the record path to put the longitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_FORMAT = new PropertyDescriptor.Builder()
+ .name("geohash-format")
+ .displayName("Geohash Format")
+ .description("In the encode mode, this property specifies the desired format for encoding geohash;"
+ + "in the decode mode, this property specifies the format of geohash provided")
+ .required(true)
+ .allowableValues(BASE32, BINARY_STRING, LONG_INT)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_LEVEL = new PropertyDescriptor.Builder()
+ .name("geohash-level")
+ .displayName("Geohash Level")
+ .description("The integer precision level(1-12) desired for encoding geohash")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(MODE, ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("geohash-record-path")
+ .displayName("Geohash Record Path")
+ .description("In the encode mode, this property specifies the record path to put the geohash value;"
+ + "in the decode mode, this property specifies the record path to retrieve the geohash value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Records that are successfully encoded or decoded will be routed to success")
+ .build();
+ public static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Records that cannot be encoded or decoded will be routed to failure")
+ .build();
+
+ public static final List<PropertyDescriptor> RECORD_PATH_PROPERTIES = Collections.unmodifiableList(Arrays.asList(
+ LATITUDE_RECORD_PATH,LONGITUDE_RECORD_PATH,GEOHASH_RECORD_PATH
+ ));
+
+ private List<PropertyDescriptor> descriptors;
+
+ private Set<Relationship> relationships;
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ descriptors = new ArrayList<>();
+ descriptors.add(MODE);
+ descriptors.add(RECORD_READER);
+ descriptors.add(RECORD_WRITER);
+ descriptors.add(LATITUDE_RECORD_PATH);
+ descriptors.add(LONGITUDE_RECORD_PATH);
+ descriptors.add(GEOHASH_FORMAT);
+ descriptors.add(GEOHASH_LEVEL);
+ descriptors.add(GEOHASH_RECORD_PATH);
+ descriptors = Collections.unmodifiableList(descriptors);
+
+ relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ relationships.add(REL_FAILURE);
+ relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) {
+ FlowFile input = session.get();
+ if ( input == null ) {
+ return;
+ }
+
+ final RecordReaderFactory readerFactory = context.getProperty(RECORD_READER).asControllerService(RecordReaderFactory.class);
+ final RecordSetWriterFactory writerFactory = context.getProperty(RECORD_WRITER).asControllerService(RecordSetWriterFactory.class);
+ final boolean encode = context.getProperty(MODE).getValue().equalsIgnoreCase(ENCODE_MODE);
+ final String format = context.getProperty(GEOHASH_FORMAT).getValue();
+
+ FlowFile output = session.create(input);
+ try (InputStream is = session.read(input);
+ OutputStream os = session.write(output)) {
+ RecordPathCache cache = new RecordPathCache(RECORD_PATH_PROPERTIES.size() + 1);
+ Map<PropertyDescriptor, RecordPath> paths = new HashMap<>();
+
+ for (PropertyDescriptor descriptor : RECORD_PATH_PROPERTIES) {
+ String rawRecordPath = context.getProperty(descriptor).evaluateAttributeExpressions(input).getValue();
+ RecordPath compiled = cache.getCompiled(rawRecordPath);
+ paths.put(descriptor, compiled);
+ }
+
+ RecordReader reader = readerFactory.createRecordReader(input, is, getLogger());
+ RecordSchema schema = writerFactory.getSchema(input.getAttributes(), reader.getSchema());
+ RecordSetWriter writer = writerFactory.createWriter(getLogger(), schema, os, output);
+
+ Record record;
+ Relationship targetRelationship = REL_FAILURE;
+ writer.beginRecordSet();
+
+ int recordsCount = 0;
+ boolean updated = false;
+ while ((record = reader.nextRecord()) != null) {
+ if(encode) {
+ final int level = Integer.parseInt(context.getProperty(GEOHASH_LEVEL).getValue());
+ final String rawLatitudePath = context.getProperty(LATITUDE_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath latitudePath = cache.getCompiled(rawLatitudePath);
+ final String rawLongitudePath = context.getProperty(LONGITUDE_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath longitudePath = cache.getCompiled(rawLongitudePath);
+
+ Object encodedGeohash = getEncodedGeohash(latitudePath, longitudePath, record, format, level);
+
+ updated = updateRecord(GEOHASH_RECORD_PATH, encodedGeohash, record, paths);
+ }else {
+ final String rawgeohashPath = context.getProperty(GEOHASH_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath geohashPath = cache.getCompiled(rawgeohashPath);
+
+ WGS84Point decodedPoint = getDecodedPointFromGeohash(geohashPath, record, format);
+
+ if(decodedPoint != null) {
+ updated = updateRecord(LATITUDE_RECORD_PATH, String.valueOf(decodedPoint.getLatitude()), record, paths)
+ && updateRecord(LONGITUDE_RECORD_PATH, String.valueOf(decodedPoint.getLongitude()), record, paths);
+ }
+ }
+
+ if (updated) {
+ targetRelationship = REL_SUCCESS;
+ }
+
+ writer.write(record);
+ recordsCount++;
+
+ }
+ writer.finishRecordSet();
+ writer.close();
+
+ is.close();
+ os.close();
Review comment:
If the implementation were refactored using a `StreamCallback` implementation, closing the InputStream and OutputStream would happen automatically.
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,367 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+ public static final String ENCODE_MODE = "Encode";
+ public static final String DECODE_MODE = "Decode";
+
+ public static final String BINARY_STRING = "BinaryString";
+ public static final String BASE32 = "Base32";
+ public static final String LONG_INT = "LongInt";
+
+ public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
+ .name("mode")
+ .displayName("Mode")
+ .description("Specifies whether to encode latitude/longitude to geohash or decode geohash to latitude/longitude")
+ .required(true)
+ .allowableValues(ENCODE_MODE, DECODE_MODE)
+ .defaultValue(ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor RECORD_READER = new PropertyDescriptor.Builder()
+ .name("record-reader")
+ .displayName("Record Reader")
+ .description("Specifies the record reader service to use for reading incoming data")
+ .required(true)
+ .identifiesControllerService(RecordReaderFactory.class)
+ .build();
+
+ public static final PropertyDescriptor RECORD_WRITER = new PropertyDescriptor.Builder()
+ .name("record-writer")
+ .displayName("Record Writer")
+ .description("Specifies the record writer service to use for writing data")
+ .required(true)
+ .identifiesControllerService(RecordSetWriterFactory.class)
+ .build();
+
+ public static final PropertyDescriptor LATITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("latitude-record-path")
+ .displayName("Latitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the latitude value;"
+ + "in the decode mode, this property specifies the record path to put the latitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor LONGITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("longitude-record-path")
+ .displayName("Longitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the longitude value;"
+ + "in the decode mode, this property specifies the record path to put the longitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_FORMAT = new PropertyDescriptor.Builder()
+ .name("geohash-format")
+ .displayName("Geohash Format")
+ .description("In the encode mode, this property specifies the desired format for encoding geohash;"
+ + "in the decode mode, this property specifies the format of geohash provided")
+ .required(true)
+ .allowableValues(BASE32, BINARY_STRING, LONG_INT)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_LEVEL = new PropertyDescriptor.Builder()
+ .name("geohash-level")
+ .displayName("Geohash Level")
+ .description("The integer precision level(1-12) desired for encoding geohash")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(MODE, ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("geohash-record-path")
+ .displayName("Geohash Record Path")
+ .description("In the encode mode, this property specifies the record path to put the geohash value;"
+ + "in the decode mode, this property specifies the record path to retrieve the geohash value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Records that are successfully encoded or decoded will be routed to success")
+ .build();
+ public static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Records that cannot be encoded or decoded will be routed to failure")
+ .build();
+
+ public static final List<PropertyDescriptor> RECORD_PATH_PROPERTIES = Collections.unmodifiableList(Arrays.asList(
+ LATITUDE_RECORD_PATH,LONGITUDE_RECORD_PATH,GEOHASH_RECORD_PATH
+ ));
+
+ private List<PropertyDescriptor> descriptors;
+
+ private Set<Relationship> relationships;
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ descriptors = new ArrayList<>();
+ descriptors.add(MODE);
+ descriptors.add(RECORD_READER);
+ descriptors.add(RECORD_WRITER);
+ descriptors.add(LATITUDE_RECORD_PATH);
+ descriptors.add(LONGITUDE_RECORD_PATH);
+ descriptors.add(GEOHASH_FORMAT);
+ descriptors.add(GEOHASH_LEVEL);
+ descriptors.add(GEOHASH_RECORD_PATH);
+ descriptors = Collections.unmodifiableList(descriptors);
+
+ relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ relationships.add(REL_FAILURE);
+ relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) {
+ FlowFile input = session.get();
+ if ( input == null ) {
+ return;
+ }
+
+ final RecordReaderFactory readerFactory = context.getProperty(RECORD_READER).asControllerService(RecordReaderFactory.class);
+ final RecordSetWriterFactory writerFactory = context.getProperty(RECORD_WRITER).asControllerService(RecordSetWriterFactory.class);
+ final boolean encode = context.getProperty(MODE).getValue().equalsIgnoreCase(ENCODE_MODE);
+ final String format = context.getProperty(GEOHASH_FORMAT).getValue();
+
+ FlowFile output = session.create(input);
+ try (InputStream is = session.read(input);
+ OutputStream os = session.write(output)) {
+ RecordPathCache cache = new RecordPathCache(RECORD_PATH_PROPERTIES.size() + 1);
+ Map<PropertyDescriptor, RecordPath> paths = new HashMap<>();
+
+ for (PropertyDescriptor descriptor : RECORD_PATH_PROPERTIES) {
+ String rawRecordPath = context.getProperty(descriptor).evaluateAttributeExpressions(input).getValue();
+ RecordPath compiled = cache.getCompiled(rawRecordPath);
+ paths.put(descriptor, compiled);
+ }
+
+ RecordReader reader = readerFactory.createRecordReader(input, is, getLogger());
+ RecordSchema schema = writerFactory.getSchema(input.getAttributes(), reader.getSchema());
+ RecordSetWriter writer = writerFactory.createWriter(getLogger(), schema, os, output);
+
+ Record record;
+ Relationship targetRelationship = REL_FAILURE;
+ writer.beginRecordSet();
+
+ int recordsCount = 0;
+ boolean updated = false;
+ while ((record = reader.nextRecord()) != null) {
+ if(encode) {
+ final int level = Integer.parseInt(context.getProperty(GEOHASH_LEVEL).getValue());
+ final String rawLatitudePath = context.getProperty(LATITUDE_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath latitudePath = cache.getCompiled(rawLatitudePath);
+ final String rawLongitudePath = context.getProperty(LONGITUDE_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath longitudePath = cache.getCompiled(rawLongitudePath);
+
+ Object encodedGeohash = getEncodedGeohash(latitudePath, longitudePath, record, format, level);
+
+ updated = updateRecord(GEOHASH_RECORD_PATH, encodedGeohash, record, paths);
+ }else {
Review comment:
Recommend running automatic formatting to correct the spacing here and on some other lines.
```suggestion
} else {
```
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,367 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+ public static final String ENCODE_MODE = "Encode";
+ public static final String DECODE_MODE = "Decode";
+
+ public static final String BINARY_STRING = "BinaryString";
+ public static final String BASE32 = "Base32";
+ public static final String LONG_INT = "LongInt";
+
+ public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
+ .name("mode")
+ .displayName("Mode")
+ .description("Specifies whether to encode latitude/longitude to geohash or decode geohash to latitude/longitude")
+ .required(true)
+ .allowableValues(ENCODE_MODE, DECODE_MODE)
+ .defaultValue(ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor RECORD_READER = new PropertyDescriptor.Builder()
+ .name("record-reader")
+ .displayName("Record Reader")
+ .description("Specifies the record reader service to use for reading incoming data")
+ .required(true)
+ .identifiesControllerService(RecordReaderFactory.class)
+ .build();
+
+ public static final PropertyDescriptor RECORD_WRITER = new PropertyDescriptor.Builder()
+ .name("record-writer")
+ .displayName("Record Writer")
+ .description("Specifies the record writer service to use for writing data")
+ .required(true)
+ .identifiesControllerService(RecordSetWriterFactory.class)
+ .build();
+
+ public static final PropertyDescriptor LATITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("latitude-record-path")
+ .displayName("Latitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the latitude value;"
+ + "in the decode mode, this property specifies the record path to put the latitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor LONGITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("longitude-record-path")
+ .displayName("Longitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the longitude value;"
+ + "in the decode mode, this property specifies the record path to put the longitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_FORMAT = new PropertyDescriptor.Builder()
+ .name("geohash-format")
+ .displayName("Geohash Format")
+ .description("In the encode mode, this property specifies the desired format for encoding geohash;"
+ + "in the decode mode, this property specifies the format of geohash provided")
+ .required(true)
+ .allowableValues(BASE32, BINARY_STRING, LONG_INT)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_LEVEL = new PropertyDescriptor.Builder()
+ .name("geohash-level")
+ .displayName("Geohash Level")
+ .description("The integer precision level(1-12) desired for encoding geohash")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(MODE, ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("geohash-record-path")
+ .displayName("Geohash Record Path")
+ .description("In the encode mode, this property specifies the record path to put the geohash value;"
+ + "in the decode mode, this property specifies the record path to retrieve the geohash value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Records that are successfully encoded or decoded will be routed to success")
+ .build();
+ public static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Records that cannot be encoded or decoded will be routed to failure")
+ .build();
+
+ public static final List<PropertyDescriptor> RECORD_PATH_PROPERTIES = Collections.unmodifiableList(Arrays.asList(
+ LATITUDE_RECORD_PATH,LONGITUDE_RECORD_PATH,GEOHASH_RECORD_PATH
+ ));
+
+ private List<PropertyDescriptor> descriptors;
+
+ private Set<Relationship> relationships;
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ descriptors = new ArrayList<>();
+ descriptors.add(MODE);
+ descriptors.add(RECORD_READER);
+ descriptors.add(RECORD_WRITER);
+ descriptors.add(LATITUDE_RECORD_PATH);
+ descriptors.add(LONGITUDE_RECORD_PATH);
+ descriptors.add(GEOHASH_FORMAT);
+ descriptors.add(GEOHASH_LEVEL);
+ descriptors.add(GEOHASH_RECORD_PATH);
+ descriptors = Collections.unmodifiableList(descriptors);
+
+ relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ relationships.add(REL_FAILURE);
+ relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) {
+ FlowFile input = session.get();
+ if ( input == null ) {
+ return;
+ }
+
+ final RecordReaderFactory readerFactory = context.getProperty(RECORD_READER).asControllerService(RecordReaderFactory.class);
+ final RecordSetWriterFactory writerFactory = context.getProperty(RECORD_WRITER).asControllerService(RecordSetWriterFactory.class);
+ final boolean encode = context.getProperty(MODE).getValue().equalsIgnoreCase(ENCODE_MODE);
+ final String format = context.getProperty(GEOHASH_FORMAT).getValue();
+
+ FlowFile output = session.create(input);
+ try (InputStream is = session.read(input);
+ OutputStream os = session.write(output)) {
+ RecordPathCache cache = new RecordPathCache(RECORD_PATH_PROPERTIES.size() + 1);
+ Map<PropertyDescriptor, RecordPath> paths = new HashMap<>();
+
+ for (PropertyDescriptor descriptor : RECORD_PATH_PROPERTIES) {
+ String rawRecordPath = context.getProperty(descriptor).evaluateAttributeExpressions(input).getValue();
+ RecordPath compiled = cache.getCompiled(rawRecordPath);
+ paths.put(descriptor, compiled);
+ }
+
+ RecordReader reader = readerFactory.createRecordReader(input, is, getLogger());
+ RecordSchema schema = writerFactory.getSchema(input.getAttributes(), reader.getSchema());
+ RecordSetWriter writer = writerFactory.createWriter(getLogger(), schema, os, output);
+
+ Record record;
+ Relationship targetRelationship = REL_FAILURE;
+ writer.beginRecordSet();
+
+ int recordsCount = 0;
+ boolean updated = false;
+ while ((record = reader.nextRecord()) != null) {
+ if(encode) {
+ final int level = Integer.parseInt(context.getProperty(GEOHASH_LEVEL).getValue());
Review comment:
This can be changed to using `PropertyValue::asInteger()`:
```suggestion
final int level = context.getProperty(GEOHASH_LEVEL).asInteger();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a change in pull request #5476: NIFI-9259: Adding GeohashRecord Processor
Posted by GitBox <gi...@apache.org>.
exceptionfactory commented on a change in pull request #5476:
URL: https://github.com/apache/nifi/pull/5476#discussion_r741239106
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,439 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.record.path.validation.RecordPathValidator;
+
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.WriteResult;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+
+ public enum ProcessingMode {
+ Encode, Decode
+ }
+
+ public enum GeohashFormat {
+ Base32, Binary_String, Long_Int
Review comment:
Recommend renaming as follows:
```suggestion
BASE_32, BINARY_STRING, LONG
```
The name "Long Int" is not very intuitive since `Long` and `Integer` are two different incompatible types in Java. Perhaps just naming this `LONG` or perhaps `NUMBER` might be more intuitive.
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,439 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.record.path.validation.RecordPathValidator;
+
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.WriteResult;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+
+ public enum ProcessingMode {
+ Encode, Decode
Review comment:
Recommend renaming these values to follow standard enum naming conventions:
```suggestion
ENCODE, DECODE
```
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,439 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.record.path.validation.RecordPathValidator;
+
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.WriteResult;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+
+ public enum ProcessingMode {
+ Encode, Decode
+ }
+
+ public enum GeohashFormat {
+ Base32, Binary_String, Long_Int
+ }
+
+ public enum RoutingStrategy {
+ Skip_Unenriched,
+ Split_Enriched_Unenriched,
+ Require_All_Enriched
Review comment:
Recommend renaming as follows:
```suggestion
SKIP_UNENRICHED,
SPLIT,
REQUIRE_ALL_ENRICHED
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on pull request #5476: NIFI-9259: Adding GeohashRecord Processor
Posted by GitBox <gi...@apache.org>.
exceptionfactory commented on pull request #5476:
URL: https://github.com/apache/nifi/pull/5476#issuecomment-976111152
Thanks for the updates @mikayla-yang! The latest version looks close to completion.
With the recent release of NiFi 1.15.0, can you rebase and update the version references from `1.15.0-SNAPSHOT` to `1.16.0-SNAPSHOT`? That should correct the issue with the current automated build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] markap14 commented on pull request #5476: NIFI-9259: Adding GeohashRecord Processor
Posted by GitBox <gi...@apache.org>.
markap14 commented on pull request #5476:
URL: https://github.com/apache/nifi/pull/5476#issuecomment-991331935
Thanks for all of your effort here @mikayla-yang ! All looks good to me now. And with a +1 from @exceptionfactory I've merged your changes to main!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] mikayla-yang commented on a change in pull request #5476: NIFI-9259: Adding GeohashRecord Processor
Posted by GitBox <gi...@apache.org>.
mikayla-yang commented on a change in pull request #5476:
URL: https://github.com/apache/nifi/pull/5476#discussion_r764415370
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/pom.xml
##########
@@ -0,0 +1,99 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-geohash-bundle</artifactId>
+ <version>1.16.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-geohash-processors</artifactId>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-utils</artifactId>
+ <version>1.16.0-SNAPSHOT</version>
+ </dependency>
+ <dependency>
+ <groupId>ch.hsr</groupId>
+ <artifactId>geohash</artifactId>
+ <version>1.4.0</version>
Review comment:
Hi @ChrisSamo632, thank you for pointing this out! Since this latest 1.5.0 version is not available in the [maven repository](https://mvnrepository.com/artifact/ch.hsr/geohash) yet, I've reached out to @markap14 and @YolandaMDavis for suggestions regarding this. We think it could be onerous to use 1.5.0 now outside of maven given that the functionalities we currently use are not changing between 1.4.0 and 1.5.0. We could update the version once it is available(also see this [issue](https://github.com/kungfoo/geohash-java/issues/56))
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] mikayla-yang commented on a change in pull request #5476: NIFI-9259: Adding GeohashRecord Processor
Posted by GitBox <gi...@apache.org>.
mikayla-yang commented on a change in pull request #5476:
URL: https://github.com/apache/nifi/pull/5476#discussion_r758592294
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,367 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+ public static final String ENCODE_MODE = "Encode";
+ public static final String DECODE_MODE = "Decode";
+
+ public static final String BINARY_STRING = "BinaryString";
+ public static final String BASE32 = "Base32";
+ public static final String LONG_INT = "LongInt";
+
+ public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
+ .name("mode")
+ .displayName("Mode")
+ .description("Specifies whether to encode latitude/longitude to geohash or decode geohash to latitude/longitude")
+ .required(true)
+ .allowableValues(ENCODE_MODE, DECODE_MODE)
+ .defaultValue(ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor RECORD_READER = new PropertyDescriptor.Builder()
+ .name("record-reader")
+ .displayName("Record Reader")
+ .description("Specifies the record reader service to use for reading incoming data")
+ .required(true)
+ .identifiesControllerService(RecordReaderFactory.class)
+ .build();
+
+ public static final PropertyDescriptor RECORD_WRITER = new PropertyDescriptor.Builder()
+ .name("record-writer")
+ .displayName("Record Writer")
+ .description("Specifies the record writer service to use for writing data")
+ .required(true)
+ .identifiesControllerService(RecordSetWriterFactory.class)
+ .build();
+
+ public static final PropertyDescriptor LATITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("latitude-record-path")
+ .displayName("Latitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the latitude value;"
+ + "in the decode mode, this property specifies the record path to put the latitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor LONGITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("longitude-record-path")
+ .displayName("Longitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the longitude value;"
+ + "in the decode mode, this property specifies the record path to put the longitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_FORMAT = new PropertyDescriptor.Builder()
+ .name("geohash-format")
+ .displayName("Geohash Format")
+ .description("In the encode mode, this property specifies the desired format for encoding geohash;"
+ + "in the decode mode, this property specifies the format of geohash provided")
+ .required(true)
+ .allowableValues(BASE32, BINARY_STRING, LONG_INT)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_LEVEL = new PropertyDescriptor.Builder()
+ .name("geohash-level")
+ .displayName("Geohash Level")
+ .description("The integer precision level(1-12) desired for encoding geohash")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(MODE, ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("geohash-record-path")
+ .displayName("Geohash Record Path")
+ .description("In the encode mode, this property specifies the record path to put the geohash value;"
+ + "in the decode mode, this property specifies the record path to retrieve the geohash value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Records that are successfully encoded or decoded will be routed to success")
+ .build();
+ public static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Records that cannot be encoded or decoded will be routed to failure")
+ .build();
+
+ public static final List<PropertyDescriptor> RECORD_PATH_PROPERTIES = Collections.unmodifiableList(Arrays.asList(
+ LATITUDE_RECORD_PATH,LONGITUDE_RECORD_PATH,GEOHASH_RECORD_PATH
+ ));
+
+ private List<PropertyDescriptor> descriptors;
+
+ private Set<Relationship> relationships;
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ descriptors = new ArrayList<>();
+ descriptors.add(MODE);
+ descriptors.add(RECORD_READER);
+ descriptors.add(RECORD_WRITER);
+ descriptors.add(LATITUDE_RECORD_PATH);
+ descriptors.add(LONGITUDE_RECORD_PATH);
+ descriptors.add(GEOHASH_FORMAT);
+ descriptors.add(GEOHASH_LEVEL);
+ descriptors.add(GEOHASH_RECORD_PATH);
+ descriptors = Collections.unmodifiableList(descriptors);
+
+ relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ relationships.add(REL_FAILURE);
+ relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) {
+ FlowFile input = session.get();
+ if ( input == null ) {
+ return;
+ }
+
+ final RecordReaderFactory readerFactory = context.getProperty(RECORD_READER).asControllerService(RecordReaderFactory.class);
+ final RecordSetWriterFactory writerFactory = context.getProperty(RECORD_WRITER).asControllerService(RecordSetWriterFactory.class);
+ final boolean encode = context.getProperty(MODE).getValue().equalsIgnoreCase(ENCODE_MODE);
+ final String format = context.getProperty(GEOHASH_FORMAT).getValue();
+
+ FlowFile output = session.create(input);
+ try (InputStream is = session.read(input);
+ OutputStream os = session.write(output)) {
+ RecordPathCache cache = new RecordPathCache(RECORD_PATH_PROPERTIES.size() + 1);
+ Map<PropertyDescriptor, RecordPath> paths = new HashMap<>();
+
+ for (PropertyDescriptor descriptor : RECORD_PATH_PROPERTIES) {
+ String rawRecordPath = context.getProperty(descriptor).evaluateAttributeExpressions(input).getValue();
+ RecordPath compiled = cache.getCompiled(rawRecordPath);
+ paths.put(descriptor, compiled);
+ }
+
+ RecordReader reader = readerFactory.createRecordReader(input, is, getLogger());
+ RecordSchema schema = writerFactory.getSchema(input.getAttributes(), reader.getSchema());
+ RecordSetWriter writer = writerFactory.createWriter(getLogger(), schema, os, output);
+
+ Record record;
+ Relationship targetRelationship = REL_FAILURE;
+ writer.beginRecordSet();
+
+ int recordsCount = 0;
+ boolean updated = false;
+ while ((record = reader.nextRecord()) != null) {
+ if(encode) {
+ final int level = Integer.parseInt(context.getProperty(GEOHASH_LEVEL).getValue());
+ final String rawLatitudePath = context.getProperty(LATITUDE_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath latitudePath = cache.getCompiled(rawLatitudePath);
+ final String rawLongitudePath = context.getProperty(LONGITUDE_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath longitudePath = cache.getCompiled(rawLongitudePath);
+
+ Object encodedGeohash = getEncodedGeohash(latitudePath, longitudePath, record, format, level);
+
+ updated = updateRecord(GEOHASH_RECORD_PATH, encodedGeohash, record, paths);
+ }else {
+ final String rawgeohashPath = context.getProperty(GEOHASH_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath geohashPath = cache.getCompiled(rawgeohashPath);
+
+ WGS84Point decodedPoint = getDecodedPointFromGeohash(geohashPath, record, format);
+
+ if(decodedPoint != null) {
+ updated = updateRecord(LATITUDE_RECORD_PATH, String.valueOf(decodedPoint.getLatitude()), record, paths)
+ && updateRecord(LONGITUDE_RECORD_PATH, String.valueOf(decodedPoint.getLongitude()), record, paths);
+ }
+ }
+
+ if (updated) {
Review comment:
Thank you @YolandaMDavis for your suggestion! I have posted two commits separately just to provide options. The last commit [97c5a5e](https://github.com/apache/nifi/pull/5476/commits/97c5a5e3acf20a5d8591ef6bc8aefbf4bf02d820) is aligned with your comment: I tried to make it parallel to the GeoErichIPRecord as you suggested, and the only difference is our “all or nothing” strategy will route the flowfile to the not found relationship if any of the record is not enriched. The second to last commit [c63b7e2](https://github.com/apache/nifi/pull/5476/commits/c63b7e275bb2406874003b08651724eda8f1b9f6) is aligned with the previous design, but I make sure we always route the original flowfiles to the original relationship under every routing strategy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] mikayla-yang commented on pull request #5476: NIFI-9259: Adding GeohashRecord Processor
Posted by GitBox <gi...@apache.org>.
mikayla-yang commented on pull request #5476:
URL: https://github.com/apache/nifi/pull/5476#issuecomment-957072411
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] mikayla-yang edited a comment on pull request #5476: NIFI-9259: Adding GeohashRecord Processor
Posted by GitBox <gi...@apache.org>.
mikayla-yang edited a comment on pull request #5476:
URL: https://github.com/apache/nifi/pull/5476#issuecomment-957996417
Test failure fixed and successfully built with`-Pcontrib-check`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] mikayla-yang commented on a change in pull request #5476: NIFI-9259: Adding GeohashRecord Processor
Posted by GitBox <gi...@apache.org>.
mikayla-yang commented on a change in pull request #5476:
URL: https://github.com/apache/nifi/pull/5476#discussion_r749780399
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,449 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.record.path.validation.RecordPathValidator;
+
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.WriteResult;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+
+ public enum ProcessingMode {
+ ENCODE, DECODE
+ }
+
+ public enum GeohashFormat {
+ BASE_32, BINARY_STRING, LONG
+ }
+
+ public enum RoutingStrategy {
+ SKIP_UNENRICHED,
+ SPLIT,
+ REQUIRE_ALL_ENRICHED
+ }
+
+ public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
+ .name("mode")
+ .displayName("Mode")
+ .description("Specifies whether to encode latitude/longitude to geohash or decode geohash to latitude/longitude")
+ .required(true)
+ .allowableValues(ProcessingMode.values())
+ .defaultValue(ProcessingMode.ENCODE.name())
+ .build();
+
+ public static final PropertyDescriptor ROUTING_STRATEGY = new PropertyDescriptor.Builder()
+ .name("routing-strategy")
+ .displayName("Routing Strategy")
+ .description("Specifies how to route records after encoding or decoding has been performed. "
+ + "Skip_Unenriched will route a flowfile to success if any of its records is enriched, otherwise it will be sent to failure. "
+ + "Split_Enriched_Unenriched will separate the records that have been enriched from those have not and send them to success, while unenriched records will be sent to failure; "
+ + "and the original flowfile will be sent to the original relationship. "
+ + "Require_All_Enriched will route a flowfile to success only if all of its records are enriched, otherwise it will be sent to failure")
+ .required(true)
+ .allowableValues(RoutingStrategy.values())
+ .defaultValue(RoutingStrategy.SKIP_UNENRICHED.name())
+ .build();
+
+ public static final PropertyDescriptor RECORD_READER = new PropertyDescriptor.Builder()
+ .name("record-reader")
+ .displayName("Record Reader")
+ .description("Specifies the record reader service to use for reading incoming data")
+ .required(true)
+ .identifiesControllerService(RecordReaderFactory.class)
+ .build();
+
+ public static final PropertyDescriptor RECORD_WRITER = new PropertyDescriptor.Builder()
+ .name("record-writer")
+ .displayName("Record Writer")
+ .description("Specifies the record writer service to use for writing data")
+ .required(true)
+ .identifiesControllerService(RecordSetWriterFactory.class)
+ .build();
+
+ public static final PropertyDescriptor LATITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("latitude-record-path")
+ .displayName("Latitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the latitude value; "
+ + "in the decode mode, this property specifies the record path to put the latitude value")
+ .required(true)
+ .addValidator(new RecordPathValidator())
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor LONGITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("longitude-record-path")
+ .displayName("Longitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the longitude value; "
+ + "in the decode mode, this property specifies the record path to put the longitude value")
+ .required(true)
+ .addValidator(new RecordPathValidator())
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_FORMAT = new PropertyDescriptor.Builder()
+ .name("geohash-format")
+ .displayName("Geohash Format")
+ .description("In the encode mode, this property specifies the desired format for encoding geohash; "
+ + "in the decode mode, this property specifies the format of geohash provided")
+ .required(true)
+ .allowableValues(GeohashFormat.values())
+ .defaultValue(GeohashFormat.BASE_32.name())
Review comment:
GeohashFormat dose not support expression language because it uses allowableValues(), just like the property MODE and ROUTING_STRATEGY
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] ChrisSamo632 commented on a change in pull request #5476: NIFI-9259: Adding GeohashRecord Processor
Posted by GitBox <gi...@apache.org>.
ChrisSamo632 commented on a change in pull request #5476:
URL: https://github.com/apache/nifi/pull/5476#discussion_r764721504
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/pom.xml
##########
@@ -0,0 +1,99 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-geohash-bundle</artifactId>
+ <version>1.16.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-geohash-processors</artifactId>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-utils</artifactId>
+ <version>1.16.0-SNAPSHOT</version>
+ </dependency>
+ <dependency>
+ <groupId>ch.hsr</groupId>
+ <artifactId>geohash</artifactId>
+ <version>1.4.0</version>
Review comment:
Yeah I'd not noticed that they've updated the README for the new version but not actually published the new version, ah well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] mikayla-yang commented on pull request #5476: NIFI-9259: Adding GeohashRecord Processor
Posted by GitBox <gi...@apache.org>.
mikayla-yang commented on pull request #5476:
URL: https://github.com/apache/nifi/pull/5476#issuecomment-976811220
Rebase and fixing the problem of routing unenriched empty flowfiles to success with SPLIT strategy in the upcoming commit...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] ChrisSamo632 commented on a change in pull request #5476: NIFI-9259: Adding GeohashRecord Processor
Posted by GitBox <gi...@apache.org>.
ChrisSamo632 commented on a change in pull request #5476:
URL: https://github.com/apache/nifi/pull/5476#discussion_r762574445
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/pom.xml
##########
@@ -0,0 +1,99 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-geohash-bundle</artifactId>
+ <version>1.16.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-geohash-processors</artifactId>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-utils</artifactId>
+ <version>1.16.0-SNAPSHOT</version>
+ </dependency>
+ <dependency>
+ <groupId>ch.hsr</groupId>
+ <artifactId>geohash</artifactId>
+ <version>1.4.0</version>
Review comment:
Version 1.5.0 now available, should we use that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] mikayla-yang commented on a change in pull request #5476: NIFI-9259: Adding GeohashRecord Processor
Posted by GitBox <gi...@apache.org>.
mikayla-yang commented on a change in pull request #5476:
URL: https://github.com/apache/nifi/pull/5476#discussion_r764415370
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/pom.xml
##########
@@ -0,0 +1,99 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-geohash-bundle</artifactId>
+ <version>1.16.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-geohash-processors</artifactId>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-utils</artifactId>
+ <version>1.16.0-SNAPSHOT</version>
+ </dependency>
+ <dependency>
+ <groupId>ch.hsr</groupId>
+ <artifactId>geohash</artifactId>
+ <version>1.4.0</version>
Review comment:
Hi @ChrisSamo632, thank you for pointing this out! Since this latest 1.5.0 version is not available in the [maven repository](https://mvnrepository.com/artifact/ch.hsr/geohash) yet, I've reached out to @markap14 and @YolandaMDavis for suggestions regarding this. We think it could be onerous to use 1.5.0 now outside of maven given that the functionalities we currently use are not changing between 1.4.0 and 1.5.0. We could update the version once it is available.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] markap14 merged pull request #5476: NIFI-9259: Adding GeohashRecord Processor
Posted by GitBox <gi...@apache.org>.
markap14 merged pull request #5476:
URL: https://github.com/apache/nifi/pull/5476
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] mikayla-yang commented on a change in pull request #5476: NIFI-9259: Adding GeohashRecord Processor
Posted by GitBox <gi...@apache.org>.
mikayla-yang commented on a change in pull request #5476:
URL: https://github.com/apache/nifi/pull/5476#discussion_r740697078
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,367 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+ public static final String ENCODE_MODE = "Encode";
+ public static final String DECODE_MODE = "Decode";
+
+ public static final String BINARY_STRING = "BinaryString";
+ public static final String BASE32 = "Base32";
+ public static final String LONG_INT = "LongInt";
+
+ public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
+ .name("mode")
+ .displayName("Mode")
+ .description("Specifies whether to encode latitude/longitude to geohash or decode geohash to latitude/longitude")
+ .required(true)
+ .allowableValues(ENCODE_MODE, DECODE_MODE)
+ .defaultValue(ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor RECORD_READER = new PropertyDescriptor.Builder()
+ .name("record-reader")
+ .displayName("Record Reader")
+ .description("Specifies the record reader service to use for reading incoming data")
+ .required(true)
+ .identifiesControllerService(RecordReaderFactory.class)
+ .build();
+
+ public static final PropertyDescriptor RECORD_WRITER = new PropertyDescriptor.Builder()
+ .name("record-writer")
+ .displayName("Record Writer")
+ .description("Specifies the record writer service to use for writing data")
+ .required(true)
+ .identifiesControllerService(RecordSetWriterFactory.class)
+ .build();
+
+ public static final PropertyDescriptor LATITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("latitude-record-path")
+ .displayName("Latitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the latitude value;"
+ + "in the decode mode, this property specifies the record path to put the latitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor LONGITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("longitude-record-path")
+ .displayName("Longitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the longitude value;"
+ + "in the decode mode, this property specifies the record path to put the longitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_FORMAT = new PropertyDescriptor.Builder()
+ .name("geohash-format")
+ .displayName("Geohash Format")
+ .description("In the encode mode, this property specifies the desired format for encoding geohash;"
+ + "in the decode mode, this property specifies the format of geohash provided")
+ .required(true)
+ .allowableValues(BASE32, BINARY_STRING, LONG_INT)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_LEVEL = new PropertyDescriptor.Builder()
+ .name("geohash-level")
+ .displayName("Geohash Level")
+ .description("The integer precision level(1-12) desired for encoding geohash")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(MODE, ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("geohash-record-path")
+ .displayName("Geohash Record Path")
+ .description("In the encode mode, this property specifies the record path to put the geohash value;"
+ + "in the decode mode, this property specifies the record path to retrieve the geohash value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Records that are successfully encoded or decoded will be routed to success")
+ .build();
+ public static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Records that cannot be encoded or decoded will be routed to failure")
+ .build();
+
+ public static final List<PropertyDescriptor> RECORD_PATH_PROPERTIES = Collections.unmodifiableList(Arrays.asList(
+ LATITUDE_RECORD_PATH,LONGITUDE_RECORD_PATH,GEOHASH_RECORD_PATH
+ ));
+
+ private List<PropertyDescriptor> descriptors;
+
+ private Set<Relationship> relationships;
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ descriptors = new ArrayList<>();
+ descriptors.add(MODE);
+ descriptors.add(RECORD_READER);
+ descriptors.add(RECORD_WRITER);
+ descriptors.add(LATITUDE_RECORD_PATH);
+ descriptors.add(LONGITUDE_RECORD_PATH);
+ descriptors.add(GEOHASH_FORMAT);
+ descriptors.add(GEOHASH_LEVEL);
+ descriptors.add(GEOHASH_RECORD_PATH);
+ descriptors = Collections.unmodifiableList(descriptors);
+
+ relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ relationships.add(REL_FAILURE);
+ relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) {
+ FlowFile input = session.get();
+ if ( input == null ) {
+ return;
+ }
+
+ final RecordReaderFactory readerFactory = context.getProperty(RECORD_READER).asControllerService(RecordReaderFactory.class);
+ final RecordSetWriterFactory writerFactory = context.getProperty(RECORD_WRITER).asControllerService(RecordSetWriterFactory.class);
+ final boolean encode = context.getProperty(MODE).getValue().equalsIgnoreCase(ENCODE_MODE);
+ final String format = context.getProperty(GEOHASH_FORMAT).getValue();
+
+ FlowFile output = session.create(input);
+ try (InputStream is = session.read(input);
+ OutputStream os = session.write(output)) {
+ RecordPathCache cache = new RecordPathCache(RECORD_PATH_PROPERTIES.size() + 1);
+ Map<PropertyDescriptor, RecordPath> paths = new HashMap<>();
+
+ for (PropertyDescriptor descriptor : RECORD_PATH_PROPERTIES) {
+ String rawRecordPath = context.getProperty(descriptor).evaluateAttributeExpressions(input).getValue();
+ RecordPath compiled = cache.getCompiled(rawRecordPath);
+ paths.put(descriptor, compiled);
+ }
+
+ RecordReader reader = readerFactory.createRecordReader(input, is, getLogger());
+ RecordSchema schema = writerFactory.getSchema(input.getAttributes(), reader.getSchema());
+ RecordSetWriter writer = writerFactory.createWriter(getLogger(), schema, os, output);
+
+ Record record;
+ Relationship targetRelationship = REL_FAILURE;
+ writer.beginRecordSet();
+
+ int recordsCount = 0;
+ boolean updated = false;
+ while ((record = reader.nextRecord()) != null) {
+ if(encode) {
+ final int level = Integer.parseInt(context.getProperty(GEOHASH_LEVEL).getValue());
+ final String rawLatitudePath = context.getProperty(LATITUDE_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath latitudePath = cache.getCompiled(rawLatitudePath);
+ final String rawLongitudePath = context.getProperty(LONGITUDE_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath longitudePath = cache.getCompiled(rawLongitudePath);
+
+ Object encodedGeohash = getEncodedGeohash(latitudePath, longitudePath, record, format, level);
+
+ updated = updateRecord(GEOHASH_RECORD_PATH, encodedGeohash, record, paths);
+ }else {
+ final String rawgeohashPath = context.getProperty(GEOHASH_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath geohashPath = cache.getCompiled(rawgeohashPath);
+
+ WGS84Point decodedPoint = getDecodedPointFromGeohash(geohashPath, record, format);
+
+ if(decodedPoint != null) {
+ updated = updateRecord(LATITUDE_RECORD_PATH, String.valueOf(decodedPoint.getLatitude()), record, paths)
+ && updateRecord(LONGITUDE_RECORD_PATH, String.valueOf(decodedPoint.getLongitude()), record, paths);
+ }
+ }
+
+ if (updated) {
Review comment:
I have added a ROUTING_STRATEGY property so that users can specify how to route the flowfile, which I believe covers all the user cases you mentioned:
- _Skip_Unenriched_: "Some of the Records have the latitude/longitude elements and some don't. If the elements are there, enrich them, otherwise skip them."
- _Require_All_Enriched_: “I expect all records to be enriched. Otherwise, something is wrong and I want the whole FlowFile to be failed."
- _Split_Enriched_Unenriched_: split the records that are enriched from those are not. The flowfile including successfully enriched records will be sent to ‘success’, the flowfile including records that are unable to be enriched will be sent to ‘failure’, and the original flowfile will be sent to ‘original’.
Just one quick note on the original code here, the destination relationship for all records will be 'success' as long as one record has been updated, which is identical to the current Skip_Unenriched strategy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] mikayla-yang commented on a change in pull request #5476: NIFI-9259: Adding GeohashRecord Processor
Posted by GitBox <gi...@apache.org>.
mikayla-yang commented on a change in pull request #5476:
URL: https://github.com/apache/nifi/pull/5476#discussion_r740697576
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,367 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+ public static final String ENCODE_MODE = "Encode";
+ public static final String DECODE_MODE = "Decode";
+
+ public static final String BINARY_STRING = "BinaryString";
+ public static final String BASE32 = "Base32";
+ public static final String LONG_INT = "LongInt";
+
+ public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
+ .name("mode")
+ .displayName("Mode")
+ .description("Specifies whether to encode latitude/longitude to geohash or decode geohash to latitude/longitude")
+ .required(true)
+ .allowableValues(ENCODE_MODE, DECODE_MODE)
+ .defaultValue(ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor RECORD_READER = new PropertyDescriptor.Builder()
+ .name("record-reader")
+ .displayName("Record Reader")
+ .description("Specifies the record reader service to use for reading incoming data")
+ .required(true)
+ .identifiesControllerService(RecordReaderFactory.class)
+ .build();
+
+ public static final PropertyDescriptor RECORD_WRITER = new PropertyDescriptor.Builder()
+ .name("record-writer")
+ .displayName("Record Writer")
+ .description("Specifies the record writer service to use for writing data")
+ .required(true)
+ .identifiesControllerService(RecordSetWriterFactory.class)
+ .build();
+
+ public static final PropertyDescriptor LATITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("latitude-record-path")
+ .displayName("Latitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the latitude value;"
+ + "in the decode mode, this property specifies the record path to put the latitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor LONGITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("longitude-record-path")
+ .displayName("Longitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the longitude value;"
+ + "in the decode mode, this property specifies the record path to put the longitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_FORMAT = new PropertyDescriptor.Builder()
+ .name("geohash-format")
+ .displayName("Geohash Format")
+ .description("In the encode mode, this property specifies the desired format for encoding geohash;"
+ + "in the decode mode, this property specifies the format of geohash provided")
+ .required(true)
+ .allowableValues(BASE32, BINARY_STRING, LONG_INT)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_LEVEL = new PropertyDescriptor.Builder()
+ .name("geohash-level")
+ .displayName("Geohash Level")
+ .description("The integer precision level(1-12) desired for encoding geohash")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(MODE, ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("geohash-record-path")
+ .displayName("Geohash Record Path")
+ .description("In the encode mode, this property specifies the record path to put the geohash value;"
+ + "in the decode mode, this property specifies the record path to retrieve the geohash value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Records that are successfully encoded or decoded will be routed to success")
+ .build();
+ public static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Records that cannot be encoded or decoded will be routed to failure")
+ .build();
+
+ public static final List<PropertyDescriptor> RECORD_PATH_PROPERTIES = Collections.unmodifiableList(Arrays.asList(
+ LATITUDE_RECORD_PATH,LONGITUDE_RECORD_PATH,GEOHASH_RECORD_PATH
+ ));
+
+ private List<PropertyDescriptor> descriptors;
+
+ private Set<Relationship> relationships;
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ descriptors = new ArrayList<>();
+ descriptors.add(MODE);
+ descriptors.add(RECORD_READER);
+ descriptors.add(RECORD_WRITER);
+ descriptors.add(LATITUDE_RECORD_PATH);
+ descriptors.add(LONGITUDE_RECORD_PATH);
+ descriptors.add(GEOHASH_FORMAT);
+ descriptors.add(GEOHASH_LEVEL);
+ descriptors.add(GEOHASH_RECORD_PATH);
+ descriptors = Collections.unmodifiableList(descriptors);
+
+ relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ relationships.add(REL_FAILURE);
+ relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) {
+ FlowFile input = session.get();
+ if ( input == null ) {
+ return;
+ }
+
+ final RecordReaderFactory readerFactory = context.getProperty(RECORD_READER).asControllerService(RecordReaderFactory.class);
+ final RecordSetWriterFactory writerFactory = context.getProperty(RECORD_WRITER).asControllerService(RecordSetWriterFactory.class);
+ final boolean encode = context.getProperty(MODE).getValue().equalsIgnoreCase(ENCODE_MODE);
+ final String format = context.getProperty(GEOHASH_FORMAT).getValue();
+
+ FlowFile output = session.create(input);
Review comment:
Now a ROUTING_STRATEGY property is added, and it includes three strategy options for users to specify. We need to process the InputStream and OuputStream separately when using the _Split_Enriched_Unenriched_ strategy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] mikayla-yang commented on a change in pull request #5476: NIFI-9259: Adding GeohashRecord Processor
Posted by GitBox <gi...@apache.org>.
mikayla-yang commented on a change in pull request #5476:
URL: https://github.com/apache/nifi/pull/5476#discussion_r741257919
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,439 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.record.path.validation.RecordPathValidator;
+
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.WriteResult;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+
+ public enum ProcessingMode {
+ Encode, Decode
+ }
+
+ public enum GeohashFormat {
+ Base32, Binary_String, Long_Int
Review comment:
I'll change it to LONG here, because the integer version of Geohash should be a 64-bit integer. So I think LONG makes more sense compared to NUMBER.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] mikayla-yang commented on a change in pull request #5476: NIFI-9259: Adding GeohashRecord Processor
Posted by GitBox <gi...@apache.org>.
mikayla-yang commented on a change in pull request #5476:
URL: https://github.com/apache/nifi/pull/5476#discussion_r740696092
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,367 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+ public static final String ENCODE_MODE = "Encode";
+ public static final String DECODE_MODE = "Decode";
+
+ public static final String BINARY_STRING = "BinaryString";
+ public static final String BASE32 = "Base32";
+ public static final String LONG_INT = "LongInt";
+
+ public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
+ .name("mode")
+ .displayName("Mode")
+ .description("Specifies whether to encode latitude/longitude to geohash or decode geohash to latitude/longitude")
+ .required(true)
+ .allowableValues(ENCODE_MODE, DECODE_MODE)
+ .defaultValue(ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor RECORD_READER = new PropertyDescriptor.Builder()
+ .name("record-reader")
+ .displayName("Record Reader")
+ .description("Specifies the record reader service to use for reading incoming data")
+ .required(true)
+ .identifiesControllerService(RecordReaderFactory.class)
+ .build();
+
+ public static final PropertyDescriptor RECORD_WRITER = new PropertyDescriptor.Builder()
+ .name("record-writer")
+ .displayName("Record Writer")
+ .description("Specifies the record writer service to use for writing data")
+ .required(true)
+ .identifiesControllerService(RecordSetWriterFactory.class)
+ .build();
+
+ public static final PropertyDescriptor LATITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("latitude-record-path")
+ .displayName("Latitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the latitude value;"
+ + "in the decode mode, this property specifies the record path to put the latitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor LONGITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("longitude-record-path")
+ .displayName("Longitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the longitude value;"
+ + "in the decode mode, this property specifies the record path to put the longitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_FORMAT = new PropertyDescriptor.Builder()
+ .name("geohash-format")
+ .displayName("Geohash Format")
+ .description("In the encode mode, this property specifies the desired format for encoding geohash;"
+ + "in the decode mode, this property specifies the format of geohash provided")
+ .required(true)
+ .allowableValues(BASE32, BINARY_STRING, LONG_INT)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_LEVEL = new PropertyDescriptor.Builder()
+ .name("geohash-level")
+ .displayName("Geohash Level")
+ .description("The integer precision level(1-12) desired for encoding geohash")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(MODE, ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("geohash-record-path")
+ .displayName("Geohash Record Path")
+ .description("In the encode mode, this property specifies the record path to put the geohash value;"
+ + "in the decode mode, this property specifies the record path to retrieve the geohash value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Records that are successfully encoded or decoded will be routed to success")
+ .build();
+ public static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Records that cannot be encoded or decoded will be routed to failure")
+ .build();
+
+ public static final List<PropertyDescriptor> RECORD_PATH_PROPERTIES = Collections.unmodifiableList(Arrays.asList(
+ LATITUDE_RECORD_PATH,LONGITUDE_RECORD_PATH,GEOHASH_RECORD_PATH
+ ));
+
+ private List<PropertyDescriptor> descriptors;
+
+ private Set<Relationship> relationships;
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ descriptors = new ArrayList<>();
+ descriptors.add(MODE);
+ descriptors.add(RECORD_READER);
+ descriptors.add(RECORD_WRITER);
+ descriptors.add(LATITUDE_RECORD_PATH);
+ descriptors.add(LONGITUDE_RECORD_PATH);
+ descriptors.add(GEOHASH_FORMAT);
+ descriptors.add(GEOHASH_LEVEL);
+ descriptors.add(GEOHASH_RECORD_PATH);
+ descriptors = Collections.unmodifiableList(descriptors);
+
+ relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ relationships.add(REL_FAILURE);
+ relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) {
+ FlowFile input = session.get();
+ if ( input == null ) {
+ return;
+ }
+
+ final RecordReaderFactory readerFactory = context.getProperty(RECORD_READER).asControllerService(RecordReaderFactory.class);
+ final RecordSetWriterFactory writerFactory = context.getProperty(RECORD_WRITER).asControllerService(RecordSetWriterFactory.class);
+ final boolean encode = context.getProperty(MODE).getValue().equalsIgnoreCase(ENCODE_MODE);
+ final String format = context.getProperty(GEOHASH_FORMAT).getValue();
+
+ FlowFile output = session.create(input);
+ try (InputStream is = session.read(input);
+ OutputStream os = session.write(output)) {
+ RecordPathCache cache = new RecordPathCache(RECORD_PATH_PROPERTIES.size() + 1);
+ Map<PropertyDescriptor, RecordPath> paths = new HashMap<>();
+
+ for (PropertyDescriptor descriptor : RECORD_PATH_PROPERTIES) {
+ String rawRecordPath = context.getProperty(descriptor).evaluateAttributeExpressions(input).getValue();
+ RecordPath compiled = cache.getCompiled(rawRecordPath);
+ paths.put(descriptor, compiled);
+ }
+
+ RecordReader reader = readerFactory.createRecordReader(input, is, getLogger());
+ RecordSchema schema = writerFactory.getSchema(input.getAttributes(), reader.getSchema());
+ RecordSetWriter writer = writerFactory.createWriter(getLogger(), schema, os, output);
Review comment:
All the stream, reader and writer now are declared in the try-with-resources statement for automatically closing them.
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,367 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+ public static final String ENCODE_MODE = "Encode";
+ public static final String DECODE_MODE = "Decode";
+
+ public static final String BINARY_STRING = "BinaryString";
+ public static final String BASE32 = "Base32";
+ public static final String LONG_INT = "LongInt";
+
+ public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
+ .name("mode")
+ .displayName("Mode")
+ .description("Specifies whether to encode latitude/longitude to geohash or decode geohash to latitude/longitude")
+ .required(true)
+ .allowableValues(ENCODE_MODE, DECODE_MODE)
+ .defaultValue(ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor RECORD_READER = new PropertyDescriptor.Builder()
+ .name("record-reader")
+ .displayName("Record Reader")
+ .description("Specifies the record reader service to use for reading incoming data")
+ .required(true)
+ .identifiesControllerService(RecordReaderFactory.class)
+ .build();
+
+ public static final PropertyDescriptor RECORD_WRITER = new PropertyDescriptor.Builder()
+ .name("record-writer")
+ .displayName("Record Writer")
+ .description("Specifies the record writer service to use for writing data")
+ .required(true)
+ .identifiesControllerService(RecordSetWriterFactory.class)
+ .build();
+
+ public static final PropertyDescriptor LATITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("latitude-record-path")
+ .displayName("Latitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the latitude value;"
+ + "in the decode mode, this property specifies the record path to put the latitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor LONGITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("longitude-record-path")
+ .displayName("Longitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the longitude value;"
+ + "in the decode mode, this property specifies the record path to put the longitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_FORMAT = new PropertyDescriptor.Builder()
+ .name("geohash-format")
+ .displayName("Geohash Format")
+ .description("In the encode mode, this property specifies the desired format for encoding geohash;"
+ + "in the decode mode, this property specifies the format of geohash provided")
+ .required(true)
+ .allowableValues(BASE32, BINARY_STRING, LONG_INT)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_LEVEL = new PropertyDescriptor.Builder()
+ .name("geohash-level")
+ .displayName("Geohash Level")
+ .description("The integer precision level(1-12) desired for encoding geohash")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(MODE, ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("geohash-record-path")
+ .displayName("Geohash Record Path")
+ .description("In the encode mode, this property specifies the record path to put the geohash value;"
+ + "in the decode mode, this property specifies the record path to retrieve the geohash value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Records that are successfully encoded or decoded will be routed to success")
+ .build();
+ public static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Records that cannot be encoded or decoded will be routed to failure")
+ .build();
+
+ public static final List<PropertyDescriptor> RECORD_PATH_PROPERTIES = Collections.unmodifiableList(Arrays.asList(
+ LATITUDE_RECORD_PATH,LONGITUDE_RECORD_PATH,GEOHASH_RECORD_PATH
+ ));
+
+ private List<PropertyDescriptor> descriptors;
+
+ private Set<Relationship> relationships;
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ descriptors = new ArrayList<>();
+ descriptors.add(MODE);
+ descriptors.add(RECORD_READER);
+ descriptors.add(RECORD_WRITER);
+ descriptors.add(LATITUDE_RECORD_PATH);
+ descriptors.add(LONGITUDE_RECORD_PATH);
+ descriptors.add(GEOHASH_FORMAT);
+ descriptors.add(GEOHASH_LEVEL);
+ descriptors.add(GEOHASH_RECORD_PATH);
+ descriptors = Collections.unmodifiableList(descriptors);
+
+ relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ relationships.add(REL_FAILURE);
+ relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) {
+ FlowFile input = session.get();
+ if ( input == null ) {
+ return;
+ }
+
+ final RecordReaderFactory readerFactory = context.getProperty(RECORD_READER).asControllerService(RecordReaderFactory.class);
+ final RecordSetWriterFactory writerFactory = context.getProperty(RECORD_WRITER).asControllerService(RecordSetWriterFactory.class);
+ final boolean encode = context.getProperty(MODE).getValue().equalsIgnoreCase(ENCODE_MODE);
+ final String format = context.getProperty(GEOHASH_FORMAT).getValue();
+
+ FlowFile output = session.create(input);
+ try (InputStream is = session.read(input);
+ OutputStream os = session.write(output)) {
+ RecordPathCache cache = new RecordPathCache(RECORD_PATH_PROPERTIES.size() + 1);
+ Map<PropertyDescriptor, RecordPath> paths = new HashMap<>();
+
+ for (PropertyDescriptor descriptor : RECORD_PATH_PROPERTIES) {
+ String rawRecordPath = context.getProperty(descriptor).evaluateAttributeExpressions(input).getValue();
+ RecordPath compiled = cache.getCompiled(rawRecordPath);
+ paths.put(descriptor, compiled);
+ }
+
+ RecordReader reader = readerFactory.createRecordReader(input, is, getLogger());
+ RecordSchema schema = writerFactory.getSchema(input.getAttributes(), reader.getSchema());
+ RecordSetWriter writer = writerFactory.createWriter(getLogger(), schema, os, output);
+
+ Record record;
+ Relationship targetRelationship = REL_FAILURE;
+ writer.beginRecordSet();
+
+ int recordsCount = 0;
+ boolean updated = false;
+ while ((record = reader.nextRecord()) != null) {
+ if(encode) {
+ final int level = Integer.parseInt(context.getProperty(GEOHASH_LEVEL).getValue());
+ final String rawLatitudePath = context.getProperty(LATITUDE_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath latitudePath = cache.getCompiled(rawLatitudePath);
+ final String rawLongitudePath = context.getProperty(LONGITUDE_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath longitudePath = cache.getCompiled(rawLongitudePath);
+
+ Object encodedGeohash = getEncodedGeohash(latitudePath, longitudePath, record, format, level);
+
+ updated = updateRecord(GEOHASH_RECORD_PATH, encodedGeohash, record, paths);
+ }else {
+ final String rawgeohashPath = context.getProperty(GEOHASH_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath geohashPath = cache.getCompiled(rawgeohashPath);
+
+ WGS84Point decodedPoint = getDecodedPointFromGeohash(geohashPath, record, format);
+
+ if(decodedPoint != null) {
+ updated = updateRecord(LATITUDE_RECORD_PATH, String.valueOf(decodedPoint.getLatitude()), record, paths)
+ && updateRecord(LONGITUDE_RECORD_PATH, String.valueOf(decodedPoint.getLongitude()), record, paths);
+ }
+ }
+
+ if (updated) {
Review comment:
I have added a ROUTING_STRATEGY property so that users can specify how to route the flowfile, which I believe covers all the user cases you mentioned:
- _Skip_Unenriched_: "Some of the Records have the latitude/longitude elements and some don't. If the elements are there, enrich them, otherwise skip them."
- _Require_All_Enriched_: “I expect all records to be enriched. Otherwise, something is wrong and I want the whole FlowFile to be failed."
- _Split_Enriched_Unenriched_: split the records that are enriched from those are not. The flowfile including successfully enriched records will be sent to ‘success’, the flowfile including records that are unable to be enriched will be sent to ‘failure’, and the original flowfile will be sent to ‘original’.
Just one quick note on the original code here, the destination relationship for all records will be 'success' as long as one record has been updated, which is identical to the current Skip_Unenriched strategy.
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,367 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+ public static final String ENCODE_MODE = "Encode";
+ public static final String DECODE_MODE = "Decode";
+
+ public static final String BINARY_STRING = "BinaryString";
+ public static final String BASE32 = "Base32";
+ public static final String LONG_INT = "LongInt";
+
+ public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
+ .name("mode")
+ .displayName("Mode")
+ .description("Specifies whether to encode latitude/longitude to geohash or decode geohash to latitude/longitude")
+ .required(true)
+ .allowableValues(ENCODE_MODE, DECODE_MODE)
+ .defaultValue(ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor RECORD_READER = new PropertyDescriptor.Builder()
+ .name("record-reader")
+ .displayName("Record Reader")
+ .description("Specifies the record reader service to use for reading incoming data")
+ .required(true)
+ .identifiesControllerService(RecordReaderFactory.class)
+ .build();
+
+ public static final PropertyDescriptor RECORD_WRITER = new PropertyDescriptor.Builder()
+ .name("record-writer")
+ .displayName("Record Writer")
+ .description("Specifies the record writer service to use for writing data")
+ .required(true)
+ .identifiesControllerService(RecordSetWriterFactory.class)
+ .build();
+
+ public static final PropertyDescriptor LATITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("latitude-record-path")
+ .displayName("Latitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the latitude value;"
+ + "in the decode mode, this property specifies the record path to put the latitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor LONGITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("longitude-record-path")
+ .displayName("Longitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the longitude value;"
+ + "in the decode mode, this property specifies the record path to put the longitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_FORMAT = new PropertyDescriptor.Builder()
+ .name("geohash-format")
+ .displayName("Geohash Format")
+ .description("In the encode mode, this property specifies the desired format for encoding geohash;"
+ + "in the decode mode, this property specifies the format of geohash provided")
+ .required(true)
+ .allowableValues(BASE32, BINARY_STRING, LONG_INT)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_LEVEL = new PropertyDescriptor.Builder()
+ .name("geohash-level")
+ .displayName("Geohash Level")
+ .description("The integer precision level(1-12) desired for encoding geohash")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(MODE, ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("geohash-record-path")
+ .displayName("Geohash Record Path")
+ .description("In the encode mode, this property specifies the record path to put the geohash value;"
+ + "in the decode mode, this property specifies the record path to retrieve the geohash value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Records that are successfully encoded or decoded will be routed to success")
+ .build();
+ public static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Records that cannot be encoded or decoded will be routed to failure")
+ .build();
+
+ public static final List<PropertyDescriptor> RECORD_PATH_PROPERTIES = Collections.unmodifiableList(Arrays.asList(
+ LATITUDE_RECORD_PATH,LONGITUDE_RECORD_PATH,GEOHASH_RECORD_PATH
+ ));
+
+ private List<PropertyDescriptor> descriptors;
+
+ private Set<Relationship> relationships;
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ descriptors = new ArrayList<>();
+ descriptors.add(MODE);
+ descriptors.add(RECORD_READER);
+ descriptors.add(RECORD_WRITER);
+ descriptors.add(LATITUDE_RECORD_PATH);
+ descriptors.add(LONGITUDE_RECORD_PATH);
+ descriptors.add(GEOHASH_FORMAT);
+ descriptors.add(GEOHASH_LEVEL);
+ descriptors.add(GEOHASH_RECORD_PATH);
+ descriptors = Collections.unmodifiableList(descriptors);
+
+ relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ relationships.add(REL_FAILURE);
+ relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) {
+ FlowFile input = session.get();
+ if ( input == null ) {
+ return;
+ }
+
+ final RecordReaderFactory readerFactory = context.getProperty(RECORD_READER).asControllerService(RecordReaderFactory.class);
+ final RecordSetWriterFactory writerFactory = context.getProperty(RECORD_WRITER).asControllerService(RecordSetWriterFactory.class);
+ final boolean encode = context.getProperty(MODE).getValue().equalsIgnoreCase(ENCODE_MODE);
+ final String format = context.getProperty(GEOHASH_FORMAT).getValue();
+
+ FlowFile output = session.create(input);
Review comment:
Now a ROUTING_STRATEGY property is added, and it includes three strategy options for users to specify. We need to process the InputStream and OuputStream separately when using the _Split_Enriched_Unenriched_ strategy.
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,439 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.record.path.validation.RecordPathValidator;
+
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.WriteResult;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+
+ public enum ProcessingMode {
+ Encode, Decode
+ }
+
+ public enum GeohashFormat {
+ Base32, Binary_String, Long_Int
Review comment:
I'll change it to LONG here, because the integer version of Geohash should be a 64-bit integer. So I think LONG makes more sense compared to NUMBER.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a change in pull request #5476: NIFI-9259: Adding GeohashRecord Processor
Posted by GitBox <gi...@apache.org>.
exceptionfactory commented on a change in pull request #5476:
URL: https://github.com/apache/nifi/pull/5476#discussion_r744967480
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/pom.xml
##########
@@ -0,0 +1,98 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-geohash-bundle</artifactId>
+ <version>1.15.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-geohash-processors</artifactId>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-utils</artifactId>
+ <version>1.15.0-SNAPSHOT</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-mock</artifactId>
+ <version>1.15.0-SNAPSHOT</version>
+ <scope>test</scope>
+ </dependency>
Review comment:
For clearer organization, it would be helpful to move all of the `test` scope dependencies after the standard compile dependencies.
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/test/java/org/apache/nifi/processors/geohash/GeohashRecordTest.java
##########
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.avro.Schema;
+import org.apache.commons.io.IOUtils;
+import org.apache.nifi.avro.AvroTypeUtil;
+import org.apache.nifi.controller.ControllerService;
+import org.apache.nifi.json.JsonRecordSetWriter;
+import org.apache.nifi.json.JsonTreeReader;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.schema.access.SchemaAccessUtils;
+import org.apache.nifi.serialization.record.MockSchemaRegistry;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.codehaus.jackson.map.ObjectMapper;
+
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+
+import java.io.InputStream;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+public class GeohashRecordTest {
+
+ private final TestRunner runner = TestRunners.newTestRunner(GeohashRecord.class);
Review comment:
To avoid unexpected behavior, recommend removing the `final` keyword and moving the `newTestRunner()` assignment into the `setUp()` method.
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,449 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.record.path.validation.RecordPathValidator;
+
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.WriteResult;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+
+ public enum ProcessingMode {
+ ENCODE, DECODE
+ }
+
+ public enum GeohashFormat {
+ BASE_32, BINARY_STRING, LONG
+ }
+
+ public enum RoutingStrategy {
+ SKIP_UNENRICHED,
+ SPLIT,
+ REQUIRE_ALL_ENRICHED
+ }
+
+ public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
+ .name("mode")
+ .displayName("Mode")
+ .description("Specifies whether to encode latitude/longitude to geohash or decode geohash to latitude/longitude")
+ .required(true)
+ .allowableValues(ProcessingMode.values())
+ .defaultValue(ProcessingMode.ENCODE.name())
+ .build();
+
+ public static final PropertyDescriptor ROUTING_STRATEGY = new PropertyDescriptor.Builder()
+ .name("routing-strategy")
+ .displayName("Routing Strategy")
+ .description("Specifies how to route records after encoding or decoding has been performed. "
+ + "Skip_Unenriched will route a flowfile to success if any of its records is enriched, otherwise it will be sent to failure. "
+ + "Split_Enriched_Unenriched will separate the records that have been enriched from those have not and send them to success, while unenriched records will be sent to failure; "
+ + "and the original flowfile will be sent to the original relationship. "
+ + "Require_All_Enriched will route a flowfile to success only if all of its records are enriched, otherwise it will be sent to failure")
+ .required(true)
+ .allowableValues(RoutingStrategy.values())
+ .defaultValue(RoutingStrategy.SKIP_UNENRICHED.name())
+ .build();
+
+ public static final PropertyDescriptor RECORD_READER = new PropertyDescriptor.Builder()
+ .name("record-reader")
+ .displayName("Record Reader")
+ .description("Specifies the record reader service to use for reading incoming data")
+ .required(true)
+ .identifiesControllerService(RecordReaderFactory.class)
+ .build();
+
+ public static final PropertyDescriptor RECORD_WRITER = new PropertyDescriptor.Builder()
+ .name("record-writer")
+ .displayName("Record Writer")
+ .description("Specifies the record writer service to use for writing data")
+ .required(true)
+ .identifiesControllerService(RecordSetWriterFactory.class)
+ .build();
+
+ public static final PropertyDescriptor LATITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("latitude-record-path")
+ .displayName("Latitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the latitude value; "
+ + "in the decode mode, this property specifies the record path to put the latitude value")
+ .required(true)
+ .addValidator(new RecordPathValidator())
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor LONGITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("longitude-record-path")
+ .displayName("Longitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the longitude value; "
+ + "in the decode mode, this property specifies the record path to put the longitude value")
+ .required(true)
+ .addValidator(new RecordPathValidator())
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_FORMAT = new PropertyDescriptor.Builder()
+ .name("geohash-format")
+ .displayName("Geohash Format")
+ .description("In the encode mode, this property specifies the desired format for encoding geohash; "
+ + "in the decode mode, this property specifies the format of geohash provided")
+ .required(true)
+ .allowableValues(GeohashFormat.values())
+ .defaultValue(GeohashFormat.BASE_32.name())
Review comment:
Is there a reason why this property does not support expression language? It looks like most of the other properties support FlowFile attributes with EL.
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,449 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.record.path.validation.RecordPathValidator;
+
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.WriteResult;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+
+ public enum ProcessingMode {
+ ENCODE, DECODE
+ }
+
+ public enum GeohashFormat {
+ BASE_32, BINARY_STRING, LONG
+ }
+
+ public enum RoutingStrategy {
+ SKIP_UNENRICHED,
+ SPLIT,
+ REQUIRE_ALL_ENRICHED
+ }
+
+ public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
+ .name("mode")
+ .displayName("Mode")
+ .description("Specifies whether to encode latitude/longitude to geohash or decode geohash to latitude/longitude")
+ .required(true)
+ .allowableValues(ProcessingMode.values())
+ .defaultValue(ProcessingMode.ENCODE.name())
+ .build();
+
+ public static final PropertyDescriptor ROUTING_STRATEGY = new PropertyDescriptor.Builder()
+ .name("routing-strategy")
+ .displayName("Routing Strategy")
+ .description("Specifies how to route records after encoding or decoding has been performed. "
+ + "Skip_Unenriched will route a flowfile to success if any of its records is enriched, otherwise it will be sent to failure. "
+ + "Split_Enriched_Unenriched will separate the records that have been enriched from those have not and send them to success, while unenriched records will be sent to failure; "
+ + "and the original flowfile will be sent to the original relationship. "
+ + "Require_All_Enriched will route a flowfile to success only if all of its records are enriched, otherwise it will be sent to failure")
+ .required(true)
+ .allowableValues(RoutingStrategy.values())
+ .defaultValue(RoutingStrategy.SKIP_UNENRICHED.name())
+ .build();
+
+ public static final PropertyDescriptor RECORD_READER = new PropertyDescriptor.Builder()
+ .name("record-reader")
+ .displayName("Record Reader")
+ .description("Specifies the record reader service to use for reading incoming data")
+ .required(true)
+ .identifiesControllerService(RecordReaderFactory.class)
+ .build();
+
+ public static final PropertyDescriptor RECORD_WRITER = new PropertyDescriptor.Builder()
+ .name("record-writer")
+ .displayName("Record Writer")
+ .description("Specifies the record writer service to use for writing data")
+ .required(true)
+ .identifiesControllerService(RecordSetWriterFactory.class)
+ .build();
+
+ public static final PropertyDescriptor LATITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("latitude-record-path")
+ .displayName("Latitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the latitude value; "
+ + "in the decode mode, this property specifies the record path to put the latitude value")
+ .required(true)
+ .addValidator(new RecordPathValidator())
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor LONGITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("longitude-record-path")
+ .displayName("Longitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the longitude value; "
+ + "in the decode mode, this property specifies the record path to put the longitude value")
+ .required(true)
+ .addValidator(new RecordPathValidator())
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_FORMAT = new PropertyDescriptor.Builder()
+ .name("geohash-format")
+ .displayName("Geohash Format")
+ .description("In the encode mode, this property specifies the desired format for encoding geohash; "
+ + "in the decode mode, this property specifies the format of geohash provided")
+ .required(true)
+ .allowableValues(GeohashFormat.values())
+ .defaultValue(GeohashFormat.BASE_32.name())
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_LEVEL = new PropertyDescriptor.Builder()
+ .name("geohash-level")
+ .displayName("Geohash Level")
+ .description("The integer precision level(1-12) desired for encoding geohash")
+ .required(true)
+ .addValidator(StandardValidators.createLongValidator(1, 12, true))
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .dependsOn(MODE, ProcessingMode.ENCODE.name())
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("geohash-record-path")
+ .displayName("Geohash Record Path")
+ .description("In the encode mode, this property specifies the record path to put the geohash value; "
+ + "in the decode mode, this property specifies the record path to retrieve the geohash value")
+ .required(true)
+ .addValidator(new RecordPathValidator())
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Records that are successfully encoded or decoded will be routed to success")
+ .build();
+
+ public static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Records that cannot be encoded or decoded will be routed to failure")
+ .build();
+
+ public static final Relationship REL_ORIGINAL = new Relationship.Builder()
+ .name("original")
+ .description("With the Split_Enriched_Unenriched strategy, the original input flowfile will be sent to this relationship regardless of whether it was enriched or not.")
Review comment:
```suggestion
.description("With the SPLIT strategy, the original input flowfile will be sent to this relationship regardless of whether it was enriched or not.")
```
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,449 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.record.path.validation.RecordPathValidator;
+
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.WriteResult;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+
+ public enum ProcessingMode {
+ ENCODE, DECODE
+ }
+
+ public enum GeohashFormat {
+ BASE_32, BINARY_STRING, LONG
+ }
+
+ public enum RoutingStrategy {
+ SKIP_UNENRICHED,
+ SPLIT,
+ REQUIRE_ALL_ENRICHED
+ }
+
+ public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
+ .name("mode")
+ .displayName("Mode")
+ .description("Specifies whether to encode latitude/longitude to geohash or decode geohash to latitude/longitude")
+ .required(true)
+ .allowableValues(ProcessingMode.values())
+ .defaultValue(ProcessingMode.ENCODE.name())
+ .build();
+
+ public static final PropertyDescriptor ROUTING_STRATEGY = new PropertyDescriptor.Builder()
+ .name("routing-strategy")
+ .displayName("Routing Strategy")
+ .description("Specifies how to route records after encoding or decoding has been performed. "
+ + "Skip_Unenriched will route a flowfile to success if any of its records is enriched, otherwise it will be sent to failure. "
+ + "Split_Enriched_Unenriched will separate the records that have been enriched from those have not and send them to success, while unenriched records will be sent to failure; "
Review comment:
```suggestion
+ "SPLIT will separate the records that have been enriched from those have not and send them to success, while unenriched records will be sent to failure; "
```
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/pom.xml
##########
@@ -0,0 +1,98 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-geohash-bundle</artifactId>
+ <version>1.15.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-geohash-processors</artifactId>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-utils</artifactId>
+ <version>1.15.0-SNAPSHOT</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-mock</artifactId>
+ <version>1.15.0-SNAPSHOT</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>ch.hsr</groupId>
+ <artifactId>geohash</artifactId>
+ <version>1.4.0</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-record-serialization-service-api</artifactId>
Review comment:
This dependency can use the `provided` scope since it is included in the parent `nifi-standard-services-api-nar`.
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,449 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.record.path.validation.RecordPathValidator;
+
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.WriteResult;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+
+ public enum ProcessingMode {
+ ENCODE, DECODE
+ }
+
+ public enum GeohashFormat {
+ BASE_32, BINARY_STRING, LONG
+ }
+
+ public enum RoutingStrategy {
+ SKIP_UNENRICHED,
+ SPLIT,
+ REQUIRE_ALL_ENRICHED
+ }
+
+ public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
+ .name("mode")
+ .displayName("Mode")
+ .description("Specifies whether to encode latitude/longitude to geohash or decode geohash to latitude/longitude")
+ .required(true)
+ .allowableValues(ProcessingMode.values())
+ .defaultValue(ProcessingMode.ENCODE.name())
+ .build();
+
+ public static final PropertyDescriptor ROUTING_STRATEGY = new PropertyDescriptor.Builder()
+ .name("routing-strategy")
+ .displayName("Routing Strategy")
+ .description("Specifies how to route records after encoding or decoding has been performed. "
+ + "Skip_Unenriched will route a flowfile to success if any of its records is enriched, otherwise it will be sent to failure. "
+ + "Split_Enriched_Unenriched will separate the records that have been enriched from those have not and send them to success, while unenriched records will be sent to failure; "
+ + "and the original flowfile will be sent to the original relationship. "
+ + "Require_All_Enriched will route a flowfile to success only if all of its records are enriched, otherwise it will be sent to failure")
Review comment:
```suggestion
+ "REQUIRE_ALL_ENRICHED will route a flowfile to success only if all of its records are enriched, otherwise it will be sent to failure")
```
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,449 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.record.path.validation.RecordPathValidator;
+
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.WriteResult;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+
+ public enum ProcessingMode {
+ ENCODE, DECODE
+ }
+
+ public enum GeohashFormat {
+ BASE_32, BINARY_STRING, LONG
+ }
+
+ public enum RoutingStrategy {
+ SKIP_UNENRICHED,
+ SPLIT,
+ REQUIRE_ALL_ENRICHED
+ }
+
+ public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
+ .name("mode")
+ .displayName("Mode")
+ .description("Specifies whether to encode latitude/longitude to geohash or decode geohash to latitude/longitude")
+ .required(true)
+ .allowableValues(ProcessingMode.values())
+ .defaultValue(ProcessingMode.ENCODE.name())
+ .build();
+
+ public static final PropertyDescriptor ROUTING_STRATEGY = new PropertyDescriptor.Builder()
+ .name("routing-strategy")
+ .displayName("Routing Strategy")
+ .description("Specifies how to route records after encoding or decoding has been performed. "
+ + "Skip_Unenriched will route a flowfile to success if any of its records is enriched, otherwise it will be sent to failure. "
Review comment:
```suggestion
+ "SKIP_UNENRICHED will route a flowfile to success if any of its records is enriched, otherwise it will be sent to failure. "
```
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,449 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.record.path.validation.RecordPathValidator;
+
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.WriteResult;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+
+ public enum ProcessingMode {
+ ENCODE, DECODE
+ }
+
+ public enum GeohashFormat {
+ BASE_32, BINARY_STRING, LONG
+ }
+
+ public enum RoutingStrategy {
+ SKIP_UNENRICHED,
+ SPLIT,
+ REQUIRE_ALL_ENRICHED
+ }
+
+ public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
+ .name("mode")
+ .displayName("Mode")
+ .description("Specifies whether to encode latitude/longitude to geohash or decode geohash to latitude/longitude")
+ .required(true)
+ .allowableValues(ProcessingMode.values())
+ .defaultValue(ProcessingMode.ENCODE.name())
+ .build();
+
+ public static final PropertyDescriptor ROUTING_STRATEGY = new PropertyDescriptor.Builder()
+ .name("routing-strategy")
+ .displayName("Routing Strategy")
+ .description("Specifies how to route records after encoding or decoding has been performed. "
+ + "Skip_Unenriched will route a flowfile to success if any of its records is enriched, otherwise it will be sent to failure. "
+ + "Split_Enriched_Unenriched will separate the records that have been enriched from those have not and send them to success, while unenriched records will be sent to failure; "
+ + "and the original flowfile will be sent to the original relationship. "
+ + "Require_All_Enriched will route a flowfile to success only if all of its records are enriched, otherwise it will be sent to failure")
+ .required(true)
+ .allowableValues(RoutingStrategy.values())
+ .defaultValue(RoutingStrategy.SKIP_UNENRICHED.name())
+ .build();
+
+ public static final PropertyDescriptor RECORD_READER = new PropertyDescriptor.Builder()
+ .name("record-reader")
+ .displayName("Record Reader")
+ .description("Specifies the record reader service to use for reading incoming data")
+ .required(true)
+ .identifiesControllerService(RecordReaderFactory.class)
+ .build();
+
+ public static final PropertyDescriptor RECORD_WRITER = new PropertyDescriptor.Builder()
+ .name("record-writer")
+ .displayName("Record Writer")
+ .description("Specifies the record writer service to use for writing data")
+ .required(true)
+ .identifiesControllerService(RecordSetWriterFactory.class)
+ .build();
+
+ public static final PropertyDescriptor LATITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("latitude-record-path")
+ .displayName("Latitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the latitude value; "
+ + "in the decode mode, this property specifies the record path to put the latitude value")
+ .required(true)
+ .addValidator(new RecordPathValidator())
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor LONGITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("longitude-record-path")
+ .displayName("Longitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the longitude value; "
+ + "in the decode mode, this property specifies the record path to put the longitude value")
+ .required(true)
+ .addValidator(new RecordPathValidator())
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_FORMAT = new PropertyDescriptor.Builder()
+ .name("geohash-format")
+ .displayName("Geohash Format")
+ .description("In the encode mode, this property specifies the desired format for encoding geohash; "
+ + "in the decode mode, this property specifies the format of geohash provided")
+ .required(true)
+ .allowableValues(GeohashFormat.values())
+ .defaultValue(GeohashFormat.BASE_32.name())
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_LEVEL = new PropertyDescriptor.Builder()
+ .name("geohash-level")
+ .displayName("Geohash Level")
+ .description("The integer precision level(1-12) desired for encoding geohash")
+ .required(true)
+ .addValidator(StandardValidators.createLongValidator(1, 12, true))
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .dependsOn(MODE, ProcessingMode.ENCODE.name())
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("geohash-record-path")
+ .displayName("Geohash Record Path")
+ .description("In the encode mode, this property specifies the record path to put the geohash value; "
+ + "in the decode mode, this property specifies the record path to retrieve the geohash value")
+ .required(true)
+ .addValidator(new RecordPathValidator())
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Records that are successfully encoded or decoded will be routed to success")
+ .build();
+
+ public static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Records that cannot be encoded or decoded will be routed to failure")
+ .build();
+
+ public static final Relationship REL_ORIGINAL = new Relationship.Builder()
+ .name("original")
+ .description("With the Split_Enriched_Unenriched strategy, the original input flowfile will be sent to this relationship regardless of whether it was enriched or not.")
+ .build();
+
+ private static final List<PropertyDescriptor> RECORD_PATH_PROPERTIES = Collections.unmodifiableList(Arrays.asList(
+ LATITUDE_RECORD_PATH, LONGITUDE_RECORD_PATH, GEOHASH_RECORD_PATH
+ ));
+
+ private final RecordPathCache cache = new RecordPathCache(100);
+
+ private List<PropertyDescriptor> descriptors;
+
+ private Set<Relationship> relationships;
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ descriptors = new ArrayList<>();
+ descriptors.add(MODE);
+ descriptors.add(RECORD_READER);
+ descriptors.add(RECORD_WRITER);
+ descriptors.add(ROUTING_STRATEGY);
+ descriptors.add(LATITUDE_RECORD_PATH);
+ descriptors.add(LONGITUDE_RECORD_PATH);
+ descriptors.add(GEOHASH_RECORD_PATH);
+ descriptors.add(GEOHASH_FORMAT);
+ descriptors.add(GEOHASH_LEVEL);
+ descriptors = Collections.unmodifiableList(descriptors);
+
+ relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ relationships.add(REL_FAILURE);
+ relationships.add(REL_ORIGINAL);
+ relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) {
+ FlowFile input = session.get();
+ if (input == null) {
+ return;
+ }
+
+ final RecordReaderFactory readerFactory = context.getProperty(RECORD_READER).asControllerService(RecordReaderFactory.class);
+ final RecordSetWriterFactory writerFactory = context.getProperty(RECORD_WRITER).asControllerService(RecordSetWriterFactory.class);
+ final boolean encode = context.getProperty(MODE).getValue().equalsIgnoreCase(ProcessingMode.ENCODE.toString());
+ final RoutingStrategy routingStrategy = RoutingStrategy.valueOf(context.getProperty(ROUTING_STRATEGY).getValue());
+ final String format = context.getProperty(GEOHASH_FORMAT).getValue();
+
+ FlowFile output = session.create(input);
+ FlowFile notFound = routingStrategy == RoutingStrategy.SPLIT ? session.create(input) : null;
+
+ try (InputStream is = session.read(input);
+ RecordReader reader = readerFactory.createRecordReader(input, is, getLogger());
+ OutputStream os = session.write(output);
+ RecordSetWriter writer = writerFactory.createWriter(getLogger(), writerFactory.getSchema(input.getAttributes(), reader.getSchema()), os, output);
+ OutputStream osNotFound = routingStrategy == RoutingStrategy.SPLIT ? session.write(notFound) : null;
+ RecordSetWriter notFoundWriter = routingStrategy == RoutingStrategy.SPLIT ? writerFactory.createWriter(getLogger(), reader.getSchema(), osNotFound, notFound) : null) {
Review comment:
Recommend adding `final` to these declarations:
```suggestion
try (final InputStream is = session.read(input);
final RecordReader reader = readerFactory.createRecordReader(input, is, getLogger());
final OutputStream os = session.write(output);
final RecordSetWriter writer = writerFactory.createWriter(getLogger(), writerFactory.getSchema(input.getAttributes(), reader.getSchema()), os, output);
final OutputStream osNotFound = routingStrategy == RoutingStrategy.SPLIT ? session.write(notFound) : null;
final RecordSetWriter notFoundWriter = routingStrategy == RoutingStrategy.SPLIT ? writerFactory.createWriter(getLogger(), reader.getSchema(), osNotFound, notFound) : null) {
```
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/test/java/org/apache/nifi/processors/geohash/GeohashRecordTest.java
##########
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.avro.Schema;
+import org.apache.commons.io.IOUtils;
+import org.apache.nifi.avro.AvroTypeUtil;
+import org.apache.nifi.controller.ControllerService;
+import org.apache.nifi.json.JsonRecordSetWriter;
+import org.apache.nifi.json.JsonTreeReader;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.schema.access.SchemaAccessUtils;
+import org.apache.nifi.serialization.record.MockSchemaRegistry;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.codehaus.jackson.map.ObjectMapper;
+
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+
+import java.io.InputStream;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+public class GeohashRecordTest {
+
+ private final TestRunner runner = TestRunners.newTestRunner(GeohashRecord.class);
+
+ @BeforeEach
+ public void setUp() throws InitializationException {
+ ControllerService reader = new JsonTreeReader();
+ ControllerService writer = new JsonRecordSetWriter();
+ ControllerService registry = new MockSchemaRegistry();
+ runner.addControllerService("reader", reader);
+ runner.addControllerService("writer", writer);
+ runner.addControllerService("registry", registry);
+
+ try (InputStream is = getClass().getResourceAsStream("/record_schema.avsc")) {
+ String raw = IOUtils.toString(is, "UTF-8");
+ RecordSchema parsed = AvroTypeUtil.createSchema(new Schema.Parser().parse(raw));
+ ((MockSchemaRegistry) registry).addSchema("record", parsed);
+
+ } catch (Exception ex) {
+ throw new RuntimeException(ex);
+ }
+
+ runner.setProperty(reader, SchemaAccessUtils.SCHEMA_REGISTRY, "registry");
+ runner.setProperty(writer, SchemaAccessUtils.SCHEMA_ACCESS_STRATEGY, SchemaAccessUtils.SCHEMA_NAME_PROPERTY);
+ runner.setProperty(writer, SchemaAccessUtils.SCHEMA_REGISTRY, "registry");
+
+ runner.setProperty(GeohashRecord.RECORD_READER, "reader");
+ runner.setProperty(GeohashRecord.RECORD_WRITER, "writer");
+ runner.enableControllerService(registry);
+ runner.enableControllerService(reader);
+ runner.enableControllerService(writer);
+
+ runner.setProperty(GeohashRecord.LATITUDE_RECORD_PATH, "/latitude");
+ runner.setProperty(GeohashRecord.LONGITUDE_RECORD_PATH, "/longitude");
+ runner.setProperty(GeohashRecord.GEOHASH_RECORD_PATH, "/geohash");
+ runner.setProperty(GeohashRecord.GEOHASH_FORMAT, GeohashRecord.GeohashFormat.BASE_32.toString());
+ runner.setProperty(GeohashRecord.GEOHASH_LEVEL, "12");
+ }
+
+ private void commonTest(String path, int failure, int success, int original) {
Review comment:
For clarity of implementation, recommend renaming this to something like `run()` or `assertTransfers()`
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,449 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.record.path.validation.RecordPathValidator;
+
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.WriteResult;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
Review comment:
An additional `WritesAttributes` annotation should be added to indicate the attributes that can be written during processing, such as `record.count`.
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,449 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.record.path.validation.RecordPathValidator;
+
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.WriteResult;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+
+ public enum ProcessingMode {
+ ENCODE, DECODE
+ }
+
+ public enum GeohashFormat {
+ BASE_32, BINARY_STRING, LONG
+ }
+
+ public enum RoutingStrategy {
+ SKIP_UNENRICHED,
+ SPLIT,
+ REQUIRE_ALL_ENRICHED
+ }
+
+ public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
+ .name("mode")
+ .displayName("Mode")
+ .description("Specifies whether to encode latitude/longitude to geohash or decode geohash to latitude/longitude")
+ .required(true)
+ .allowableValues(ProcessingMode.values())
+ .defaultValue(ProcessingMode.ENCODE.name())
+ .build();
+
+ public static final PropertyDescriptor ROUTING_STRATEGY = new PropertyDescriptor.Builder()
+ .name("routing-strategy")
+ .displayName("Routing Strategy")
+ .description("Specifies how to route records after encoding or decoding has been performed. "
+ + "Skip_Unenriched will route a flowfile to success if any of its records is enriched, otherwise it will be sent to failure. "
+ + "Split_Enriched_Unenriched will separate the records that have been enriched from those have not and send them to success, while unenriched records will be sent to failure; "
+ + "and the original flowfile will be sent to the original relationship. "
+ + "Require_All_Enriched will route a flowfile to success only if all of its records are enriched, otherwise it will be sent to failure")
+ .required(true)
+ .allowableValues(RoutingStrategy.values())
+ .defaultValue(RoutingStrategy.SKIP_UNENRICHED.name())
+ .build();
+
+ public static final PropertyDescriptor RECORD_READER = new PropertyDescriptor.Builder()
+ .name("record-reader")
+ .displayName("Record Reader")
+ .description("Specifies the record reader service to use for reading incoming data")
+ .required(true)
+ .identifiesControllerService(RecordReaderFactory.class)
+ .build();
+
+ public static final PropertyDescriptor RECORD_WRITER = new PropertyDescriptor.Builder()
+ .name("record-writer")
+ .displayName("Record Writer")
+ .description("Specifies the record writer service to use for writing data")
+ .required(true)
+ .identifiesControllerService(RecordSetWriterFactory.class)
+ .build();
+
+ public static final PropertyDescriptor LATITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("latitude-record-path")
+ .displayName("Latitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the latitude value; "
+ + "in the decode mode, this property specifies the record path to put the latitude value")
+ .required(true)
+ .addValidator(new RecordPathValidator())
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor LONGITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("longitude-record-path")
+ .displayName("Longitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the longitude value; "
+ + "in the decode mode, this property specifies the record path to put the longitude value")
+ .required(true)
+ .addValidator(new RecordPathValidator())
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_FORMAT = new PropertyDescriptor.Builder()
+ .name("geohash-format")
+ .displayName("Geohash Format")
+ .description("In the encode mode, this property specifies the desired format for encoding geohash; "
+ + "in the decode mode, this property specifies the format of geohash provided")
+ .required(true)
+ .allowableValues(GeohashFormat.values())
+ .defaultValue(GeohashFormat.BASE_32.name())
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_LEVEL = new PropertyDescriptor.Builder()
+ .name("geohash-level")
+ .displayName("Geohash Level")
+ .description("The integer precision level(1-12) desired for encoding geohash")
+ .required(true)
+ .addValidator(StandardValidators.createLongValidator(1, 12, true))
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .dependsOn(MODE, ProcessingMode.ENCODE.name())
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("geohash-record-path")
+ .displayName("Geohash Record Path")
+ .description("In the encode mode, this property specifies the record path to put the geohash value; "
+ + "in the decode mode, this property specifies the record path to retrieve the geohash value")
+ .required(true)
+ .addValidator(new RecordPathValidator())
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Records that are successfully encoded or decoded will be routed to success")
+ .build();
+
+ public static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Records that cannot be encoded or decoded will be routed to failure")
+ .build();
+
+ public static final Relationship REL_ORIGINAL = new Relationship.Builder()
+ .name("original")
+ .description("With the Split_Enriched_Unenriched strategy, the original input flowfile will be sent to this relationship regardless of whether it was enriched or not.")
+ .build();
+
+ private static final List<PropertyDescriptor> RECORD_PATH_PROPERTIES = Collections.unmodifiableList(Arrays.asList(
+ LATITUDE_RECORD_PATH, LONGITUDE_RECORD_PATH, GEOHASH_RECORD_PATH
+ ));
+
+ private final RecordPathCache cache = new RecordPathCache(100);
+
+ private List<PropertyDescriptor> descriptors;
+
+ private Set<Relationship> relationships;
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ descriptors = new ArrayList<>();
+ descriptors.add(MODE);
+ descriptors.add(RECORD_READER);
+ descriptors.add(RECORD_WRITER);
+ descriptors.add(ROUTING_STRATEGY);
+ descriptors.add(LATITUDE_RECORD_PATH);
+ descriptors.add(LONGITUDE_RECORD_PATH);
+ descriptors.add(GEOHASH_RECORD_PATH);
+ descriptors.add(GEOHASH_FORMAT);
+ descriptors.add(GEOHASH_LEVEL);
+ descriptors = Collections.unmodifiableList(descriptors);
+
+ relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ relationships.add(REL_FAILURE);
+ relationships.add(REL_ORIGINAL);
+ relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) {
+ FlowFile input = session.get();
+ if (input == null) {
+ return;
+ }
+
+ final RecordReaderFactory readerFactory = context.getProperty(RECORD_READER).asControllerService(RecordReaderFactory.class);
+ final RecordSetWriterFactory writerFactory = context.getProperty(RECORD_WRITER).asControllerService(RecordSetWriterFactory.class);
+ final boolean encode = context.getProperty(MODE).getValue().equalsIgnoreCase(ProcessingMode.ENCODE.toString());
+ final RoutingStrategy routingStrategy = RoutingStrategy.valueOf(context.getProperty(ROUTING_STRATEGY).getValue());
+ final String format = context.getProperty(GEOHASH_FORMAT).getValue();
+
+ FlowFile output = session.create(input);
+ FlowFile notFound = routingStrategy == RoutingStrategy.SPLIT ? session.create(input) : null;
+
+ try (InputStream is = session.read(input);
+ RecordReader reader = readerFactory.createRecordReader(input, is, getLogger());
+ OutputStream os = session.write(output);
+ RecordSetWriter writer = writerFactory.createWriter(getLogger(), writerFactory.getSchema(input.getAttributes(), reader.getSchema()), os, output);
+ OutputStream osNotFound = routingStrategy == RoutingStrategy.SPLIT ? session.write(notFound) : null;
+ RecordSetWriter notFoundWriter = routingStrategy == RoutingStrategy.SPLIT ? writerFactory.createWriter(getLogger(), reader.getSchema(), osNotFound, notFound) : null) {
+
+ Map<PropertyDescriptor, RecordPath> paths = new HashMap<>();
+ for (PropertyDescriptor descriptor : RECORD_PATH_PROPERTIES) {
+ String rawRecordPath = context.getProperty(descriptor).evaluateAttributeExpressions(input).getValue();
+ RecordPath compiled = cache.getCompiled(rawRecordPath);
+ paths.put(descriptor, compiled);
+ }
+
+ Record record;
+
+ //The overall relationship used by the Require_All_Enriched and Skip_Unenriched routing strategies to transfer Flowfiles.
+ //For the Split_Enriched_Unenriched strategy, the transfer of Flowfiles does not rely on this overall relationship. Instead, each individual record will be sent to success or failure.
+ Relationship targetRelationship = routingStrategy == RoutingStrategy.REQUIRE_ALL_ENRICHED ? REL_SUCCESS : REL_FAILURE;
+
+ writer.beginRecordSet();
+
+ if (notFoundWriter != null) {
+ notFoundWriter.beginRecordSet();
+ }
+
+ int foundCount = 0;
+ int notFoundCount = 0;
+
+ int level = context.getProperty(GEOHASH_LEVEL).asInteger();
+ final String rawLatitudePath = context.getProperty(LATITUDE_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath latitudePath = cache.getCompiled(rawLatitudePath);
+ final String rawLongitudePath = context.getProperty(LONGITUDE_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath longitudePath = cache.getCompiled(rawLongitudePath);
+ final String rawgeohashPath = context.getProperty(GEOHASH_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath geohashPath = cache.getCompiled(rawgeohashPath);
+
+ while ((record = reader.nextRecord()) != null) {
+ boolean updated = false;
+ if (encode) {
+ Object encodedGeohash = getEncodedGeohash(latitudePath, longitudePath, record, format, level);
+ updated = updateRecord(GEOHASH_RECORD_PATH, encodedGeohash, record, paths);
+ } else {
+ WGS84Point decodedPoint = getDecodedPointFromGeohash(geohashPath, record, format);
+ if (decodedPoint != null) {
+ updated = updateRecord(LATITUDE_RECORD_PATH, String.valueOf(decodedPoint.getLatitude()), record, paths)
+ && updateRecord(LONGITUDE_RECORD_PATH, String.valueOf(decodedPoint.getLongitude()), record, paths);
+ }
+ }
+
+ if (routingStrategy == RoutingStrategy.REQUIRE_ALL_ENRICHED) {
+ if (!updated) { //If the routing strategy is Require_All_Enriched and there exists a record that is not updated, the entire flowfile should be route to REL_FAILURE
+ targetRelationship = REL_FAILURE;
+ }
+ } else {
+ if (updated) {
+ //If the routing strategy is Skip_Unenriched and there exists a record that is updated, the entire flowfile should be route to REL_SUCCESS
+ //If the routing strategy is Split_Enriched_Unenriched and there exists a record that is updated, this record should be route to REL_SUCCESS
+ targetRelationship = REL_SUCCESS;
+ }
+ }
+
+ if (routingStrategy != RoutingStrategy.SPLIT || updated) {
+ writer.write(record);
+ foundCount++;
+ } else { //if the routing strategy is Split_Enriched_Unenriched and the record is not updated
+ notFoundWriter.write(record);
+ notFoundCount++;
+ }
+ }
+
+ final WriteResult writeResult = writer.finishRecordSet();
+ writer.close();
+
+ WriteResult notFoundWriterResult = null;
+
+ if (notFoundWriter != null) {
+ notFoundWriterResult = notFoundWriter.finishRecordSet();
+ notFoundWriter.close();
+ }
+
+ is.close();
+ os.close();
+
+ if (osNotFound != null) {
+ osNotFound.close();
+ }
Review comment:
Can those `close()` method calls be removed since the streams are declared in the enclosing `try` block and should be closed automatically?
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,449 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.record.path.validation.RecordPathValidator;
+
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.WriteResult;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+
+ public enum ProcessingMode {
+ ENCODE, DECODE
+ }
+
+ public enum GeohashFormat {
+ BASE_32, BINARY_STRING, LONG
+ }
+
+ public enum RoutingStrategy {
+ SKIP_UNENRICHED,
+ SPLIT,
+ REQUIRE_ALL_ENRICHED
+ }
+
+ public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
+ .name("mode")
+ .displayName("Mode")
+ .description("Specifies whether to encode latitude/longitude to geohash or decode geohash to latitude/longitude")
+ .required(true)
+ .allowableValues(ProcessingMode.values())
+ .defaultValue(ProcessingMode.ENCODE.name())
+ .build();
+
+ public static final PropertyDescriptor ROUTING_STRATEGY = new PropertyDescriptor.Builder()
+ .name("routing-strategy")
+ .displayName("Routing Strategy")
+ .description("Specifies how to route records after encoding or decoding has been performed. "
+ + "Skip_Unenriched will route a flowfile to success if any of its records is enriched, otherwise it will be sent to failure. "
+ + "Split_Enriched_Unenriched will separate the records that have been enriched from those have not and send them to success, while unenriched records will be sent to failure; "
+ + "and the original flowfile will be sent to the original relationship. "
+ + "Require_All_Enriched will route a flowfile to success only if all of its records are enriched, otherwise it will be sent to failure")
+ .required(true)
+ .allowableValues(RoutingStrategy.values())
+ .defaultValue(RoutingStrategy.SKIP_UNENRICHED.name())
+ .build();
+
+ public static final PropertyDescriptor RECORD_READER = new PropertyDescriptor.Builder()
+ .name("record-reader")
+ .displayName("Record Reader")
+ .description("Specifies the record reader service to use for reading incoming data")
+ .required(true)
+ .identifiesControllerService(RecordReaderFactory.class)
+ .build();
+
+ public static final PropertyDescriptor RECORD_WRITER = new PropertyDescriptor.Builder()
+ .name("record-writer")
+ .displayName("Record Writer")
+ .description("Specifies the record writer service to use for writing data")
+ .required(true)
+ .identifiesControllerService(RecordSetWriterFactory.class)
+ .build();
+
+ public static final PropertyDescriptor LATITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("latitude-record-path")
+ .displayName("Latitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the latitude value; "
+ + "in the decode mode, this property specifies the record path to put the latitude value")
+ .required(true)
+ .addValidator(new RecordPathValidator())
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor LONGITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("longitude-record-path")
+ .displayName("Longitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the longitude value; "
+ + "in the decode mode, this property specifies the record path to put the longitude value")
+ .required(true)
+ .addValidator(new RecordPathValidator())
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_FORMAT = new PropertyDescriptor.Builder()
+ .name("geohash-format")
+ .displayName("Geohash Format")
+ .description("In the encode mode, this property specifies the desired format for encoding geohash; "
+ + "in the decode mode, this property specifies the format of geohash provided")
+ .required(true)
+ .allowableValues(GeohashFormat.values())
+ .defaultValue(GeohashFormat.BASE_32.name())
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_LEVEL = new PropertyDescriptor.Builder()
+ .name("geohash-level")
+ .displayName("Geohash Level")
+ .description("The integer precision level(1-12) desired for encoding geohash")
+ .required(true)
+ .addValidator(StandardValidators.createLongValidator(1, 12, true))
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .dependsOn(MODE, ProcessingMode.ENCODE.name())
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("geohash-record-path")
+ .displayName("Geohash Record Path")
+ .description("In the encode mode, this property specifies the record path to put the geohash value; "
+ + "in the decode mode, this property specifies the record path to retrieve the geohash value")
+ .required(true)
+ .addValidator(new RecordPathValidator())
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Records that are successfully encoded or decoded will be routed to success")
+ .build();
+
+ public static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Records that cannot be encoded or decoded will be routed to failure")
+ .build();
+
+ public static final Relationship REL_ORIGINAL = new Relationship.Builder()
+ .name("original")
+ .description("With the Split_Enriched_Unenriched strategy, the original input flowfile will be sent to this relationship regardless of whether it was enriched or not.")
+ .build();
+
+ private static final List<PropertyDescriptor> RECORD_PATH_PROPERTIES = Collections.unmodifiableList(Arrays.asList(
+ LATITUDE_RECORD_PATH, LONGITUDE_RECORD_PATH, GEOHASH_RECORD_PATH
+ ));
+
+ private final RecordPathCache cache = new RecordPathCache(100);
+
+ private List<PropertyDescriptor> descriptors;
+
+ private Set<Relationship> relationships;
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ descriptors = new ArrayList<>();
+ descriptors.add(MODE);
+ descriptors.add(RECORD_READER);
+ descriptors.add(RECORD_WRITER);
+ descriptors.add(ROUTING_STRATEGY);
+ descriptors.add(LATITUDE_RECORD_PATH);
+ descriptors.add(LONGITUDE_RECORD_PATH);
+ descriptors.add(GEOHASH_RECORD_PATH);
+ descriptors.add(GEOHASH_FORMAT);
+ descriptors.add(GEOHASH_LEVEL);
+ descriptors = Collections.unmodifiableList(descriptors);
+
+ relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ relationships.add(REL_FAILURE);
+ relationships.add(REL_ORIGINAL);
+ relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) {
+ FlowFile input = session.get();
+ if (input == null) {
+ return;
+ }
+
+ final RecordReaderFactory readerFactory = context.getProperty(RECORD_READER).asControllerService(RecordReaderFactory.class);
+ final RecordSetWriterFactory writerFactory = context.getProperty(RECORD_WRITER).asControllerService(RecordSetWriterFactory.class);
+ final boolean encode = context.getProperty(MODE).getValue().equalsIgnoreCase(ProcessingMode.ENCODE.toString());
+ final RoutingStrategy routingStrategy = RoutingStrategy.valueOf(context.getProperty(ROUTING_STRATEGY).getValue());
Review comment:
The Property Descriptors indicate that these properties support expression language, but `evaluateAttributeExpression()` is not called.
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,449 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.record.path.validation.RecordPathValidator;
+
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.WriteResult;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+
+ public enum ProcessingMode {
+ ENCODE, DECODE
+ }
+
+ public enum GeohashFormat {
+ BASE_32, BINARY_STRING, LONG
+ }
+
+ public enum RoutingStrategy {
+ SKIP_UNENRICHED,
+ SPLIT,
+ REQUIRE_ALL_ENRICHED
+ }
+
+ public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
+ .name("mode")
+ .displayName("Mode")
+ .description("Specifies whether to encode latitude/longitude to geohash or decode geohash to latitude/longitude")
+ .required(true)
+ .allowableValues(ProcessingMode.values())
+ .defaultValue(ProcessingMode.ENCODE.name())
+ .build();
+
+ public static final PropertyDescriptor ROUTING_STRATEGY = new PropertyDescriptor.Builder()
+ .name("routing-strategy")
+ .displayName("Routing Strategy")
+ .description("Specifies how to route records after encoding or decoding has been performed. "
+ + "Skip_Unenriched will route a flowfile to success if any of its records is enriched, otherwise it will be sent to failure. "
+ + "Split_Enriched_Unenriched will separate the records that have been enriched from those have not and send them to success, while unenriched records will be sent to failure; "
+ + "and the original flowfile will be sent to the original relationship. "
+ + "Require_All_Enriched will route a flowfile to success only if all of its records are enriched, otherwise it will be sent to failure")
+ .required(true)
+ .allowableValues(RoutingStrategy.values())
+ .defaultValue(RoutingStrategy.SKIP_UNENRICHED.name())
+ .build();
+
+ public static final PropertyDescriptor RECORD_READER = new PropertyDescriptor.Builder()
+ .name("record-reader")
+ .displayName("Record Reader")
+ .description("Specifies the record reader service to use for reading incoming data")
+ .required(true)
+ .identifiesControllerService(RecordReaderFactory.class)
+ .build();
+
+ public static final PropertyDescriptor RECORD_WRITER = new PropertyDescriptor.Builder()
+ .name("record-writer")
+ .displayName("Record Writer")
+ .description("Specifies the record writer service to use for writing data")
+ .required(true)
+ .identifiesControllerService(RecordSetWriterFactory.class)
+ .build();
+
+ public static final PropertyDescriptor LATITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("latitude-record-path")
+ .displayName("Latitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the latitude value; "
+ + "in the decode mode, this property specifies the record path to put the latitude value")
+ .required(true)
+ .addValidator(new RecordPathValidator())
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor LONGITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("longitude-record-path")
+ .displayName("Longitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the longitude value; "
+ + "in the decode mode, this property specifies the record path to put the longitude value")
+ .required(true)
+ .addValidator(new RecordPathValidator())
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_FORMAT = new PropertyDescriptor.Builder()
+ .name("geohash-format")
+ .displayName("Geohash Format")
+ .description("In the encode mode, this property specifies the desired format for encoding geohash; "
+ + "in the decode mode, this property specifies the format of geohash provided")
+ .required(true)
+ .allowableValues(GeohashFormat.values())
+ .defaultValue(GeohashFormat.BASE_32.name())
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_LEVEL = new PropertyDescriptor.Builder()
+ .name("geohash-level")
+ .displayName("Geohash Level")
+ .description("The integer precision level(1-12) desired for encoding geohash")
+ .required(true)
+ .addValidator(StandardValidators.createLongValidator(1, 12, true))
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .dependsOn(MODE, ProcessingMode.ENCODE.name())
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("geohash-record-path")
+ .displayName("Geohash Record Path")
+ .description("In the encode mode, this property specifies the record path to put the geohash value; "
+ + "in the decode mode, this property specifies the record path to retrieve the geohash value")
+ .required(true)
+ .addValidator(new RecordPathValidator())
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Records that are successfully encoded or decoded will be routed to success")
+ .build();
+
+ public static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Records that cannot be encoded or decoded will be routed to failure")
+ .build();
+
+ public static final Relationship REL_ORIGINAL = new Relationship.Builder()
+ .name("original")
+ .description("With the Split_Enriched_Unenriched strategy, the original input flowfile will be sent to this relationship regardless of whether it was enriched or not.")
+ .build();
+
+ private static final List<PropertyDescriptor> RECORD_PATH_PROPERTIES = Collections.unmodifiableList(Arrays.asList(
+ LATITUDE_RECORD_PATH, LONGITUDE_RECORD_PATH, GEOHASH_RECORD_PATH
+ ));
+
+ private final RecordPathCache cache = new RecordPathCache(100);
+
+ private List<PropertyDescriptor> descriptors;
+
+ private Set<Relationship> relationships;
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ descriptors = new ArrayList<>();
+ descriptors.add(MODE);
+ descriptors.add(RECORD_READER);
+ descriptors.add(RECORD_WRITER);
+ descriptors.add(ROUTING_STRATEGY);
+ descriptors.add(LATITUDE_RECORD_PATH);
+ descriptors.add(LONGITUDE_RECORD_PATH);
+ descriptors.add(GEOHASH_RECORD_PATH);
+ descriptors.add(GEOHASH_FORMAT);
+ descriptors.add(GEOHASH_LEVEL);
+ descriptors = Collections.unmodifiableList(descriptors);
+
+ relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ relationships.add(REL_FAILURE);
+ relationships.add(REL_ORIGINAL);
+ relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) {
+ FlowFile input = session.get();
+ if (input == null) {
+ return;
+ }
+
+ final RecordReaderFactory readerFactory = context.getProperty(RECORD_READER).asControllerService(RecordReaderFactory.class);
+ final RecordSetWriterFactory writerFactory = context.getProperty(RECORD_WRITER).asControllerService(RecordSetWriterFactory.class);
+ final boolean encode = context.getProperty(MODE).getValue().equalsIgnoreCase(ProcessingMode.ENCODE.toString());
+ final RoutingStrategy routingStrategy = RoutingStrategy.valueOf(context.getProperty(ROUTING_STRATEGY).getValue());
+ final String format = context.getProperty(GEOHASH_FORMAT).getValue();
+
+ FlowFile output = session.create(input);
+ FlowFile notFound = routingStrategy == RoutingStrategy.SPLIT ? session.create(input) : null;
+
+ try (InputStream is = session.read(input);
+ RecordReader reader = readerFactory.createRecordReader(input, is, getLogger());
+ OutputStream os = session.write(output);
+ RecordSetWriter writer = writerFactory.createWriter(getLogger(), writerFactory.getSchema(input.getAttributes(), reader.getSchema()), os, output);
+ OutputStream osNotFound = routingStrategy == RoutingStrategy.SPLIT ? session.write(notFound) : null;
+ RecordSetWriter notFoundWriter = routingStrategy == RoutingStrategy.SPLIT ? writerFactory.createWriter(getLogger(), reader.getSchema(), osNotFound, notFound) : null) {
+
+ Map<PropertyDescriptor, RecordPath> paths = new HashMap<>();
+ for (PropertyDescriptor descriptor : RECORD_PATH_PROPERTIES) {
+ String rawRecordPath = context.getProperty(descriptor).evaluateAttributeExpressions(input).getValue();
+ RecordPath compiled = cache.getCompiled(rawRecordPath);
+ paths.put(descriptor, compiled);
+ }
+
+ Record record;
+
+ //The overall relationship used by the Require_All_Enriched and Skip_Unenriched routing strategies to transfer Flowfiles.
+ //For the Split_Enriched_Unenriched strategy, the transfer of Flowfiles does not rely on this overall relationship. Instead, each individual record will be sent to success or failure.
+ Relationship targetRelationship = routingStrategy == RoutingStrategy.REQUIRE_ALL_ENRICHED ? REL_SUCCESS : REL_FAILURE;
+
+ writer.beginRecordSet();
+
+ if (notFoundWriter != null) {
+ notFoundWriter.beginRecordSet();
+ }
+
+ int foundCount = 0;
+ int notFoundCount = 0;
+
+ int level = context.getProperty(GEOHASH_LEVEL).asInteger();
+ final String rawLatitudePath = context.getProperty(LATITUDE_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath latitudePath = cache.getCompiled(rawLatitudePath);
+ final String rawLongitudePath = context.getProperty(LONGITUDE_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath longitudePath = cache.getCompiled(rawLongitudePath);
+ final String rawgeohashPath = context.getProperty(GEOHASH_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath geohashPath = cache.getCompiled(rawgeohashPath);
+
+ while ((record = reader.nextRecord()) != null) {
+ boolean updated = false;
+ if (encode) {
+ Object encodedGeohash = getEncodedGeohash(latitudePath, longitudePath, record, format, level);
+ updated = updateRecord(GEOHASH_RECORD_PATH, encodedGeohash, record, paths);
+ } else {
+ WGS84Point decodedPoint = getDecodedPointFromGeohash(geohashPath, record, format);
+ if (decodedPoint != null) {
+ updated = updateRecord(LATITUDE_RECORD_PATH, String.valueOf(decodedPoint.getLatitude()), record, paths)
+ && updateRecord(LONGITUDE_RECORD_PATH, String.valueOf(decodedPoint.getLongitude()), record, paths);
+ }
+ }
+
+ if (routingStrategy == RoutingStrategy.REQUIRE_ALL_ENRICHED) {
+ if (!updated) { //If the routing strategy is Require_All_Enriched and there exists a record that is not updated, the entire flowfile should be route to REL_FAILURE
+ targetRelationship = REL_FAILURE;
+ }
+ } else {
+ if (updated) {
+ //If the routing strategy is Skip_Unenriched and there exists a record that is updated, the entire flowfile should be route to REL_SUCCESS
+ //If the routing strategy is Split_Enriched_Unenriched and there exists a record that is updated, this record should be route to REL_SUCCESS
+ targetRelationship = REL_SUCCESS;
+ }
+ }
+
+ if (routingStrategy != RoutingStrategy.SPLIT || updated) {
+ writer.write(record);
+ foundCount++;
+ } else { //if the routing strategy is Split_Enriched_Unenriched and the record is not updated
+ notFoundWriter.write(record);
+ notFoundCount++;
+ }
+ }
+
+ final WriteResult writeResult = writer.finishRecordSet();
+ writer.close();
+
+ WriteResult notFoundWriterResult = null;
+
+ if (notFoundWriter != null) {
+ notFoundWriterResult = notFoundWriter.finishRecordSet();
+ notFoundWriter.close();
+ }
+
+ is.close();
+ os.close();
+
+ if (osNotFound != null) {
+ osNotFound.close();
+ }
+
+ output = session.putAllAttributes(output, buildAttributes(foundCount, writer.getMimeType(), writeResult));
+ if (routingStrategy != RoutingStrategy.SPLIT) {
+ session.transfer(output, targetRelationship);
+ session.remove(input);
+ } else {
+ if (notFoundCount > 0) {
+ notFound = session.putAllAttributes(notFound, buildAttributes(notFoundCount, writer.getMimeType(), notFoundWriterResult));
+ session.transfer(notFound, REL_FAILURE);
+ } else {
+ session.remove(notFound);
+ }
+ session.transfer(output, REL_SUCCESS);
+ session.transfer(input, REL_ORIGINAL);
+ }
+
+ } catch (Exception ex) {
+ getLogger().error("Failed to {} due to {}", encode ? "encode" : "decode", ex.getLocalizedMessage(), ex);
+ }
+ }
+
+
+ private Object getEncodedGeohash(RecordPath latitudePath, RecordPath longitudePath, Record record, String format, int level) {
+ RecordPathResult latitudeResult = latitudePath.evaluate(record);
+ RecordPathResult longitudeResult = longitudePath.evaluate(record);
+ Optional<FieldValue> latitudeField = latitudeResult.getSelectedFields().findFirst();
+ Optional<FieldValue> longitudeField = longitudeResult.getSelectedFields().findFirst();
+
+ if (!latitudeField.isPresent() || !longitudeField.isPresent()) {
+ return null;
+ }
+
+ FieldValue latitudeValue = latitudeField.get();
+ FieldValue longitudeValue = longitudeField.get();
+ Object latitudeVal = latitudeValue.getValue();
+ Object longitudeVal = longitudeValue.getValue();
+
+ if (latitudeVal == null || longitudeVal == null) {
+ return null;
+ }
+
+ double realLatValue = Double.parseDouble(latitudeVal.toString());
+ double realLongValue = Double.parseDouble(longitudeVal.toString());
+ GeoHash gh = GeoHash.withCharacterPrecision(realLatValue, realLongValue, level);
+
+ switch (GeohashFormat.valueOf(format)) {
+ case BINARY_STRING:
+ return gh.toBinaryString();
+ case LONG:
+ return gh.longValue();
+ default:
+ return gh.toBase32();
+ }
+ }
+
+ private WGS84Point getDecodedPointFromGeohash(RecordPath geohashPath, Record record, String format) {
Review comment:
As mentioned above, recommend using `GeohashFormat`:
```suggestion
private WGS84Point getDecodedPointFromGeohash(RecordPath geohashPath, Record record, GeohashFormat format) {
```
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,449 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.record.path.validation.RecordPathValidator;
+
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.WriteResult;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+
+ public enum ProcessingMode {
+ ENCODE, DECODE
+ }
+
+ public enum GeohashFormat {
+ BASE_32, BINARY_STRING, LONG
+ }
+
+ public enum RoutingStrategy {
+ SKIP_UNENRICHED,
+ SPLIT,
+ REQUIRE_ALL_ENRICHED
+ }
+
+ public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
+ .name("mode")
+ .displayName("Mode")
+ .description("Specifies whether to encode latitude/longitude to geohash or decode geohash to latitude/longitude")
+ .required(true)
+ .allowableValues(ProcessingMode.values())
+ .defaultValue(ProcessingMode.ENCODE.name())
+ .build();
+
+ public static final PropertyDescriptor ROUTING_STRATEGY = new PropertyDescriptor.Builder()
+ .name("routing-strategy")
+ .displayName("Routing Strategy")
+ .description("Specifies how to route records after encoding or decoding has been performed. "
+ + "Skip_Unenriched will route a flowfile to success if any of its records is enriched, otherwise it will be sent to failure. "
+ + "Split_Enriched_Unenriched will separate the records that have been enriched from those have not and send them to success, while unenriched records will be sent to failure; "
+ + "and the original flowfile will be sent to the original relationship. "
+ + "Require_All_Enriched will route a flowfile to success only if all of its records are enriched, otherwise it will be sent to failure")
+ .required(true)
+ .allowableValues(RoutingStrategy.values())
+ .defaultValue(RoutingStrategy.SKIP_UNENRICHED.name())
+ .build();
+
+ public static final PropertyDescriptor RECORD_READER = new PropertyDescriptor.Builder()
+ .name("record-reader")
+ .displayName("Record Reader")
+ .description("Specifies the record reader service to use for reading incoming data")
+ .required(true)
+ .identifiesControllerService(RecordReaderFactory.class)
+ .build();
+
+ public static final PropertyDescriptor RECORD_WRITER = new PropertyDescriptor.Builder()
+ .name("record-writer")
+ .displayName("Record Writer")
+ .description("Specifies the record writer service to use for writing data")
+ .required(true)
+ .identifiesControllerService(RecordSetWriterFactory.class)
+ .build();
+
+ public static final PropertyDescriptor LATITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("latitude-record-path")
+ .displayName("Latitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the latitude value; "
+ + "in the decode mode, this property specifies the record path to put the latitude value")
+ .required(true)
+ .addValidator(new RecordPathValidator())
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor LONGITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("longitude-record-path")
+ .displayName("Longitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the longitude value; "
+ + "in the decode mode, this property specifies the record path to put the longitude value")
+ .required(true)
+ .addValidator(new RecordPathValidator())
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_FORMAT = new PropertyDescriptor.Builder()
+ .name("geohash-format")
+ .displayName("Geohash Format")
+ .description("In the encode mode, this property specifies the desired format for encoding geohash; "
+ + "in the decode mode, this property specifies the format of geohash provided")
+ .required(true)
+ .allowableValues(GeohashFormat.values())
+ .defaultValue(GeohashFormat.BASE_32.name())
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_LEVEL = new PropertyDescriptor.Builder()
+ .name("geohash-level")
+ .displayName("Geohash Level")
+ .description("The integer precision level(1-12) desired for encoding geohash")
+ .required(true)
+ .addValidator(StandardValidators.createLongValidator(1, 12, true))
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .dependsOn(MODE, ProcessingMode.ENCODE.name())
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("geohash-record-path")
+ .displayName("Geohash Record Path")
+ .description("In the encode mode, this property specifies the record path to put the geohash value; "
+ + "in the decode mode, this property specifies the record path to retrieve the geohash value")
+ .required(true)
+ .addValidator(new RecordPathValidator())
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Records that are successfully encoded or decoded will be routed to success")
+ .build();
+
+ public static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Records that cannot be encoded or decoded will be routed to failure")
+ .build();
+
+ public static final Relationship REL_ORIGINAL = new Relationship.Builder()
+ .name("original")
+ .description("With the Split_Enriched_Unenriched strategy, the original input flowfile will be sent to this relationship regardless of whether it was enriched or not.")
+ .build();
+
+ private static final List<PropertyDescriptor> RECORD_PATH_PROPERTIES = Collections.unmodifiableList(Arrays.asList(
+ LATITUDE_RECORD_PATH, LONGITUDE_RECORD_PATH, GEOHASH_RECORD_PATH
+ ));
+
+ private final RecordPathCache cache = new RecordPathCache(100);
+
+ private List<PropertyDescriptor> descriptors;
+
+ private Set<Relationship> relationships;
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ descriptors = new ArrayList<>();
+ descriptors.add(MODE);
+ descriptors.add(RECORD_READER);
+ descriptors.add(RECORD_WRITER);
+ descriptors.add(ROUTING_STRATEGY);
+ descriptors.add(LATITUDE_RECORD_PATH);
+ descriptors.add(LONGITUDE_RECORD_PATH);
+ descriptors.add(GEOHASH_RECORD_PATH);
+ descriptors.add(GEOHASH_FORMAT);
+ descriptors.add(GEOHASH_LEVEL);
+ descriptors = Collections.unmodifiableList(descriptors);
+
+ relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ relationships.add(REL_FAILURE);
+ relationships.add(REL_ORIGINAL);
+ relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) {
+ FlowFile input = session.get();
+ if (input == null) {
+ return;
+ }
+
+ final RecordReaderFactory readerFactory = context.getProperty(RECORD_READER).asControllerService(RecordReaderFactory.class);
+ final RecordSetWriterFactory writerFactory = context.getProperty(RECORD_WRITER).asControllerService(RecordSetWriterFactory.class);
+ final boolean encode = context.getProperty(MODE).getValue().equalsIgnoreCase(ProcessingMode.ENCODE.toString());
+ final RoutingStrategy routingStrategy = RoutingStrategy.valueOf(context.getProperty(ROUTING_STRATEGY).getValue());
+ final String format = context.getProperty(GEOHASH_FORMAT).getValue();
+
+ FlowFile output = session.create(input);
+ FlowFile notFound = routingStrategy == RoutingStrategy.SPLIT ? session.create(input) : null;
+
+ try (InputStream is = session.read(input);
+ RecordReader reader = readerFactory.createRecordReader(input, is, getLogger());
+ OutputStream os = session.write(output);
+ RecordSetWriter writer = writerFactory.createWriter(getLogger(), writerFactory.getSchema(input.getAttributes(), reader.getSchema()), os, output);
+ OutputStream osNotFound = routingStrategy == RoutingStrategy.SPLIT ? session.write(notFound) : null;
+ RecordSetWriter notFoundWriter = routingStrategy == RoutingStrategy.SPLIT ? writerFactory.createWriter(getLogger(), reader.getSchema(), osNotFound, notFound) : null) {
+
+ Map<PropertyDescriptor, RecordPath> paths = new HashMap<>();
+ for (PropertyDescriptor descriptor : RECORD_PATH_PROPERTIES) {
+ String rawRecordPath = context.getProperty(descriptor).evaluateAttributeExpressions(input).getValue();
+ RecordPath compiled = cache.getCompiled(rawRecordPath);
+ paths.put(descriptor, compiled);
+ }
+
+ Record record;
+
+ //The overall relationship used by the Require_All_Enriched and Skip_Unenriched routing strategies to transfer Flowfiles.
+ //For the Split_Enriched_Unenriched strategy, the transfer of Flowfiles does not rely on this overall relationship. Instead, each individual record will be sent to success or failure.
+ Relationship targetRelationship = routingStrategy == RoutingStrategy.REQUIRE_ALL_ENRICHED ? REL_SUCCESS : REL_FAILURE;
+
+ writer.beginRecordSet();
+
+ if (notFoundWriter != null) {
+ notFoundWriter.beginRecordSet();
+ }
+
+ int foundCount = 0;
+ int notFoundCount = 0;
+
+ int level = context.getProperty(GEOHASH_LEVEL).asInteger();
+ final String rawLatitudePath = context.getProperty(LATITUDE_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath latitudePath = cache.getCompiled(rawLatitudePath);
+ final String rawLongitudePath = context.getProperty(LONGITUDE_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath longitudePath = cache.getCompiled(rawLongitudePath);
+ final String rawgeohashPath = context.getProperty(GEOHASH_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath geohashPath = cache.getCompiled(rawgeohashPath);
+
+ while ((record = reader.nextRecord()) != null) {
+ boolean updated = false;
+ if (encode) {
+ Object encodedGeohash = getEncodedGeohash(latitudePath, longitudePath, record, format, level);
+ updated = updateRecord(GEOHASH_RECORD_PATH, encodedGeohash, record, paths);
+ } else {
+ WGS84Point decodedPoint = getDecodedPointFromGeohash(geohashPath, record, format);
+ if (decodedPoint != null) {
+ updated = updateRecord(LATITUDE_RECORD_PATH, String.valueOf(decodedPoint.getLatitude()), record, paths)
+ && updateRecord(LONGITUDE_RECORD_PATH, String.valueOf(decodedPoint.getLongitude()), record, paths);
+ }
+ }
+
+ if (routingStrategy == RoutingStrategy.REQUIRE_ALL_ENRICHED) {
+ if (!updated) { //If the routing strategy is Require_All_Enriched and there exists a record that is not updated, the entire flowfile should be route to REL_FAILURE
+ targetRelationship = REL_FAILURE;
+ }
+ } else {
+ if (updated) {
+ //If the routing strategy is Skip_Unenriched and there exists a record that is updated, the entire flowfile should be route to REL_SUCCESS
+ //If the routing strategy is Split_Enriched_Unenriched and there exists a record that is updated, this record should be route to REL_SUCCESS
+ targetRelationship = REL_SUCCESS;
+ }
+ }
+
+ if (routingStrategy != RoutingStrategy.SPLIT || updated) {
+ writer.write(record);
+ foundCount++;
+ } else { //if the routing strategy is Split_Enriched_Unenriched and the record is not updated
+ notFoundWriter.write(record);
+ notFoundCount++;
+ }
+ }
+
+ final WriteResult writeResult = writer.finishRecordSet();
+ writer.close();
+
+ WriteResult notFoundWriterResult = null;
+
+ if (notFoundWriter != null) {
+ notFoundWriterResult = notFoundWriter.finishRecordSet();
+ notFoundWriter.close();
+ }
+
+ is.close();
+ os.close();
+
+ if (osNotFound != null) {
+ osNotFound.close();
+ }
+
+ output = session.putAllAttributes(output, buildAttributes(foundCount, writer.getMimeType(), writeResult));
+ if (routingStrategy != RoutingStrategy.SPLIT) {
+ session.transfer(output, targetRelationship);
+ session.remove(input);
+ } else {
+ if (notFoundCount > 0) {
+ notFound = session.putAllAttributes(notFound, buildAttributes(notFoundCount, writer.getMimeType(), notFoundWriterResult));
+ session.transfer(notFound, REL_FAILURE);
+ } else {
+ session.remove(notFound);
+ }
+ session.transfer(output, REL_SUCCESS);
+ session.transfer(input, REL_ORIGINAL);
+ }
+
+ } catch (Exception ex) {
+ getLogger().error("Failed to {} due to {}", encode ? "encode" : "decode", ex.getLocalizedMessage(), ex);
+ }
+ }
+
+
+ private Object getEncodedGeohash(RecordPath latitudePath, RecordPath longitudePath, Record record, String format, int level) {
Review comment:
Recommend changing `format` to `GeohashFormat` instead of `String` to avoid calling `GeohashFormat.valueOf()` for each record evaluation.
```suggestion
private Object getEncodedGeohash(RecordPath latitudePath, RecordPath longitudePath, Record record, GeohashFormat format, int level) {
```
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/resources/docs/org.apache.nifi.processors.geohash.GeohashRecord/additionalDetails.html
##########
@@ -0,0 +1,55 @@
+<!DOCTYPE html>
+<html lang="en">
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<head>
+ <meta charset="utf-8"/>
+ <title>GeohashRecord</title>
+ <link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"/>
+</head>
+
+<body>
+<h1>Overview</h1>
+<p>
+ A Geohash value corresponds to an specific area with pre-defined granularity, and is widely used in identifying,
+ representing, and indexing geospatial objects.
+ This GeohashRecord processor provides the ability to encode and decode Geohashes with desired format and precision.
+</p>
+<h3>Formats supported</h3>
+<p>
+<ul>
+ <li>Base32 string: The most commonly used alphanumeric version. It is compact and more human-readale by discarding
+ some letters(such as "a" and "o", "i" and "l") that might cause confusion.
+ </li>
+ <li>Binary String: This format is generated by directly interleaving latitude and longitude binary string. The even
+ bits in the binary strings corresponds to the longitude, while the odd digits corresponds to the latitude.
+ </li>
+ <li>Long Integer: Although this 64-bit integer format is not human-readale, it can be calculated very fast and is
Review comment:
```suggestion
<li>Long: Although this 64-bit number format is not human-readale, it can be calculated very fast and is
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] mikayla-yang commented on a change in pull request #5476: NIFI-9259: Adding GeohashRecord Processor
Posted by GitBox <gi...@apache.org>.
mikayla-yang commented on a change in pull request #5476:
URL: https://github.com/apache/nifi/pull/5476#discussion_r749781535
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,449 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.record.path.validation.RecordPathValidator;
+
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.WriteResult;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+
+ public enum ProcessingMode {
+ ENCODE, DECODE
+ }
+
+ public enum GeohashFormat {
+ BASE_32, BINARY_STRING, LONG
+ }
+
+ public enum RoutingStrategy {
+ SKIP_UNENRICHED,
+ SPLIT,
+ REQUIRE_ALL_ENRICHED
+ }
+
+ public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
+ .name("mode")
+ .displayName("Mode")
+ .description("Specifies whether to encode latitude/longitude to geohash or decode geohash to latitude/longitude")
+ .required(true)
+ .allowableValues(ProcessingMode.values())
+ .defaultValue(ProcessingMode.ENCODE.name())
+ .build();
+
+ public static final PropertyDescriptor ROUTING_STRATEGY = new PropertyDescriptor.Builder()
+ .name("routing-strategy")
+ .displayName("Routing Strategy")
+ .description("Specifies how to route records after encoding or decoding has been performed. "
+ + "Skip_Unenriched will route a flowfile to success if any of its records is enriched, otherwise it will be sent to failure. "
+ + "Split_Enriched_Unenriched will separate the records that have been enriched from those have not and send them to success, while unenriched records will be sent to failure; "
+ + "and the original flowfile will be sent to the original relationship. "
+ + "Require_All_Enriched will route a flowfile to success only if all of its records are enriched, otherwise it will be sent to failure")
+ .required(true)
+ .allowableValues(RoutingStrategy.values())
+ .defaultValue(RoutingStrategy.SKIP_UNENRICHED.name())
+ .build();
+
+ public static final PropertyDescriptor RECORD_READER = new PropertyDescriptor.Builder()
+ .name("record-reader")
+ .displayName("Record Reader")
+ .description("Specifies the record reader service to use for reading incoming data")
+ .required(true)
+ .identifiesControllerService(RecordReaderFactory.class)
+ .build();
+
+ public static final PropertyDescriptor RECORD_WRITER = new PropertyDescriptor.Builder()
+ .name("record-writer")
+ .displayName("Record Writer")
+ .description("Specifies the record writer service to use for writing data")
+ .required(true)
+ .identifiesControllerService(RecordSetWriterFactory.class)
+ .build();
+
+ public static final PropertyDescriptor LATITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("latitude-record-path")
+ .displayName("Latitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the latitude value; "
+ + "in the decode mode, this property specifies the record path to put the latitude value")
+ .required(true)
+ .addValidator(new RecordPathValidator())
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor LONGITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("longitude-record-path")
+ .displayName("Longitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the longitude value; "
+ + "in the decode mode, this property specifies the record path to put the longitude value")
+ .required(true)
+ .addValidator(new RecordPathValidator())
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_FORMAT = new PropertyDescriptor.Builder()
+ .name("geohash-format")
+ .displayName("Geohash Format")
+ .description("In the encode mode, this property specifies the desired format for encoding geohash; "
+ + "in the decode mode, this property specifies the format of geohash provided")
+ .required(true)
+ .allowableValues(GeohashFormat.values())
+ .defaultValue(GeohashFormat.BASE_32.name())
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_LEVEL = new PropertyDescriptor.Builder()
+ .name("geohash-level")
+ .displayName("Geohash Level")
+ .description("The integer precision level(1-12) desired for encoding geohash")
+ .required(true)
+ .addValidator(StandardValidators.createLongValidator(1, 12, true))
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .dependsOn(MODE, ProcessingMode.ENCODE.name())
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("geohash-record-path")
+ .displayName("Geohash Record Path")
+ .description("In the encode mode, this property specifies the record path to put the geohash value; "
+ + "in the decode mode, this property specifies the record path to retrieve the geohash value")
+ .required(true)
+ .addValidator(new RecordPathValidator())
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Records that are successfully encoded or decoded will be routed to success")
+ .build();
+
+ public static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Records that cannot be encoded or decoded will be routed to failure")
+ .build();
+
+ public static final Relationship REL_ORIGINAL = new Relationship.Builder()
+ .name("original")
+ .description("With the Split_Enriched_Unenriched strategy, the original input flowfile will be sent to this relationship regardless of whether it was enriched or not.")
+ .build();
+
+ private static final List<PropertyDescriptor> RECORD_PATH_PROPERTIES = Collections.unmodifiableList(Arrays.asList(
+ LATITUDE_RECORD_PATH, LONGITUDE_RECORD_PATH, GEOHASH_RECORD_PATH
+ ));
+
+ private final RecordPathCache cache = new RecordPathCache(100);
+
+ private List<PropertyDescriptor> descriptors;
+
+ private Set<Relationship> relationships;
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ descriptors = new ArrayList<>();
+ descriptors.add(MODE);
+ descriptors.add(RECORD_READER);
+ descriptors.add(RECORD_WRITER);
+ descriptors.add(ROUTING_STRATEGY);
+ descriptors.add(LATITUDE_RECORD_PATH);
+ descriptors.add(LONGITUDE_RECORD_PATH);
+ descriptors.add(GEOHASH_RECORD_PATH);
+ descriptors.add(GEOHASH_FORMAT);
+ descriptors.add(GEOHASH_LEVEL);
+ descriptors = Collections.unmodifiableList(descriptors);
+
+ relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ relationships.add(REL_FAILURE);
+ relationships.add(REL_ORIGINAL);
+ relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) {
+ FlowFile input = session.get();
+ if (input == null) {
+ return;
+ }
+
+ final RecordReaderFactory readerFactory = context.getProperty(RECORD_READER).asControllerService(RecordReaderFactory.class);
+ final RecordSetWriterFactory writerFactory = context.getProperty(RECORD_WRITER).asControllerService(RecordSetWriterFactory.class);
+ final boolean encode = context.getProperty(MODE).getValue().equalsIgnoreCase(ProcessingMode.ENCODE.toString());
+ final RoutingStrategy routingStrategy = RoutingStrategy.valueOf(context.getProperty(ROUTING_STRATEGY).getValue());
+ final String format = context.getProperty(GEOHASH_FORMAT).getValue();
+
+ FlowFile output = session.create(input);
+ FlowFile notFound = routingStrategy == RoutingStrategy.SPLIT ? session.create(input) : null;
+
+ try (InputStream is = session.read(input);
+ RecordReader reader = readerFactory.createRecordReader(input, is, getLogger());
+ OutputStream os = session.write(output);
+ RecordSetWriter writer = writerFactory.createWriter(getLogger(), writerFactory.getSchema(input.getAttributes(), reader.getSchema()), os, output);
+ OutputStream osNotFound = routingStrategy == RoutingStrategy.SPLIT ? session.write(notFound) : null;
+ RecordSetWriter notFoundWriter = routingStrategy == RoutingStrategy.SPLIT ? writerFactory.createWriter(getLogger(), reader.getSchema(), osNotFound, notFound) : null) {
+
+ Map<PropertyDescriptor, RecordPath> paths = new HashMap<>();
+ for (PropertyDescriptor descriptor : RECORD_PATH_PROPERTIES) {
+ String rawRecordPath = context.getProperty(descriptor).evaluateAttributeExpressions(input).getValue();
+ RecordPath compiled = cache.getCompiled(rawRecordPath);
+ paths.put(descriptor, compiled);
+ }
+
+ Record record;
+
+ //The overall relationship used by the Require_All_Enriched and Skip_Unenriched routing strategies to transfer Flowfiles.
+ //For the Split_Enriched_Unenriched strategy, the transfer of Flowfiles does not rely on this overall relationship. Instead, each individual record will be sent to success or failure.
+ Relationship targetRelationship = routingStrategy == RoutingStrategy.REQUIRE_ALL_ENRICHED ? REL_SUCCESS : REL_FAILURE;
+
+ writer.beginRecordSet();
+
+ if (notFoundWriter != null) {
+ notFoundWriter.beginRecordSet();
+ }
+
+ int foundCount = 0;
+ int notFoundCount = 0;
+
+ int level = context.getProperty(GEOHASH_LEVEL).asInteger();
+ final String rawLatitudePath = context.getProperty(LATITUDE_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath latitudePath = cache.getCompiled(rawLatitudePath);
+ final String rawLongitudePath = context.getProperty(LONGITUDE_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath longitudePath = cache.getCompiled(rawLongitudePath);
+ final String rawgeohashPath = context.getProperty(GEOHASH_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath geohashPath = cache.getCompiled(rawgeohashPath);
+
+ while ((record = reader.nextRecord()) != null) {
+ boolean updated = false;
+ if (encode) {
+ Object encodedGeohash = getEncodedGeohash(latitudePath, longitudePath, record, format, level);
+ updated = updateRecord(GEOHASH_RECORD_PATH, encodedGeohash, record, paths);
+ } else {
+ WGS84Point decodedPoint = getDecodedPointFromGeohash(geohashPath, record, format);
+ if (decodedPoint != null) {
+ updated = updateRecord(LATITUDE_RECORD_PATH, String.valueOf(decodedPoint.getLatitude()), record, paths)
+ && updateRecord(LONGITUDE_RECORD_PATH, String.valueOf(decodedPoint.getLongitude()), record, paths);
+ }
+ }
+
+ if (routingStrategy == RoutingStrategy.REQUIRE_ALL_ENRICHED) {
+ if (!updated) { //If the routing strategy is Require_All_Enriched and there exists a record that is not updated, the entire flowfile should be route to REL_FAILURE
+ targetRelationship = REL_FAILURE;
+ }
+ } else {
+ if (updated) {
+ //If the routing strategy is Skip_Unenriched and there exists a record that is updated, the entire flowfile should be route to REL_SUCCESS
+ //If the routing strategy is Split_Enriched_Unenriched and there exists a record that is updated, this record should be route to REL_SUCCESS
+ targetRelationship = REL_SUCCESS;
+ }
+ }
+
+ if (routingStrategy != RoutingStrategy.SPLIT || updated) {
+ writer.write(record);
+ foundCount++;
+ } else { //if the routing strategy is Split_Enriched_Unenriched and the record is not updated
+ notFoundWriter.write(record);
+ notFoundCount++;
+ }
+ }
+
+ final WriteResult writeResult = writer.finishRecordSet();
+ writer.close();
+
+ WriteResult notFoundWriterResult = null;
+
+ if (notFoundWriter != null) {
+ notFoundWriterResult = notFoundWriter.finishRecordSet();
+ notFoundWriter.close();
+ }
+
+ is.close();
+ os.close();
+
+ if (osNotFound != null) {
+ osNotFound.close();
+ }
Review comment:
Thanks for the comment! I have removed all the close() except the ones for `writer` and `notFoundWriter` because they need to be closed manually.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] YolandaMDavis commented on a change in pull request #5476: NIFI-9259: Adding GeohashRecord Processor
Posted by GitBox <gi...@apache.org>.
YolandaMDavis commented on a change in pull request #5476:
URL: https://github.com/apache/nifi/pull/5476#discussion_r756159196
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,367 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+ public static final String ENCODE_MODE = "Encode";
+ public static final String DECODE_MODE = "Decode";
+
+ public static final String BINARY_STRING = "BinaryString";
+ public static final String BASE32 = "Base32";
+ public static final String LONG_INT = "LongInt";
+
+ public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
+ .name("mode")
+ .displayName("Mode")
+ .description("Specifies whether to encode latitude/longitude to geohash or decode geohash to latitude/longitude")
+ .required(true)
+ .allowableValues(ENCODE_MODE, DECODE_MODE)
+ .defaultValue(ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor RECORD_READER = new PropertyDescriptor.Builder()
+ .name("record-reader")
+ .displayName("Record Reader")
+ .description("Specifies the record reader service to use for reading incoming data")
+ .required(true)
+ .identifiesControllerService(RecordReaderFactory.class)
+ .build();
+
+ public static final PropertyDescriptor RECORD_WRITER = new PropertyDescriptor.Builder()
+ .name("record-writer")
+ .displayName("Record Writer")
+ .description("Specifies the record writer service to use for writing data")
+ .required(true)
+ .identifiesControllerService(RecordSetWriterFactory.class)
+ .build();
+
+ public static final PropertyDescriptor LATITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("latitude-record-path")
+ .displayName("Latitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the latitude value;"
+ + "in the decode mode, this property specifies the record path to put the latitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor LONGITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("longitude-record-path")
+ .displayName("Longitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the longitude value;"
+ + "in the decode mode, this property specifies the record path to put the longitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_FORMAT = new PropertyDescriptor.Builder()
+ .name("geohash-format")
+ .displayName("Geohash Format")
+ .description("In the encode mode, this property specifies the desired format for encoding geohash;"
+ + "in the decode mode, this property specifies the format of geohash provided")
+ .required(true)
+ .allowableValues(BASE32, BINARY_STRING, LONG_INT)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_LEVEL = new PropertyDescriptor.Builder()
+ .name("geohash-level")
+ .displayName("Geohash Level")
+ .description("The integer precision level(1-12) desired for encoding geohash")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(MODE, ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("geohash-record-path")
+ .displayName("Geohash Record Path")
+ .description("In the encode mode, this property specifies the record path to put the geohash value;"
+ + "in the decode mode, this property specifies the record path to retrieve the geohash value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Records that are successfully encoded or decoded will be routed to success")
+ .build();
+ public static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Records that cannot be encoded or decoded will be routed to failure")
+ .build();
+
+ public static final List<PropertyDescriptor> RECORD_PATH_PROPERTIES = Collections.unmodifiableList(Arrays.asList(
+ LATITUDE_RECORD_PATH,LONGITUDE_RECORD_PATH,GEOHASH_RECORD_PATH
+ ));
+
+ private List<PropertyDescriptor> descriptors;
+
+ private Set<Relationship> relationships;
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ descriptors = new ArrayList<>();
+ descriptors.add(MODE);
+ descriptors.add(RECORD_READER);
+ descriptors.add(RECORD_WRITER);
+ descriptors.add(LATITUDE_RECORD_PATH);
+ descriptors.add(LONGITUDE_RECORD_PATH);
+ descriptors.add(GEOHASH_FORMAT);
+ descriptors.add(GEOHASH_LEVEL);
+ descriptors.add(GEOHASH_RECORD_PATH);
+ descriptors = Collections.unmodifiableList(descriptors);
+
+ relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ relationships.add(REL_FAILURE);
+ relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) {
+ FlowFile input = session.get();
+ if ( input == null ) {
+ return;
+ }
+
+ final RecordReaderFactory readerFactory = context.getProperty(RECORD_READER).asControllerService(RecordReaderFactory.class);
+ final RecordSetWriterFactory writerFactory = context.getProperty(RECORD_WRITER).asControllerService(RecordSetWriterFactory.class);
+ final boolean encode = context.getProperty(MODE).getValue().equalsIgnoreCase(ENCODE_MODE);
+ final String format = context.getProperty(GEOHASH_FORMAT).getValue();
+
+ FlowFile output = session.create(input);
+ try (InputStream is = session.read(input);
+ OutputStream os = session.write(output)) {
+ RecordPathCache cache = new RecordPathCache(RECORD_PATH_PROPERTIES.size() + 1);
+ Map<PropertyDescriptor, RecordPath> paths = new HashMap<>();
+
+ for (PropertyDescriptor descriptor : RECORD_PATH_PROPERTIES) {
+ String rawRecordPath = context.getProperty(descriptor).evaluateAttributeExpressions(input).getValue();
+ RecordPath compiled = cache.getCompiled(rawRecordPath);
+ paths.put(descriptor, compiled);
+ }
+
+ RecordReader reader = readerFactory.createRecordReader(input, is, getLogger());
+ RecordSchema schema = writerFactory.getSchema(input.getAttributes(), reader.getSchema());
+ RecordSetWriter writer = writerFactory.createWriter(getLogger(), schema, os, output);
+
+ Record record;
+ Relationship targetRelationship = REL_FAILURE;
+ writer.beginRecordSet();
+
+ int recordsCount = 0;
+ boolean updated = false;
+ while ((record = reader.nextRecord()) != null) {
+ if(encode) {
+ final int level = Integer.parseInt(context.getProperty(GEOHASH_LEVEL).getValue());
+ final String rawLatitudePath = context.getProperty(LATITUDE_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath latitudePath = cache.getCompiled(rawLatitudePath);
+ final String rawLongitudePath = context.getProperty(LONGITUDE_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath longitudePath = cache.getCompiled(rawLongitudePath);
+
+ Object encodedGeohash = getEncodedGeohash(latitudePath, longitudePath, record, format, level);
+
+ updated = updateRecord(GEOHASH_RECORD_PATH, encodedGeohash, record, paths);
+ }else {
+ final String rawgeohashPath = context.getProperty(GEOHASH_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath geohashPath = cache.getCompiled(rawgeohashPath);
+
+ WGS84Point decodedPoint = getDecodedPointFromGeohash(geohashPath, record, format);
+
+ if(decodedPoint != null) {
+ updated = updateRecord(LATITUDE_RECORD_PATH, String.valueOf(decodedPoint.getLatitude()), record, paths)
+ && updateRecord(LONGITUDE_RECORD_PATH, String.valueOf(decodedPoint.getLongitude()), record, paths);
+ }
+ }
+
+ if (updated) {
Review comment:
@mikayla-yang I'd like to recommend another alternative which I think would simplify a bit the approach here, yet still aligns with what @markap14 described. I think this can be similar to what was done in GeoEnrichIPRecord, by having a property that indicates whether or not processing should be "all or nothing" or to allow individual record processing. And instead of having the success or fail have a "found", "not found" and "original" relationships.
In the instance where it's all or nothing and we have one record that did not produce a geohash, the entire flow file would land on the "not found" relationship, otherwise it would land in "found". In the case where we'd support individual record processing, it would then ensure to split records into found or not found as needed. And it would always populate the original connection with the original payload no matter what option is selected.
What do you think of this? @markap14 does this align with the scenario you described? @mikayla-yang would this be something you could migrate to if @markap14 agrees it's a fit?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] markap14 commented on pull request #5476: NIFI-9259: Adding GeohashRecord Processor
Posted by GitBox <gi...@apache.org>.
markap14 commented on pull request #5476:
URL: https://github.com/apache/nifi/pull/5476#issuecomment-991331935
Thanks for all of your effort here @mikayla-yang ! All looks good to me now. And with a +1 from @exceptionfactory I've merged your changes to main!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] markap14 merged pull request #5476: NIFI-9259: Adding GeohashRecord Processor
Posted by GitBox <gi...@apache.org>.
markap14 merged pull request #5476:
URL: https://github.com/apache/nifi/pull/5476
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] mikayla-yang commented on pull request #5476: NIFI-9259: Adding GeohashRecord Processor
Posted by GitBox <gi...@apache.org>.
mikayla-yang commented on pull request #5476:
URL: https://github.com/apache/nifi/pull/5476#issuecomment-957072411
@markap14 @exceptionfactory Thank you for the comments! I have made changes according to your reviews.
@markap14 For handling the failure conditions, I made the change based on your suggestion and also referred to the logic of GeoEnrichIPRecord. I didn’t refactor the code to use the Map structure as we discussed because for the _Require_All_Enriched_ routing strategy, we will not know which RecordSetWriter to use until we finish processing all records. However, using the Map structure requires us to know the final relationship every time we write a record in order to use a proper RecordSetWriter.
LISCENCE and NOTICE files have been added in the NAR module. The NOTICE under nifi-assembly has been updated as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a change in pull request #5476: NIFI-9259: Adding GeohashRecord Processor
Posted by GitBox <gi...@apache.org>.
exceptionfactory commented on a change in pull request #5476:
URL: https://github.com/apache/nifi/pull/5476#discussion_r741239106
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,439 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.record.path.validation.RecordPathValidator;
+
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.WriteResult;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+
+ public enum ProcessingMode {
+ Encode, Decode
+ }
+
+ public enum GeohashFormat {
+ Base32, Binary_String, Long_Int
Review comment:
Recommend renaming as follows:
```suggestion
BASE_32, BINARY_STRING, LONG
```
The name "Long Int" is not very intuitive since `Long` and `Integer` are two different incompatible types in Java. Perhaps just naming this `LONG` or perhaps `NUMBER` might be more intuitive.
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,439 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.record.path.validation.RecordPathValidator;
+
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.WriteResult;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+
+ public enum ProcessingMode {
+ Encode, Decode
Review comment:
Recommend renaming these values to follow standard enum naming conventions:
```suggestion
ENCODE, DECODE
```
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,439 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.record.path.validation.RecordPathValidator;
+
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.WriteResult;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+
+ public enum ProcessingMode {
+ Encode, Decode
+ }
+
+ public enum GeohashFormat {
+ Base32, Binary_String, Long_Int
+ }
+
+ public enum RoutingStrategy {
+ Skip_Unenriched,
+ Split_Enriched_Unenriched,
+ Require_All_Enriched
Review comment:
Recommend renaming as follows:
```suggestion
SKIP_UNENRICHED,
SPLIT,
REQUIRE_ALL_ENRICHED
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a change in pull request #5476: NIFI-9259: Adding GeohashRecord Processor
Posted by GitBox <gi...@apache.org>.
exceptionfactory commented on a change in pull request #5476:
URL: https://github.com/apache/nifi/pull/5476#discussion_r741239106
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,439 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.record.path.validation.RecordPathValidator;
+
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.WriteResult;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+
+ public enum ProcessingMode {
+ Encode, Decode
+ }
+
+ public enum GeohashFormat {
+ Base32, Binary_String, Long_Int
Review comment:
Recommend renaming as follows:
```suggestion
BASE_32, BINARY_STRING, LONG
```
The name "Long Int" is not very intuitive since `Long` and `Integer` are two different incompatible types in Java. Perhaps just naming this `LONG` or perhaps `NUMBER` might be more intuitive.
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,439 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.record.path.validation.RecordPathValidator;
+
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.WriteResult;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+
+ public enum ProcessingMode {
+ Encode, Decode
Review comment:
Recommend renaming these values to follow standard enum naming conventions:
```suggestion
ENCODE, DECODE
```
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,439 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.record.path.validation.RecordPathValidator;
+
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.WriteResult;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+
+ public enum ProcessingMode {
+ Encode, Decode
+ }
+
+ public enum GeohashFormat {
+ Base32, Binary_String, Long_Int
+ }
+
+ public enum RoutingStrategy {
+ Skip_Unenriched,
+ Split_Enriched_Unenriched,
+ Require_All_Enriched
Review comment:
Recommend renaming as follows:
```suggestion
SKIP_UNENRICHED,
SPLIT,
REQUIRE_ALL_ENRICHED
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] markap14 commented on a change in pull request #5476: NIFI-9259: Adding GeohashRecord Processor
Posted by GitBox <gi...@apache.org>.
markap14 commented on a change in pull request #5476:
URL: https://github.com/apache/nifi/pull/5476#discussion_r766170394
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,541 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.record.path.validation.RecordPathValidator;
+
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.serialization.MalformedRecordException;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.WriteResult;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@WritesAttributes({
+ @WritesAttribute(attribute = "mime.type", description = "The MIME type indicated by the record writer"),
+ @WritesAttribute(attribute = "record.count", description = "The number of records in the resulting flow file")
+})
+public class GeohashRecord extends AbstractProcessor {
+
+ public enum ProcessingMode {
+ ENCODE, DECODE
+ }
+
+ public enum GeohashFormat {
+ BASE32, BINARY, LONG
+ }
+
+ public enum RoutingStrategy {
+ SKIP,
+ SPLIT,
+ REQUIRE
+ }
+
+ public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
+ .name("mode")
+ .displayName("Mode")
+ .description("Specifies whether to encode latitude/longitude to geohash or decode geohash to latitude/longitude")
+ .required(true)
+ .allowableValues(ProcessingMode.values())
+ .defaultValue(ProcessingMode.ENCODE.name())
+ .build();
+
+ public static final PropertyDescriptor ROUTING_STRATEGY = new PropertyDescriptor.Builder()
+ .name("routing-strategy")
+ .displayName("Routing Strategy")
+ .description("Specifies how to route flowfiles after encoding or decoding being performed. "
+ + "SKIP will enrich those records that can be enriched and skip the rest. "
+ + "The SKIP strategy will route a flowfile to failure only if unable to parse the data. "
+ + "Otherwise, it will route the enriched flowfile to success, and the original input to original. "
+ + "SPLIT will separate the records that have been enriched from those that have not and send them to matched, while unenriched records will be sent to unmatched; "
+ + "the original input flowfile will be sent to original. The SPLIT strategy will route a flowfile to failure only if unable to parse the data. "
+ + "REQUIRE will route a flowfile to success only if all of its records are enriched, and the original input will be sent to original. "
+ + "The REQUIRE strategy will route the original input flowfile to failure if any of its records cannot be enriched or unable to be parsed")
+ .required(true)
+ .allowableValues(RoutingStrategy.values())
+ .defaultValue(RoutingStrategy.SKIP.name())
+ .build();
+
+ public static final PropertyDescriptor RECORD_READER = new PropertyDescriptor.Builder()
+ .name("record-reader")
+ .displayName("Record Reader")
+ .description("Specifies the record reader service to use for reading incoming data")
+ .required(true)
+ .identifiesControllerService(RecordReaderFactory.class)
+ .build();
+
+ public static final PropertyDescriptor RECORD_WRITER = new PropertyDescriptor.Builder()
+ .name("record-writer")
+ .displayName("Record Writer")
+ .description("Specifies the record writer service to use for writing data")
+ .required(true)
+ .identifiesControllerService(RecordSetWriterFactory.class)
+ .build();
+
+ public static final PropertyDescriptor LATITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("latitude-record-path")
+ .displayName("Latitude Record Path")
+ .description("In the ENCODE mode, this property specifies the record path to retrieve the latitude values. "
+ + "Latitude values should be in the range of [-90, 90]; invalid values will be logged at warn level. "
+ + "In the DECODE mode, this property specifies the record path to put the latitude value")
+ .required(true)
+ .addValidator(new RecordPathValidator())
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor LONGITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("longitude-record-path")
+ .displayName("Longitude Record Path")
+ .description("In the ENCODE mode, this property specifies the record path to retrieve the longitude values; "
+ + "Longitude values should be in the range of [-180, 180]; invalid values will be logged at warn level. "
+ + "In the DECODE mode, this property specifies the record path to put the longitude value")
+ .required(true)
+ .addValidator(new RecordPathValidator())
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("geohash-record-path")
+ .displayName("Geohash Record Path")
+ .description("In the ENCODE mode, this property specifies the record path to put the geohash value; "
+ + "in the DECODE mode, this property specifies the record path to retrieve the geohash value")
+ .required(true)
+ .addValidator(new RecordPathValidator())
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_FORMAT = new PropertyDescriptor.Builder()
+ .name("geohash-format")
+ .displayName("Geohash Format")
+ .description("In the ENCODE mode, this property specifies the desired format for encoding geohash; "
+ + "in the DECODE mode, this property specifies the format of geohash provided")
+ .required(true)
+ .allowableValues(GeohashFormat.values())
+ .defaultValue(GeohashFormat.BASE32.name())
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_LEVEL = new PropertyDescriptor.Builder()
+ .name("geohash-level")
+ .displayName("Geohash Level")
+ .description("The integer precision level(1-12) desired for encoding geohash")
+ .required(true)
+ .addValidator(StandardValidators.createLongValidator(1, 12, true))
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .dependsOn(MODE, ProcessingMode.ENCODE.name())
+ .build();
+
+ public static final Relationship REL_NOT_MATCHED = new Relationship.Builder()
+ .name("not matched")
+ .description("Using the SPLIT strategy, flowfiles that cannot be encoded or decoded due to the lack of lat/lon or geohashes will be routed to not matched")
+ .build();
+
+ public static final Relationship REL_MATCHED = new Relationship.Builder()
+ .name("matched")
+ .description("Using the SPLIT strategy, flowfiles with lat/lon or geohashes provided that are successfully encoded or decoded will be routed to matched")
+ .build();
+
+ public static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Flowfiles that cannot be encoded or decoded will be routed to failure")
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Flowfiles that are successfully encoded or decoded will be routed to success")
+ .build();
+
+ public static final Relationship REL_ORIGINAL = new Relationship.Builder()
+ .name("original")
+ .description("The original input flowfile will be sent to this relationship")
+ .build();
+
+ private static final List<PropertyDescriptor> RECORD_PATH_PROPERTIES = Collections.unmodifiableList(Arrays.asList(
+ LATITUDE_RECORD_PATH, LONGITUDE_RECORD_PATH, GEOHASH_RECORD_PATH
+ ));
+
+ private static final Set<Relationship> RELATIONSHIPS = Collections.unmodifiableSet(new HashSet<>(Arrays.asList(REL_SUCCESS, REL_ORIGINAL, REL_FAILURE)));
+ private static final Set<Relationship> SPLIT_RELATIONSHIPS = Collections.unmodifiableSet(new HashSet<>(Arrays.asList(REL_MATCHED, REL_NOT_MATCHED, REL_ORIGINAL, REL_FAILURE)));
+
+ private RoutingStrategyExecutor routingStrategyExecutor;
+ private static boolean isSplit;
+ private static Integer enrichedCount, unenrichedCount;
+
+ private final RecordPathCache cache = new RecordPathCache(100);
+
+ private List<PropertyDescriptor> descriptors;
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ descriptors = new ArrayList<>();
+ descriptors.add(MODE);
+ descriptors.add(RECORD_READER);
+ descriptors.add(RECORD_WRITER);
+ descriptors.add(ROUTING_STRATEGY);
+ descriptors.add(LATITUDE_RECORD_PATH);
+ descriptors.add(LONGITUDE_RECORD_PATH);
+ descriptors.add(GEOHASH_RECORD_PATH);
+ descriptors.add(GEOHASH_FORMAT);
+ descriptors.add(GEOHASH_LEVEL);
+ descriptors = Collections.unmodifiableList(descriptors);
+ }
+
+ @Override
+ public void onPropertyModified(final PropertyDescriptor descriptor, final String oldValue, final String newValue) {
+ if (descriptor.equals(ROUTING_STRATEGY)) {
+ isSplit = RoutingStrategy.SPLIT.name().equals(newValue);
+ }
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return isSplit ? SPLIT_RELATIONSHIPS : RELATIONSHIPS;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @OnScheduled
+ public void setup(ProcessContext context) {
+ final RoutingStrategy routingStrategy = RoutingStrategy.valueOf(context.getProperty(ROUTING_STRATEGY).getValue());
+ switch (routingStrategy) {
+ case REQUIRE:
+ routingStrategyExecutor = new RequireRoutingStrategyExecutor();
+ break;
+ case SKIP:
+ routingStrategyExecutor = new SkipRoutingStrategyExecutor();
+ break;
+ case SPLIT:
+ routingStrategyExecutor = new SplitRoutingStrategyExecutor();
+ break;
+ default:
+ throw new AssertionError();
+ }
+ enrichedCount = 0;
+ unenrichedCount = 0;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) {
+ FlowFile input = session.get();
+ if (input == null) {
+ return;
+ }
+
+ final RecordReaderFactory readerFactory = context.getProperty(RECORD_READER).asControllerService(RecordReaderFactory.class);
+ final RecordSetWriterFactory writerFactory = context.getProperty(RECORD_WRITER).asControllerService(RecordSetWriterFactory.class);
+ final boolean encode = context.getProperty(MODE).getValue().equalsIgnoreCase(ProcessingMode.ENCODE.toString());
+ final RoutingStrategy routingStrategy = RoutingStrategy.valueOf(context.getProperty(ROUTING_STRATEGY).getValue());
+ final GeohashFormat format = GeohashFormat.valueOf(context.getProperty(GEOHASH_FORMAT).getValue());
+
+ FlowFile output = session.create(input);
+ FlowFile notMatched = routingStrategy == RoutingStrategy.SPLIT ? session.create(input) : null;
+
+ try (final InputStream is = session.read(input);
+ final RecordReader reader = readerFactory.createRecordReader(input, is, getLogger());
+ final OutputStream os = session.write(output);
+ final OutputStream osNotFound = routingStrategy == RoutingStrategy.SPLIT ? session.write(notMatched) : null) {
+
+ final RecordSetWriter writer = writerFactory.createWriter(getLogger(), writerFactory.getSchema(input.getAttributes(), reader.getSchema()), os, output);
+ final RecordSetWriter notMatchedWriter = routingStrategy == RoutingStrategy.SPLIT ? writerFactory.createWriter(getLogger(), reader.getSchema(), osNotFound, notMatched) : null;
+
+ Map<PropertyDescriptor, RecordPath> paths = new HashMap<>();
+ for (PropertyDescriptor descriptor : RECORD_PATH_PROPERTIES) {
+ String rawRecordPath = context.getProperty(descriptor).evaluateAttributeExpressions(input).getValue();
+ RecordPath compiled = cache.getCompiled(rawRecordPath);
+ paths.put(descriptor, compiled);
+ }
+
+ Record record;
+
+ writer.beginRecordSet();
+
+ if (notMatchedWriter != null) {
+ notMatchedWriter.beginRecordSet();
+ }
+
+ int level = context.getProperty(GEOHASH_LEVEL).evaluateAttributeExpressions(input).asInteger();
+ final String rawLatitudePath = context.getProperty(LATITUDE_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath latitudePath = cache.getCompiled(rawLatitudePath);
+ final String rawLongitudePath = context.getProperty(LONGITUDE_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath longitudePath = cache.getCompiled(rawLongitudePath);
+ final String rawGeohashPath = context.getProperty(GEOHASH_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath geohashPath = cache.getCompiled(rawGeohashPath);
+
+ while ((record = reader.nextRecord()) != null) {
+ boolean updated = false;
+
+ try {
+ if (encode) {
+ Object encodedGeohash = getEncodedGeohash(latitudePath, longitudePath, record, format, level);
+ updated = updateRecord(GEOHASH_RECORD_PATH, encodedGeohash, record, paths);
+ } else {
+ WGS84Point decodedPoint = getDecodedPointFromGeohash(geohashPath, record, format);
+ if (decodedPoint != null) {
+ updated = updateRecord(LATITUDE_RECORD_PATH, String.valueOf(decodedPoint.getLatitude()), record, paths)
+ && updateRecord(LONGITUDE_RECORD_PATH, String.valueOf(decodedPoint.getLongitude()), record, paths);
+ }
+ }
+ } catch (IllegalArgumentException e) {
+ //lat/lon/geohash values out of range or is not valid
+ getLogger().warn(e.getLocalizedMessage(), e);
+ }
+
+ routingStrategyExecutor.writeFlowFiles(record, writer, notMatchedWriter, updated);
+ }
+
+ final WriteResult writeResult = writer.finishRecordSet();
+ writer.close();
+ output = session.putAllAttributes(output, buildAttributes(writeResult.getRecordCount(), writer.getMimeType(), writeResult));
+
+ WriteResult notMatchedWriterResult;
+
+ if (notMatchedWriter != null) {
+ notMatchedWriterResult = notMatchedWriter.finishRecordSet();
+ notMatchedWriter.close();
+ if (notMatchedWriterResult.getRecordCount() > 0) {
+ notMatched = session.putAllAttributes(notMatched, buildAttributes(notMatchedWriterResult.getRecordCount(), writer.getMimeType(), notMatchedWriterResult));
+ }
+ }
+ } catch (IOException | SchemaNotFoundException | MalformedRecordException e) {
+ //cannot parse incoming data
+ getLogger().error("Cannot parse the incoming data", e.getLocalizedMessage(), e);
+ session.remove(output);
+ if (notMatched != null) {
+ session.remove(notMatched);
+ }
+ session.transfer(input, REL_FAILURE);
+ return;
+ }
+
+ //Transfer Flowfiles by routing strategy
+ routingStrategyExecutor.transferFlowFiles(session, input, output, notMatched);
+ }
+
+ private interface RoutingStrategyExecutor {
+ void writeFlowFiles(Record record, RecordSetWriter writer, RecordSetWriter notMatchedWriter, boolean updated) throws IOException;
+
+ void transferFlowFiles(final ProcessSession session, FlowFile input, FlowFile output, FlowFile notMatched);
+ }
+
+ private static class SkipRoutingStrategyExecutor implements RoutingStrategyExecutor {
+ @Override
+ public void writeFlowFiles(Record record, RecordSetWriter writer, RecordSetWriter notMatchedWriter, boolean updated) throws IOException {
+ writer.write(record);
+ }
+
+ @Override
+ public void transferFlowFiles(final ProcessSession session, FlowFile input, FlowFile output, FlowFile notMatched) {
+ session.transfer(output, REL_SUCCESS);
+ session.transfer(input, REL_ORIGINAL);
+ }
+ }
+
+ private static class SplitRoutingStrategyExecutor implements RoutingStrategyExecutor {
+ @Override
+ public void writeFlowFiles(Record record, RecordSetWriter writer, RecordSetWriter notMatchedWriter, boolean updated) throws IOException {
+ if (updated) {
+ enrichedCount++;
+ writer.write(record);
+ } else {
+ unenrichedCount++;
+ notMatchedWriter.write(record);
+ }
+ }
+
+ @Override
+ public void transferFlowFiles(final ProcessSession session, FlowFile input, FlowFile output, FlowFile notMatched) {
+ if (unenrichedCount > 0) {
+ session.transfer(notMatched, REL_NOT_MATCHED);
+ } else {
+ session.remove(notMatched);
+ }
+ if (enrichedCount > 0) {
+ session.transfer(output, REL_MATCHED);
+ } else {
+ session.remove(output);
+ }
+ session.transfer(input, REL_ORIGINAL);
+ }
+ }
+
+ private static class RequireRoutingStrategyExecutor implements RoutingStrategyExecutor {
+ @Override
+ public void writeFlowFiles(Record record, RecordSetWriter writer, RecordSetWriter notMatchedWriter, boolean updated) throws IOException {
+ if (updated) {
+ writer.write(record);
+ } else {
+ unenrichedCount++;
+ }
+ }
+
+ @Override
+ public void transferFlowFiles(final ProcessSession session, FlowFile input, FlowFile output, FlowFile notMatched) {
+ if (unenrichedCount > 0) {
+ session.remove(output);
+ session.transfer(input, REL_FAILURE);
Review comment:
Any time we route to 'failure' we should log a warning or error indicating why it was routed there.
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,541 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.record.path.validation.RecordPathValidator;
+
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.serialization.MalformedRecordException;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.WriteResult;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@WritesAttributes({
+ @WritesAttribute(attribute = "mime.type", description = "The MIME type indicated by the record writer"),
+ @WritesAttribute(attribute = "record.count", description = "The number of records in the resulting flow file")
+})
+public class GeohashRecord extends AbstractProcessor {
+
+ public enum ProcessingMode {
+ ENCODE, DECODE
+ }
+
+ public enum GeohashFormat {
+ BASE32, BINARY, LONG
+ }
+
+ public enum RoutingStrategy {
+ SKIP,
+ SPLIT,
+ REQUIRE
+ }
+
+ public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
+ .name("mode")
+ .displayName("Mode")
+ .description("Specifies whether to encode latitude/longitude to geohash or decode geohash to latitude/longitude")
+ .required(true)
+ .allowableValues(ProcessingMode.values())
+ .defaultValue(ProcessingMode.ENCODE.name())
+ .build();
+
+ public static final PropertyDescriptor ROUTING_STRATEGY = new PropertyDescriptor.Builder()
+ .name("routing-strategy")
+ .displayName("Routing Strategy")
+ .description("Specifies how to route flowfiles after encoding or decoding being performed. "
+ + "SKIP will enrich those records that can be enriched and skip the rest. "
+ + "The SKIP strategy will route a flowfile to failure only if unable to parse the data. "
+ + "Otherwise, it will route the enriched flowfile to success, and the original input to original. "
+ + "SPLIT will separate the records that have been enriched from those that have not and send them to matched, while unenriched records will be sent to unmatched; "
+ + "the original input flowfile will be sent to original. The SPLIT strategy will route a flowfile to failure only if unable to parse the data. "
+ + "REQUIRE will route a flowfile to success only if all of its records are enriched, and the original input will be sent to original. "
+ + "The REQUIRE strategy will route the original input flowfile to failure if any of its records cannot be enriched or unable to be parsed")
+ .required(true)
+ .allowableValues(RoutingStrategy.values())
+ .defaultValue(RoutingStrategy.SKIP.name())
+ .build();
+
+ public static final PropertyDescriptor RECORD_READER = new PropertyDescriptor.Builder()
+ .name("record-reader")
+ .displayName("Record Reader")
+ .description("Specifies the record reader service to use for reading incoming data")
+ .required(true)
+ .identifiesControllerService(RecordReaderFactory.class)
+ .build();
+
+ public static final PropertyDescriptor RECORD_WRITER = new PropertyDescriptor.Builder()
+ .name("record-writer")
+ .displayName("Record Writer")
+ .description("Specifies the record writer service to use for writing data")
+ .required(true)
+ .identifiesControllerService(RecordSetWriterFactory.class)
+ .build();
+
+ public static final PropertyDescriptor LATITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("latitude-record-path")
+ .displayName("Latitude Record Path")
+ .description("In the ENCODE mode, this property specifies the record path to retrieve the latitude values. "
+ + "Latitude values should be in the range of [-90, 90]; invalid values will be logged at warn level. "
+ + "In the DECODE mode, this property specifies the record path to put the latitude value")
+ .required(true)
+ .addValidator(new RecordPathValidator())
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor LONGITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("longitude-record-path")
+ .displayName("Longitude Record Path")
+ .description("In the ENCODE mode, this property specifies the record path to retrieve the longitude values; "
+ + "Longitude values should be in the range of [-180, 180]; invalid values will be logged at warn level. "
+ + "In the DECODE mode, this property specifies the record path to put the longitude value")
+ .required(true)
+ .addValidator(new RecordPathValidator())
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("geohash-record-path")
+ .displayName("Geohash Record Path")
+ .description("In the ENCODE mode, this property specifies the record path to put the geohash value; "
+ + "in the DECODE mode, this property specifies the record path to retrieve the geohash value")
+ .required(true)
+ .addValidator(new RecordPathValidator())
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_FORMAT = new PropertyDescriptor.Builder()
+ .name("geohash-format")
+ .displayName("Geohash Format")
+ .description("In the ENCODE mode, this property specifies the desired format for encoding geohash; "
+ + "in the DECODE mode, this property specifies the format of geohash provided")
+ .required(true)
+ .allowableValues(GeohashFormat.values())
+ .defaultValue(GeohashFormat.BASE32.name())
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_LEVEL = new PropertyDescriptor.Builder()
+ .name("geohash-level")
+ .displayName("Geohash Level")
+ .description("The integer precision level(1-12) desired for encoding geohash")
+ .required(true)
+ .addValidator(StandardValidators.createLongValidator(1, 12, true))
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .dependsOn(MODE, ProcessingMode.ENCODE.name())
+ .build();
+
+ public static final Relationship REL_NOT_MATCHED = new Relationship.Builder()
+ .name("not matched")
+ .description("Using the SPLIT strategy, flowfiles that cannot be encoded or decoded due to the lack of lat/lon or geohashes will be routed to not matched")
+ .build();
+
+ public static final Relationship REL_MATCHED = new Relationship.Builder()
+ .name("matched")
+ .description("Using the SPLIT strategy, flowfiles with lat/lon or geohashes provided that are successfully encoded or decoded will be routed to matched")
+ .build();
+
+ public static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Flowfiles that cannot be encoded or decoded will be routed to failure")
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Flowfiles that are successfully encoded or decoded will be routed to success")
+ .build();
+
+ public static final Relationship REL_ORIGINAL = new Relationship.Builder()
+ .name("original")
+ .description("The original input flowfile will be sent to this relationship")
+ .build();
+
+ private static final List<PropertyDescriptor> RECORD_PATH_PROPERTIES = Collections.unmodifiableList(Arrays.asList(
+ LATITUDE_RECORD_PATH, LONGITUDE_RECORD_PATH, GEOHASH_RECORD_PATH
+ ));
+
+ private static final Set<Relationship> RELATIONSHIPS = Collections.unmodifiableSet(new HashSet<>(Arrays.asList(REL_SUCCESS, REL_ORIGINAL, REL_FAILURE)));
+ private static final Set<Relationship> SPLIT_RELATIONSHIPS = Collections.unmodifiableSet(new HashSet<>(Arrays.asList(REL_MATCHED, REL_NOT_MATCHED, REL_ORIGINAL, REL_FAILURE)));
+
+ private RoutingStrategyExecutor routingStrategyExecutor;
+ private static boolean isSplit;
+ private static Integer enrichedCount, unenrichedCount;
+
+ private final RecordPathCache cache = new RecordPathCache(100);
+
+ private List<PropertyDescriptor> descriptors;
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ descriptors = new ArrayList<>();
+ descriptors.add(MODE);
+ descriptors.add(RECORD_READER);
+ descriptors.add(RECORD_WRITER);
+ descriptors.add(ROUTING_STRATEGY);
+ descriptors.add(LATITUDE_RECORD_PATH);
+ descriptors.add(LONGITUDE_RECORD_PATH);
+ descriptors.add(GEOHASH_RECORD_PATH);
+ descriptors.add(GEOHASH_FORMAT);
+ descriptors.add(GEOHASH_LEVEL);
+ descriptors = Collections.unmodifiableList(descriptors);
+ }
+
+ @Override
+ public void onPropertyModified(final PropertyDescriptor descriptor, final String oldValue, final String newValue) {
+ if (descriptor.equals(ROUTING_STRATEGY)) {
+ isSplit = RoutingStrategy.SPLIT.name().equals(newValue);
+ }
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return isSplit ? SPLIT_RELATIONSHIPS : RELATIONSHIPS;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @OnScheduled
+ public void setup(ProcessContext context) {
+ final RoutingStrategy routingStrategy = RoutingStrategy.valueOf(context.getProperty(ROUTING_STRATEGY).getValue());
+ switch (routingStrategy) {
+ case REQUIRE:
+ routingStrategyExecutor = new RequireRoutingStrategyExecutor();
+ break;
+ case SKIP:
+ routingStrategyExecutor = new SkipRoutingStrategyExecutor();
+ break;
+ case SPLIT:
+ routingStrategyExecutor = new SplitRoutingStrategyExecutor();
+ break;
+ default:
+ throw new AssertionError();
+ }
+ enrichedCount = 0;
+ unenrichedCount = 0;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) {
+ FlowFile input = session.get();
+ if (input == null) {
+ return;
+ }
+
+ final RecordReaderFactory readerFactory = context.getProperty(RECORD_READER).asControllerService(RecordReaderFactory.class);
+ final RecordSetWriterFactory writerFactory = context.getProperty(RECORD_WRITER).asControllerService(RecordSetWriterFactory.class);
+ final boolean encode = context.getProperty(MODE).getValue().equalsIgnoreCase(ProcessingMode.ENCODE.toString());
+ final RoutingStrategy routingStrategy = RoutingStrategy.valueOf(context.getProperty(ROUTING_STRATEGY).getValue());
+ final GeohashFormat format = GeohashFormat.valueOf(context.getProperty(GEOHASH_FORMAT).getValue());
+
+ FlowFile output = session.create(input);
+ FlowFile notMatched = routingStrategy == RoutingStrategy.SPLIT ? session.create(input) : null;
+
+ try (final InputStream is = session.read(input);
+ final RecordReader reader = readerFactory.createRecordReader(input, is, getLogger());
+ final OutputStream os = session.write(output);
+ final OutputStream osNotFound = routingStrategy == RoutingStrategy.SPLIT ? session.write(notMatched) : null) {
+
+ final RecordSetWriter writer = writerFactory.createWriter(getLogger(), writerFactory.getSchema(input.getAttributes(), reader.getSchema()), os, output);
+ final RecordSetWriter notMatchedWriter = routingStrategy == RoutingStrategy.SPLIT ? writerFactory.createWriter(getLogger(), reader.getSchema(), osNotFound, notMatched) : null;
+
+ Map<PropertyDescriptor, RecordPath> paths = new HashMap<>();
+ for (PropertyDescriptor descriptor : RECORD_PATH_PROPERTIES) {
+ String rawRecordPath = context.getProperty(descriptor).evaluateAttributeExpressions(input).getValue();
+ RecordPath compiled = cache.getCompiled(rawRecordPath);
+ paths.put(descriptor, compiled);
+ }
+
+ Record record;
+
+ writer.beginRecordSet();
+
+ if (notMatchedWriter != null) {
+ notMatchedWriter.beginRecordSet();
+ }
+
+ int level = context.getProperty(GEOHASH_LEVEL).evaluateAttributeExpressions(input).asInteger();
+ final String rawLatitudePath = context.getProperty(LATITUDE_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath latitudePath = cache.getCompiled(rawLatitudePath);
+ final String rawLongitudePath = context.getProperty(LONGITUDE_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath longitudePath = cache.getCompiled(rawLongitudePath);
+ final String rawGeohashPath = context.getProperty(GEOHASH_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath geohashPath = cache.getCompiled(rawGeohashPath);
+
+ while ((record = reader.nextRecord()) != null) {
+ boolean updated = false;
+
+ try {
+ if (encode) {
+ Object encodedGeohash = getEncodedGeohash(latitudePath, longitudePath, record, format, level);
+ updated = updateRecord(GEOHASH_RECORD_PATH, encodedGeohash, record, paths);
+ } else {
+ WGS84Point decodedPoint = getDecodedPointFromGeohash(geohashPath, record, format);
+ if (decodedPoint != null) {
+ updated = updateRecord(LATITUDE_RECORD_PATH, String.valueOf(decodedPoint.getLatitude()), record, paths)
+ && updateRecord(LONGITUDE_RECORD_PATH, String.valueOf(decodedPoint.getLongitude()), record, paths);
+ }
+ }
+ } catch (IllegalArgumentException e) {
+ //lat/lon/geohash values out of range or is not valid
+ getLogger().warn(e.getLocalizedMessage(), e);
+ }
+
+ routingStrategyExecutor.writeFlowFiles(record, writer, notMatchedWriter, updated);
+ }
+
+ final WriteResult writeResult = writer.finishRecordSet();
+ writer.close();
+ output = session.putAllAttributes(output, buildAttributes(writeResult.getRecordCount(), writer.getMimeType(), writeResult));
+
+ WriteResult notMatchedWriterResult;
+
+ if (notMatchedWriter != null) {
+ notMatchedWriterResult = notMatchedWriter.finishRecordSet();
+ notMatchedWriter.close();
+ if (notMatchedWriterResult.getRecordCount() > 0) {
+ notMatched = session.putAllAttributes(notMatched, buildAttributes(notMatchedWriterResult.getRecordCount(), writer.getMimeType(), notMatchedWriterResult));
+ }
+ }
+ } catch (IOException | SchemaNotFoundException | MalformedRecordException e) {
+ //cannot parse incoming data
+ getLogger().error("Cannot parse the incoming data", e.getLocalizedMessage(), e);
Review comment:
Mismatched argument count here: No need for the `e.getLocalizedMessage()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] mikayla-yang commented on pull request #5476: NIFI-9259: Adding GeohashRecord Processor
Posted by GitBox <gi...@apache.org>.
mikayla-yang commented on pull request #5476:
URL: https://github.com/apache/nifi/pull/5476#issuecomment-969502105
Thank you @exceptionfactory for the comments! I have made changes accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] markap14 commented on a change in pull request #5476: NIFI-9259: Adding GeohashRecord Processor
Posted by GitBox <gi...@apache.org>.
markap14 commented on a change in pull request #5476:
URL: https://github.com/apache/nifi/pull/5476#discussion_r737816742
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,367 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+ public static final String ENCODE_MODE = "Encode";
+ public static final String DECODE_MODE = "Decode";
+
+ public static final String BINARY_STRING = "BinaryString";
+ public static final String BASE32 = "Base32";
+ public static final String LONG_INT = "LongInt";
+
+ public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
+ .name("mode")
+ .displayName("Mode")
+ .description("Specifies whether to encode latitude/longitude to geohash or decode geohash to latitude/longitude")
+ .required(true)
+ .allowableValues(ENCODE_MODE, DECODE_MODE)
+ .defaultValue(ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor RECORD_READER = new PropertyDescriptor.Builder()
+ .name("record-reader")
+ .displayName("Record Reader")
+ .description("Specifies the record reader service to use for reading incoming data")
+ .required(true)
+ .identifiesControllerService(RecordReaderFactory.class)
+ .build();
+
+ public static final PropertyDescriptor RECORD_WRITER = new PropertyDescriptor.Builder()
+ .name("record-writer")
+ .displayName("Record Writer")
+ .description("Specifies the record writer service to use for writing data")
+ .required(true)
+ .identifiesControllerService(RecordSetWriterFactory.class)
+ .build();
+
+ public static final PropertyDescriptor LATITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("latitude-record-path")
+ .displayName("Latitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the latitude value;"
+ + "in the decode mode, this property specifies the record path to put the latitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor LONGITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("longitude-record-path")
+ .displayName("Longitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the longitude value;"
Review comment:
space after semicolon
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,367 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+ public static final String ENCODE_MODE = "Encode";
+ public static final String DECODE_MODE = "Decode";
+
+ public static final String BINARY_STRING = "BinaryString";
+ public static final String BASE32 = "Base32";
+ public static final String LONG_INT = "LongInt";
+
+ public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
+ .name("mode")
+ .displayName("Mode")
+ .description("Specifies whether to encode latitude/longitude to geohash or decode geohash to latitude/longitude")
+ .required(true)
+ .allowableValues(ENCODE_MODE, DECODE_MODE)
+ .defaultValue(ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor RECORD_READER = new PropertyDescriptor.Builder()
+ .name("record-reader")
+ .displayName("Record Reader")
+ .description("Specifies the record reader service to use for reading incoming data")
+ .required(true)
+ .identifiesControllerService(RecordReaderFactory.class)
+ .build();
+
+ public static final PropertyDescriptor RECORD_WRITER = new PropertyDescriptor.Builder()
+ .name("record-writer")
+ .displayName("Record Writer")
+ .description("Specifies the record writer service to use for writing data")
+ .required(true)
+ .identifiesControllerService(RecordSetWriterFactory.class)
+ .build();
+
+ public static final PropertyDescriptor LATITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("latitude-record-path")
+ .displayName("Latitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the latitude value;"
+ + "in the decode mode, this property specifies the record path to put the latitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor LONGITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("longitude-record-path")
+ .displayName("Longitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the longitude value;"
+ + "in the decode mode, this property specifies the record path to put the longitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
Review comment:
Should use `new RecordPathValidator()` here
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,367 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+ public static final String ENCODE_MODE = "Encode";
+ public static final String DECODE_MODE = "Decode";
+
+ public static final String BINARY_STRING = "BinaryString";
+ public static final String BASE32 = "Base32";
+ public static final String LONG_INT = "LongInt";
+
+ public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
+ .name("mode")
+ .displayName("Mode")
+ .description("Specifies whether to encode latitude/longitude to geohash or decode geohash to latitude/longitude")
+ .required(true)
+ .allowableValues(ENCODE_MODE, DECODE_MODE)
+ .defaultValue(ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor RECORD_READER = new PropertyDescriptor.Builder()
+ .name("record-reader")
+ .displayName("Record Reader")
+ .description("Specifies the record reader service to use for reading incoming data")
+ .required(true)
+ .identifiesControllerService(RecordReaderFactory.class)
+ .build();
+
+ public static final PropertyDescriptor RECORD_WRITER = new PropertyDescriptor.Builder()
+ .name("record-writer")
+ .displayName("Record Writer")
+ .description("Specifies the record writer service to use for writing data")
+ .required(true)
+ .identifiesControllerService(RecordSetWriterFactory.class)
+ .build();
+
+ public static final PropertyDescriptor LATITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("latitude-record-path")
+ .displayName("Latitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the latitude value;"
+ + "in the decode mode, this property specifies the record path to put the latitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor LONGITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("longitude-record-path")
+ .displayName("Longitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the longitude value;"
+ + "in the decode mode, this property specifies the record path to put the longitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_FORMAT = new PropertyDescriptor.Builder()
+ .name("geohash-format")
+ .displayName("Geohash Format")
+ .description("In the encode mode, this property specifies the desired format for encoding geohash;"
+ + "in the decode mode, this property specifies the format of geohash provided")
+ .required(true)
+ .allowableValues(BASE32, BINARY_STRING, LONG_INT)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_LEVEL = new PropertyDescriptor.Builder()
+ .name("geohash-level")
+ .displayName("Geohash Level")
+ .description("The integer precision level(1-12) desired for encoding geohash")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(MODE, ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("geohash-record-path")
+ .displayName("Geohash Record Path")
+ .description("In the encode mode, this property specifies the record path to put the geohash value;"
+ + "in the decode mode, this property specifies the record path to retrieve the geohash value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Records that are successfully encoded or decoded will be routed to success")
+ .build();
+ public static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Records that cannot be encoded or decoded will be routed to failure")
+ .build();
+
+ public static final List<PropertyDescriptor> RECORD_PATH_PROPERTIES = Collections.unmodifiableList(Arrays.asList(
+ LATITUDE_RECORD_PATH,LONGITUDE_RECORD_PATH,GEOHASH_RECORD_PATH
+ ));
+
+ private List<PropertyDescriptor> descriptors;
+
+ private Set<Relationship> relationships;
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ descriptors = new ArrayList<>();
+ descriptors.add(MODE);
+ descriptors.add(RECORD_READER);
+ descriptors.add(RECORD_WRITER);
+ descriptors.add(LATITUDE_RECORD_PATH);
+ descriptors.add(LONGITUDE_RECORD_PATH);
+ descriptors.add(GEOHASH_FORMAT);
+ descriptors.add(GEOHASH_LEVEL);
+ descriptors.add(GEOHASH_RECORD_PATH);
Review comment:
Would recommend adding GEOHASH_RECORD_PATH immediately following LONGITUDE_RECORD_PATH so that the RecordPath properties are provided together
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,367 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+ public static final String ENCODE_MODE = "Encode";
+ public static final String DECODE_MODE = "Decode";
+
+ public static final String BINARY_STRING = "BinaryString";
+ public static final String BASE32 = "Base32";
+ public static final String LONG_INT = "LongInt";
+
+ public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
+ .name("mode")
+ .displayName("Mode")
+ .description("Specifies whether to encode latitude/longitude to geohash or decode geohash to latitude/longitude")
+ .required(true)
+ .allowableValues(ENCODE_MODE, DECODE_MODE)
+ .defaultValue(ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor RECORD_READER = new PropertyDescriptor.Builder()
+ .name("record-reader")
+ .displayName("Record Reader")
+ .description("Specifies the record reader service to use for reading incoming data")
+ .required(true)
+ .identifiesControllerService(RecordReaderFactory.class)
+ .build();
+
+ public static final PropertyDescriptor RECORD_WRITER = new PropertyDescriptor.Builder()
+ .name("record-writer")
+ .displayName("Record Writer")
+ .description("Specifies the record writer service to use for writing data")
+ .required(true)
+ .identifiesControllerService(RecordSetWriterFactory.class)
+ .build();
+
+ public static final PropertyDescriptor LATITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("latitude-record-path")
+ .displayName("Latitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the latitude value;"
+ + "in the decode mode, this property specifies the record path to put the latitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor LONGITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("longitude-record-path")
+ .displayName("Longitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the longitude value;"
+ + "in the decode mode, this property specifies the record path to put the longitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_FORMAT = new PropertyDescriptor.Builder()
+ .name("geohash-format")
+ .displayName("Geohash Format")
+ .description("In the encode mode, this property specifies the desired format for encoding geohash;"
+ + "in the decode mode, this property specifies the format of geohash provided")
+ .required(true)
+ .allowableValues(BASE32, BINARY_STRING, LONG_INT)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_LEVEL = new PropertyDescriptor.Builder()
+ .name("geohash-level")
+ .displayName("Geohash Level")
+ .description("The integer precision level(1-12) desired for encoding geohash")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(MODE, ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("geohash-record-path")
+ .displayName("Geohash Record Path")
+ .description("In the encode mode, this property specifies the record path to put the geohash value;"
+ + "in the decode mode, this property specifies the record path to retrieve the geohash value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Records that are successfully encoded or decoded will be routed to success")
+ .build();
+ public static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Records that cannot be encoded or decoded will be routed to failure")
+ .build();
+
+ public static final List<PropertyDescriptor> RECORD_PATH_PROPERTIES = Collections.unmodifiableList(Arrays.asList(
+ LATITUDE_RECORD_PATH,LONGITUDE_RECORD_PATH,GEOHASH_RECORD_PATH
+ ));
+
+ private List<PropertyDescriptor> descriptors;
+
+ private Set<Relationship> relationships;
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ descriptors = new ArrayList<>();
+ descriptors.add(MODE);
+ descriptors.add(RECORD_READER);
+ descriptors.add(RECORD_WRITER);
+ descriptors.add(LATITUDE_RECORD_PATH);
+ descriptors.add(LONGITUDE_RECORD_PATH);
+ descriptors.add(GEOHASH_FORMAT);
+ descriptors.add(GEOHASH_LEVEL);
+ descriptors.add(GEOHASH_RECORD_PATH);
+ descriptors = Collections.unmodifiableList(descriptors);
+
+ relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ relationships.add(REL_FAILURE);
+ relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) {
+ FlowFile input = session.get();
+ if ( input == null ) {
+ return;
+ }
+
+ final RecordReaderFactory readerFactory = context.getProperty(RECORD_READER).asControllerService(RecordReaderFactory.class);
+ final RecordSetWriterFactory writerFactory = context.getProperty(RECORD_WRITER).asControllerService(RecordSetWriterFactory.class);
+ final boolean encode = context.getProperty(MODE).getValue().equalsIgnoreCase(ENCODE_MODE);
+ final String format = context.getProperty(GEOHASH_FORMAT).getValue();
+
+ FlowFile output = session.create(input);
+ try (InputStream is = session.read(input);
+ OutputStream os = session.write(output)) {
+ RecordPathCache cache = new RecordPathCache(RECORD_PATH_PROPERTIES.size() + 1);
+ Map<PropertyDescriptor, RecordPath> paths = new HashMap<>();
+
+ for (PropertyDescriptor descriptor : RECORD_PATH_PROPERTIES) {
+ String rawRecordPath = context.getProperty(descriptor).evaluateAttributeExpressions(input).getValue();
+ RecordPath compiled = cache.getCompiled(rawRecordPath);
+ paths.put(descriptor, compiled);
+ }
+
+ RecordReader reader = readerFactory.createRecordReader(input, is, getLogger());
+ RecordSchema schema = writerFactory.getSchema(input.getAttributes(), reader.getSchema());
+ RecordSetWriter writer = writerFactory.createWriter(getLogger(), schema, os, output);
+
+ Record record;
+ Relationship targetRelationship = REL_FAILURE;
+ writer.beginRecordSet();
+
+ int recordsCount = 0;
+ boolean updated = false;
+ while ((record = reader.nextRecord()) != null) {
+ if(encode) {
+ final int level = Integer.parseInt(context.getProperty(GEOHASH_LEVEL).getValue());
+ final String rawLatitudePath = context.getProperty(LATITUDE_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath latitudePath = cache.getCompiled(rawLatitudePath);
+ final String rawLongitudePath = context.getProperty(LONGITUDE_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath longitudePath = cache.getCompiled(rawLongitudePath);
+
+ Object encodedGeohash = getEncodedGeohash(latitudePath, longitudePath, record, format, level);
+
+ updated = updateRecord(GEOHASH_RECORD_PATH, encodedGeohash, record, paths);
+ }else {
+ final String rawgeohashPath = context.getProperty(GEOHASH_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath geohashPath = cache.getCompiled(rawgeohashPath);
+
+ WGS84Point decodedPoint = getDecodedPointFromGeohash(geohashPath, record, format);
+
+ if(decodedPoint != null) {
+ updated = updateRecord(LATITUDE_RECORD_PATH, String.valueOf(decodedPoint.getLatitude()), record, paths)
+ && updateRecord(LONGITUDE_RECORD_PATH, String.valueOf(decodedPoint.getLongitude()), record, paths);
+ }
+ }
+
+ if (updated) {
+ targetRelationship = REL_SUCCESS;
+ }
+
+ writer.write(record);
+ recordsCount++;
+
+ }
+ writer.finishRecordSet();
+ writer.close();
+
+ is.close();
+ os.close();
+
+ output = session.putAllAttributes(output, buildAttributes(recordsCount, writer.getMimeType()));
+
+ session.transfer(output, targetRelationship);
+ session.remove(input);
+
+ session.getProvenanceReporter().modifyContent(output);
+ } catch (Exception ex) {
+ getLogger().error("Failed to {}; will route to failure", new Object[]{encode ? "encode" : "decode" ,ex});
Review comment:
No need for the explicit array constructor -- can just give `encode ? "encode" : "decode", ex` as the argument list.
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,367 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+ public static final String ENCODE_MODE = "Encode";
+ public static final String DECODE_MODE = "Decode";
+
+ public static final String BINARY_STRING = "BinaryString";
+ public static final String BASE32 = "Base32";
+ public static final String LONG_INT = "LongInt";
+
+ public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
+ .name("mode")
+ .displayName("Mode")
+ .description("Specifies whether to encode latitude/longitude to geohash or decode geohash to latitude/longitude")
+ .required(true)
+ .allowableValues(ENCODE_MODE, DECODE_MODE)
+ .defaultValue(ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor RECORD_READER = new PropertyDescriptor.Builder()
+ .name("record-reader")
+ .displayName("Record Reader")
+ .description("Specifies the record reader service to use for reading incoming data")
+ .required(true)
+ .identifiesControllerService(RecordReaderFactory.class)
+ .build();
+
+ public static final PropertyDescriptor RECORD_WRITER = new PropertyDescriptor.Builder()
+ .name("record-writer")
+ .displayName("Record Writer")
+ .description("Specifies the record writer service to use for writing data")
+ .required(true)
+ .identifiesControllerService(RecordSetWriterFactory.class)
+ .build();
+
+ public static final PropertyDescriptor LATITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("latitude-record-path")
+ .displayName("Latitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the latitude value;"
+ + "in the decode mode, this property specifies the record path to put the latitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor LONGITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("longitude-record-path")
+ .displayName("Longitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the longitude value;"
+ + "in the decode mode, this property specifies the record path to put the longitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_FORMAT = new PropertyDescriptor.Builder()
+ .name("geohash-format")
+ .displayName("Geohash Format")
+ .description("In the encode mode, this property specifies the desired format for encoding geohash;"
+ + "in the decode mode, this property specifies the format of geohash provided")
+ .required(true)
+ .allowableValues(BASE32, BINARY_STRING, LONG_INT)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_LEVEL = new PropertyDescriptor.Builder()
+ .name("geohash-level")
+ .displayName("Geohash Level")
+ .description("The integer precision level(1-12) desired for encoding geohash")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(MODE, ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("geohash-record-path")
+ .displayName("Geohash Record Path")
+ .description("In the encode mode, this property specifies the record path to put the geohash value;"
+ + "in the decode mode, this property specifies the record path to retrieve the geohash value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Records that are successfully encoded or decoded will be routed to success")
+ .build();
+ public static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Records that cannot be encoded or decoded will be routed to failure")
+ .build();
+
+ public static final List<PropertyDescriptor> RECORD_PATH_PROPERTIES = Collections.unmodifiableList(Arrays.asList(
+ LATITUDE_RECORD_PATH,LONGITUDE_RECORD_PATH,GEOHASH_RECORD_PATH
+ ));
+
+ private List<PropertyDescriptor> descriptors;
+
+ private Set<Relationship> relationships;
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ descriptors = new ArrayList<>();
+ descriptors.add(MODE);
+ descriptors.add(RECORD_READER);
+ descriptors.add(RECORD_WRITER);
+ descriptors.add(LATITUDE_RECORD_PATH);
+ descriptors.add(LONGITUDE_RECORD_PATH);
+ descriptors.add(GEOHASH_FORMAT);
+ descriptors.add(GEOHASH_LEVEL);
+ descriptors.add(GEOHASH_RECORD_PATH);
+ descriptors = Collections.unmodifiableList(descriptors);
+
+ relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ relationships.add(REL_FAILURE);
+ relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) {
+ FlowFile input = session.get();
+ if ( input == null ) {
+ return;
+ }
+
+ final RecordReaderFactory readerFactory = context.getProperty(RECORD_READER).asControllerService(RecordReaderFactory.class);
+ final RecordSetWriterFactory writerFactory = context.getProperty(RECORD_WRITER).asControllerService(RecordSetWriterFactory.class);
+ final boolean encode = context.getProperty(MODE).getValue().equalsIgnoreCase(ENCODE_MODE);
+ final String format = context.getProperty(GEOHASH_FORMAT).getValue();
+
+ FlowFile output = session.create(input);
+ try (InputStream is = session.read(input);
+ OutputStream os = session.write(output)) {
+ RecordPathCache cache = new RecordPathCache(RECORD_PATH_PROPERTIES.size() + 1);
+ Map<PropertyDescriptor, RecordPath> paths = new HashMap<>();
+
+ for (PropertyDescriptor descriptor : RECORD_PATH_PROPERTIES) {
+ String rawRecordPath = context.getProperty(descriptor).evaluateAttributeExpressions(input).getValue();
+ RecordPath compiled = cache.getCompiled(rawRecordPath);
+ paths.put(descriptor, compiled);
+ }
+
+ RecordReader reader = readerFactory.createRecordReader(input, is, getLogger());
+ RecordSchema schema = writerFactory.getSchema(input.getAttributes(), reader.getSchema());
+ RecordSetWriter writer = writerFactory.createWriter(getLogger(), schema, os, output);
+
+ Record record;
+ Relationship targetRelationship = REL_FAILURE;
+ writer.beginRecordSet();
+
+ int recordsCount = 0;
+ boolean updated = false;
+ while ((record = reader.nextRecord()) != null) {
+ if(encode) {
+ final int level = Integer.parseInt(context.getProperty(GEOHASH_LEVEL).getValue());
+ final String rawLatitudePath = context.getProperty(LATITUDE_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath latitudePath = cache.getCompiled(rawLatitudePath);
+ final String rawLongitudePath = context.getProperty(LONGITUDE_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath longitudePath = cache.getCompiled(rawLongitudePath);
+
+ Object encodedGeohash = getEncodedGeohash(latitudePath, longitudePath, record, format, level);
+
+ updated = updateRecord(GEOHASH_RECORD_PATH, encodedGeohash, record, paths);
+ }else {
+ final String rawgeohashPath = context.getProperty(GEOHASH_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath geohashPath = cache.getCompiled(rawgeohashPath);
+
+ WGS84Point decodedPoint = getDecodedPointFromGeohash(geohashPath, record, format);
+
+ if(decodedPoint != null) {
+ updated = updateRecord(LATITUDE_RECORD_PATH, String.valueOf(decodedPoint.getLatitude()), record, paths)
+ && updateRecord(LONGITUDE_RECORD_PATH, String.valueOf(decodedPoint.getLongitude()), record, paths);
+ }
+ }
+
+ if (updated) {
+ targetRelationship = REL_SUCCESS;
+ }
+
+ writer.write(record);
+ recordsCount++;
+
+ }
+ writer.finishRecordSet();
+ writer.close();
+
+ is.close();
+ os.close();
+
+ output = session.putAllAttributes(output, buildAttributes(recordsCount, writer.getMimeType()));
+
+ session.transfer(output, targetRelationship);
+ session.remove(input);
+
+ session.getProvenanceReporter().modifyContent(output);
+ } catch (Exception ex) {
+ getLogger().error("Failed to {}; will route to failure", new Object[]{encode ? "encode" : "decode" ,ex});
+ session.rollback();
+ context.yield();
+ }
+ }
+
+ private Object getEncodedGeohash(RecordPath latitudePath, RecordPath longitudePath, Record record, String format, int level) {
+ RecordPathResult latitudeResult = latitudePath.evaluate(record);
+ RecordPathResult longitudeResult = longitudePath.evaluate(record);
+ Optional<FieldValue> latitudeField = latitudeResult.getSelectedFields().findFirst();
+ Optional<FieldValue> longitudeField = longitudeResult.getSelectedFields().findFirst();
+
+ if (latitudeField.isPresent() && longitudeField.isPresent()) {
+ FieldValue latitudeValue = latitudeField.get();
+ FieldValue longitudeValue = longitudeField.get();
+ Object latitudeVal = latitudeValue.getValue();
+ Object longitudeVal = longitudeValue.getValue();
+
+ if (latitudeVal == null || longitudeVal == null) {
+ return null;
+ }
+
+ double realLatValue = Double.parseDouble(latitudeVal.toString());
+ double realLongValue = Double.parseDouble(longitudeVal.toString());
+ GeoHash gh = GeoHash.withCharacterPrecision(realLatValue, realLongValue, level);
+
+ switch (format) {
+ case BINARY_STRING:
+ return gh.toBinaryString();
+ case LONG_INT:
+ return gh.longValue();
+ default:
+ return gh.toBase32();
+ }
+ } else {
+ return null;
+ }
+ }
+
+ private WGS84Point getDecodedPointFromGeohash(RecordPath geohashPath, Record record, String format) {
+ RecordPathResult geohashResult = geohashPath.evaluate(record);
+ Optional<FieldValue> geohashField = geohashResult.getSelectedFields().findFirst();
+ if (geohashField.isPresent()) {
+ FieldValue geohashFieldValue = geohashField.get();
+ Object geohashVal = geohashFieldValue.getValue();
+ if(geohashVal == null) {
+ return null;
+ }
+
+ String geohashString = geohashVal.toString();
+ GeoHash decodedHash;
+
+ switch (format) {
+ case BINARY_STRING:
+ decodedHash = GeoHash.fromBinaryString(geohashString);
+ break;
+ case LONG_INT:
+ String binaryString = Long.toBinaryString(Long.parseLong(geohashString));
+ decodedHash = GeoHash.fromBinaryString(binaryString);
+ break;
+ default:
+ decodedHash = GeoHash.fromGeohashString(geohashString);
+ }
+
+ return decodedHash.getBoundingBoxCenter();
+
+ }else {
+ return null;
+ }
+ }
+
+ private boolean updateRecord(PropertyDescriptor descriptor, Object newValue, Record record, Map<PropertyDescriptor, RecordPath> cached) {
+ if (!cached.containsKey(descriptor) || newValue == null) {
+ return false;
+ }
+ RecordPath path = cached.get(descriptor);
+ RecordPathResult result = path.evaluate(record);
+ FieldValue fieldValue = result.getSelectedFields().findFirst().get();
+
+ if (fieldValue.getParent().get().getValue() == null) {
Review comment:
If `fieldValue` were to point to the root record (i.e., user configured it as `/`) then `getParent()` would return an empty optional and the call here to `get()` will throw `NoSuchElementException`
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,367 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+ public static final String ENCODE_MODE = "Encode";
+ public static final String DECODE_MODE = "Decode";
+
+ public static final String BINARY_STRING = "BinaryString";
+ public static final String BASE32 = "Base32";
+ public static final String LONG_INT = "LongInt";
+
+ public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
+ .name("mode")
+ .displayName("Mode")
+ .description("Specifies whether to encode latitude/longitude to geohash or decode geohash to latitude/longitude")
+ .required(true)
+ .allowableValues(ENCODE_MODE, DECODE_MODE)
+ .defaultValue(ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor RECORD_READER = new PropertyDescriptor.Builder()
+ .name("record-reader")
+ .displayName("Record Reader")
+ .description("Specifies the record reader service to use for reading incoming data")
+ .required(true)
+ .identifiesControllerService(RecordReaderFactory.class)
+ .build();
+
+ public static final PropertyDescriptor RECORD_WRITER = new PropertyDescriptor.Builder()
+ .name("record-writer")
+ .displayName("Record Writer")
+ .description("Specifies the record writer service to use for writing data")
+ .required(true)
+ .identifiesControllerService(RecordSetWriterFactory.class)
+ .build();
+
+ public static final PropertyDescriptor LATITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("latitude-record-path")
+ .displayName("Latitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the latitude value;"
+ + "in the decode mode, this property specifies the record path to put the latitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
Review comment:
Should use `new RecordPathValidator()` here
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,367 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+ public static final String ENCODE_MODE = "Encode";
+ public static final String DECODE_MODE = "Decode";
+
+ public static final String BINARY_STRING = "BinaryString";
+ public static final String BASE32 = "Base32";
+ public static final String LONG_INT = "LongInt";
+
+ public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
+ .name("mode")
+ .displayName("Mode")
+ .description("Specifies whether to encode latitude/longitude to geohash or decode geohash to latitude/longitude")
+ .required(true)
+ .allowableValues(ENCODE_MODE, DECODE_MODE)
+ .defaultValue(ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor RECORD_READER = new PropertyDescriptor.Builder()
+ .name("record-reader")
+ .displayName("Record Reader")
+ .description("Specifies the record reader service to use for reading incoming data")
+ .required(true)
+ .identifiesControllerService(RecordReaderFactory.class)
+ .build();
+
+ public static final PropertyDescriptor RECORD_WRITER = new PropertyDescriptor.Builder()
+ .name("record-writer")
+ .displayName("Record Writer")
+ .description("Specifies the record writer service to use for writing data")
+ .required(true)
+ .identifiesControllerService(RecordSetWriterFactory.class)
+ .build();
+
+ public static final PropertyDescriptor LATITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("latitude-record-path")
+ .displayName("Latitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the latitude value;"
Review comment:
should add a space after the semicolon :)
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,367 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+ public static final String ENCODE_MODE = "Encode";
+ public static final String DECODE_MODE = "Decode";
+
+ public static final String BINARY_STRING = "BinaryString";
+ public static final String BASE32 = "Base32";
+ public static final String LONG_INT = "LongInt";
+
+ public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
+ .name("mode")
+ .displayName("Mode")
+ .description("Specifies whether to encode latitude/longitude to geohash or decode geohash to latitude/longitude")
+ .required(true)
+ .allowableValues(ENCODE_MODE, DECODE_MODE)
+ .defaultValue(ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor RECORD_READER = new PropertyDescriptor.Builder()
+ .name("record-reader")
+ .displayName("Record Reader")
+ .description("Specifies the record reader service to use for reading incoming data")
+ .required(true)
+ .identifiesControllerService(RecordReaderFactory.class)
+ .build();
+
+ public static final PropertyDescriptor RECORD_WRITER = new PropertyDescriptor.Builder()
+ .name("record-writer")
+ .displayName("Record Writer")
+ .description("Specifies the record writer service to use for writing data")
+ .required(true)
+ .identifiesControllerService(RecordSetWriterFactory.class)
+ .build();
+
+ public static final PropertyDescriptor LATITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("latitude-record-path")
+ .displayName("Latitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the latitude value;"
+ + "in the decode mode, this property specifies the record path to put the latitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor LONGITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("longitude-record-path")
+ .displayName("Longitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the longitude value;"
+ + "in the decode mode, this property specifies the record path to put the longitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_FORMAT = new PropertyDescriptor.Builder()
+ .name("geohash-format")
+ .displayName("Geohash Format")
+ .description("In the encode mode, this property specifies the desired format for encoding geohash;"
+ + "in the decode mode, this property specifies the format of geohash provided")
+ .required(true)
+ .allowableValues(BASE32, BINARY_STRING, LONG_INT)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_LEVEL = new PropertyDescriptor.Builder()
+ .name("geohash-level")
+ .displayName("Geohash Level")
+ .description("The integer precision level(1-12) desired for encoding geohash")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(MODE, ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("geohash-record-path")
+ .displayName("Geohash Record Path")
+ .description("In the encode mode, this property specifies the record path to put the geohash value;"
+ + "in the decode mode, this property specifies the record path to retrieve the geohash value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Records that are successfully encoded or decoded will be routed to success")
+ .build();
+ public static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Records that cannot be encoded or decoded will be routed to failure")
+ .build();
+
+ public static final List<PropertyDescriptor> RECORD_PATH_PROPERTIES = Collections.unmodifiableList(Arrays.asList(
+ LATITUDE_RECORD_PATH,LONGITUDE_RECORD_PATH,GEOHASH_RECORD_PATH
+ ));
+
+ private List<PropertyDescriptor> descriptors;
+
+ private Set<Relationship> relationships;
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ descriptors = new ArrayList<>();
+ descriptors.add(MODE);
+ descriptors.add(RECORD_READER);
+ descriptors.add(RECORD_WRITER);
+ descriptors.add(LATITUDE_RECORD_PATH);
+ descriptors.add(LONGITUDE_RECORD_PATH);
+ descriptors.add(GEOHASH_FORMAT);
+ descriptors.add(GEOHASH_LEVEL);
+ descriptors.add(GEOHASH_RECORD_PATH);
+ descriptors = Collections.unmodifiableList(descriptors);
+
+ relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ relationships.add(REL_FAILURE);
+ relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) {
+ FlowFile input = session.get();
+ if ( input == null ) {
+ return;
+ }
+
+ final RecordReaderFactory readerFactory = context.getProperty(RECORD_READER).asControllerService(RecordReaderFactory.class);
+ final RecordSetWriterFactory writerFactory = context.getProperty(RECORD_WRITER).asControllerService(RecordSetWriterFactory.class);
+ final boolean encode = context.getProperty(MODE).getValue().equalsIgnoreCase(ENCODE_MODE);
+ final String format = context.getProperty(GEOHASH_FORMAT).getValue();
+
+ FlowFile output = session.create(input);
+ try (InputStream is = session.read(input);
+ OutputStream os = session.write(output)) {
+ RecordPathCache cache = new RecordPathCache(RECORD_PATH_PROPERTIES.size() + 1);
Review comment:
Should create the cache as a member variable, not an inline variable. This way the compilation of the RecordPath is cached across multiple invocations. Not sure that it's buying us anything as-is.
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,367 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+ public static final String ENCODE_MODE = "Encode";
+ public static final String DECODE_MODE = "Decode";
+
+ public static final String BINARY_STRING = "BinaryString";
+ public static final String BASE32 = "Base32";
+ public static final String LONG_INT = "LongInt";
+
+ public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
+ .name("mode")
+ .displayName("Mode")
+ .description("Specifies whether to encode latitude/longitude to geohash or decode geohash to latitude/longitude")
+ .required(true)
+ .allowableValues(ENCODE_MODE, DECODE_MODE)
+ .defaultValue(ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor RECORD_READER = new PropertyDescriptor.Builder()
+ .name("record-reader")
+ .displayName("Record Reader")
+ .description("Specifies the record reader service to use for reading incoming data")
+ .required(true)
+ .identifiesControllerService(RecordReaderFactory.class)
+ .build();
+
+ public static final PropertyDescriptor RECORD_WRITER = new PropertyDescriptor.Builder()
+ .name("record-writer")
+ .displayName("Record Writer")
+ .description("Specifies the record writer service to use for writing data")
+ .required(true)
+ .identifiesControllerService(RecordSetWriterFactory.class)
+ .build();
+
+ public static final PropertyDescriptor LATITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("latitude-record-path")
+ .displayName("Latitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the latitude value;"
+ + "in the decode mode, this property specifies the record path to put the latitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor LONGITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("longitude-record-path")
+ .displayName("Longitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the longitude value;"
+ + "in the decode mode, this property specifies the record path to put the longitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_FORMAT = new PropertyDescriptor.Builder()
+ .name("geohash-format")
+ .displayName("Geohash Format")
+ .description("In the encode mode, this property specifies the desired format for encoding geohash;"
+ + "in the decode mode, this property specifies the format of geohash provided")
+ .required(true)
+ .allowableValues(BASE32, BINARY_STRING, LONG_INT)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_LEVEL = new PropertyDescriptor.Builder()
+ .name("geohash-level")
+ .displayName("Geohash Level")
+ .description("The integer precision level(1-12) desired for encoding geohash")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(MODE, ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("geohash-record-path")
+ .displayName("Geohash Record Path")
+ .description("In the encode mode, this property specifies the record path to put the geohash value;"
Review comment:
Missing space after semicolon
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,367 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+ public static final String ENCODE_MODE = "Encode";
+ public static final String DECODE_MODE = "Decode";
+
+ public static final String BINARY_STRING = "BinaryString";
+ public static final String BASE32 = "Base32";
+ public static final String LONG_INT = "LongInt";
+
+ public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
+ .name("mode")
+ .displayName("Mode")
+ .description("Specifies whether to encode latitude/longitude to geohash or decode geohash to latitude/longitude")
+ .required(true)
+ .allowableValues(ENCODE_MODE, DECODE_MODE)
+ .defaultValue(ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor RECORD_READER = new PropertyDescriptor.Builder()
+ .name("record-reader")
+ .displayName("Record Reader")
+ .description("Specifies the record reader service to use for reading incoming data")
+ .required(true)
+ .identifiesControllerService(RecordReaderFactory.class)
+ .build();
+
+ public static final PropertyDescriptor RECORD_WRITER = new PropertyDescriptor.Builder()
+ .name("record-writer")
+ .displayName("Record Writer")
+ .description("Specifies the record writer service to use for writing data")
+ .required(true)
+ .identifiesControllerService(RecordSetWriterFactory.class)
+ .build();
+
+ public static final PropertyDescriptor LATITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("latitude-record-path")
+ .displayName("Latitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the latitude value;"
+ + "in the decode mode, this property specifies the record path to put the latitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor LONGITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("longitude-record-path")
+ .displayName("Longitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the longitude value;"
+ + "in the decode mode, this property specifies the record path to put the longitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_FORMAT = new PropertyDescriptor.Builder()
+ .name("geohash-format")
+ .displayName("Geohash Format")
+ .description("In the encode mode, this property specifies the desired format for encoding geohash;"
+ + "in the decode mode, this property specifies the format of geohash provided")
+ .required(true)
+ .allowableValues(BASE32, BINARY_STRING, LONG_INT)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
Review comment:
No need for a validator when using `allowableValues`
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,367 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+ public static final String ENCODE_MODE = "Encode";
+ public static final String DECODE_MODE = "Decode";
+
+ public static final String BINARY_STRING = "BinaryString";
+ public static final String BASE32 = "Base32";
+ public static final String LONG_INT = "LongInt";
+
+ public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
+ .name("mode")
+ .displayName("Mode")
+ .description("Specifies whether to encode latitude/longitude to geohash or decode geohash to latitude/longitude")
+ .required(true)
+ .allowableValues(ENCODE_MODE, DECODE_MODE)
+ .defaultValue(ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor RECORD_READER = new PropertyDescriptor.Builder()
+ .name("record-reader")
+ .displayName("Record Reader")
+ .description("Specifies the record reader service to use for reading incoming data")
+ .required(true)
+ .identifiesControllerService(RecordReaderFactory.class)
+ .build();
+
+ public static final PropertyDescriptor RECORD_WRITER = new PropertyDescriptor.Builder()
+ .name("record-writer")
+ .displayName("Record Writer")
+ .description("Specifies the record writer service to use for writing data")
+ .required(true)
+ .identifiesControllerService(RecordSetWriterFactory.class)
+ .build();
+
+ public static final PropertyDescriptor LATITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("latitude-record-path")
+ .displayName("Latitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the latitude value;"
+ + "in the decode mode, this property specifies the record path to put the latitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor LONGITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("longitude-record-path")
+ .displayName("Longitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the longitude value;"
+ + "in the decode mode, this property specifies the record path to put the longitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_FORMAT = new PropertyDescriptor.Builder()
+ .name("geohash-format")
+ .displayName("Geohash Format")
+ .description("In the encode mode, this property specifies the desired format for encoding geohash;"
+ + "in the decode mode, this property specifies the format of geohash provided")
+ .required(true)
+ .allowableValues(BASE32, BINARY_STRING, LONG_INT)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_LEVEL = new PropertyDescriptor.Builder()
+ .name("geohash-level")
+ .displayName("Geohash Level")
+ .description("The integer precision level(1-12) desired for encoding geohash")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(MODE, ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("geohash-record-path")
+ .displayName("Geohash Record Path")
+ .description("In the encode mode, this property specifies the record path to put the geohash value;"
+ + "in the decode mode, this property specifies the record path to retrieve the geohash value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
Review comment:
`new RecordPathValidator()`
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,367 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+ public static final String ENCODE_MODE = "Encode";
+ public static final String DECODE_MODE = "Decode";
+
+ public static final String BINARY_STRING = "BinaryString";
+ public static final String BASE32 = "Base32";
+ public static final String LONG_INT = "LongInt";
+
+ public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
+ .name("mode")
+ .displayName("Mode")
+ .description("Specifies whether to encode latitude/longitude to geohash or decode geohash to latitude/longitude")
+ .required(true)
+ .allowableValues(ENCODE_MODE, DECODE_MODE)
+ .defaultValue(ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor RECORD_READER = new PropertyDescriptor.Builder()
+ .name("record-reader")
+ .displayName("Record Reader")
+ .description("Specifies the record reader service to use for reading incoming data")
+ .required(true)
+ .identifiesControllerService(RecordReaderFactory.class)
+ .build();
+
+ public static final PropertyDescriptor RECORD_WRITER = new PropertyDescriptor.Builder()
+ .name("record-writer")
+ .displayName("Record Writer")
+ .description("Specifies the record writer service to use for writing data")
+ .required(true)
+ .identifiesControllerService(RecordSetWriterFactory.class)
+ .build();
+
+ public static final PropertyDescriptor LATITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("latitude-record-path")
+ .displayName("Latitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the latitude value;"
+ + "in the decode mode, this property specifies the record path to put the latitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor LONGITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("longitude-record-path")
+ .displayName("Longitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the longitude value;"
+ + "in the decode mode, this property specifies the record path to put the longitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_FORMAT = new PropertyDescriptor.Builder()
+ .name("geohash-format")
+ .displayName("Geohash Format")
+ .description("In the encode mode, this property specifies the desired format for encoding geohash;"
+ + "in the decode mode, this property specifies the format of geohash provided")
+ .required(true)
+ .allowableValues(BASE32, BINARY_STRING, LONG_INT)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_LEVEL = new PropertyDescriptor.Builder()
+ .name("geohash-level")
+ .displayName("Geohash Level")
+ .description("The integer precision level(1-12) desired for encoding geohash")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
Review comment:
Should either use `StandardValidators.createLongValidator(1, 12, true)` or use `.allowableValues( "1", "2", "3", "4", "5", "6", "7", "8", "9", "10", "11", "12")` -- in this case I don't have a strong preference for which one is better. Might also make sense to allow for Expression Language (EL) to be used here, in which case the `createLongValidator` is probably better because there's no way to use EL when using allowable values
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,367 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+ public static final String ENCODE_MODE = "Encode";
+ public static final String DECODE_MODE = "Decode";
+
+ public static final String BINARY_STRING = "BinaryString";
+ public static final String BASE32 = "Base32";
+ public static final String LONG_INT = "LongInt";
+
+ public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
+ .name("mode")
+ .displayName("Mode")
+ .description("Specifies whether to encode latitude/longitude to geohash or decode geohash to latitude/longitude")
+ .required(true)
+ .allowableValues(ENCODE_MODE, DECODE_MODE)
+ .defaultValue(ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor RECORD_READER = new PropertyDescriptor.Builder()
+ .name("record-reader")
+ .displayName("Record Reader")
+ .description("Specifies the record reader service to use for reading incoming data")
+ .required(true)
+ .identifiesControllerService(RecordReaderFactory.class)
+ .build();
+
+ public static final PropertyDescriptor RECORD_WRITER = new PropertyDescriptor.Builder()
+ .name("record-writer")
+ .displayName("Record Writer")
+ .description("Specifies the record writer service to use for writing data")
+ .required(true)
+ .identifiesControllerService(RecordSetWriterFactory.class)
+ .build();
+
+ public static final PropertyDescriptor LATITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("latitude-record-path")
+ .displayName("Latitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the latitude value;"
+ + "in the decode mode, this property specifies the record path to put the latitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor LONGITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("longitude-record-path")
+ .displayName("Longitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the longitude value;"
+ + "in the decode mode, this property specifies the record path to put the longitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_FORMAT = new PropertyDescriptor.Builder()
+ .name("geohash-format")
+ .displayName("Geohash Format")
+ .description("In the encode mode, this property specifies the desired format for encoding geohash;"
+ + "in the decode mode, this property specifies the format of geohash provided")
+ .required(true)
+ .allowableValues(BASE32, BINARY_STRING, LONG_INT)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_LEVEL = new PropertyDescriptor.Builder()
+ .name("geohash-level")
+ .displayName("Geohash Level")
+ .description("The integer precision level(1-12) desired for encoding geohash")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(MODE, ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("geohash-record-path")
+ .displayName("Geohash Record Path")
+ .description("In the encode mode, this property specifies the record path to put the geohash value;"
+ + "in the decode mode, this property specifies the record path to retrieve the geohash value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Records that are successfully encoded or decoded will be routed to success")
+ .build();
+ public static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Records that cannot be encoded or decoded will be routed to failure")
+ .build();
+
+ public static final List<PropertyDescriptor> RECORD_PATH_PROPERTIES = Collections.unmodifiableList(Arrays.asList(
+ LATITUDE_RECORD_PATH,LONGITUDE_RECORD_PATH,GEOHASH_RECORD_PATH
+ ));
+
+ private List<PropertyDescriptor> descriptors;
+
+ private Set<Relationship> relationships;
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ descriptors = new ArrayList<>();
+ descriptors.add(MODE);
+ descriptors.add(RECORD_READER);
+ descriptors.add(RECORD_WRITER);
+ descriptors.add(LATITUDE_RECORD_PATH);
+ descriptors.add(LONGITUDE_RECORD_PATH);
+ descriptors.add(GEOHASH_FORMAT);
+ descriptors.add(GEOHASH_LEVEL);
+ descriptors.add(GEOHASH_RECORD_PATH);
+ descriptors = Collections.unmodifiableList(descriptors);
+
+ relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ relationships.add(REL_FAILURE);
+ relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) {
+ FlowFile input = session.get();
+ if ( input == null ) {
+ return;
+ }
+
+ final RecordReaderFactory readerFactory = context.getProperty(RECORD_READER).asControllerService(RecordReaderFactory.class);
+ final RecordSetWriterFactory writerFactory = context.getProperty(RECORD_WRITER).asControllerService(RecordSetWriterFactory.class);
+ final boolean encode = context.getProperty(MODE).getValue().equalsIgnoreCase(ENCODE_MODE);
+ final String format = context.getProperty(GEOHASH_FORMAT).getValue();
+
+ FlowFile output = session.create(input);
+ try (InputStream is = session.read(input);
+ OutputStream os = session.write(output)) {
+ RecordPathCache cache = new RecordPathCache(RECORD_PATH_PROPERTIES.size() + 1);
+ Map<PropertyDescriptor, RecordPath> paths = new HashMap<>();
+
+ for (PropertyDescriptor descriptor : RECORD_PATH_PROPERTIES) {
+ String rawRecordPath = context.getProperty(descriptor).evaluateAttributeExpressions(input).getValue();
+ RecordPath compiled = cache.getCompiled(rawRecordPath);
+ paths.put(descriptor, compiled);
+ }
+
+ RecordReader reader = readerFactory.createRecordReader(input, is, getLogger());
+ RecordSchema schema = writerFactory.getSchema(input.getAttributes(), reader.getSchema());
+ RecordSetWriter writer = writerFactory.createWriter(getLogger(), schema, os, output);
Review comment:
RecordReader and RecordWriter should be created with a try-with-resources. Need to make sure that we close these.
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,367 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+ public static final String ENCODE_MODE = "Encode";
+ public static final String DECODE_MODE = "Decode";
+
+ public static final String BINARY_STRING = "BinaryString";
+ public static final String BASE32 = "Base32";
+ public static final String LONG_INT = "LongInt";
+
+ public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
+ .name("mode")
+ .displayName("Mode")
+ .description("Specifies whether to encode latitude/longitude to geohash or decode geohash to latitude/longitude")
+ .required(true)
+ .allowableValues(ENCODE_MODE, DECODE_MODE)
+ .defaultValue(ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor RECORD_READER = new PropertyDescriptor.Builder()
+ .name("record-reader")
+ .displayName("Record Reader")
+ .description("Specifies the record reader service to use for reading incoming data")
+ .required(true)
+ .identifiesControllerService(RecordReaderFactory.class)
+ .build();
+
+ public static final PropertyDescriptor RECORD_WRITER = new PropertyDescriptor.Builder()
+ .name("record-writer")
+ .displayName("Record Writer")
+ .description("Specifies the record writer service to use for writing data")
+ .required(true)
+ .identifiesControllerService(RecordSetWriterFactory.class)
+ .build();
+
+ public static final PropertyDescriptor LATITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("latitude-record-path")
+ .displayName("Latitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the latitude value;"
+ + "in the decode mode, this property specifies the record path to put the latitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor LONGITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("longitude-record-path")
+ .displayName("Longitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the longitude value;"
+ + "in the decode mode, this property specifies the record path to put the longitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_FORMAT = new PropertyDescriptor.Builder()
+ .name("geohash-format")
+ .displayName("Geohash Format")
+ .description("In the encode mode, this property specifies the desired format for encoding geohash;"
+ + "in the decode mode, this property specifies the format of geohash provided")
+ .required(true)
+ .allowableValues(BASE32, BINARY_STRING, LONG_INT)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_LEVEL = new PropertyDescriptor.Builder()
+ .name("geohash-level")
+ .displayName("Geohash Level")
+ .description("The integer precision level(1-12) desired for encoding geohash")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(MODE, ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("geohash-record-path")
+ .displayName("Geohash Record Path")
+ .description("In the encode mode, this property specifies the record path to put the geohash value;"
+ + "in the decode mode, this property specifies the record path to retrieve the geohash value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Records that are successfully encoded or decoded will be routed to success")
+ .build();
+ public static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Records that cannot be encoded or decoded will be routed to failure")
+ .build();
+
+ public static final List<PropertyDescriptor> RECORD_PATH_PROPERTIES = Collections.unmodifiableList(Arrays.asList(
+ LATITUDE_RECORD_PATH,LONGITUDE_RECORD_PATH,GEOHASH_RECORD_PATH
+ ));
+
+ private List<PropertyDescriptor> descriptors;
+
+ private Set<Relationship> relationships;
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ descriptors = new ArrayList<>();
+ descriptors.add(MODE);
+ descriptors.add(RECORD_READER);
+ descriptors.add(RECORD_WRITER);
+ descriptors.add(LATITUDE_RECORD_PATH);
+ descriptors.add(LONGITUDE_RECORD_PATH);
+ descriptors.add(GEOHASH_FORMAT);
+ descriptors.add(GEOHASH_LEVEL);
+ descriptors.add(GEOHASH_RECORD_PATH);
+ descriptors = Collections.unmodifiableList(descriptors);
+
+ relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ relationships.add(REL_FAILURE);
+ relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) {
+ FlowFile input = session.get();
+ if ( input == null ) {
+ return;
+ }
+
+ final RecordReaderFactory readerFactory = context.getProperty(RECORD_READER).asControllerService(RecordReaderFactory.class);
+ final RecordSetWriterFactory writerFactory = context.getProperty(RECORD_WRITER).asControllerService(RecordSetWriterFactory.class);
+ final boolean encode = context.getProperty(MODE).getValue().equalsIgnoreCase(ENCODE_MODE);
+ final String format = context.getProperty(GEOHASH_FORMAT).getValue();
+
+ FlowFile output = session.create(input);
+ try (InputStream is = session.read(input);
+ OutputStream os = session.write(output)) {
+ RecordPathCache cache = new RecordPathCache(RECORD_PATH_PROPERTIES.size() + 1);
+ Map<PropertyDescriptor, RecordPath> paths = new HashMap<>();
+
+ for (PropertyDescriptor descriptor : RECORD_PATH_PROPERTIES) {
+ String rawRecordPath = context.getProperty(descriptor).evaluateAttributeExpressions(input).getValue();
+ RecordPath compiled = cache.getCompiled(rawRecordPath);
+ paths.put(descriptor, compiled);
+ }
+
+ RecordReader reader = readerFactory.createRecordReader(input, is, getLogger());
+ RecordSchema schema = writerFactory.getSchema(input.getAttributes(), reader.getSchema());
+ RecordSetWriter writer = writerFactory.createWriter(getLogger(), schema, os, output);
+
+ Record record;
+ Relationship targetRelationship = REL_FAILURE;
+ writer.beginRecordSet();
+
+ int recordsCount = 0;
+ boolean updated = false;
+ while ((record = reader.nextRecord()) != null) {
+ if(encode) {
+ final int level = Integer.parseInt(context.getProperty(GEOHASH_LEVEL).getValue());
+ final String rawLatitudePath = context.getProperty(LATITUDE_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath latitudePath = cache.getCompiled(rawLatitudePath);
+ final String rawLongitudePath = context.getProperty(LONGITUDE_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath longitudePath = cache.getCompiled(rawLongitudePath);
Review comment:
All of these values will be the same for every Record, so they should be calculated only once, outside of the loop
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,367 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+ public static final String ENCODE_MODE = "Encode";
+ public static final String DECODE_MODE = "Decode";
+
+ public static final String BINARY_STRING = "BinaryString";
+ public static final String BASE32 = "Base32";
+ public static final String LONG_INT = "LongInt";
+
+ public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
+ .name("mode")
+ .displayName("Mode")
+ .description("Specifies whether to encode latitude/longitude to geohash or decode geohash to latitude/longitude")
+ .required(true)
+ .allowableValues(ENCODE_MODE, DECODE_MODE)
+ .defaultValue(ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor RECORD_READER = new PropertyDescriptor.Builder()
+ .name("record-reader")
+ .displayName("Record Reader")
+ .description("Specifies the record reader service to use for reading incoming data")
+ .required(true)
+ .identifiesControllerService(RecordReaderFactory.class)
+ .build();
+
+ public static final PropertyDescriptor RECORD_WRITER = new PropertyDescriptor.Builder()
+ .name("record-writer")
+ .displayName("Record Writer")
+ .description("Specifies the record writer service to use for writing data")
+ .required(true)
+ .identifiesControllerService(RecordSetWriterFactory.class)
+ .build();
+
+ public static final PropertyDescriptor LATITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("latitude-record-path")
+ .displayName("Latitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the latitude value;"
+ + "in the decode mode, this property specifies the record path to put the latitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor LONGITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("longitude-record-path")
+ .displayName("Longitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the longitude value;"
+ + "in the decode mode, this property specifies the record path to put the longitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_FORMAT = new PropertyDescriptor.Builder()
+ .name("geohash-format")
+ .displayName("Geohash Format")
+ .description("In the encode mode, this property specifies the desired format for encoding geohash;"
+ + "in the decode mode, this property specifies the format of geohash provided")
+ .required(true)
+ .allowableValues(BASE32, BINARY_STRING, LONG_INT)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_LEVEL = new PropertyDescriptor.Builder()
+ .name("geohash-level")
+ .displayName("Geohash Level")
+ .description("The integer precision level(1-12) desired for encoding geohash")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(MODE, ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("geohash-record-path")
+ .displayName("Geohash Record Path")
+ .description("In the encode mode, this property specifies the record path to put the geohash value;"
+ + "in the decode mode, this property specifies the record path to retrieve the geohash value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Records that are successfully encoded or decoded will be routed to success")
+ .build();
+ public static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Records that cannot be encoded or decoded will be routed to failure")
+ .build();
+
+ public static final List<PropertyDescriptor> RECORD_PATH_PROPERTIES = Collections.unmodifiableList(Arrays.asList(
+ LATITUDE_RECORD_PATH,LONGITUDE_RECORD_PATH,GEOHASH_RECORD_PATH
+ ));
+
+ private List<PropertyDescriptor> descriptors;
+
+ private Set<Relationship> relationships;
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ descriptors = new ArrayList<>();
+ descriptors.add(MODE);
+ descriptors.add(RECORD_READER);
+ descriptors.add(RECORD_WRITER);
+ descriptors.add(LATITUDE_RECORD_PATH);
+ descriptors.add(LONGITUDE_RECORD_PATH);
+ descriptors.add(GEOHASH_FORMAT);
+ descriptors.add(GEOHASH_LEVEL);
+ descriptors.add(GEOHASH_RECORD_PATH);
+ descriptors = Collections.unmodifiableList(descriptors);
+
+ relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ relationships.add(REL_FAILURE);
+ relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) {
+ FlowFile input = session.get();
+ if ( input == null ) {
+ return;
+ }
+
+ final RecordReaderFactory readerFactory = context.getProperty(RECORD_READER).asControllerService(RecordReaderFactory.class);
+ final RecordSetWriterFactory writerFactory = context.getProperty(RECORD_WRITER).asControllerService(RecordSetWriterFactory.class);
+ final boolean encode = context.getProperty(MODE).getValue().equalsIgnoreCase(ENCODE_MODE);
+ final String format = context.getProperty(GEOHASH_FORMAT).getValue();
+
+ FlowFile output = session.create(input);
+ try (InputStream is = session.read(input);
+ OutputStream os = session.write(output)) {
+ RecordPathCache cache = new RecordPathCache(RECORD_PATH_PROPERTIES.size() + 1);
+ Map<PropertyDescriptor, RecordPath> paths = new HashMap<>();
+
+ for (PropertyDescriptor descriptor : RECORD_PATH_PROPERTIES) {
+ String rawRecordPath = context.getProperty(descriptor).evaluateAttributeExpressions(input).getValue();
+ RecordPath compiled = cache.getCompiled(rawRecordPath);
+ paths.put(descriptor, compiled);
+ }
+
+ RecordReader reader = readerFactory.createRecordReader(input, is, getLogger());
+ RecordSchema schema = writerFactory.getSchema(input.getAttributes(), reader.getSchema());
+ RecordSetWriter writer = writerFactory.createWriter(getLogger(), schema, os, output);
+
+ Record record;
+ Relationship targetRelationship = REL_FAILURE;
+ writer.beginRecordSet();
+
+ int recordsCount = 0;
+ boolean updated = false;
+ while ((record = reader.nextRecord()) != null) {
+ if(encode) {
+ final int level = Integer.parseInt(context.getProperty(GEOHASH_LEVEL).getValue());
+ final String rawLatitudePath = context.getProperty(LATITUDE_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath latitudePath = cache.getCompiled(rawLatitudePath);
+ final String rawLongitudePath = context.getProperty(LONGITUDE_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath longitudePath = cache.getCompiled(rawLongitudePath);
+
+ Object encodedGeohash = getEncodedGeohash(latitudePath, longitudePath, record, format, level);
+
+ updated = updateRecord(GEOHASH_RECORD_PATH, encodedGeohash, record, paths);
+ }else {
+ final String rawgeohashPath = context.getProperty(GEOHASH_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath geohashPath = cache.getCompiled(rawgeohashPath);
+
+ WGS84Point decodedPoint = getDecodedPointFromGeohash(geohashPath, record, format);
+
+ if(decodedPoint != null) {
+ updated = updateRecord(LATITUDE_RECORD_PATH, String.valueOf(decodedPoint.getLatitude()), record, paths)
+ && updateRecord(LONGITUDE_RECORD_PATH, String.valueOf(decodedPoint.getLongitude()), record, paths);
+ }
+ }
+
+ if (updated) {
+ targetRelationship = REL_SUCCESS;
+ }
+
+ writer.write(record);
+ recordsCount++;
+
+ }
+ writer.finishRecordSet();
+ writer.close();
+
+ is.close();
+ os.close();
+
+ output = session.putAllAttributes(output, buildAttributes(recordsCount, writer.getMimeType()));
+
+ session.transfer(output, targetRelationship);
+ session.remove(input);
+
+ session.getProvenanceReporter().modifyContent(output);
+ } catch (Exception ex) {
+ getLogger().error("Failed to {}; will route to failure", new Object[]{encode ? "encode" : "decode" ,ex});
+ session.rollback();
+ context.yield();
+ }
+ }
+
+ private Object getEncodedGeohash(RecordPath latitudePath, RecordPath longitudePath, Record record, String format, int level) {
+ RecordPathResult latitudeResult = latitudePath.evaluate(record);
+ RecordPathResult longitudeResult = longitudePath.evaluate(record);
+ Optional<FieldValue> latitudeField = latitudeResult.getSelectedFields().findFirst();
+ Optional<FieldValue> longitudeField = longitudeResult.getSelectedFields().findFirst();
+
+ if (latitudeField.isPresent() && longitudeField.isPresent()) {
+ FieldValue latitudeValue = latitudeField.get();
+ FieldValue longitudeValue = longitudeField.get();
+ Object latitudeVal = latitudeValue.getValue();
+ Object longitudeVal = longitudeValue.getValue();
+
+ if (latitudeVal == null || longitudeVal == null) {
+ return null;
+ }
+
+ double realLatValue = Double.parseDouble(latitudeVal.toString());
+ double realLongValue = Double.parseDouble(longitudeVal.toString());
+ GeoHash gh = GeoHash.withCharacterPrecision(realLatValue, realLongValue, level);
+
+ switch (format) {
+ case BINARY_STRING:
+ return gh.toBinaryString();
+ case LONG_INT:
+ return gh.longValue();
+ default:
+ return gh.toBase32();
+ }
+ } else {
+ return null;
+ }
+ }
+
+ private WGS84Point getDecodedPointFromGeohash(RecordPath geohashPath, Record record, String format) {
+ RecordPathResult geohashResult = geohashPath.evaluate(record);
+ Optional<FieldValue> geohashField = geohashResult.getSelectedFields().findFirst();
+ if (geohashField.isPresent()) {
+ FieldValue geohashFieldValue = geohashField.get();
+ Object geohashVal = geohashFieldValue.getValue();
+ if(geohashVal == null) {
+ return null;
+ }
+
+ String geohashString = geohashVal.toString();
+ GeoHash decodedHash;
+
+ switch (format) {
+ case BINARY_STRING:
+ decodedHash = GeoHash.fromBinaryString(geohashString);
+ break;
+ case LONG_INT:
+ String binaryString = Long.toBinaryString(Long.parseLong(geohashString));
+ decodedHash = GeoHash.fromBinaryString(binaryString);
+ break;
+ default:
+ decodedHash = GeoHash.fromGeohashString(geohashString);
+ }
+
+ return decodedHash.getBoundingBoxCenter();
+
+ }else {
+ return null;
+ }
+ }
+
+ private boolean updateRecord(PropertyDescriptor descriptor, Object newValue, Record record, Map<PropertyDescriptor, RecordPath> cached) {
+ if (!cached.containsKey(descriptor) || newValue == null) {
+ return false;
+ }
+ RecordPath path = cached.get(descriptor);
+ RecordPathResult result = path.evaluate(record);
+ FieldValue fieldValue = result.getSelectedFields().findFirst().get();
Review comment:
`result.getSelectedFields().findFirst()` may return an empty Optional. Probably best to do:
```
final Optional<FieldValue> fieldValueOptional = result.getSelectedFields().findFirst();
if (fieldValueOption.isEmpty()) {
return false;
}
final FieldValue fieldValue = fieldValueOptional.get();
```
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,367 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+ public static final String ENCODE_MODE = "Encode";
+ public static final String DECODE_MODE = "Decode";
+
+ public static final String BINARY_STRING = "BinaryString";
+ public static final String BASE32 = "Base32";
+ public static final String LONG_INT = "LongInt";
+
+ public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
+ .name("mode")
+ .displayName("Mode")
+ .description("Specifies whether to encode latitude/longitude to geohash or decode geohash to latitude/longitude")
+ .required(true)
+ .allowableValues(ENCODE_MODE, DECODE_MODE)
+ .defaultValue(ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor RECORD_READER = new PropertyDescriptor.Builder()
+ .name("record-reader")
+ .displayName("Record Reader")
+ .description("Specifies the record reader service to use for reading incoming data")
+ .required(true)
+ .identifiesControllerService(RecordReaderFactory.class)
+ .build();
+
+ public static final PropertyDescriptor RECORD_WRITER = new PropertyDescriptor.Builder()
+ .name("record-writer")
+ .displayName("Record Writer")
+ .description("Specifies the record writer service to use for writing data")
+ .required(true)
+ .identifiesControllerService(RecordSetWriterFactory.class)
+ .build();
+
+ public static final PropertyDescriptor LATITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("latitude-record-path")
+ .displayName("Latitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the latitude value;"
+ + "in the decode mode, this property specifies the record path to put the latitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor LONGITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("longitude-record-path")
+ .displayName("Longitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the longitude value;"
+ + "in the decode mode, this property specifies the record path to put the longitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_FORMAT = new PropertyDescriptor.Builder()
+ .name("geohash-format")
+ .displayName("Geohash Format")
+ .description("In the encode mode, this property specifies the desired format for encoding geohash;"
+ + "in the decode mode, this property specifies the format of geohash provided")
+ .required(true)
+ .allowableValues(BASE32, BINARY_STRING, LONG_INT)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_LEVEL = new PropertyDescriptor.Builder()
+ .name("geohash-level")
+ .displayName("Geohash Level")
+ .description("The integer precision level(1-12) desired for encoding geohash")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(MODE, ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("geohash-record-path")
+ .displayName("Geohash Record Path")
+ .description("In the encode mode, this property specifies the record path to put the geohash value;"
+ + "in the decode mode, this property specifies the record path to retrieve the geohash value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Records that are successfully encoded or decoded will be routed to success")
+ .build();
+ public static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Records that cannot be encoded or decoded will be routed to failure")
+ .build();
+
+ public static final List<PropertyDescriptor> RECORD_PATH_PROPERTIES = Collections.unmodifiableList(Arrays.asList(
+ LATITUDE_RECORD_PATH,LONGITUDE_RECORD_PATH,GEOHASH_RECORD_PATH
+ ));
+
+ private List<PropertyDescriptor> descriptors;
+
+ private Set<Relationship> relationships;
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ descriptors = new ArrayList<>();
+ descriptors.add(MODE);
+ descriptors.add(RECORD_READER);
+ descriptors.add(RECORD_WRITER);
+ descriptors.add(LATITUDE_RECORD_PATH);
+ descriptors.add(LONGITUDE_RECORD_PATH);
+ descriptors.add(GEOHASH_FORMAT);
+ descriptors.add(GEOHASH_LEVEL);
+ descriptors.add(GEOHASH_RECORD_PATH);
+ descriptors = Collections.unmodifiableList(descriptors);
+
+ relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ relationships.add(REL_FAILURE);
+ relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) {
+ FlowFile input = session.get();
+ if ( input == null ) {
+ return;
+ }
+
+ final RecordReaderFactory readerFactory = context.getProperty(RECORD_READER).asControllerService(RecordReaderFactory.class);
+ final RecordSetWriterFactory writerFactory = context.getProperty(RECORD_WRITER).asControllerService(RecordSetWriterFactory.class);
+ final boolean encode = context.getProperty(MODE).getValue().equalsIgnoreCase(ENCODE_MODE);
+ final String format = context.getProperty(GEOHASH_FORMAT).getValue();
+
+ FlowFile output = session.create(input);
+ try (InputStream is = session.read(input);
+ OutputStream os = session.write(output)) {
+ RecordPathCache cache = new RecordPathCache(RECORD_PATH_PROPERTIES.size() + 1);
+ Map<PropertyDescriptor, RecordPath> paths = new HashMap<>();
+
+ for (PropertyDescriptor descriptor : RECORD_PATH_PROPERTIES) {
+ String rawRecordPath = context.getProperty(descriptor).evaluateAttributeExpressions(input).getValue();
+ RecordPath compiled = cache.getCompiled(rawRecordPath);
+ paths.put(descriptor, compiled);
+ }
+
+ RecordReader reader = readerFactory.createRecordReader(input, is, getLogger());
+ RecordSchema schema = writerFactory.getSchema(input.getAttributes(), reader.getSchema());
+ RecordSetWriter writer = writerFactory.createWriter(getLogger(), schema, os, output);
+
+ Record record;
+ Relationship targetRelationship = REL_FAILURE;
+ writer.beginRecordSet();
+
+ int recordsCount = 0;
+ boolean updated = false;
+ while ((record = reader.nextRecord()) != null) {
+ if(encode) {
+ final int level = Integer.parseInt(context.getProperty(GEOHASH_LEVEL).getValue());
+ final String rawLatitudePath = context.getProperty(LATITUDE_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath latitudePath = cache.getCompiled(rawLatitudePath);
+ final String rawLongitudePath = context.getProperty(LONGITUDE_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath longitudePath = cache.getCompiled(rawLongitudePath);
+
+ Object encodedGeohash = getEncodedGeohash(latitudePath, longitudePath, record, format, level);
+
+ updated = updateRecord(GEOHASH_RECORD_PATH, encodedGeohash, record, paths);
+ }else {
+ final String rawgeohashPath = context.getProperty(GEOHASH_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath geohashPath = cache.getCompiled(rawgeohashPath);
+
+ WGS84Point decodedPoint = getDecodedPointFromGeohash(geohashPath, record, format);
+
+ if(decodedPoint != null) {
+ updated = updateRecord(LATITUDE_RECORD_PATH, String.valueOf(decodedPoint.getLatitude()), record, paths)
+ && updateRecord(LONGITUDE_RECORD_PATH, String.valueOf(decodedPoint.getLongitude()), record, paths);
+ }
+ }
+
+ if (updated) {
+ targetRelationship = REL_SUCCESS;
+ }
+
+ writer.write(record);
+ recordsCount++;
+
+ }
+ writer.finishRecordSet();
+ writer.close();
+
+ is.close();
+ os.close();
+
+ output = session.putAllAttributes(output, buildAttributes(recordsCount, writer.getMimeType()));
+
+ session.transfer(output, targetRelationship);
+ session.remove(input);
+
+ session.getProvenanceReporter().modifyContent(output);
+ } catch (Exception ex) {
+ getLogger().error("Failed to {}; will route to failure", new Object[]{encode ? "encode" : "decode" ,ex});
+ session.rollback();
+ context.yield();
+ }
+ }
+
+ private Object getEncodedGeohash(RecordPath latitudePath, RecordPath longitudePath, Record record, String format, int level) {
+ RecordPathResult latitudeResult = latitudePath.evaluate(record);
+ RecordPathResult longitudeResult = longitudePath.evaluate(record);
+ Optional<FieldValue> latitudeField = latitudeResult.getSelectedFields().findFirst();
+ Optional<FieldValue> longitudeField = longitudeResult.getSelectedFields().findFirst();
+
+ if (latitudeField.isPresent() && longitudeField.isPresent()) {
+ FieldValue latitudeValue = latitudeField.get();
+ FieldValue longitudeValue = longitudeField.get();
+ Object latitudeVal = latitudeValue.getValue();
+ Object longitudeVal = longitudeValue.getValue();
+
+ if (latitudeVal == null || longitudeVal == null) {
+ return null;
+ }
+
+ double realLatValue = Double.parseDouble(latitudeVal.toString());
+ double realLongValue = Double.parseDouble(longitudeVal.toString());
+ GeoHash gh = GeoHash.withCharacterPrecision(realLatValue, realLongValue, level);
+
+ switch (format) {
+ case BINARY_STRING:
+ return gh.toBinaryString();
+ case LONG_INT:
+ return gh.longValue();
+ default:
+ return gh.toBase32();
+ }
+ } else {
+ return null;
+ }
+ }
+
+ private WGS84Point getDecodedPointFromGeohash(RecordPath geohashPath, Record record, String format) {
+ RecordPathResult geohashResult = geohashPath.evaluate(record);
+ Optional<FieldValue> geohashField = geohashResult.getSelectedFields().findFirst();
+ if (geohashField.isPresent()) {
Review comment:
This will tend to be a bit cleaner if we reverse the conditional. I.e., instead of:
```
if (geohashField.isPresent()) {
... a bunch of logic here, with indentation ...
} else {
return null;
}
```
It's simpler if we change this to:
```
if (!geohashField.isPresent()) {
return null;
}
... the rest of the logic here...
```
Makes it very clear that the logic is "if it's not present, just return null, nothing else we can do" and also means we avoid lots of lines of nested code.
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,367 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+ public static final String ENCODE_MODE = "Encode";
+ public static final String DECODE_MODE = "Decode";
+
+ public static final String BINARY_STRING = "BinaryString";
+ public static final String BASE32 = "Base32";
+ public static final String LONG_INT = "LongInt";
+
+ public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
+ .name("mode")
+ .displayName("Mode")
+ .description("Specifies whether to encode latitude/longitude to geohash or decode geohash to latitude/longitude")
+ .required(true)
+ .allowableValues(ENCODE_MODE, DECODE_MODE)
+ .defaultValue(ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor RECORD_READER = new PropertyDescriptor.Builder()
+ .name("record-reader")
+ .displayName("Record Reader")
+ .description("Specifies the record reader service to use for reading incoming data")
+ .required(true)
+ .identifiesControllerService(RecordReaderFactory.class)
+ .build();
+
+ public static final PropertyDescriptor RECORD_WRITER = new PropertyDescriptor.Builder()
+ .name("record-writer")
+ .displayName("Record Writer")
+ .description("Specifies the record writer service to use for writing data")
+ .required(true)
+ .identifiesControllerService(RecordSetWriterFactory.class)
+ .build();
+
+ public static final PropertyDescriptor LATITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("latitude-record-path")
+ .displayName("Latitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the latitude value;"
+ + "in the decode mode, this property specifies the record path to put the latitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor LONGITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("longitude-record-path")
+ .displayName("Longitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the longitude value;"
+ + "in the decode mode, this property specifies the record path to put the longitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_FORMAT = new PropertyDescriptor.Builder()
+ .name("geohash-format")
+ .displayName("Geohash Format")
+ .description("In the encode mode, this property specifies the desired format for encoding geohash;"
+ + "in the decode mode, this property specifies the format of geohash provided")
+ .required(true)
+ .allowableValues(BASE32, BINARY_STRING, LONG_INT)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_LEVEL = new PropertyDescriptor.Builder()
+ .name("geohash-level")
+ .displayName("Geohash Level")
+ .description("The integer precision level(1-12) desired for encoding geohash")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(MODE, ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("geohash-record-path")
+ .displayName("Geohash Record Path")
+ .description("In the encode mode, this property specifies the record path to put the geohash value;"
+ + "in the decode mode, this property specifies the record path to retrieve the geohash value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Records that are successfully encoded or decoded will be routed to success")
+ .build();
+ public static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Records that cannot be encoded or decoded will be routed to failure")
+ .build();
+
+ public static final List<PropertyDescriptor> RECORD_PATH_PROPERTIES = Collections.unmodifiableList(Arrays.asList(
+ LATITUDE_RECORD_PATH,LONGITUDE_RECORD_PATH,GEOHASH_RECORD_PATH
+ ));
+
+ private List<PropertyDescriptor> descriptors;
+
+ private Set<Relationship> relationships;
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ descriptors = new ArrayList<>();
+ descriptors.add(MODE);
+ descriptors.add(RECORD_READER);
+ descriptors.add(RECORD_WRITER);
+ descriptors.add(LATITUDE_RECORD_PATH);
+ descriptors.add(LONGITUDE_RECORD_PATH);
+ descriptors.add(GEOHASH_FORMAT);
+ descriptors.add(GEOHASH_LEVEL);
+ descriptors.add(GEOHASH_RECORD_PATH);
+ descriptors = Collections.unmodifiableList(descriptors);
+
+ relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ relationships.add(REL_FAILURE);
+ relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) {
+ FlowFile input = session.get();
+ if ( input == null ) {
+ return;
+ }
+
+ final RecordReaderFactory readerFactory = context.getProperty(RECORD_READER).asControllerService(RecordReaderFactory.class);
+ final RecordSetWriterFactory writerFactory = context.getProperty(RECORD_WRITER).asControllerService(RecordSetWriterFactory.class);
+ final boolean encode = context.getProperty(MODE).getValue().equalsIgnoreCase(ENCODE_MODE);
+ final String format = context.getProperty(GEOHASH_FORMAT).getValue();
+
+ FlowFile output = session.create(input);
+ try (InputStream is = session.read(input);
+ OutputStream os = session.write(output)) {
+ RecordPathCache cache = new RecordPathCache(RECORD_PATH_PROPERTIES.size() + 1);
+ Map<PropertyDescriptor, RecordPath> paths = new HashMap<>();
+
+ for (PropertyDescriptor descriptor : RECORD_PATH_PROPERTIES) {
+ String rawRecordPath = context.getProperty(descriptor).evaluateAttributeExpressions(input).getValue();
+ RecordPath compiled = cache.getCompiled(rawRecordPath);
+ paths.put(descriptor, compiled);
+ }
+
+ RecordReader reader = readerFactory.createRecordReader(input, is, getLogger());
+ RecordSchema schema = writerFactory.getSchema(input.getAttributes(), reader.getSchema());
+ RecordSetWriter writer = writerFactory.createWriter(getLogger(), schema, os, output);
+
+ Record record;
+ Relationship targetRelationship = REL_FAILURE;
+ writer.beginRecordSet();
+
+ int recordsCount = 0;
+ boolean updated = false;
+ while ((record = reader.nextRecord()) != null) {
+ if(encode) {
+ final int level = Integer.parseInt(context.getProperty(GEOHASH_LEVEL).getValue());
+ final String rawLatitudePath = context.getProperty(LATITUDE_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath latitudePath = cache.getCompiled(rawLatitudePath);
+ final String rawLongitudePath = context.getProperty(LONGITUDE_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath longitudePath = cache.getCompiled(rawLongitudePath);
+
+ Object encodedGeohash = getEncodedGeohash(latitudePath, longitudePath, record, format, level);
+
+ updated = updateRecord(GEOHASH_RECORD_PATH, encodedGeohash, record, paths);
+ }else {
+ final String rawgeohashPath = context.getProperty(GEOHASH_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath geohashPath = cache.getCompiled(rawgeohashPath);
+
+ WGS84Point decodedPoint = getDecodedPointFromGeohash(geohashPath, record, format);
+
+ if(decodedPoint != null) {
+ updated = updateRecord(LATITUDE_RECORD_PATH, String.valueOf(decodedPoint.getLatitude()), record, paths)
+ && updateRecord(LONGITUDE_RECORD_PATH, String.valueOf(decodedPoint.getLongitude()), record, paths);
+ }
+ }
+
+ if (updated) {
+ targetRelationship = REL_SUCCESS;
+ }
+
+ writer.write(record);
+ recordsCount++;
+
+ }
+ writer.finishRecordSet();
+ writer.close();
+
+ is.close();
+ os.close();
+
+ output = session.putAllAttributes(output, buildAttributes(recordsCount, writer.getMimeType()));
+
+ session.transfer(output, targetRelationship);
+ session.remove(input);
Review comment:
We should avoid removing the original FlowFile. Instead, we should create an "original" relationship and transfer the FlowFile to that Relationship. Can use
```
new Relationship.Builder()
.name("original")
.description("The original, unenriched FlowFile will be routed to this relationship")
.autoTerminateDefault(true)
.build();
```
in order to have it auto-terminate the 'original' relationship be default. Most of the time it's not necessary to have the default relationship but it's helpful in some use cases.
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,367 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+ public static final String ENCODE_MODE = "Encode";
+ public static final String DECODE_MODE = "Decode";
+
+ public static final String BINARY_STRING = "BinaryString";
+ public static final String BASE32 = "Base32";
+ public static final String LONG_INT = "LongInt";
+
+ public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
+ .name("mode")
+ .displayName("Mode")
+ .description("Specifies whether to encode latitude/longitude to geohash or decode geohash to latitude/longitude")
+ .required(true)
+ .allowableValues(ENCODE_MODE, DECODE_MODE)
+ .defaultValue(ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor RECORD_READER = new PropertyDescriptor.Builder()
+ .name("record-reader")
+ .displayName("Record Reader")
+ .description("Specifies the record reader service to use for reading incoming data")
+ .required(true)
+ .identifiesControllerService(RecordReaderFactory.class)
+ .build();
+
+ public static final PropertyDescriptor RECORD_WRITER = new PropertyDescriptor.Builder()
+ .name("record-writer")
+ .displayName("Record Writer")
+ .description("Specifies the record writer service to use for writing data")
+ .required(true)
+ .identifiesControllerService(RecordSetWriterFactory.class)
+ .build();
+
+ public static final PropertyDescriptor LATITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("latitude-record-path")
+ .displayName("Latitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the latitude value;"
+ + "in the decode mode, this property specifies the record path to put the latitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor LONGITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("longitude-record-path")
+ .displayName("Longitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the longitude value;"
+ + "in the decode mode, this property specifies the record path to put the longitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_FORMAT = new PropertyDescriptor.Builder()
+ .name("geohash-format")
+ .displayName("Geohash Format")
+ .description("In the encode mode, this property specifies the desired format for encoding geohash;"
+ + "in the decode mode, this property specifies the format of geohash provided")
+ .required(true)
+ .allowableValues(BASE32, BINARY_STRING, LONG_INT)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_LEVEL = new PropertyDescriptor.Builder()
+ .name("geohash-level")
+ .displayName("Geohash Level")
+ .description("The integer precision level(1-12) desired for encoding geohash")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(MODE, ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("geohash-record-path")
+ .displayName("Geohash Record Path")
+ .description("In the encode mode, this property specifies the record path to put the geohash value;"
+ + "in the decode mode, this property specifies the record path to retrieve the geohash value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Records that are successfully encoded or decoded will be routed to success")
+ .build();
+ public static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Records that cannot be encoded or decoded will be routed to failure")
+ .build();
+
+ public static final List<PropertyDescriptor> RECORD_PATH_PROPERTIES = Collections.unmodifiableList(Arrays.asList(
+ LATITUDE_RECORD_PATH,LONGITUDE_RECORD_PATH,GEOHASH_RECORD_PATH
+ ));
+
+ private List<PropertyDescriptor> descriptors;
+
+ private Set<Relationship> relationships;
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ descriptors = new ArrayList<>();
+ descriptors.add(MODE);
+ descriptors.add(RECORD_READER);
+ descriptors.add(RECORD_WRITER);
+ descriptors.add(LATITUDE_RECORD_PATH);
+ descriptors.add(LONGITUDE_RECORD_PATH);
+ descriptors.add(GEOHASH_FORMAT);
+ descriptors.add(GEOHASH_LEVEL);
+ descriptors.add(GEOHASH_RECORD_PATH);
+ descriptors = Collections.unmodifiableList(descriptors);
+
+ relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ relationships.add(REL_FAILURE);
+ relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) {
+ FlowFile input = session.get();
+ if ( input == null ) {
+ return;
+ }
+
+ final RecordReaderFactory readerFactory = context.getProperty(RECORD_READER).asControllerService(RecordReaderFactory.class);
+ final RecordSetWriterFactory writerFactory = context.getProperty(RECORD_WRITER).asControllerService(RecordSetWriterFactory.class);
+ final boolean encode = context.getProperty(MODE).getValue().equalsIgnoreCase(ENCODE_MODE);
+ final String format = context.getProperty(GEOHASH_FORMAT).getValue();
+
+ FlowFile output = session.create(input);
+ try (InputStream is = session.read(input);
+ OutputStream os = session.write(output)) {
+ RecordPathCache cache = new RecordPathCache(RECORD_PATH_PROPERTIES.size() + 1);
+ Map<PropertyDescriptor, RecordPath> paths = new HashMap<>();
+
+ for (PropertyDescriptor descriptor : RECORD_PATH_PROPERTIES) {
+ String rawRecordPath = context.getProperty(descriptor).evaluateAttributeExpressions(input).getValue();
+ RecordPath compiled = cache.getCompiled(rawRecordPath);
+ paths.put(descriptor, compiled);
+ }
+
+ RecordReader reader = readerFactory.createRecordReader(input, is, getLogger());
+ RecordSchema schema = writerFactory.getSchema(input.getAttributes(), reader.getSchema());
+ RecordSetWriter writer = writerFactory.createWriter(getLogger(), schema, os, output);
+
+ Record record;
+ Relationship targetRelationship = REL_FAILURE;
+ writer.beginRecordSet();
+
+ int recordsCount = 0;
+ boolean updated = false;
+ while ((record = reader.nextRecord()) != null) {
+ if(encode) {
+ final int level = Integer.parseInt(context.getProperty(GEOHASH_LEVEL).getValue());
+ final String rawLatitudePath = context.getProperty(LATITUDE_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath latitudePath = cache.getCompiled(rawLatitudePath);
+ final String rawLongitudePath = context.getProperty(LONGITUDE_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath longitudePath = cache.getCompiled(rawLongitudePath);
+
+ Object encodedGeohash = getEncodedGeohash(latitudePath, longitudePath, record, format, level);
+
+ updated = updateRecord(GEOHASH_RECORD_PATH, encodedGeohash, record, paths);
+ }else {
+ final String rawgeohashPath = context.getProperty(GEOHASH_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath geohashPath = cache.getCompiled(rawgeohashPath);
+
+ WGS84Point decodedPoint = getDecodedPointFromGeohash(geohashPath, record, format);
+
+ if(decodedPoint != null) {
+ updated = updateRecord(LATITUDE_RECORD_PATH, String.valueOf(decodedPoint.getLatitude()), record, paths)
+ && updateRecord(LONGITUDE_RECORD_PATH, String.valueOf(decodedPoint.getLongitude()), record, paths);
+ }
+ }
+
+ if (updated) {
+ targetRelationship = REL_SUCCESS;
+ }
+
+ writer.write(record);
+ recordsCount++;
+
+ }
+ writer.finishRecordSet();
+ writer.close();
+
+ is.close();
+ os.close();
+
+ output = session.putAllAttributes(output, buildAttributes(recordsCount, writer.getMimeType()));
+
+ session.transfer(output, targetRelationship);
+ session.remove(input);
+
+ session.getProvenanceReporter().modifyContent(output);
+ } catch (Exception ex) {
+ getLogger().error("Failed to {}; will route to failure", new Object[]{encode ? "encode" : "decode" ,ex});
+ session.rollback();
+ context.yield();
Review comment:
No need to rollback or yield here. Should instead route the incoming FlowFile to 'failure'
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,367 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+ public static final String ENCODE_MODE = "Encode";
+ public static final String DECODE_MODE = "Decode";
+
+ public static final String BINARY_STRING = "BinaryString";
+ public static final String BASE32 = "Base32";
+ public static final String LONG_INT = "LongInt";
+
+ public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
+ .name("mode")
+ .displayName("Mode")
+ .description("Specifies whether to encode latitude/longitude to geohash or decode geohash to latitude/longitude")
+ .required(true)
+ .allowableValues(ENCODE_MODE, DECODE_MODE)
+ .defaultValue(ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor RECORD_READER = new PropertyDescriptor.Builder()
+ .name("record-reader")
+ .displayName("Record Reader")
+ .description("Specifies the record reader service to use for reading incoming data")
+ .required(true)
+ .identifiesControllerService(RecordReaderFactory.class)
+ .build();
+
+ public static final PropertyDescriptor RECORD_WRITER = new PropertyDescriptor.Builder()
+ .name("record-writer")
+ .displayName("Record Writer")
+ .description("Specifies the record writer service to use for writing data")
+ .required(true)
+ .identifiesControllerService(RecordSetWriterFactory.class)
+ .build();
+
+ public static final PropertyDescriptor LATITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("latitude-record-path")
+ .displayName("Latitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the latitude value;"
+ + "in the decode mode, this property specifies the record path to put the latitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor LONGITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("longitude-record-path")
+ .displayName("Longitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the longitude value;"
+ + "in the decode mode, this property specifies the record path to put the longitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_FORMAT = new PropertyDescriptor.Builder()
+ .name("geohash-format")
+ .displayName("Geohash Format")
+ .description("In the encode mode, this property specifies the desired format for encoding geohash;"
+ + "in the decode mode, this property specifies the format of geohash provided")
+ .required(true)
+ .allowableValues(BASE32, BINARY_STRING, LONG_INT)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_LEVEL = new PropertyDescriptor.Builder()
+ .name("geohash-level")
+ .displayName("Geohash Level")
+ .description("The integer precision level(1-12) desired for encoding geohash")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(MODE, ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("geohash-record-path")
+ .displayName("Geohash Record Path")
+ .description("In the encode mode, this property specifies the record path to put the geohash value;"
+ + "in the decode mode, this property specifies the record path to retrieve the geohash value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Records that are successfully encoded or decoded will be routed to success")
+ .build();
+ public static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Records that cannot be encoded or decoded will be routed to failure")
+ .build();
+
+ public static final List<PropertyDescriptor> RECORD_PATH_PROPERTIES = Collections.unmodifiableList(Arrays.asList(
+ LATITUDE_RECORD_PATH,LONGITUDE_RECORD_PATH,GEOHASH_RECORD_PATH
+ ));
+
+ private List<PropertyDescriptor> descriptors;
+
+ private Set<Relationship> relationships;
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ descriptors = new ArrayList<>();
+ descriptors.add(MODE);
+ descriptors.add(RECORD_READER);
+ descriptors.add(RECORD_WRITER);
+ descriptors.add(LATITUDE_RECORD_PATH);
+ descriptors.add(LONGITUDE_RECORD_PATH);
+ descriptors.add(GEOHASH_FORMAT);
+ descriptors.add(GEOHASH_LEVEL);
+ descriptors.add(GEOHASH_RECORD_PATH);
+ descriptors = Collections.unmodifiableList(descriptors);
+
+ relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ relationships.add(REL_FAILURE);
+ relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) {
+ FlowFile input = session.get();
+ if ( input == null ) {
+ return;
+ }
+
+ final RecordReaderFactory readerFactory = context.getProperty(RECORD_READER).asControllerService(RecordReaderFactory.class);
+ final RecordSetWriterFactory writerFactory = context.getProperty(RECORD_WRITER).asControllerService(RecordSetWriterFactory.class);
+ final boolean encode = context.getProperty(MODE).getValue().equalsIgnoreCase(ENCODE_MODE);
+ final String format = context.getProperty(GEOHASH_FORMAT).getValue();
+
+ FlowFile output = session.create(input);
+ try (InputStream is = session.read(input);
+ OutputStream os = session.write(output)) {
+ RecordPathCache cache = new RecordPathCache(RECORD_PATH_PROPERTIES.size() + 1);
+ Map<PropertyDescriptor, RecordPath> paths = new HashMap<>();
+
+ for (PropertyDescriptor descriptor : RECORD_PATH_PROPERTIES) {
+ String rawRecordPath = context.getProperty(descriptor).evaluateAttributeExpressions(input).getValue();
+ RecordPath compiled = cache.getCompiled(rawRecordPath);
+ paths.put(descriptor, compiled);
+ }
+
+ RecordReader reader = readerFactory.createRecordReader(input, is, getLogger());
+ RecordSchema schema = writerFactory.getSchema(input.getAttributes(), reader.getSchema());
+ RecordSetWriter writer = writerFactory.createWriter(getLogger(), schema, os, output);
+
+ Record record;
+ Relationship targetRelationship = REL_FAILURE;
+ writer.beginRecordSet();
+
+ int recordsCount = 0;
+ boolean updated = false;
+ while ((record = reader.nextRecord()) != null) {
+ if(encode) {
+ final int level = Integer.parseInt(context.getProperty(GEOHASH_LEVEL).getValue());
+ final String rawLatitudePath = context.getProperty(LATITUDE_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath latitudePath = cache.getCompiled(rawLatitudePath);
+ final String rawLongitudePath = context.getProperty(LONGITUDE_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath longitudePath = cache.getCompiled(rawLongitudePath);
+
+ Object encodedGeohash = getEncodedGeohash(latitudePath, longitudePath, record, format, level);
+
+ updated = updateRecord(GEOHASH_RECORD_PATH, encodedGeohash, record, paths);
+ }else {
+ final String rawgeohashPath = context.getProperty(GEOHASH_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath geohashPath = cache.getCompiled(rawgeohashPath);
+
+ WGS84Point decodedPoint = getDecodedPointFromGeohash(geohashPath, record, format);
+
+ if(decodedPoint != null) {
+ updated = updateRecord(LATITUDE_RECORD_PATH, String.valueOf(decodedPoint.getLatitude()), record, paths)
+ && updateRecord(LONGITUDE_RECORD_PATH, String.valueOf(decodedPoint.getLongitude()), record, paths);
+ }
+ }
+
+ if (updated) {
+ targetRelationship = REL_SUCCESS;
+ }
+
+ writer.write(record);
+ recordsCount++;
+
+ }
+ writer.finishRecordSet();
Review comment:
This call to `finishRecordSet()` will return a `WriteResult`. It will be important to capture that `WriteResult` and call `getAttributes()` on it and include those values in the call to `buildAttributes` below. This allows the RecordWriter to provide attributes that need to go onto the FlowFile, such as the schema, etc.
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,367 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+ public static final String ENCODE_MODE = "Encode";
+ public static final String DECODE_MODE = "Decode";
+
+ public static final String BINARY_STRING = "BinaryString";
+ public static final String BASE32 = "Base32";
+ public static final String LONG_INT = "LongInt";
+
+ public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
+ .name("mode")
+ .displayName("Mode")
+ .description("Specifies whether to encode latitude/longitude to geohash or decode geohash to latitude/longitude")
+ .required(true)
+ .allowableValues(ENCODE_MODE, DECODE_MODE)
+ .defaultValue(ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor RECORD_READER = new PropertyDescriptor.Builder()
+ .name("record-reader")
+ .displayName("Record Reader")
+ .description("Specifies the record reader service to use for reading incoming data")
+ .required(true)
+ .identifiesControllerService(RecordReaderFactory.class)
+ .build();
+
+ public static final PropertyDescriptor RECORD_WRITER = new PropertyDescriptor.Builder()
+ .name("record-writer")
+ .displayName("Record Writer")
+ .description("Specifies the record writer service to use for writing data")
+ .required(true)
+ .identifiesControllerService(RecordSetWriterFactory.class)
+ .build();
+
+ public static final PropertyDescriptor LATITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("latitude-record-path")
+ .displayName("Latitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the latitude value;"
+ + "in the decode mode, this property specifies the record path to put the latitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor LONGITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("longitude-record-path")
+ .displayName("Longitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the longitude value;"
+ + "in the decode mode, this property specifies the record path to put the longitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_FORMAT = new PropertyDescriptor.Builder()
+ .name("geohash-format")
+ .displayName("Geohash Format")
+ .description("In the encode mode, this property specifies the desired format for encoding geohash;"
+ + "in the decode mode, this property specifies the format of geohash provided")
+ .required(true)
+ .allowableValues(BASE32, BINARY_STRING, LONG_INT)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_LEVEL = new PropertyDescriptor.Builder()
+ .name("geohash-level")
+ .displayName("Geohash Level")
+ .description("The integer precision level(1-12) desired for encoding geohash")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(MODE, ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("geohash-record-path")
+ .displayName("Geohash Record Path")
+ .description("In the encode mode, this property specifies the record path to put the geohash value;"
+ + "in the decode mode, this property specifies the record path to retrieve the geohash value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Records that are successfully encoded or decoded will be routed to success")
+ .build();
+ public static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Records that cannot be encoded or decoded will be routed to failure")
+ .build();
+
+ public static final List<PropertyDescriptor> RECORD_PATH_PROPERTIES = Collections.unmodifiableList(Arrays.asList(
+ LATITUDE_RECORD_PATH,LONGITUDE_RECORD_PATH,GEOHASH_RECORD_PATH
+ ));
+
+ private List<PropertyDescriptor> descriptors;
+
+ private Set<Relationship> relationships;
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ descriptors = new ArrayList<>();
+ descriptors.add(MODE);
+ descriptors.add(RECORD_READER);
+ descriptors.add(RECORD_WRITER);
+ descriptors.add(LATITUDE_RECORD_PATH);
+ descriptors.add(LONGITUDE_RECORD_PATH);
+ descriptors.add(GEOHASH_FORMAT);
+ descriptors.add(GEOHASH_LEVEL);
+ descriptors.add(GEOHASH_RECORD_PATH);
+ descriptors = Collections.unmodifiableList(descriptors);
+
+ relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ relationships.add(REL_FAILURE);
+ relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) {
+ FlowFile input = session.get();
+ if ( input == null ) {
+ return;
+ }
+
+ final RecordReaderFactory readerFactory = context.getProperty(RECORD_READER).asControllerService(RecordReaderFactory.class);
+ final RecordSetWriterFactory writerFactory = context.getProperty(RECORD_WRITER).asControllerService(RecordSetWriterFactory.class);
+ final boolean encode = context.getProperty(MODE).getValue().equalsIgnoreCase(ENCODE_MODE);
+ final String format = context.getProperty(GEOHASH_FORMAT).getValue();
+
+ FlowFile output = session.create(input);
+ try (InputStream is = session.read(input);
+ OutputStream os = session.write(output)) {
+ RecordPathCache cache = new RecordPathCache(RECORD_PATH_PROPERTIES.size() + 1);
+ Map<PropertyDescriptor, RecordPath> paths = new HashMap<>();
+
+ for (PropertyDescriptor descriptor : RECORD_PATH_PROPERTIES) {
+ String rawRecordPath = context.getProperty(descriptor).evaluateAttributeExpressions(input).getValue();
+ RecordPath compiled = cache.getCompiled(rawRecordPath);
+ paths.put(descriptor, compiled);
+ }
+
+ RecordReader reader = readerFactory.createRecordReader(input, is, getLogger());
+ RecordSchema schema = writerFactory.getSchema(input.getAttributes(), reader.getSchema());
+ RecordSetWriter writer = writerFactory.createWriter(getLogger(), schema, os, output);
+
+ Record record;
+ Relationship targetRelationship = REL_FAILURE;
+ writer.beginRecordSet();
+
+ int recordsCount = 0;
+ boolean updated = false;
+ while ((record = reader.nextRecord()) != null) {
+ if(encode) {
+ final int level = Integer.parseInt(context.getProperty(GEOHASH_LEVEL).getValue());
+ final String rawLatitudePath = context.getProperty(LATITUDE_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath latitudePath = cache.getCompiled(rawLatitudePath);
+ final String rawLongitudePath = context.getProperty(LONGITUDE_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath longitudePath = cache.getCompiled(rawLongitudePath);
+
+ Object encodedGeohash = getEncodedGeohash(latitudePath, longitudePath, record, format, level);
+
+ updated = updateRecord(GEOHASH_RECORD_PATH, encodedGeohash, record, paths);
+ }else {
+ final String rawgeohashPath = context.getProperty(GEOHASH_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath geohashPath = cache.getCompiled(rawgeohashPath);
+
+ WGS84Point decodedPoint = getDecodedPointFromGeohash(geohashPath, record, format);
+
+ if(decodedPoint != null) {
+ updated = updateRecord(LATITUDE_RECORD_PATH, String.valueOf(decodedPoint.getLatitude()), record, paths)
+ && updateRecord(LONGITUDE_RECORD_PATH, String.valueOf(decodedPoint.getLongitude()), record, paths);
+ }
+ }
+
+ if (updated) {
Review comment:
As this is written, the destination relationship for ALL records will be either 'success' or 'failure' depending on whether or not the last Record in the FlowFile is successful. We should route the original FlowFile to 'failure' if any Record cannot be enriched, OR we should route each individual Record either to 'success' or 'failure'. Or we should simply enrich the records that we could enrich and skip others. I would lean toward providing a property to allow the user to decide. I can definitely envision a use case where users want to say "I expect all records to be enriched. Otherwise, something is wrong and I want the whole FlowFile to be failed." And I can also imagine other use cases where the user will say "Some of the Records have the latitude/longitude elements and some don't. If the elements are there, enrich them, otherwise skip them." So a property gives the user the freedom to configure it as necessary for their use case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] mikayla-yang commented on pull request #5476: NIFI-9259: Adding GeohashRecord Processor
Posted by GitBox <gi...@apache.org>.
mikayla-yang commented on pull request #5476:
URL: https://github.com/apache/nifi/pull/5476#issuecomment-957072411
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] mikayla-yang commented on a change in pull request #5476: NIFI-9259: Adding GeohashRecord Processor
Posted by GitBox <gi...@apache.org>.
mikayla-yang commented on a change in pull request #5476:
URL: https://github.com/apache/nifi/pull/5476#discussion_r740696092
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,367 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+ public static final String ENCODE_MODE = "Encode";
+ public static final String DECODE_MODE = "Decode";
+
+ public static final String BINARY_STRING = "BinaryString";
+ public static final String BASE32 = "Base32";
+ public static final String LONG_INT = "LongInt";
+
+ public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
+ .name("mode")
+ .displayName("Mode")
+ .description("Specifies whether to encode latitude/longitude to geohash or decode geohash to latitude/longitude")
+ .required(true)
+ .allowableValues(ENCODE_MODE, DECODE_MODE)
+ .defaultValue(ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor RECORD_READER = new PropertyDescriptor.Builder()
+ .name("record-reader")
+ .displayName("Record Reader")
+ .description("Specifies the record reader service to use for reading incoming data")
+ .required(true)
+ .identifiesControllerService(RecordReaderFactory.class)
+ .build();
+
+ public static final PropertyDescriptor RECORD_WRITER = new PropertyDescriptor.Builder()
+ .name("record-writer")
+ .displayName("Record Writer")
+ .description("Specifies the record writer service to use for writing data")
+ .required(true)
+ .identifiesControllerService(RecordSetWriterFactory.class)
+ .build();
+
+ public static final PropertyDescriptor LATITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("latitude-record-path")
+ .displayName("Latitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the latitude value;"
+ + "in the decode mode, this property specifies the record path to put the latitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor LONGITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("longitude-record-path")
+ .displayName("Longitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the longitude value;"
+ + "in the decode mode, this property specifies the record path to put the longitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_FORMAT = new PropertyDescriptor.Builder()
+ .name("geohash-format")
+ .displayName("Geohash Format")
+ .description("In the encode mode, this property specifies the desired format for encoding geohash;"
+ + "in the decode mode, this property specifies the format of geohash provided")
+ .required(true)
+ .allowableValues(BASE32, BINARY_STRING, LONG_INT)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_LEVEL = new PropertyDescriptor.Builder()
+ .name("geohash-level")
+ .displayName("Geohash Level")
+ .description("The integer precision level(1-12) desired for encoding geohash")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(MODE, ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("geohash-record-path")
+ .displayName("Geohash Record Path")
+ .description("In the encode mode, this property specifies the record path to put the geohash value;"
+ + "in the decode mode, this property specifies the record path to retrieve the geohash value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Records that are successfully encoded or decoded will be routed to success")
+ .build();
+ public static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Records that cannot be encoded or decoded will be routed to failure")
+ .build();
+
+ public static final List<PropertyDescriptor> RECORD_PATH_PROPERTIES = Collections.unmodifiableList(Arrays.asList(
+ LATITUDE_RECORD_PATH,LONGITUDE_RECORD_PATH,GEOHASH_RECORD_PATH
+ ));
+
+ private List<PropertyDescriptor> descriptors;
+
+ private Set<Relationship> relationships;
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ descriptors = new ArrayList<>();
+ descriptors.add(MODE);
+ descriptors.add(RECORD_READER);
+ descriptors.add(RECORD_WRITER);
+ descriptors.add(LATITUDE_RECORD_PATH);
+ descriptors.add(LONGITUDE_RECORD_PATH);
+ descriptors.add(GEOHASH_FORMAT);
+ descriptors.add(GEOHASH_LEVEL);
+ descriptors.add(GEOHASH_RECORD_PATH);
+ descriptors = Collections.unmodifiableList(descriptors);
+
+ relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ relationships.add(REL_FAILURE);
+ relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) {
+ FlowFile input = session.get();
+ if ( input == null ) {
+ return;
+ }
+
+ final RecordReaderFactory readerFactory = context.getProperty(RECORD_READER).asControllerService(RecordReaderFactory.class);
+ final RecordSetWriterFactory writerFactory = context.getProperty(RECORD_WRITER).asControllerService(RecordSetWriterFactory.class);
+ final boolean encode = context.getProperty(MODE).getValue().equalsIgnoreCase(ENCODE_MODE);
+ final String format = context.getProperty(GEOHASH_FORMAT).getValue();
+
+ FlowFile output = session.create(input);
+ try (InputStream is = session.read(input);
+ OutputStream os = session.write(output)) {
+ RecordPathCache cache = new RecordPathCache(RECORD_PATH_PROPERTIES.size() + 1);
+ Map<PropertyDescriptor, RecordPath> paths = new HashMap<>();
+
+ for (PropertyDescriptor descriptor : RECORD_PATH_PROPERTIES) {
+ String rawRecordPath = context.getProperty(descriptor).evaluateAttributeExpressions(input).getValue();
+ RecordPath compiled = cache.getCompiled(rawRecordPath);
+ paths.put(descriptor, compiled);
+ }
+
+ RecordReader reader = readerFactory.createRecordReader(input, is, getLogger());
+ RecordSchema schema = writerFactory.getSchema(input.getAttributes(), reader.getSchema());
+ RecordSetWriter writer = writerFactory.createWriter(getLogger(), schema, os, output);
Review comment:
All the stream, reader and writer now are declared in the try-with-resources statement for automatically closing them.
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,367 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+ public static final String ENCODE_MODE = "Encode";
+ public static final String DECODE_MODE = "Decode";
+
+ public static final String BINARY_STRING = "BinaryString";
+ public static final String BASE32 = "Base32";
+ public static final String LONG_INT = "LongInt";
+
+ public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
+ .name("mode")
+ .displayName("Mode")
+ .description("Specifies whether to encode latitude/longitude to geohash or decode geohash to latitude/longitude")
+ .required(true)
+ .allowableValues(ENCODE_MODE, DECODE_MODE)
+ .defaultValue(ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor RECORD_READER = new PropertyDescriptor.Builder()
+ .name("record-reader")
+ .displayName("Record Reader")
+ .description("Specifies the record reader service to use for reading incoming data")
+ .required(true)
+ .identifiesControllerService(RecordReaderFactory.class)
+ .build();
+
+ public static final PropertyDescriptor RECORD_WRITER = new PropertyDescriptor.Builder()
+ .name("record-writer")
+ .displayName("Record Writer")
+ .description("Specifies the record writer service to use for writing data")
+ .required(true)
+ .identifiesControllerService(RecordSetWriterFactory.class)
+ .build();
+
+ public static final PropertyDescriptor LATITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("latitude-record-path")
+ .displayName("Latitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the latitude value;"
+ + "in the decode mode, this property specifies the record path to put the latitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor LONGITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("longitude-record-path")
+ .displayName("Longitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the longitude value;"
+ + "in the decode mode, this property specifies the record path to put the longitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_FORMAT = new PropertyDescriptor.Builder()
+ .name("geohash-format")
+ .displayName("Geohash Format")
+ .description("In the encode mode, this property specifies the desired format for encoding geohash;"
+ + "in the decode mode, this property specifies the format of geohash provided")
+ .required(true)
+ .allowableValues(BASE32, BINARY_STRING, LONG_INT)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_LEVEL = new PropertyDescriptor.Builder()
+ .name("geohash-level")
+ .displayName("Geohash Level")
+ .description("The integer precision level(1-12) desired for encoding geohash")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(MODE, ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("geohash-record-path")
+ .displayName("Geohash Record Path")
+ .description("In the encode mode, this property specifies the record path to put the geohash value;"
+ + "in the decode mode, this property specifies the record path to retrieve the geohash value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Records that are successfully encoded or decoded will be routed to success")
+ .build();
+ public static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Records that cannot be encoded or decoded will be routed to failure")
+ .build();
+
+ public static final List<PropertyDescriptor> RECORD_PATH_PROPERTIES = Collections.unmodifiableList(Arrays.asList(
+ LATITUDE_RECORD_PATH,LONGITUDE_RECORD_PATH,GEOHASH_RECORD_PATH
+ ));
+
+ private List<PropertyDescriptor> descriptors;
+
+ private Set<Relationship> relationships;
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ descriptors = new ArrayList<>();
+ descriptors.add(MODE);
+ descriptors.add(RECORD_READER);
+ descriptors.add(RECORD_WRITER);
+ descriptors.add(LATITUDE_RECORD_PATH);
+ descriptors.add(LONGITUDE_RECORD_PATH);
+ descriptors.add(GEOHASH_FORMAT);
+ descriptors.add(GEOHASH_LEVEL);
+ descriptors.add(GEOHASH_RECORD_PATH);
+ descriptors = Collections.unmodifiableList(descriptors);
+
+ relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ relationships.add(REL_FAILURE);
+ relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) {
+ FlowFile input = session.get();
+ if ( input == null ) {
+ return;
+ }
+
+ final RecordReaderFactory readerFactory = context.getProperty(RECORD_READER).asControllerService(RecordReaderFactory.class);
+ final RecordSetWriterFactory writerFactory = context.getProperty(RECORD_WRITER).asControllerService(RecordSetWriterFactory.class);
+ final boolean encode = context.getProperty(MODE).getValue().equalsIgnoreCase(ENCODE_MODE);
+ final String format = context.getProperty(GEOHASH_FORMAT).getValue();
+
+ FlowFile output = session.create(input);
+ try (InputStream is = session.read(input);
+ OutputStream os = session.write(output)) {
+ RecordPathCache cache = new RecordPathCache(RECORD_PATH_PROPERTIES.size() + 1);
+ Map<PropertyDescriptor, RecordPath> paths = new HashMap<>();
+
+ for (PropertyDescriptor descriptor : RECORD_PATH_PROPERTIES) {
+ String rawRecordPath = context.getProperty(descriptor).evaluateAttributeExpressions(input).getValue();
+ RecordPath compiled = cache.getCompiled(rawRecordPath);
+ paths.put(descriptor, compiled);
+ }
+
+ RecordReader reader = readerFactory.createRecordReader(input, is, getLogger());
+ RecordSchema schema = writerFactory.getSchema(input.getAttributes(), reader.getSchema());
+ RecordSetWriter writer = writerFactory.createWriter(getLogger(), schema, os, output);
+
+ Record record;
+ Relationship targetRelationship = REL_FAILURE;
+ writer.beginRecordSet();
+
+ int recordsCount = 0;
+ boolean updated = false;
+ while ((record = reader.nextRecord()) != null) {
+ if(encode) {
+ final int level = Integer.parseInt(context.getProperty(GEOHASH_LEVEL).getValue());
+ final String rawLatitudePath = context.getProperty(LATITUDE_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath latitudePath = cache.getCompiled(rawLatitudePath);
+ final String rawLongitudePath = context.getProperty(LONGITUDE_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath longitudePath = cache.getCompiled(rawLongitudePath);
+
+ Object encodedGeohash = getEncodedGeohash(latitudePath, longitudePath, record, format, level);
+
+ updated = updateRecord(GEOHASH_RECORD_PATH, encodedGeohash, record, paths);
+ }else {
+ final String rawgeohashPath = context.getProperty(GEOHASH_RECORD_PATH).evaluateAttributeExpressions(input).getValue();
+ RecordPath geohashPath = cache.getCompiled(rawgeohashPath);
+
+ WGS84Point decodedPoint = getDecodedPointFromGeohash(geohashPath, record, format);
+
+ if(decodedPoint != null) {
+ updated = updateRecord(LATITUDE_RECORD_PATH, String.valueOf(decodedPoint.getLatitude()), record, paths)
+ && updateRecord(LONGITUDE_RECORD_PATH, String.valueOf(decodedPoint.getLongitude()), record, paths);
+ }
+ }
+
+ if (updated) {
Review comment:
I have added a ROUTING_STRATEGY property so that users can specify how to route the flowfile, which I believe covers all the user cases you mentioned:
- _Skip_Unenriched_: "Some of the Records have the latitude/longitude elements and some don't. If the elements are there, enrich them, otherwise skip them."
- _Require_All_Enriched_: “I expect all records to be enriched. Otherwise, something is wrong and I want the whole FlowFile to be failed."
- _Split_Enriched_Unenriched_: split the records that are enriched from those are not. The flowfile including successfully enriched records will be sent to ‘success’, the flowfile including records that are unable to be enriched will be sent to ‘failure’, and the original flowfile will be sent to ‘original’.
Just one quick note on the original code here, the destination relationship for all records will be 'success' as long as one record has been updated, which is identical to the current Skip_Unenriched strategy.
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,367 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+ public static final String ENCODE_MODE = "Encode";
+ public static final String DECODE_MODE = "Decode";
+
+ public static final String BINARY_STRING = "BinaryString";
+ public static final String BASE32 = "Base32";
+ public static final String LONG_INT = "LongInt";
+
+ public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
+ .name("mode")
+ .displayName("Mode")
+ .description("Specifies whether to encode latitude/longitude to geohash or decode geohash to latitude/longitude")
+ .required(true)
+ .allowableValues(ENCODE_MODE, DECODE_MODE)
+ .defaultValue(ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor RECORD_READER = new PropertyDescriptor.Builder()
+ .name("record-reader")
+ .displayName("Record Reader")
+ .description("Specifies the record reader service to use for reading incoming data")
+ .required(true)
+ .identifiesControllerService(RecordReaderFactory.class)
+ .build();
+
+ public static final PropertyDescriptor RECORD_WRITER = new PropertyDescriptor.Builder()
+ .name("record-writer")
+ .displayName("Record Writer")
+ .description("Specifies the record writer service to use for writing data")
+ .required(true)
+ .identifiesControllerService(RecordSetWriterFactory.class)
+ .build();
+
+ public static final PropertyDescriptor LATITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("latitude-record-path")
+ .displayName("Latitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the latitude value;"
+ + "in the decode mode, this property specifies the record path to put the latitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor LONGITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("longitude-record-path")
+ .displayName("Longitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the longitude value;"
+ + "in the decode mode, this property specifies the record path to put the longitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_FORMAT = new PropertyDescriptor.Builder()
+ .name("geohash-format")
+ .displayName("Geohash Format")
+ .description("In the encode mode, this property specifies the desired format for encoding geohash;"
+ + "in the decode mode, this property specifies the format of geohash provided")
+ .required(true)
+ .allowableValues(BASE32, BINARY_STRING, LONG_INT)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_LEVEL = new PropertyDescriptor.Builder()
+ .name("geohash-level")
+ .displayName("Geohash Level")
+ .description("The integer precision level(1-12) desired for encoding geohash")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(MODE, ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("geohash-record-path")
+ .displayName("Geohash Record Path")
+ .description("In the encode mode, this property specifies the record path to put the geohash value;"
+ + "in the decode mode, this property specifies the record path to retrieve the geohash value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Records that are successfully encoded or decoded will be routed to success")
+ .build();
+ public static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Records that cannot be encoded or decoded will be routed to failure")
+ .build();
+
+ public static final List<PropertyDescriptor> RECORD_PATH_PROPERTIES = Collections.unmodifiableList(Arrays.asList(
+ LATITUDE_RECORD_PATH,LONGITUDE_RECORD_PATH,GEOHASH_RECORD_PATH
+ ));
+
+ private List<PropertyDescriptor> descriptors;
+
+ private Set<Relationship> relationships;
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ descriptors = new ArrayList<>();
+ descriptors.add(MODE);
+ descriptors.add(RECORD_READER);
+ descriptors.add(RECORD_WRITER);
+ descriptors.add(LATITUDE_RECORD_PATH);
+ descriptors.add(LONGITUDE_RECORD_PATH);
+ descriptors.add(GEOHASH_FORMAT);
+ descriptors.add(GEOHASH_LEVEL);
+ descriptors.add(GEOHASH_RECORD_PATH);
+ descriptors = Collections.unmodifiableList(descriptors);
+
+ relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ relationships.add(REL_FAILURE);
+ relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) {
+ FlowFile input = session.get();
+ if ( input == null ) {
+ return;
+ }
+
+ final RecordReaderFactory readerFactory = context.getProperty(RECORD_READER).asControllerService(RecordReaderFactory.class);
+ final RecordSetWriterFactory writerFactory = context.getProperty(RECORD_WRITER).asControllerService(RecordSetWriterFactory.class);
+ final boolean encode = context.getProperty(MODE).getValue().equalsIgnoreCase(ENCODE_MODE);
+ final String format = context.getProperty(GEOHASH_FORMAT).getValue();
+
+ FlowFile output = session.create(input);
Review comment:
Now a ROUTING_STRATEGY property is added, and it includes three strategy options for users to specify. We need to process the InputStream and OuputStream separately when using the _Split_Enriched_Unenriched_ strategy.
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,439 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.record.path.validation.RecordPathValidator;
+
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.WriteResult;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+
+ public enum ProcessingMode {
+ Encode, Decode
+ }
+
+ public enum GeohashFormat {
+ Base32, Binary_String, Long_Int
Review comment:
I'll change it to LONG here, because the integer version of Geohash should be a 64-bit integer. So I think LONG makes more sense compared to NUMBER.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] mikayla-yang commented on a change in pull request #5476: NIFI-9259: Adding GeohashRecord Processor
Posted by GitBox <gi...@apache.org>.
mikayla-yang commented on a change in pull request #5476:
URL: https://github.com/apache/nifi/pull/5476#discussion_r740696092
##########
File path: nifi-nar-bundles/nifi-geohash-bundle/nifi-geohash-processors/src/main/java/org/apache/nifi/processors/geohash/GeohashRecord.java
##########
@@ -0,0 +1,367 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.geohash;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SideEffectFree;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+import ch.hsr.geohash.GeoHash;
+import ch.hsr.geohash.WGS84Point;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.Optional;
+
+@EventDriven
+@SideEffectFree
+@SupportsBatching
+@Tags({"geo", "geohash", "record"})
+@CapabilityDescription("A record-based processor that encodes and decodes Geohashes from and to latitude/longitude coordinates.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class GeohashRecord extends AbstractProcessor {
+ public static final String ENCODE_MODE = "Encode";
+ public static final String DECODE_MODE = "Decode";
+
+ public static final String BINARY_STRING = "BinaryString";
+ public static final String BASE32 = "Base32";
+ public static final String LONG_INT = "LongInt";
+
+ public static final PropertyDescriptor MODE = new PropertyDescriptor.Builder()
+ .name("mode")
+ .displayName("Mode")
+ .description("Specifies whether to encode latitude/longitude to geohash or decode geohash to latitude/longitude")
+ .required(true)
+ .allowableValues(ENCODE_MODE, DECODE_MODE)
+ .defaultValue(ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor RECORD_READER = new PropertyDescriptor.Builder()
+ .name("record-reader")
+ .displayName("Record Reader")
+ .description("Specifies the record reader service to use for reading incoming data")
+ .required(true)
+ .identifiesControllerService(RecordReaderFactory.class)
+ .build();
+
+ public static final PropertyDescriptor RECORD_WRITER = new PropertyDescriptor.Builder()
+ .name("record-writer")
+ .displayName("Record Writer")
+ .description("Specifies the record writer service to use for writing data")
+ .required(true)
+ .identifiesControllerService(RecordSetWriterFactory.class)
+ .build();
+
+ public static final PropertyDescriptor LATITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("latitude-record-path")
+ .displayName("Latitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the latitude value;"
+ + "in the decode mode, this property specifies the record path to put the latitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor LONGITUDE_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("longitude-record-path")
+ .displayName("Longitude Record Path")
+ .description("In the encode mode, this property specifies the record path to retrieve the longitude value;"
+ + "in the decode mode, this property specifies the record path to put the longitude value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_FORMAT = new PropertyDescriptor.Builder()
+ .name("geohash-format")
+ .displayName("Geohash Format")
+ .description("In the encode mode, this property specifies the desired format for encoding geohash;"
+ + "in the decode mode, this property specifies the format of geohash provided")
+ .required(true)
+ .allowableValues(BASE32, BINARY_STRING, LONG_INT)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_LEVEL = new PropertyDescriptor.Builder()
+ .name("geohash-level")
+ .displayName("Geohash Level")
+ .description("The integer precision level(1-12) desired for encoding geohash")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(MODE, ENCODE_MODE)
+ .build();
+
+ public static final PropertyDescriptor GEOHASH_RECORD_PATH = new PropertyDescriptor.Builder()
+ .name("geohash-record-path")
+ .displayName("Geohash Record Path")
+ .description("In the encode mode, this property specifies the record path to put the geohash value;"
+ + "in the decode mode, this property specifies the record path to retrieve the geohash value")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Records that are successfully encoded or decoded will be routed to success")
+ .build();
+ public static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Records that cannot be encoded or decoded will be routed to failure")
+ .build();
+
+ public static final List<PropertyDescriptor> RECORD_PATH_PROPERTIES = Collections.unmodifiableList(Arrays.asList(
+ LATITUDE_RECORD_PATH,LONGITUDE_RECORD_PATH,GEOHASH_RECORD_PATH
+ ));
+
+ private List<PropertyDescriptor> descriptors;
+
+ private Set<Relationship> relationships;
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ descriptors = new ArrayList<>();
+ descriptors.add(MODE);
+ descriptors.add(RECORD_READER);
+ descriptors.add(RECORD_WRITER);
+ descriptors.add(LATITUDE_RECORD_PATH);
+ descriptors.add(LONGITUDE_RECORD_PATH);
+ descriptors.add(GEOHASH_FORMAT);
+ descriptors.add(GEOHASH_LEVEL);
+ descriptors.add(GEOHASH_RECORD_PATH);
+ descriptors = Collections.unmodifiableList(descriptors);
+
+ relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ relationships.add(REL_FAILURE);
+ relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) {
+ FlowFile input = session.get();
+ if ( input == null ) {
+ return;
+ }
+
+ final RecordReaderFactory readerFactory = context.getProperty(RECORD_READER).asControllerService(RecordReaderFactory.class);
+ final RecordSetWriterFactory writerFactory = context.getProperty(RECORD_WRITER).asControllerService(RecordSetWriterFactory.class);
+ final boolean encode = context.getProperty(MODE).getValue().equalsIgnoreCase(ENCODE_MODE);
+ final String format = context.getProperty(GEOHASH_FORMAT).getValue();
+
+ FlowFile output = session.create(input);
+ try (InputStream is = session.read(input);
+ OutputStream os = session.write(output)) {
+ RecordPathCache cache = new RecordPathCache(RECORD_PATH_PROPERTIES.size() + 1);
+ Map<PropertyDescriptor, RecordPath> paths = new HashMap<>();
+
+ for (PropertyDescriptor descriptor : RECORD_PATH_PROPERTIES) {
+ String rawRecordPath = context.getProperty(descriptor).evaluateAttributeExpressions(input).getValue();
+ RecordPath compiled = cache.getCompiled(rawRecordPath);
+ paths.put(descriptor, compiled);
+ }
+
+ RecordReader reader = readerFactory.createRecordReader(input, is, getLogger());
+ RecordSchema schema = writerFactory.getSchema(input.getAttributes(), reader.getSchema());
+ RecordSetWriter writer = writerFactory.createWriter(getLogger(), schema, os, output);
Review comment:
All the stream, reader and writer now are declared in the try-with-resources statement for automatically closing them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org