You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by GitBox <gi...@apache.org> on 2022/08/15 18:27:34 UTC
[GitHub] [nifi] Lehel44 opened a new pull request, #6301: NIFI-10356: Create GetHubSpot processor
Lehel44 opened a new pull request, #6301:
URL: https://github.com/apache/nifi/pull/6301
<!-- Licensed to the Apache Software Foundation (ASF) under one or more -->
<!-- contributor license agreements. See the NOTICE file distributed with -->
<!-- this work for additional information regarding copyright ownership. -->
<!-- The ASF licenses this file to You under the Apache License, Version 2.0 -->
<!-- (the "License"); you may not use this file except in compliance with -->
<!-- the License. You may obtain a copy of the License at -->
<!-- http://www.apache.org/licenses/LICENSE-2.0 -->
<!-- Unless required by applicable law or agreed to in writing, software -->
<!-- distributed under the License is distributed on an "AS IS" BASIS, -->
<!-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -->
<!-- See the License for the specific language governing permissions and -->
<!-- limitations under the License. -->
# Summary
[NIFI-10356](https://issues.apache.org/jira/browse/NIFI-10356)
# Tracking
Please complete the following tracking steps prior to pull request creation.
### Issue Tracking
- [ ] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI-10356) issue created
### Pull Request Tracking
- [ ] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-00000`
- [ ] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-00000`
### Pull Request Formatting
- [ ] Pull Request based on current revision of the `main` branch
- [ ] Pull Request refers to a feature branch with one commit containing changes
# Verification
Please indicate the verification steps performed prior to pull request creation.
### Build
- [ ] Build completed using `mvn clean install -P contrib-check`
- [ ] JDK 8
- [ ] JDK 11
- [ ] JDK 17
### Licensing
- [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html)
- [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files
### Documentation
- [ ] Documentation formatting appears as expected in rendered files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] Lehel44 commented on a diff in pull request #6301: NIFI-10356: Create GetHubSpot processor
Posted by GitBox <gi...@apache.org>.
Lehel44 commented on code in PR #6301:
URL: https://github.com/apache/nifi/pull/6301#discussion_r952586880
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.configuration.DefaultSettings;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"hubspot"})
+@CapabilityDescription("Retrieves JSON data from a private HubSpot application."
+ + " Supports incremental retrieval: Users can set the \"limit\" property which serves as the upper limit of the retrieved objects."
+ + " When this property is set the processor will retrieve new records. This processor is intended to be run on the Primary Node only.")
+@Stateful(scopes = Scope.CLUSTER, description = "When the 'Limit' attribute is set, the paging cursor is saved after executing a request."
+ + " Only the objects after the paging cursor will be retrieved. The maximum number of retrieved objects is the 'Limit' attribute."
+ + " State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected,"
+ + " the new node can pick up where the previous node left off, without duplicating the data.")
+@DefaultSettings(yieldDuration = "10 sec")
+public class GetHubSpot extends AbstractProcessor {
+
+ static final PropertyDescriptor ACCESS_TOKEN = new PropertyDescriptor.Builder()
+ .name("admin-api-access-token")
+ .displayName("Access Token")
+ .description("Access Token to authenticate requests")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+ .build();
+
+ static final PropertyDescriptor CRM_ENDPOINT = new PropertyDescriptor.Builder()
+ .name("crm-endpoint")
+ .displayName("HubSpot CRM API Endpoint")
+ .description("The HubSpot CRM API endpoint to get")
+ .required(true)
+ .allowableValues(CrmEndpoint.class)
+ .build();
+
+ static final PropertyDescriptor LIMIT = new PropertyDescriptor.Builder()
+ .name("crm-limit")
+ .displayName("Limit")
+ .description("The maximum number of results to display per page")
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor WEB_CLIENT_SERVICE_PROVIDER = new PropertyDescriptor.Builder()
+ .name("web-client-service-provider")
+ .displayName("Web Client Service Provider")
+ .description("Controller service for HTTP client operations")
+ .identifiesControllerService(WebClientServiceProvider.class)
+ .required(true)
+ .build();
+
+ static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("For FlowFiles created as a result of a successful HTTP request.")
+ .build();
+
+ static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("In case of HTTP client errors the flowfile will be routed to this relationship")
+ .build();
+
+ private static final String API_BASE_URI = "api.hubapi.com";
+ private static final String HTTPS = "https";
+ private static final String CURSOR_PARAMETER = "after";
+ private static final String LIMIT_PARAMETER = "limit";
+ private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
+ private static final JsonFactory JSON_FACTORY = OBJECT_MAPPER.getFactory();
+
+ private volatile WebClientServiceProvider webClientServiceProvider;
+
+ private static final List<PropertyDescriptor> PROPERTY_DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ ACCESS_TOKEN,
+ CRM_ENDPOINT,
+ LIMIT,
+ WEB_CLIENT_SERVICE_PROVIDER
+ ));
+
+ private static final Set<Relationship> RELATIONSHIPS = Collections.unmodifiableSet(new HashSet<>(Arrays.asList(
+ REL_SUCCESS,
+ REL_FAILURE
+ )));
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ webClientServiceProvider = context.getProperty(WEB_CLIENT_SERVICE_PROVIDER).asControllerService(WebClientServiceProvider.class);
+ }
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTY_DESCRIPTORS;
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return RELATIONSHIPS;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ final String accessToken = context.getProperty(ACCESS_TOKEN).getValue();
+ final String endpoint = context.getProperty(CRM_ENDPOINT).getValue();
+
+ final StateMap state = getStateMap(context);
+ final URI uri = createUri(context, state);
+
+ final HttpResponseEntity response = getHttpResponseEntity(accessToken, uri);
+ final AtomicInteger objectCountHolder = new AtomicInteger();
+
+ FlowFile flowFile = session.create();
+ flowFile = session.putAttribute(flowFile, "statusCode", String.valueOf(response.statusCode()));
+
+ flowFile = session.write(flowFile, out -> {
+
+ try (JsonParser jsonParser = JSON_FACTORY.createParser(response.body());
+ final JsonGenerator jsonGenerator = JSON_FACTORY.createGenerator(out, JsonEncoding.UTF8)) {
+ while (jsonParser.nextToken() != null) {
+ if (jsonParser.getCurrentToken() == JsonToken.FIELD_NAME && jsonParser.getCurrentName().equals("results")) {
+ jsonParser.nextToken();
+ jsonGenerator.copyCurrentStructure(jsonParser);
+ objectCountHolder.incrementAndGet();
+ }
+ String fieldName = jsonParser.getCurrentName();
+ if (CURSOR_PARAMETER.equals(fieldName)) {
+ jsonParser.nextToken();
+ Map<String, String> newStateMap = new HashMap<>(state.toMap());
+ newStateMap.put(endpoint, jsonParser.getText());
+ updateState(context, newStateMap);
+ break;
+ }
+ }
+ }
+ });
+ if (response.statusCode() >= 400) {
+ if (response.statusCode() == 429) {
+ context.yield();
+ throw new ProcessException("Rate limit exceeded, yielding for 10 seconds before retrying request.");
Review Comment:
I think changing the default configuration is the user's responsibility as we are not aware of what kind of accounts are used and they come with different rate limits. The current default yield time is adjusted to HubSpot Starter Account's Rate Limit (100 requests/10sec). The exception is thrown to retry the request after yielding in case of HTTP 429 (reaching rate limit).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] Lehel44 commented on a diff in pull request #6301: NIFI-10356: Create GetHubSpot processor
Posted by GitBox <gi...@apache.org>.
Lehel44 commented on code in PR #6301:
URL: https://github.com/apache/nifi/pull/6301#discussion_r956753475
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.configuration.DefaultSettings;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpResponseStatus;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"hubspot"})
+@CapabilityDescription("Retrieves JSON data from a private HubSpot application."
+ + " Supports incremental retrieval: Users can set the \"limit\" property which serves as the upper limit of the retrieved objects."
+ + " When this property is set the processor will retrieve new records. This processor is intended to be run on the Primary Node only.")
+@Stateful(scopes = Scope.CLUSTER, description = "When the 'Limit' attribute is set, the paging cursor is saved after executing a request."
+ + " Only the objects after the paging cursor will be retrieved. The maximum number of retrieved objects is the 'Limit' attribute."
+ + " State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected,"
+ + " the new node can pick up where the previous node left off, without duplicating the data.")
+@DefaultSettings(yieldDuration = "10 sec")
+public class GetHubSpot extends AbstractProcessor {
Review Comment:
I don't think it's worth creating another processor if more endpoints are added later. I renamed the Crm Object Type to Object Type to make it more generic but left the CrmEndpoint class as it is for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #6301: NIFI-10356: Create GetHubSpot processor
Posted by GitBox <gi...@apache.org>.
exceptionfactory commented on code in PR #6301:
URL: https://github.com/apache/nifi/pull/6301#discussion_r946210770
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,320 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"hubspot"})
+@CapabilityDescription("Retrieves JSON data from a private HubSpot application."
+ + " Supports incremental retrieval: Users can set the \"limit\" property which serves as the upper limit of the retrieved objects."
+ + " When this property is set the processor will retrieve new records. This processor is intended to be run on the Primary Node only.")
+@Stateful(scopes = Scope.CLUSTER, description = "When the 'Limit' attribute is set, the paging cursor is saved after executing a request."
+ + " Only the objects after the paging cursor will be retrieved. The maximum number of retrieved objects is the 'Limit' attribute."
+ + " State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected,"
+ + " the new node can pick up where the previous node left off, without duplicating the data.")
+public class GetHubSpot extends AbstractProcessor {
+
+ // OBJECTS
+
+ static final AllowableValue COMPANIES = new AllowableValue(
+ "/crm/v3/objects/companies",
+ "Companies",
+ "In HubSpot, the companies object is a standard CRM object. Individual company records can be used to store information about businesses" +
+ " and organizations within company properties."
+ );
+ static final AllowableValue CONTACTS = new AllowableValue(
+ "/crm/v3/objects/contacts",
+ "Contacts",
+ "In HubSpot, contacts store information about individuals. From marketing automation to smart content, the lead-specific data found in" +
+ " contact records helps users leverage much of HubSpot's functionality."
+ );
+ static final AllowableValue DEALS = new AllowableValue(
+ "/crm/v3/objects/deals",
+ "In HubSpot, a deal represents an ongoing transaction that a sales team is pursuing with a contact or company. It’s tracked through" +
+ " pipeline stages until won or lost."
+ );
+ static final AllowableValue FEEDBACK_SUBMISSIONS = new AllowableValue(
+ "/crm/v3/objects/feedback_submissions",
+ "In HubSpot, feedback submissions are an object which stores information submitted to a feedback survey. This includes Net Promoter Score (NPS)," +
+ " Customer Satisfaction (CSAT), Customer Effort Score (CES) and Custom Surveys."
+ );
+ static final AllowableValue LINE_ITEMS = new AllowableValue(
+ "/crm/v3/objects/line_items",
+ "Line Items",
+ "In HubSpot, line items can be thought of as a subset of products. When a product is attached to a deal, it becomes a line item. Line items can" +
+ " be created that are unique to an individual quote, but they will not be added to the product library."
+ );
+ static final AllowableValue PRODUCTS = new AllowableValue(
+ "/crm/v3/objects/products",
+ "Products",
+ "In HubSpot, products represent the goods or services to be sold. Building a product library allows the user to quickly add products to deals," +
+ " generate quotes, and report on product performance."
+ );
+ static final AllowableValue TICKETS = new AllowableValue(
+ "/crm/v3/objects/tickets",
+ "Tickets",
+ "In HubSpot, a ticket represents a customer request for help or support."
+ );
+ static final AllowableValue QUOTES = new AllowableValue(
+ "/crm/v3/objects/quotes",
+ "Quotes",
+ "In HubSpot, quotes are used to share pricing information with potential buyers."
+ );
+
+ // ENGAGEMENTS
+
+ private static final AllowableValue CALLS = new AllowableValue(
+ "/crm/v3/objects/calls",
+ "Calls",
+ "Get calls on CRM records and on the calls index page."
+ );
+ private static final AllowableValue EMAILS = new AllowableValue(
+ "/crm/v3/objects/emails",
+ "Emails",
+ "Get emails on CRM records."
+ );
+ private static final AllowableValue MEETINGS = new AllowableValue(
+ "/crm/v3/objects/meetings",
+ "Meetings",
+ "Get meetings on CRM records."
+ );
+ private static final AllowableValue NOTES = new AllowableValue(
+ "/crm/v3/objects/notes",
+ "Notes",
+ "Get notes on CRM records."
+ );
+ private static final AllowableValue TASKS = new AllowableValue(
+ "/crm/v3/objects/tasks",
+ "Tasks",
+ "Get tasks on CRM records."
+ );
+
+ // OTHER
+
+ private static final AllowableValue OWNERS = new AllowableValue(
+ "/crm/v3/owners/",
+ "Owners",
+ "HubSpot uses owners to assign specific users to contacts, companies, deals, tickets, or engagements. Any HubSpot user with access to contacts" +
+ " can be assigned as an owner, and multiple owners can be assigned to an object by creating a custom property for this purpose."
+ );
+
+ static final PropertyDescriptor ACCESS_TOKEN = new PropertyDescriptor.Builder()
Review Comment:
Thanks for the feedback @Lehel44, that is good to know. In that case, the current implementation makes sense. Just a question on the wording, should the property be named `Access Token` as opposed to `Admin API Access Token`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] Lehel44 commented on a diff in pull request #6301: NIFI-10356: Create GetHubSpot processor
Posted by GitBox <gi...@apache.org>.
Lehel44 commented on code in PR #6301:
URL: https://github.com/apache/nifi/pull/6301#discussion_r956751098
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.configuration.DefaultSettings;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpResponseStatus;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"hubspot"})
+@CapabilityDescription("Retrieves JSON data from a private HubSpot application."
+ + " Supports incremental retrieval: Users can set the \"limit\" property which serves as the upper limit of the retrieved objects."
+ + " When this property is set the processor will retrieve new records. This processor is intended to be run on the Primary Node only.")
+@Stateful(scopes = Scope.CLUSTER, description = "When the 'Limit' attribute is set, the paging cursor is saved after executing a request."
+ + " Only the objects after the paging cursor will be retrieved. The maximum number of retrieved objects is the 'Limit' attribute."
+ + " State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected,"
+ + " the new node can pick up where the previous node left off, without duplicating the data.")
+@DefaultSettings(yieldDuration = "10 sec")
+public class GetHubSpot extends AbstractProcessor {
+
+ static final PropertyDescriptor CRM_ENDPOINT = new PropertyDescriptor.Builder()
+ .name("crm-endpoint")
+ .displayName("HubSpot CRM API Endpoint")
+ .description("The HubSpot CRM API endpoint to which the Processor will send requests")
Review Comment:
Thanks for the suggestion. I agree that the user does not necessarily needs to know about endpoints and APIs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] Lehel44 commented on a diff in pull request #6301: NIFI-10356: Create GetHubSpot processor
Posted by GitBox <gi...@apache.org>.
Lehel44 commented on code in PR #6301:
URL: https://github.com/apache/nifi/pull/6301#discussion_r956753475
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.configuration.DefaultSettings;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpResponseStatus;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"hubspot"})
+@CapabilityDescription("Retrieves JSON data from a private HubSpot application."
+ + " Supports incremental retrieval: Users can set the \"limit\" property which serves as the upper limit of the retrieved objects."
+ + " When this property is set the processor will retrieve new records. This processor is intended to be run on the Primary Node only.")
+@Stateful(scopes = Scope.CLUSTER, description = "When the 'Limit' attribute is set, the paging cursor is saved after executing a request."
+ + " Only the objects after the paging cursor will be retrieved. The maximum number of retrieved objects is the 'Limit' attribute."
+ + " State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected,"
+ + " the new node can pick up where the previous node left off, without duplicating the data.")
+@DefaultSettings(yieldDuration = "10 sec")
+public class GetHubSpot extends AbstractProcessor {
Review Comment:
That's a very good question. I don't think it's worth creating another processor if more endpoints are added later. I renamed the Crm Object Type to Object Type to make it more generic but left the CrmEndpoint class as it is for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #6301: NIFI-10356: Create GetHubSpot processor
Posted by GitBox <gi...@apache.org>.
exceptionfactory commented on code in PR #6301:
URL: https://github.com/apache/nifi/pull/6301#discussion_r946166142
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,320 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"hubspot"})
+@CapabilityDescription("Retrieves JSON data from a private HubSpot application."
+ + " Supports incremental retrieval: Users can set the \"limit\" property which serves as the upper limit of the retrieved objects."
+ + " When this property is set the processor will retrieve new records. This processor is intended to be run on the Primary Node only.")
+@Stateful(scopes = Scope.CLUSTER, description = "When the 'Limit' attribute is set, the paging cursor is saved after executing a request."
+ + " Only the objects after the paging cursor will be retrieved. The maximum number of retrieved objects is the 'Limit' attribute."
+ + " State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected,"
+ + " the new node can pick up where the previous node left off, without duplicating the data.")
+public class GetHubSpot extends AbstractProcessor {
+
+ // OBJECTS
+
+ static final AllowableValue COMPANIES = new AllowableValue(
+ "/crm/v3/objects/companies",
+ "Companies",
+ "In HubSpot, the companies object is a standard CRM object. Individual company records can be used to store information about businesses" +
+ " and organizations within company properties."
+ );
+ static final AllowableValue CONTACTS = new AllowableValue(
+ "/crm/v3/objects/contacts",
+ "Contacts",
+ "In HubSpot, contacts store information about individuals. From marketing automation to smart content, the lead-specific data found in" +
+ " contact records helps users leverage much of HubSpot's functionality."
+ );
+ static final AllowableValue DEALS = new AllowableValue(
+ "/crm/v3/objects/deals",
+ "In HubSpot, a deal represents an ongoing transaction that a sales team is pursuing with a contact or company. It’s tracked through" +
+ " pipeline stages until won or lost."
+ );
+ static final AllowableValue FEEDBACK_SUBMISSIONS = new AllowableValue(
+ "/crm/v3/objects/feedback_submissions",
+ "In HubSpot, feedback submissions are an object which stores information submitted to a feedback survey. This includes Net Promoter Score (NPS)," +
+ " Customer Satisfaction (CSAT), Customer Effort Score (CES) and Custom Surveys."
+ );
+ static final AllowableValue LINE_ITEMS = new AllowableValue(
+ "/crm/v3/objects/line_items",
+ "Line Items",
+ "In HubSpot, line items can be thought of as a subset of products. When a product is attached to a deal, it becomes a line item. Line items can" +
+ " be created that are unique to an individual quote, but they will not be added to the product library."
+ );
+ static final AllowableValue PRODUCTS = new AllowableValue(
+ "/crm/v3/objects/products",
+ "Products",
+ "In HubSpot, products represent the goods or services to be sold. Building a product library allows the user to quickly add products to deals," +
+ " generate quotes, and report on product performance."
+ );
+ static final AllowableValue TICKETS = new AllowableValue(
+ "/crm/v3/objects/tickets",
+ "Tickets",
+ "In HubSpot, a ticket represents a customer request for help or support."
+ );
+ static final AllowableValue QUOTES = new AllowableValue(
+ "/crm/v3/objects/quotes",
+ "Quotes",
+ "In HubSpot, quotes are used to share pricing information with potential buyers."
+ );
+
+ // ENGAGEMENTS
+
+ private static final AllowableValue CALLS = new AllowableValue(
+ "/crm/v3/objects/calls",
+ "Calls",
+ "Get calls on CRM records and on the calls index page."
+ );
+ private static final AllowableValue EMAILS = new AllowableValue(
+ "/crm/v3/objects/emails",
+ "Emails",
+ "Get emails on CRM records."
+ );
+ private static final AllowableValue MEETINGS = new AllowableValue(
+ "/crm/v3/objects/meetings",
+ "Meetings",
+ "Get meetings on CRM records."
+ );
+ private static final AllowableValue NOTES = new AllowableValue(
+ "/crm/v3/objects/notes",
+ "Notes",
+ "Get notes on CRM records."
+ );
+ private static final AllowableValue TASKS = new AllowableValue(
+ "/crm/v3/objects/tasks",
+ "Tasks",
+ "Get tasks on CRM records."
+ );
+
+ // OTHER
+
+ private static final AllowableValue OWNERS = new AllowableValue(
+ "/crm/v3/owners/",
+ "Owners",
+ "HubSpot uses owners to assign specific users to contacts, companies, deals, tickets, or engagements. Any HubSpot user with access to contacts" +
+ " can be assigned as an owner, and multiple owners can be assigned to an object by creating a custom property for this purpose."
+ );
+
+ static final PropertyDescriptor ACCESS_TOKEN = new PropertyDescriptor.Builder()
Review Comment:
The [HubSpot documentation](https://developers.hubspot.com/docs/api/working-with-oauth) indicates support for OAuth 2, as well as [Private access tokens](https://developers.hubspot.com/docs/api/intro-to-auth). At minimum, it would be helpful to have an `Authentication Strategy` property, which would provide the opportunity to configure with the access tokens, or OAuth 2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] Lehel44 commented on a diff in pull request #6301: NIFI-10356: Create GetHubSpot processor
Posted by GitBox <gi...@apache.org>.
Lehel44 commented on code in PR #6301:
URL: https://github.com/apache/nifi/pull/6301#discussion_r952496841
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.configuration.DefaultSettings;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"hubspot"})
+@CapabilityDescription("Retrieves JSON data from a private HubSpot application."
+ + " Supports incremental retrieval: Users can set the \"limit\" property which serves as the upper limit of the retrieved objects."
+ + " When this property is set the processor will retrieve new records. This processor is intended to be run on the Primary Node only.")
+@Stateful(scopes = Scope.CLUSTER, description = "When the 'Limit' attribute is set, the paging cursor is saved after executing a request."
+ + " Only the objects after the paging cursor will be retrieved. The maximum number of retrieved objects is the 'Limit' attribute."
+ + " State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected,"
+ + " the new node can pick up where the previous node left off, without duplicating the data.")
+@DefaultSettings(yieldDuration = "10 sec")
+public class GetHubSpot extends AbstractProcessor {
+
+ static final PropertyDescriptor ACCESS_TOKEN = new PropertyDescriptor.Builder()
+ .name("admin-api-access-token")
+ .displayName("Access Token")
+ .description("Access Token to authenticate requests")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+ .build();
+
+ static final PropertyDescriptor CRM_ENDPOINT = new PropertyDescriptor.Builder()
+ .name("crm-endpoint")
+ .displayName("HubSpot CRM API Endpoint")
+ .description("The HubSpot CRM API endpoint to get")
+ .required(true)
+ .allowableValues(CrmEndpoint.class)
+ .build();
+
+ static final PropertyDescriptor LIMIT = new PropertyDescriptor.Builder()
+ .name("crm-limit")
+ .displayName("Limit")
+ .description("The maximum number of results to display per page")
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor WEB_CLIENT_SERVICE_PROVIDER = new PropertyDescriptor.Builder()
+ .name("web-client-service-provider")
+ .displayName("Web Client Service Provider")
+ .description("Controller service for HTTP client operations")
+ .identifiesControllerService(WebClientServiceProvider.class)
+ .required(true)
+ .build();
+
+ static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("For FlowFiles created as a result of a successful HTTP request.")
+ .build();
+
+ static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("In case of HTTP client errors the flowfile will be routed to this relationship")
+ .build();
+
+ private static final String API_BASE_URI = "api.hubapi.com";
+ private static final String HTTPS = "https";
+ private static final String CURSOR_PARAMETER = "after";
+ private static final String LIMIT_PARAMETER = "limit";
+ private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
+ private static final JsonFactory JSON_FACTORY = OBJECT_MAPPER.getFactory();
+
+ private volatile WebClientServiceProvider webClientServiceProvider;
+
+ private static final List<PropertyDescriptor> PROPERTY_DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ ACCESS_TOKEN,
+ CRM_ENDPOINT,
+ LIMIT,
+ WEB_CLIENT_SERVICE_PROVIDER
+ ));
+
+ private static final Set<Relationship> RELATIONSHIPS = Collections.unmodifiableSet(new HashSet<>(Arrays.asList(
+ REL_SUCCESS,
+ REL_FAILURE
+ )));
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ webClientServiceProvider = context.getProperty(WEB_CLIENT_SERVICE_PROVIDER).asControllerService(WebClientServiceProvider.class);
+ }
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTY_DESCRIPTORS;
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return RELATIONSHIPS;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ final String accessToken = context.getProperty(ACCESS_TOKEN).getValue();
+ final String endpoint = context.getProperty(CRM_ENDPOINT).getValue();
+
+ final StateMap state = getStateMap(context);
+ final URI uri = createUri(context, state);
+
+ final HttpResponseEntity response = getHttpResponseEntity(accessToken, uri);
+ final AtomicInteger objectCountHolder = new AtomicInteger();
+
+ FlowFile flowFile = session.create();
+ flowFile = session.putAttribute(flowFile, "statusCode", String.valueOf(response.statusCode()));
+
+ flowFile = session.write(flowFile, out -> {
+
+ try (JsonParser jsonParser = JSON_FACTORY.createParser(response.body());
+ final JsonGenerator jsonGenerator = JSON_FACTORY.createGenerator(out, JsonEncoding.UTF8)) {
+ while (jsonParser.nextToken() != null) {
+ if (jsonParser.getCurrentToken() == JsonToken.FIELD_NAME && jsonParser.getCurrentName().equals("results")) {
+ jsonParser.nextToken();
+ jsonGenerator.copyCurrentStructure(jsonParser);
+ objectCountHolder.incrementAndGet();
+ }
+ String fieldName = jsonParser.getCurrentName();
+ if (CURSOR_PARAMETER.equals(fieldName)) {
+ jsonParser.nextToken();
+ Map<String, String> newStateMap = new HashMap<>(state.toMap());
+ newStateMap.put(endpoint, jsonParser.getText());
+ updateState(context, newStateMap);
+ break;
+ }
+ }
+ }
+ });
+ if (response.statusCode() >= 400) {
+ if (response.statusCode() == 429) {
+ context.yield();
+ throw new ProcessException("Rate limit exceeded, yielding for 10 seconds before retrying request.");
+ } else {
+ getLogger().warn("HTTP [{}] client error occurred at endpoint [{}]", response.statusCode(), endpoint);
Review Comment:
I usually avoid logging URIs for security but in this case the base URI is public. only the headers contain sensitive information. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] Lehel44 commented on a diff in pull request #6301: NIFI-10356: Create GetHubSpot processor
Posted by GitBox <gi...@apache.org>.
Lehel44 commented on code in PR #6301:
URL: https://github.com/apache/nifi/pull/6301#discussion_r952592756
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.configuration.DefaultSettings;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"hubspot"})
+@CapabilityDescription("Retrieves JSON data from a private HubSpot application."
+ + " Supports incremental retrieval: Users can set the \"limit\" property which serves as the upper limit of the retrieved objects."
+ + " When this property is set the processor will retrieve new records. This processor is intended to be run on the Primary Node only.")
+@Stateful(scopes = Scope.CLUSTER, description = "When the 'Limit' attribute is set, the paging cursor is saved after executing a request."
+ + " Only the objects after the paging cursor will be retrieved. The maximum number of retrieved objects is the 'Limit' attribute."
+ + " State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected,"
+ + " the new node can pick up where the previous node left off, without duplicating the data.")
+@DefaultSettings(yieldDuration = "10 sec")
+public class GetHubSpot extends AbstractProcessor {
+
+ static final PropertyDescriptor ACCESS_TOKEN = new PropertyDescriptor.Builder()
+ .name("admin-api-access-token")
+ .displayName("Access Token")
+ .description("Access Token to authenticate requests")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+ .build();
+
+ static final PropertyDescriptor CRM_ENDPOINT = new PropertyDescriptor.Builder()
+ .name("crm-endpoint")
+ .displayName("HubSpot CRM API Endpoint")
+ .description("The HubSpot CRM API endpoint to get")
+ .required(true)
+ .allowableValues(CrmEndpoint.class)
+ .build();
+
+ static final PropertyDescriptor LIMIT = new PropertyDescriptor.Builder()
+ .name("crm-limit")
+ .displayName("Limit")
+ .description("The maximum number of results to display per page")
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor WEB_CLIENT_SERVICE_PROVIDER = new PropertyDescriptor.Builder()
+ .name("web-client-service-provider")
+ .displayName("Web Client Service Provider")
+ .description("Controller service for HTTP client operations")
+ .identifiesControllerService(WebClientServiceProvider.class)
+ .required(true)
+ .build();
+
+ static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("For FlowFiles created as a result of a successful HTTP request.")
+ .build();
+
+ static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("In case of HTTP client errors the flowfile will be routed to this relationship")
+ .build();
+
+ private static final String API_BASE_URI = "api.hubapi.com";
+ private static final String HTTPS = "https";
+ private static final String CURSOR_PARAMETER = "after";
+ private static final String LIMIT_PARAMETER = "limit";
+ private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
+ private static final JsonFactory JSON_FACTORY = OBJECT_MAPPER.getFactory();
+
+ private volatile WebClientServiceProvider webClientServiceProvider;
+
+ private static final List<PropertyDescriptor> PROPERTY_DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ ACCESS_TOKEN,
+ CRM_ENDPOINT,
+ LIMIT,
+ WEB_CLIENT_SERVICE_PROVIDER
+ ));
+
+ private static final Set<Relationship> RELATIONSHIPS = Collections.unmodifiableSet(new HashSet<>(Arrays.asList(
+ REL_SUCCESS,
+ REL_FAILURE
+ )));
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ webClientServiceProvider = context.getProperty(WEB_CLIENT_SERVICE_PROVIDER).asControllerService(WebClientServiceProvider.class);
+ }
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTY_DESCRIPTORS;
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return RELATIONSHIPS;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ final String accessToken = context.getProperty(ACCESS_TOKEN).getValue();
+ final String endpoint = context.getProperty(CRM_ENDPOINT).getValue();
+
+ final StateMap state = getStateMap(context);
+ final URI uri = createUri(context, state);
+
+ final HttpResponseEntity response = getHttpResponseEntity(accessToken, uri);
+ final AtomicInteger objectCountHolder = new AtomicInteger();
+
+ FlowFile flowFile = session.create();
+ flowFile = session.putAttribute(flowFile, "statusCode", String.valueOf(response.statusCode()));
+
+ flowFile = session.write(flowFile, out -> {
+
+ try (JsonParser jsonParser = JSON_FACTORY.createParser(response.body());
+ final JsonGenerator jsonGenerator = JSON_FACTORY.createGenerator(out, JsonEncoding.UTF8)) {
+ while (jsonParser.nextToken() != null) {
+ if (jsonParser.getCurrentToken() == JsonToken.FIELD_NAME && jsonParser.getCurrentName().equals("results")) {
+ jsonParser.nextToken();
+ jsonGenerator.copyCurrentStructure(jsonParser);
+ objectCountHolder.incrementAndGet();
+ }
+ String fieldName = jsonParser.getCurrentName();
+ if (CURSOR_PARAMETER.equals(fieldName)) {
+ jsonParser.nextToken();
+ Map<String, String> newStateMap = new HashMap<>(state.toMap());
+ newStateMap.put(endpoint, jsonParser.getText());
+ updateState(context, newStateMap);
+ break;
+ }
+ }
+ }
+ });
+ if (response.statusCode() >= 400) {
+ if (response.statusCode() == 429) {
+ context.yield();
+ throw new ProcessException("Rate limit exceeded, yielding for 10 seconds before retrying request.");
Review Comment:
I removed the 10 seconds from the exception message and added the status code and uri.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] Lehel44 commented on pull request #6301: NIFI-10356: Create GetHubSpot processor
Posted by GitBox <gi...@apache.org>.
Lehel44 commented on PR #6301:
URL: https://github.com/apache/nifi/pull/6301#issuecomment-1220039086
Thanks for the comments @exceptionfactory . I also added Rate Limit handling to the processor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] Lehel44 commented on a diff in pull request #6301: NIFI-10356: Create GetHubSpot processor
Posted by GitBox <gi...@apache.org>.
Lehel44 commented on code in PR #6301:
URL: https://github.com/apache/nifi/pull/6301#discussion_r952403150
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-nar/pom.xml:
##########
@@ -0,0 +1,46 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>nifi-hubspot-bundle</artifactId>
+ <groupId>org.apache.nifi</groupId>
+ <version>1.18.0-SNAPSHOT</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>nifi-hubspot-nar</artifactId>
+
+ <packaging>nar</packaging>
+ <properties>
+ <maven.javadoc.skip>true</maven.javadoc.skip>
+ <source.skip>true</source.skip>
+ </properties>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-hubspot-processors</artifactId>
+ <version>1.18.0-SNAPSHOT</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-record-serialization-services-nar</artifactId>
Review Comment:
That's a nice catch, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] Lehel44 commented on a diff in pull request #6301: NIFI-10356: Create GetHubSpot processor
Posted by GitBox <gi...@apache.org>.
Lehel44 commented on code in PR #6301:
URL: https://github.com/apache/nifi/pull/6301#discussion_r952591579
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/resources/docs/org.apache.nifi.processors.hubspot.GetHubSpot/additionalDetails.html:
##########
@@ -0,0 +1,143 @@
+<!DOCTYPE html>
+<html lang="en">
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<head>
+ <meta charset="utf-8"/>
+ <title>GetHubSpot</title>
+ <link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"/>
+</head>
+
+<body>
+<h2>Incremental Loading</h2>
+<p>
+ Some resources can be processed incrementally by NiFi. This means that only resources created or modified after the last run
+ time of the processor are displayed. The processor state can be reset in the context menu. The following list shows which
+ date-time fields are incremented for which resources.
+<ul>
+ <li>Access
+ <ul>
+ <li>Access Scope: none</li>
+ <li>StoreFront Access Token: none</li>
+ </ul>
+ </li>
+ <li>Analytics
+ <ul>
+ <li>Reports: updated_at_min</li>
+ </ul>
+ </li>
+ <li>Billing
+ <ul>
+ <li>Application Charge: none</li>
+ <li>Application Credit: none</li>
+ <li>Recurring Application Charge: none</li>
+ </ul>
+ </li>
+ <li>Customers
+ <ul>
+ <li>Customers: updated_at_min</li>
+ <li>Customer Saved Searches: none</li>
+ </ul>
+ </li>
+ <li>Discounts
+ <ul>
+ <li>Price Rules: updated_at_min</li>
+ </ul>
+ </li>
+ <li>Events
+ <ul>
+ <li>Events: created_at_min</li>
+ </ul>
+ </li>
+ <li>Inventory
+ <ul>
+ <li>Inventory Levels: updated_at_min</li>
+ <li>Locations: none</li>
+ </ul>
+ </li>
+ <li>Marketing Event
+ <ul>
+ <li>Marketing Events: none</li>
+ </ul>
+ </li>
+ <li>Metafields
+ <ul>
+ <li>Metafields: updated_at_min</li>
+ </ul>
+ </li>
+ <li>Online Store
+ <ul>
+ <li>Blogs: none</li>
+ <li>Comment: none</li>
+ <li>Pages: none</li>
+ <li>Redirects: none</li>
+ <li>Script Tags: updated_at_min</li>
+ <li>Themes: none</li>
+ </ul>
+ </li>
+ <li>Orders
+ <ul>
+ <li>Abandoned Checkouts: updated_at_min</li>
+ <li>Draft Orders: updated_at_min</li>
+ <li>Orders: updated_at_min</li>
+ </ul>
+ </li>
+ <li>Plus
+ <ul>
+ <li>Gift Cards: none</li>
+ <li>Users: none</li>
+ </ul>
+ </li>
+ <li>Product
+ <ul>
+ <li>Collects: none</li>
+ <li>Custom Collections: updated_at_min</li>
+ <li>Products: updated_at_min</li>
+ <li>Smart Collections: updated_at_min</li>
+ </ul>
+ </li>
+ <li>Sales Channels
+ <ul>
+ <li>Collection Listings: none</li>
+ <li>Mobile Platform Applications: none</li>
+ <li>Product Listings: updated_at_min</li>
+ <li>Resource Feedbacks: none</li>
+ </ul>
+ </li>
+ <li>Shipping and Fulfillments
+ <ul>
+ <li>Carrier Services: none</li>
+ </ul>
+ </li>
+ <li>Store Properties
+ <ul>
+ <li>Countries: none</li>
+ <li>Currencies: none</li>
+ <li>Policies: none</li>
+ <li>Shipping Zones: updated_at_min</li>
+ <li>Shop: none</li>
+ </ul>
+ </li>
+ <li>Tender Transactions
+ <ul>
+ <li>Tender Transactions: processed_at_min</li>
+ </ul>
+ </li>
+</ul>
+
+Specific trap type can set in case of Enterprise Specific generic trap type is chosen.
Review Comment:
It should be removed, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] Lehel44 commented on a diff in pull request #6301: NIFI-10356: Create GetHubSpot processor
Posted by GitBox <gi...@apache.org>.
Lehel44 commented on code in PR #6301:
URL: https://github.com/apache/nifi/pull/6301#discussion_r955491219
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.configuration.DefaultSettings;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpResponseStatus;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"hubspot"})
+@CapabilityDescription("Retrieves JSON data from a private HubSpot application."
+ + " Supports incremental retrieval: Users can set the \"limit\" property which serves as the upper limit of the retrieved objects."
+ + " When this property is set the processor will retrieve new records. This processor is intended to be run on the Primary Node only.")
+@Stateful(scopes = Scope.CLUSTER, description = "When the 'Limit' attribute is set, the paging cursor is saved after executing a request."
+ + " Only the objects after the paging cursor will be retrieved. The maximum number of retrieved objects is the 'Limit' attribute."
+ + " State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected,"
+ + " the new node can pick up where the previous node left off, without duplicating the data.")
+@DefaultSettings(yieldDuration = "10 sec")
+public class GetHubSpot extends AbstractProcessor {
+
+ static final PropertyDescriptor CRM_ENDPOINT = new PropertyDescriptor.Builder()
+ .name("crm-endpoint")
+ .displayName("HubSpot CRM API Endpoint")
+ .description("The HubSpot CRM API endpoint to which the Processor will send requests")
+ .required(true)
+ .allowableValues(CrmEndpoint.class)
+ .build();
+
+ static final PropertyDescriptor ACCESS_TOKEN = new PropertyDescriptor.Builder()
+ .name("access-token")
+ .displayName("Access Token")
+ .description("Access Token to authenticate requests")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+ .build();
+
+ static final PropertyDescriptor LIMIT = new PropertyDescriptor.Builder()
+ .name("crm-limit")
+ .displayName("Limit")
+ .description("The maximum number of results to request for each invocation of the Processor")
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor WEB_CLIENT_SERVICE_PROVIDER = new PropertyDescriptor.Builder()
+ .name("web-client-service-provider")
+ .displayName("Web Client Service Provider")
+ .description("Controller service for HTTP client operations")
+ .identifiesControllerService(WebClientServiceProvider.class)
+ .required(true)
+ .build();
+
+ static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("For FlowFiles created as a result of a successful HTTP request.")
+ .build();
+
+ static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("In case of HTTP client errors the flowfile will be routed to this relationship")
+ .build();
+
+ private static final String API_BASE_URI = "api.hubapi.com";
+ private static final String HTTPS = "https";
+ private static final String CURSOR_PARAMETER = "after";
+ private static final String LIMIT_PARAMETER = "limit";
+ private static final int TOO_MANY_REQUESTS = 429;
+ private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
+ private static final JsonFactory JSON_FACTORY = OBJECT_MAPPER.getFactory();
+
+ private volatile WebClientServiceProvider webClientServiceProvider;
+
+ private static final List<PropertyDescriptor> PROPERTY_DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ CRM_ENDPOINT,
+ ACCESS_TOKEN,
+ LIMIT,
+ WEB_CLIENT_SERVICE_PROVIDER
+ ));
+
+ private static final Set<Relationship> RELATIONSHIPS = Collections.unmodifiableSet(new HashSet<>(
+ Collections.singletonList(
+ REL_SUCCESS
+ )));
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ webClientServiceProvider = context.getProperty(WEB_CLIENT_SERVICE_PROVIDER).asControllerService(WebClientServiceProvider.class);
+ }
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTY_DESCRIPTORS;
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return RELATIONSHIPS;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ final String accessToken = context.getProperty(ACCESS_TOKEN).getValue();
+ final String endpoint = context.getProperty(CRM_ENDPOINT).getValue();
+
+ final StateMap state = getStateMap(context);
+ final URI uri = createUri(context, state);
+
+ final HttpResponseEntity response = getHttpResponseEntity(accessToken, uri);
+ final AtomicInteger objectCountHolder = new AtomicInteger();
+
+ if (response.statusCode() == HttpResponseStatus.OK.getCode()) {
+ FlowFile flowFile = session.create();
+ flowFile = session.write(flowFile, parseHttpResponse(context, endpoint, state, response, objectCountHolder));
+ if (objectCountHolder.get() > 0) {
+ session.transfer(flowFile, REL_SUCCESS);
+ } else {
+ getLogger().debug("Empty response when requested HubSpot endpoint: [{}]", endpoint);
+ }
+ } else if (response.statusCode() >= 400) {
+ if (response.statusCode() == TOO_MANY_REQUESTS) {
+ context.yield();
+ throw new ProcessException(String.format("Rate limit exceeded, yielding before retrying request. HTTP %d error for requested URI [%s]", response.statusCode(), uri));
+ } else {
+ getLogger().warn("HTTP {} error for requested URI [{}]", response.statusCode(), uri);
+ }
+ }
+ }
+
+ private OutputStreamCallback parseHttpResponse(ProcessContext context, String endpoint, StateMap state, HttpResponseEntity response, AtomicInteger objectCountHolder) {
+ return out -> {
+ try (JsonParser jsonParser = JSON_FACTORY.createParser(response.body());
+ final JsonGenerator jsonGenerator = JSON_FACTORY.createGenerator(out, JsonEncoding.UTF8)) {
Review Comment:
I reformatted again, hope it get fixed.
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.configuration.DefaultSettings;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpResponseStatus;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"hubspot"})
+@CapabilityDescription("Retrieves JSON data from a private HubSpot application."
+ + " Supports incremental retrieval: Users can set the \"limit\" property which serves as the upper limit of the retrieved objects."
+ + " When this property is set the processor will retrieve new records. This processor is intended to be run on the Primary Node only.")
+@Stateful(scopes = Scope.CLUSTER, description = "When the 'Limit' attribute is set, the paging cursor is saved after executing a request."
+ + " Only the objects after the paging cursor will be retrieved. The maximum number of retrieved objects is the 'Limit' attribute."
+ + " State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected,"
+ + " the new node can pick up where the previous node left off, without duplicating the data.")
+@DefaultSettings(yieldDuration = "10 sec")
+public class GetHubSpot extends AbstractProcessor {
+
+ static final PropertyDescriptor CRM_ENDPOINT = new PropertyDescriptor.Builder()
+ .name("crm-endpoint")
+ .displayName("HubSpot CRM API Endpoint")
+ .description("The HubSpot CRM API endpoint to which the Processor will send requests")
+ .required(true)
+ .allowableValues(CrmEndpoint.class)
+ .build();
+
+ static final PropertyDescriptor ACCESS_TOKEN = new PropertyDescriptor.Builder()
+ .name("access-token")
+ .displayName("Access Token")
+ .description("Access Token to authenticate requests")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+ .build();
+
+ static final PropertyDescriptor LIMIT = new PropertyDescriptor.Builder()
+ .name("crm-limit")
+ .displayName("Limit")
+ .description("The maximum number of results to request for each invocation of the Processor")
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor WEB_CLIENT_SERVICE_PROVIDER = new PropertyDescriptor.Builder()
+ .name("web-client-service-provider")
+ .displayName("Web Client Service Provider")
+ .description("Controller service for HTTP client operations")
+ .identifiesControllerService(WebClientServiceProvider.class)
+ .required(true)
+ .build();
+
+ static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("For FlowFiles created as a result of a successful HTTP request.")
+ .build();
+
+ static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("In case of HTTP client errors the flowfile will be routed to this relationship")
+ .build();
+
+ private static final String API_BASE_URI = "api.hubapi.com";
+ private static final String HTTPS = "https";
+ private static final String CURSOR_PARAMETER = "after";
+ private static final String LIMIT_PARAMETER = "limit";
+ private static final int TOO_MANY_REQUESTS = 429;
+ private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
+ private static final JsonFactory JSON_FACTORY = OBJECT_MAPPER.getFactory();
+
+ private volatile WebClientServiceProvider webClientServiceProvider;
+
+ private static final List<PropertyDescriptor> PROPERTY_DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ CRM_ENDPOINT,
+ ACCESS_TOKEN,
+ LIMIT,
+ WEB_CLIENT_SERVICE_PROVIDER
+ ));
+
+ private static final Set<Relationship> RELATIONSHIPS = Collections.unmodifiableSet(new HashSet<>(
+ Collections.singletonList(
+ REL_SUCCESS
+ )));
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ webClientServiceProvider = context.getProperty(WEB_CLIENT_SERVICE_PROVIDER).asControllerService(WebClientServiceProvider.class);
+ }
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTY_DESCRIPTORS;
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return RELATIONSHIPS;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ final String accessToken = context.getProperty(ACCESS_TOKEN).getValue();
+ final String endpoint = context.getProperty(CRM_ENDPOINT).getValue();
+
+ final StateMap state = getStateMap(context);
+ final URI uri = createUri(context, state);
+
+ final HttpResponseEntity response = getHttpResponseEntity(accessToken, uri);
+ final AtomicInteger objectCountHolder = new AtomicInteger();
+
+ if (response.statusCode() == HttpResponseStatus.OK.getCode()) {
+ FlowFile flowFile = session.create();
+ flowFile = session.write(flowFile, parseHttpResponse(context, endpoint, state, response, objectCountHolder));
+ if (objectCountHolder.get() > 0) {
+ session.transfer(flowFile, REL_SUCCESS);
+ } else {
+ getLogger().debug("Empty response when requested HubSpot endpoint: [{}]", endpoint);
+ }
+ } else if (response.statusCode() >= 400) {
+ if (response.statusCode() == TOO_MANY_REQUESTS) {
+ context.yield();
+ throw new ProcessException(String.format("Rate limit exceeded, yielding before retrying request. HTTP %d error for requested URI [%s]", response.statusCode(), uri));
+ } else {
+ getLogger().warn("HTTP {} error for requested URI [{}]", response.statusCode(), uri);
+ }
+ }
+ }
+
+ private OutputStreamCallback parseHttpResponse(ProcessContext context, String endpoint, StateMap state, HttpResponseEntity response, AtomicInteger objectCountHolder) {
+ return out -> {
+ try (JsonParser jsonParser = JSON_FACTORY.createParser(response.body());
+ final JsonGenerator jsonGenerator = JSON_FACTORY.createGenerator(out, JsonEncoding.UTF8)) {
Review Comment:
I reformatted again, hope it gets fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #6301: NIFI-10356: Create GetHubSpot processor
Posted by GitBox <gi...@apache.org>.
exceptionfactory commented on code in PR #6301:
URL: https://github.com/apache/nifi/pull/6301#discussion_r953102705
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.configuration.DefaultSettings;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpResponseStatus;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"hubspot"})
+@CapabilityDescription("Retrieves JSON data from a private HubSpot application."
+ + " Supports incremental retrieval: Users can set the \"limit\" property which serves as the upper limit of the retrieved objects."
+ + " When this property is set the processor will retrieve new records. This processor is intended to be run on the Primary Node only.")
+@Stateful(scopes = Scope.CLUSTER, description = "When the 'Limit' attribute is set, the paging cursor is saved after executing a request."
+ + " Only the objects after the paging cursor will be retrieved. The maximum number of retrieved objects is the 'Limit' attribute."
+ + " State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected,"
+ + " the new node can pick up where the previous node left off, without duplicating the data.")
+@DefaultSettings(yieldDuration = "10 sec")
+public class GetHubSpot extends AbstractProcessor {
+
+ static final PropertyDescriptor CRM_ENDPOINT = new PropertyDescriptor.Builder()
+ .name("crm-endpoint")
+ .displayName("HubSpot CRM API Endpoint")
+ .description("The HubSpot CRM API endpoint to which the Processor will send requests")
+ .required(true)
+ .allowableValues(CrmEndpoint.class)
+ .build();
+
+ static final PropertyDescriptor ACCESS_TOKEN = new PropertyDescriptor.Builder()
+ .name("access-token")
+ .displayName("Access Token")
+ .description("Access Token to authenticate requests")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+ .build();
+
+ static final PropertyDescriptor LIMIT = new PropertyDescriptor.Builder()
+ .name("crm-limit")
+ .displayName("Limit")
+ .description("The maximum number of results to request for each invocation of the Processor")
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor WEB_CLIENT_SERVICE_PROVIDER = new PropertyDescriptor.Builder()
+ .name("web-client-service-provider")
+ .displayName("Web Client Service Provider")
+ .description("Controller service for HTTP client operations")
+ .identifiesControllerService(WebClientServiceProvider.class)
+ .required(true)
+ .build();
+
+ static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("For FlowFiles created as a result of a successful HTTP request.")
+ .build();
+
+ static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("In case of HTTP client errors the flowfile will be routed to this relationship")
+ .build();
+
+ private static final String API_BASE_URI = "api.hubapi.com";
+ private static final String HTTPS = "https";
+ private static final String CURSOR_PARAMETER = "after";
+ private static final String LIMIT_PARAMETER = "limit";
+ private static final int TOO_MANY_REQUESTS = 429;
+ private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
+ private static final JsonFactory JSON_FACTORY = OBJECT_MAPPER.getFactory();
+
+ private volatile WebClientServiceProvider webClientServiceProvider;
+
+ private static final List<PropertyDescriptor> PROPERTY_DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ CRM_ENDPOINT,
+ ACCESS_TOKEN,
+ LIMIT,
+ WEB_CLIENT_SERVICE_PROVIDER
+ ));
+
+ private static final Set<Relationship> RELATIONSHIPS = Collections.unmodifiableSet(new HashSet<>(
+ Collections.singletonList(
+ REL_SUCCESS
+ )));
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ webClientServiceProvider = context.getProperty(WEB_CLIENT_SERVICE_PROVIDER).asControllerService(WebClientServiceProvider.class);
+ }
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTY_DESCRIPTORS;
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return RELATIONSHIPS;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ final String accessToken = context.getProperty(ACCESS_TOKEN).getValue();
+ final String endpoint = context.getProperty(CRM_ENDPOINT).getValue();
+
+ final StateMap state = getStateMap(context);
+ final URI uri = createUri(context, state);
+
+ final HttpResponseEntity response = getHttpResponseEntity(accessToken, uri);
+ final AtomicInteger objectCountHolder = new AtomicInteger();
+
+ if (response.statusCode() == HttpResponseStatus.OK.getCode()) {
+ FlowFile flowFile = session.create();
+ flowFile = session.putAttribute(flowFile, "statusCode", String.valueOf(response.statusCode()));
Review Comment:
This FlowFile attribute seems unnecessary since all successes will have a value of `200`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory closed pull request #6301: NIFI-10356: Create GetHubSpot processor
Posted by GitBox <gi...@apache.org>.
exceptionfactory closed pull request #6301: NIFI-10356: Create GetHubSpot processor
URL: https://github.com/apache/nifi/pull/6301
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] dam4rus commented on a diff in pull request #6301: NIFI-10356: Create GetHubSpot processor
Posted by GitBox <gi...@apache.org>.
dam4rus commented on code in PR #6301:
URL: https://github.com/apache/nifi/pull/6301#discussion_r954889290
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.configuration.DefaultSettings;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpResponseStatus;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"hubspot"})
+@CapabilityDescription("Retrieves JSON data from a private HubSpot application."
+ + " Supports incremental retrieval: Users can set the \"limit\" property which serves as the upper limit of the retrieved objects."
+ + " When this property is set the processor will retrieve new records. This processor is intended to be run on the Primary Node only.")
+@Stateful(scopes = Scope.CLUSTER, description = "When the 'Limit' attribute is set, the paging cursor is saved after executing a request."
+ + " Only the objects after the paging cursor will be retrieved. The maximum number of retrieved objects is the 'Limit' attribute."
+ + " State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected,"
+ + " the new node can pick up where the previous node left off, without duplicating the data.")
+@DefaultSettings(yieldDuration = "10 sec")
+public class GetHubSpot extends AbstractProcessor {
+
+ static final PropertyDescriptor CRM_ENDPOINT = new PropertyDescriptor.Builder()
+ .name("crm-endpoint")
+ .displayName("HubSpot CRM API Endpoint")
+ .description("The HubSpot CRM API endpoint to which the Processor will send requests")
+ .required(true)
+ .allowableValues(CrmEndpoint.class)
+ .build();
+
+ static final PropertyDescriptor ACCESS_TOKEN = new PropertyDescriptor.Builder()
+ .name("access-token")
+ .displayName("Access Token")
+ .description("Access Token to authenticate requests")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+ .build();
+
+ static final PropertyDescriptor LIMIT = new PropertyDescriptor.Builder()
+ .name("crm-limit")
+ .displayName("Limit")
+ .description("The maximum number of results to request for each invocation of the Processor")
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor WEB_CLIENT_SERVICE_PROVIDER = new PropertyDescriptor.Builder()
+ .name("web-client-service-provider")
+ .displayName("Web Client Service Provider")
+ .description("Controller service for HTTP client operations")
+ .identifiesControllerService(WebClientServiceProvider.class)
+ .required(true)
+ .build();
+
+ static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("For FlowFiles created as a result of a successful HTTP request.")
+ .build();
+
+ static final Relationship REL_FAILURE = new Relationship.Builder()
Review Comment:
Unused field
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/test/java/org/apache/nifi/processors/hubspot/GetHubSpotTest.java:
##########
@@ -0,0 +1,199 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import okhttp3.HttpUrl;
+import okhttp3.mockwebserver.MockResponse;
+import okhttp3.mockwebserver.MockWebServer;
+import okhttp3.mockwebserver.RecordedRequest;
+import org.apache.commons.io.IOUtils;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.apache.nifi.web.client.StandardHttpUriBuilder;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.service.StandardWebClientServiceProvider;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Collections;
+import java.util.List;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+class GetHubSpotTest {
+
+ public static final String BASE_URL = "/test/hubspot";
+ public static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
+ private static MockWebServer server;
+ private static HttpUrl baseUrl;
+
+ @BeforeEach
+ void setup() throws IOException {
+ server = new MockWebServer();
+ server.start();
+ baseUrl = server.url(BASE_URL);
+ }
+
+ @AfterEach
+ void tearDown() throws IOException {
+ if (server != null) {
+ server.shutdown();
+ server = null;
+ }
+ }
+
+ @Test
+ void testLimitIsAddedToUrl() throws InitializationException, InterruptedException, IOException {
+
+ final String response = getResourceAsString("response-without-paging-cursor.json");
+ server.enqueue(new MockResponse().setResponseCode(200).setBody(response));
+
+ final StandardWebClientServiceProvider standardWebClientServiceProvider = new StandardWebClientServiceProvider();
Review Comment:
This code block is duplicated in every test case. You can extract it to a method or set it up in `void setup()`
```java
final StandardWebClientServiceProvider standardWebClientServiceProvider = new StandardWebClientServiceProvider();
final MockGetHubSpot mockGetHubSpot = new MockGetHubSpot();
TestRunner runner = TestRunners.newTestRunner(mockGetHubSpot);
runner.addControllerService("standardWebClientServiceProvider", standardWebClientServiceProvider);
runner.enableControllerService(standardWebClientServiceProvider);
runner.setProperty(GetHubSpot.WEB_CLIENT_SERVICE_PROVIDER, standardWebClientServiceProvider.getIdentifier());
runner.setProperty(GetHubSpot.ACCESS_TOKEN, "testToken");
runner.setProperty(GetHubSpot.CRM_ENDPOINT, CrmEndpoint.COMPANIES.getValue());
runner.setProperty(GetHubSpot.LIMIT, "1");
```
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.configuration.DefaultSettings;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpResponseStatus;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"hubspot"})
+@CapabilityDescription("Retrieves JSON data from a private HubSpot application."
+ + " Supports incremental retrieval: Users can set the \"limit\" property which serves as the upper limit of the retrieved objects."
+ + " When this property is set the processor will retrieve new records. This processor is intended to be run on the Primary Node only.")
+@Stateful(scopes = Scope.CLUSTER, description = "When the 'Limit' attribute is set, the paging cursor is saved after executing a request."
+ + " Only the objects after the paging cursor will be retrieved. The maximum number of retrieved objects is the 'Limit' attribute."
+ + " State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected,"
+ + " the new node can pick up where the previous node left off, without duplicating the data.")
+@DefaultSettings(yieldDuration = "10 sec")
+public class GetHubSpot extends AbstractProcessor {
+
+ static final PropertyDescriptor CRM_ENDPOINT = new PropertyDescriptor.Builder()
+ .name("crm-endpoint")
+ .displayName("HubSpot CRM API Endpoint")
+ .description("The HubSpot CRM API endpoint to which the Processor will send requests")
+ .required(true)
+ .allowableValues(CrmEndpoint.class)
+ .build();
+
+ static final PropertyDescriptor ACCESS_TOKEN = new PropertyDescriptor.Builder()
+ .name("access-token")
+ .displayName("Access Token")
+ .description("Access Token to authenticate requests")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+ .build();
+
+ static final PropertyDescriptor LIMIT = new PropertyDescriptor.Builder()
+ .name("crm-limit")
+ .displayName("Limit")
+ .description("The maximum number of results to request for each invocation of the Processor")
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor WEB_CLIENT_SERVICE_PROVIDER = new PropertyDescriptor.Builder()
+ .name("web-client-service-provider")
+ .displayName("Web Client Service Provider")
+ .description("Controller service for HTTP client operations")
+ .identifiesControllerService(WebClientServiceProvider.class)
+ .required(true)
+ .build();
+
+ static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("For FlowFiles created as a result of a successful HTTP request.")
+ .build();
+
+ static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("In case of HTTP client errors the flowfile will be routed to this relationship")
+ .build();
+
+ private static final String API_BASE_URI = "api.hubapi.com";
+ private static final String HTTPS = "https";
+ private static final String CURSOR_PARAMETER = "after";
+ private static final String LIMIT_PARAMETER = "limit";
+ private static final int TOO_MANY_REQUESTS = 429;
+ private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
+ private static final JsonFactory JSON_FACTORY = OBJECT_MAPPER.getFactory();
+
+ private volatile WebClientServiceProvider webClientServiceProvider;
+
+ private static final List<PropertyDescriptor> PROPERTY_DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ CRM_ENDPOINT,
+ ACCESS_TOKEN,
+ LIMIT,
+ WEB_CLIENT_SERVICE_PROVIDER
+ ));
+
+ private static final Set<Relationship> RELATIONSHIPS = Collections.unmodifiableSet(new HashSet<>(
+ Collections.singletonList(
+ REL_SUCCESS
+ )));
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ webClientServiceProvider = context.getProperty(WEB_CLIENT_SERVICE_PROVIDER).asControllerService(WebClientServiceProvider.class);
+ }
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTY_DESCRIPTORS;
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return RELATIONSHIPS;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ final String accessToken = context.getProperty(ACCESS_TOKEN).getValue();
+ final String endpoint = context.getProperty(CRM_ENDPOINT).getValue();
+
+ final StateMap state = getStateMap(context);
+ final URI uri = createUri(context, state);
+
+ final HttpResponseEntity response = getHttpResponseEntity(accessToken, uri);
+ final AtomicInteger objectCountHolder = new AtomicInteger();
+
+ if (response.statusCode() == HttpResponseStatus.OK.getCode()) {
+ FlowFile flowFile = session.create();
+ flowFile = session.write(flowFile, parseHttpResponse(context, endpoint, state, response, objectCountHolder));
+ if (objectCountHolder.get() > 0) {
+ session.transfer(flowFile, REL_SUCCESS);
+ } else {
+ getLogger().debug("Empty response when requested HubSpot endpoint: [{}]", endpoint);
+ }
+ } else if (response.statusCode() >= 400) {
+ if (response.statusCode() == TOO_MANY_REQUESTS) {
+ context.yield();
+ throw new ProcessException(String.format("Rate limit exceeded, yielding before retrying request. HTTP %d error for requested URI [%s]", response.statusCode(), uri));
+ } else {
+ getLogger().warn("HTTP {} error for requested URI [{}]", response.statusCode(), uri);
+ }
+ }
+ }
+
+ private OutputStreamCallback parseHttpResponse(ProcessContext context, String endpoint, StateMap state, HttpResponseEntity response, AtomicInteger objectCountHolder) {
+ return out -> {
+ try (JsonParser jsonParser = JSON_FACTORY.createParser(response.body());
Review Comment:
Missing `final`
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.configuration.DefaultSettings;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpResponseStatus;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"hubspot"})
+@CapabilityDescription("Retrieves JSON data from a private HubSpot application."
+ + " Supports incremental retrieval: Users can set the \"limit\" property which serves as the upper limit of the retrieved objects."
+ + " When this property is set the processor will retrieve new records. This processor is intended to be run on the Primary Node only.")
+@Stateful(scopes = Scope.CLUSTER, description = "When the 'Limit' attribute is set, the paging cursor is saved after executing a request."
+ + " Only the objects after the paging cursor will be retrieved. The maximum number of retrieved objects is the 'Limit' attribute."
+ + " State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected,"
+ + " the new node can pick up where the previous node left off, without duplicating the data.")
+@DefaultSettings(yieldDuration = "10 sec")
+public class GetHubSpot extends AbstractProcessor {
+
+ static final PropertyDescriptor CRM_ENDPOINT = new PropertyDescriptor.Builder()
+ .name("crm-endpoint")
+ .displayName("HubSpot CRM API Endpoint")
+ .description("The HubSpot CRM API endpoint to which the Processor will send requests")
+ .required(true)
+ .allowableValues(CrmEndpoint.class)
+ .build();
+
+ static final PropertyDescriptor ACCESS_TOKEN = new PropertyDescriptor.Builder()
+ .name("access-token")
+ .displayName("Access Token")
+ .description("Access Token to authenticate requests")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+ .build();
+
+ static final PropertyDescriptor LIMIT = new PropertyDescriptor.Builder()
+ .name("crm-limit")
+ .displayName("Limit")
+ .description("The maximum number of results to request for each invocation of the Processor")
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor WEB_CLIENT_SERVICE_PROVIDER = new PropertyDescriptor.Builder()
+ .name("web-client-service-provider")
+ .displayName("Web Client Service Provider")
+ .description("Controller service for HTTP client operations")
+ .identifiesControllerService(WebClientServiceProvider.class)
+ .required(true)
+ .build();
+
+ static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("For FlowFiles created as a result of a successful HTTP request.")
+ .build();
+
+ static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("In case of HTTP client errors the flowfile will be routed to this relationship")
+ .build();
+
+ private static final String API_BASE_URI = "api.hubapi.com";
+ private static final String HTTPS = "https";
+ private static final String CURSOR_PARAMETER = "after";
+ private static final String LIMIT_PARAMETER = "limit";
+ private static final int TOO_MANY_REQUESTS = 429;
+ private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
+ private static final JsonFactory JSON_FACTORY = OBJECT_MAPPER.getFactory();
+
+ private volatile WebClientServiceProvider webClientServiceProvider;
+
+ private static final List<PropertyDescriptor> PROPERTY_DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ CRM_ENDPOINT,
+ ACCESS_TOKEN,
+ LIMIT,
+ WEB_CLIENT_SERVICE_PROVIDER
+ ));
+
+ private static final Set<Relationship> RELATIONSHIPS = Collections.unmodifiableSet(new HashSet<>(
+ Collections.singletonList(
+ REL_SUCCESS
+ )));
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ webClientServiceProvider = context.getProperty(WEB_CLIENT_SERVICE_PROVIDER).asControllerService(WebClientServiceProvider.class);
+ }
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTY_DESCRIPTORS;
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return RELATIONSHIPS;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ final String accessToken = context.getProperty(ACCESS_TOKEN).getValue();
+ final String endpoint = context.getProperty(CRM_ENDPOINT).getValue();
+
+ final StateMap state = getStateMap(context);
+ final URI uri = createUri(context, state);
+
+ final HttpResponseEntity response = getHttpResponseEntity(accessToken, uri);
+ final AtomicInteger objectCountHolder = new AtomicInteger();
+
+ if (response.statusCode() == HttpResponseStatus.OK.getCode()) {
+ FlowFile flowFile = session.create();
+ flowFile = session.write(flowFile, parseHttpResponse(context, endpoint, state, response, objectCountHolder));
+ if (objectCountHolder.get() > 0) {
+ session.transfer(flowFile, REL_SUCCESS);
+ } else {
+ getLogger().debug("Empty response when requested HubSpot endpoint: [{}]", endpoint);
+ }
+ } else if (response.statusCode() >= 400) {
+ if (response.statusCode() == TOO_MANY_REQUESTS) {
+ context.yield();
+ throw new ProcessException(String.format("Rate limit exceeded, yielding before retrying request. HTTP %d error for requested URI [%s]", response.statusCode(), uri));
+ } else {
+ getLogger().warn("HTTP {} error for requested URI [{}]", response.statusCode(), uri);
+ }
+ }
+ }
+
+ private OutputStreamCallback parseHttpResponse(ProcessContext context, String endpoint, StateMap state, HttpResponseEntity response, AtomicInteger objectCountHolder) {
+ return out -> {
+ try (JsonParser jsonParser = JSON_FACTORY.createParser(response.body());
+ final JsonGenerator jsonGenerator = JSON_FACTORY.createGenerator(out, JsonEncoding.UTF8)) {
+ while (jsonParser.nextToken() != null) {
+ if (jsonParser.getCurrentToken() == JsonToken.FIELD_NAME && jsonParser.getCurrentName()
+ .equals("results")) {
+ jsonParser.nextToken();
+ jsonGenerator.copyCurrentStructure(jsonParser);
+ objectCountHolder.incrementAndGet();
+ }
+ String fieldName = jsonParser.getCurrentName();
Review Comment:
Missing `final`
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.configuration.DefaultSettings;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpResponseStatus;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"hubspot"})
+@CapabilityDescription("Retrieves JSON data from a private HubSpot application."
+ + " Supports incremental retrieval: Users can set the \"limit\" property which serves as the upper limit of the retrieved objects."
+ + " When this property is set the processor will retrieve new records. This processor is intended to be run on the Primary Node only.")
+@Stateful(scopes = Scope.CLUSTER, description = "When the 'Limit' attribute is set, the paging cursor is saved after executing a request."
+ + " Only the objects after the paging cursor will be retrieved. The maximum number of retrieved objects is the 'Limit' attribute."
+ + " State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected,"
+ + " the new node can pick up where the previous node left off, without duplicating the data.")
+@DefaultSettings(yieldDuration = "10 sec")
+public class GetHubSpot extends AbstractProcessor {
+
+ static final PropertyDescriptor CRM_ENDPOINT = new PropertyDescriptor.Builder()
+ .name("crm-endpoint")
+ .displayName("HubSpot CRM API Endpoint")
+ .description("The HubSpot CRM API endpoint to which the Processor will send requests")
+ .required(true)
+ .allowableValues(CrmEndpoint.class)
+ .build();
+
+ static final PropertyDescriptor ACCESS_TOKEN = new PropertyDescriptor.Builder()
+ .name("access-token")
+ .displayName("Access Token")
+ .description("Access Token to authenticate requests")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+ .build();
+
+ static final PropertyDescriptor LIMIT = new PropertyDescriptor.Builder()
+ .name("crm-limit")
+ .displayName("Limit")
+ .description("The maximum number of results to request for each invocation of the Processor")
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor WEB_CLIENT_SERVICE_PROVIDER = new PropertyDescriptor.Builder()
+ .name("web-client-service-provider")
+ .displayName("Web Client Service Provider")
+ .description("Controller service for HTTP client operations")
+ .identifiesControllerService(WebClientServiceProvider.class)
+ .required(true)
+ .build();
+
+ static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("For FlowFiles created as a result of a successful HTTP request.")
+ .build();
+
+ static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("In case of HTTP client errors the flowfile will be routed to this relationship")
+ .build();
+
+ private static final String API_BASE_URI = "api.hubapi.com";
+ private static final String HTTPS = "https";
+ private static final String CURSOR_PARAMETER = "after";
+ private static final String LIMIT_PARAMETER = "limit";
+ private static final int TOO_MANY_REQUESTS = 429;
+ private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
+ private static final JsonFactory JSON_FACTORY = OBJECT_MAPPER.getFactory();
+
+ private volatile WebClientServiceProvider webClientServiceProvider;
+
+ private static final List<PropertyDescriptor> PROPERTY_DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ CRM_ENDPOINT,
+ ACCESS_TOKEN,
+ LIMIT,
+ WEB_CLIENT_SERVICE_PROVIDER
+ ));
+
+ private static final Set<Relationship> RELATIONSHIPS = Collections.unmodifiableSet(new HashSet<>(
+ Collections.singletonList(
+ REL_SUCCESS
+ )));
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ webClientServiceProvider = context.getProperty(WEB_CLIENT_SERVICE_PROVIDER).asControllerService(WebClientServiceProvider.class);
+ }
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTY_DESCRIPTORS;
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return RELATIONSHIPS;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ final String accessToken = context.getProperty(ACCESS_TOKEN).getValue();
+ final String endpoint = context.getProperty(CRM_ENDPOINT).getValue();
+
+ final StateMap state = getStateMap(context);
+ final URI uri = createUri(context, state);
+
+ final HttpResponseEntity response = getHttpResponseEntity(accessToken, uri);
+ final AtomicInteger objectCountHolder = new AtomicInteger();
+
+ if (response.statusCode() == HttpResponseStatus.OK.getCode()) {
+ FlowFile flowFile = session.create();
+ flowFile = session.write(flowFile, parseHttpResponse(context, endpoint, state, response, objectCountHolder));
+ if (objectCountHolder.get() > 0) {
+ session.transfer(flowFile, REL_SUCCESS);
+ } else {
+ getLogger().debug("Empty response when requested HubSpot endpoint: [{}]", endpoint);
+ }
+ } else if (response.statusCode() >= 400) {
+ if (response.statusCode() == TOO_MANY_REQUESTS) {
+ context.yield();
+ throw new ProcessException(String.format("Rate limit exceeded, yielding before retrying request. HTTP %d error for requested URI [%s]", response.statusCode(), uri));
+ } else {
+ getLogger().warn("HTTP {} error for requested URI [{}]", response.statusCode(), uri);
+ }
+ }
+ }
+
+ private OutputStreamCallback parseHttpResponse(ProcessContext context, String endpoint, StateMap state, HttpResponseEntity response, AtomicInteger objectCountHolder) {
+ return out -> {
+ try (JsonParser jsonParser = JSON_FACTORY.createParser(response.body());
+ final JsonGenerator jsonGenerator = JSON_FACTORY.createGenerator(out, JsonEncoding.UTF8)) {
+ while (jsonParser.nextToken() != null) {
+ if (jsonParser.getCurrentToken() == JsonToken.FIELD_NAME && jsonParser.getCurrentName()
+ .equals("results")) {
+ jsonParser.nextToken();
+ jsonGenerator.copyCurrentStructure(jsonParser);
+ objectCountHolder.incrementAndGet();
+ }
+ String fieldName = jsonParser.getCurrentName();
+ if (CURSOR_PARAMETER.equals(fieldName)) {
+ jsonParser.nextToken();
+ Map<String, String> newStateMap = new HashMap<>(state.toMap());
+ newStateMap.put(endpoint, jsonParser.getText());
+ updateState(context, newStateMap);
+ break;
+ }
+ }
+ }
+ };
+ }
+
+ HttpUriBuilder getBaseUri(final ProcessContext context) {
+ final String path = context.getProperty(CRM_ENDPOINT).getValue();
+ return webClientServiceProvider.getHttpUriBuilder()
+ .scheme(HTTPS)
+ .host(API_BASE_URI)
+ .encodedPath(path);
+ }
+
+ private HttpResponseEntity getHttpResponseEntity(final String accessToken, final URI uri) {
+ return webClientServiceProvider.getWebClientService()
+ .get()
+ .uri(uri)
+ .header("Authorization", "Bearer " + accessToken)
+ .retrieve();
+ }
+
+ private URI createUri(final ProcessContext context, final StateMap state) {
+ final String path = context.getProperty(CRM_ENDPOINT).getValue();
+ final HttpUriBuilder uriBuilder = getBaseUri(context);
+
+ final boolean isLimitSet = context.getProperty(LIMIT).isSet();
+ if (isLimitSet) {
+ final String limit = context.getProperty(LIMIT).getValue();
Review Comment:
Since `LIMIT` supports EL shouldn't `PropertyValue.evaluateAttributeExpressions()` be called before getting the value?
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/test/java/org/apache/nifi/processors/hubspot/GetHubSpotTest.java:
##########
@@ -0,0 +1,199 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import okhttp3.HttpUrl;
+import okhttp3.mockwebserver.MockResponse;
+import okhttp3.mockwebserver.MockWebServer;
+import okhttp3.mockwebserver.RecordedRequest;
+import org.apache.commons.io.IOUtils;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.apache.nifi.web.client.StandardHttpUriBuilder;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.service.StandardWebClientServiceProvider;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Collections;
+import java.util.List;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+class GetHubSpotTest {
+
+ public static final String BASE_URL = "/test/hubspot";
+ public static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
+ private static MockWebServer server;
+ private static HttpUrl baseUrl;
+
+ @BeforeEach
+ void setup() throws IOException {
+ server = new MockWebServer();
+ server.start();
+ baseUrl = server.url(BASE_URL);
+ }
+
+ @AfterEach
+ void tearDown() throws IOException {
+ if (server != null) {
+ server.shutdown();
+ server = null;
+ }
+ }
+
+ @Test
+ void testLimitIsAddedToUrl() throws InitializationException, InterruptedException, IOException {
+
+ final String response = getResourceAsString("response-without-paging-cursor.json");
+ server.enqueue(new MockResponse().setResponseCode(200).setBody(response));
+
+ final StandardWebClientServiceProvider standardWebClientServiceProvider = new StandardWebClientServiceProvider();
+ final MockGetHubSpot mockGetHubSpot = new MockGetHubSpot();
+
+ TestRunner runner = TestRunners.newTestRunner(mockGetHubSpot);
+ runner.addControllerService("standardWebClientServiceProvider", standardWebClientServiceProvider);
+ runner.enableControllerService(standardWebClientServiceProvider);
+
+ runner.setProperty(GetHubSpot.WEB_CLIENT_SERVICE_PROVIDER, standardWebClientServiceProvider.getIdentifier());
+ runner.setProperty(GetHubSpot.ACCESS_TOKEN, "testToken");
+ runner.setProperty(GetHubSpot.CRM_ENDPOINT, CrmEndpoint.COMPANIES.getValue());
+ runner.setProperty(GetHubSpot.LIMIT, "1");
+
+ runner.run(1);
+
+ RecordedRequest request = server.takeRequest();
+ assertEquals(BASE_URL + "?limit=1", request.getPath());
+ }
+
+ @Test
+ void testPageCursorIsAddedToUrlFromState() throws InitializationException, InterruptedException, IOException {
+
+ final String response = getResourceAsString("response-without-paging-cursor.json");
+ server.enqueue(new MockResponse().setBody(response));
+
+ final StandardWebClientServiceProvider standardWebClientServiceProvider = new StandardWebClientServiceProvider();
+ final MockGetHubSpot mockGetHubSpot = new MockGetHubSpot();
+
+ TestRunner runner = TestRunners.newTestRunner(mockGetHubSpot);
+ runner.addControllerService("standardWebClientServiceProvider", standardWebClientServiceProvider);
+ runner.enableControllerService(standardWebClientServiceProvider);
+
+ runner.setProperty(GetHubSpot.WEB_CLIENT_SERVICE_PROVIDER, standardWebClientServiceProvider.getIdentifier());
+ runner.setProperty(GetHubSpot.ACCESS_TOKEN, "testToken");
+ runner.setProperty(GetHubSpot.CRM_ENDPOINT, CrmEndpoint.COMPANIES.getValue());
+ runner.setProperty(GetHubSpot.LIMIT, "1");
+
+ runner.getStateManager().setState(Collections.singletonMap(CrmEndpoint.COMPANIES.getValue(), "12345"), Scope.CLUSTER);
+
+ runner.run(1);
+
+ RecordedRequest request = server.takeRequest();
+ assertEquals(BASE_URL + "?limit=1&after=12345", request.getPath());
+ }
+
+ @Test
+ void testFlowFileContainsResultsArray() throws InitializationException, IOException {
+
+ final String response = getResourceAsString("response-with-paging-cursor.json");
Review Comment:
Please extract this to a constant `"response-with-paging-cursor.json"`
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.configuration.DefaultSettings;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpResponseStatus;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"hubspot"})
+@CapabilityDescription("Retrieves JSON data from a private HubSpot application."
+ + " Supports incremental retrieval: Users can set the \"limit\" property which serves as the upper limit of the retrieved objects."
+ + " When this property is set the processor will retrieve new records. This processor is intended to be run on the Primary Node only.")
+@Stateful(scopes = Scope.CLUSTER, description = "When the 'Limit' attribute is set, the paging cursor is saved after executing a request."
+ + " Only the objects after the paging cursor will be retrieved. The maximum number of retrieved objects is the 'Limit' attribute."
+ + " State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected,"
+ + " the new node can pick up where the previous node left off, without duplicating the data.")
+@DefaultSettings(yieldDuration = "10 sec")
+public class GetHubSpot extends AbstractProcessor {
+
+ static final PropertyDescriptor CRM_ENDPOINT = new PropertyDescriptor.Builder()
+ .name("crm-endpoint")
+ .displayName("HubSpot CRM API Endpoint")
+ .description("The HubSpot CRM API endpoint to which the Processor will send requests")
+ .required(true)
+ .allowableValues(CrmEndpoint.class)
+ .build();
+
+ static final PropertyDescriptor ACCESS_TOKEN = new PropertyDescriptor.Builder()
+ .name("access-token")
+ .displayName("Access Token")
+ .description("Access Token to authenticate requests")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+ .build();
+
+ static final PropertyDescriptor LIMIT = new PropertyDescriptor.Builder()
+ .name("crm-limit")
+ .displayName("Limit")
+ .description("The maximum number of results to request for each invocation of the Processor")
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor WEB_CLIENT_SERVICE_PROVIDER = new PropertyDescriptor.Builder()
+ .name("web-client-service-provider")
+ .displayName("Web Client Service Provider")
+ .description("Controller service for HTTP client operations")
+ .identifiesControllerService(WebClientServiceProvider.class)
+ .required(true)
+ .build();
+
+ static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("For FlowFiles created as a result of a successful HTTP request.")
+ .build();
+
+ static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("In case of HTTP client errors the flowfile will be routed to this relationship")
+ .build();
+
+ private static final String API_BASE_URI = "api.hubapi.com";
+ private static final String HTTPS = "https";
+ private static final String CURSOR_PARAMETER = "after";
+ private static final String LIMIT_PARAMETER = "limit";
+ private static final int TOO_MANY_REQUESTS = 429;
+ private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
+ private static final JsonFactory JSON_FACTORY = OBJECT_MAPPER.getFactory();
+
+ private volatile WebClientServiceProvider webClientServiceProvider;
+
+ private static final List<PropertyDescriptor> PROPERTY_DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ CRM_ENDPOINT,
+ ACCESS_TOKEN,
+ LIMIT,
+ WEB_CLIENT_SERVICE_PROVIDER
+ ));
+
+ private static final Set<Relationship> RELATIONSHIPS = Collections.unmodifiableSet(new HashSet<>(
+ Collections.singletonList(
+ REL_SUCCESS
+ )));
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ webClientServiceProvider = context.getProperty(WEB_CLIENT_SERVICE_PROVIDER).asControllerService(WebClientServiceProvider.class);
+ }
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTY_DESCRIPTORS;
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return RELATIONSHIPS;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ final String accessToken = context.getProperty(ACCESS_TOKEN).getValue();
+ final String endpoint = context.getProperty(CRM_ENDPOINT).getValue();
+
+ final StateMap state = getStateMap(context);
+ final URI uri = createUri(context, state);
+
+ final HttpResponseEntity response = getHttpResponseEntity(accessToken, uri);
+ final AtomicInteger objectCountHolder = new AtomicInteger();
+
+ if (response.statusCode() == HttpResponseStatus.OK.getCode()) {
+ FlowFile flowFile = session.create();
+ flowFile = session.write(flowFile, parseHttpResponse(context, endpoint, state, response, objectCountHolder));
+ if (objectCountHolder.get() > 0) {
+ session.transfer(flowFile, REL_SUCCESS);
+ } else {
+ getLogger().debug("Empty response when requested HubSpot endpoint: [{}]", endpoint);
+ }
+ } else if (response.statusCode() >= 400) {
+ if (response.statusCode() == TOO_MANY_REQUESTS) {
+ context.yield();
+ throw new ProcessException(String.format("Rate limit exceeded, yielding before retrying request. HTTP %d error for requested URI [%s]", response.statusCode(), uri));
+ } else {
+ getLogger().warn("HTTP {} error for requested URI [{}]", response.statusCode(), uri);
+ }
+ }
+ }
+
+ private OutputStreamCallback parseHttpResponse(ProcessContext context, String endpoint, StateMap state, HttpResponseEntity response, AtomicInteger objectCountHolder) {
+ return out -> {
+ try (JsonParser jsonParser = JSON_FACTORY.createParser(response.body());
+ final JsonGenerator jsonGenerator = JSON_FACTORY.createGenerator(out, JsonEncoding.UTF8)) {
Review Comment:
Indentation is 5 space instead of 4
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/test/java/org/apache/nifi/processors/hubspot/GetHubSpotTest.java:
##########
@@ -0,0 +1,199 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import okhttp3.HttpUrl;
+import okhttp3.mockwebserver.MockResponse;
+import okhttp3.mockwebserver.MockWebServer;
+import okhttp3.mockwebserver.RecordedRequest;
+import org.apache.commons.io.IOUtils;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.apache.nifi.web.client.StandardHttpUriBuilder;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.service.StandardWebClientServiceProvider;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Collections;
+import java.util.List;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+class GetHubSpotTest {
+
+ public static final String BASE_URL = "/test/hubspot";
+ public static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
+ private static MockWebServer server;
+ private static HttpUrl baseUrl;
+
+ @BeforeEach
+ void setup() throws IOException {
+ server = new MockWebServer();
+ server.start();
+ baseUrl = server.url(BASE_URL);
+ }
+
+ @AfterEach
+ void tearDown() throws IOException {
+ if (server != null) {
+ server.shutdown();
+ server = null;
+ }
+ }
+
+ @Test
+ void testLimitIsAddedToUrl() throws InitializationException, InterruptedException, IOException {
+
+ final String response = getResourceAsString("response-without-paging-cursor.json");
Review Comment:
Please extract this to a constant `"response-without-paging-cursor.json"`
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.configuration.DefaultSettings;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpResponseStatus;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"hubspot"})
+@CapabilityDescription("Retrieves JSON data from a private HubSpot application."
+ + " Supports incremental retrieval: Users can set the \"limit\" property which serves as the upper limit of the retrieved objects."
+ + " When this property is set the processor will retrieve new records. This processor is intended to be run on the Primary Node only.")
+@Stateful(scopes = Scope.CLUSTER, description = "When the 'Limit' attribute is set, the paging cursor is saved after executing a request."
+ + " Only the objects after the paging cursor will be retrieved. The maximum number of retrieved objects is the 'Limit' attribute."
+ + " State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected,"
+ + " the new node can pick up where the previous node left off, without duplicating the data.")
+@DefaultSettings(yieldDuration = "10 sec")
+public class GetHubSpot extends AbstractProcessor {
+
+ static final PropertyDescriptor CRM_ENDPOINT = new PropertyDescriptor.Builder()
+ .name("crm-endpoint")
+ .displayName("HubSpot CRM API Endpoint")
+ .description("The HubSpot CRM API endpoint to which the Processor will send requests")
+ .required(true)
+ .allowableValues(CrmEndpoint.class)
+ .build();
+
+ static final PropertyDescriptor ACCESS_TOKEN = new PropertyDescriptor.Builder()
+ .name("access-token")
+ .displayName("Access Token")
+ .description("Access Token to authenticate requests")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+ .build();
+
+ static final PropertyDescriptor LIMIT = new PropertyDescriptor.Builder()
+ .name("crm-limit")
+ .displayName("Limit")
+ .description("The maximum number of results to request for each invocation of the Processor")
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor WEB_CLIENT_SERVICE_PROVIDER = new PropertyDescriptor.Builder()
+ .name("web-client-service-provider")
+ .displayName("Web Client Service Provider")
+ .description("Controller service for HTTP client operations")
+ .identifiesControllerService(WebClientServiceProvider.class)
+ .required(true)
+ .build();
+
+ static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("For FlowFiles created as a result of a successful HTTP request.")
+ .build();
+
+ static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("In case of HTTP client errors the flowfile will be routed to this relationship")
+ .build();
+
+ private static final String API_BASE_URI = "api.hubapi.com";
+ private static final String HTTPS = "https";
+ private static final String CURSOR_PARAMETER = "after";
+ private static final String LIMIT_PARAMETER = "limit";
+ private static final int TOO_MANY_REQUESTS = 429;
+ private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
+ private static final JsonFactory JSON_FACTORY = OBJECT_MAPPER.getFactory();
+
+ private volatile WebClientServiceProvider webClientServiceProvider;
+
+ private static final List<PropertyDescriptor> PROPERTY_DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ CRM_ENDPOINT,
+ ACCESS_TOKEN,
+ LIMIT,
+ WEB_CLIENT_SERVICE_PROVIDER
+ ));
+
+ private static final Set<Relationship> RELATIONSHIPS = Collections.unmodifiableSet(new HashSet<>(
Review Comment:
You can use `Collections.singleton(REL_SUCCESS)` to instantiate a singleton immutable set
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #6301: NIFI-10356: Create GetHubSpot processor
Posted by GitBox <gi...@apache.org>.
exceptionfactory commented on code in PR #6301:
URL: https://github.com/apache/nifi/pull/6301#discussion_r953100112
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.configuration.DefaultSettings;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+
+@PrimaryNodeOnly
+@TriggerSerially
Review Comment:
Thanks for the reply @Lehel44, these annotations make sense given the approach implemented in other source processors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #6301: NIFI-10356: Create GetHubSpot processor
Posted by GitBox <gi...@apache.org>.
exceptionfactory commented on code in PR #6301:
URL: https://github.com/apache/nifi/pull/6301#discussion_r950160505
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.configuration.DefaultSettings;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+
+@PrimaryNodeOnly
+@TriggerSerially
Review Comment:
Is this annotation necessary? It seems safe to have concurrent invocations in some scenarios, unless HubSpot only allows one concurrent request per client.
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-nar/pom.xml:
##########
@@ -0,0 +1,46 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>nifi-hubspot-bundle</artifactId>
+ <groupId>org.apache.nifi</groupId>
+ <version>1.18.0-SNAPSHOT</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>nifi-hubspot-nar</artifactId>
+
+ <packaging>nar</packaging>
+ <properties>
+ <maven.javadoc.skip>true</maven.javadoc.skip>
+ <source.skip>true</source.skip>
+ </properties>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-hubspot-processors</artifactId>
+ <version>1.18.0-SNAPSHOT</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-record-serialization-services-nar</artifactId>
Review Comment:
This dependency should be changed to the `nifi-standard-services-api-nar`.
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.configuration.DefaultSettings;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"hubspot"})
+@CapabilityDescription("Retrieves JSON data from a private HubSpot application."
+ + " Supports incremental retrieval: Users can set the \"limit\" property which serves as the upper limit of the retrieved objects."
+ + " When this property is set the processor will retrieve new records. This processor is intended to be run on the Primary Node only.")
+@Stateful(scopes = Scope.CLUSTER, description = "When the 'Limit' attribute is set, the paging cursor is saved after executing a request."
+ + " Only the objects after the paging cursor will be retrieved. The maximum number of retrieved objects is the 'Limit' attribute."
+ + " State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected,"
+ + " the new node can pick up where the previous node left off, without duplicating the data.")
+@DefaultSettings(yieldDuration = "10 sec")
+public class GetHubSpot extends AbstractProcessor {
+
+ static final PropertyDescriptor ACCESS_TOKEN = new PropertyDescriptor.Builder()
+ .name("admin-api-access-token")
Review Comment:
This should be changed to `access-token`:
```suggestion
.name("access-token")
```
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/CrmEndpoint.java:
##########
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.DescribedValue;
+
+public enum CrmEndpoint implements DescribedValue {
+
+ COMPANIES(
+ "/crm/v3/objects/companies",
+ "Companies",
+ "In HubSpot, the companies object is a standard CRM object. Individual company records can be used to store information about businesses" +
+ " and organizations within company properties."
+ ),
+ CONTACTS(
+ "/crm/v3/objects/contacts",
+ "Contacts",
+ "In HubSpot, contacts store information about individuals. From marketing automation to smart content, the lead-specific data found in" +
+ " contact records helps users leverage much of HubSpot's functionality."
+ ),
+ DEALS(
+ "/crm/v3/objects/deals",
+ "Deals",
+ "In HubSpot, a deal represents an ongoing transaction that a sales team is pursuing with a contact or company. It’s tracked through" +
+ " pipeline stages until won or lost."
+ ),
+ FEEDBACK_SUBMISSIONS(
+ "/crm/v3/objects/feedback_submissions",
+ "Feedback Submissions",
+ "In HubSpot, feedback submissions are an object which stores information submitted to a feedback survey. This includes Net Promoter Score (NPS)," +
+ " Customer Satisfaction (CSAT), Customer Effort Score (CES) and Custom Surveys."
+ ),
+ LINE_ITEMS(
+ "/crm/v3/objects/line_items",
+ "Line Items",
+ "In HubSpot, line items can be thought of as a subset of products. When a product is attached to a deal, it becomes a line item. Line items can" +
+ " be created that are unique to an individual quote, but they will not be added to the product library."
+ ),
+ PRODUCTS(
+ "/crm/v3/objects/products",
+ "Products",
+ "In HubSpot, products represent the goods or services to be sold. Building a product library allows the user to quickly add products to deals," +
+ " generate quotes, and report on product performance."
+ ),
+ TICKETS(
+ "/crm/v3/objects/tickets",
+ "Tickets",
+ "In HubSpot, a ticket represents a customer request for help or support."
+ ),
+ QUOTES(
+ "/crm/v3/objects/quotes",
+ "Quotes",
+ "In HubSpot, quotes are used to share pricing information with potential buyers."
+ ),
+
+ CALLS(
+ "/crm/v3/objects/calls",
+ "Calls",
+ "Get calls on CRM records and on the calls index page."
+ ),
+ EMAILS(
+ "/crm/v3/objects/emails",
+ "Emails",
+ "Get emails on CRM records."
+ ),
+ MEETINGS(
+ "/crm/v3/objects/meetings",
+ "Meetings",
+ "Get meetings on CRM records."
+ ),
+ NOTES(
+ "/crm/v3/objects/notes",
+ "Notes",
+ "Get notes on CRM records."
+ ),
+ TASKS(
+ "/crm/v3/objects/tasks",
+ "Tasks",
+ "Get tasks on CRM records."
+ ),
+
+ OWNERS(
+ "/crm/v3/owners/",
+ "Owners",
+ "HubSpot uses owners to assign specific users to contacts, companies, deals, tickets, or engagements. Any HubSpot user with access to contacts" +
+ " can be assigned as an owner, and multiple owners can be assigned to an object by creating a custom property for this purpose."
+ );
+
+
+ private final String value;
+ private final String displayName;
+ private final String description;
+
+ CrmEndpoint(final String value, final String displayName, final String description) {
+ this.value = value;
+ this.displayName = displayName;
+ this.description = description;
+ }
+
+ @Override
+ public String getValue() {
+ return value;
+ }
+
+ @Override
+ public String getDisplayName() {
+ return displayName;
+ }
+
+ @Override
+ public String getDescription() {
+ return description;
+ }
+
+ public AllowableValue getAllowableValue() {
+ return new AllowableValue(value, displayName, description);
+ }
Review Comment:
This method seems unnecessary, for references in test code, it should be sufficient to use `getValue()`.
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.configuration.DefaultSettings;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"hubspot"})
+@CapabilityDescription("Retrieves JSON data from a private HubSpot application."
+ + " Supports incremental retrieval: Users can set the \"limit\" property which serves as the upper limit of the retrieved objects."
+ + " When this property is set the processor will retrieve new records. This processor is intended to be run on the Primary Node only.")
+@Stateful(scopes = Scope.CLUSTER, description = "When the 'Limit' attribute is set, the paging cursor is saved after executing a request."
+ + " Only the objects after the paging cursor will be retrieved. The maximum number of retrieved objects is the 'Limit' attribute."
+ + " State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected,"
+ + " the new node can pick up where the previous node left off, without duplicating the data.")
+@DefaultSettings(yieldDuration = "10 sec")
+public class GetHubSpot extends AbstractProcessor {
+
+ static final PropertyDescriptor ACCESS_TOKEN = new PropertyDescriptor.Builder()
+ .name("admin-api-access-token")
+ .displayName("Access Token")
+ .description("Access Token to authenticate requests")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+ .build();
+
+ static final PropertyDescriptor CRM_ENDPOINT = new PropertyDescriptor.Builder()
+ .name("crm-endpoint")
+ .displayName("HubSpot CRM API Endpoint")
+ .description("The HubSpot CRM API endpoint to get")
+ .required(true)
+ .allowableValues(CrmEndpoint.class)
+ .build();
+
+ static final PropertyDescriptor LIMIT = new PropertyDescriptor.Builder()
+ .name("crm-limit")
+ .displayName("Limit")
+ .description("The maximum number of results to display per page")
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor WEB_CLIENT_SERVICE_PROVIDER = new PropertyDescriptor.Builder()
+ .name("web-client-service-provider")
+ .displayName("Web Client Service Provider")
+ .description("Controller service for HTTP client operations")
+ .identifiesControllerService(WebClientServiceProvider.class)
+ .required(true)
+ .build();
+
+ static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("For FlowFiles created as a result of a successful HTTP request.")
+ .build();
+
+ static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("In case of HTTP client errors the flowfile will be routed to this relationship")
+ .build();
+
+ private static final String API_BASE_URI = "api.hubapi.com";
+ private static final String HTTPS = "https";
+ private static final String CURSOR_PARAMETER = "after";
+ private static final String LIMIT_PARAMETER = "limit";
+ private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
+ private static final JsonFactory JSON_FACTORY = OBJECT_MAPPER.getFactory();
+
+ private volatile WebClientServiceProvider webClientServiceProvider;
+
+ private static final List<PropertyDescriptor> PROPERTY_DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ ACCESS_TOKEN,
+ CRM_ENDPOINT,
+ LIMIT,
+ WEB_CLIENT_SERVICE_PROVIDER
+ ));
+
+ private static final Set<Relationship> RELATIONSHIPS = Collections.unmodifiableSet(new HashSet<>(Arrays.asList(
+ REL_SUCCESS,
+ REL_FAILURE
+ )));
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ webClientServiceProvider = context.getProperty(WEB_CLIENT_SERVICE_PROVIDER).asControllerService(WebClientServiceProvider.class);
+ }
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTY_DESCRIPTORS;
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return RELATIONSHIPS;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ final String accessToken = context.getProperty(ACCESS_TOKEN).getValue();
+ final String endpoint = context.getProperty(CRM_ENDPOINT).getValue();
+
+ final StateMap state = getStateMap(context);
+ final URI uri = createUri(context, state);
+
+ final HttpResponseEntity response = getHttpResponseEntity(accessToken, uri);
+ final AtomicInteger objectCountHolder = new AtomicInteger();
+
+ FlowFile flowFile = session.create();
+ flowFile = session.putAttribute(flowFile, "statusCode", String.valueOf(response.statusCode()));
+
+ flowFile = session.write(flowFile, out -> {
Review Comment:
Based on other comments, removing the failure relationship would allow FlowFile creation to be moved inside of the conditional check for a successful response status code.
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.configuration.DefaultSettings;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"hubspot"})
+@CapabilityDescription("Retrieves JSON data from a private HubSpot application."
+ + " Supports incremental retrieval: Users can set the \"limit\" property which serves as the upper limit of the retrieved objects."
+ + " When this property is set the processor will retrieve new records. This processor is intended to be run on the Primary Node only.")
+@Stateful(scopes = Scope.CLUSTER, description = "When the 'Limit' attribute is set, the paging cursor is saved after executing a request."
+ + " Only the objects after the paging cursor will be retrieved. The maximum number of retrieved objects is the 'Limit' attribute."
+ + " State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected,"
+ + " the new node can pick up where the previous node left off, without duplicating the data.")
+@DefaultSettings(yieldDuration = "10 sec")
+public class GetHubSpot extends AbstractProcessor {
+
+ static final PropertyDescriptor ACCESS_TOKEN = new PropertyDescriptor.Builder()
+ .name("admin-api-access-token")
+ .displayName("Access Token")
+ .description("Access Token to authenticate requests")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+ .build();
+
+ static final PropertyDescriptor CRM_ENDPOINT = new PropertyDescriptor.Builder()
+ .name("crm-endpoint")
+ .displayName("HubSpot CRM API Endpoint")
+ .description("The HubSpot CRM API endpoint to get")
+ .required(true)
+ .allowableValues(CrmEndpoint.class)
+ .build();
+
+ static final PropertyDescriptor LIMIT = new PropertyDescriptor.Builder()
+ .name("crm-limit")
+ .displayName("Limit")
+ .description("The maximum number of results to display per page")
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor WEB_CLIENT_SERVICE_PROVIDER = new PropertyDescriptor.Builder()
+ .name("web-client-service-provider")
+ .displayName("Web Client Service Provider")
+ .description("Controller service for HTTP client operations")
+ .identifiesControllerService(WebClientServiceProvider.class)
+ .required(true)
+ .build();
+
+ static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("For FlowFiles created as a result of a successful HTTP request.")
+ .build();
+
+ static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("In case of HTTP client errors the flowfile will be routed to this relationship")
+ .build();
Review Comment:
As a source Processor with input forbidden, having a failure relationship does not seem to follow the model of other processors. Recommend removing this relationship given the general design.
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.configuration.DefaultSettings;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"hubspot"})
+@CapabilityDescription("Retrieves JSON data from a private HubSpot application."
+ + " Supports incremental retrieval: Users can set the \"limit\" property which serves as the upper limit of the retrieved objects."
+ + " When this property is set the processor will retrieve new records. This processor is intended to be run on the Primary Node only.")
+@Stateful(scopes = Scope.CLUSTER, description = "When the 'Limit' attribute is set, the paging cursor is saved after executing a request."
+ + " Only the objects after the paging cursor will be retrieved. The maximum number of retrieved objects is the 'Limit' attribute."
+ + " State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected,"
+ + " the new node can pick up where the previous node left off, without duplicating the data.")
+@DefaultSettings(yieldDuration = "10 sec")
+public class GetHubSpot extends AbstractProcessor {
+
+ static final PropertyDescriptor ACCESS_TOKEN = new PropertyDescriptor.Builder()
+ .name("admin-api-access-token")
+ .displayName("Access Token")
+ .description("Access Token to authenticate requests")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+ .build();
+
+ static final PropertyDescriptor CRM_ENDPOINT = new PropertyDescriptor.Builder()
+ .name("crm-endpoint")
+ .displayName("HubSpot CRM API Endpoint")
+ .description("The HubSpot CRM API endpoint to get")
+ .required(true)
+ .allowableValues(CrmEndpoint.class)
+ .build();
+
+ static final PropertyDescriptor LIMIT = new PropertyDescriptor.Builder()
+ .name("crm-limit")
+ .displayName("Limit")
+ .description("The maximum number of results to display per page")
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor WEB_CLIENT_SERVICE_PROVIDER = new PropertyDescriptor.Builder()
+ .name("web-client-service-provider")
+ .displayName("Web Client Service Provider")
+ .description("Controller service for HTTP client operations")
+ .identifiesControllerService(WebClientServiceProvider.class)
+ .required(true)
+ .build();
+
+ static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("For FlowFiles created as a result of a successful HTTP request.")
+ .build();
+
+ static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("In case of HTTP client errors the flowfile will be routed to this relationship")
+ .build();
+
+ private static final String API_BASE_URI = "api.hubapi.com";
+ private static final String HTTPS = "https";
+ private static final String CURSOR_PARAMETER = "after";
+ private static final String LIMIT_PARAMETER = "limit";
+ private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
+ private static final JsonFactory JSON_FACTORY = OBJECT_MAPPER.getFactory();
+
+ private volatile WebClientServiceProvider webClientServiceProvider;
+
+ private static final List<PropertyDescriptor> PROPERTY_DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ ACCESS_TOKEN,
+ CRM_ENDPOINT,
+ LIMIT,
+ WEB_CLIENT_SERVICE_PROVIDER
+ ));
+
+ private static final Set<Relationship> RELATIONSHIPS = Collections.unmodifiableSet(new HashSet<>(Arrays.asList(
+ REL_SUCCESS,
+ REL_FAILURE
+ )));
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ webClientServiceProvider = context.getProperty(WEB_CLIENT_SERVICE_PROVIDER).asControllerService(WebClientServiceProvider.class);
+ }
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTY_DESCRIPTORS;
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return RELATIONSHIPS;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ final String accessToken = context.getProperty(ACCESS_TOKEN).getValue();
+ final String endpoint = context.getProperty(CRM_ENDPOINT).getValue();
+
+ final StateMap state = getStateMap(context);
+ final URI uri = createUri(context, state);
+
+ final HttpResponseEntity response = getHttpResponseEntity(accessToken, uri);
+ final AtomicInteger objectCountHolder = new AtomicInteger();
+
+ FlowFile flowFile = session.create();
+ flowFile = session.putAttribute(flowFile, "statusCode", String.valueOf(response.statusCode()));
+
+ flowFile = session.write(flowFile, out -> {
+
+ try (JsonParser jsonParser = JSON_FACTORY.createParser(response.body());
+ final JsonGenerator jsonGenerator = JSON_FACTORY.createGenerator(out, JsonEncoding.UTF8)) {
+ while (jsonParser.nextToken() != null) {
+ if (jsonParser.getCurrentToken() == JsonToken.FIELD_NAME && jsonParser.getCurrentName().equals("results")) {
+ jsonParser.nextToken();
+ jsonGenerator.copyCurrentStructure(jsonParser);
+ objectCountHolder.incrementAndGet();
+ }
+ String fieldName = jsonParser.getCurrentName();
+ if (CURSOR_PARAMETER.equals(fieldName)) {
+ jsonParser.nextToken();
+ Map<String, String> newStateMap = new HashMap<>(state.toMap());
+ newStateMap.put(endpoint, jsonParser.getText());
+ updateState(context, newStateMap);
+ break;
+ }
+ }
+ }
+ });
+ if (response.statusCode() >= 400) {
+ if (response.statusCode() == 429) {
+ context.yield();
+ throw new ProcessException("Rate limit exceeded, yielding for 10 seconds before retrying request.");
+ } else {
+ getLogger().warn("HTTP [{}] client error occurred at endpoint [{}]", response.statusCode(), endpoint);
+ session.transfer(flowFile, REL_FAILURE);
Review Comment:
As mentioned on the relationship, changing the implementation to only create a FlowFile on success follows the pattern of other "source" Processors, so this could be removed.
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.configuration.DefaultSettings;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"hubspot"})
+@CapabilityDescription("Retrieves JSON data from a private HubSpot application."
+ + " Supports incremental retrieval: Users can set the \"limit\" property which serves as the upper limit of the retrieved objects."
+ + " When this property is set the processor will retrieve new records. This processor is intended to be run on the Primary Node only.")
+@Stateful(scopes = Scope.CLUSTER, description = "When the 'Limit' attribute is set, the paging cursor is saved after executing a request."
+ + " Only the objects after the paging cursor will be retrieved. The maximum number of retrieved objects is the 'Limit' attribute."
+ + " State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected,"
+ + " the new node can pick up where the previous node left off, without duplicating the data.")
+@DefaultSettings(yieldDuration = "10 sec")
+public class GetHubSpot extends AbstractProcessor {
+
+ static final PropertyDescriptor ACCESS_TOKEN = new PropertyDescriptor.Builder()
+ .name("admin-api-access-token")
+ .displayName("Access Token")
+ .description("Access Token to authenticate requests")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+ .build();
+
+ static final PropertyDescriptor CRM_ENDPOINT = new PropertyDescriptor.Builder()
+ .name("crm-endpoint")
+ .displayName("HubSpot CRM API Endpoint")
+ .description("The HubSpot CRM API endpoint to get")
+ .required(true)
+ .allowableValues(CrmEndpoint.class)
+ .build();
+
+ static final PropertyDescriptor LIMIT = new PropertyDescriptor.Builder()
+ .name("crm-limit")
+ .displayName("Limit")
+ .description("The maximum number of results to display per page")
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor WEB_CLIENT_SERVICE_PROVIDER = new PropertyDescriptor.Builder()
+ .name("web-client-service-provider")
+ .displayName("Web Client Service Provider")
+ .description("Controller service for HTTP client operations")
+ .identifiesControllerService(WebClientServiceProvider.class)
+ .required(true)
+ .build();
+
+ static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("For FlowFiles created as a result of a successful HTTP request.")
+ .build();
+
+ static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("In case of HTTP client errors the flowfile will be routed to this relationship")
+ .build();
+
+ private static final String API_BASE_URI = "api.hubapi.com";
+ private static final String HTTPS = "https";
+ private static final String CURSOR_PARAMETER = "after";
+ private static final String LIMIT_PARAMETER = "limit";
+ private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
+ private static final JsonFactory JSON_FACTORY = OBJECT_MAPPER.getFactory();
+
+ private volatile WebClientServiceProvider webClientServiceProvider;
+
+ private static final List<PropertyDescriptor> PROPERTY_DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ ACCESS_TOKEN,
+ CRM_ENDPOINT,
+ LIMIT,
+ WEB_CLIENT_SERVICE_PROVIDER
+ ));
+
+ private static final Set<Relationship> RELATIONSHIPS = Collections.unmodifiableSet(new HashSet<>(Arrays.asList(
+ REL_SUCCESS,
+ REL_FAILURE
+ )));
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ webClientServiceProvider = context.getProperty(WEB_CLIENT_SERVICE_PROVIDER).asControllerService(WebClientServiceProvider.class);
+ }
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTY_DESCRIPTORS;
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return RELATIONSHIPS;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ final String accessToken = context.getProperty(ACCESS_TOKEN).getValue();
+ final String endpoint = context.getProperty(CRM_ENDPOINT).getValue();
+
+ final StateMap state = getStateMap(context);
+ final URI uri = createUri(context, state);
+
+ final HttpResponseEntity response = getHttpResponseEntity(accessToken, uri);
+ final AtomicInteger objectCountHolder = new AtomicInteger();
+
+ FlowFile flowFile = session.create();
+ flowFile = session.putAttribute(flowFile, "statusCode", String.valueOf(response.statusCode()));
+
+ flowFile = session.write(flowFile, out -> {
+
+ try (JsonParser jsonParser = JSON_FACTORY.createParser(response.body());
+ final JsonGenerator jsonGenerator = JSON_FACTORY.createGenerator(out, JsonEncoding.UTF8)) {
+ while (jsonParser.nextToken() != null) {
+ if (jsonParser.getCurrentToken() == JsonToken.FIELD_NAME && jsonParser.getCurrentName().equals("results")) {
+ jsonParser.nextToken();
+ jsonGenerator.copyCurrentStructure(jsonParser);
+ objectCountHolder.incrementAndGet();
+ }
+ String fieldName = jsonParser.getCurrentName();
+ if (CURSOR_PARAMETER.equals(fieldName)) {
+ jsonParser.nextToken();
+ Map<String, String> newStateMap = new HashMap<>(state.toMap());
+ newStateMap.put(endpoint, jsonParser.getText());
+ updateState(context, newStateMap);
+ break;
+ }
+ }
+ }
+ });
+ if (response.statusCode() >= 400) {
+ if (response.statusCode() == 429) {
+ context.yield();
+ throw new ProcessException("Rate limit exceeded, yielding for 10 seconds before retrying request.");
+ } else {
+ getLogger().warn("HTTP [{}] client error occurred at endpoint [{}]", response.statusCode(), endpoint);
Review Comment:
Recommend a minor adjustment to remove the `[]` characters around the status code since it will always be a number, and also adjusting the wording to include the complete URI.
```suggestion
getLogger().warn("HTTP {} error for requested URI [{}]", response.statusCode(), uri);
```
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/resources/docs/org.apache.nifi.processors.hubspot.GetHubSpot/additionalDetails.html:
##########
@@ -0,0 +1,143 @@
+<!DOCTYPE html>
+<html lang="en">
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<head>
+ <meta charset="utf-8"/>
+ <title>GetHubSpot</title>
+ <link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"/>
+</head>
+
+<body>
+<h2>Incremental Loading</h2>
+<p>
+ Some resources can be processed incrementally by NiFi. This means that only resources created or modified after the last run
+ time of the processor are displayed. The processor state can be reset in the context menu. The following list shows which
+ date-time fields are incremented for which resources.
+<ul>
+ <li>Access
+ <ul>
+ <li>Access Scope: none</li>
+ <li>StoreFront Access Token: none</li>
+ </ul>
+ </li>
+ <li>Analytics
+ <ul>
+ <li>Reports: updated_at_min</li>
+ </ul>
+ </li>
+ <li>Billing
+ <ul>
+ <li>Application Charge: none</li>
+ <li>Application Credit: none</li>
+ <li>Recurring Application Charge: none</li>
+ </ul>
+ </li>
+ <li>Customers
+ <ul>
+ <li>Customers: updated_at_min</li>
+ <li>Customer Saved Searches: none</li>
+ </ul>
+ </li>
+ <li>Discounts
+ <ul>
+ <li>Price Rules: updated_at_min</li>
+ </ul>
+ </li>
+ <li>Events
+ <ul>
+ <li>Events: created_at_min</li>
+ </ul>
+ </li>
+ <li>Inventory
+ <ul>
+ <li>Inventory Levels: updated_at_min</li>
+ <li>Locations: none</li>
+ </ul>
+ </li>
+ <li>Marketing Event
+ <ul>
+ <li>Marketing Events: none</li>
+ </ul>
+ </li>
+ <li>Metafields
+ <ul>
+ <li>Metafields: updated_at_min</li>
+ </ul>
+ </li>
+ <li>Online Store
+ <ul>
+ <li>Blogs: none</li>
+ <li>Comment: none</li>
+ <li>Pages: none</li>
+ <li>Redirects: none</li>
+ <li>Script Tags: updated_at_min</li>
+ <li>Themes: none</li>
+ </ul>
+ </li>
+ <li>Orders
+ <ul>
+ <li>Abandoned Checkouts: updated_at_min</li>
+ <li>Draft Orders: updated_at_min</li>
+ <li>Orders: updated_at_min</li>
+ </ul>
+ </li>
+ <li>Plus
+ <ul>
+ <li>Gift Cards: none</li>
+ <li>Users: none</li>
+ </ul>
+ </li>
+ <li>Product
+ <ul>
+ <li>Collects: none</li>
+ <li>Custom Collections: updated_at_min</li>
+ <li>Products: updated_at_min</li>
+ <li>Smart Collections: updated_at_min</li>
+ </ul>
+ </li>
+ <li>Sales Channels
+ <ul>
+ <li>Collection Listings: none</li>
+ <li>Mobile Platform Applications: none</li>
+ <li>Product Listings: updated_at_min</li>
+ <li>Resource Feedbacks: none</li>
+ </ul>
+ </li>
+ <li>Shipping and Fulfillments
+ <ul>
+ <li>Carrier Services: none</li>
+ </ul>
+ </li>
+ <li>Store Properties
+ <ul>
+ <li>Countries: none</li>
+ <li>Currencies: none</li>
+ <li>Policies: none</li>
+ <li>Shipping Zones: updated_at_min</li>
+ <li>Shop: none</li>
+ </ul>
+ </li>
+ <li>Tender Transactions
+ <ul>
+ <li>Tender Transactions: processed_at_min</li>
+ </ul>
+ </li>
+</ul>
+
+Specific trap type can set in case of Enterprise Specific generic trap type is chosen.
Review Comment:
Does this comment apply to this processor, or should it be removed?
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.configuration.DefaultSettings;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"hubspot"})
+@CapabilityDescription("Retrieves JSON data from a private HubSpot application."
+ + " Supports incremental retrieval: Users can set the \"limit\" property which serves as the upper limit of the retrieved objects."
+ + " When this property is set the processor will retrieve new records. This processor is intended to be run on the Primary Node only.")
+@Stateful(scopes = Scope.CLUSTER, description = "When the 'Limit' attribute is set, the paging cursor is saved after executing a request."
+ + " Only the objects after the paging cursor will be retrieved. The maximum number of retrieved objects is the 'Limit' attribute."
+ + " State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected,"
+ + " the new node can pick up where the previous node left off, without duplicating the data.")
+@DefaultSettings(yieldDuration = "10 sec")
+public class GetHubSpot extends AbstractProcessor {
+
+ static final PropertyDescriptor ACCESS_TOKEN = new PropertyDescriptor.Builder()
+ .name("admin-api-access-token")
+ .displayName("Access Token")
+ .description("Access Token to authenticate requests")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+ .build();
+
+ static final PropertyDescriptor CRM_ENDPOINT = new PropertyDescriptor.Builder()
+ .name("crm-endpoint")
+ .displayName("HubSpot CRM API Endpoint")
+ .description("The HubSpot CRM API endpoint to get")
+ .required(true)
+ .allowableValues(CrmEndpoint.class)
+ .build();
+
+ static final PropertyDescriptor LIMIT = new PropertyDescriptor.Builder()
+ .name("crm-limit")
+ .displayName("Limit")
+ .description("The maximum number of results to display per page")
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor WEB_CLIENT_SERVICE_PROVIDER = new PropertyDescriptor.Builder()
+ .name("web-client-service-provider")
+ .displayName("Web Client Service Provider")
+ .description("Controller service for HTTP client operations")
+ .identifiesControllerService(WebClientServiceProvider.class)
+ .required(true)
+ .build();
+
+ static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("For FlowFiles created as a result of a successful HTTP request.")
+ .build();
+
+ static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("In case of HTTP client errors the flowfile will be routed to this relationship")
+ .build();
+
+ private static final String API_BASE_URI = "api.hubapi.com";
+ private static final String HTTPS = "https";
+ private static final String CURSOR_PARAMETER = "after";
+ private static final String LIMIT_PARAMETER = "limit";
+ private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
+ private static final JsonFactory JSON_FACTORY = OBJECT_MAPPER.getFactory();
+
+ private volatile WebClientServiceProvider webClientServiceProvider;
+
+ private static final List<PropertyDescriptor> PROPERTY_DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ ACCESS_TOKEN,
+ CRM_ENDPOINT,
Review Comment:
As mentioned, recommend placing the CRM_ENDPOINT first in the list of properties.
```suggestion
CRM_ENDPOINT,
ACCESS_TOKEN,
```
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.configuration.DefaultSettings;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"hubspot"})
+@CapabilityDescription("Retrieves JSON data from a private HubSpot application."
+ + " Supports incremental retrieval: Users can set the \"limit\" property which serves as the upper limit of the retrieved objects."
+ + " When this property is set the processor will retrieve new records. This processor is intended to be run on the Primary Node only.")
+@Stateful(scopes = Scope.CLUSTER, description = "When the 'Limit' attribute is set, the paging cursor is saved after executing a request."
+ + " Only the objects after the paging cursor will be retrieved. The maximum number of retrieved objects is the 'Limit' attribute."
+ + " State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected,"
+ + " the new node can pick up where the previous node left off, without duplicating the data.")
+@DefaultSettings(yieldDuration = "10 sec")
+public class GetHubSpot extends AbstractProcessor {
+
+ static final PropertyDescriptor ACCESS_TOKEN = new PropertyDescriptor.Builder()
+ .name("admin-api-access-token")
+ .displayName("Access Token")
+ .description("Access Token to authenticate requests")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+ .build();
+
+ static final PropertyDescriptor CRM_ENDPOINT = new PropertyDescriptor.Builder()
Review Comment:
Recommend moving this property before `ACCESS_TOKEN`.
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.configuration.DefaultSettings;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"hubspot"})
+@CapabilityDescription("Retrieves JSON data from a private HubSpot application."
+ + " Supports incremental retrieval: Users can set the \"limit\" property which serves as the upper limit of the retrieved objects."
+ + " When this property is set the processor will retrieve new records. This processor is intended to be run on the Primary Node only.")
+@Stateful(scopes = Scope.CLUSTER, description = "When the 'Limit' attribute is set, the paging cursor is saved after executing a request."
+ + " Only the objects after the paging cursor will be retrieved. The maximum number of retrieved objects is the 'Limit' attribute."
+ + " State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected,"
+ + " the new node can pick up where the previous node left off, without duplicating the data.")
+@DefaultSettings(yieldDuration = "10 sec")
+public class GetHubSpot extends AbstractProcessor {
+
+ static final PropertyDescriptor ACCESS_TOKEN = new PropertyDescriptor.Builder()
+ .name("admin-api-access-token")
+ .displayName("Access Token")
+ .description("Access Token to authenticate requests")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+ .build();
+
+ static final PropertyDescriptor CRM_ENDPOINT = new PropertyDescriptor.Builder()
+ .name("crm-endpoint")
+ .displayName("HubSpot CRM API Endpoint")
+ .description("The HubSpot CRM API endpoint to get")
Review Comment:
```suggestion
.description("The HubSpot CRM API endpoint to which the Processor will send requests")
```
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.configuration.DefaultSettings;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"hubspot"})
+@CapabilityDescription("Retrieves JSON data from a private HubSpot application."
+ + " Supports incremental retrieval: Users can set the \"limit\" property which serves as the upper limit of the retrieved objects."
+ + " When this property is set the processor will retrieve new records. This processor is intended to be run on the Primary Node only.")
+@Stateful(scopes = Scope.CLUSTER, description = "When the 'Limit' attribute is set, the paging cursor is saved after executing a request."
+ + " Only the objects after the paging cursor will be retrieved. The maximum number of retrieved objects is the 'Limit' attribute."
+ + " State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected,"
+ + " the new node can pick up where the previous node left off, without duplicating the data.")
+@DefaultSettings(yieldDuration = "10 sec")
+public class GetHubSpot extends AbstractProcessor {
+
+ static final PropertyDescriptor ACCESS_TOKEN = new PropertyDescriptor.Builder()
+ .name("admin-api-access-token")
+ .displayName("Access Token")
+ .description("Access Token to authenticate requests")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+ .build();
+
+ static final PropertyDescriptor CRM_ENDPOINT = new PropertyDescriptor.Builder()
+ .name("crm-endpoint")
+ .displayName("HubSpot CRM API Endpoint")
+ .description("The HubSpot CRM API endpoint to get")
+ .required(true)
+ .allowableValues(CrmEndpoint.class)
+ .build();
+
+ static final PropertyDescriptor LIMIT = new PropertyDescriptor.Builder()
+ .name("crm-limit")
+ .displayName("Limit")
+ .description("The maximum number of results to display per page")
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor WEB_CLIENT_SERVICE_PROVIDER = new PropertyDescriptor.Builder()
+ .name("web-client-service-provider")
+ .displayName("Web Client Service Provider")
+ .description("Controller service for HTTP client operations")
+ .identifiesControllerService(WebClientServiceProvider.class)
+ .required(true)
+ .build();
+
+ static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("For FlowFiles created as a result of a successful HTTP request.")
+ .build();
+
+ static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("In case of HTTP client errors the flowfile will be routed to this relationship")
+ .build();
+
+ private static final String API_BASE_URI = "api.hubapi.com";
+ private static final String HTTPS = "https";
+ private static final String CURSOR_PARAMETER = "after";
+ private static final String LIMIT_PARAMETER = "limit";
+ private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
+ private static final JsonFactory JSON_FACTORY = OBJECT_MAPPER.getFactory();
+
+ private volatile WebClientServiceProvider webClientServiceProvider;
+
+ private static final List<PropertyDescriptor> PROPERTY_DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ ACCESS_TOKEN,
+ CRM_ENDPOINT,
+ LIMIT,
+ WEB_CLIENT_SERVICE_PROVIDER
+ ));
+
+ private static final Set<Relationship> RELATIONSHIPS = Collections.unmodifiableSet(new HashSet<>(Arrays.asList(
+ REL_SUCCESS,
+ REL_FAILURE
+ )));
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ webClientServiceProvider = context.getProperty(WEB_CLIENT_SERVICE_PROVIDER).asControllerService(WebClientServiceProvider.class);
+ }
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTY_DESCRIPTORS;
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return RELATIONSHIPS;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ final String accessToken = context.getProperty(ACCESS_TOKEN).getValue();
+ final String endpoint = context.getProperty(CRM_ENDPOINT).getValue();
+
+ final StateMap state = getStateMap(context);
+ final URI uri = createUri(context, state);
+
+ final HttpResponseEntity response = getHttpResponseEntity(accessToken, uri);
+ final AtomicInteger objectCountHolder = new AtomicInteger();
+
+ FlowFile flowFile = session.create();
+ flowFile = session.putAttribute(flowFile, "statusCode", String.valueOf(response.statusCode()));
+
+ flowFile = session.write(flowFile, out -> {
+
+ try (JsonParser jsonParser = JSON_FACTORY.createParser(response.body());
+ final JsonGenerator jsonGenerator = JSON_FACTORY.createGenerator(out, JsonEncoding.UTF8)) {
+ while (jsonParser.nextToken() != null) {
+ if (jsonParser.getCurrentToken() == JsonToken.FIELD_NAME && jsonParser.getCurrentName().equals("results")) {
+ jsonParser.nextToken();
+ jsonGenerator.copyCurrentStructure(jsonParser);
+ objectCountHolder.incrementAndGet();
+ }
+ String fieldName = jsonParser.getCurrentName();
+ if (CURSOR_PARAMETER.equals(fieldName)) {
+ jsonParser.nextToken();
+ Map<String, String> newStateMap = new HashMap<>(state.toMap());
+ newStateMap.put(endpoint, jsonParser.getText());
+ updateState(context, newStateMap);
+ break;
+ }
+ }
+ }
+ });
+ if (response.statusCode() >= 400) {
+ if (response.statusCode() == 429) {
+ context.yield();
+ throw new ProcessException("Rate limit exceeded, yielding for 10 seconds before retrying request.");
+ } else {
+ getLogger().warn("HTTP [{}] client error occurred at endpoint [{}]", response.statusCode(), endpoint);
+ session.transfer(flowFile, REL_FAILURE);
+ }
+ } else if (objectCountHolder.get() > 0) {
+ session.transfer(flowFile, REL_SUCCESS);
+ } else {
+ getLogger().debug("Empty response when requested HubSpot endpoint: [{}]", endpoint);
+ }
+ }
+
+ HttpResponseEntity getHttpResponseEntity(final String accessToken, final URI uri) {
+ return webClientServiceProvider.getWebClientService()
+ .get()
+ .uri(uri)
+ .header("Authorization", "Bearer " + accessToken)
+ .retrieve();
+ }
+
+ HttpUriBuilder getBaseUri(final ProcessContext context) {
+ final String path = context.getProperty(CRM_ENDPOINT).getValue();
+ return webClientServiceProvider.getHttpUriBuilder()
+ .scheme(HTTPS)
+ .host(API_BASE_URI)
+ .encodedPath(path);
+ }
+
+ URI createUri(final ProcessContext context, final StateMap state) {
Review Comment:
It looks like these methods should be marked `private`.
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.configuration.DefaultSettings;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"hubspot"})
+@CapabilityDescription("Retrieves JSON data from a private HubSpot application."
+ + " Supports incremental retrieval: Users can set the \"limit\" property which serves as the upper limit of the retrieved objects."
+ + " When this property is set the processor will retrieve new records. This processor is intended to be run on the Primary Node only.")
+@Stateful(scopes = Scope.CLUSTER, description = "When the 'Limit' attribute is set, the paging cursor is saved after executing a request."
+ + " Only the objects after the paging cursor will be retrieved. The maximum number of retrieved objects is the 'Limit' attribute."
+ + " State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected,"
+ + " the new node can pick up where the previous node left off, without duplicating the data.")
+@DefaultSettings(yieldDuration = "10 sec")
+public class GetHubSpot extends AbstractProcessor {
+
+ static final PropertyDescriptor ACCESS_TOKEN = new PropertyDescriptor.Builder()
+ .name("admin-api-access-token")
+ .displayName("Access Token")
+ .description("Access Token to authenticate requests")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+ .build();
+
+ static final PropertyDescriptor CRM_ENDPOINT = new PropertyDescriptor.Builder()
+ .name("crm-endpoint")
+ .displayName("HubSpot CRM API Endpoint")
+ .description("The HubSpot CRM API endpoint to get")
+ .required(true)
+ .allowableValues(CrmEndpoint.class)
+ .build();
+
+ static final PropertyDescriptor LIMIT = new PropertyDescriptor.Builder()
+ .name("crm-limit")
+ .displayName("Limit")
+ .description("The maximum number of results to display per page")
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor WEB_CLIENT_SERVICE_PROVIDER = new PropertyDescriptor.Builder()
+ .name("web-client-service-provider")
+ .displayName("Web Client Service Provider")
+ .description("Controller service for HTTP client operations")
+ .identifiesControllerService(WebClientServiceProvider.class)
+ .required(true)
+ .build();
+
+ static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("For FlowFiles created as a result of a successful HTTP request.")
+ .build();
+
+ static final Relationship REL_FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("In case of HTTP client errors the flowfile will be routed to this relationship")
+ .build();
+
+ private static final String API_BASE_URI = "api.hubapi.com";
+ private static final String HTTPS = "https";
+ private static final String CURSOR_PARAMETER = "after";
+ private static final String LIMIT_PARAMETER = "limit";
+ private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
+ private static final JsonFactory JSON_FACTORY = OBJECT_MAPPER.getFactory();
+
+ private volatile WebClientServiceProvider webClientServiceProvider;
+
+ private static final List<PropertyDescriptor> PROPERTY_DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ ACCESS_TOKEN,
+ CRM_ENDPOINT,
+ LIMIT,
+ WEB_CLIENT_SERVICE_PROVIDER
+ ));
+
+ private static final Set<Relationship> RELATIONSHIPS = Collections.unmodifiableSet(new HashSet<>(Arrays.asList(
+ REL_SUCCESS,
+ REL_FAILURE
+ )));
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ webClientServiceProvider = context.getProperty(WEB_CLIENT_SERVICE_PROVIDER).asControllerService(WebClientServiceProvider.class);
+ }
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTY_DESCRIPTORS;
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return RELATIONSHIPS;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ final String accessToken = context.getProperty(ACCESS_TOKEN).getValue();
+ final String endpoint = context.getProperty(CRM_ENDPOINT).getValue();
+
+ final StateMap state = getStateMap(context);
+ final URI uri = createUri(context, state);
+
+ final HttpResponseEntity response = getHttpResponseEntity(accessToken, uri);
+ final AtomicInteger objectCountHolder = new AtomicInteger();
+
+ FlowFile flowFile = session.create();
+ flowFile = session.putAttribute(flowFile, "statusCode", String.valueOf(response.statusCode()));
+
+ flowFile = session.write(flowFile, out -> {
+
+ try (JsonParser jsonParser = JSON_FACTORY.createParser(response.body());
+ final JsonGenerator jsonGenerator = JSON_FACTORY.createGenerator(out, JsonEncoding.UTF8)) {
+ while (jsonParser.nextToken() != null) {
+ if (jsonParser.getCurrentToken() == JsonToken.FIELD_NAME && jsonParser.getCurrentName().equals("results")) {
+ jsonParser.nextToken();
+ jsonGenerator.copyCurrentStructure(jsonParser);
+ objectCountHolder.incrementAndGet();
+ }
+ String fieldName = jsonParser.getCurrentName();
+ if (CURSOR_PARAMETER.equals(fieldName)) {
+ jsonParser.nextToken();
+ Map<String, String> newStateMap = new HashMap<>(state.toMap());
+ newStateMap.put(endpoint, jsonParser.getText());
+ updateState(context, newStateMap);
+ break;
+ }
+ }
+ }
+ });
+ if (response.statusCode() >= 400) {
+ if (response.statusCode() == 429) {
+ context.yield();
+ throw new ProcessException("Rate limit exceeded, yielding for 10 seconds before retrying request.");
Review Comment:
Although `10 seconds` is the default configuration, this can be changed. Instead of throwing an exception, it should be sufficient to log a warning. Also recommend including the status code and URI in a warning.
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.configuration.DefaultSettings;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"hubspot"})
+@CapabilityDescription("Retrieves JSON data from a private HubSpot application."
+ + " Supports incremental retrieval: Users can set the \"limit\" property which serves as the upper limit of the retrieved objects."
+ + " When this property is set the processor will retrieve new records. This processor is intended to be run on the Primary Node only.")
+@Stateful(scopes = Scope.CLUSTER, description = "When the 'Limit' attribute is set, the paging cursor is saved after executing a request."
+ + " Only the objects after the paging cursor will be retrieved. The maximum number of retrieved objects is the 'Limit' attribute."
+ + " State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected,"
+ + " the new node can pick up where the previous node left off, without duplicating the data.")
+@DefaultSettings(yieldDuration = "10 sec")
+public class GetHubSpot extends AbstractProcessor {
+
+ static final PropertyDescriptor ACCESS_TOKEN = new PropertyDescriptor.Builder()
+ .name("admin-api-access-token")
+ .displayName("Access Token")
+ .description("Access Token to authenticate requests")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+ .build();
+
+ static final PropertyDescriptor CRM_ENDPOINT = new PropertyDescriptor.Builder()
+ .name("crm-endpoint")
+ .displayName("HubSpot CRM API Endpoint")
+ .description("The HubSpot CRM API endpoint to get")
+ .required(true)
+ .allowableValues(CrmEndpoint.class)
+ .build();
+
+ static final PropertyDescriptor LIMIT = new PropertyDescriptor.Builder()
+ .name("crm-limit")
+ .displayName("Limit")
+ .description("The maximum number of results to display per page")
Review Comment:
The `display per page` wording sounds more like a user interface, so recommend something along the following lines:
```suggestion
.description("The maximum number of results to request for each invocation of the Processor")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] Lehel44 commented on a diff in pull request #6301: NIFI-10356: Create GetHubSpot processor
Posted by GitBox <gi...@apache.org>.
Lehel44 commented on code in PR #6301:
URL: https://github.com/apache/nifi/pull/6301#discussion_r947285954
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,320 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"hubspot"})
+@CapabilityDescription("Retrieves JSON data from a private HubSpot application."
+ + " Supports incremental retrieval: Users can set the \"limit\" property which serves as the upper limit of the retrieved objects."
+ + " When this property is set the processor will retrieve new records. This processor is intended to be run on the Primary Node only.")
+@Stateful(scopes = Scope.CLUSTER, description = "When the 'Limit' attribute is set, the paging cursor is saved after executing a request."
+ + " Only the objects after the paging cursor will be retrieved. The maximum number of retrieved objects is the 'Limit' attribute."
+ + " State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected,"
+ + " the new node can pick up where the previous node left off, without duplicating the data.")
+public class GetHubSpot extends AbstractProcessor {
+
+ // OBJECTS
+
+ static final AllowableValue COMPANIES = new AllowableValue(
+ "/crm/v3/objects/companies",
+ "Companies",
+ "In HubSpot, the companies object is a standard CRM object. Individual company records can be used to store information about businesses" +
+ " and organizations within company properties."
+ );
+ static final AllowableValue CONTACTS = new AllowableValue(
+ "/crm/v3/objects/contacts",
+ "Contacts",
+ "In HubSpot, contacts store information about individuals. From marketing automation to smart content, the lead-specific data found in" +
+ " contact records helps users leverage much of HubSpot's functionality."
+ );
+ static final AllowableValue DEALS = new AllowableValue(
+ "/crm/v3/objects/deals",
+ "In HubSpot, a deal represents an ongoing transaction that a sales team is pursuing with a contact or company. It’s tracked through" +
+ " pipeline stages until won or lost."
+ );
+ static final AllowableValue FEEDBACK_SUBMISSIONS = new AllowableValue(
+ "/crm/v3/objects/feedback_submissions",
+ "In HubSpot, feedback submissions are an object which stores information submitted to a feedback survey. This includes Net Promoter Score (NPS)," +
+ " Customer Satisfaction (CSAT), Customer Effort Score (CES) and Custom Surveys."
+ );
+ static final AllowableValue LINE_ITEMS = new AllowableValue(
+ "/crm/v3/objects/line_items",
+ "Line Items",
+ "In HubSpot, line items can be thought of as a subset of products. When a product is attached to a deal, it becomes a line item. Line items can" +
+ " be created that are unique to an individual quote, but they will not be added to the product library."
+ );
+ static final AllowableValue PRODUCTS = new AllowableValue(
+ "/crm/v3/objects/products",
+ "Products",
+ "In HubSpot, products represent the goods or services to be sold. Building a product library allows the user to quickly add products to deals," +
+ " generate quotes, and report on product performance."
+ );
+ static final AllowableValue TICKETS = new AllowableValue(
+ "/crm/v3/objects/tickets",
+ "Tickets",
+ "In HubSpot, a ticket represents a customer request for help or support."
+ );
+ static final AllowableValue QUOTES = new AllowableValue(
+ "/crm/v3/objects/quotes",
+ "Quotes",
+ "In HubSpot, quotes are used to share pricing information with potential buyers."
+ );
+
+ // ENGAGEMENTS
+
+ private static final AllowableValue CALLS = new AllowableValue(
+ "/crm/v3/objects/calls",
+ "Calls",
+ "Get calls on CRM records and on the calls index page."
+ );
+ private static final AllowableValue EMAILS = new AllowableValue(
+ "/crm/v3/objects/emails",
+ "Emails",
+ "Get emails on CRM records."
+ );
+ private static final AllowableValue MEETINGS = new AllowableValue(
+ "/crm/v3/objects/meetings",
+ "Meetings",
+ "Get meetings on CRM records."
+ );
+ private static final AllowableValue NOTES = new AllowableValue(
+ "/crm/v3/objects/notes",
+ "Notes",
+ "Get notes on CRM records."
+ );
+ private static final AllowableValue TASKS = new AllowableValue(
+ "/crm/v3/objects/tasks",
+ "Tasks",
+ "Get tasks on CRM records."
+ );
+
+ // OTHER
+
+ private static final AllowableValue OWNERS = new AllowableValue(
+ "/crm/v3/owners/",
+ "Owners",
+ "HubSpot uses owners to assign specific users to contacts, companies, deals, tickets, or engagements. Any HubSpot user with access to contacts" +
+ " can be assigned as an owner, and multiple owners can be assigned to an object by creating a custom property for this purpose."
+ );
+
+ static final PropertyDescriptor ACCESS_TOKEN = new PropertyDescriptor.Builder()
Review Comment:
Thanks, it should be simply Access Token, I renamed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] Lehel44 commented on a diff in pull request #6301: NIFI-10356: Create GetHubSpot processor
Posted by GitBox <gi...@apache.org>.
Lehel44 commented on code in PR #6301:
URL: https://github.com/apache/nifi/pull/6301#discussion_r952475997
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.configuration.DefaultSettings;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+
+@PrimaryNodeOnly
+@TriggerSerially
Review Comment:
I'm not sure, I followed the stateful processor annotation pattern. "State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected" sand usually @PrimaryNodeOnly processors run on 1 thread. What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #6301: NIFI-10356: Create GetHubSpot processor
Posted by GitBox <gi...@apache.org>.
exceptionfactory commented on code in PR #6301:
URL: https://github.com/apache/nifi/pull/6301#discussion_r956421187
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.configuration.DefaultSettings;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpResponseStatus;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"hubspot"})
+@CapabilityDescription("Retrieves JSON data from a private HubSpot application."
+ + " Supports incremental retrieval: Users can set the \"limit\" property which serves as the upper limit of the retrieved objects."
+ + " When this property is set the processor will retrieve new records. This processor is intended to be run on the Primary Node only.")
+@Stateful(scopes = Scope.CLUSTER, description = "When the 'Limit' attribute is set, the paging cursor is saved after executing a request."
+ + " Only the objects after the paging cursor will be retrieved. The maximum number of retrieved objects is the 'Limit' attribute."
+ + " State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected,"
+ + " the new node can pick up where the previous node left off, without duplicating the data.")
+@DefaultSettings(yieldDuration = "10 sec")
+public class GetHubSpot extends AbstractProcessor {
+
+ static final PropertyDescriptor CRM_ENDPOINT = new PropertyDescriptor.Builder()
+ .name("crm-endpoint")
+ .displayName("HubSpot CRM API Endpoint")
+ .description("The HubSpot CRM API endpoint to which the Processor will send requests")
+ .required(true)
+ .allowableValues(CrmEndpoint.class)
+ .build();
+
+ static final PropertyDescriptor ACCESS_TOKEN = new PropertyDescriptor.Builder()
+ .name("access-token")
+ .displayName("Access Token")
+ .description("Access Token to authenticate requests")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+ .build();
+
+ static final PropertyDescriptor LIMIT = new PropertyDescriptor.Builder()
+ .name("crm-limit")
+ .displayName("Limit")
+ .description("The maximum number of results to request for each invocation of the Processor")
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .required(true)
Review Comment:
The check for `isLimitSet` in the `createUri` method implies this property is optional, so it looks like this should be changed.
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.configuration.DefaultSettings;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpResponseStatus;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"hubspot"})
+@CapabilityDescription("Retrieves JSON data from a private HubSpot application."
+ + " Supports incremental retrieval: Users can set the \"limit\" property which serves as the upper limit of the retrieved objects."
+ + " When this property is set the processor will retrieve new records. This processor is intended to be run on the Primary Node only.")
+@Stateful(scopes = Scope.CLUSTER, description = "When the 'Limit' attribute is set, the paging cursor is saved after executing a request."
+ + " Only the objects after the paging cursor will be retrieved. The maximum number of retrieved objects is the 'Limit' attribute."
+ + " State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected,"
+ + " the new node can pick up where the previous node left off, without duplicating the data.")
+@DefaultSettings(yieldDuration = "10 sec")
+public class GetHubSpot extends AbstractProcessor {
+
+ static final PropertyDescriptor CRM_ENDPOINT = new PropertyDescriptor.Builder()
+ .name("crm-endpoint")
+ .displayName("HubSpot CRM API Endpoint")
+ .description("The HubSpot CRM API endpoint to which the Processor will send requests")
Review Comment:
Recommend renaming this property to `CRM Object Type` to align with the display name presented when selecting the available property. Although the internal value is a path, the display name is the object type.
```suggestion
static final PropertyDescriptor CRM_OBJECT_TYPE = new PropertyDescriptor.Builder()
.name("crm-object-type")
.displayName("CRM Object Type")
.description("HubSpot CRM Object Type requested")
```
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.configuration.DefaultSettings;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpResponseStatus;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"hubspot"})
+@CapabilityDescription("Retrieves JSON data from a private HubSpot application."
+ + " Supports incremental retrieval: Users can set the \"limit\" property which serves as the upper limit of the retrieved objects."
+ + " When this property is set the processor will retrieve new records. This processor is intended to be run on the Primary Node only.")
+@Stateful(scopes = Scope.CLUSTER, description = "When the 'Limit' attribute is set, the paging cursor is saved after executing a request."
+ + " Only the objects after the paging cursor will be retrieved. The maximum number of retrieved objects is the 'Limit' attribute."
+ + " State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected,"
+ + " the new node can pick up where the previous node left off, without duplicating the data.")
+@DefaultSettings(yieldDuration = "10 sec")
+public class GetHubSpot extends AbstractProcessor {
+
+ static final PropertyDescriptor CRM_ENDPOINT = new PropertyDescriptor.Builder()
+ .name("crm-endpoint")
+ .displayName("HubSpot CRM API Endpoint")
+ .description("The HubSpot CRM API endpoint to which the Processor will send requests")
+ .required(true)
+ .allowableValues(CrmEndpoint.class)
+ .build();
+
+ static final PropertyDescriptor ACCESS_TOKEN = new PropertyDescriptor.Builder()
+ .name("access-token")
+ .displayName("Access Token")
+ .description("Access Token to authenticate requests")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+ .build();
+
+ static final PropertyDescriptor LIMIT = new PropertyDescriptor.Builder()
+ .name("crm-limit")
+ .displayName("Limit")
+ .description("The maximum number of results to request for each invocation of the Processor")
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor WEB_CLIENT_SERVICE_PROVIDER = new PropertyDescriptor.Builder()
+ .name("web-client-service-provider")
+ .displayName("Web Client Service Provider")
+ .description("Controller service for HTTP client operations")
+ .identifiesControllerService(WebClientServiceProvider.class)
+ .required(true)
+ .build();
+
+ static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("For FlowFiles created as a result of a successful HTTP request.")
+ .build();
+
+ private static final String API_BASE_URI = "api.hubapi.com";
+ private static final String HTTPS = "https";
+ private static final String CURSOR_PARAMETER = "after";
+ private static final String LIMIT_PARAMETER = "limit";
+ private static final int TOO_MANY_REQUESTS = 429;
+ private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
+ private static final JsonFactory JSON_FACTORY = OBJECT_MAPPER.getFactory();
+
+ private volatile WebClientServiceProvider webClientServiceProvider;
+
+ private static final List<PropertyDescriptor> PROPERTY_DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ CRM_ENDPOINT,
+ ACCESS_TOKEN,
+ LIMIT,
+ WEB_CLIENT_SERVICE_PROVIDER
+ ));
+
+ private static final Set<Relationship> RELATIONSHIPS = Collections.singleton(REL_SUCCESS);
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ webClientServiceProvider = context.getProperty(WEB_CLIENT_SERVICE_PROVIDER).asControllerService(WebClientServiceProvider.class);
+ }
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTY_DESCRIPTORS;
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return RELATIONSHIPS;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ final String accessToken = context.getProperty(ACCESS_TOKEN).getValue();
+ final String endpoint = context.getProperty(CRM_ENDPOINT).getValue();
+
+ final StateMap state = getStateMap(context);
+ final URI uri = createUri(context, state);
+
+ final HttpResponseEntity response = getHttpResponseEntity(accessToken, uri);
+ final AtomicInteger objectCountHolder = new AtomicInteger();
+
+ if (response.statusCode() == HttpResponseStatus.OK.getCode()) {
+ FlowFile flowFile = session.create();
+ flowFile = session.write(flowFile, parseHttpResponse(context, endpoint, state, response, objectCountHolder));
+ if (objectCountHolder.get() > 0) {
+ session.transfer(flowFile, REL_SUCCESS);
+ } else {
+ getLogger().debug("Empty response when requested HubSpot endpoint: [{}]", endpoint);
+ session.remove(flowFile);
+ }
+ } else if (response.statusCode() >= 400) {
Review Comment:
This should be changed to a more generalized `else` condition.
```suggestion
} else {
```
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.configuration.DefaultSettings;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpResponseStatus;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"hubspot"})
+@CapabilityDescription("Retrieves JSON data from a private HubSpot application."
+ + " Supports incremental retrieval: Users can set the \"limit\" property which serves as the upper limit of the retrieved objects."
+ + " When this property is set the processor will retrieve new records. This processor is intended to be run on the Primary Node only.")
+@Stateful(scopes = Scope.CLUSTER, description = "When the 'Limit' attribute is set, the paging cursor is saved after executing a request."
+ + " Only the objects after the paging cursor will be retrieved. The maximum number of retrieved objects is the 'Limit' attribute."
+ + " State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected,"
+ + " the new node can pick up where the previous node left off, without duplicating the data.")
+@DefaultSettings(yieldDuration = "10 sec")
+public class GetHubSpot extends AbstractProcessor {
Review Comment:
The current version of this processor is limited to HubSpot [CRM](https://developers.hubspot.com/docs/api/crm/understanding-the-crm) operations, but there are other parts of the HubSpot API. In light of that fact, it seems like the processor should be renamed to `GetHubSpotCRM`, or the Object Type property should be renamed to something more general. What do you think?
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.configuration.DefaultSettings;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpResponseStatus;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"hubspot"})
+@CapabilityDescription("Retrieves JSON data from a private HubSpot application."
+ + " Supports incremental retrieval: Users can set the \"limit\" property which serves as the upper limit of the retrieved objects."
+ + " When this property is set the processor will retrieve new records. This processor is intended to be run on the Primary Node only.")
+@Stateful(scopes = Scope.CLUSTER, description = "When the 'Limit' attribute is set, the paging cursor is saved after executing a request."
+ + " Only the objects after the paging cursor will be retrieved. The maximum number of retrieved objects is the 'Limit' attribute."
+ + " State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected,"
+ + " the new node can pick up where the previous node left off, without duplicating the data.")
+@DefaultSettings(yieldDuration = "10 sec")
+public class GetHubSpot extends AbstractProcessor {
+
+ static final PropertyDescriptor CRM_ENDPOINT = new PropertyDescriptor.Builder()
+ .name("crm-endpoint")
+ .displayName("HubSpot CRM API Endpoint")
+ .description("The HubSpot CRM API endpoint to which the Processor will send requests")
+ .required(true)
+ .allowableValues(CrmEndpoint.class)
+ .build();
+
+ static final PropertyDescriptor ACCESS_TOKEN = new PropertyDescriptor.Builder()
+ .name("access-token")
+ .displayName("Access Token")
+ .description("Access Token to authenticate requests")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+ .build();
+
+ static final PropertyDescriptor LIMIT = new PropertyDescriptor.Builder()
+ .name("crm-limit")
+ .displayName("Limit")
Review Comment:
Recommend renaming this property to `Result Limit` for better clarity of function.
```suggestion
static final PropertyDescriptor RESULT_LIMIT = new PropertyDescriptor.Builder()
.name("result-limit")
.displayName("Result Limit")
```
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.configuration.DefaultSettings;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpResponseStatus;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"hubspot"})
+@CapabilityDescription("Retrieves JSON data from a private HubSpot application."
+ + " Supports incremental retrieval: Users can set the \"limit\" property which serves as the upper limit of the retrieved objects."
Review Comment:
Recommend adjusting the wording to avoid the quotes and use capitalization for the Result Limit property instead:
```suggestion
+ " Configuring the Result Limit property enables incremental retrieval of results."
```
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.configuration.DefaultSettings;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpResponseStatus;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"hubspot"})
+@CapabilityDescription("Retrieves JSON data from a private HubSpot application."
+ + " Supports incremental retrieval: Users can set the \"limit\" property which serves as the upper limit of the retrieved objects."
+ + " When this property is set the processor will retrieve new records. This processor is intended to be run on the Primary Node only.")
+@Stateful(scopes = Scope.CLUSTER, description = "When the 'Limit' attribute is set, the paging cursor is saved after executing a request."
+ + " Only the objects after the paging cursor will be retrieved. The maximum number of retrieved objects is the 'Limit' attribute."
+ + " State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected,"
+ + " the new node can pick up where the previous node left off, without duplicating the data.")
+@DefaultSettings(yieldDuration = "10 sec")
+public class GetHubSpot extends AbstractProcessor {
+
+ static final PropertyDescriptor CRM_ENDPOINT = new PropertyDescriptor.Builder()
+ .name("crm-endpoint")
+ .displayName("HubSpot CRM API Endpoint")
+ .description("The HubSpot CRM API endpoint to which the Processor will send requests")
+ .required(true)
+ .allowableValues(CrmEndpoint.class)
+ .build();
+
+ static final PropertyDescriptor ACCESS_TOKEN = new PropertyDescriptor.Builder()
+ .name("access-token")
+ .displayName("Access Token")
+ .description("Access Token to authenticate requests")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.NONE)
+ .build();
+
+ static final PropertyDescriptor LIMIT = new PropertyDescriptor.Builder()
+ .name("crm-limit")
+ .displayName("Limit")
+ .description("The maximum number of results to request for each invocation of the Processor")
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor WEB_CLIENT_SERVICE_PROVIDER = new PropertyDescriptor.Builder()
+ .name("web-client-service-provider")
+ .displayName("Web Client Service Provider")
+ .description("Controller service for HTTP client operations")
+ .identifiesControllerService(WebClientServiceProvider.class)
+ .required(true)
+ .build();
+
+ static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("For FlowFiles created as a result of a successful HTTP request.")
+ .build();
+
+ private static final String API_BASE_URI = "api.hubapi.com";
+ private static final String HTTPS = "https";
+ private static final String CURSOR_PARAMETER = "after";
+ private static final String LIMIT_PARAMETER = "limit";
+ private static final int TOO_MANY_REQUESTS = 429;
+ private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
+ private static final JsonFactory JSON_FACTORY = OBJECT_MAPPER.getFactory();
+
+ private volatile WebClientServiceProvider webClientServiceProvider;
+
+ private static final List<PropertyDescriptor> PROPERTY_DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ CRM_ENDPOINT,
+ ACCESS_TOKEN,
+ LIMIT,
+ WEB_CLIENT_SERVICE_PROVIDER
+ ));
+
+ private static final Set<Relationship> RELATIONSHIPS = Collections.singleton(REL_SUCCESS);
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ webClientServiceProvider = context.getProperty(WEB_CLIENT_SERVICE_PROVIDER).asControllerService(WebClientServiceProvider.class);
+ }
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTY_DESCRIPTORS;
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return RELATIONSHIPS;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ final String accessToken = context.getProperty(ACCESS_TOKEN).getValue();
+ final String endpoint = context.getProperty(CRM_ENDPOINT).getValue();
+
+ final StateMap state = getStateMap(context);
+ final URI uri = createUri(context, state);
+
+ final HttpResponseEntity response = getHttpResponseEntity(accessToken, uri);
+ final AtomicInteger objectCountHolder = new AtomicInteger();
+
+ if (response.statusCode() == HttpResponseStatus.OK.getCode()) {
+ FlowFile flowFile = session.create();
+ flowFile = session.write(flowFile, parseHttpResponse(context, endpoint, state, response, objectCountHolder));
+ if (objectCountHolder.get() > 0) {
+ session.transfer(flowFile, REL_SUCCESS);
+ } else {
+ getLogger().debug("Empty response when requested HubSpot endpoint: [{}]", endpoint);
+ session.remove(flowFile);
+ }
+ } else if (response.statusCode() >= 400) {
+ if (response.statusCode() == TOO_MANY_REQUESTS) {
+ context.yield();
+ throw new ProcessException(String.format("Rate limit exceeded, yielding before retrying request. HTTP %d error for requested URI [%s]", response.statusCode(), uri));
+ } else {
+ getLogger().warn("HTTP {} error for requested URI [{}]", response.statusCode(), uri);
Review Comment:
It would be helpful to read the HTTP response body to a string and include it in the warning. It contains useful troubleshooting details. This would also handle consuming and closing the response body InputStream in failure scenarios.
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.configuration.DefaultSettings;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpResponseStatus;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"hubspot"})
+@CapabilityDescription("Retrieves JSON data from a private HubSpot application."
+ + " Supports incremental retrieval: Users can set the \"limit\" property which serves as the upper limit of the retrieved objects."
+ + " When this property is set the processor will retrieve new records. This processor is intended to be run on the Primary Node only.")
+@Stateful(scopes = Scope.CLUSTER, description = "When the 'Limit' attribute is set, the paging cursor is saved after executing a request."
+ + " Only the objects after the paging cursor will be retrieved. The maximum number of retrieved objects is the 'Limit' attribute."
+ + " State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected,"
+ + " the new node can pick up where the previous node left off, without duplicating the data.")
Review Comment:
These two lines can be removed since they are implied as part of cluster state storage for all components.
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/resources/docs/org.apache.nifi.processors.hubspot.GetHubSpot/additionalDetails.html:
##########
@@ -0,0 +1,143 @@
+<!DOCTYPE html>
+<html lang="en">
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<head>
+ <meta charset="utf-8"/>
+ <title>GetHubSpot</title>
+ <link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"/>
+</head>
+
+<body>
+<h2>Incremental Loading</h2>
Review Comment:
Recommend adding a section at the beginning with a link to the HubSpot [Authentication Methods](https://developers.hubspot.com/docs/api/intro-to-auth) documentation, which describes how to obtain an Access Token.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #6301: NIFI-10356: Create GetHubSpot processor
Posted by GitBox <gi...@apache.org>.
exceptionfactory commented on code in PR #6301:
URL: https://github.com/apache/nifi/pull/6301#discussion_r946137053
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/pom.xml:
##########
@@ -0,0 +1,104 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xmlns="http://maven.apache.org/POM/4.0.0"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>nifi-hubspot-bundle</artifactId>
+ <groupId>org.apache.nifi</groupId>
+ <version>1.18.0-SNAPSHOT</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>nifi-hubspot-processors</artifactId>
+
+ <dependencies>
+ <dependency>
+ <groupId>commons-io</groupId>
+ <artifactId>commons-io</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-web-client-api</artifactId>
+ <version>1.18.0-SNAPSHOT</version>
+ <scope>compile</scope>
Review Comment:
This dependency should be marked as `provided` since it comes from the `nifi-standard-services-api-nar`.
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/pom.xml:
##########
@@ -0,0 +1,104 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xmlns="http://maven.apache.org/POM/4.0.0"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>nifi-hubspot-bundle</artifactId>
+ <groupId>org.apache.nifi</groupId>
+ <version>1.18.0-SNAPSHOT</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>nifi-hubspot-processors</artifactId>
+
+ <dependencies>
+ <dependency>
+ <groupId>commons-io</groupId>
+ <artifactId>commons-io</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-web-client-api</artifactId>
+ <version>1.18.0-SNAPSHOT</version>
+ <scope>compile</scope>
+ </dependency>
+ <dependency>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>jackson-databind</artifactId>
+ </dependency>
+ <!-- Test dependencies -->
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-mock</artifactId>
+ <version>1.18.0-SNAPSHOT</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-web-client-provider-api</artifactId>
+ <version>1.18.0-SNAPSHOT</version>
+ <scope>compile</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-utils</artifactId>
+ <version>1.18.0-SNAPSHOT</version>
+ <scope>compile</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-web-client-provider-service</artifactId>
+ <version>1.18.0-SNAPSHOT</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-ssl-context-service-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-ssl-context-service</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-proxy-configuration-api</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>com.squareup.okhttp3</groupId>
+ <artifactId>mockwebserver</artifactId>
+ <version>4.10.0</version>
Review Comment:
This version number can be removed as it is managed in the root Maven configuration.
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,320 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"hubspot"})
+@CapabilityDescription("Retrieves JSON data from a private HubSpot application."
+ + " Supports incremental retrieval: Users can set the \"limit\" property which serves as the upper limit of the retrieved objects."
+ + " When this property is set the processor will retrieve new records. This processor is intended to be run on the Primary Node only.")
+@Stateful(scopes = Scope.CLUSTER, description = "When the 'Limit' attribute is set, the paging cursor is saved after executing a request."
+ + " Only the objects after the paging cursor will be retrieved. The maximum number of retrieved objects is the 'Limit' attribute."
+ + " State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected,"
+ + " the new node can pick up where the previous node left off, without duplicating the data.")
+public class GetHubSpot extends AbstractProcessor {
+
+ // OBJECTS
+
+ static final AllowableValue COMPANIES = new AllowableValue(
+ "/crm/v3/objects/companies",
+ "Companies",
+ "In HubSpot, the companies object is a standard CRM object. Individual company records can be used to store information about businesses" +
+ " and organizations within company properties."
+ );
+ static final AllowableValue CONTACTS = new AllowableValue(
+ "/crm/v3/objects/contacts",
+ "Contacts",
+ "In HubSpot, contacts store information about individuals. From marketing automation to smart content, the lead-specific data found in" +
+ " contact records helps users leverage much of HubSpot's functionality."
+ );
+ static final AllowableValue DEALS = new AllowableValue(
+ "/crm/v3/objects/deals",
+ "In HubSpot, a deal represents an ongoing transaction that a sales team is pursuing with a contact or company. It’s tracked through" +
+ " pipeline stages until won or lost."
+ );
+ static final AllowableValue FEEDBACK_SUBMISSIONS = new AllowableValue(
+ "/crm/v3/objects/feedback_submissions",
+ "In HubSpot, feedback submissions are an object which stores information submitted to a feedback survey. This includes Net Promoter Score (NPS)," +
+ " Customer Satisfaction (CSAT), Customer Effort Score (CES) and Custom Surveys."
+ );
+ static final AllowableValue LINE_ITEMS = new AllowableValue(
+ "/crm/v3/objects/line_items",
+ "Line Items",
+ "In HubSpot, line items can be thought of as a subset of products. When a product is attached to a deal, it becomes a line item. Line items can" +
+ " be created that are unique to an individual quote, but they will not be added to the product library."
+ );
+ static final AllowableValue PRODUCTS = new AllowableValue(
+ "/crm/v3/objects/products",
+ "Products",
+ "In HubSpot, products represent the goods or services to be sold. Building a product library allows the user to quickly add products to deals," +
+ " generate quotes, and report on product performance."
+ );
+ static final AllowableValue TICKETS = new AllowableValue(
+ "/crm/v3/objects/tickets",
+ "Tickets",
+ "In HubSpot, a ticket represents a customer request for help or support."
+ );
+ static final AllowableValue QUOTES = new AllowableValue(
+ "/crm/v3/objects/quotes",
+ "Quotes",
+ "In HubSpot, quotes are used to share pricing information with potential buyers."
+ );
+
+ // ENGAGEMENTS
+
+ private static final AllowableValue CALLS = new AllowableValue(
+ "/crm/v3/objects/calls",
+ "Calls",
+ "Get calls on CRM records and on the calls index page."
+ );
+ private static final AllowableValue EMAILS = new AllowableValue(
+ "/crm/v3/objects/emails",
+ "Emails",
+ "Get emails on CRM records."
+ );
+ private static final AllowableValue MEETINGS = new AllowableValue(
+ "/crm/v3/objects/meetings",
+ "Meetings",
+ "Get meetings on CRM records."
+ );
+ private static final AllowableValue NOTES = new AllowableValue(
+ "/crm/v3/objects/notes",
+ "Notes",
+ "Get notes on CRM records."
+ );
+ private static final AllowableValue TASKS = new AllowableValue(
+ "/crm/v3/objects/tasks",
+ "Tasks",
+ "Get tasks on CRM records."
+ );
+
+ // OTHER
+
+ private static final AllowableValue OWNERS = new AllowableValue(
+ "/crm/v3/owners/",
+ "Owners",
+ "HubSpot uses owners to assign specific users to contacts, companies, deals, tickets, or engagements. Any HubSpot user with access to contacts" +
+ " can be assigned as an owner, and multiple owners can be assigned to an object by creating a custom property for this purpose."
+ );
+
+ static final PropertyDescriptor ACCESS_TOKEN = new PropertyDescriptor.Builder()
+ .name("hubspot-admin-api-access-token")
+ .displayName("Admin API Access Token")
+ .description("")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
+ .build();
+
+ static final PropertyDescriptor CRM_ENDPOINT = new PropertyDescriptor.Builder()
+ .name("hubspot-crm-endpoint")
+ .displayName("HubSpot CRM API Endpoint")
+ .description("The HubSpot CRM API endpoint to get")
+ .required(true)
+ .allowableValues(COMPANIES, CONTACTS, DEALS, FEEDBACK_SUBMISSIONS, LINE_ITEMS, PRODUCTS, TICKETS, QUOTES,
+ CALLS, EMAILS, MEETINGS, NOTES, TASKS, OWNERS)
+ .build();
+
+ static final PropertyDescriptor LIMIT = new PropertyDescriptor.Builder()
+ .name("hubspot-crm-limit")
+ .displayName("Limit")
+ .description("The maximum number of results to display per page")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor WEB_CLIENT_PROVIDER = new PropertyDescriptor.Builder()
+ .name("nifi-web-client")
+ .displayName("NiFi Web Client")
+ .description("Controller service for HTTP client operations")
+ .identifiesControllerService(WebClientServiceProvider.class)
+ .required(true)
+ .build();
+
+ static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("For FlowFiles created as a result of a successful query.")
+ .build();
+
+ private static final List<PropertyDescriptor> PROPERTY_DESCRIPTORS = createPropertyDescriptors();
+ public static final String API_BASE_URI = "api.hubapi.com";
+ public static final String HTTPS = "https";
+
+ private volatile WebClientServiceProvider webClientServiceProvider;
+
+ private static List<PropertyDescriptor> createPropertyDescriptors() {
+ final List<PropertyDescriptor> propertyDescriptors = new ArrayList<>(Arrays.asList(
+ ACCESS_TOKEN,
+ CRM_ENDPOINT,
+ LIMIT,
+ WEB_CLIENT_PROVIDER
+ ));
+ return Collections.unmodifiableList(propertyDescriptors);
+ }
+
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ webClientServiceProvider = context.getProperty(WEB_CLIENT_PROVIDER).asControllerService(WebClientServiceProvider.class);
+ }
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTY_DESCRIPTORS;
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ final Set<Relationship> relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ return relationships;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ final String accessToken = context.getProperty(ACCESS_TOKEN).getValue();
+ final String endpoint = context.getProperty(CRM_ENDPOINT).getValue();
+
+ final StateMap state = getStateMap(context);
+ final URI uri = createUri(context, state);
+
+ final HttpResponseEntity response = getHttpResponseEntity(accessToken, uri);
+ final ObjectMapper objectMapper = new ObjectMapper();
+ final JsonFactory jsonFactory = objectMapper.getFactory();
+ FlowFile flowFile = session.create();
+
+ flowFile = session.write(flowFile, out -> {
+
+
+ try (JsonParser jsonParser = jsonFactory.createParser(response.body());
+ final JsonGenerator jsonGenerator = jsonFactory.createGenerator(out, JsonEncoding.UTF8)) {
+ while (jsonParser.nextToken() != null) {
+ if (jsonParser.getCurrentToken() == JsonToken.FIELD_NAME && jsonParser.getCurrentName().equals("results")) {
+ jsonParser.nextToken();
+ jsonGenerator.copyCurrentStructure(jsonParser);
+ }
+ String fieldname = jsonParser.getCurrentName();
+ if ("after".equals(fieldname)) {
+ jsonParser.nextToken();
+ Map<String, String> newStateMap = new HashMap<>(state.toMap());
+ newStateMap.put(endpoint, jsonParser.getText());
+ updateState(context, newStateMap);
+ break;
+ }
+ }
+ }
+ });
+ session.transfer(flowFile, REL_SUCCESS);
Review Comment:
It is possible for the query to return zero results? If so, that condition should be checked before transferring.
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/pom.xml:
##########
@@ -0,0 +1,104 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xmlns="http://maven.apache.org/POM/4.0.0"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>nifi-hubspot-bundle</artifactId>
+ <groupId>org.apache.nifi</groupId>
+ <version>1.18.0-SNAPSHOT</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>nifi-hubspot-processors</artifactId>
+
+ <dependencies>
+ <dependency>
+ <groupId>commons-io</groupId>
+ <artifactId>commons-io</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-web-client-api</artifactId>
+ <version>1.18.0-SNAPSHOT</version>
+ <scope>compile</scope>
+ </dependency>
+ <dependency>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>jackson-databind</artifactId>
+ </dependency>
+ <!-- Test dependencies -->
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-mock</artifactId>
+ <version>1.18.0-SNAPSHOT</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-web-client-provider-api</artifactId>
+ <version>1.18.0-SNAPSHOT</version>
+ <scope>compile</scope>
+ </dependency>
Review Comment:
This should be moved up in the order to keep test dependencies together.
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/pom.xml:
##########
@@ -0,0 +1,104 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xmlns="http://maven.apache.org/POM/4.0.0"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>nifi-hubspot-bundle</artifactId>
+ <groupId>org.apache.nifi</groupId>
+ <version>1.18.0-SNAPSHOT</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>nifi-hubspot-processors</artifactId>
+
+ <dependencies>
+ <dependency>
+ <groupId>commons-io</groupId>
+ <artifactId>commons-io</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-web-client-api</artifactId>
+ <version>1.18.0-SNAPSHOT</version>
+ <scope>compile</scope>
+ </dependency>
+ <dependency>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>jackson-databind</artifactId>
+ </dependency>
+ <!-- Test dependencies -->
Review Comment:
This comment should be removed since the following dependencies include non-test libraries.
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/pom.xml:
##########
@@ -0,0 +1,104 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xmlns="http://maven.apache.org/POM/4.0.0"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>nifi-hubspot-bundle</artifactId>
+ <groupId>org.apache.nifi</groupId>
+ <version>1.18.0-SNAPSHOT</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>nifi-hubspot-processors</artifactId>
+
+ <dependencies>
+ <dependency>
+ <groupId>commons-io</groupId>
+ <artifactId>commons-io</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-web-client-api</artifactId>
+ <version>1.18.0-SNAPSHOT</version>
+ <scope>compile</scope>
+ </dependency>
+ <dependency>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>jackson-databind</artifactId>
+ </dependency>
+ <!-- Test dependencies -->
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-mock</artifactId>
+ <version>1.18.0-SNAPSHOT</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-web-client-provider-api</artifactId>
+ <version>1.18.0-SNAPSHOT</version>
+ <scope>compile</scope>
Review Comment:
This dependency should be marked as `provided`.
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,320 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"hubspot"})
+@CapabilityDescription("Retrieves JSON data from a private HubSpot application."
+ + " Supports incremental retrieval: Users can set the \"limit\" property which serves as the upper limit of the retrieved objects."
+ + " When this property is set the processor will retrieve new records. This processor is intended to be run on the Primary Node only.")
+@Stateful(scopes = Scope.CLUSTER, description = "When the 'Limit' attribute is set, the paging cursor is saved after executing a request."
+ + " Only the objects after the paging cursor will be retrieved. The maximum number of retrieved objects is the 'Limit' attribute."
+ + " State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected,"
+ + " the new node can pick up where the previous node left off, without duplicating the data.")
+public class GetHubSpot extends AbstractProcessor {
+
+ // OBJECTS
+
+ static final AllowableValue COMPANIES = new AllowableValue(
+ "/crm/v3/objects/companies",
+ "Companies",
+ "In HubSpot, the companies object is a standard CRM object. Individual company records can be used to store information about businesses" +
+ " and organizations within company properties."
+ );
+ static final AllowableValue CONTACTS = new AllowableValue(
+ "/crm/v3/objects/contacts",
+ "Contacts",
+ "In HubSpot, contacts store information about individuals. From marketing automation to smart content, the lead-specific data found in" +
+ " contact records helps users leverage much of HubSpot's functionality."
+ );
+ static final AllowableValue DEALS = new AllowableValue(
+ "/crm/v3/objects/deals",
+ "In HubSpot, a deal represents an ongoing transaction that a sales team is pursuing with a contact or company. It’s tracked through" +
+ " pipeline stages until won or lost."
+ );
+ static final AllowableValue FEEDBACK_SUBMISSIONS = new AllowableValue(
+ "/crm/v3/objects/feedback_submissions",
+ "In HubSpot, feedback submissions are an object which stores information submitted to a feedback survey. This includes Net Promoter Score (NPS)," +
+ " Customer Satisfaction (CSAT), Customer Effort Score (CES) and Custom Surveys."
+ );
+ static final AllowableValue LINE_ITEMS = new AllowableValue(
+ "/crm/v3/objects/line_items",
+ "Line Items",
+ "In HubSpot, line items can be thought of as a subset of products. When a product is attached to a deal, it becomes a line item. Line items can" +
+ " be created that are unique to an individual quote, but they will not be added to the product library."
+ );
+ static final AllowableValue PRODUCTS = new AllowableValue(
+ "/crm/v3/objects/products",
+ "Products",
+ "In HubSpot, products represent the goods or services to be sold. Building a product library allows the user to quickly add products to deals," +
+ " generate quotes, and report on product performance."
+ );
+ static final AllowableValue TICKETS = new AllowableValue(
+ "/crm/v3/objects/tickets",
+ "Tickets",
+ "In HubSpot, a ticket represents a customer request for help or support."
+ );
+ static final AllowableValue QUOTES = new AllowableValue(
+ "/crm/v3/objects/quotes",
+ "Quotes",
+ "In HubSpot, quotes are used to share pricing information with potential buyers."
+ );
+
+ // ENGAGEMENTS
+
+ private static final AllowableValue CALLS = new AllowableValue(
+ "/crm/v3/objects/calls",
+ "Calls",
+ "Get calls on CRM records and on the calls index page."
+ );
+ private static final AllowableValue EMAILS = new AllowableValue(
+ "/crm/v3/objects/emails",
+ "Emails",
+ "Get emails on CRM records."
+ );
+ private static final AllowableValue MEETINGS = new AllowableValue(
+ "/crm/v3/objects/meetings",
+ "Meetings",
+ "Get meetings on CRM records."
+ );
+ private static final AllowableValue NOTES = new AllowableValue(
+ "/crm/v3/objects/notes",
+ "Notes",
+ "Get notes on CRM records."
+ );
+ private static final AllowableValue TASKS = new AllowableValue(
+ "/crm/v3/objects/tasks",
+ "Tasks",
+ "Get tasks on CRM records."
+ );
+
+ // OTHER
+
+ private static final AllowableValue OWNERS = new AllowableValue(
+ "/crm/v3/owners/",
+ "Owners",
+ "HubSpot uses owners to assign specific users to contacts, companies, deals, tickets, or engagements. Any HubSpot user with access to contacts" +
+ " can be assigned as an owner, and multiple owners can be assigned to an object by creating a custom property for this purpose."
+ );
+
+ static final PropertyDescriptor ACCESS_TOKEN = new PropertyDescriptor.Builder()
+ .name("hubspot-admin-api-access-token")
+ .displayName("Admin API Access Token")
+ .description("")
Review Comment:
A description should be added
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,320 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"hubspot"})
+@CapabilityDescription("Retrieves JSON data from a private HubSpot application."
+ + " Supports incremental retrieval: Users can set the \"limit\" property which serves as the upper limit of the retrieved objects."
+ + " When this property is set the processor will retrieve new records. This processor is intended to be run on the Primary Node only.")
+@Stateful(scopes = Scope.CLUSTER, description = "When the 'Limit' attribute is set, the paging cursor is saved after executing a request."
+ + " Only the objects after the paging cursor will be retrieved. The maximum number of retrieved objects is the 'Limit' attribute."
+ + " State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected,"
+ + " the new node can pick up where the previous node left off, without duplicating the data.")
+public class GetHubSpot extends AbstractProcessor {
+
+ // OBJECTS
+
+ static final AllowableValue COMPANIES = new AllowableValue(
+ "/crm/v3/objects/companies",
+ "Companies",
+ "In HubSpot, the companies object is a standard CRM object. Individual company records can be used to store information about businesses" +
+ " and organizations within company properties."
+ );
+ static final AllowableValue CONTACTS = new AllowableValue(
+ "/crm/v3/objects/contacts",
+ "Contacts",
+ "In HubSpot, contacts store information about individuals. From marketing automation to smart content, the lead-specific data found in" +
+ " contact records helps users leverage much of HubSpot's functionality."
+ );
+ static final AllowableValue DEALS = new AllowableValue(
+ "/crm/v3/objects/deals",
+ "In HubSpot, a deal represents an ongoing transaction that a sales team is pursuing with a contact or company. It’s tracked through" +
+ " pipeline stages until won or lost."
+ );
+ static final AllowableValue FEEDBACK_SUBMISSIONS = new AllowableValue(
+ "/crm/v3/objects/feedback_submissions",
+ "In HubSpot, feedback submissions are an object which stores information submitted to a feedback survey. This includes Net Promoter Score (NPS)," +
+ " Customer Satisfaction (CSAT), Customer Effort Score (CES) and Custom Surveys."
+ );
+ static final AllowableValue LINE_ITEMS = new AllowableValue(
+ "/crm/v3/objects/line_items",
+ "Line Items",
+ "In HubSpot, line items can be thought of as a subset of products. When a product is attached to a deal, it becomes a line item. Line items can" +
+ " be created that are unique to an individual quote, but they will not be added to the product library."
+ );
+ static final AllowableValue PRODUCTS = new AllowableValue(
+ "/crm/v3/objects/products",
+ "Products",
+ "In HubSpot, products represent the goods or services to be sold. Building a product library allows the user to quickly add products to deals," +
+ " generate quotes, and report on product performance."
+ );
+ static final AllowableValue TICKETS = new AllowableValue(
+ "/crm/v3/objects/tickets",
+ "Tickets",
+ "In HubSpot, a ticket represents a customer request for help or support."
+ );
+ static final AllowableValue QUOTES = new AllowableValue(
+ "/crm/v3/objects/quotes",
+ "Quotes",
+ "In HubSpot, quotes are used to share pricing information with potential buyers."
+ );
+
+ // ENGAGEMENTS
+
+ private static final AllowableValue CALLS = new AllowableValue(
+ "/crm/v3/objects/calls",
+ "Calls",
+ "Get calls on CRM records and on the calls index page."
+ );
+ private static final AllowableValue EMAILS = new AllowableValue(
+ "/crm/v3/objects/emails",
+ "Emails",
+ "Get emails on CRM records."
+ );
+ private static final AllowableValue MEETINGS = new AllowableValue(
+ "/crm/v3/objects/meetings",
+ "Meetings",
+ "Get meetings on CRM records."
+ );
+ private static final AllowableValue NOTES = new AllowableValue(
+ "/crm/v3/objects/notes",
+ "Notes",
+ "Get notes on CRM records."
+ );
+ private static final AllowableValue TASKS = new AllowableValue(
+ "/crm/v3/objects/tasks",
+ "Tasks",
+ "Get tasks on CRM records."
+ );
+
+ // OTHER
+
+ private static final AllowableValue OWNERS = new AllowableValue(
+ "/crm/v3/owners/",
+ "Owners",
+ "HubSpot uses owners to assign specific users to contacts, companies, deals, tickets, or engagements. Any HubSpot user with access to contacts" +
+ " can be assigned as an owner, and multiple owners can be assigned to an object by creating a custom property for this purpose."
+ );
+
+ static final PropertyDescriptor ACCESS_TOKEN = new PropertyDescriptor.Builder()
+ .name("hubspot-admin-api-access-token")
Review Comment:
Prefixing all of the property names with `hubspot` is not necessary.
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,320 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"hubspot"})
+@CapabilityDescription("Retrieves JSON data from a private HubSpot application."
+ + " Supports incremental retrieval: Users can set the \"limit\" property which serves as the upper limit of the retrieved objects."
+ + " When this property is set the processor will retrieve new records. This processor is intended to be run on the Primary Node only.")
+@Stateful(scopes = Scope.CLUSTER, description = "When the 'Limit' attribute is set, the paging cursor is saved after executing a request."
+ + " Only the objects after the paging cursor will be retrieved. The maximum number of retrieved objects is the 'Limit' attribute."
+ + " State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected,"
+ + " the new node can pick up where the previous node left off, without duplicating the data.")
+public class GetHubSpot extends AbstractProcessor {
+
+ // OBJECTS
+
+ static final AllowableValue COMPANIES = new AllowableValue(
+ "/crm/v3/objects/companies",
+ "Companies",
+ "In HubSpot, the companies object is a standard CRM object. Individual company records can be used to store information about businesses" +
+ " and organizations within company properties."
+ );
+ static final AllowableValue CONTACTS = new AllowableValue(
+ "/crm/v3/objects/contacts",
+ "Contacts",
+ "In HubSpot, contacts store information about individuals. From marketing automation to smart content, the lead-specific data found in" +
+ " contact records helps users leverage much of HubSpot's functionality."
+ );
+ static final AllowableValue DEALS = new AllowableValue(
+ "/crm/v3/objects/deals",
+ "In HubSpot, a deal represents an ongoing transaction that a sales team is pursuing with a contact or company. It’s tracked through" +
+ " pipeline stages until won or lost."
+ );
+ static final AllowableValue FEEDBACK_SUBMISSIONS = new AllowableValue(
+ "/crm/v3/objects/feedback_submissions",
+ "In HubSpot, feedback submissions are an object which stores information submitted to a feedback survey. This includes Net Promoter Score (NPS)," +
+ " Customer Satisfaction (CSAT), Customer Effort Score (CES) and Custom Surveys."
+ );
+ static final AllowableValue LINE_ITEMS = new AllowableValue(
+ "/crm/v3/objects/line_items",
+ "Line Items",
+ "In HubSpot, line items can be thought of as a subset of products. When a product is attached to a deal, it becomes a line item. Line items can" +
+ " be created that are unique to an individual quote, but they will not be added to the product library."
+ );
+ static final AllowableValue PRODUCTS = new AllowableValue(
+ "/crm/v3/objects/products",
+ "Products",
+ "In HubSpot, products represent the goods or services to be sold. Building a product library allows the user to quickly add products to deals," +
+ " generate quotes, and report on product performance."
+ );
+ static final AllowableValue TICKETS = new AllowableValue(
+ "/crm/v3/objects/tickets",
+ "Tickets",
+ "In HubSpot, a ticket represents a customer request for help or support."
+ );
+ static final AllowableValue QUOTES = new AllowableValue(
+ "/crm/v3/objects/quotes",
+ "Quotes",
+ "In HubSpot, quotes are used to share pricing information with potential buyers."
+ );
+
+ // ENGAGEMENTS
+
+ private static final AllowableValue CALLS = new AllowableValue(
+ "/crm/v3/objects/calls",
+ "Calls",
+ "Get calls on CRM records and on the calls index page."
+ );
+ private static final AllowableValue EMAILS = new AllowableValue(
+ "/crm/v3/objects/emails",
+ "Emails",
+ "Get emails on CRM records."
+ );
+ private static final AllowableValue MEETINGS = new AllowableValue(
+ "/crm/v3/objects/meetings",
+ "Meetings",
+ "Get meetings on CRM records."
+ );
+ private static final AllowableValue NOTES = new AllowableValue(
+ "/crm/v3/objects/notes",
+ "Notes",
+ "Get notes on CRM records."
+ );
+ private static final AllowableValue TASKS = new AllowableValue(
+ "/crm/v3/objects/tasks",
+ "Tasks",
+ "Get tasks on CRM records."
+ );
+
+ // OTHER
+
+ private static final AllowableValue OWNERS = new AllowableValue(
+ "/crm/v3/owners/",
+ "Owners",
+ "HubSpot uses owners to assign specific users to contacts, companies, deals, tickets, or engagements. Any HubSpot user with access to contacts" +
+ " can be assigned as an owner, and multiple owners can be assigned to an object by creating a custom property for this purpose."
+ );
+
+ static final PropertyDescriptor ACCESS_TOKEN = new PropertyDescriptor.Builder()
+ .name("hubspot-admin-api-access-token")
+ .displayName("Admin API Access Token")
+ .description("")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
Review Comment:
Sensitive properties should not support expression language due to security concerns. Sensitive properties can be externalized using Parameter Contexts instead.
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,320 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"hubspot"})
+@CapabilityDescription("Retrieves JSON data from a private HubSpot application."
+ + " Supports incremental retrieval: Users can set the \"limit\" property which serves as the upper limit of the retrieved objects."
+ + " When this property is set the processor will retrieve new records. This processor is intended to be run on the Primary Node only.")
+@Stateful(scopes = Scope.CLUSTER, description = "When the 'Limit' attribute is set, the paging cursor is saved after executing a request."
+ + " Only the objects after the paging cursor will be retrieved. The maximum number of retrieved objects is the 'Limit' attribute."
+ + " State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected,"
+ + " the new node can pick up where the previous node left off, without duplicating the data.")
+public class GetHubSpot extends AbstractProcessor {
+
+ // OBJECTS
Review Comment:
This comment does not provide much value and should be removed.
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,320 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"hubspot"})
+@CapabilityDescription("Retrieves JSON data from a private HubSpot application."
+ + " Supports incremental retrieval: Users can set the \"limit\" property which serves as the upper limit of the retrieved objects."
+ + " When this property is set the processor will retrieve new records. This processor is intended to be run on the Primary Node only.")
+@Stateful(scopes = Scope.CLUSTER, description = "When the 'Limit' attribute is set, the paging cursor is saved after executing a request."
+ + " Only the objects after the paging cursor will be retrieved. The maximum number of retrieved objects is the 'Limit' attribute."
+ + " State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected,"
+ + " the new node can pick up where the previous node left off, without duplicating the data.")
+public class GetHubSpot extends AbstractProcessor {
+
+ // OBJECTS
+
+ static final AllowableValue COMPANIES = new AllowableValue(
+ "/crm/v3/objects/companies",
+ "Companies",
+ "In HubSpot, the companies object is a standard CRM object. Individual company records can be used to store information about businesses" +
+ " and organizations within company properties."
+ );
+ static final AllowableValue CONTACTS = new AllowableValue(
+ "/crm/v3/objects/contacts",
+ "Contacts",
+ "In HubSpot, contacts store information about individuals. From marketing automation to smart content, the lead-specific data found in" +
+ " contact records helps users leverage much of HubSpot's functionality."
+ );
+ static final AllowableValue DEALS = new AllowableValue(
+ "/crm/v3/objects/deals",
+ "In HubSpot, a deal represents an ongoing transaction that a sales team is pursuing with a contact or company. It’s tracked through" +
+ " pipeline stages until won or lost."
+ );
+ static final AllowableValue FEEDBACK_SUBMISSIONS = new AllowableValue(
+ "/crm/v3/objects/feedback_submissions",
+ "In HubSpot, feedback submissions are an object which stores information submitted to a feedback survey. This includes Net Promoter Score (NPS)," +
+ " Customer Satisfaction (CSAT), Customer Effort Score (CES) and Custom Surveys."
+ );
+ static final AllowableValue LINE_ITEMS = new AllowableValue(
+ "/crm/v3/objects/line_items",
+ "Line Items",
+ "In HubSpot, line items can be thought of as a subset of products. When a product is attached to a deal, it becomes a line item. Line items can" +
+ " be created that are unique to an individual quote, but they will not be added to the product library."
+ );
+ static final AllowableValue PRODUCTS = new AllowableValue(
+ "/crm/v3/objects/products",
+ "Products",
+ "In HubSpot, products represent the goods or services to be sold. Building a product library allows the user to quickly add products to deals," +
+ " generate quotes, and report on product performance."
+ );
+ static final AllowableValue TICKETS = new AllowableValue(
+ "/crm/v3/objects/tickets",
+ "Tickets",
+ "In HubSpot, a ticket represents a customer request for help or support."
+ );
+ static final AllowableValue QUOTES = new AllowableValue(
+ "/crm/v3/objects/quotes",
+ "Quotes",
+ "In HubSpot, quotes are used to share pricing information with potential buyers."
+ );
+
+ // ENGAGEMENTS
+
+ private static final AllowableValue CALLS = new AllowableValue(
+ "/crm/v3/objects/calls",
+ "Calls",
+ "Get calls on CRM records and on the calls index page."
+ );
+ private static final AllowableValue EMAILS = new AllowableValue(
+ "/crm/v3/objects/emails",
+ "Emails",
+ "Get emails on CRM records."
+ );
+ private static final AllowableValue MEETINGS = new AllowableValue(
+ "/crm/v3/objects/meetings",
+ "Meetings",
+ "Get meetings on CRM records."
+ );
+ private static final AllowableValue NOTES = new AllowableValue(
+ "/crm/v3/objects/notes",
+ "Notes",
+ "Get notes on CRM records."
+ );
+ private static final AllowableValue TASKS = new AllowableValue(
+ "/crm/v3/objects/tasks",
+ "Tasks",
+ "Get tasks on CRM records."
+ );
+
+ // OTHER
+
+ private static final AllowableValue OWNERS = new AllowableValue(
+ "/crm/v3/owners/",
+ "Owners",
+ "HubSpot uses owners to assign specific users to contacts, companies, deals, tickets, or engagements. Any HubSpot user with access to contacts" +
+ " can be assigned as an owner, and multiple owners can be assigned to an object by creating a custom property for this purpose."
+ );
+
+ static final PropertyDescriptor ACCESS_TOKEN = new PropertyDescriptor.Builder()
+ .name("hubspot-admin-api-access-token")
+ .displayName("Admin API Access Token")
+ .description("")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
+ .build();
+
+ static final PropertyDescriptor CRM_ENDPOINT = new PropertyDescriptor.Builder()
+ .name("hubspot-crm-endpoint")
+ .displayName("HubSpot CRM API Endpoint")
+ .description("The HubSpot CRM API endpoint to get")
+ .required(true)
+ .allowableValues(COMPANIES, CONTACTS, DEALS, FEEDBACK_SUBMISSIONS, LINE_ITEMS, PRODUCTS, TICKETS, QUOTES,
+ CALLS, EMAILS, MEETINGS, NOTES, TASKS, OWNERS)
+ .build();
+
+ static final PropertyDescriptor LIMIT = new PropertyDescriptor.Builder()
+ .name("hubspot-crm-limit")
+ .displayName("Limit")
+ .description("The maximum number of results to display per page")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor WEB_CLIENT_PROVIDER = new PropertyDescriptor.Builder()
+ .name("nifi-web-client")
+ .displayName("NiFi Web Client")
+ .description("Controller service for HTTP client operations")
+ .identifiesControllerService(WebClientServiceProvider.class)
+ .required(true)
+ .build();
+
+ static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("For FlowFiles created as a result of a successful query.")
+ .build();
+
+ private static final List<PropertyDescriptor> PROPERTY_DESCRIPTORS = createPropertyDescriptors();
+ public static final String API_BASE_URI = "api.hubapi.com";
+ public static final String HTTPS = "https";
+
+ private volatile WebClientServiceProvider webClientServiceProvider;
+
+ private static List<PropertyDescriptor> createPropertyDescriptors() {
+ final List<PropertyDescriptor> propertyDescriptors = new ArrayList<>(Arrays.asList(
+ ACCESS_TOKEN,
+ CRM_ENDPOINT,
+ LIMIT,
+ WEB_CLIENT_PROVIDER
+ ));
+ return Collections.unmodifiableList(propertyDescriptors);
+ }
+
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ webClientServiceProvider = context.getProperty(WEB_CLIENT_PROVIDER).asControllerService(WebClientServiceProvider.class);
+ }
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTY_DESCRIPTORS;
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ final Set<Relationship> relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ return relationships;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ final String accessToken = context.getProperty(ACCESS_TOKEN).getValue();
+ final String endpoint = context.getProperty(CRM_ENDPOINT).getValue();
+
+ final StateMap state = getStateMap(context);
+ final URI uri = createUri(context, state);
+
+ final HttpResponseEntity response = getHttpResponseEntity(accessToken, uri);
+ final ObjectMapper objectMapper = new ObjectMapper();
Review Comment:
The `ObjectMapper` can be created once as a static member variable and reused.
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,320 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"hubspot"})
+@CapabilityDescription("Retrieves JSON data from a private HubSpot application."
+ + " Supports incremental retrieval: Users can set the \"limit\" property which serves as the upper limit of the retrieved objects."
+ + " When this property is set the processor will retrieve new records. This processor is intended to be run on the Primary Node only.")
+@Stateful(scopes = Scope.CLUSTER, description = "When the 'Limit' attribute is set, the paging cursor is saved after executing a request."
+ + " Only the objects after the paging cursor will be retrieved. The maximum number of retrieved objects is the 'Limit' attribute."
+ + " State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected,"
+ + " the new node can pick up where the previous node left off, without duplicating the data.")
+public class GetHubSpot extends AbstractProcessor {
+
+ // OBJECTS
+
+ static final AllowableValue COMPANIES = new AllowableValue(
+ "/crm/v3/objects/companies",
+ "Companies",
+ "In HubSpot, the companies object is a standard CRM object. Individual company records can be used to store information about businesses" +
+ " and organizations within company properties."
+ );
+ static final AllowableValue CONTACTS = new AllowableValue(
+ "/crm/v3/objects/contacts",
+ "Contacts",
+ "In HubSpot, contacts store information about individuals. From marketing automation to smart content, the lead-specific data found in" +
+ " contact records helps users leverage much of HubSpot's functionality."
+ );
+ static final AllowableValue DEALS = new AllowableValue(
+ "/crm/v3/objects/deals",
+ "In HubSpot, a deal represents an ongoing transaction that a sales team is pursuing with a contact or company. It’s tracked through" +
+ " pipeline stages until won or lost."
+ );
+ static final AllowableValue FEEDBACK_SUBMISSIONS = new AllowableValue(
+ "/crm/v3/objects/feedback_submissions",
+ "In HubSpot, feedback submissions are an object which stores information submitted to a feedback survey. This includes Net Promoter Score (NPS)," +
+ " Customer Satisfaction (CSAT), Customer Effort Score (CES) and Custom Surveys."
+ );
+ static final AllowableValue LINE_ITEMS = new AllowableValue(
+ "/crm/v3/objects/line_items",
+ "Line Items",
+ "In HubSpot, line items can be thought of as a subset of products. When a product is attached to a deal, it becomes a line item. Line items can" +
+ " be created that are unique to an individual quote, but they will not be added to the product library."
+ );
+ static final AllowableValue PRODUCTS = new AllowableValue(
+ "/crm/v3/objects/products",
+ "Products",
+ "In HubSpot, products represent the goods or services to be sold. Building a product library allows the user to quickly add products to deals," +
+ " generate quotes, and report on product performance."
+ );
+ static final AllowableValue TICKETS = new AllowableValue(
+ "/crm/v3/objects/tickets",
+ "Tickets",
+ "In HubSpot, a ticket represents a customer request for help or support."
+ );
+ static final AllowableValue QUOTES = new AllowableValue(
+ "/crm/v3/objects/quotes",
+ "Quotes",
+ "In HubSpot, quotes are used to share pricing information with potential buyers."
+ );
+
+ // ENGAGEMENTS
+
+ private static final AllowableValue CALLS = new AllowableValue(
+ "/crm/v3/objects/calls",
+ "Calls",
+ "Get calls on CRM records and on the calls index page."
+ );
+ private static final AllowableValue EMAILS = new AllowableValue(
+ "/crm/v3/objects/emails",
+ "Emails",
+ "Get emails on CRM records."
+ );
+ private static final AllowableValue MEETINGS = new AllowableValue(
+ "/crm/v3/objects/meetings",
+ "Meetings",
+ "Get meetings on CRM records."
+ );
+ private static final AllowableValue NOTES = new AllowableValue(
+ "/crm/v3/objects/notes",
+ "Notes",
+ "Get notes on CRM records."
+ );
+ private static final AllowableValue TASKS = new AllowableValue(
+ "/crm/v3/objects/tasks",
+ "Tasks",
+ "Get tasks on CRM records."
+ );
+
+ // OTHER
+
+ private static final AllowableValue OWNERS = new AllowableValue(
+ "/crm/v3/owners/",
+ "Owners",
+ "HubSpot uses owners to assign specific users to contacts, companies, deals, tickets, or engagements. Any HubSpot user with access to contacts" +
+ " can be assigned as an owner, and multiple owners can be assigned to an object by creating a custom property for this purpose."
+ );
+
+ static final PropertyDescriptor ACCESS_TOKEN = new PropertyDescriptor.Builder()
+ .name("hubspot-admin-api-access-token")
+ .displayName("Admin API Access Token")
+ .description("")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
+ .build();
+
+ static final PropertyDescriptor CRM_ENDPOINT = new PropertyDescriptor.Builder()
+ .name("hubspot-crm-endpoint")
+ .displayName("HubSpot CRM API Endpoint")
+ .description("The HubSpot CRM API endpoint to get")
+ .required(true)
+ .allowableValues(COMPANIES, CONTACTS, DEALS, FEEDBACK_SUBMISSIONS, LINE_ITEMS, PRODUCTS, TICKETS, QUOTES,
+ CALLS, EMAILS, MEETINGS, NOTES, TASKS, OWNERS)
+ .build();
+
+ static final PropertyDescriptor LIMIT = new PropertyDescriptor.Builder()
+ .name("hubspot-crm-limit")
+ .displayName("Limit")
+ .description("The maximum number of results to display per page")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor WEB_CLIENT_PROVIDER = new PropertyDescriptor.Builder()
+ .name("nifi-web-client")
+ .displayName("NiFi Web Client")
Review Comment:
This should be named `Web Client Service Provider` following the service name:
```suggestion
.name("web-client-service-provider")
.displayName("Web Client Service Provider")
```
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/resources/docs/org.apache.nifi.processors.hubspot.GetHubSpot/additionalDetails.html:
##########
@@ -0,0 +1,143 @@
+<!DOCTYPE html>
+<html lang="en">
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<head>
+ <meta charset="utf-8"/>
+ <title>GetShopify</title>
Review Comment:
This title should be corrected
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,320 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"hubspot"})
+@CapabilityDescription("Retrieves JSON data from a private HubSpot application."
+ + " Supports incremental retrieval: Users can set the \"limit\" property which serves as the upper limit of the retrieved objects."
+ + " When this property is set the processor will retrieve new records. This processor is intended to be run on the Primary Node only.")
+@Stateful(scopes = Scope.CLUSTER, description = "When the 'Limit' attribute is set, the paging cursor is saved after executing a request."
+ + " Only the objects after the paging cursor will be retrieved. The maximum number of retrieved objects is the 'Limit' attribute."
+ + " State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected,"
+ + " the new node can pick up where the previous node left off, without duplicating the data.")
+public class GetHubSpot extends AbstractProcessor {
+
+ // OBJECTS
+
+ static final AllowableValue COMPANIES = new AllowableValue(
+ "/crm/v3/objects/companies",
+ "Companies",
+ "In HubSpot, the companies object is a standard CRM object. Individual company records can be used to store information about businesses" +
+ " and organizations within company properties."
+ );
+ static final AllowableValue CONTACTS = new AllowableValue(
+ "/crm/v3/objects/contacts",
+ "Contacts",
+ "In HubSpot, contacts store information about individuals. From marketing automation to smart content, the lead-specific data found in" +
+ " contact records helps users leverage much of HubSpot's functionality."
+ );
+ static final AllowableValue DEALS = new AllowableValue(
+ "/crm/v3/objects/deals",
+ "In HubSpot, a deal represents an ongoing transaction that a sales team is pursuing with a contact or company. It’s tracked through" +
+ " pipeline stages until won or lost."
+ );
+ static final AllowableValue FEEDBACK_SUBMISSIONS = new AllowableValue(
+ "/crm/v3/objects/feedback_submissions",
+ "In HubSpot, feedback submissions are an object which stores information submitted to a feedback survey. This includes Net Promoter Score (NPS)," +
+ " Customer Satisfaction (CSAT), Customer Effort Score (CES) and Custom Surveys."
+ );
+ static final AllowableValue LINE_ITEMS = new AllowableValue(
+ "/crm/v3/objects/line_items",
+ "Line Items",
+ "In HubSpot, line items can be thought of as a subset of products. When a product is attached to a deal, it becomes a line item. Line items can" +
+ " be created that are unique to an individual quote, but they will not be added to the product library."
+ );
+ static final AllowableValue PRODUCTS = new AllowableValue(
+ "/crm/v3/objects/products",
+ "Products",
+ "In HubSpot, products represent the goods or services to be sold. Building a product library allows the user to quickly add products to deals," +
+ " generate quotes, and report on product performance."
+ );
+ static final AllowableValue TICKETS = new AllowableValue(
+ "/crm/v3/objects/tickets",
+ "Tickets",
+ "In HubSpot, a ticket represents a customer request for help or support."
+ );
+ static final AllowableValue QUOTES = new AllowableValue(
+ "/crm/v3/objects/quotes",
+ "Quotes",
+ "In HubSpot, quotes are used to share pricing information with potential buyers."
+ );
+
+ // ENGAGEMENTS
+
+ private static final AllowableValue CALLS = new AllowableValue(
+ "/crm/v3/objects/calls",
+ "Calls",
+ "Get calls on CRM records and on the calls index page."
+ );
+ private static final AllowableValue EMAILS = new AllowableValue(
+ "/crm/v3/objects/emails",
+ "Emails",
+ "Get emails on CRM records."
+ );
+ private static final AllowableValue MEETINGS = new AllowableValue(
+ "/crm/v3/objects/meetings",
+ "Meetings",
+ "Get meetings on CRM records."
+ );
+ private static final AllowableValue NOTES = new AllowableValue(
+ "/crm/v3/objects/notes",
+ "Notes",
+ "Get notes on CRM records."
+ );
+ private static final AllowableValue TASKS = new AllowableValue(
+ "/crm/v3/objects/tasks",
+ "Tasks",
+ "Get tasks on CRM records."
+ );
+
+ // OTHER
+
+ private static final AllowableValue OWNERS = new AllowableValue(
+ "/crm/v3/owners/",
+ "Owners",
+ "HubSpot uses owners to assign specific users to contacts, companies, deals, tickets, or engagements. Any HubSpot user with access to contacts" +
+ " can be assigned as an owner, and multiple owners can be assigned to an object by creating a custom property for this purpose."
+ );
+
+ static final PropertyDescriptor ACCESS_TOKEN = new PropertyDescriptor.Builder()
+ .name("hubspot-admin-api-access-token")
+ .displayName("Admin API Access Token")
+ .description("")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
+ .build();
+
+ static final PropertyDescriptor CRM_ENDPOINT = new PropertyDescriptor.Builder()
+ .name("hubspot-crm-endpoint")
+ .displayName("HubSpot CRM API Endpoint")
+ .description("The HubSpot CRM API endpoint to get")
+ .required(true)
+ .allowableValues(COMPANIES, CONTACTS, DEALS, FEEDBACK_SUBMISSIONS, LINE_ITEMS, PRODUCTS, TICKETS, QUOTES,
+ CALLS, EMAILS, MEETINGS, NOTES, TASKS, OWNERS)
+ .build();
+
+ static final PropertyDescriptor LIMIT = new PropertyDescriptor.Builder()
+ .name("hubspot-crm-limit")
+ .displayName("Limit")
+ .description("The maximum number of results to display per page")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor WEB_CLIENT_PROVIDER = new PropertyDescriptor.Builder()
+ .name("nifi-web-client")
+ .displayName("NiFi Web Client")
+ .description("Controller service for HTTP client operations")
+ .identifiesControllerService(WebClientServiceProvider.class)
+ .required(true)
+ .build();
+
+ static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("For FlowFiles created as a result of a successful query.")
+ .build();
+
+ private static final List<PropertyDescriptor> PROPERTY_DESCRIPTORS = createPropertyDescriptors();
+ public static final String API_BASE_URI = "api.hubapi.com";
+ public static final String HTTPS = "https";
+
+ private volatile WebClientServiceProvider webClientServiceProvider;
+
+ private static List<PropertyDescriptor> createPropertyDescriptors() {
+ final List<PropertyDescriptor> propertyDescriptors = new ArrayList<>(Arrays.asList(
+ ACCESS_TOKEN,
+ CRM_ENDPOINT,
+ LIMIT,
+ WEB_CLIENT_PROVIDER
+ ));
+ return Collections.unmodifiableList(propertyDescriptors);
+ }
+
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ webClientServiceProvider = context.getProperty(WEB_CLIENT_PROVIDER).asControllerService(WebClientServiceProvider.class);
+ }
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTY_DESCRIPTORS;
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ final Set<Relationship> relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ return relationships;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ final String accessToken = context.getProperty(ACCESS_TOKEN).getValue();
+ final String endpoint = context.getProperty(CRM_ENDPOINT).getValue();
+
+ final StateMap state = getStateMap(context);
+ final URI uri = createUri(context, state);
+
+ final HttpResponseEntity response = getHttpResponseEntity(accessToken, uri);
+ final ObjectMapper objectMapper = new ObjectMapper();
+ final JsonFactory jsonFactory = objectMapper.getFactory();
+ FlowFile flowFile = session.create();
+
+ flowFile = session.write(flowFile, out -> {
+
+
+ try (JsonParser jsonParser = jsonFactory.createParser(response.body());
+ final JsonGenerator jsonGenerator = jsonFactory.createGenerator(out, JsonEncoding.UTF8)) {
+ while (jsonParser.nextToken() != null) {
+ if (jsonParser.getCurrentToken() == JsonToken.FIELD_NAME && jsonParser.getCurrentName().equals("results")) {
+ jsonParser.nextToken();
+ jsonGenerator.copyCurrentStructure(jsonParser);
+ }
+ String fieldname = jsonParser.getCurrentName();
+ if ("after".equals(fieldname)) {
+ jsonParser.nextToken();
+ Map<String, String> newStateMap = new HashMap<>(state.toMap());
+ newStateMap.put(endpoint, jsonParser.getText());
+ updateState(context, newStateMap);
+ break;
+ }
+ }
+ }
+ });
+ session.transfer(flowFile, REL_SUCCESS);
+ }
+
+ HttpResponseEntity getHttpResponseEntity(final String accessToken, final URI uri) {
+ return webClientServiceProvider.getWebClientService()
+ .get()
+ .uri(uri)
+ .header("Authorization", "Bearer " + accessToken)
+ .retrieve();
+ }
+
+ HttpUriBuilder getBaseUri(final ProcessContext context) {
+ final String path = context.getProperty(CRM_ENDPOINT).getValue();
+ return webClientServiceProvider.getHttpUriBuilder()
+ .scheme(HTTPS)
+ .host(API_BASE_URI)
+ .encodedPath(path);
+ }
+
+ URI createUri(final ProcessContext context, final StateMap state) {
+ final String path = context.getProperty(CRM_ENDPOINT).getValue();
+ final HttpUriBuilder uriBuilder = getBaseUri(context);
+
+ final boolean isLimitSet = context.getProperty(LIMIT).isSet();
+ if (isLimitSet) {
+ final String limit = context.getProperty(LIMIT).getValue();
+ uriBuilder.addQueryParameter("limit", limit);
+ }
+
+ final String cursor = state.get(path);
+ if (cursor != null) {
+ uriBuilder.addQueryParameter("after", cursor);
Review Comment:
Recommend creating a static variable and reusing for `after` in this method and the JSON parsing block.
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,320 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"hubspot"})
+@CapabilityDescription("Retrieves JSON data from a private HubSpot application."
+ + " Supports incremental retrieval: Users can set the \"limit\" property which serves as the upper limit of the retrieved objects."
+ + " When this property is set the processor will retrieve new records. This processor is intended to be run on the Primary Node only.")
+@Stateful(scopes = Scope.CLUSTER, description = "When the 'Limit' attribute is set, the paging cursor is saved after executing a request."
+ + " Only the objects after the paging cursor will be retrieved. The maximum number of retrieved objects is the 'Limit' attribute."
+ + " State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected,"
+ + " the new node can pick up where the previous node left off, without duplicating the data.")
+public class GetHubSpot extends AbstractProcessor {
+
+ // OBJECTS
+
+ static final AllowableValue COMPANIES = new AllowableValue(
+ "/crm/v3/objects/companies",
+ "Companies",
+ "In HubSpot, the companies object is a standard CRM object. Individual company records can be used to store information about businesses" +
+ " and organizations within company properties."
+ );
+ static final AllowableValue CONTACTS = new AllowableValue(
+ "/crm/v3/objects/contacts",
+ "Contacts",
+ "In HubSpot, contacts store information about individuals. From marketing automation to smart content, the lead-specific data found in" +
+ " contact records helps users leverage much of HubSpot's functionality."
+ );
+ static final AllowableValue DEALS = new AllowableValue(
+ "/crm/v3/objects/deals",
+ "In HubSpot, a deal represents an ongoing transaction that a sales team is pursuing with a contact or company. It’s tracked through" +
+ " pipeline stages until won or lost."
+ );
+ static final AllowableValue FEEDBACK_SUBMISSIONS = new AllowableValue(
+ "/crm/v3/objects/feedback_submissions",
+ "In HubSpot, feedback submissions are an object which stores information submitted to a feedback survey. This includes Net Promoter Score (NPS)," +
+ " Customer Satisfaction (CSAT), Customer Effort Score (CES) and Custom Surveys."
+ );
+ static final AllowableValue LINE_ITEMS = new AllowableValue(
+ "/crm/v3/objects/line_items",
+ "Line Items",
+ "In HubSpot, line items can be thought of as a subset of products. When a product is attached to a deal, it becomes a line item. Line items can" +
+ " be created that are unique to an individual quote, but they will not be added to the product library."
+ );
+ static final AllowableValue PRODUCTS = new AllowableValue(
+ "/crm/v3/objects/products",
+ "Products",
+ "In HubSpot, products represent the goods or services to be sold. Building a product library allows the user to quickly add products to deals," +
+ " generate quotes, and report on product performance."
+ );
+ static final AllowableValue TICKETS = new AllowableValue(
+ "/crm/v3/objects/tickets",
+ "Tickets",
+ "In HubSpot, a ticket represents a customer request for help or support."
+ );
+ static final AllowableValue QUOTES = new AllowableValue(
+ "/crm/v3/objects/quotes",
+ "Quotes",
+ "In HubSpot, quotes are used to share pricing information with potential buyers."
+ );
+
+ // ENGAGEMENTS
+
+ private static final AllowableValue CALLS = new AllowableValue(
+ "/crm/v3/objects/calls",
+ "Calls",
+ "Get calls on CRM records and on the calls index page."
+ );
+ private static final AllowableValue EMAILS = new AllowableValue(
+ "/crm/v3/objects/emails",
+ "Emails",
+ "Get emails on CRM records."
+ );
+ private static final AllowableValue MEETINGS = new AllowableValue(
+ "/crm/v3/objects/meetings",
+ "Meetings",
+ "Get meetings on CRM records."
+ );
+ private static final AllowableValue NOTES = new AllowableValue(
+ "/crm/v3/objects/notes",
+ "Notes",
+ "Get notes on CRM records."
+ );
+ private static final AllowableValue TASKS = new AllowableValue(
+ "/crm/v3/objects/tasks",
+ "Tasks",
+ "Get tasks on CRM records."
+ );
+
+ // OTHER
+
+ private static final AllowableValue OWNERS = new AllowableValue(
+ "/crm/v3/owners/",
+ "Owners",
+ "HubSpot uses owners to assign specific users to contacts, companies, deals, tickets, or engagements. Any HubSpot user with access to contacts" +
+ " can be assigned as an owner, and multiple owners can be assigned to an object by creating a custom property for this purpose."
+ );
Review Comment:
What do you think about creating an `enum` that implements `DescribedValue` and externalizing these values?
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,320 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"hubspot"})
+@CapabilityDescription("Retrieves JSON data from a private HubSpot application."
+ + " Supports incremental retrieval: Users can set the \"limit\" property which serves as the upper limit of the retrieved objects."
+ + " When this property is set the processor will retrieve new records. This processor is intended to be run on the Primary Node only.")
+@Stateful(scopes = Scope.CLUSTER, description = "When the 'Limit' attribute is set, the paging cursor is saved after executing a request."
+ + " Only the objects after the paging cursor will be retrieved. The maximum number of retrieved objects is the 'Limit' attribute."
+ + " State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected,"
+ + " the new node can pick up where the previous node left off, without duplicating the data.")
+public class GetHubSpot extends AbstractProcessor {
+
+ // OBJECTS
+
+ static final AllowableValue COMPANIES = new AllowableValue(
+ "/crm/v3/objects/companies",
+ "Companies",
+ "In HubSpot, the companies object is a standard CRM object. Individual company records can be used to store information about businesses" +
+ " and organizations within company properties."
+ );
+ static final AllowableValue CONTACTS = new AllowableValue(
+ "/crm/v3/objects/contacts",
+ "Contacts",
+ "In HubSpot, contacts store information about individuals. From marketing automation to smart content, the lead-specific data found in" +
+ " contact records helps users leverage much of HubSpot's functionality."
+ );
+ static final AllowableValue DEALS = new AllowableValue(
+ "/crm/v3/objects/deals",
+ "In HubSpot, a deal represents an ongoing transaction that a sales team is pursuing with a contact or company. It’s tracked through" +
+ " pipeline stages until won or lost."
+ );
+ static final AllowableValue FEEDBACK_SUBMISSIONS = new AllowableValue(
+ "/crm/v3/objects/feedback_submissions",
+ "In HubSpot, feedback submissions are an object which stores information submitted to a feedback survey. This includes Net Promoter Score (NPS)," +
+ " Customer Satisfaction (CSAT), Customer Effort Score (CES) and Custom Surveys."
+ );
+ static final AllowableValue LINE_ITEMS = new AllowableValue(
+ "/crm/v3/objects/line_items",
+ "Line Items",
+ "In HubSpot, line items can be thought of as a subset of products. When a product is attached to a deal, it becomes a line item. Line items can" +
+ " be created that are unique to an individual quote, but they will not be added to the product library."
+ );
+ static final AllowableValue PRODUCTS = new AllowableValue(
+ "/crm/v3/objects/products",
+ "Products",
+ "In HubSpot, products represent the goods or services to be sold. Building a product library allows the user to quickly add products to deals," +
+ " generate quotes, and report on product performance."
+ );
+ static final AllowableValue TICKETS = new AllowableValue(
+ "/crm/v3/objects/tickets",
+ "Tickets",
+ "In HubSpot, a ticket represents a customer request for help or support."
+ );
+ static final AllowableValue QUOTES = new AllowableValue(
+ "/crm/v3/objects/quotes",
+ "Quotes",
+ "In HubSpot, quotes are used to share pricing information with potential buyers."
+ );
+
+ // ENGAGEMENTS
+
+ private static final AllowableValue CALLS = new AllowableValue(
+ "/crm/v3/objects/calls",
+ "Calls",
+ "Get calls on CRM records and on the calls index page."
+ );
+ private static final AllowableValue EMAILS = new AllowableValue(
+ "/crm/v3/objects/emails",
+ "Emails",
+ "Get emails on CRM records."
+ );
+ private static final AllowableValue MEETINGS = new AllowableValue(
+ "/crm/v3/objects/meetings",
+ "Meetings",
+ "Get meetings on CRM records."
+ );
+ private static final AllowableValue NOTES = new AllowableValue(
+ "/crm/v3/objects/notes",
+ "Notes",
+ "Get notes on CRM records."
+ );
+ private static final AllowableValue TASKS = new AllowableValue(
+ "/crm/v3/objects/tasks",
+ "Tasks",
+ "Get tasks on CRM records."
+ );
+
+ // OTHER
+
+ private static final AllowableValue OWNERS = new AllowableValue(
+ "/crm/v3/owners/",
+ "Owners",
+ "HubSpot uses owners to assign specific users to contacts, companies, deals, tickets, or engagements. Any HubSpot user with access to contacts" +
+ " can be assigned as an owner, and multiple owners can be assigned to an object by creating a custom property for this purpose."
+ );
+
+ static final PropertyDescriptor ACCESS_TOKEN = new PropertyDescriptor.Builder()
+ .name("hubspot-admin-api-access-token")
+ .displayName("Admin API Access Token")
+ .description("")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
+ .build();
+
+ static final PropertyDescriptor CRM_ENDPOINT = new PropertyDescriptor.Builder()
+ .name("hubspot-crm-endpoint")
+ .displayName("HubSpot CRM API Endpoint")
+ .description("The HubSpot CRM API endpoint to get")
+ .required(true)
+ .allowableValues(COMPANIES, CONTACTS, DEALS, FEEDBACK_SUBMISSIONS, LINE_ITEMS, PRODUCTS, TICKETS, QUOTES,
+ CALLS, EMAILS, MEETINGS, NOTES, TASKS, OWNERS)
+ .build();
+
+ static final PropertyDescriptor LIMIT = new PropertyDescriptor.Builder()
+ .name("hubspot-crm-limit")
+ .displayName("Limit")
+ .description("The maximum number of results to display per page")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor WEB_CLIENT_PROVIDER = new PropertyDescriptor.Builder()
+ .name("nifi-web-client")
+ .displayName("NiFi Web Client")
+ .description("Controller service for HTTP client operations")
+ .identifiesControllerService(WebClientServiceProvider.class)
+ .required(true)
+ .build();
+
+ static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("For FlowFiles created as a result of a successful query.")
+ .build();
+
+ private static final List<PropertyDescriptor> PROPERTY_DESCRIPTORS = createPropertyDescriptors();
+ public static final String API_BASE_URI = "api.hubapi.com";
+ public static final String HTTPS = "https";
+
+ private volatile WebClientServiceProvider webClientServiceProvider;
+
+ private static List<PropertyDescriptor> createPropertyDescriptors() {
+ final List<PropertyDescriptor> propertyDescriptors = new ArrayList<>(Arrays.asList(
+ ACCESS_TOKEN,
+ CRM_ENDPOINT,
+ LIMIT,
+ WEB_CLIENT_PROVIDER
+ ));
+ return Collections.unmodifiableList(propertyDescriptors);
+ }
+
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ webClientServiceProvider = context.getProperty(WEB_CLIENT_PROVIDER).asControllerService(WebClientServiceProvider.class);
+ }
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTY_DESCRIPTORS;
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ final Set<Relationship> relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ return relationships;
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ final String accessToken = context.getProperty(ACCESS_TOKEN).getValue();
+ final String endpoint = context.getProperty(CRM_ENDPOINT).getValue();
+
+ final StateMap state = getStateMap(context);
+ final URI uri = createUri(context, state);
+
+ final HttpResponseEntity response = getHttpResponseEntity(accessToken, uri);
+ final ObjectMapper objectMapper = new ObjectMapper();
+ final JsonFactory jsonFactory = objectMapper.getFactory();
+ FlowFile flowFile = session.create();
+
+ flowFile = session.write(flowFile, out -> {
+
+
+ try (JsonParser jsonParser = jsonFactory.createParser(response.body());
+ final JsonGenerator jsonGenerator = jsonFactory.createGenerator(out, JsonEncoding.UTF8)) {
+ while (jsonParser.nextToken() != null) {
+ if (jsonParser.getCurrentToken() == JsonToken.FIELD_NAME && jsonParser.getCurrentName().equals("results")) {
+ jsonParser.nextToken();
+ jsonGenerator.copyCurrentStructure(jsonParser);
+ }
+ String fieldname = jsonParser.getCurrentName();
+ if ("after".equals(fieldname)) {
+ jsonParser.nextToken();
+ Map<String, String> newStateMap = new HashMap<>(state.toMap());
+ newStateMap.put(endpoint, jsonParser.getText());
+ updateState(context, newStateMap);
+ break;
+ }
+ }
+ }
+ });
+ session.transfer(flowFile, REL_SUCCESS);
+ }
+
+ HttpResponseEntity getHttpResponseEntity(final String accessToken, final URI uri) {
+ return webClientServiceProvider.getWebClientService()
+ .get()
+ .uri(uri)
+ .header("Authorization", "Bearer " + accessToken)
+ .retrieve();
+ }
+
+ HttpUriBuilder getBaseUri(final ProcessContext context) {
+ final String path = context.getProperty(CRM_ENDPOINT).getValue();
+ return webClientServiceProvider.getHttpUriBuilder()
+ .scheme(HTTPS)
+ .host(API_BASE_URI)
+ .encodedPath(path);
+ }
+
+ URI createUri(final ProcessContext context, final StateMap state) {
+ final String path = context.getProperty(CRM_ENDPOINT).getValue();
+ final HttpUriBuilder uriBuilder = getBaseUri(context);
+
+ final boolean isLimitSet = context.getProperty(LIMIT).isSet();
+ if (isLimitSet) {
+ final String limit = context.getProperty(LIMIT).getValue();
+ uriBuilder.addQueryParameter("limit", limit);
Review Comment:
Recommend creating a static variable and reusing for `limit` in this method and the JSON parsing block.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] Lehel44 commented on a diff in pull request #6301: NIFI-10356: Create GetHubSpot processor
Posted by GitBox <gi...@apache.org>.
Lehel44 commented on code in PR #6301:
URL: https://github.com/apache/nifi/pull/6301#discussion_r947270902
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,320 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"hubspot"})
+@CapabilityDescription("Retrieves JSON data from a private HubSpot application."
+ + " Supports incremental retrieval: Users can set the \"limit\" property which serves as the upper limit of the retrieved objects."
+ + " When this property is set the processor will retrieve new records. This processor is intended to be run on the Primary Node only.")
+@Stateful(scopes = Scope.CLUSTER, description = "When the 'Limit' attribute is set, the paging cursor is saved after executing a request."
+ + " Only the objects after the paging cursor will be retrieved. The maximum number of retrieved objects is the 'Limit' attribute."
+ + " State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected,"
+ + " the new node can pick up where the previous node left off, without duplicating the data.")
+public class GetHubSpot extends AbstractProcessor {
+
+ // OBJECTS
+
+ static final AllowableValue COMPANIES = new AllowableValue(
+ "/crm/v3/objects/companies",
+ "Companies",
+ "In HubSpot, the companies object is a standard CRM object. Individual company records can be used to store information about businesses" +
+ " and organizations within company properties."
+ );
+ static final AllowableValue CONTACTS = new AllowableValue(
+ "/crm/v3/objects/contacts",
+ "Contacts",
+ "In HubSpot, contacts store information about individuals. From marketing automation to smart content, the lead-specific data found in" +
+ " contact records helps users leverage much of HubSpot's functionality."
+ );
+ static final AllowableValue DEALS = new AllowableValue(
+ "/crm/v3/objects/deals",
+ "In HubSpot, a deal represents an ongoing transaction that a sales team is pursuing with a contact or company. It’s tracked through" +
+ " pipeline stages until won or lost."
+ );
+ static final AllowableValue FEEDBACK_SUBMISSIONS = new AllowableValue(
+ "/crm/v3/objects/feedback_submissions",
+ "In HubSpot, feedback submissions are an object which stores information submitted to a feedback survey. This includes Net Promoter Score (NPS)," +
+ " Customer Satisfaction (CSAT), Customer Effort Score (CES) and Custom Surveys."
+ );
+ static final AllowableValue LINE_ITEMS = new AllowableValue(
+ "/crm/v3/objects/line_items",
+ "Line Items",
+ "In HubSpot, line items can be thought of as a subset of products. When a product is attached to a deal, it becomes a line item. Line items can" +
+ " be created that are unique to an individual quote, but they will not be added to the product library."
+ );
+ static final AllowableValue PRODUCTS = new AllowableValue(
+ "/crm/v3/objects/products",
+ "Products",
+ "In HubSpot, products represent the goods or services to be sold. Building a product library allows the user to quickly add products to deals," +
+ " generate quotes, and report on product performance."
+ );
+ static final AllowableValue TICKETS = new AllowableValue(
+ "/crm/v3/objects/tickets",
+ "Tickets",
+ "In HubSpot, a ticket represents a customer request for help or support."
+ );
+ static final AllowableValue QUOTES = new AllowableValue(
+ "/crm/v3/objects/quotes",
+ "Quotes",
+ "In HubSpot, quotes are used to share pricing information with potential buyers."
+ );
+
+ // ENGAGEMENTS
+
+ private static final AllowableValue CALLS = new AllowableValue(
+ "/crm/v3/objects/calls",
+ "Calls",
+ "Get calls on CRM records and on the calls index page."
+ );
+ private static final AllowableValue EMAILS = new AllowableValue(
+ "/crm/v3/objects/emails",
+ "Emails",
+ "Get emails on CRM records."
+ );
+ private static final AllowableValue MEETINGS = new AllowableValue(
+ "/crm/v3/objects/meetings",
+ "Meetings",
+ "Get meetings on CRM records."
+ );
+ private static final AllowableValue NOTES = new AllowableValue(
+ "/crm/v3/objects/notes",
+ "Notes",
+ "Get notes on CRM records."
+ );
+ private static final AllowableValue TASKS = new AllowableValue(
+ "/crm/v3/objects/tasks",
+ "Tasks",
+ "Get tasks on CRM records."
+ );
+
+ // OTHER
+
+ private static final AllowableValue OWNERS = new AllowableValue(
+ "/crm/v3/owners/",
+ "Owners",
+ "HubSpot uses owners to assign specific users to contacts, companies, deals, tickets, or engagements. Any HubSpot user with access to contacts" +
+ " can be assigned as an owner, and multiple owners can be assigned to an object by creating a custom property for this purpose."
+ );
Review Comment:
I usually extract them if there are conditions on the allowable values. Although, there are many Allowable values listed here and for better separation I can extract them to an enum.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] Lehel44 commented on a diff in pull request #6301: NIFI-10356: Create GetHubSpot processor
Posted by GitBox <gi...@apache.org>.
Lehel44 commented on code in PR #6301:
URL: https://github.com/apache/nifi/pull/6301#discussion_r956755047
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/resources/docs/org.apache.nifi.processors.hubspot.GetHubSpot/additionalDetails.html:
##########
@@ -0,0 +1,143 @@
+<!DOCTYPE html>
+<html lang="en">
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<head>
+ <meta charset="utf-8"/>
+ <title>GetHubSpot</title>
+ <link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"/>
+</head>
+
+<body>
+<h2>Incremental Loading</h2>
Review Comment:
That's a good idea. I added a section mentioning HubSpot's Private App documentation. The API keys are getting deprecated and the username-pw OAuth flow isn't supported, so the only authentication are the App Access Tokens.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] Lehel44 commented on a diff in pull request #6301: NIFI-10356: Create GetHubSpot processor
Posted by GitBox <gi...@apache.org>.
Lehel44 commented on code in PR #6301:
URL: https://github.com/apache/nifi/pull/6301#discussion_r946196450
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,320 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"hubspot"})
+@CapabilityDescription("Retrieves JSON data from a private HubSpot application."
+ + " Supports incremental retrieval: Users can set the \"limit\" property which serves as the upper limit of the retrieved objects."
+ + " When this property is set the processor will retrieve new records. This processor is intended to be run on the Primary Node only.")
+@Stateful(scopes = Scope.CLUSTER, description = "When the 'Limit' attribute is set, the paging cursor is saved after executing a request."
+ + " Only the objects after the paging cursor will be retrieved. The maximum number of retrieved objects is the 'Limit' attribute."
+ + " State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected,"
+ + " the new node can pick up where the previous node left off, without duplicating the data.")
+public class GetHubSpot extends AbstractProcessor {
+
+ // OBJECTS
+
+ static final AllowableValue COMPANIES = new AllowableValue(
+ "/crm/v3/objects/companies",
+ "Companies",
+ "In HubSpot, the companies object is a standard CRM object. Individual company records can be used to store information about businesses" +
+ " and organizations within company properties."
+ );
+ static final AllowableValue CONTACTS = new AllowableValue(
+ "/crm/v3/objects/contacts",
+ "Contacts",
+ "In HubSpot, contacts store information about individuals. From marketing automation to smart content, the lead-specific data found in" +
+ " contact records helps users leverage much of HubSpot's functionality."
+ );
+ static final AllowableValue DEALS = new AllowableValue(
+ "/crm/v3/objects/deals",
+ "In HubSpot, a deal represents an ongoing transaction that a sales team is pursuing with a contact or company. It’s tracked through" +
+ " pipeline stages until won or lost."
+ );
+ static final AllowableValue FEEDBACK_SUBMISSIONS = new AllowableValue(
+ "/crm/v3/objects/feedback_submissions",
+ "In HubSpot, feedback submissions are an object which stores information submitted to a feedback survey. This includes Net Promoter Score (NPS)," +
+ " Customer Satisfaction (CSAT), Customer Effort Score (CES) and Custom Surveys."
+ );
+ static final AllowableValue LINE_ITEMS = new AllowableValue(
+ "/crm/v3/objects/line_items",
+ "Line Items",
+ "In HubSpot, line items can be thought of as a subset of products. When a product is attached to a deal, it becomes a line item. Line items can" +
+ " be created that are unique to an individual quote, but they will not be added to the product library."
+ );
+ static final AllowableValue PRODUCTS = new AllowableValue(
+ "/crm/v3/objects/products",
+ "Products",
+ "In HubSpot, products represent the goods or services to be sold. Building a product library allows the user to quickly add products to deals," +
+ " generate quotes, and report on product performance."
+ );
+ static final AllowableValue TICKETS = new AllowableValue(
+ "/crm/v3/objects/tickets",
+ "Tickets",
+ "In HubSpot, a ticket represents a customer request for help or support."
+ );
+ static final AllowableValue QUOTES = new AllowableValue(
+ "/crm/v3/objects/quotes",
+ "Quotes",
+ "In HubSpot, quotes are used to share pricing information with potential buyers."
+ );
+
+ // ENGAGEMENTS
+
+ private static final AllowableValue CALLS = new AllowableValue(
+ "/crm/v3/objects/calls",
+ "Calls",
+ "Get calls on CRM records and on the calls index page."
+ );
+ private static final AllowableValue EMAILS = new AllowableValue(
+ "/crm/v3/objects/emails",
+ "Emails",
+ "Get emails on CRM records."
+ );
+ private static final AllowableValue MEETINGS = new AllowableValue(
+ "/crm/v3/objects/meetings",
+ "Meetings",
+ "Get meetings on CRM records."
+ );
+ private static final AllowableValue NOTES = new AllowableValue(
+ "/crm/v3/objects/notes",
+ "Notes",
+ "Get notes on CRM records."
+ );
+ private static final AllowableValue TASKS = new AllowableValue(
+ "/crm/v3/objects/tasks",
+ "Tasks",
+ "Get tasks on CRM records."
+ );
+
+ // OTHER
+
+ private static final AllowableValue OWNERS = new AllowableValue(
+ "/crm/v3/owners/",
+ "Owners",
+ "HubSpot uses owners to assign specific users to contacts, companies, deals, tickets, or engagements. Any HubSpot user with access to contacts" +
+ " can be assigned as an owner, and multiple owners can be assigned to an object by creating a custom property for this purpose."
+ );
+
+ static final PropertyDescriptor ACCESS_TOKEN = new PropertyDescriptor.Builder()
Review Comment:
Thanks for the suggestion. Unfortunately they only support the Authorization Code grant type and our OAuthTokenProvider service works with Password and Client Credentials grant types.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] Lehel44 commented on a diff in pull request #6301: NIFI-10356: Create GetHubSpot processor
Posted by GitBox <gi...@apache.org>.
Lehel44 commented on code in PR #6301:
URL: https://github.com/apache/nifi/pull/6301#discussion_r947260325
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/pom.xml:
##########
@@ -0,0 +1,104 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xmlns="http://maven.apache.org/POM/4.0.0"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>nifi-hubspot-bundle</artifactId>
+ <groupId>org.apache.nifi</groupId>
+ <version>1.18.0-SNAPSHOT</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>nifi-hubspot-processors</artifactId>
+
+ <dependencies>
+ <dependency>
+ <groupId>commons-io</groupId>
+ <artifactId>commons-io</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-web-client-api</artifactId>
+ <version>1.18.0-SNAPSHOT</version>
+ <scope>compile</scope>
+ </dependency>
+ <dependency>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>jackson-databind</artifactId>
+ </dependency>
+ <!-- Test dependencies -->
Review Comment:
I would leave it there to separate test and non test dependencies, but I reordered them correctly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] Lehel44 commented on a diff in pull request #6301: NIFI-10356: Create GetHubSpot processor
Posted by GitBox <gi...@apache.org>.
Lehel44 commented on code in PR #6301:
URL: https://github.com/apache/nifi/pull/6301#discussion_r947270902
##########
nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/GetHubSpot.java:
##########
@@ -0,0 +1,320 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hubspot;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.HttpUriBuilder;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+@PrimaryNodeOnly
+@TriggerSerially
+@TriggerWhenEmpty
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"hubspot"})
+@CapabilityDescription("Retrieves JSON data from a private HubSpot application."
+ + " Supports incremental retrieval: Users can set the \"limit\" property which serves as the upper limit of the retrieved objects."
+ + " When this property is set the processor will retrieve new records. This processor is intended to be run on the Primary Node only.")
+@Stateful(scopes = Scope.CLUSTER, description = "When the 'Limit' attribute is set, the paging cursor is saved after executing a request."
+ + " Only the objects after the paging cursor will be retrieved. The maximum number of retrieved objects is the 'Limit' attribute."
+ + " State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected,"
+ + " the new node can pick up where the previous node left off, without duplicating the data.")
+public class GetHubSpot extends AbstractProcessor {
+
+ // OBJECTS
+
+ static final AllowableValue COMPANIES = new AllowableValue(
+ "/crm/v3/objects/companies",
+ "Companies",
+ "In HubSpot, the companies object is a standard CRM object. Individual company records can be used to store information about businesses" +
+ " and organizations within company properties."
+ );
+ static final AllowableValue CONTACTS = new AllowableValue(
+ "/crm/v3/objects/contacts",
+ "Contacts",
+ "In HubSpot, contacts store information about individuals. From marketing automation to smart content, the lead-specific data found in" +
+ " contact records helps users leverage much of HubSpot's functionality."
+ );
+ static final AllowableValue DEALS = new AllowableValue(
+ "/crm/v3/objects/deals",
+ "In HubSpot, a deal represents an ongoing transaction that a sales team is pursuing with a contact or company. It’s tracked through" +
+ " pipeline stages until won or lost."
+ );
+ static final AllowableValue FEEDBACK_SUBMISSIONS = new AllowableValue(
+ "/crm/v3/objects/feedback_submissions",
+ "In HubSpot, feedback submissions are an object which stores information submitted to a feedback survey. This includes Net Promoter Score (NPS)," +
+ " Customer Satisfaction (CSAT), Customer Effort Score (CES) and Custom Surveys."
+ );
+ static final AllowableValue LINE_ITEMS = new AllowableValue(
+ "/crm/v3/objects/line_items",
+ "Line Items",
+ "In HubSpot, line items can be thought of as a subset of products. When a product is attached to a deal, it becomes a line item. Line items can" +
+ " be created that are unique to an individual quote, but they will not be added to the product library."
+ );
+ static final AllowableValue PRODUCTS = new AllowableValue(
+ "/crm/v3/objects/products",
+ "Products",
+ "In HubSpot, products represent the goods or services to be sold. Building a product library allows the user to quickly add products to deals," +
+ " generate quotes, and report on product performance."
+ );
+ static final AllowableValue TICKETS = new AllowableValue(
+ "/crm/v3/objects/tickets",
+ "Tickets",
+ "In HubSpot, a ticket represents a customer request for help or support."
+ );
+ static final AllowableValue QUOTES = new AllowableValue(
+ "/crm/v3/objects/quotes",
+ "Quotes",
+ "In HubSpot, quotes are used to share pricing information with potential buyers."
+ );
+
+ // ENGAGEMENTS
+
+ private static final AllowableValue CALLS = new AllowableValue(
+ "/crm/v3/objects/calls",
+ "Calls",
+ "Get calls on CRM records and on the calls index page."
+ );
+ private static final AllowableValue EMAILS = new AllowableValue(
+ "/crm/v3/objects/emails",
+ "Emails",
+ "Get emails on CRM records."
+ );
+ private static final AllowableValue MEETINGS = new AllowableValue(
+ "/crm/v3/objects/meetings",
+ "Meetings",
+ "Get meetings on CRM records."
+ );
+ private static final AllowableValue NOTES = new AllowableValue(
+ "/crm/v3/objects/notes",
+ "Notes",
+ "Get notes on CRM records."
+ );
+ private static final AllowableValue TASKS = new AllowableValue(
+ "/crm/v3/objects/tasks",
+ "Tasks",
+ "Get tasks on CRM records."
+ );
+
+ // OTHER
+
+ private static final AllowableValue OWNERS = new AllowableValue(
+ "/crm/v3/owners/",
+ "Owners",
+ "HubSpot uses owners to assign specific users to contacts, companies, deals, tickets, or engagements. Any HubSpot user with access to contacts" +
+ " can be assigned as an owner, and multiple owners can be assigned to an object by creating a custom property for this purpose."
+ );
Review Comment:
I usually extract them if there are conditions on the allowable values.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org