You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by GitBox <gi...@apache.org> on 2022/10/10 18:03:17 UTC
[GitHub] [nifi] takraj opened a new pull request, #6504: NIFI-10618: Add Asana connector
takraj opened a new pull request, #6504:
URL: https://github.com/apache/nifi/pull/6504
<!-- Licensed to the Apache Software Foundation (ASF) under one or more -->
<!-- contributor license agreements. See the NOTICE file distributed with -->
<!-- this work for additional information regarding copyright ownership. -->
<!-- The ASF licenses this file to You under the Apache License, Version 2.0 -->
<!-- (the "License"); you may not use this file except in compliance with -->
<!-- the License. You may obtain a copy of the License at -->
<!-- http://www.apache.org/licenses/LICENSE-2.0 -->
<!-- Unless required by applicable law or agreed to in writing, software -->
<!-- distributed under the License is distributed on an "AS IS" BASIS, -->
<!-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -->
<!-- See the License for the specific language governing permissions and -->
<!-- limitations under the License. -->
# Summary
Connector to incrementally fetch data from Asana.
[NIFI-10618](https://issues.apache.org/jira/browse/NIFI-10618)
# Tracking
Please complete the following tracking steps prior to pull request creation.
### Issue Tracking
- [x] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created
### Pull Request Tracking
- [x] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-00000`
- [x] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-00000`
### Pull Request Formatting
- [x] Pull Request based on current revision of the `main` branch
- [x] Pull Request refers to a feature branch with one commit containing changes
# Verification
Please indicate the verification steps performed prior to pull request creation.
### Build
- [x] Build completed using `mvn clean install -P contrib-check`
- [x] JDK 8
- [ ] JDK 11
- [ ] JDK 17
### Licensing
- [x] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html)
- [x] New dependencies are documented in applicable `LICENSE` and `NOTICE` files
### Documentation
- [x] Documentation formatting appears as expected in rendered files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r1042719736
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,405 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import static java.lang.String.format;
+import static java.lang.String.join;
+import static java.util.Collections.singletonMap;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_EVENTS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_MEMBERS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_STATUS_ATTACHMENTS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_STATUS_UPDATES;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_STORIES;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_TASKS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_TASK_ATTACHMENTS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_TEAM_MEMBERS;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientProviderService;
+import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaObjectState;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+@TriggerSerially
+@PrimaryNodeOnly
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.")
+@Tags({"asana", "source", "ingest"})
+@CapabilityDescription("This processor collects data from Asana")
+public class GetAsanaObject extends AbstractProcessor {
+
+ protected static final String ASANA_GID = "asana.gid";
+ protected static final String ASANA_CLIENT_SERVICE = "asana-controller-service";
+ protected static final String DISTRIBUTED_CACHE_SERVICE = "distributed-cache-service";
+ protected static final String ASANA_OBJECT_TYPE = "asana-object-type";
+ protected static final String ASANA_PROJECT_NAME = "asana-project-name";
+ protected static final String ASANA_SECTION_NAME = "asana-section-name";
+ protected static final String ASANA_TAG_NAME = "asana-tag-name";
+ protected static final String ASANA_TEAM_NAME = "asana-team-name";
+ protected static final String ASANA_OUTPUT_BATCH_SIZE = "asana-output-batch-size";
+ protected static final String REL_NAME_NEW = "new";
+ protected static final String REL_NAME_UPDATED = "updated";
+ protected static final String REL_NAME_REMOVED = "removed";
+
+ protected static final PropertyDescriptor PROP_ASANA_CLIENT_SERVICE = new PropertyDescriptor.Builder()
+ .name(ASANA_CLIENT_SERVICE)
+ .displayName("Asana Client Service")
+ .description("Specify which controller service to use for accessing Asana.")
+ .required(true)
+ .identifiesControllerService(AsanaClientProviderService.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_DISTRIBUTED_CACHE_SERVICE = new Builder()
+ .name(DISTRIBUTED_CACHE_SERVICE)
+ .displayName("Distributed Cache Service")
+ .description("Cache service to store fetched item fingerprints. These, from the last successful query"
+ + " are stored, in order to enable incremental loading and change detection.")
+ .required(true)
+ .identifiesControllerService(DistributedMapCacheClient.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OBJECT_TYPE = new PropertyDescriptor.Builder()
+ .name(ASANA_OBJECT_TYPE)
+ .displayName("Object Type")
+ .description("Specify what kind of objects to be collected from Asana")
+ .required(true)
+ .allowableValues(AsanaObjectType.class)
+ .defaultValue(AV_COLLECT_TASKS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_PROJECT = new PropertyDescriptor.Builder()
+ .name(ASANA_PROJECT_NAME)
+ .displayName("Project Name")
+ .description("Fetch only objects in this project. Case sensitive.")
+ .required(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(
+ PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_PROJECT_MEMBERS,
+ AV_COLLECT_STORIES,
+ AV_COLLECT_PROJECT_STATUS_UPDATES,
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ AV_COLLECT_PROJECT_EVENTS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_SECTION = new PropertyDescriptor.Builder()
+ .name(ASANA_SECTION_NAME)
+ .displayName("Section Name")
+ .description("Fetch only objects in this section. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TAG = new PropertyDescriptor.Builder()
+ .name(ASANA_TAG_NAME)
+ .displayName("Tag")
+ .description("Fetch only objects having this tag. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TEAM_NAME = new PropertyDescriptor.Builder()
+ .name(ASANA_TEAM_NAME)
+ .displayName("Team")
+ .description("Team name. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TEAM_MEMBERS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OUTPUT_BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name(ASANA_OUTPUT_BATCH_SIZE)
+ .displayName("Output Batch Size")
+ .description("The number of items batched together in a single Flow File. If set to 1 (default), then each item is"
+ + " transferred in a separate Flow File and each will have an asana.gid attribute, to help identifying"
+ + " the fetched item on the server side, if needed. If the batch size is greater than 1, then the"
+ + " specified amount of items are batched together in a single Flow File as a Json array, and the"
+ + " Flow Files won't have the asana.gid attribute.")
+ .defaultValue("1")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+
+ protected static final List<PropertyDescriptor> DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ PROP_ASANA_CLIENT_SERVICE,
+ PROP_DISTRIBUTED_CACHE_SERVICE,
+ PROP_ASANA_OBJECT_TYPE,
+ PROP_ASANA_PROJECT,
+ PROP_ASANA_SECTION,
+ PROP_ASANA_TEAM_NAME,
+ PROP_ASANA_TAG,
+ PROP_ASANA_OUTPUT_BATCH_SIZE
+ ));
+
+ protected static final Relationship REL_NEW = new Relationship.Builder()
+ .name(REL_NAME_NEW)
+ .description("Newly collected objects are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_UPDATED = new Relationship.Builder()
+ .name(REL_NAME_UPDATED)
+ .description("Objects that have already been collected earlier, but were updated since, are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_REMOVED = new Relationship.Builder()
+ .name(REL_NAME_REMOVED)
+ .description("Notification about deleted objects are routed to this relationship. "
+ + "Flow files will not have any payload. IDs of the resources no longer exist "
+ + "are carried by the asana.gid attribute of the generated FlowFiles.")
+ .build();
+
+ protected static final Set<Relationship> RELATIONSHIPS = Collections.unmodifiableSet(new HashSet<>(Arrays.asList(
+ REL_NEW,
+ REL_UPDATED,
+ REL_REMOVED
+ )));
+ protected static final GenericObjectSerDe<String> STATE_MAP_KEY_SERIALIZER = new GenericObjectSerDe<>();
+ protected static final GenericObjectSerDe<Map<String, String>> STATE_MAP_VALUE_SERIALIZER = new GenericObjectSerDe<>();
+
+ private volatile AsanaObjectFetcher objectFetcher;
+ private volatile Integer batchSize;
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return RELATIONSHIPS;
+ }
+
+ @Override
+ public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return DESCRIPTORS;
+ }
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) throws InitializationException {
+ AsanaClientProviderService controllerService = context.getProperty(PROP_ASANA_CLIENT_SERVICE).asControllerService(AsanaClientProviderService.class);
+ AsanaClient client = controllerService.createClient();
+ batchSize = context.getProperty(PROP_ASANA_OUTPUT_BATCH_SIZE).asInteger();
+
+ try {
+ getLogger().debug("Initializing object fetcher...");
+ objectFetcher = createObjectFetcher(context, client);
+ } catch (Exception e) {
+ throw new InitializationException(e);
+ }
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ try {
+ Map<String, String> state = recoverState(context).orElse(Collections.emptyMap());
+ getLogger().debug("Attempting to load state: {}", state);
+ objectFetcher.loadState(state);
Review Comment:
Done in e8f6176
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r995754501
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientServiceApi;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@TriggerSerially
+@Stateful(scopes = {Scope.LOCAL}, description = "Fingerprints of items in the last successful query are stored in order to enable incremental loading and change detection.")
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.")
+@Tags({"asana", "source", "connector", "ingest"})
+@CapabilityDescription("This processor collects data from Asana")
+public class GetAsanaObject extends AbstractProcessor {
+
+ protected static final String ASANA_GID = "asana.gid";
+ protected static final String AV_NAME_COLLECT_TASKS = "asana-collect-tasks";
+ protected static final String AV_NAME_COLLECT_TASK_ATTACHMENTS = "asana-collect-task-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECTS = "asana-collect-projects";
+ protected static final String AV_NAME_COLLECT_TAGS = "asana-collect-tags";
+ protected static final String AV_NAME_COLLECT_USERS = "asana-collect-users";
+ protected static final String AV_NAME_COLLECT_PROJECT_MEMBERS = "asana-collect-project-members";
+ protected static final String AV_NAME_COLLECT_TEAMS = "asana-collect-teams";
+ protected static final String AV_NAME_COLLECT_TEAM_MEMBERS = "asana-collect-team-members";
+ protected static final String AV_NAME_COLLECT_STORIES = "asana-collect-stories";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_UPDATES = "asana-collect-project-status-updates";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS = "asana-collect-project-status-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECT_EVENTS = "asana-collect-project-events";
+ protected static final String ASANA_CONTROLLER_SERVICE = "asana-controller-service";
+ protected static final String ASANA_OBJECT_TYPE = "asana-object-type";
+ protected static final String ASANA_PROJECT_NAME = "asana-project-name";
+ protected static final String ASANA_SECTION_NAME = "asana-section-name";
+ protected static final String ASANA_TAG_NAME = "asana-tag-name";
+ protected static final String ASANA_TEAM_NAME = "asana-team-name";
+ protected static final String ASANA_OUTPUT_BATCH_SIZE = "asana-output-batch-size";
+ protected static final String REL_NAME_NEW = "new";
+ protected static final String REL_NAME_UPDATED = "updated";
+ protected static final String REL_NAME_REMOVED = "removed";
+
+ protected static final AllowableValue AV_COLLECT_TASKS = new AllowableValue(
+ AV_NAME_COLLECT_TASKS,
+ "Tasks",
+ "Collect tasks matching to the specified conditions.");
+
+ protected static final AllowableValue AV_COLLECT_TASK_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_TASK_ATTACHMENTS,
+ "Task attachments",
+ "Collect attached files of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECTS,
+ "Projects",
+ "Collect projects of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TAGS = new AllowableValue(
+ AV_NAME_COLLECT_TAGS,
+ "Tags",
+ "Collect tags of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_USERS = new AllowableValue(
+ AV_NAME_COLLECT_USERS,
+ "Users",
+ "Collect users assigned to the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_MEMBERS,
+ "Members of a project",
+ "Collect users assigned to the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAMS = new AllowableValue(
+ AV_NAME_COLLECT_TEAMS,
+ "Teams",
+ "Collect teams of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAM_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_TEAM_MEMBERS,
+ "Team members",
+ "Collect users assigned to the specified team."
+ );
+
+ protected static final AllowableValue AV_COLLECT_STORIES = new AllowableValue(
+ AV_NAME_COLLECT_STORIES,
+ "Stories of tasks",
+ "Collect stories (comments) of of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_UPDATES = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_UPDATES,
+ "Status updates of a project",
+ "Collect status updates of the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ "Attachments of status updates",
+ "Collect attached files of project status updates."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_EVENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_EVENTS,
+ "Events of a project",
+ "Collect various events happening on the specified project and on its' tasks."
+ );
+
+ protected static final PropertyDescriptor PROP_ASANA_CONTROLLER_SERVICE = new PropertyDescriptor.Builder()
+ .name(ASANA_CONTROLLER_SERVICE)
+ .displayName("Controller service")
Review Comment:
Done in [dc2a050](https://github.com/apache/nifi/pull/6504/commits/dc2a050548a389d89181670371748af741eefd61)
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientServiceApi;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@TriggerSerially
+@Stateful(scopes = {Scope.LOCAL}, description = "Fingerprints of items in the last successful query are stored in order to enable incremental loading and change detection.")
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.")
+@Tags({"asana", "source", "connector", "ingest"})
+@CapabilityDescription("This processor collects data from Asana")
+public class GetAsanaObject extends AbstractProcessor {
+
+ protected static final String ASANA_GID = "asana.gid";
+ protected static final String AV_NAME_COLLECT_TASKS = "asana-collect-tasks";
+ protected static final String AV_NAME_COLLECT_TASK_ATTACHMENTS = "asana-collect-task-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECTS = "asana-collect-projects";
+ protected static final String AV_NAME_COLLECT_TAGS = "asana-collect-tags";
+ protected static final String AV_NAME_COLLECT_USERS = "asana-collect-users";
+ protected static final String AV_NAME_COLLECT_PROJECT_MEMBERS = "asana-collect-project-members";
+ protected static final String AV_NAME_COLLECT_TEAMS = "asana-collect-teams";
+ protected static final String AV_NAME_COLLECT_TEAM_MEMBERS = "asana-collect-team-members";
+ protected static final String AV_NAME_COLLECT_STORIES = "asana-collect-stories";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_UPDATES = "asana-collect-project-status-updates";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS = "asana-collect-project-status-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECT_EVENTS = "asana-collect-project-events";
+ protected static final String ASANA_CONTROLLER_SERVICE = "asana-controller-service";
+ protected static final String ASANA_OBJECT_TYPE = "asana-object-type";
+ protected static final String ASANA_PROJECT_NAME = "asana-project-name";
+ protected static final String ASANA_SECTION_NAME = "asana-section-name";
+ protected static final String ASANA_TAG_NAME = "asana-tag-name";
+ protected static final String ASANA_TEAM_NAME = "asana-team-name";
+ protected static final String ASANA_OUTPUT_BATCH_SIZE = "asana-output-batch-size";
+ protected static final String REL_NAME_NEW = "new";
+ protected static final String REL_NAME_UPDATED = "updated";
+ protected static final String REL_NAME_REMOVED = "removed";
+
+ protected static final AllowableValue AV_COLLECT_TASKS = new AllowableValue(
+ AV_NAME_COLLECT_TASKS,
+ "Tasks",
+ "Collect tasks matching to the specified conditions.");
+
+ protected static final AllowableValue AV_COLLECT_TASK_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_TASK_ATTACHMENTS,
+ "Task attachments",
+ "Collect attached files of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECTS,
+ "Projects",
+ "Collect projects of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TAGS = new AllowableValue(
+ AV_NAME_COLLECT_TAGS,
+ "Tags",
+ "Collect tags of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_USERS = new AllowableValue(
+ AV_NAME_COLLECT_USERS,
+ "Users",
+ "Collect users assigned to the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_MEMBERS,
+ "Members of a project",
+ "Collect users assigned to the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAMS = new AllowableValue(
+ AV_NAME_COLLECT_TEAMS,
+ "Teams",
+ "Collect teams of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAM_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_TEAM_MEMBERS,
+ "Team members",
+ "Collect users assigned to the specified team."
+ );
+
+ protected static final AllowableValue AV_COLLECT_STORIES = new AllowableValue(
+ AV_NAME_COLLECT_STORIES,
+ "Stories of tasks",
+ "Collect stories (comments) of of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_UPDATES = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_UPDATES,
+ "Status updates of a project",
+ "Collect status updates of the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ "Attachments of status updates",
+ "Collect attached files of project status updates."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_EVENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_EVENTS,
+ "Events of a project",
+ "Collect various events happening on the specified project and on its' tasks."
+ );
+
+ protected static final PropertyDescriptor PROP_ASANA_CONTROLLER_SERVICE = new PropertyDescriptor.Builder()
+ .name(ASANA_CONTROLLER_SERVICE)
+ .displayName("Controller service")
+ .description("Specify which controller service to use for accessing Asana.")
+ .required(true)
+ .identifiesControllerService(AsanaClientServiceApi.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OBJECT_TYPE = new PropertyDescriptor.Builder()
+ .name(ASANA_OBJECT_TYPE)
+ .displayName("Object type to be collected")
Review Comment:
Done in [dc2a050](https://github.com/apache/nifi/pull/6504/commits/dc2a050548a389d89181670371748af741eefd61)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r1037448185
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,393 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import static java.lang.String.format;
+import static java.util.Collections.singletonMap;
+import static java.util.stream.Collectors.groupingBy;
+import static java.util.stream.Collectors.joining;
+import static org.apache.commons.collections4.ListUtils.partition;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_EVENTS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_MEMBERS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_STATUS_ATTACHMENTS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_STATUS_UPDATES;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_STORIES;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_TASKS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_TASK_ATTACHMENTS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_TEAM_MEMBERS;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientProviderService;
+import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaObjectState;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+@TriggerSerially
+@PrimaryNodeOnly
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.")
+@Tags({"asana", "source", "ingest"})
+@CapabilityDescription("This processor collects data from Asana")
+public class GetAsanaObject extends AbstractProcessor {
+
+ protected static final String ASANA_GID = "asana.gid";
+ protected static final String ASANA_CLIENT_SERVICE = "asana-controller-service";
+ protected static final String DISTRIBUTED_CACHE_SERVICE = "distributed-cache-service";
+ protected static final String ASANA_OBJECT_TYPE = "asana-object-type";
+ protected static final String ASANA_PROJECT_NAME = "asana-project-name";
+ protected static final String ASANA_SECTION_NAME = "asana-section-name";
+ protected static final String ASANA_TAG_NAME = "asana-tag-name";
+ protected static final String ASANA_TEAM_NAME = "asana-team-name";
+ protected static final String ASANA_OUTPUT_BATCH_SIZE = "asana-output-batch-size";
+ protected static final String REL_NAME_NEW = "new";
+ protected static final String REL_NAME_UPDATED = "updated";
+ protected static final String REL_NAME_REMOVED = "removed";
+
+ protected static final PropertyDescriptor PROP_ASANA_CLIENT_SERVICE = new PropertyDescriptor.Builder()
+ .name(ASANA_CLIENT_SERVICE)
+ .displayName("Asana Client Service")
+ .description("Specify which controller service to use for accessing Asana.")
+ .required(true)
+ .identifiesControllerService(AsanaClientProviderService.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_DISTRIBUTED_CACHE_SERVICE = new Builder()
+ .name(DISTRIBUTED_CACHE_SERVICE)
+ .displayName("Distributed Cache Service")
+ .description("Cache service to store fetched item fingerprints. These, from the last successful query"
+ + " are stored, in order to enable incremental loading and change detection.")
+ .required(true)
+ .identifiesControllerService(DistributedMapCacheClient.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OBJECT_TYPE = new PropertyDescriptor.Builder()
+ .name(ASANA_OBJECT_TYPE)
+ .displayName("Object Type")
+ .description("Specify what kind of objects to be collected from Asana")
+ .required(true)
+ .allowableValues(AsanaObjectType.class)
+ .defaultValue(AV_COLLECT_TASKS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_PROJECT = new PropertyDescriptor.Builder()
+ .name(ASANA_PROJECT_NAME)
+ .displayName("Project Name")
+ .description("Fetch only objects in this project. Case sensitive.")
+ .required(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(
+ PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS.getValue(),
+ AV_COLLECT_TASK_ATTACHMENTS.getValue(),
+ AV_COLLECT_PROJECT_MEMBERS.getValue(),
+ AV_COLLECT_STORIES.getValue(),
+ AV_COLLECT_PROJECT_STATUS_UPDATES.getValue(),
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS.getValue(),
+ AV_COLLECT_PROJECT_EVENTS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_SECTION = new PropertyDescriptor.Builder()
+ .name(ASANA_SECTION_NAME)
+ .displayName("Section Name")
+ .description("Fetch only objects in this section. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS.getValue(),
+ AV_COLLECT_TASK_ATTACHMENTS.getValue(),
+ AV_COLLECT_STORIES.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TAG = new PropertyDescriptor.Builder()
+ .name(ASANA_TAG_NAME)
+ .displayName("Tag")
+ .description("Fetch only objects having this tag. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS.getValue(),
+ AV_COLLECT_TASK_ATTACHMENTS.getValue(),
+ AV_COLLECT_STORIES.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TEAM_NAME = new PropertyDescriptor.Builder()
+ .name(ASANA_TEAM_NAME)
+ .displayName("Team")
+ .description("Team name. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TEAM_MEMBERS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OUTPUT_BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name(ASANA_OUTPUT_BATCH_SIZE)
+ .displayName("Output Batch Size")
+ .description("The number of items batched together in a single Flow File. If set to 1 (default), then each item is"
+ + " transferred in a separate Flow File and each will have an asana.gid attribute, to help identifying"
+ + " the fetched item on the server side, if needed. If the batch size is greater than 1, then the"
+ + " specified amount of items are batched together in a single Flow File as a Json array, and the"
+ + " Flow Files won't have the asana.gid attribute.")
+ .defaultValue("1")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+
+ protected static final List<PropertyDescriptor> DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ PROP_ASANA_CLIENT_SERVICE,
+ PROP_DISTRIBUTED_CACHE_SERVICE,
+ PROP_ASANA_OBJECT_TYPE,
+ PROP_ASANA_PROJECT,
+ PROP_ASANA_SECTION,
+ PROP_ASANA_TEAM_NAME,
+ PROP_ASANA_TAG,
+ PROP_ASANA_OUTPUT_BATCH_SIZE
+ ));
+
+ protected static final Relationship REL_NEW = new Relationship.Builder()
+ .name(REL_NAME_NEW)
+ .description("Newly collected objects are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_UPDATED = new Relationship.Builder()
+ .name(REL_NAME_UPDATED)
+ .description("Objects that have already been collected earlier, but were updated since, are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_REMOVED = new Relationship.Builder()
+ .name(REL_NAME_REMOVED)
+ .description("Notification about deleted objects are routed to this relationship. "
+ + "Flow files will not have any payload. IDs of the resources no longer exist "
+ + "are carried by the asana.gid attribute of the generated FlowFiles.")
+ .build();
+
+ protected static final Set<Relationship> RELATIONSHIPS = Collections.unmodifiableSet(new HashSet<>(Arrays.asList(
+ REL_NEW,
+ REL_UPDATED,
+ REL_REMOVED
+ )));
+
+ private volatile AsanaObjectFetcher objectFetcher;
+ private volatile Integer batchSize;
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return RELATIONSHIPS;
+ }
+
+ @Override
+ public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return DESCRIPTORS;
+ }
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) throws InitializationException {
+ AsanaClientProviderService controllerService = context.getProperty(PROP_ASANA_CLIENT_SERVICE).asControllerService(AsanaClientProviderService.class);
+ AsanaClient client = controllerService.createClient();
+ batchSize = context.getProperty(PROP_ASANA_OUTPUT_BATCH_SIZE).asInteger();
+
+ try {
+ getLogger().debug("Initializing object fetcher...");
+ objectFetcher = createObjectFetcher(context, client);
+ } catch (Exception e) {
+ throw new InitializationException(e);
+ }
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ try {
+ Map<String, String> state = recoverState(context).orElse(Collections.emptyMap());
+ getLogger().debug("Attempting to load state: {}", state);
+ objectFetcher.loadState(state);
+ } catch (Exception e) {
+ getLogger().info("Failed to recover state. Falling back to clean start.");
+ objectFetcher.clearState();
+ }
+ getLogger().debug("Initial state: {}", objectFetcher.saveState());
+
+ Collection<FlowFile> newItems = new ArrayList<>();
+ Collection<FlowFile> updatedItems = new ArrayList<>();
+ Collection<FlowFile> removedItems = new ArrayList<>();
+ Map<AsanaObjectState, Collection<FlowFile>> flowFiles = new HashMap<>();
+ flowFiles.put(AsanaObjectState.NEW, newItems);
+ flowFiles.put(AsanaObjectState.UPDATED, updatedItems);
+ flowFiles.put(AsanaObjectState.REMOVED, removedItems);
+
+ List<AsanaObject> allObjects = new ArrayList<>();
+
+ AsanaObject nextObject;
+ while ((nextObject = objectFetcher.fetchNext()) != null) {
+ allObjects.add(nextObject);
Review Comment:
@turcsanyip Implemented in cc0997d.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r1031398934
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-services/pom.xml:
##########
@@ -0,0 +1,64 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-asana-bundle</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-asana-services</artifactId>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-api</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ </dependency>
Review Comment:
Done in e96631a
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r1014225968
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,429 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientProviderService;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@TriggerSerially
+@PrimaryNodeOnly
+@Stateful(scopes = {Scope.CLUSTER}, description = "Fingerprints of items in the last successful query are stored in order to enable incremental loading and change detection.")
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.")
+@Tags({"asana", "source", "ingest"})
+@CapabilityDescription("This processor collects data from Asana")
+public class GetAsanaObject extends AbstractProcessor {
+
+ protected static final String ASANA_GID = "asana.gid";
+ protected static final String AV_NAME_COLLECT_TASKS = "asana-collect-tasks";
+ protected static final String AV_NAME_COLLECT_TASK_ATTACHMENTS = "asana-collect-task-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECTS = "asana-collect-projects";
+ protected static final String AV_NAME_COLLECT_TAGS = "asana-collect-tags";
+ protected static final String AV_NAME_COLLECT_USERS = "asana-collect-users";
+ protected static final String AV_NAME_COLLECT_PROJECT_MEMBERS = "asana-collect-project-members";
+ protected static final String AV_NAME_COLLECT_TEAMS = "asana-collect-teams";
+ protected static final String AV_NAME_COLLECT_TEAM_MEMBERS = "asana-collect-team-members";
+ protected static final String AV_NAME_COLLECT_STORIES = "asana-collect-stories";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_UPDATES = "asana-collect-project-status-updates";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS = "asana-collect-project-status-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECT_EVENTS = "asana-collect-project-events";
+ protected static final String ASANA_CONTROLLER_SERVICE = "asana-controller-service";
+ protected static final String ASANA_OBJECT_TYPE = "asana-object-type";
+ protected static final String ASANA_PROJECT_NAME = "asana-project-name";
+ protected static final String ASANA_SECTION_NAME = "asana-section-name";
+ protected static final String ASANA_TAG_NAME = "asana-tag-name";
+ protected static final String ASANA_TEAM_NAME = "asana-team-name";
+ protected static final String ASANA_OUTPUT_BATCH_SIZE = "asana-output-batch-size";
+ protected static final String REL_NAME_NEW = "new";
+ protected static final String REL_NAME_UPDATED = "updated";
+ protected static final String REL_NAME_REMOVED = "removed";
+
+ protected static final AllowableValue AV_COLLECT_TASKS = new AllowableValue(
+ AV_NAME_COLLECT_TASKS,
+ "Tasks",
+ "Collect tasks matching to the specified conditions.");
+
+ protected static final AllowableValue AV_COLLECT_TASK_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_TASK_ATTACHMENTS,
+ "Task Attachments",
+ "Collect attached files of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECTS,
+ "Projects",
+ "Collect projects of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TAGS = new AllowableValue(
+ AV_NAME_COLLECT_TAGS,
+ "Tags",
+ "Collect tags of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_USERS = new AllowableValue(
+ AV_NAME_COLLECT_USERS,
+ "Users",
+ "Collect users assigned to the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_MEMBERS,
+ "Members of a Project",
+ "Collect users assigned to the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAMS = new AllowableValue(
+ AV_NAME_COLLECT_TEAMS,
+ "Teams",
+ "Collect teams of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAM_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_TEAM_MEMBERS,
+ "Team Members",
+ "Collect users assigned to the specified team."
+ );
+
+ protected static final AllowableValue AV_COLLECT_STORIES = new AllowableValue(
+ AV_NAME_COLLECT_STORIES,
+ "Stories of Tasks",
+ "Collect stories (comments) of of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_UPDATES = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_UPDATES,
+ "Status Updates of a Project",
+ "Collect status updates of the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ "Attachments of Status Updates",
+ "Collect attached files of project status updates."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_EVENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_EVENTS,
+ "Events of a Project",
+ "Collect various events happening on the specified project and on its' tasks."
+ );
+
+ protected static final PropertyDescriptor PROP_ASANA_CONTROLLER_SERVICE = new PropertyDescriptor.Builder()
+ .name(ASANA_CONTROLLER_SERVICE)
+ .displayName("Asana Controller Service")
+ .description("Specify which controller service to use for accessing Asana.")
+ .required(true)
+ .identifiesControllerService(AsanaClientProviderService.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OBJECT_TYPE = new PropertyDescriptor.Builder()
+ .name(ASANA_OBJECT_TYPE)
+ .displayName("Object Type")
+ .description("Specify what kind of objects to be collected from Asana")
+ .required(true)
+ .allowableValues(
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_PROJECTS,
+ AV_COLLECT_TAGS,
+ AV_COLLECT_USERS,
+ AV_COLLECT_PROJECT_MEMBERS,
+ AV_COLLECT_TEAMS,
+ AV_COLLECT_TEAM_MEMBERS,
+ AV_COLLECT_STORIES,
+ AV_COLLECT_PROJECT_STATUS_UPDATES,
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ AV_COLLECT_PROJECT_EVENTS)
+ .defaultValue(AV_COLLECT_TASKS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_PROJECT = new PropertyDescriptor.Builder()
+ .name(ASANA_PROJECT_NAME)
+ .displayName("Project Name")
+ .description("Fetch only objects in this project. Case sensitive.")
+ .required(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(
+ PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_PROJECT_MEMBERS,
+ AV_COLLECT_STORIES,
+ AV_COLLECT_PROJECT_STATUS_UPDATES,
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ AV_COLLECT_PROJECT_EVENTS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_SECTION = new PropertyDescriptor.Builder()
+ .name(ASANA_SECTION_NAME)
+ .displayName("Section Name")
+ .description("Fetch only objects in this section. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TASKS, AV_COLLECT_TASK_ATTACHMENTS, AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TAG = new PropertyDescriptor.Builder()
+ .name(ASANA_TAG_NAME)
+ .displayName("Tag")
+ .description("Fetch only objects having this tag. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TASKS, AV_COLLECT_TASK_ATTACHMENTS, AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TEAM_NAME = new PropertyDescriptor.Builder()
+ .name(ASANA_TEAM_NAME)
+ .displayName("Team")
+ .description("Team name. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TEAM_MEMBERS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OUTPUT_BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name(ASANA_OUTPUT_BATCH_SIZE)
+ .displayName("Output Batch Size")
+ .description("The number of output FlowFiles to queue before committing the process session. When set to zero, the session will be committed when all result set rows "
+ + "have been processed and the output FlowFiles are ready for transfer to the downstream relationship. For large result sets, this can cause a large burst of FlowFiles "
+ + "to be transferred at the end of processor execution. If this property is set, then when the specified number of FlowFiles are ready for transfer, then the session will "
+ + "be committed, thus releasing the FlowFiles to the downstream relationship.")
+ .defaultValue("0")
+ .required(true)
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+
+ protected static final List<PropertyDescriptor> DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ PROP_ASANA_CONTROLLER_SERVICE,
+ PROP_ASANA_OBJECT_TYPE,
+ PROP_ASANA_PROJECT,
+ PROP_ASANA_SECTION,
+ PROP_ASANA_TEAM_NAME,
+ PROP_ASANA_TAG,
+ PROP_ASANA_OUTPUT_BATCH_SIZE
+ ));
+
+ protected static final Relationship REL_NEW = new Relationship.Builder()
+ .name(REL_NAME_NEW)
+ .description("Newly collected objects are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_UPDATED = new Relationship.Builder()
+ .name(REL_NAME_UPDATED)
+ .description("Objects that have already been collected earlier, but were updated since, are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_REMOVED = new Relationship.Builder()
+ .name(REL_NAME_REMOVED)
+ .description("Notification about deleted objects are routed to this relationship. "
+ + "Flow files will not have any payload. IDs of the resources no longer exist "
+ + "are carried by the asana.gid attribute of the generated FlowFiles.")
+ .build();
+
+ protected static final Set<Relationship> RELATIONSHIPS = Collections.unmodifiableSet(new HashSet<>(Arrays.asList(
+ REL_NEW,
+ REL_UPDATED,
+ REL_REMOVED
+ )));
+
+ private static final Scope STATE_STORAGE_SCOPE = Scope.CLUSTER;
+
+ private volatile AsanaObjectFetcher objectFetcher;
+ private volatile Integer batchSize;
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return RELATIONSHIPS;
+ }
+
+ @Override
+ public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return DESCRIPTORS;
+ }
+
+ @OnScheduled
+ public synchronized void onScheduled(final ProcessContext context) throws InitializationException {
+ AsanaClientProviderService controllerService = context.getProperty(PROP_ASANA_CONTROLLER_SERVICE).asControllerService(AsanaClientProviderService.class);
+ AsanaClient client = controllerService.createClient();
+ batchSize = context.getProperty(PROP_ASANA_OUTPUT_BATCH_SIZE).asInteger();
+
+ try {
+ getLogger().debug("Initializing object fetcher...");
+ objectFetcher = createObjectFetcher(context, client);
+ } catch (Exception e) {
+ throw new InitializationException(e);
+ }
+ }
+
+ @Override
+ public synchronized void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ try {
+ Map<String, String> state = recoverState(context).orElse(Collections.emptyMap());
+ getLogger().debug("Attempting to load state: {}", state);
+ objectFetcher.loadState(state);
+ } catch (Exception e) {
+ getLogger().info("Failed to recover state. Falling back to clean start.");
+ objectFetcher.clearState();
+ }
+ getLogger().debug("Initial state: {}", objectFetcher.saveState());
+
+ Collection<FlowFile> newItems = new ArrayList<>();
+ Collection<FlowFile> updatedItems = new ArrayList<>();
+ Collection<FlowFile> removedItems = new ArrayList<>();
+
+ AsanaObject nextObject;
+ while ((batchSize == 0 || (batchSize > (newItems.size() + updatedItems.size() + removedItems.size()))) && (nextObject = objectFetcher.fetchNext()) != null) {
+ final Map<String, String> attributes = new HashMap<>(2);
+ attributes.put(CoreAttributes.MIME_TYPE.key(), ContentType.APPLICATION_JSON.getMimeType());
+ attributes.put(ASANA_GID, nextObject.getGid());
+ FlowFile flowFile = createFlowFileWithStringPayload(session, nextObject.getContent());
Review Comment:
Thanks for pointing this out. I'll attempt to change the behavior to batch multiple items together in a single FlowFile. The max amount of items in a single batch will be controllable via Batch size attribute.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r1035778300
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-services-api/src/main/java/org/apache/nifi/controller/asana/AsanaClient.java:
##########
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.controller.asana;
+
+import com.asana.models.Attachment;
+import com.asana.models.Project;
+import com.asana.models.ProjectMembership;
+import com.asana.models.ProjectStatus;
+import com.asana.models.Section;
+import com.asana.models.Story;
+import com.asana.models.Tag;
+import com.asana.models.Task;
+import com.asana.models.Team;
+import com.asana.models.User;
+
+import java.util.Map;
+
+/**
+ * This interface represents a client to Asana REST server, with some basic filtering options built in.
+ */
+public interface AsanaClient {
+ /**
+ * Find & retrieve an Asana project by its name. If multiple projects match, returns the first.
+ * If there is no match, then {@link RuntimeException} is thrown. Note that constant ordering
Review Comment:
Done in 6c03215
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
exceptionfactory commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r1012362369
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-services/src/main/java/org/apache/nifi/controller/asana/StandardAsanaClient.java:
##########
@@ -0,0 +1,303 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.controller.asana;
+
+import com.asana.Client;
+import com.asana.errors.InvalidTokenError;
+import com.asana.models.Attachment;
+import com.asana.models.Event;
+import com.asana.models.Project;
+import com.asana.models.ProjectMembership;
+import com.asana.models.ProjectStatus;
+import com.asana.models.Resource;
+import com.asana.models.ResultBodyCollection;
+import com.asana.models.Section;
+import com.asana.models.Story;
+import com.asana.models.Tag;
+import com.asana.models.Task;
+import com.asana.models.Team;
+import com.asana.models.User;
+import com.asana.models.Workspace;
+import com.asana.requests.CollectionRequest;
+import com.asana.requests.EventsRequest;
+import com.google.gson.annotations.SerializedName;
+
+import java.io.IOException;
+import java.lang.reflect.Field;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.StreamSupport;
+
+public class StandardAsanaClient implements AsanaClient {
+
+ static final String ASANA_CLIENT_OPTION_BASE_URL = "base_url";
+
+ private final Client client;
+ private final Workspace workspace;
+
+ public StandardAsanaClient(String personalAccessToken, String workspaceName, String baseUrl) {
+ client = Client.accessToken(personalAccessToken);
+ if (baseUrl != null) {
+ client.options.put(ASANA_CLIENT_OPTION_BASE_URL, baseUrl);
+ }
+ workspace = getWorkspaceByName(workspaceName);
+ }
+
+ @Override
+ public Project getProjectByName(String projectName) {
+ return getProjects()
+ .values()
+ .stream()
+ .filter(p -> p.name.equals(projectName))
+ .findFirst()
+ .orElseThrow(() -> new RuntimeException("No such project: " + projectName));
+ }
+
+ @Override
+ public Map<String, Project> getProjects() {
+ try {
+ return collectionRequestToMap(
+ client.projects.getProjects(null, null, workspace.gid, null, null, getSerializedFieldNames(Project.class), false)
+ );
+ } catch (IOException e) {
+ throw new RuntimeException(e);
Review Comment:
`RuntimeException` should be replaced with `UncheckedIOException` to provide a better representation of the wrapped exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r1031544762
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-services-api-nar/src/main/resources/META-INF/NOTICE:
##########
@@ -0,0 +1,51 @@
+nifi-asana-nar
+Copyright 2015-2022 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+===========================================
+Apache Software License v2
+===========================================
+
+The following binary components are provided under the Apache Software License v2
+
+ (ASLv2) Apache Commons IO
+ The following NOTICE information applies:
+ Apache Commons IO
+ Copyright 2002-2017 The Apache Software Foundation
+
+ (ASLv2) Jackson JSON processor
+ The following NOTICE information applies:
+ # Jackson JSON processor
+
+ Jackson is a high-performance, Free/Open Source JSON processing library.
+ It was originally written by Tatu Saloranta (tatu.saloranta@iki.fi), and has
+ been in development since 2007.
+ It is currently developed by a community of developers, as well as supported
+ commercially by FasterXML.com.
+
+ ## Licensing
+
+ Jackson core and extension components may licensed under different licenses.
+ To find the details that apply to this artifact see the accompanying LICENSE file.
+ For more information, including possible other licensing options, contact
+ FasterXML.com (http://fasterxml.com).
+
+ ## Credits
+
+ A list of contributors may be found from CREDITS file, which is included
+ in some artifacts (usually source distributions); but is always available
+ from the source code management (SCM) system project uses.
+
+
+===========================================
+The MIT License
+===========================================
+
+The following binary components are provided under the MIT License
+
+ (MIT License) Java client library for the Asana API
+ The following NOTICE information applies:
+ Asana
+ Copyright (c) 2015
Review Comment:
Added in 06fe489
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r995756472
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientServiceApi;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@TriggerSerially
+@Stateful(scopes = {Scope.LOCAL}, description = "Fingerprints of items in the last successful query are stored in order to enable incremental loading and change detection.")
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.")
+@Tags({"asana", "source", "connector", "ingest"})
+@CapabilityDescription("This processor collects data from Asana")
+public class GetAsanaObject extends AbstractProcessor {
+
+ protected static final String ASANA_GID = "asana.gid";
+ protected static final String AV_NAME_COLLECT_TASKS = "asana-collect-tasks";
+ protected static final String AV_NAME_COLLECT_TASK_ATTACHMENTS = "asana-collect-task-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECTS = "asana-collect-projects";
+ protected static final String AV_NAME_COLLECT_TAGS = "asana-collect-tags";
+ protected static final String AV_NAME_COLLECT_USERS = "asana-collect-users";
+ protected static final String AV_NAME_COLLECT_PROJECT_MEMBERS = "asana-collect-project-members";
+ protected static final String AV_NAME_COLLECT_TEAMS = "asana-collect-teams";
+ protected static final String AV_NAME_COLLECT_TEAM_MEMBERS = "asana-collect-team-members";
+ protected static final String AV_NAME_COLLECT_STORIES = "asana-collect-stories";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_UPDATES = "asana-collect-project-status-updates";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS = "asana-collect-project-status-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECT_EVENTS = "asana-collect-project-events";
+ protected static final String ASANA_CONTROLLER_SERVICE = "asana-controller-service";
+ protected static final String ASANA_OBJECT_TYPE = "asana-object-type";
+ protected static final String ASANA_PROJECT_NAME = "asana-project-name";
+ protected static final String ASANA_SECTION_NAME = "asana-section-name";
+ protected static final String ASANA_TAG_NAME = "asana-tag-name";
+ protected static final String ASANA_TEAM_NAME = "asana-team-name";
+ protected static final String ASANA_OUTPUT_BATCH_SIZE = "asana-output-batch-size";
+ protected static final String REL_NAME_NEW = "new";
+ protected static final String REL_NAME_UPDATED = "updated";
+ protected static final String REL_NAME_REMOVED = "removed";
+
+ protected static final AllowableValue AV_COLLECT_TASKS = new AllowableValue(
+ AV_NAME_COLLECT_TASKS,
+ "Tasks",
+ "Collect tasks matching to the specified conditions.");
+
+ protected static final AllowableValue AV_COLLECT_TASK_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_TASK_ATTACHMENTS,
+ "Task attachments",
+ "Collect attached files of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECTS,
+ "Projects",
+ "Collect projects of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TAGS = new AllowableValue(
+ AV_NAME_COLLECT_TAGS,
+ "Tags",
+ "Collect tags of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_USERS = new AllowableValue(
+ AV_NAME_COLLECT_USERS,
+ "Users",
+ "Collect users assigned to the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_MEMBERS,
+ "Members of a project",
+ "Collect users assigned to the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAMS = new AllowableValue(
+ AV_NAME_COLLECT_TEAMS,
+ "Teams",
+ "Collect teams of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAM_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_TEAM_MEMBERS,
+ "Team members",
+ "Collect users assigned to the specified team."
+ );
+
+ protected static final AllowableValue AV_COLLECT_STORIES = new AllowableValue(
+ AV_NAME_COLLECT_STORIES,
+ "Stories of tasks",
+ "Collect stories (comments) of of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_UPDATES = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_UPDATES,
+ "Status updates of a project",
+ "Collect status updates of the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ "Attachments of status updates",
+ "Collect attached files of project status updates."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_EVENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_EVENTS,
+ "Events of a project",
+ "Collect various events happening on the specified project and on its' tasks."
+ );
+
+ protected static final PropertyDescriptor PROP_ASANA_CONTROLLER_SERVICE = new PropertyDescriptor.Builder()
+ .name(ASANA_CONTROLLER_SERVICE)
+ .displayName("Controller service")
+ .description("Specify which controller service to use for accessing Asana.")
+ .required(true)
+ .identifiesControllerService(AsanaClientServiceApi.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OBJECT_TYPE = new PropertyDescriptor.Builder()
+ .name(ASANA_OBJECT_TYPE)
+ .displayName("Object type to be collected")
+ .description("Specify what kind of objects to be collected from Asana")
+ .required(true)
+ .allowableValues(
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_PROJECTS,
+ AV_COLLECT_TAGS,
+ AV_COLLECT_USERS,
+ AV_COLLECT_PROJECT_MEMBERS,
+ AV_COLLECT_TEAMS,
+ AV_COLLECT_TEAM_MEMBERS,
+ AV_COLLECT_STORIES,
+ AV_COLLECT_PROJECT_STATUS_UPDATES,
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ AV_COLLECT_PROJECT_EVENTS)
+ .defaultValue(AV_COLLECT_TASKS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_PROJECT = new PropertyDescriptor.Builder()
+ .name(ASANA_PROJECT_NAME)
+ .displayName("Project name")
+ .description("Fetch only objects in this project. Case sensitive.")
+ .required(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(
+ PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_PROJECT_MEMBERS,
+ AV_COLLECT_STORIES,
+ AV_COLLECT_PROJECT_STATUS_UPDATES,
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ AV_COLLECT_PROJECT_EVENTS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_SECTION = new PropertyDescriptor.Builder()
+ .name(ASANA_SECTION_NAME)
+ .displayName("Section name")
+ .description("Fetch only objects in this section. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TASKS, AV_COLLECT_TASK_ATTACHMENTS, AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TAG = new PropertyDescriptor.Builder()
+ .name(ASANA_TAG_NAME)
+ .displayName("Tag")
+ .description("Fetch only objects having this tag. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TASKS, AV_COLLECT_TASK_ATTACHMENTS, AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TEAM_NAME = new PropertyDescriptor.Builder()
+ .name(ASANA_TEAM_NAME)
+ .displayName("Team")
+ .description("Team name. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TEAM_MEMBERS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OUTPUT_BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name(ASANA_OUTPUT_BATCH_SIZE)
+ .displayName("Output Batch Size")
+ .description("The number of output FlowFiles to queue before committing the process session. When set to zero, the session will be committed when all result set rows "
+ + "have been processed and the output FlowFiles are ready for transfer to the downstream relationship. For large result sets, this can cause a large burst of FlowFiles "
+ + "to be transferred at the end of processor execution. If this property is set, then when the specified number of FlowFiles are ready for transfer, then the session will "
+ + "be committed, thus releasing the FlowFiles to the downstream relationship.")
+ .defaultValue("0")
+ .required(true)
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+
+ protected static final List<PropertyDescriptor> DESCRIPTORS = Collections.unmodifiableList(Lists.newArrayList(
+ PROP_ASANA_CONTROLLER_SERVICE,
+ PROP_ASANA_OBJECT_TYPE,
+ PROP_ASANA_PROJECT,
+ PROP_ASANA_SECTION,
+ PROP_ASANA_TEAM_NAME,
+ PROP_ASANA_TAG,
+ PROP_ASANA_OUTPUT_BATCH_SIZE
+ ));
+
+ protected static final Relationship REL_NEW = new Relationship.Builder()
+ .name(REL_NAME_NEW)
+ .description("Newly collected objects are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_UPDATED = new Relationship.Builder()
+ .name(REL_NAME_UPDATED)
+ .description("Objects that have already been collected earlier, but were updated since, are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_REMOVED = new Relationship.Builder()
+ .name(REL_NAME_REMOVED)
+ .description("Notification about deleted objects are routed to this relationship. "
+ + "Flow files will not have any payload. IDs of the resources no longer exist "
+ + "are carried by the asana.gid attribute of the generated FlowFiles.")
+ .build();
+
+ protected static final Set<Relationship> RELATIONSHIPS = Collections.unmodifiableSet(Sets.newHashSet(
+ REL_NEW,
+ REL_UPDATED,
+ REL_REMOVED
+ ));
+
+ final Scope STATE_STORAGE_SCOPE = Scope.LOCAL;
+
+ AsanaClientServiceApi controllerService;
+ AsanaObjectFetcher objectFetcher;
+ private Integer batchSize;
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return RELATIONSHIPS;
+ }
+
+ @Override
+ public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return DESCRIPTORS;
+ }
+
+ @OnScheduled
+ public synchronized void onScheduled(final ProcessContext context) throws InitializationException {
+ controllerService = context.getProperty(PROP_ASANA_CONTROLLER_SERVICE).asControllerService(AsanaClientServiceApi.class);
+ AsanaClient client = controllerService.createClient();
+ batchSize = context.getProperty(PROP_ASANA_OUTPUT_BATCH_SIZE).asInteger();
+
+ try {
+ getLogger().debug("Initializing object fetcher...");
+ objectFetcher = createObjectFetcher(context, client);
+ } catch (Exception e) {
+ throw new InitializationException(e);
+ }
+ }
+
+ @Override
+ public synchronized void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ try {
+ Map<String, String> state = recoverState(context).orElse(Collections.emptyMap());
+ getLogger().debug("Attempting to load state: {}", state);
+ objectFetcher.loadState(state);
+ } catch (Exception e) {
+ getLogger().info("Failed to recover state. Falling back to clean start.");
+ objectFetcher.clearState();
+ }
+ getLogger().debug("Initial state: {}", objectFetcher.saveState());
+
+ Collection<FlowFile> newItems = new ArrayList<>();
+ Collection<FlowFile> updatedItems = new ArrayList<>();
+ Collection<FlowFile> removedItems = new ArrayList<>();
+
+ AsanaObject nextObject;
+ while ((batchSize == 0 || (batchSize > (newItems.size() + updatedItems.size() + removedItems.size()))) && (nextObject = objectFetcher.fetchNext()) != null) {
+ final Map<String, String> attributes = new HashMap<>(2);
+ attributes.put(CoreAttributes.MIME_TYPE.key(), ContentType.APPLICATION_JSON.getMimeType());
+ attributes.put(ASANA_GID, nextObject.getGid());
+ FlowFile flowFile = createFlowFileWithStringPayload(session, nextObject.getContent());
+ flowFile = session.putAllAttributes(flowFile, attributes);
+
+ switch (nextObject.getState()) {
+ case NEW:
+ newItems.add(flowFile);
+ break;
+ case REMOVED:
+ removedItems.add(flowFile);
+ break;
+ default:
+ updatedItems.add(flowFile);
+ }
+ }
+
+ if (newItems.isEmpty() && updatedItems.isEmpty() && removedItems.isEmpty()) {
+ context.yield();
+ getLogger().debug("Yielding, as there are no new FlowFiles.");
+ return;
+ }
+
+ session.transfer(newItems, REL_NEW);
+ session.transfer(updatedItems, REL_UPDATED);
+ session.transfer(removedItems, REL_REMOVED);
+
+ session.commitAsync();
+ Map<String, String> state = objectFetcher.saveState();
+ try {
+ persistState(state, context);
+ } catch (IOException e) {
+ throw new ProcessException(e);
+ }
+ getLogger().debug(
+ "New state after transferring {} new, {} updated, and {} removed items: {}",
+ newItems.size(), updatedItems.size(), removedItems.size(), state);
+ }
+
+ protected AsanaObjectFetcher createObjectFetcher(final ProcessContext context, AsanaClient client) {
+ final String objectType = context.getProperty(PROP_ASANA_OBJECT_TYPE).getValue();
+ final String projectName = context.getProperty(PROP_ASANA_PROJECT).getValue();
+ final String sectionName = context.getProperty(PROP_ASANA_SECTION).getValue();
+ final String teamName = context.getProperty(PROP_ASANA_TEAM_NAME).getValue();
+ final String tagName = context.getProperty(PROP_ASANA_TAG).getValue();
+
+ switch (objectType) {
+ case AV_NAME_COLLECT_TASKS:
+ return new AsanaTaskFetcher(client, projectName, sectionName, tagName);
+ case AV_NAME_COLLECT_PROJECTS:
+ return new AsanaProjectFetcher(client);
+ case AV_NAME_COLLECT_PROJECT_EVENTS:
+ return new AsanaProjectEventFetcher(client, projectName);
+ case AV_NAME_COLLECT_PROJECT_MEMBERS:
+ return new AsanaProjectMembershipFetcher(client, projectName);
+ case AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS:
+ return new AsanaProjectStatusAttachmentFetcher(client, projectName);
+ case AV_NAME_COLLECT_PROJECT_STATUS_UPDATES:
+ return new AsanaProjectStatusFetcher(client, projectName);
+ case AV_NAME_COLLECT_STORIES:
+ return new AsanaStoryFetcher(client, projectName, sectionName, tagName);
+ case AV_NAME_COLLECT_TAGS:
+ return new AsanaTagFetcher(client);
+ case AV_NAME_COLLECT_TASK_ATTACHMENTS:
+ return new AsanaTaskAttachmentFetcher(client, projectName, sectionName, tagName);
+ case AV_NAME_COLLECT_TEAMS:
+ return new AsanaTeamFetcher(client);
+ case AV_NAME_COLLECT_TEAM_MEMBERS:
+ return new AsanaTeamMemberFetcher(client, teamName);
+ case AV_NAME_COLLECT_USERS:
+ return new AsanaUserFetcher(client);
+ }
+
+ throw new RuntimeException("Cannot fetch objects of type: " + objectType);
Review Comment:
Done in [dc2a050](https://github.com/apache/nifi/pull/6504/commits/dc2a050548a389d89181670371748af741eefd61)
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientServiceApi;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@TriggerSerially
+@Stateful(scopes = {Scope.LOCAL}, description = "Fingerprints of items in the last successful query are stored in order to enable incremental loading and change detection.")
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.")
+@Tags({"asana", "source", "connector", "ingest"})
Review Comment:
Done in [dc2a050](https://github.com/apache/nifi/pull/6504/commits/dc2a050548a389d89181670371748af741eefd61)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
exceptionfactory commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r1012362913
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-services/src/main/java/org/apache/nifi/controller/asana/StandardAsanaClientProviderService.java:
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.controller.asana;
+
+import com.asana.Client;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.resource.ResourceCardinality;
+import org.apache.nifi.components.resource.ResourceType;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+
+import static org.apache.nifi.controller.asana.StandardAsanaClient.ASANA_CLIENT_OPTION_BASE_URL;
+
+@CapabilityDescription("Common service to authenticate with Asana, and to work on a specified workspace.")
+@Tags({"asana", "service", "authentication"})
+public class StandardAsanaClientProviderService extends AbstractControllerService implements AsanaClientProviderService {
+
+ protected static final String ASANA_API_URL = "asana-api-url";
+ protected static final String ASANA_PERSONAL_ACCESS_TOKEN = "asana-personal-access-token";
+ protected static final String ASANA_WORKSPACE_NAME = "asana-workspace-name";
+
+ protected static final PropertyDescriptor PROP_ASANA_API_BASE_URL = new PropertyDescriptor.Builder()
+ .name(ASANA_API_URL)
+ .displayName("API URL")
+ .description("Base URL of Asana API. Leave it as default, unless you have your own Asana instance "
+ + "serving on a different URL. (typical for on-premise installations)")
+ .required(true)
+ .defaultValue(Client.DEFAULTS.get(ASANA_CLIENT_OPTION_BASE_URL).toString())
+ .identifiesExternalResource(ResourceCardinality.SINGLE, ResourceType.URL)
Review Comment:
This attribute should be removed, as the External Resource feature is used for retrieving content, not for referencing a URL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] turcsanyip commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
turcsanyip commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r1027320316
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-services-api/pom.xml:
##########
@@ -0,0 +1,39 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-asana-bundle</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-asana-services-api</artifactId>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-api</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ </dependency>
Review Comment:
`nifi-api`'s version and scope are defined in `dependecyManagement` in parent pom and they do not need to be specified here.
```suggestion
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-api</artifactId>
</dependency>
```
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-services/pom.xml:
##########
@@ -0,0 +1,64 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-asana-bundle</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-asana-services</artifactId>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-api</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-mock</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>com.squareup.okhttp3</groupId>
+ <artifactId>mockwebserver</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>commons-io</groupId>
+ <artifactId>commons-io</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>com.asana</groupId>
+ <artifactId>asana</artifactId>
+ </dependency>
Review Comment:
The `asana` dependency should be `provided` here because it is available from `nifi-asana-services-api-nar` which is the parent nar of this nar (jar-s from the parent nar will be loaded first and they are available in the child nar-s).
```suggestion
<dependency>
<groupId>com.asana</groupId>
<artifactId>asana</artifactId>
<scope>provided</scope>
</dependency>
```
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/pom.xml:
##########
@@ -0,0 +1,69 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-asana-bundle</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-asana-processors</artifactId>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-api</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-mock</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>commons-io</groupId>
+ <artifactId>commons-io</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.commons</groupId>
+ <artifactId>commons-collections4</artifactId>
+ <version>4.4</version>
+ </dependency>
+ <dependency>
+ <groupId>com.asana</groupId>
+ <artifactId>asana</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-utils</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ <scope>compile</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-asana-services-api</artifactId>
+ </dependency>
Review Comment:
Similar to `nifi-asana-services`' pom:
- no version needed for `nifi-api`
- `compile` scope for `nifi-utils` is redundant
- `asana` and `nifi-asana-services-api` should be provided
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,457 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import static java.lang.String.format;
+import static java.util.Collections.singletonMap;
+import static java.util.stream.Collectors.groupingBy;
+import static java.util.stream.Collectors.joining;
+import static org.apache.commons.collections4.ListUtils.partition;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientProviderService;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaObjectState;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+@TriggerSerially
+@PrimaryNodeOnly
+@Stateful(scopes = {Scope.CLUSTER}, description = "Fingerprints of items in the last successful query are stored in order to enable incremental loading and change detection.")
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.")
+@Tags({"asana", "source", "ingest"})
+@CapabilityDescription("This processor collects data from Asana")
+public class GetAsanaObject extends AbstractProcessor {
+
+ protected static final String ASANA_GID = "asana.gid";
+ protected static final String AV_NAME_COLLECT_TASKS = "asana-collect-tasks";
+ protected static final String AV_NAME_COLLECT_TASK_ATTACHMENTS = "asana-collect-task-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECTS = "asana-collect-projects";
+ protected static final String AV_NAME_COLLECT_TAGS = "asana-collect-tags";
+ protected static final String AV_NAME_COLLECT_USERS = "asana-collect-users";
+ protected static final String AV_NAME_COLLECT_PROJECT_MEMBERS = "asana-collect-project-members";
+ protected static final String AV_NAME_COLLECT_TEAMS = "asana-collect-teams";
+ protected static final String AV_NAME_COLLECT_TEAM_MEMBERS = "asana-collect-team-members";
+ protected static final String AV_NAME_COLLECT_STORIES = "asana-collect-stories";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_UPDATES = "asana-collect-project-status-updates";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS = "asana-collect-project-status-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECT_EVENTS = "asana-collect-project-events";
+ protected static final String ASANA_CONTROLLER_SERVICE = "asana-controller-service";
+ protected static final String ASANA_OBJECT_TYPE = "asana-object-type";
+ protected static final String ASANA_PROJECT_NAME = "asana-project-name";
+ protected static final String ASANA_SECTION_NAME = "asana-section-name";
+ protected static final String ASANA_TAG_NAME = "asana-tag-name";
+ protected static final String ASANA_TEAM_NAME = "asana-team-name";
+ protected static final String ASANA_OUTPUT_BATCH_SIZE = "asana-output-batch-size";
+ protected static final String REL_NAME_NEW = "new";
+ protected static final String REL_NAME_UPDATED = "updated";
+ protected static final String REL_NAME_REMOVED = "removed";
+
+ protected static final AllowableValue AV_COLLECT_TASKS = new AllowableValue(
+ AV_NAME_COLLECT_TASKS,
+ "Tasks",
+ "Collect tasks matching to the specified conditions.");
+
+ protected static final AllowableValue AV_COLLECT_TASK_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_TASK_ATTACHMENTS,
+ "Task Attachments",
+ "Collect attached files of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECTS,
+ "Projects",
+ "Collect projects of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TAGS = new AllowableValue(
+ AV_NAME_COLLECT_TAGS,
+ "Tags",
+ "Collect tags of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_USERS = new AllowableValue(
+ AV_NAME_COLLECT_USERS,
+ "Users",
+ "Collect users assigned to the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_MEMBERS,
+ "Members of a Project",
+ "Collect users assigned to the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAMS = new AllowableValue(
+ AV_NAME_COLLECT_TEAMS,
+ "Teams",
+ "Collect teams of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAM_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_TEAM_MEMBERS,
+ "Team Members",
+ "Collect users assigned to the specified team."
+ );
+
+ protected static final AllowableValue AV_COLLECT_STORIES = new AllowableValue(
+ AV_NAME_COLLECT_STORIES,
+ "Stories of Tasks",
+ "Collect stories (comments) of of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_UPDATES = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_UPDATES,
+ "Status Updates of a Project",
+ "Collect status updates of the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ "Attachments of Status Updates",
+ "Collect attached files of project status updates."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_EVENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_EVENTS,
+ "Events of a Project",
+ "Collect various events happening on the specified project and on its' tasks."
+ );
+
+ protected static final PropertyDescriptor PROP_ASANA_CONTROLLER_SERVICE = new PropertyDescriptor.Builder()
+ .name(ASANA_CONTROLLER_SERVICE)
+ .displayName("Asana Controller Service")
+ .description("Specify which controller service to use for accessing Asana.")
+ .required(true)
+ .identifiesControllerService(AsanaClientProviderService.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OBJECT_TYPE = new PropertyDescriptor.Builder()
+ .name(ASANA_OBJECT_TYPE)
+ .displayName("Object Type")
+ .description("Specify what kind of objects to be collected from Asana")
+ .required(true)
+ .allowableValues(
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_PROJECTS,
+ AV_COLLECT_TAGS,
+ AV_COLLECT_USERS,
+ AV_COLLECT_PROJECT_MEMBERS,
+ AV_COLLECT_TEAMS,
+ AV_COLLECT_TEAM_MEMBERS,
+ AV_COLLECT_STORIES,
+ AV_COLLECT_PROJECT_STATUS_UPDATES,
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ AV_COLLECT_PROJECT_EVENTS)
+ .defaultValue(AV_COLLECT_TASKS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_PROJECT = new PropertyDescriptor.Builder()
+ .name(ASANA_PROJECT_NAME)
+ .displayName("Project Name")
+ .description("Fetch only objects in this project. Case sensitive.")
+ .required(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(
+ PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_PROJECT_MEMBERS,
+ AV_COLLECT_STORIES,
+ AV_COLLECT_PROJECT_STATUS_UPDATES,
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ AV_COLLECT_PROJECT_EVENTS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_SECTION = new PropertyDescriptor.Builder()
+ .name(ASANA_SECTION_NAME)
+ .displayName("Section Name")
+ .description("Fetch only objects in this section. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TASKS, AV_COLLECT_TASK_ATTACHMENTS, AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TAG = new PropertyDescriptor.Builder()
+ .name(ASANA_TAG_NAME)
+ .displayName("Tag")
+ .description("Fetch only objects having this tag. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TASKS, AV_COLLECT_TASK_ATTACHMENTS, AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TEAM_NAME = new PropertyDescriptor.Builder()
+ .name(ASANA_TEAM_NAME)
+ .displayName("Team")
+ .description("Team name. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TEAM_MEMBERS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OUTPUT_BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name(ASANA_OUTPUT_BATCH_SIZE)
+ .displayName("Output Batch Size")
+ .description("The number of items batched together in a single Flow File. If set to 1 (default), then each item is"
+ + " transferred in a separate Flow File and each will have an asana.gid attribute, to help identifying"
+ + " the fetched item on the server side, if needed. If the batch size is greater than 1, then the"
+ + " specified amount of items are batched together in a single Flow File as a Json array, and the"
+ + " Flow Files won't have the asana.gid attribute.")
+ .defaultValue("1")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+
+ protected static final List<PropertyDescriptor> DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ PROP_ASANA_CONTROLLER_SERVICE,
+ PROP_ASANA_OBJECT_TYPE,
+ PROP_ASANA_PROJECT,
+ PROP_ASANA_SECTION,
+ PROP_ASANA_TEAM_NAME,
+ PROP_ASANA_TAG,
+ PROP_ASANA_OUTPUT_BATCH_SIZE
+ ));
+
+ protected static final Relationship REL_NEW = new Relationship.Builder()
+ .name(REL_NAME_NEW)
+ .description("Newly collected objects are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_UPDATED = new Relationship.Builder()
+ .name(REL_NAME_UPDATED)
+ .description("Objects that have already been collected earlier, but were updated since, are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_REMOVED = new Relationship.Builder()
+ .name(REL_NAME_REMOVED)
+ .description("Notification about deleted objects are routed to this relationship. "
+ + "Flow files will not have any payload. IDs of the resources no longer exist "
+ + "are carried by the asana.gid attribute of the generated FlowFiles.")
+ .build();
+
+ protected static final Set<Relationship> RELATIONSHIPS = Collections.unmodifiableSet(new HashSet<>(Arrays.asList(
+ REL_NEW,
+ REL_UPDATED,
+ REL_REMOVED
+ )));
+
+ private static final Scope STATE_STORAGE_SCOPE = Scope.CLUSTER;
+
+ private volatile AsanaObjectFetcher objectFetcher;
+ private volatile Integer batchSize;
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return RELATIONSHIPS;
+ }
+
+ @Override
+ public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return DESCRIPTORS;
+ }
+
+ @OnScheduled
+ public synchronized void onScheduled(final ProcessContext context) throws InitializationException {
Review Comment:
We don't use `synchronized` for `onScheduled()`. It is not necessary, because the framework always calls it on a single thread.
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-services-api-nar/src/main/resources/META-INF/NOTICE:
##########
@@ -0,0 +1,51 @@
+nifi-asana-nar
+Copyright 2015-2022 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+===========================================
+Apache Software License v2
+===========================================
+
+The following binary components are provided under the Apache Software License v2
+
+ (ASLv2) Apache Commons IO
+ The following NOTICE information applies:
+ Apache Commons IO
+ Copyright 2002-2017 The Apache Software Foundation
+
+ (ASLv2) Jackson JSON processor
+ The following NOTICE information applies:
+ # Jackson JSON processor
+
+ Jackson is a high-performance, Free/Open Source JSON processing library.
+ It was originally written by Tatu Saloranta (tatu.saloranta@iki.fi), and has
+ been in development since 2007.
+ It is currently developed by a community of developers, as well as supported
+ commercially by FasterXML.com.
+
+ ## Licensing
+
+ Jackson core and extension components may licensed under different licenses.
+ To find the details that apply to this artifact see the accompanying LICENSE file.
+ For more information, including possible other licensing options, contact
+ FasterXML.com (http://fasterxml.com).
+
+ ## Credits
+
+ A list of contributors may be found from CREDITS file, which is included
+ in some artifacts (usually source distributions); but is always available
+ from the source code management (SCM) system project uses.
+
+
+===========================================
+The MIT License
+===========================================
+
+The following binary components are provided under the MIT License
+
+ (MIT License) Java client library for the Asana API
+ The following NOTICE information applies:
+ Asana
+ Copyright (c) 2015
Review Comment:
Asana library brings in a couple of transitive dependencies that also need to be added in the NOTICE file (e.g. checker-qual). Please check the jar list in the nar file after the build and add the necessary entries in the NOTICE file.
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-services/pom.xml:
##########
@@ -0,0 +1,64 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-asana-bundle</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-asana-services</artifactId>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-api</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-mock</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>com.squareup.okhttp3</groupId>
+ <artifactId>mockwebserver</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>commons-io</groupId>
+ <artifactId>commons-io</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>com.asana</groupId>
+ <artifactId>asana</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-utils</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ <scope>compile</scope>
+ </dependency>
Review Comment:
`compile` scope is the default and we don't use it explicitly if not needed.
```suggestion
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-utils</artifactId>
<version>1.19.0-SNAPSHOT</version>
</dependency>
```
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-services/pom.xml:
##########
@@ -0,0 +1,64 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-asana-bundle</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-asana-services</artifactId>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-api</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ </dependency>
Review Comment:
```suggestion
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-api</artifactId>
</dependency>
```
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-services-api-nar/src/main/resources/META-INF/NOTICE:
##########
@@ -0,0 +1,51 @@
+nifi-asana-nar
+Copyright 2015-2022 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+===========================================
+Apache Software License v2
+===========================================
+
+The following binary components are provided under the Apache Software License v2
+
+ (ASLv2) Apache Commons IO
+ The following NOTICE information applies:
+ Apache Commons IO
+ Copyright 2002-2017 The Apache Software Foundation
+
+ (ASLv2) Jackson JSON processor
+ The following NOTICE information applies:
+ # Jackson JSON processor
+
+ Jackson is a high-performance, Free/Open Source JSON processing library.
+ It was originally written by Tatu Saloranta (tatu.saloranta@iki.fi), and has
+ been in development since 2007.
+ It is currently developed by a community of developers, as well as supported
+ commercially by FasterXML.com.
+
+ ## Licensing
+
+ Jackson core and extension components may licensed under different licenses.
+ To find the details that apply to this artifact see the accompanying LICENSE file.
+ For more information, including possible other licensing options, contact
+ FasterXML.com (http://fasterxml.com).
+
+ ## Credits
+
+ A list of contributors may be found from CREDITS file, which is included
+ in some artifacts (usually source distributions); but is always available
+ from the source code management (SCM) system project uses.
Review Comment:
As far as I see, Jackson is not present in the built nar so this entry should be deleted.
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/pom.xml:
##########
@@ -0,0 +1,69 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-asana-bundle</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-asana-processors</artifactId>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-api</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-mock</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>commons-io</groupId>
+ <artifactId>commons-io</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.commons</groupId>
+ <artifactId>commons-collections4</artifactId>
+ <version>4.4</version>
+ </dependency>
+ <dependency>
+ <groupId>com.asana</groupId>
+ <artifactId>asana</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-utils</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ <scope>compile</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-asana-services-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.mockito</groupId>
+ <artifactId>mockito-junit-jupiter</artifactId>
+ <scope>test</scope>
+ </dependency>
Review Comment:
This is a duplicated dependency and should be deleted. `mockito-junit-jupiter` comes from the project root pom.
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-services/pom.xml:
##########
@@ -0,0 +1,64 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-asana-bundle</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-asana-services</artifactId>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-api</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-mock</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ <scope>test</scope>
+ </dependency>
Review Comment:
Could you please organize the dependencies in all the poms using the following order:
- nifi prod dependencies
- 3rd party prod dependencies
- test dependencies
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-services/pom.xml:
##########
@@ -0,0 +1,64 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-asana-bundle</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-asana-services</artifactId>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-api</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-mock</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>com.squareup.okhttp3</groupId>
+ <artifactId>mockwebserver</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>commons-io</groupId>
+ <artifactId>commons-io</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>com.asana</groupId>
+ <artifactId>asana</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-utils</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ <scope>compile</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-asana-services-api</artifactId>
+ </dependency>
Review Comment:
Similar to the `asana` dependency, it should be `provided`.
```suggestion
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-asana-services-api</artifactId>
<scope>provided</scope>
</dependency>
```
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors-nar/src/main/resources/META-INF/NOTICE:
##########
@@ -0,0 +1,51 @@
+nifi-asana-nar
Review Comment:
```suggestion
nifi-asana-processors-nar
```
Please fix the header for the other NOTICE files too. I must match the name of the nar bundle.
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-services-nar/src/main/resources/META-INF/NOTICE:
##########
@@ -0,0 +1,51 @@
+nifi-asana-nar
+Copyright 2015-2022 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+===========================================
+Apache Software License v2
+===========================================
+
+The following binary components are provided under the Apache Software License v2
+
+ (ASLv2) Apache Commons IO
+ The following NOTICE information applies:
+ Apache Commons IO
+ Copyright 2002-2017 The Apache Software Foundation
+
+ (ASLv2) Jackson JSON processor
+ The following NOTICE information applies:
+ # Jackson JSON processor
+
+ Jackson is a high-performance, Free/Open Source JSON processing library.
+ It was originally written by Tatu Saloranta (tatu.saloranta@iki.fi), and has
+ been in development since 2007.
+ It is currently developed by a community of developers, as well as supported
+ commercially by FasterXML.com.
+
+ ## Licensing
+
+ Jackson core and extension components may licensed under different licenses.
+ To find the details that apply to this artifact see the accompanying LICENSE file.
+ For more information, including possible other licensing options, contact
+ FasterXML.com (http://fasterxml.com).
+
+ ## Credits
+
+ A list of contributors may be found from CREDITS file, which is included
+ in some artifacts (usually source distributions); but is always available
+ from the source code management (SCM) system project uses.
Review Comment:
Jackson is not bundled, it should be deleted.
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-services-nar/src/main/resources/META-INF/NOTICE:
##########
@@ -0,0 +1,51 @@
+nifi-asana-nar
+Copyright 2015-2022 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+===========================================
+Apache Software License v2
+===========================================
+
+The following binary components are provided under the Apache Software License v2
+
+ (ASLv2) Apache Commons IO
+ The following NOTICE information applies:
+ Apache Commons IO
+ Copyright 2002-2017 The Apache Software Foundation
+
+ (ASLv2) Jackson JSON processor
+ The following NOTICE information applies:
+ # Jackson JSON processor
+
+ Jackson is a high-performance, Free/Open Source JSON processing library.
+ It was originally written by Tatu Saloranta (tatu.saloranta@iki.fi), and has
+ been in development since 2007.
+ It is currently developed by a community of developers, as well as supported
+ commercially by FasterXML.com.
+
+ ## Licensing
+
+ Jackson core and extension components may licensed under different licenses.
+ To find the details that apply to this artifact see the accompanying LICENSE file.
+ For more information, including possible other licensing options, contact
+ FasterXML.com (http://fasterxml.com).
+
+ ## Credits
+
+ A list of contributors may be found from CREDITS file, which is included
+ in some artifacts (usually source distributions); but is always available
+ from the source code management (SCM) system project uses.
+
+
+===========================================
+The MIT License
+===========================================
+
+The following binary components are provided under the MIT License
+
+ (MIT License) Java client library for the Asana API
+ The following NOTICE information applies:
+ Asana
+ Copyright (c) 2015
Review Comment:
`asana` dependency is provided in this nar (see my earlier comment) and therefore this entry should be deleted.
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-services-api/src/main/java/org/apache/nifi/controller/asana/AsanaClient.java:
##########
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.controller.asana;
+
+import com.asana.models.Attachment;
+import com.asana.models.Project;
+import com.asana.models.ProjectMembership;
+import com.asana.models.ProjectStatus;
+import com.asana.models.Section;
+import com.asana.models.Story;
+import com.asana.models.Tag;
+import com.asana.models.Task;
+import com.asana.models.Team;
+import com.asana.models.User;
+
+import java.util.Map;
+
+/**
+ * This interface represents a client to Asana REST server, with some basic filtering options built in.
+ */
+public interface AsanaClient {
+ /**
+ * Find & retrieve an Asana project by its name. If multiple projects match, returns the first.
+ * If there is no match, then {@link RuntimeException} is thrown. Note that constant ordering
Review Comment:
Please do not throw raw `RuntimeException`s in the code. Please use `ProcessException` or `UncheckedIOException` instead, or you can also define a custom runtime exception like `AsanaException`.
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,457 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import static java.lang.String.format;
+import static java.util.Collections.singletonMap;
+import static java.util.stream.Collectors.groupingBy;
+import static java.util.stream.Collectors.joining;
+import static org.apache.commons.collections4.ListUtils.partition;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientProviderService;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaObjectState;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+@TriggerSerially
+@PrimaryNodeOnly
+@Stateful(scopes = {Scope.CLUSTER}, description = "Fingerprints of items in the last successful query are stored in order to enable incremental loading and change detection.")
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.")
+@Tags({"asana", "source", "ingest"})
+@CapabilityDescription("This processor collects data from Asana")
+public class GetAsanaObject extends AbstractProcessor {
+
+ protected static final String ASANA_GID = "asana.gid";
+ protected static final String AV_NAME_COLLECT_TASKS = "asana-collect-tasks";
+ protected static final String AV_NAME_COLLECT_TASK_ATTACHMENTS = "asana-collect-task-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECTS = "asana-collect-projects";
+ protected static final String AV_NAME_COLLECT_TAGS = "asana-collect-tags";
+ protected static final String AV_NAME_COLLECT_USERS = "asana-collect-users";
+ protected static final String AV_NAME_COLLECT_PROJECT_MEMBERS = "asana-collect-project-members";
+ protected static final String AV_NAME_COLLECT_TEAMS = "asana-collect-teams";
+ protected static final String AV_NAME_COLLECT_TEAM_MEMBERS = "asana-collect-team-members";
+ protected static final String AV_NAME_COLLECT_STORIES = "asana-collect-stories";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_UPDATES = "asana-collect-project-status-updates";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS = "asana-collect-project-status-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECT_EVENTS = "asana-collect-project-events";
Review Comment:
I'd suggest creating an `enum` for these allowable values, like [HubSpotObjectType](https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-hubspot-bundle/nifi-hubspot-processors/src/main/java/org/apache/nifi/processors/hubspot/HubSpotObjectType.java).
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors-nar/src/main/resources/META-INF/NOTICE:
##########
@@ -0,0 +1,51 @@
+nifi-asana-nar
+Copyright 2015-2022 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+===========================================
+Apache Software License v2
+===========================================
+
+The following binary components are provided under the Apache Software License v2
+
+ (ASLv2) Apache Commons IO
+ The following NOTICE information applies:
+ Apache Commons IO
+ Copyright 2002-2017 The Apache Software Foundation
+
+ (ASLv2) Jackson JSON processor
+ The following NOTICE information applies:
+ # Jackson JSON processor
+
+ Jackson is a high-performance, Free/Open Source JSON processing library.
+ It was originally written by Tatu Saloranta (tatu.saloranta@iki.fi), and has
+ been in development since 2007.
+ It is currently developed by a community of developers, as well as supported
+ commercially by FasterXML.com.
+
+ ## Licensing
+
+ Jackson core and extension components may licensed under different licenses.
+ To find the details that apply to this artifact see the accompanying LICENSE file.
+ For more information, including possible other licensing options, contact
+ FasterXML.com (http://fasterxml.com).
+
+ ## Credits
+
+ A list of contributors may be found from CREDITS file, which is included
+ in some artifacts (usually source distributions); but is always available
+ from the source code management (SCM) system project uses.
+
+
+===========================================
+The MIT License
+===========================================
+
+The following binary components are provided under the MIT License
+
+ (MIT License) Java client library for the Asana API
+ The following NOTICE information applies:
+ Asana
+ Copyright (c) 2015
Review Comment:
Similar to `nifi-asana-services-nar`'s NOTICE file:
- Jackson is not present
- Asana is provided and should be delete from here
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,457 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import static java.lang.String.format;
+import static java.util.Collections.singletonMap;
+import static java.util.stream.Collectors.groupingBy;
+import static java.util.stream.Collectors.joining;
+import static org.apache.commons.collections4.ListUtils.partition;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientProviderService;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaObjectState;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+@TriggerSerially
+@PrimaryNodeOnly
+@Stateful(scopes = {Scope.CLUSTER}, description = "Fingerprints of items in the last successful query are stored in order to enable incremental loading and change detection.")
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.")
+@Tags({"asana", "source", "ingest"})
+@CapabilityDescription("This processor collects data from Asana")
+public class GetAsanaObject extends AbstractProcessor {
+
+ protected static final String ASANA_GID = "asana.gid";
+ protected static final String AV_NAME_COLLECT_TASKS = "asana-collect-tasks";
+ protected static final String AV_NAME_COLLECT_TASK_ATTACHMENTS = "asana-collect-task-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECTS = "asana-collect-projects";
+ protected static final String AV_NAME_COLLECT_TAGS = "asana-collect-tags";
+ protected static final String AV_NAME_COLLECT_USERS = "asana-collect-users";
+ protected static final String AV_NAME_COLLECT_PROJECT_MEMBERS = "asana-collect-project-members";
+ protected static final String AV_NAME_COLLECT_TEAMS = "asana-collect-teams";
+ protected static final String AV_NAME_COLLECT_TEAM_MEMBERS = "asana-collect-team-members";
+ protected static final String AV_NAME_COLLECT_STORIES = "asana-collect-stories";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_UPDATES = "asana-collect-project-status-updates";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS = "asana-collect-project-status-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECT_EVENTS = "asana-collect-project-events";
+ protected static final String ASANA_CONTROLLER_SERVICE = "asana-controller-service";
+ protected static final String ASANA_OBJECT_TYPE = "asana-object-type";
+ protected static final String ASANA_PROJECT_NAME = "asana-project-name";
+ protected static final String ASANA_SECTION_NAME = "asana-section-name";
+ protected static final String ASANA_TAG_NAME = "asana-tag-name";
+ protected static final String ASANA_TEAM_NAME = "asana-team-name";
+ protected static final String ASANA_OUTPUT_BATCH_SIZE = "asana-output-batch-size";
+ protected static final String REL_NAME_NEW = "new";
+ protected static final String REL_NAME_UPDATED = "updated";
+ protected static final String REL_NAME_REMOVED = "removed";
+
+ protected static final AllowableValue AV_COLLECT_TASKS = new AllowableValue(
+ AV_NAME_COLLECT_TASKS,
+ "Tasks",
+ "Collect tasks matching to the specified conditions.");
+
+ protected static final AllowableValue AV_COLLECT_TASK_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_TASK_ATTACHMENTS,
+ "Task Attachments",
+ "Collect attached files of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECTS,
+ "Projects",
+ "Collect projects of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TAGS = new AllowableValue(
+ AV_NAME_COLLECT_TAGS,
+ "Tags",
+ "Collect tags of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_USERS = new AllowableValue(
+ AV_NAME_COLLECT_USERS,
+ "Users",
+ "Collect users assigned to the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_MEMBERS,
+ "Members of a Project",
+ "Collect users assigned to the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAMS = new AllowableValue(
+ AV_NAME_COLLECT_TEAMS,
+ "Teams",
+ "Collect teams of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAM_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_TEAM_MEMBERS,
+ "Team Members",
+ "Collect users assigned to the specified team."
+ );
+
+ protected static final AllowableValue AV_COLLECT_STORIES = new AllowableValue(
+ AV_NAME_COLLECT_STORIES,
+ "Stories of Tasks",
+ "Collect stories (comments) of of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_UPDATES = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_UPDATES,
+ "Status Updates of a Project",
+ "Collect status updates of the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ "Attachments of Status Updates",
+ "Collect attached files of project status updates."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_EVENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_EVENTS,
+ "Events of a Project",
+ "Collect various events happening on the specified project and on its' tasks."
+ );
+
+ protected static final PropertyDescriptor PROP_ASANA_CONTROLLER_SERVICE = new PropertyDescriptor.Builder()
+ .name(ASANA_CONTROLLER_SERVICE)
+ .displayName("Asana Controller Service")
+ .description("Specify which controller service to use for accessing Asana.")
+ .required(true)
+ .identifiesControllerService(AsanaClientProviderService.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OBJECT_TYPE = new PropertyDescriptor.Builder()
+ .name(ASANA_OBJECT_TYPE)
+ .displayName("Object Type")
+ .description("Specify what kind of objects to be collected from Asana")
+ .required(true)
+ .allowableValues(
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_PROJECTS,
+ AV_COLLECT_TAGS,
+ AV_COLLECT_USERS,
+ AV_COLLECT_PROJECT_MEMBERS,
+ AV_COLLECT_TEAMS,
+ AV_COLLECT_TEAM_MEMBERS,
+ AV_COLLECT_STORIES,
+ AV_COLLECT_PROJECT_STATUS_UPDATES,
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ AV_COLLECT_PROJECT_EVENTS)
+ .defaultValue(AV_COLLECT_TASKS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_PROJECT = new PropertyDescriptor.Builder()
+ .name(ASANA_PROJECT_NAME)
+ .displayName("Project Name")
+ .description("Fetch only objects in this project. Case sensitive.")
+ .required(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(
+ PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_PROJECT_MEMBERS,
+ AV_COLLECT_STORIES,
+ AV_COLLECT_PROJECT_STATUS_UPDATES,
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ AV_COLLECT_PROJECT_EVENTS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_SECTION = new PropertyDescriptor.Builder()
+ .name(ASANA_SECTION_NAME)
+ .displayName("Section Name")
+ .description("Fetch only objects in this section. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TASKS, AV_COLLECT_TASK_ATTACHMENTS, AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TAG = new PropertyDescriptor.Builder()
+ .name(ASANA_TAG_NAME)
+ .displayName("Tag")
+ .description("Fetch only objects having this tag. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TASKS, AV_COLLECT_TASK_ATTACHMENTS, AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TEAM_NAME = new PropertyDescriptor.Builder()
+ .name(ASANA_TEAM_NAME)
+ .displayName("Team")
+ .description("Team name. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TEAM_MEMBERS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OUTPUT_BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name(ASANA_OUTPUT_BATCH_SIZE)
+ .displayName("Output Batch Size")
+ .description("The number of items batched together in a single Flow File. If set to 1 (default), then each item is"
+ + " transferred in a separate Flow File and each will have an asana.gid attribute, to help identifying"
+ + " the fetched item on the server side, if needed. If the batch size is greater than 1, then the"
+ + " specified amount of items are batched together in a single Flow File as a Json array, and the"
+ + " Flow Files won't have the asana.gid attribute.")
+ .defaultValue("1")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+
+ protected static final List<PropertyDescriptor> DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ PROP_ASANA_CONTROLLER_SERVICE,
+ PROP_ASANA_OBJECT_TYPE,
+ PROP_ASANA_PROJECT,
+ PROP_ASANA_SECTION,
+ PROP_ASANA_TEAM_NAME,
+ PROP_ASANA_TAG,
+ PROP_ASANA_OUTPUT_BATCH_SIZE
+ ));
+
+ protected static final Relationship REL_NEW = new Relationship.Builder()
+ .name(REL_NAME_NEW)
+ .description("Newly collected objects are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_UPDATED = new Relationship.Builder()
+ .name(REL_NAME_UPDATED)
+ .description("Objects that have already been collected earlier, but were updated since, are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_REMOVED = new Relationship.Builder()
+ .name(REL_NAME_REMOVED)
+ .description("Notification about deleted objects are routed to this relationship. "
+ + "Flow files will not have any payload. IDs of the resources no longer exist "
+ + "are carried by the asana.gid attribute of the generated FlowFiles.")
+ .build();
+
+ protected static final Set<Relationship> RELATIONSHIPS = Collections.unmodifiableSet(new HashSet<>(Arrays.asList(
+ REL_NEW,
+ REL_UPDATED,
+ REL_REMOVED
+ )));
+
+ private static final Scope STATE_STORAGE_SCOPE = Scope.CLUSTER;
+
+ private volatile AsanaObjectFetcher objectFetcher;
+ private volatile Integer batchSize;
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return RELATIONSHIPS;
+ }
+
+ @Override
+ public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return DESCRIPTORS;
+ }
+
+ @OnScheduled
+ public synchronized void onScheduled(final ProcessContext context) throws InitializationException {
+ AsanaClientProviderService controllerService = context.getProperty(PROP_ASANA_CONTROLLER_SERVICE).asControllerService(AsanaClientProviderService.class);
+ AsanaClient client = controllerService.createClient();
+ batchSize = context.getProperty(PROP_ASANA_OUTPUT_BATCH_SIZE).asInteger();
+
+ try {
+ getLogger().debug("Initializing object fetcher...");
+ objectFetcher = createObjectFetcher(context, client);
+ } catch (Exception e) {
+ throw new InitializationException(e);
+ }
+ }
+
+ @Override
+ public synchronized void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
Review Comment:
We don't use `synchronized` for `onTrigger()`. `@TriggerSerially` ensures that the framework will call it on a single thread at a time.
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,457 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import static java.lang.String.format;
+import static java.util.Collections.singletonMap;
+import static java.util.stream.Collectors.groupingBy;
+import static java.util.stream.Collectors.joining;
+import static org.apache.commons.collections4.ListUtils.partition;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientProviderService;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaObjectState;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+@TriggerSerially
+@PrimaryNodeOnly
+@Stateful(scopes = {Scope.CLUSTER}, description = "Fingerprints of items in the last successful query are stored in order to enable incremental loading and change detection.")
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.")
+@Tags({"asana", "source", "ingest"})
+@CapabilityDescription("This processor collects data from Asana")
+public class GetAsanaObject extends AbstractProcessor {
+
+ protected static final String ASANA_GID = "asana.gid";
+ protected static final String AV_NAME_COLLECT_TASKS = "asana-collect-tasks";
+ protected static final String AV_NAME_COLLECT_TASK_ATTACHMENTS = "asana-collect-task-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECTS = "asana-collect-projects";
+ protected static final String AV_NAME_COLLECT_TAGS = "asana-collect-tags";
+ protected static final String AV_NAME_COLLECT_USERS = "asana-collect-users";
+ protected static final String AV_NAME_COLLECT_PROJECT_MEMBERS = "asana-collect-project-members";
+ protected static final String AV_NAME_COLLECT_TEAMS = "asana-collect-teams";
+ protected static final String AV_NAME_COLLECT_TEAM_MEMBERS = "asana-collect-team-members";
+ protected static final String AV_NAME_COLLECT_STORIES = "asana-collect-stories";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_UPDATES = "asana-collect-project-status-updates";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS = "asana-collect-project-status-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECT_EVENTS = "asana-collect-project-events";
+ protected static final String ASANA_CONTROLLER_SERVICE = "asana-controller-service";
+ protected static final String ASANA_OBJECT_TYPE = "asana-object-type";
+ protected static final String ASANA_PROJECT_NAME = "asana-project-name";
+ protected static final String ASANA_SECTION_NAME = "asana-section-name";
+ protected static final String ASANA_TAG_NAME = "asana-tag-name";
+ protected static final String ASANA_TEAM_NAME = "asana-team-name";
+ protected static final String ASANA_OUTPUT_BATCH_SIZE = "asana-output-batch-size";
+ protected static final String REL_NAME_NEW = "new";
+ protected static final String REL_NAME_UPDATED = "updated";
+ protected static final String REL_NAME_REMOVED = "removed";
+
+ protected static final AllowableValue AV_COLLECT_TASKS = new AllowableValue(
+ AV_NAME_COLLECT_TASKS,
+ "Tasks",
+ "Collect tasks matching to the specified conditions.");
+
+ protected static final AllowableValue AV_COLLECT_TASK_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_TASK_ATTACHMENTS,
+ "Task Attachments",
+ "Collect attached files of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECTS,
+ "Projects",
+ "Collect projects of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TAGS = new AllowableValue(
+ AV_NAME_COLLECT_TAGS,
+ "Tags",
+ "Collect tags of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_USERS = new AllowableValue(
+ AV_NAME_COLLECT_USERS,
+ "Users",
+ "Collect users assigned to the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_MEMBERS,
+ "Members of a Project",
+ "Collect users assigned to the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAMS = new AllowableValue(
+ AV_NAME_COLLECT_TEAMS,
+ "Teams",
+ "Collect teams of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAM_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_TEAM_MEMBERS,
+ "Team Members",
+ "Collect users assigned to the specified team."
+ );
+
+ protected static final AllowableValue AV_COLLECT_STORIES = new AllowableValue(
+ AV_NAME_COLLECT_STORIES,
+ "Stories of Tasks",
+ "Collect stories (comments) of of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_UPDATES = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_UPDATES,
+ "Status Updates of a Project",
+ "Collect status updates of the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ "Attachments of Status Updates",
+ "Collect attached files of project status updates."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_EVENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_EVENTS,
+ "Events of a Project",
+ "Collect various events happening on the specified project and on its' tasks."
+ );
+
+ protected static final PropertyDescriptor PROP_ASANA_CONTROLLER_SERVICE = new PropertyDescriptor.Builder()
+ .name(ASANA_CONTROLLER_SERVICE)
+ .displayName("Asana Controller Service")
+ .description("Specify which controller service to use for accessing Asana.")
+ .required(true)
+ .identifiesControllerService(AsanaClientProviderService.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OBJECT_TYPE = new PropertyDescriptor.Builder()
+ .name(ASANA_OBJECT_TYPE)
+ .displayName("Object Type")
+ .description("Specify what kind of objects to be collected from Asana")
+ .required(true)
+ .allowableValues(
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_PROJECTS,
+ AV_COLLECT_TAGS,
+ AV_COLLECT_USERS,
+ AV_COLLECT_PROJECT_MEMBERS,
+ AV_COLLECT_TEAMS,
+ AV_COLLECT_TEAM_MEMBERS,
+ AV_COLLECT_STORIES,
+ AV_COLLECT_PROJECT_STATUS_UPDATES,
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ AV_COLLECT_PROJECT_EVENTS)
+ .defaultValue(AV_COLLECT_TASKS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_PROJECT = new PropertyDescriptor.Builder()
+ .name(ASANA_PROJECT_NAME)
+ .displayName("Project Name")
+ .description("Fetch only objects in this project. Case sensitive.")
+ .required(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(
+ PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_PROJECT_MEMBERS,
+ AV_COLLECT_STORIES,
+ AV_COLLECT_PROJECT_STATUS_UPDATES,
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ AV_COLLECT_PROJECT_EVENTS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_SECTION = new PropertyDescriptor.Builder()
+ .name(ASANA_SECTION_NAME)
+ .displayName("Section Name")
+ .description("Fetch only objects in this section. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TASKS, AV_COLLECT_TASK_ATTACHMENTS, AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TAG = new PropertyDescriptor.Builder()
+ .name(ASANA_TAG_NAME)
+ .displayName("Tag")
+ .description("Fetch only objects having this tag. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TASKS, AV_COLLECT_TASK_ATTACHMENTS, AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TEAM_NAME = new PropertyDescriptor.Builder()
+ .name(ASANA_TEAM_NAME)
+ .displayName("Team")
+ .description("Team name. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TEAM_MEMBERS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OUTPUT_BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name(ASANA_OUTPUT_BATCH_SIZE)
+ .displayName("Output Batch Size")
+ .description("The number of items batched together in a single Flow File. If set to 1 (default), then each item is"
+ + " transferred in a separate Flow File and each will have an asana.gid attribute, to help identifying"
+ + " the fetched item on the server side, if needed. If the batch size is greater than 1, then the"
+ + " specified amount of items are batched together in a single Flow File as a Json array, and the"
+ + " Flow Files won't have the asana.gid attribute.")
+ .defaultValue("1")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+
+ protected static final List<PropertyDescriptor> DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ PROP_ASANA_CONTROLLER_SERVICE,
+ PROP_ASANA_OBJECT_TYPE,
+ PROP_ASANA_PROJECT,
+ PROP_ASANA_SECTION,
+ PROP_ASANA_TEAM_NAME,
+ PROP_ASANA_TAG,
+ PROP_ASANA_OUTPUT_BATCH_SIZE
+ ));
+
+ protected static final Relationship REL_NEW = new Relationship.Builder()
+ .name(REL_NAME_NEW)
+ .description("Newly collected objects are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_UPDATED = new Relationship.Builder()
+ .name(REL_NAME_UPDATED)
+ .description("Objects that have already been collected earlier, but were updated since, are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_REMOVED = new Relationship.Builder()
+ .name(REL_NAME_REMOVED)
+ .description("Notification about deleted objects are routed to this relationship. "
+ + "Flow files will not have any payload. IDs of the resources no longer exist "
+ + "are carried by the asana.gid attribute of the generated FlowFiles.")
+ .build();
+
+ protected static final Set<Relationship> RELATIONSHIPS = Collections.unmodifiableSet(new HashSet<>(Arrays.asList(
+ REL_NEW,
+ REL_UPDATED,
+ REL_REMOVED
+ )));
+
+ private static final Scope STATE_STORAGE_SCOPE = Scope.CLUSTER;
+
+ private volatile AsanaObjectFetcher objectFetcher;
+ private volatile Integer batchSize;
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return RELATIONSHIPS;
+ }
+
+ @Override
+ public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return DESCRIPTORS;
+ }
+
+ @OnScheduled
+ public synchronized void onScheduled(final ProcessContext context) throws InitializationException {
+ AsanaClientProviderService controllerService = context.getProperty(PROP_ASANA_CONTROLLER_SERVICE).asControllerService(AsanaClientProviderService.class);
+ AsanaClient client = controllerService.createClient();
+ batchSize = context.getProperty(PROP_ASANA_OUTPUT_BATCH_SIZE).asInteger();
+
+ try {
+ getLogger().debug("Initializing object fetcher...");
+ objectFetcher = createObjectFetcher(context, client);
+ } catch (Exception e) {
+ throw new InitializationException(e);
+ }
+ }
+
+ @Override
+ public synchronized void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ try {
+ Map<String, String> state = recoverState(context).orElse(Collections.emptyMap());
+ getLogger().debug("Attempting to load state: {}", state);
+ objectFetcher.loadState(state);
+ } catch (Exception e) {
+ getLogger().info("Failed to recover state. Falling back to clean start.");
+ objectFetcher.clearState();
+ }
+ getLogger().debug("Initial state: {}", objectFetcher.saveState());
+
+ Collection<FlowFile> newItems = new ArrayList<>();
+ Collection<FlowFile> updatedItems = new ArrayList<>();
+ Collection<FlowFile> removedItems = new ArrayList<>();
+ Map<AsanaObjectState, Collection<FlowFile>> flowFiles = new HashMap<>();
+ flowFiles.put(AsanaObjectState.NEW, newItems);
+ flowFiles.put(AsanaObjectState.UPDATED, updatedItems);
+ flowFiles.put(AsanaObjectState.REMOVED, removedItems);
+
+ List<AsanaObject> allObjects = new ArrayList<>();
+
+ AsanaObject nextObject;
+ while ((nextObject = objectFetcher.fetchNext()) != null) {
+ allObjects.add(nextObject);
+ }
+
+ Map<AsanaObjectState, List<AsanaObject>> allObjectsByState = allObjects.stream()
+ .collect(groupingBy(AsanaObject::getState));
+
+ if (batchSize == 1) {
+ allObjectsByState
+ .forEach((asanaObjectState, asanaObjects) -> asanaObjects.forEach(
+ asanaObject -> {
+ final Map<String, String> attributes = new HashMap<>(2);
+ attributes.put(CoreAttributes.MIME_TYPE.key(), ContentType.APPLICATION_JSON.getMimeType());
+ attributes.put(ASANA_GID, asanaObject.getGid());
+ FlowFile flowFile = createFlowFileWithStringPayload(session, asanaObject.getContent());
+ flowFile = session.putAllAttributes(flowFile, attributes);
+ flowFiles.get(asanaObject.getState()).add(flowFile);
+ }
+ ));
+ } else {
+ allObjectsByState
+ .forEach((asanaObjectState, asanaObjects) -> partition(asanaObjects, batchSize).forEach(
+ asanaObjectsInPartition -> {
+ FlowFile flowFile = createFlowFileWithStringPayload(session, format("[%s]",
+ asanaObjectsInPartition.stream().map(AsanaObject::getContent)
+ .collect(joining(","))));
+ flowFile = session.putAllAttributes(flowFile,
+ singletonMap(CoreAttributes.MIME_TYPE.key(),
+ ContentType.APPLICATION_JSON.getMimeType()));
+ flowFiles.get(asanaObjectState).add(flowFile);
+ }
+ ));
+ }
+
+ if (flowFiles.values().stream().allMatch(Collection::isEmpty)) {
+ context.yield();
+ getLogger().debug("Yielding, as there are no new FlowFiles.");
Review Comment:
I believe we can return at this point.
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,457 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import static java.lang.String.format;
+import static java.util.Collections.singletonMap;
+import static java.util.stream.Collectors.groupingBy;
+import static java.util.stream.Collectors.joining;
+import static org.apache.commons.collections4.ListUtils.partition;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientProviderService;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaObjectState;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+@TriggerSerially
+@PrimaryNodeOnly
+@Stateful(scopes = {Scope.CLUSTER}, description = "Fingerprints of items in the last successful query are stored in order to enable incremental loading and change detection.")
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.")
+@Tags({"asana", "source", "ingest"})
+@CapabilityDescription("This processor collects data from Asana")
+public class GetAsanaObject extends AbstractProcessor {
+
+ protected static final String ASANA_GID = "asana.gid";
+ protected static final String AV_NAME_COLLECT_TASKS = "asana-collect-tasks";
+ protected static final String AV_NAME_COLLECT_TASK_ATTACHMENTS = "asana-collect-task-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECTS = "asana-collect-projects";
+ protected static final String AV_NAME_COLLECT_TAGS = "asana-collect-tags";
+ protected static final String AV_NAME_COLLECT_USERS = "asana-collect-users";
+ protected static final String AV_NAME_COLLECT_PROJECT_MEMBERS = "asana-collect-project-members";
+ protected static final String AV_NAME_COLLECT_TEAMS = "asana-collect-teams";
+ protected static final String AV_NAME_COLLECT_TEAM_MEMBERS = "asana-collect-team-members";
+ protected static final String AV_NAME_COLLECT_STORIES = "asana-collect-stories";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_UPDATES = "asana-collect-project-status-updates";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS = "asana-collect-project-status-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECT_EVENTS = "asana-collect-project-events";
+ protected static final String ASANA_CONTROLLER_SERVICE = "asana-controller-service";
+ protected static final String ASANA_OBJECT_TYPE = "asana-object-type";
+ protected static final String ASANA_PROJECT_NAME = "asana-project-name";
+ protected static final String ASANA_SECTION_NAME = "asana-section-name";
+ protected static final String ASANA_TAG_NAME = "asana-tag-name";
+ protected static final String ASANA_TEAM_NAME = "asana-team-name";
+ protected static final String ASANA_OUTPUT_BATCH_SIZE = "asana-output-batch-size";
+ protected static final String REL_NAME_NEW = "new";
+ protected static final String REL_NAME_UPDATED = "updated";
+ protected static final String REL_NAME_REMOVED = "removed";
+
+ protected static final AllowableValue AV_COLLECT_TASKS = new AllowableValue(
+ AV_NAME_COLLECT_TASKS,
+ "Tasks",
+ "Collect tasks matching to the specified conditions.");
+
+ protected static final AllowableValue AV_COLLECT_TASK_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_TASK_ATTACHMENTS,
+ "Task Attachments",
+ "Collect attached files of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECTS,
+ "Projects",
+ "Collect projects of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TAGS = new AllowableValue(
+ AV_NAME_COLLECT_TAGS,
+ "Tags",
+ "Collect tags of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_USERS = new AllowableValue(
+ AV_NAME_COLLECT_USERS,
+ "Users",
+ "Collect users assigned to the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_MEMBERS,
+ "Members of a Project",
+ "Collect users assigned to the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAMS = new AllowableValue(
+ AV_NAME_COLLECT_TEAMS,
+ "Teams",
+ "Collect teams of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAM_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_TEAM_MEMBERS,
+ "Team Members",
+ "Collect users assigned to the specified team."
+ );
+
+ protected static final AllowableValue AV_COLLECT_STORIES = new AllowableValue(
+ AV_NAME_COLLECT_STORIES,
+ "Stories of Tasks",
+ "Collect stories (comments) of of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_UPDATES = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_UPDATES,
+ "Status Updates of a Project",
+ "Collect status updates of the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ "Attachments of Status Updates",
+ "Collect attached files of project status updates."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_EVENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_EVENTS,
+ "Events of a Project",
+ "Collect various events happening on the specified project and on its' tasks."
+ );
+
+ protected static final PropertyDescriptor PROP_ASANA_CONTROLLER_SERVICE = new PropertyDescriptor.Builder()
+ .name(ASANA_CONTROLLER_SERVICE)
+ .displayName("Asana Controller Service")
+ .description("Specify which controller service to use for accessing Asana.")
+ .required(true)
+ .identifiesControllerService(AsanaClientProviderService.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OBJECT_TYPE = new PropertyDescriptor.Builder()
+ .name(ASANA_OBJECT_TYPE)
+ .displayName("Object Type")
+ .description("Specify what kind of objects to be collected from Asana")
+ .required(true)
+ .allowableValues(
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_PROJECTS,
+ AV_COLLECT_TAGS,
+ AV_COLLECT_USERS,
+ AV_COLLECT_PROJECT_MEMBERS,
+ AV_COLLECT_TEAMS,
+ AV_COLLECT_TEAM_MEMBERS,
+ AV_COLLECT_STORIES,
+ AV_COLLECT_PROJECT_STATUS_UPDATES,
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ AV_COLLECT_PROJECT_EVENTS)
+ .defaultValue(AV_COLLECT_TASKS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_PROJECT = new PropertyDescriptor.Builder()
+ .name(ASANA_PROJECT_NAME)
+ .displayName("Project Name")
+ .description("Fetch only objects in this project. Case sensitive.")
+ .required(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(
+ PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_PROJECT_MEMBERS,
+ AV_COLLECT_STORIES,
+ AV_COLLECT_PROJECT_STATUS_UPDATES,
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ AV_COLLECT_PROJECT_EVENTS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_SECTION = new PropertyDescriptor.Builder()
+ .name(ASANA_SECTION_NAME)
+ .displayName("Section Name")
+ .description("Fetch only objects in this section. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TASKS, AV_COLLECT_TASK_ATTACHMENTS, AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TAG = new PropertyDescriptor.Builder()
+ .name(ASANA_TAG_NAME)
+ .displayName("Tag")
+ .description("Fetch only objects having this tag. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TASKS, AV_COLLECT_TASK_ATTACHMENTS, AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TEAM_NAME = new PropertyDescriptor.Builder()
+ .name(ASANA_TEAM_NAME)
+ .displayName("Team")
+ .description("Team name. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TEAM_MEMBERS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OUTPUT_BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name(ASANA_OUTPUT_BATCH_SIZE)
+ .displayName("Output Batch Size")
+ .description("The number of items batched together in a single Flow File. If set to 1 (default), then each item is"
+ + " transferred in a separate Flow File and each will have an asana.gid attribute, to help identifying"
+ + " the fetched item on the server side, if needed. If the batch size is greater than 1, then the"
+ + " specified amount of items are batched together in a single Flow File as a Json array, and the"
+ + " Flow Files won't have the asana.gid attribute.")
+ .defaultValue("1")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+
+ protected static final List<PropertyDescriptor> DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ PROP_ASANA_CONTROLLER_SERVICE,
+ PROP_ASANA_OBJECT_TYPE,
+ PROP_ASANA_PROJECT,
+ PROP_ASANA_SECTION,
+ PROP_ASANA_TEAM_NAME,
+ PROP_ASANA_TAG,
+ PROP_ASANA_OUTPUT_BATCH_SIZE
+ ));
+
+ protected static final Relationship REL_NEW = new Relationship.Builder()
+ .name(REL_NAME_NEW)
+ .description("Newly collected objects are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_UPDATED = new Relationship.Builder()
+ .name(REL_NAME_UPDATED)
+ .description("Objects that have already been collected earlier, but were updated since, are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_REMOVED = new Relationship.Builder()
+ .name(REL_NAME_REMOVED)
+ .description("Notification about deleted objects are routed to this relationship. "
+ + "Flow files will not have any payload. IDs of the resources no longer exist "
+ + "are carried by the asana.gid attribute of the generated FlowFiles.")
+ .build();
+
+ protected static final Set<Relationship> RELATIONSHIPS = Collections.unmodifiableSet(new HashSet<>(Arrays.asList(
+ REL_NEW,
+ REL_UPDATED,
+ REL_REMOVED
+ )));
+
+ private static final Scope STATE_STORAGE_SCOPE = Scope.CLUSTER;
+
+ private volatile AsanaObjectFetcher objectFetcher;
+ private volatile Integer batchSize;
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return RELATIONSHIPS;
+ }
+
+ @Override
+ public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return DESCRIPTORS;
+ }
+
+ @OnScheduled
+ public synchronized void onScheduled(final ProcessContext context) throws InitializationException {
+ AsanaClientProviderService controllerService = context.getProperty(PROP_ASANA_CONTROLLER_SERVICE).asControllerService(AsanaClientProviderService.class);
+ AsanaClient client = controllerService.createClient();
+ batchSize = context.getProperty(PROP_ASANA_OUTPUT_BATCH_SIZE).asInteger();
+
+ try {
+ getLogger().debug("Initializing object fetcher...");
+ objectFetcher = createObjectFetcher(context, client);
+ } catch (Exception e) {
+ throw new InitializationException(e);
+ }
+ }
+
+ @Override
+ public synchronized void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ try {
+ Map<String, String> state = recoverState(context).orElse(Collections.emptyMap());
+ getLogger().debug("Attempting to load state: {}", state);
+ objectFetcher.loadState(state);
+ } catch (Exception e) {
+ getLogger().info("Failed to recover state. Falling back to clean start.");
+ objectFetcher.clearState();
+ }
+ getLogger().debug("Initial state: {}", objectFetcher.saveState());
+
+ Collection<FlowFile> newItems = new ArrayList<>();
+ Collection<FlowFile> updatedItems = new ArrayList<>();
+ Collection<FlowFile> removedItems = new ArrayList<>();
+ Map<AsanaObjectState, Collection<FlowFile>> flowFiles = new HashMap<>();
+ flowFiles.put(AsanaObjectState.NEW, newItems);
+ flowFiles.put(AsanaObjectState.UPDATED, updatedItems);
+ flowFiles.put(AsanaObjectState.REMOVED, removedItems);
+
+ List<AsanaObject> allObjects = new ArrayList<>();
+
+ AsanaObject nextObject;
+ while ((nextObject = objectFetcher.fetchNext()) != null) {
+ allObjects.add(nextObject);
+ }
+
+ Map<AsanaObjectState, List<AsanaObject>> allObjectsByState = allObjects.stream()
+ .collect(groupingBy(AsanaObject::getState));
+
+ if (batchSize == 1) {
+ allObjectsByState
+ .forEach((asanaObjectState, asanaObjects) -> asanaObjects.forEach(
+ asanaObject -> {
+ final Map<String, String> attributes = new HashMap<>(2);
+ attributes.put(CoreAttributes.MIME_TYPE.key(), ContentType.APPLICATION_JSON.getMimeType());
+ attributes.put(ASANA_GID, asanaObject.getGid());
+ FlowFile flowFile = createFlowFileWithStringPayload(session, asanaObject.getContent());
+ flowFile = session.putAllAttributes(flowFile, attributes);
+ flowFiles.get(asanaObject.getState()).add(flowFile);
+ }
+ ));
+ } else {
+ allObjectsByState
+ .forEach((asanaObjectState, asanaObjects) -> partition(asanaObjects, batchSize).forEach(
+ asanaObjectsInPartition -> {
+ FlowFile flowFile = createFlowFileWithStringPayload(session, format("[%s]",
+ asanaObjectsInPartition.stream().map(AsanaObject::getContent)
+ .collect(joining(","))));
+ flowFile = session.putAllAttributes(flowFile,
+ singletonMap(CoreAttributes.MIME_TYPE.key(),
+ ContentType.APPLICATION_JSON.getMimeType()));
+ flowFiles.get(asanaObjectState).add(flowFile);
+ }
+ ));
+ }
+
+ if (flowFiles.values().stream().allMatch(Collection::isEmpty)) {
+ context.yield();
+ getLogger().debug("Yielding, as there are no new FlowFiles.");
+ } else {
+ session.transfer(newItems, REL_NEW);
+ session.transfer(updatedItems, REL_UPDATED);
+ session.transfer(removedItems, REL_REMOVED);
+ session.commitAsync();
+ }
+ Map<String, String> state = objectFetcher.saveState();
+ try {
+ persistState(state, context);
+ } catch (IOException e) {
+ throw new ProcessException(e);
+ }
Review Comment:
You can handle the state in a transaction with the session if you use `session.setState()` instead of `context.getStateManager().setState()` in `persistState()`. I would also move the exception handling into that method.
The code would look like in that case:
```suggestion
} else {
session.transfer(newItems, REL_NEW);
session.transfer(updatedItems, REL_UPDATED);
session.transfer(removedItems, REL_REMOVED);
Map<String, String> state = objectFetcher.saveState();
persistState(state, session);
session.commitAsync();
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r1042720308
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/utils/PollableAsanaObjectFetcher.java:
##########
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana.utils;
+
+import static java.util.Collections.emptyIterator;
+
+import java.util.Iterator;
+
+public abstract class PollableAsanaObjectFetcher implements AsanaObjectFetcher {
+
+ private Iterator<AsanaObject> pending;
+
+ public PollableAsanaObjectFetcher() {
+ pending = emptyIterator();
+ }
+
+ @Override
+ public AsanaObject fetchNext() {
+ if (!pending.hasNext()) {
+ pending = poll();
+ }
+ return pending.hasNext() ? pending.next() : null;
+ }
Review Comment:
Done in 2f3a60c
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r1030379015
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,457 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import static java.lang.String.format;
+import static java.util.Collections.singletonMap;
+import static java.util.stream.Collectors.groupingBy;
+import static java.util.stream.Collectors.joining;
+import static org.apache.commons.collections4.ListUtils.partition;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientProviderService;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaObjectState;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+@TriggerSerially
+@PrimaryNodeOnly
+@Stateful(scopes = {Scope.CLUSTER}, description = "Fingerprints of items in the last successful query are stored in order to enable incremental loading and change detection.")
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.")
+@Tags({"asana", "source", "ingest"})
+@CapabilityDescription("This processor collects data from Asana")
+public class GetAsanaObject extends AbstractProcessor {
+
+ protected static final String ASANA_GID = "asana.gid";
+ protected static final String AV_NAME_COLLECT_TASKS = "asana-collect-tasks";
+ protected static final String AV_NAME_COLLECT_TASK_ATTACHMENTS = "asana-collect-task-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECTS = "asana-collect-projects";
+ protected static final String AV_NAME_COLLECT_TAGS = "asana-collect-tags";
+ protected static final String AV_NAME_COLLECT_USERS = "asana-collect-users";
+ protected static final String AV_NAME_COLLECT_PROJECT_MEMBERS = "asana-collect-project-members";
+ protected static final String AV_NAME_COLLECT_TEAMS = "asana-collect-teams";
+ protected static final String AV_NAME_COLLECT_TEAM_MEMBERS = "asana-collect-team-members";
+ protected static final String AV_NAME_COLLECT_STORIES = "asana-collect-stories";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_UPDATES = "asana-collect-project-status-updates";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS = "asana-collect-project-status-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECT_EVENTS = "asana-collect-project-events";
+ protected static final String ASANA_CONTROLLER_SERVICE = "asana-controller-service";
+ protected static final String ASANA_OBJECT_TYPE = "asana-object-type";
+ protected static final String ASANA_PROJECT_NAME = "asana-project-name";
+ protected static final String ASANA_SECTION_NAME = "asana-section-name";
+ protected static final String ASANA_TAG_NAME = "asana-tag-name";
+ protected static final String ASANA_TEAM_NAME = "asana-team-name";
+ protected static final String ASANA_OUTPUT_BATCH_SIZE = "asana-output-batch-size";
+ protected static final String REL_NAME_NEW = "new";
+ protected static final String REL_NAME_UPDATED = "updated";
+ protected static final String REL_NAME_REMOVED = "removed";
+
+ protected static final AllowableValue AV_COLLECT_TASKS = new AllowableValue(
+ AV_NAME_COLLECT_TASKS,
+ "Tasks",
+ "Collect tasks matching to the specified conditions.");
+
+ protected static final AllowableValue AV_COLLECT_TASK_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_TASK_ATTACHMENTS,
+ "Task Attachments",
+ "Collect attached files of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECTS,
+ "Projects",
+ "Collect projects of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TAGS = new AllowableValue(
+ AV_NAME_COLLECT_TAGS,
+ "Tags",
+ "Collect tags of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_USERS = new AllowableValue(
+ AV_NAME_COLLECT_USERS,
+ "Users",
+ "Collect users assigned to the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_MEMBERS,
+ "Members of a Project",
+ "Collect users assigned to the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAMS = new AllowableValue(
+ AV_NAME_COLLECT_TEAMS,
+ "Teams",
+ "Collect teams of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAM_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_TEAM_MEMBERS,
+ "Team Members",
+ "Collect users assigned to the specified team."
+ );
+
+ protected static final AllowableValue AV_COLLECT_STORIES = new AllowableValue(
+ AV_NAME_COLLECT_STORIES,
+ "Stories of Tasks",
+ "Collect stories (comments) of of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_UPDATES = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_UPDATES,
+ "Status Updates of a Project",
+ "Collect status updates of the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ "Attachments of Status Updates",
+ "Collect attached files of project status updates."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_EVENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_EVENTS,
+ "Events of a Project",
+ "Collect various events happening on the specified project and on its' tasks."
+ );
+
+ protected static final PropertyDescriptor PROP_ASANA_CONTROLLER_SERVICE = new PropertyDescriptor.Builder()
+ .name(ASANA_CONTROLLER_SERVICE)
+ .displayName("Asana Controller Service")
+ .description("Specify which controller service to use for accessing Asana.")
+ .required(true)
+ .identifiesControllerService(AsanaClientProviderService.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OBJECT_TYPE = new PropertyDescriptor.Builder()
+ .name(ASANA_OBJECT_TYPE)
+ .displayName("Object Type")
+ .description("Specify what kind of objects to be collected from Asana")
+ .required(true)
+ .allowableValues(
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_PROJECTS,
+ AV_COLLECT_TAGS,
+ AV_COLLECT_USERS,
+ AV_COLLECT_PROJECT_MEMBERS,
+ AV_COLLECT_TEAMS,
+ AV_COLLECT_TEAM_MEMBERS,
+ AV_COLLECT_STORIES,
+ AV_COLLECT_PROJECT_STATUS_UPDATES,
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ AV_COLLECT_PROJECT_EVENTS)
+ .defaultValue(AV_COLLECT_TASKS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_PROJECT = new PropertyDescriptor.Builder()
+ .name(ASANA_PROJECT_NAME)
+ .displayName("Project Name")
+ .description("Fetch only objects in this project. Case sensitive.")
+ .required(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(
+ PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_PROJECT_MEMBERS,
+ AV_COLLECT_STORIES,
+ AV_COLLECT_PROJECT_STATUS_UPDATES,
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ AV_COLLECT_PROJECT_EVENTS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_SECTION = new PropertyDescriptor.Builder()
+ .name(ASANA_SECTION_NAME)
+ .displayName("Section Name")
+ .description("Fetch only objects in this section. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TASKS, AV_COLLECT_TASK_ATTACHMENTS, AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TAG = new PropertyDescriptor.Builder()
+ .name(ASANA_TAG_NAME)
+ .displayName("Tag")
+ .description("Fetch only objects having this tag. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TASKS, AV_COLLECT_TASK_ATTACHMENTS, AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TEAM_NAME = new PropertyDescriptor.Builder()
+ .name(ASANA_TEAM_NAME)
+ .displayName("Team")
+ .description("Team name. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TEAM_MEMBERS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OUTPUT_BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name(ASANA_OUTPUT_BATCH_SIZE)
+ .displayName("Output Batch Size")
+ .description("The number of items batched together in a single Flow File. If set to 1 (default), then each item is"
+ + " transferred in a separate Flow File and each will have an asana.gid attribute, to help identifying"
+ + " the fetched item on the server side, if needed. If the batch size is greater than 1, then the"
+ + " specified amount of items are batched together in a single Flow File as a Json array, and the"
+ + " Flow Files won't have the asana.gid attribute.")
+ .defaultValue("1")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+
+ protected static final List<PropertyDescriptor> DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ PROP_ASANA_CONTROLLER_SERVICE,
+ PROP_ASANA_OBJECT_TYPE,
+ PROP_ASANA_PROJECT,
+ PROP_ASANA_SECTION,
+ PROP_ASANA_TEAM_NAME,
+ PROP_ASANA_TAG,
+ PROP_ASANA_OUTPUT_BATCH_SIZE
+ ));
+
+ protected static final Relationship REL_NEW = new Relationship.Builder()
+ .name(REL_NAME_NEW)
+ .description("Newly collected objects are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_UPDATED = new Relationship.Builder()
+ .name(REL_NAME_UPDATED)
+ .description("Objects that have already been collected earlier, but were updated since, are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_REMOVED = new Relationship.Builder()
+ .name(REL_NAME_REMOVED)
+ .description("Notification about deleted objects are routed to this relationship. "
+ + "Flow files will not have any payload. IDs of the resources no longer exist "
+ + "are carried by the asana.gid attribute of the generated FlowFiles.")
+ .build();
+
+ protected static final Set<Relationship> RELATIONSHIPS = Collections.unmodifiableSet(new HashSet<>(Arrays.asList(
+ REL_NEW,
+ REL_UPDATED,
+ REL_REMOVED
+ )));
+
+ private static final Scope STATE_STORAGE_SCOPE = Scope.CLUSTER;
+
+ private volatile AsanaObjectFetcher objectFetcher;
+ private volatile Integer batchSize;
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return RELATIONSHIPS;
+ }
+
+ @Override
+ public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return DESCRIPTORS;
+ }
+
+ @OnScheduled
+ public synchronized void onScheduled(final ProcessContext context) throws InitializationException {
+ AsanaClientProviderService controllerService = context.getProperty(PROP_ASANA_CONTROLLER_SERVICE).asControllerService(AsanaClientProviderService.class);
+ AsanaClient client = controllerService.createClient();
+ batchSize = context.getProperty(PROP_ASANA_OUTPUT_BATCH_SIZE).asInteger();
+
+ try {
+ getLogger().debug("Initializing object fetcher...");
+ objectFetcher = createObjectFetcher(context, client);
+ } catch (Exception e) {
+ throw new InitializationException(e);
+ }
+ }
+
+ @Override
+ public synchronized void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ try {
+ Map<String, String> state = recoverState(context).orElse(Collections.emptyMap());
+ getLogger().debug("Attempting to load state: {}", state);
+ objectFetcher.loadState(state);
+ } catch (Exception e) {
+ getLogger().info("Failed to recover state. Falling back to clean start.");
+ objectFetcher.clearState();
+ }
+ getLogger().debug("Initial state: {}", objectFetcher.saveState());
+
+ Collection<FlowFile> newItems = new ArrayList<>();
+ Collection<FlowFile> updatedItems = new ArrayList<>();
+ Collection<FlowFile> removedItems = new ArrayList<>();
+ Map<AsanaObjectState, Collection<FlowFile>> flowFiles = new HashMap<>();
+ flowFiles.put(AsanaObjectState.NEW, newItems);
+ flowFiles.put(AsanaObjectState.UPDATED, updatedItems);
+ flowFiles.put(AsanaObjectState.REMOVED, removedItems);
+
+ List<AsanaObject> allObjects = new ArrayList<>();
+
+ AsanaObject nextObject;
+ while ((nextObject = objectFetcher.fetchNext()) != null) {
+ allObjects.add(nextObject);
+ }
+
+ Map<AsanaObjectState, List<AsanaObject>> allObjectsByState = allObjects.stream()
+ .collect(groupingBy(AsanaObject::getState));
+
+ if (batchSize == 1) {
+ allObjectsByState
+ .forEach((asanaObjectState, asanaObjects) -> asanaObjects.forEach(
+ asanaObject -> {
+ final Map<String, String> attributes = new HashMap<>(2);
+ attributes.put(CoreAttributes.MIME_TYPE.key(), ContentType.APPLICATION_JSON.getMimeType());
+ attributes.put(ASANA_GID, asanaObject.getGid());
+ FlowFile flowFile = createFlowFileWithStringPayload(session, asanaObject.getContent());
+ flowFile = session.putAllAttributes(flowFile, attributes);
+ flowFiles.get(asanaObject.getState()).add(flowFile);
+ }
+ ));
+ } else {
+ allObjectsByState
+ .forEach((asanaObjectState, asanaObjects) -> partition(asanaObjects, batchSize).forEach(
+ asanaObjectsInPartition -> {
+ FlowFile flowFile = createFlowFileWithStringPayload(session, format("[%s]",
+ asanaObjectsInPartition.stream().map(AsanaObject::getContent)
+ .collect(joining(","))));
+ flowFile = session.putAllAttributes(flowFile,
+ singletonMap(CoreAttributes.MIME_TYPE.key(),
+ ContentType.APPLICATION_JSON.getMimeType()));
+ flowFiles.get(asanaObjectState).add(flowFile);
+ }
+ ));
+ }
+
+ if (flowFiles.values().stream().allMatch(Collection::isEmpty)) {
+ context.yield();
+ getLogger().debug("Yielding, as there are no new FlowFiles.");
+ } else {
+ session.transfer(newItems, REL_NEW);
+ session.transfer(updatedItems, REL_UPDATED);
+ session.transfer(removedItems, REL_REMOVED);
+ session.commitAsync();
+ }
+ Map<String, String> state = objectFetcher.saveState();
+ try {
+ persistState(state, context);
+ } catch (IOException e) {
+ throw new ProcessException(e);
+ }
Review Comment:
Tried to apply this suggestion in df3c96e . but since `[AsanaProjectEventFetcher](https://github.com/apache/nifi/blob/126b1b5aaf2158e6c1f68854bb1058f0971d3578/nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/utils/AsanaProjectEventFetcher.java)` generates new state even if there are no new `FlowFiles` are generated, it would break, because the new state would not be persisted.
Added 126b1b5 to address this issue. Only parts of your suggestion have been applied.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r1035780114
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,393 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import static java.lang.String.format;
+import static java.util.Collections.singletonMap;
+import static java.util.stream.Collectors.groupingBy;
+import static java.util.stream.Collectors.joining;
+import static org.apache.commons.collections4.ListUtils.partition;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_EVENTS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_MEMBERS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_STATUS_ATTACHMENTS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_STATUS_UPDATES;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_STORIES;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_TASKS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_TASK_ATTACHMENTS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_TEAM_MEMBERS;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientProviderService;
+import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaObjectState;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+@TriggerSerially
+@PrimaryNodeOnly
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.")
+@Tags({"asana", "source", "ingest"})
+@CapabilityDescription("This processor collects data from Asana")
+public class GetAsanaObject extends AbstractProcessor {
+
+ protected static final String ASANA_GID = "asana.gid";
+ protected static final String ASANA_CLIENT_SERVICE = "asana-controller-service";
+ protected static final String DISTRIBUTED_CACHE_SERVICE = "distributed-cache-service";
+ protected static final String ASANA_OBJECT_TYPE = "asana-object-type";
+ protected static final String ASANA_PROJECT_NAME = "asana-project-name";
+ protected static final String ASANA_SECTION_NAME = "asana-section-name";
+ protected static final String ASANA_TAG_NAME = "asana-tag-name";
+ protected static final String ASANA_TEAM_NAME = "asana-team-name";
+ protected static final String ASANA_OUTPUT_BATCH_SIZE = "asana-output-batch-size";
+ protected static final String REL_NAME_NEW = "new";
+ protected static final String REL_NAME_UPDATED = "updated";
+ protected static final String REL_NAME_REMOVED = "removed";
+
+ protected static final PropertyDescriptor PROP_ASANA_CLIENT_SERVICE = new PropertyDescriptor.Builder()
+ .name(ASANA_CLIENT_SERVICE)
+ .displayName("Asana Client Service")
+ .description("Specify which controller service to use for accessing Asana.")
+ .required(true)
+ .identifiesControllerService(AsanaClientProviderService.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_DISTRIBUTED_CACHE_SERVICE = new Builder()
+ .name(DISTRIBUTED_CACHE_SERVICE)
+ .displayName("Distributed Cache Service")
+ .description("Cache service to store fetched item fingerprints. These, from the last successful query"
+ + " are stored, in order to enable incremental loading and change detection.")
+ .required(true)
+ .identifiesControllerService(DistributedMapCacheClient.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OBJECT_TYPE = new PropertyDescriptor.Builder()
+ .name(ASANA_OBJECT_TYPE)
+ .displayName("Object Type")
+ .description("Specify what kind of objects to be collected from Asana")
+ .required(true)
+ .allowableValues(AsanaObjectType.class)
+ .defaultValue(AV_COLLECT_TASKS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_PROJECT = new PropertyDescriptor.Builder()
+ .name(ASANA_PROJECT_NAME)
+ .displayName("Project Name")
+ .description("Fetch only objects in this project. Case sensitive.")
+ .required(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(
+ PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS.getValue(),
+ AV_COLLECT_TASK_ATTACHMENTS.getValue(),
+ AV_COLLECT_PROJECT_MEMBERS.getValue(),
+ AV_COLLECT_STORIES.getValue(),
+ AV_COLLECT_PROJECT_STATUS_UPDATES.getValue(),
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS.getValue(),
+ AV_COLLECT_PROJECT_EVENTS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_SECTION = new PropertyDescriptor.Builder()
+ .name(ASANA_SECTION_NAME)
+ .displayName("Section Name")
+ .description("Fetch only objects in this section. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS.getValue(),
+ AV_COLLECT_TASK_ATTACHMENTS.getValue(),
+ AV_COLLECT_STORIES.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TAG = new PropertyDescriptor.Builder()
+ .name(ASANA_TAG_NAME)
+ .displayName("Tag")
+ .description("Fetch only objects having this tag. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS.getValue(),
+ AV_COLLECT_TASK_ATTACHMENTS.getValue(),
+ AV_COLLECT_STORIES.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TEAM_NAME = new PropertyDescriptor.Builder()
+ .name(ASANA_TEAM_NAME)
+ .displayName("Team")
+ .description("Team name. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TEAM_MEMBERS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OUTPUT_BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name(ASANA_OUTPUT_BATCH_SIZE)
+ .displayName("Output Batch Size")
+ .description("The number of items batched together in a single Flow File. If set to 1 (default), then each item is"
+ + " transferred in a separate Flow File and each will have an asana.gid attribute, to help identifying"
+ + " the fetched item on the server side, if needed. If the batch size is greater than 1, then the"
+ + " specified amount of items are batched together in a single Flow File as a Json array, and the"
+ + " Flow Files won't have the asana.gid attribute.")
+ .defaultValue("1")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+
+ protected static final List<PropertyDescriptor> DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ PROP_ASANA_CLIENT_SERVICE,
+ PROP_DISTRIBUTED_CACHE_SERVICE,
+ PROP_ASANA_OBJECT_TYPE,
+ PROP_ASANA_PROJECT,
+ PROP_ASANA_SECTION,
+ PROP_ASANA_TEAM_NAME,
+ PROP_ASANA_TAG,
+ PROP_ASANA_OUTPUT_BATCH_SIZE
+ ));
+
+ protected static final Relationship REL_NEW = new Relationship.Builder()
+ .name(REL_NAME_NEW)
+ .description("Newly collected objects are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_UPDATED = new Relationship.Builder()
+ .name(REL_NAME_UPDATED)
+ .description("Objects that have already been collected earlier, but were updated since, are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_REMOVED = new Relationship.Builder()
+ .name(REL_NAME_REMOVED)
+ .description("Notification about deleted objects are routed to this relationship. "
+ + "Flow files will not have any payload. IDs of the resources no longer exist "
+ + "are carried by the asana.gid attribute of the generated FlowFiles.")
+ .build();
+
+ protected static final Set<Relationship> RELATIONSHIPS = Collections.unmodifiableSet(new HashSet<>(Arrays.asList(
+ REL_NEW,
+ REL_UPDATED,
+ REL_REMOVED
+ )));
+
+ private volatile AsanaObjectFetcher objectFetcher;
+ private volatile Integer batchSize;
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return RELATIONSHIPS;
+ }
+
+ @Override
+ public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return DESCRIPTORS;
+ }
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) throws InitializationException {
+ AsanaClientProviderService controllerService = context.getProperty(PROP_ASANA_CLIENT_SERVICE).asControllerService(AsanaClientProviderService.class);
+ AsanaClient client = controllerService.createClient();
+ batchSize = context.getProperty(PROP_ASANA_OUTPUT_BATCH_SIZE).asInteger();
+
+ try {
+ getLogger().debug("Initializing object fetcher...");
+ objectFetcher = createObjectFetcher(context, client);
+ } catch (Exception e) {
+ throw new InitializationException(e);
+ }
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ try {
+ Map<String, String> state = recoverState(context).orElse(Collections.emptyMap());
+ getLogger().debug("Attempting to load state: {}", state);
+ objectFetcher.loadState(state);
+ } catch (Exception e) {
+ getLogger().info("Failed to recover state. Falling back to clean start.");
+ objectFetcher.clearState();
+ }
+ getLogger().debug("Initial state: {}", objectFetcher.saveState());
+
+ Collection<FlowFile> newItems = new ArrayList<>();
+ Collection<FlowFile> updatedItems = new ArrayList<>();
+ Collection<FlowFile> removedItems = new ArrayList<>();
+ Map<AsanaObjectState, Collection<FlowFile>> flowFiles = new HashMap<>();
+ flowFiles.put(AsanaObjectState.NEW, newItems);
+ flowFiles.put(AsanaObjectState.UPDATED, updatedItems);
+ flowFiles.put(AsanaObjectState.REMOVED, removedItems);
+
+ List<AsanaObject> allObjects = new ArrayList<>();
+
+ AsanaObject nextObject;
+ while ((nextObject = objectFetcher.fetchNext()) != null) {
+ allObjects.add(nextObject);
+ }
+
+ Map<AsanaObjectState, List<AsanaObject>> allObjectsByState = allObjects.stream()
+ .collect(groupingBy(AsanaObject::getState));
+
+ if (batchSize == 1) {
+ allObjectsByState
+ .forEach((asanaObjectState, asanaObjects) -> asanaObjects.forEach(
+ asanaObject -> {
+ final Map<String, String> attributes = new HashMap<>(2);
+ attributes.put(CoreAttributes.MIME_TYPE.key(), ContentType.APPLICATION_JSON.getMimeType());
+ attributes.put(ASANA_GID, asanaObject.getGid());
+ FlowFile flowFile = createFlowFileWithStringPayload(session, asanaObject.getContent());
+ flowFile = session.putAllAttributes(flowFile, attributes);
+ flowFiles.get(asanaObject.getState()).add(flowFile);
+ }
+ ));
+ } else {
+ allObjectsByState
+ .forEach((asanaObjectState, asanaObjects) -> partition(asanaObjects, batchSize).forEach(
+ asanaObjectsInPartition -> {
+ FlowFile flowFile = createFlowFileWithStringPayload(session, format("[%s]",
+ asanaObjectsInPartition.stream().map(AsanaObject::getContent)
+ .collect(joining(","))));
+ flowFile = session.putAllAttributes(flowFile,
+ singletonMap(CoreAttributes.MIME_TYPE.key(),
+ ContentType.APPLICATION_JSON.getMimeType()));
+ flowFiles.get(asanaObjectState).add(flowFile);
+ }
+ ));
+ }
+
+ if (flowFiles.values().stream().allMatch(Collection::isEmpty)) {
+ context.yield();
+ getLogger().debug("Yielding, as there are no new FlowFiles.");
+ } else {
+ session.transfer(newItems, REL_NEW);
+ session.transfer(updatedItems, REL_UPDATED);
+ session.transfer(removedItems, REL_REMOVED);
+ }
+
+ Map<String, String> state = objectFetcher.saveState();
+ persistState(state, context);
+
+ getLogger().debug(
+ "New state after transferring {} new, {} updated, and {} removed items: {}",
+ newItems.size(), updatedItems.size(), removedItems.size(), state);
+
+ session.commitAsync();
Review Comment:
Done in 29673c5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r1035778832
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/test/java/org/apache/nifi/processors/asana/AsanaTaskFetcherTest.java:
##########
@@ -0,0 +1,436 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import com.asana.models.Project;
+import com.asana.models.Section;
+import com.asana.models.Tag;
+import com.asana.models.Task;
+import com.google.api.client.util.DateTime;
+import org.apache.groovy.util.Maps;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaObjectState;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.mockito.Mock;
+import org.mockito.junit.jupiter.MockitoExtension;
+import org.mockito.junit.jupiter.MockitoSettings;
+import org.mockito.quality.Strictness;
+
+import static java.util.Collections.emptyMap;
+import static java.util.Collections.singletonMap;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.atLeastOnce;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.verifyNoMoreInteractions;
+import static org.mockito.Mockito.when;
+
+@ExtendWith(MockitoExtension.class)
+@MockitoSettings(strictness = Strictness.LENIENT)
+public class AsanaTaskFetcherTest {
+
+ @Mock
+ private AsanaClient client;
+ private Project project;
+ private Section section;
+ private Tag tag;
+
+ @BeforeEach
+ public void init() {
+ project = new Project();
+ project.gid = "123";
+ project.modifiedAt = new DateTime(123456789);
+ project.name = "My Project";
+
+ when(client.getProjectByName(project.name)).thenReturn(project);
+
+ section = new Section();
+ section.gid = "456";
+ section.project = project;
+ section.name = "Some section";
+ section.createdAt = new DateTime(123456789);
+
+ when(client.getSections(project)).thenReturn(singletonMap(section.gid, section));
+ when(client.getSectionByName(project, section.name)).thenReturn(section);
+
+ tag = new Tag();
+ tag.gid = "9876";
+ tag.name = "Foo";
+ tag.createdAt = new DateTime(123456789);
+
+ when(client.getTags()).thenReturn(singletonMap(tag.gid, tag));
+ }
+
+ @Test
+ public void testNoObjectsFetchedWhenNoTasksReturned() {
+ when(client.getTasks(any(Project.class))).thenReturn(emptyMap());
+
+ final AsanaObjectFetcher fetcher = new AsanaTaskFetcher(client, project.name, null, null);
+ assertNull(fetcher.fetchNext());
+
+ verify(client, atLeastOnce()).getProjectByName(project.name);
+ verify(client, times(1)).getTasks(project);
+ verifyNoMoreInteractions(client);
+ }
+
+ @Test
+ public void testNoObjectsFetchedWhenNoTasksReturnedBySection() {
+ when(client.getTasks(any(Section.class))).thenReturn(emptyMap());
+
+ final AsanaObjectFetcher fetcher = new AsanaTaskFetcher(client, project.name, section.name, null);
+ assertNull(fetcher.fetchNext());
+
+ verify(client, atLeastOnce()).getProjectByName(project.name);
+ verify(client, atLeastOnce()).getSectionByName(project, section.name);
+ verify(client, times(1)).getTasks(section);
+ verifyNoMoreInteractions(client);
+ }
+
+ @Test
+ public void testNoObjectsFetchedWhenNoTasksReturnedByTag() {
+ when(client.getTasks(any(Project.class))).thenReturn(emptyMap());
+
+ final AsanaObjectFetcher fetcher = new AsanaTaskFetcher(client, project.name, null, tag.name);
+ assertNull(fetcher.fetchNext());
+
+ verify(client, atLeastOnce()).getProjectByName(project.name);
+ verify(client, atLeastOnce()).getTags();
+ verify(client, times(1)).getTasks(project);
+ verify(client, times(1)).getTasks(tag);
+ verifyNoMoreInteractions(client);
+ }
+
+ @Test
+ public void testSingleTaskFetched() {
+ final Task task = new Task();
+ task.gid = "1234";
+ task.name = "My first task";
+ task.modifiedAt = new DateTime(123456789);
+
+ when(client.getTasks(any(Project.class))).thenReturn(singletonMap(task.gid, task));
+
+ final AsanaObjectFetcher fetcher = new AsanaTaskFetcher(client, project.name, null, null);
+ final AsanaObject object = fetcher.fetchNext();
+
+ assertEquals(AsanaObjectState.NEW, object.getState());
+ assertEquals(task.gid, object.getGid());
+
+ verify(client, atLeastOnce()).getProjectByName(project.name);
+ verify(client, times(1)).getTasks(project);
+ verifyNoMoreInteractions(client);
+ }
+
+ @Test
+ public void testSingleTaskFetchedBySection() {
+ final Task task = new Task();
+ task.gid = "1234";
+ task.name = "My first task";
+ task.modifiedAt = new DateTime(123456789);
+
+ when(client.getTasks(any(Section.class))).thenReturn(singletonMap(task.gid, task));
+
+ final AsanaObjectFetcher fetcher = new AsanaTaskFetcher(client, project.name, section.name, null);
+ final AsanaObject object = fetcher.fetchNext();
+
+ assertEquals(AsanaObjectState.NEW, object.getState());
+ assertEquals(task.gid, object.getGid());
+
+ verify(client, atLeastOnce()).getProjectByName(project.name);
+ verify(client, atLeastOnce()).getSectionByName(project, section.name);
+ verify(client, times(1)).getTasks(section);
+ verifyNoMoreInteractions(client);
+ }
+
+ @Test
+ public void testSingleTaskFetchedByTag() {
+ final Task task = new Task();
+ task.gid = "1234";
+ task.name = "My first task";
+ task.modifiedAt = new DateTime(123456789);
+
+ when(client.getTasks(any(Project.class))).thenReturn(singletonMap(task.gid, task));
+ when(client.getTasks(any(Tag.class))).thenReturn(singletonMap(task.gid, task));
+
+ final AsanaObjectFetcher fetcher = new AsanaTaskFetcher(client, project.name, null, tag.name);
+ final AsanaObject object = fetcher.fetchNext();
+
+ assertEquals(AsanaObjectState.NEW, object.getState());
+ assertEquals(task.gid, object.getGid());
+
+ verify(client, atLeastOnce()).getProjectByName(project.name);
+ verify(client, atLeastOnce()).getTags();
+ verify(client, times(1)).getTasks(project);
+ verify(client, times(1)).getTasks(tag);
+ verifyNoMoreInteractions(client);
+ }
+
+ @Test
+ public void testNoTaskFetchedByNonMatchingTag() {
+ final Task task1 = new Task();
+ task1.gid = "1234";
+ task1.name = "My first task";
+ task1.modifiedAt = new DateTime(123456789);
+
+ final Task task2 = new Task();
+ task2.gid = "5678";
+ task2.name = "My other task";
+ task2.modifiedAt = new DateTime(123456789);
+
+ when(client.getTasks(any(Project.class))).thenReturn(singletonMap(task1.gid, task1));
+ when(client.getTasks(any(Tag.class))).thenReturn(singletonMap(task2.gid, task2));
+
+ final AsanaObjectFetcher fetcher = new AsanaTaskFetcher(client, project.name, null, tag.name);
+ assertNull(fetcher.fetchNext());
+
+ verify(client, atLeastOnce()).getProjectByName(project.name);
+ verify(client, atLeastOnce()).getTags();
+ verify(client, times(1)).getTasks(project);
+ verify(client, times(1)).getTasks(tag);
+ verifyNoMoreInteractions(client);
+ }
+
+ @Test
+ public void testTaskRemovedFromSection() {
+ final Task task = new Task();
+ task.gid = "1234";
+ task.name = "My first task";
+ task.modifiedAt = new DateTime(123456789);
+
+ when(client.getTasks(any(Section.class))).thenReturn(singletonMap(task.gid, task));
+
+ final AsanaObjectFetcher fetcher = new AsanaTaskFetcher(client, project.name, section.name, null);
+ assertNotNull(fetcher.fetchNext());
+
+ when(client.getTasks(any(Section.class))).thenReturn(emptyMap());
+
+ final AsanaObject object = fetcher.fetchNext();
+ assertEquals(AsanaObjectState.REMOVED, object.getState());
+ assertEquals(task.gid, object.getGid());
+
+ verify(client, atLeastOnce()).getProjectByName(project.name);
+ verify(client, atLeastOnce()).getSectionByName(project, section.name);
+ verify(client, times(2)).getTasks(section);
+ verifyNoMoreInteractions(client);
+ }
+
+ @Test
+ public void testTaskUntagged() {
+ final Task task = new Task();
+ task.gid = "1234";
+ task.name = "My first task";
+ task.modifiedAt = new DateTime(123456789);
+
+ when(client.getTasks(any(Project.class))).thenReturn(singletonMap(task.gid, task));
+ when(client.getTasks(any(Tag.class))).thenReturn(singletonMap(task.gid, task));
+
+ final AsanaObjectFetcher fetcher = new AsanaTaskFetcher(client, project.name, null, tag.name);
+ assertNotNull(fetcher.fetchNext());
+
+ when(client.getTasks(any(Tag.class))).thenReturn(emptyMap());
+
+ final AsanaObject object = fetcher.fetchNext();
+ assertEquals(AsanaObjectState.REMOVED, object.getState());
+ assertEquals(task.gid, object.getGid());
+
+ verify(client, atLeastOnce()).getProjectByName(project.name);
+ verify(client, atLeastOnce()).getTags();
+ verify(client, times(2)).getTasks(project);
+ verify(client, times(2)).getTasks(tag);
+ verifyNoMoreInteractions(client);
+ }
+
+ @Test
+ public void testCollectMultipleTasksWithSameTagAndFilterOutDuplicates() {
+ final Tag anotherTagWithSameName = new Tag();
+ anotherTagWithSameName.gid = "555";
+ anotherTagWithSameName.name = tag.name;
+
+ when(client.getTags()).thenReturn(Maps.of(tag.gid, tag, anotherTagWithSameName.gid, anotherTagWithSameName));
+
+ final Task task1 = new Task();
+ task1.gid = "1234";
+ task1.name = "My first task";
+ task1.modifiedAt = new DateTime(123456789);
+
+ final Task task2 = new Task();
+ task2.gid = "1212";
+ task2.name = "My other task";
+ task2.modifiedAt = new DateTime(234567891);
+
+ final Task task3 = new Task();
+ task3.gid = "333";
+ task3.name = "My third task";
+ task3.modifiedAt = new DateTime(345678912);
+
+ final Task task4 = new Task();
+ task4.gid = "444";
+ task4.name = "A task without tag";
+ task4.modifiedAt = new DateTime(456789123);
+
+ when(client.getTasks(any(Project.class))).thenReturn(Maps.of(task1.gid, task1, task2.gid, task2, task3.gid, task3, task4.gid, task4));
+ when(client.getTasks(tag)).thenReturn(singletonMap(task1.gid, task1));
+ when(client.getTasks(anotherTagWithSameName)).thenReturn(Maps.of(task1.gid, task1, task2.gid, task2, task3.gid, task3));
+
+ final AsanaObjectFetcher fetcher = new AsanaTaskFetcher(client, project.name, null, tag.name);
+
+ final AsanaObject object1 = fetcher.fetchNext();
+ assertEquals(AsanaObjectState.NEW, object1.getState());
+
+ final AsanaObject object2 = fetcher.fetchNext();
+ assertEquals(AsanaObjectState.NEW, object2.getState());
+ assertNotEquals(object1, object2);
+
+ final AsanaObject object3 = fetcher.fetchNext();
+ assertEquals(AsanaObjectState.NEW, object3.getState());
+ assertNotEquals(object1, object3);
+ assertNotEquals(object2, object3);
+
+ assertNull(fetcher.fetchNext());
+
+ verify(client, atLeastOnce()).getProjectByName(project.name);
+ verify(client, atLeastOnce()).getTags();
+ verify(client, times(2)).getTasks(project);
+ verify(client, times(2)).getTasks(tag);
+ verify(client, times(2)).getTasks(anotherTagWithSameName);
+ verifyNoMoreInteractions(client);
+ }
+
+ @Test
+ public void testTaskUpdatedOnlyWhenModificationDateChanges() {
+ final Task task = new Task();
+ task.gid = "1234";
+ task.name = "My first task";
+ task.modifiedAt = new DateTime(123456789);
+
+ when(client.getTasks(any(Project.class))).thenReturn(singletonMap(task.gid, task));
+
+ final AsanaObjectFetcher fetcher = new AsanaTaskFetcher(client, project.name, null, null);
+ assertNotNull(fetcher.fetchNext());
+ assertNull(fetcher.fetchNext());
+
+ task.name = "Update my task";
+ assertNull(fetcher.fetchNext());
+
+ task.modifiedAt = new DateTime(234567891);
+ final AsanaObject object = fetcher.fetchNext();
+
+ assertEquals(AsanaObjectState.UPDATED, object.getState());
+ assertEquals(task.gid, object.getGid());
+
+ verify(client, atLeastOnce()).getProjectByName(project.name);
+ verify(client, times(4)).getTasks(project);
+ verifyNoMoreInteractions(client);
+ }
+
+ @Test
+ public void testRestoreStateAndContinue() {
+ final Task task = new Task();
+ task.gid = "1234";
+ task.name = "My first task";
+ task.modifiedAt = new DateTime(123456789);
+
+ when(client.getTasks(any(Project.class))).thenReturn(singletonMap(task.gid, task));
+
+ final AsanaObjectFetcher fetcher1 = new AsanaTaskFetcher(client, project.name, null, null);
+ assertNotNull(fetcher1.fetchNext());
+
+ final AsanaObjectFetcher fetcher2 = new AsanaTaskFetcher(client, project.name, null, null);
+ fetcher2.loadState(fetcher1.saveState());
+
+ task.modifiedAt = new DateTime(234567891);
+ final AsanaObject object = fetcher2.fetchNext();
+
+ assertEquals(AsanaObjectState.UPDATED, object.getState());
+ assertEquals(task.gid, object.getGid());
+
+ verify(client, atLeastOnce()).getProjectByName(project.name);
+ verify(client, times(2)).getTasks(project);
+ verifyNoMoreInteractions(client);
+ }
+
+ @Test
+ public void testClearState() {
+ final Task task = new Task();
+ task.gid = "1234";
+ task.name = "My first task";
+ task.modifiedAt = new DateTime(123456789);
+
+ when(client.getTasks(any(Project.class))).thenReturn(singletonMap(task.gid, task));
+
+ final AsanaObjectFetcher fetcher = new AsanaTaskFetcher(client, project.name, null, null);
+ assertNotNull(fetcher.fetchNext());
+
+ fetcher.clearState();
+
+ task.modifiedAt = new DateTime(234567891);
+ final AsanaObject object = fetcher.fetchNext();
+
+ assertEquals(AsanaObjectState.NEW, object.getState());
+ assertEquals(task.gid, object.getGid());
+
+ verify(client, atLeastOnce()).getProjectByName(project.name);
+ verify(client, times(2)).getTasks(project);
+ verifyNoMoreInteractions(client);
+ }
+
+ @Test
+ public void testWrongStateForConfigurationThrows() {
+ final Project otherProject = new Project();
+ otherProject.gid = "999";
+ otherProject.name = "Other Project";
+
+ final Section otherSection = new Section();
+ otherSection.gid = "888";
+ otherSection.name = "Other Section";
+
+ final Tag otherTag = new Tag();
+ otherTag.gid = "777";
+ otherTag.name = "Other Tag";
+
+ when(client.getProjectByName(otherProject.name)).thenReturn(otherProject);
+ when(client.getSectionByName(project, otherSection.name)).thenReturn(otherSection);
+
+ final AsanaObjectFetcher fetcher1 = new AsanaTaskFetcher(client, project.name, null, null);
+ final AsanaObjectFetcher fetcher2 = new AsanaTaskFetcher(client, otherProject.name, null, null);
+ assertThrows(RuntimeException.class, () -> fetcher2.loadState(fetcher1.saveState()));
Review Comment:
Done in 46c9391
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r1035777727
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-services-api-nar/src/main/resources/META-INF/NOTICE:
##########
@@ -0,0 +1,108 @@
+nifi-asana-services-api-nar
+Copyright 2015-2022 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+===========================================
+Apache Software License v2
+===========================================
+
+The following binary components are provided under the Apache Software License v2
+
+ (ASLv2) Apache Commons IO
+ The following NOTICE information applies:
+ Apache Commons IO
+ Copyright 2002-2022 The Apache Software Foundation
Review Comment:
First part done in 885f3b6
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r1030297591
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-services/pom.xml:
##########
@@ -0,0 +1,64 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-asana-bundle</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-asana-services</artifactId>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-api</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-mock</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ <scope>test</scope>
+ </dependency>
Review Comment:
Done in c72eeab
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r995755072
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientServiceApi;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@TriggerSerially
+@Stateful(scopes = {Scope.LOCAL}, description = "Fingerprints of items in the last successful query are stored in order to enable incremental loading and change detection.")
Review Comment:
Done in [dc2a050](https://github.com/apache/nifi/pull/6504/commits/dc2a050548a389d89181670371748af741eefd61)
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-services-api/src/main/java/org/apache/nifi/controller/asana/AsanaClientServiceApi.java:
##########
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.controller.asana;
+
+import org.apache.nifi.controller.ControllerService;
+
+/**
+ * This interface represents an API to the controller service of Asana processors.
+ */
+public interface AsanaClientServiceApi extends ControllerService {
Review Comment:
Done in [dc2a050](https://github.com/apache/nifi/pull/6504/commits/dc2a050548a389d89181670371748af741eefd61)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r1014225968
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,429 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientProviderService;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@TriggerSerially
+@PrimaryNodeOnly
+@Stateful(scopes = {Scope.CLUSTER}, description = "Fingerprints of items in the last successful query are stored in order to enable incremental loading and change detection.")
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.")
+@Tags({"asana", "source", "ingest"})
+@CapabilityDescription("This processor collects data from Asana")
+public class GetAsanaObject extends AbstractProcessor {
+
+ protected static final String ASANA_GID = "asana.gid";
+ protected static final String AV_NAME_COLLECT_TASKS = "asana-collect-tasks";
+ protected static final String AV_NAME_COLLECT_TASK_ATTACHMENTS = "asana-collect-task-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECTS = "asana-collect-projects";
+ protected static final String AV_NAME_COLLECT_TAGS = "asana-collect-tags";
+ protected static final String AV_NAME_COLLECT_USERS = "asana-collect-users";
+ protected static final String AV_NAME_COLLECT_PROJECT_MEMBERS = "asana-collect-project-members";
+ protected static final String AV_NAME_COLLECT_TEAMS = "asana-collect-teams";
+ protected static final String AV_NAME_COLLECT_TEAM_MEMBERS = "asana-collect-team-members";
+ protected static final String AV_NAME_COLLECT_STORIES = "asana-collect-stories";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_UPDATES = "asana-collect-project-status-updates";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS = "asana-collect-project-status-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECT_EVENTS = "asana-collect-project-events";
+ protected static final String ASANA_CONTROLLER_SERVICE = "asana-controller-service";
+ protected static final String ASANA_OBJECT_TYPE = "asana-object-type";
+ protected static final String ASANA_PROJECT_NAME = "asana-project-name";
+ protected static final String ASANA_SECTION_NAME = "asana-section-name";
+ protected static final String ASANA_TAG_NAME = "asana-tag-name";
+ protected static final String ASANA_TEAM_NAME = "asana-team-name";
+ protected static final String ASANA_OUTPUT_BATCH_SIZE = "asana-output-batch-size";
+ protected static final String REL_NAME_NEW = "new";
+ protected static final String REL_NAME_UPDATED = "updated";
+ protected static final String REL_NAME_REMOVED = "removed";
+
+ protected static final AllowableValue AV_COLLECT_TASKS = new AllowableValue(
+ AV_NAME_COLLECT_TASKS,
+ "Tasks",
+ "Collect tasks matching to the specified conditions.");
+
+ protected static final AllowableValue AV_COLLECT_TASK_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_TASK_ATTACHMENTS,
+ "Task Attachments",
+ "Collect attached files of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECTS,
+ "Projects",
+ "Collect projects of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TAGS = new AllowableValue(
+ AV_NAME_COLLECT_TAGS,
+ "Tags",
+ "Collect tags of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_USERS = new AllowableValue(
+ AV_NAME_COLLECT_USERS,
+ "Users",
+ "Collect users assigned to the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_MEMBERS,
+ "Members of a Project",
+ "Collect users assigned to the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAMS = new AllowableValue(
+ AV_NAME_COLLECT_TEAMS,
+ "Teams",
+ "Collect teams of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAM_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_TEAM_MEMBERS,
+ "Team Members",
+ "Collect users assigned to the specified team."
+ );
+
+ protected static final AllowableValue AV_COLLECT_STORIES = new AllowableValue(
+ AV_NAME_COLLECT_STORIES,
+ "Stories of Tasks",
+ "Collect stories (comments) of of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_UPDATES = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_UPDATES,
+ "Status Updates of a Project",
+ "Collect status updates of the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ "Attachments of Status Updates",
+ "Collect attached files of project status updates."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_EVENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_EVENTS,
+ "Events of a Project",
+ "Collect various events happening on the specified project and on its' tasks."
+ );
+
+ protected static final PropertyDescriptor PROP_ASANA_CONTROLLER_SERVICE = new PropertyDescriptor.Builder()
+ .name(ASANA_CONTROLLER_SERVICE)
+ .displayName("Asana Controller Service")
+ .description("Specify which controller service to use for accessing Asana.")
+ .required(true)
+ .identifiesControllerService(AsanaClientProviderService.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OBJECT_TYPE = new PropertyDescriptor.Builder()
+ .name(ASANA_OBJECT_TYPE)
+ .displayName("Object Type")
+ .description("Specify what kind of objects to be collected from Asana")
+ .required(true)
+ .allowableValues(
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_PROJECTS,
+ AV_COLLECT_TAGS,
+ AV_COLLECT_USERS,
+ AV_COLLECT_PROJECT_MEMBERS,
+ AV_COLLECT_TEAMS,
+ AV_COLLECT_TEAM_MEMBERS,
+ AV_COLLECT_STORIES,
+ AV_COLLECT_PROJECT_STATUS_UPDATES,
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ AV_COLLECT_PROJECT_EVENTS)
+ .defaultValue(AV_COLLECT_TASKS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_PROJECT = new PropertyDescriptor.Builder()
+ .name(ASANA_PROJECT_NAME)
+ .displayName("Project Name")
+ .description("Fetch only objects in this project. Case sensitive.")
+ .required(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(
+ PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_PROJECT_MEMBERS,
+ AV_COLLECT_STORIES,
+ AV_COLLECT_PROJECT_STATUS_UPDATES,
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ AV_COLLECT_PROJECT_EVENTS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_SECTION = new PropertyDescriptor.Builder()
+ .name(ASANA_SECTION_NAME)
+ .displayName("Section Name")
+ .description("Fetch only objects in this section. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TASKS, AV_COLLECT_TASK_ATTACHMENTS, AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TAG = new PropertyDescriptor.Builder()
+ .name(ASANA_TAG_NAME)
+ .displayName("Tag")
+ .description("Fetch only objects having this tag. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TASKS, AV_COLLECT_TASK_ATTACHMENTS, AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TEAM_NAME = new PropertyDescriptor.Builder()
+ .name(ASANA_TEAM_NAME)
+ .displayName("Team")
+ .description("Team name. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TEAM_MEMBERS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OUTPUT_BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name(ASANA_OUTPUT_BATCH_SIZE)
+ .displayName("Output Batch Size")
+ .description("The number of output FlowFiles to queue before committing the process session. When set to zero, the session will be committed when all result set rows "
+ + "have been processed and the output FlowFiles are ready for transfer to the downstream relationship. For large result sets, this can cause a large burst of FlowFiles "
+ + "to be transferred at the end of processor execution. If this property is set, then when the specified number of FlowFiles are ready for transfer, then the session will "
+ + "be committed, thus releasing the FlowFiles to the downstream relationship.")
+ .defaultValue("0")
+ .required(true)
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+
+ protected static final List<PropertyDescriptor> DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ PROP_ASANA_CONTROLLER_SERVICE,
+ PROP_ASANA_OBJECT_TYPE,
+ PROP_ASANA_PROJECT,
+ PROP_ASANA_SECTION,
+ PROP_ASANA_TEAM_NAME,
+ PROP_ASANA_TAG,
+ PROP_ASANA_OUTPUT_BATCH_SIZE
+ ));
+
+ protected static final Relationship REL_NEW = new Relationship.Builder()
+ .name(REL_NAME_NEW)
+ .description("Newly collected objects are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_UPDATED = new Relationship.Builder()
+ .name(REL_NAME_UPDATED)
+ .description("Objects that have already been collected earlier, but were updated since, are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_REMOVED = new Relationship.Builder()
+ .name(REL_NAME_REMOVED)
+ .description("Notification about deleted objects are routed to this relationship. "
+ + "Flow files will not have any payload. IDs of the resources no longer exist "
+ + "are carried by the asana.gid attribute of the generated FlowFiles.")
+ .build();
+
+ protected static final Set<Relationship> RELATIONSHIPS = Collections.unmodifiableSet(new HashSet<>(Arrays.asList(
+ REL_NEW,
+ REL_UPDATED,
+ REL_REMOVED
+ )));
+
+ private static final Scope STATE_STORAGE_SCOPE = Scope.CLUSTER;
+
+ private volatile AsanaObjectFetcher objectFetcher;
+ private volatile Integer batchSize;
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return RELATIONSHIPS;
+ }
+
+ @Override
+ public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return DESCRIPTORS;
+ }
+
+ @OnScheduled
+ public synchronized void onScheduled(final ProcessContext context) throws InitializationException {
+ AsanaClientProviderService controllerService = context.getProperty(PROP_ASANA_CONTROLLER_SERVICE).asControllerService(AsanaClientProviderService.class);
+ AsanaClient client = controllerService.createClient();
+ batchSize = context.getProperty(PROP_ASANA_OUTPUT_BATCH_SIZE).asInteger();
+
+ try {
+ getLogger().debug("Initializing object fetcher...");
+ objectFetcher = createObjectFetcher(context, client);
+ } catch (Exception e) {
+ throw new InitializationException(e);
+ }
+ }
+
+ @Override
+ public synchronized void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ try {
+ Map<String, String> state = recoverState(context).orElse(Collections.emptyMap());
+ getLogger().debug("Attempting to load state: {}", state);
+ objectFetcher.loadState(state);
+ } catch (Exception e) {
+ getLogger().info("Failed to recover state. Falling back to clean start.");
+ objectFetcher.clearState();
+ }
+ getLogger().debug("Initial state: {}", objectFetcher.saveState());
+
+ Collection<FlowFile> newItems = new ArrayList<>();
+ Collection<FlowFile> updatedItems = new ArrayList<>();
+ Collection<FlowFile> removedItems = new ArrayList<>();
+
+ AsanaObject nextObject;
+ while ((batchSize == 0 || (batchSize > (newItems.size() + updatedItems.size() + removedItems.size()))) && (nextObject = objectFetcher.fetchNext()) != null) {
+ final Map<String, String> attributes = new HashMap<>(2);
+ attributes.put(CoreAttributes.MIME_TYPE.key(), ContentType.APPLICATION_JSON.getMimeType());
+ attributes.put(ASANA_GID, nextObject.getGid());
+ FlowFile flowFile = createFlowFileWithStringPayload(session, nextObject.getContent());
Review Comment:
Thanks for pointing this out. I'll attempt to change the behavior like so.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] asfgit closed pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #6504: NIFI-10618: Add Asana connector
URL: https://github.com/apache/nifi/pull/6504
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] turcsanyip commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
turcsanyip commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r1039539797
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/utils/PollableAsanaObjectFetcher.java:
##########
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana.utils;
+
+import static java.util.Collections.emptyIterator;
+
+import java.util.Iterator;
+
+public abstract class PollableAsanaObjectFetcher implements AsanaObjectFetcher {
+
+ private Iterator<AsanaObject> pending;
+
+ public PollableAsanaObjectFetcher() {
+ pending = emptyIterator();
+ }
+
+ @Override
+ public AsanaObject fetchNext() {
+ if (!pending.hasNext()) {
+ pending = poll();
+ }
+ return pending.hasNext() ? pending.next() : null;
+ }
+
+ protected abstract Iterator<AsanaObject> poll();
Review Comment:
The current call chain is `fetchNext()` => `poll()` => `refreshObjects()` => `fetchXXX()`.
I would somehow try to consolidate these various names because they refer to the same process.
E.g. `fetchNext()` => `fetch()` => `fetchObjects()` => `fetchXXX()`
Not sure but `Pollable-` may not be relevant any more due to the on-the-fly processing and also because it should process a single iteration. Simply `Abstract-` might be a better name.
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/utils/PollableAsanaObjectFetcher.java:
##########
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana.utils;
+
+import static java.util.Collections.emptyIterator;
+
+import java.util.Iterator;
+
+public abstract class PollableAsanaObjectFetcher implements AsanaObjectFetcher {
+
+ private Iterator<AsanaObject> pending;
+
+ public PollableAsanaObjectFetcher() {
+ pending = emptyIterator();
+ }
+
+ @Override
+ public AsanaObject fetchNext() {
+ if (!pending.hasNext()) {
+ pending = poll();
+ }
+ return pending.hasNext() ? pending.next() : null;
+ }
Review Comment:
The `Iterator` should be retrieved only once per `onTrigger` call. The returned `Iterator` can iterate over all the currently available Asana objects.
Now we have a second call (when the first `Iterator` has no more elements) but it unnecessarily queries the same Asana objects again which items will be filtered out on the NiFi side.
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,405 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import static java.lang.String.format;
+import static java.lang.String.join;
+import static java.util.Collections.singletonMap;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_EVENTS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_MEMBERS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_STATUS_ATTACHMENTS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_STATUS_UPDATES;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_STORIES;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_TASKS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_TASK_ATTACHMENTS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_TEAM_MEMBERS;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientProviderService;
+import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaObjectState;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+@TriggerSerially
+@PrimaryNodeOnly
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.")
+@Tags({"asana", "source", "ingest"})
+@CapabilityDescription("This processor collects data from Asana")
+public class GetAsanaObject extends AbstractProcessor {
Review Comment:
Though the processor stores its state in an external cache but it is loaded int memory while the onTrigger is running. This state object can consume high memory and for this reason please add `@SystemResourceConsideration` annotation with `SystemResource.MEMORY`.
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,405 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import static java.lang.String.format;
+import static java.lang.String.join;
+import static java.util.Collections.singletonMap;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_EVENTS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_MEMBERS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_STATUS_ATTACHMENTS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_STATUS_UPDATES;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_STORIES;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_TASKS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_TASK_ATTACHMENTS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_TEAM_MEMBERS;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientProviderService;
+import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaObjectState;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+@TriggerSerially
+@PrimaryNodeOnly
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.")
+@Tags({"asana", "source", "ingest"})
+@CapabilityDescription("This processor collects data from Asana")
+public class GetAsanaObject extends AbstractProcessor {
+
+ protected static final String ASANA_GID = "asana.gid";
+ protected static final String ASANA_CLIENT_SERVICE = "asana-controller-service";
+ protected static final String DISTRIBUTED_CACHE_SERVICE = "distributed-cache-service";
+ protected static final String ASANA_OBJECT_TYPE = "asana-object-type";
+ protected static final String ASANA_PROJECT_NAME = "asana-project-name";
+ protected static final String ASANA_SECTION_NAME = "asana-section-name";
+ protected static final String ASANA_TAG_NAME = "asana-tag-name";
+ protected static final String ASANA_TEAM_NAME = "asana-team-name";
+ protected static final String ASANA_OUTPUT_BATCH_SIZE = "asana-output-batch-size";
+ protected static final String REL_NAME_NEW = "new";
+ protected static final String REL_NAME_UPDATED = "updated";
+ protected static final String REL_NAME_REMOVED = "removed";
+
+ protected static final PropertyDescriptor PROP_ASANA_CLIENT_SERVICE = new PropertyDescriptor.Builder()
+ .name(ASANA_CLIENT_SERVICE)
+ .displayName("Asana Client Service")
+ .description("Specify which controller service to use for accessing Asana.")
+ .required(true)
+ .identifiesControllerService(AsanaClientProviderService.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_DISTRIBUTED_CACHE_SERVICE = new Builder()
+ .name(DISTRIBUTED_CACHE_SERVICE)
+ .displayName("Distributed Cache Service")
+ .description("Cache service to store fetched item fingerprints. These, from the last successful query"
+ + " are stored, in order to enable incremental loading and change detection.")
+ .required(true)
+ .identifiesControllerService(DistributedMapCacheClient.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OBJECT_TYPE = new PropertyDescriptor.Builder()
+ .name(ASANA_OBJECT_TYPE)
+ .displayName("Object Type")
+ .description("Specify what kind of objects to be collected from Asana")
+ .required(true)
+ .allowableValues(AsanaObjectType.class)
+ .defaultValue(AV_COLLECT_TASKS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_PROJECT = new PropertyDescriptor.Builder()
+ .name(ASANA_PROJECT_NAME)
+ .displayName("Project Name")
+ .description("Fetch only objects in this project. Case sensitive.")
+ .required(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(
+ PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_PROJECT_MEMBERS,
+ AV_COLLECT_STORIES,
+ AV_COLLECT_PROJECT_STATUS_UPDATES,
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ AV_COLLECT_PROJECT_EVENTS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_SECTION = new PropertyDescriptor.Builder()
+ .name(ASANA_SECTION_NAME)
+ .displayName("Section Name")
+ .description("Fetch only objects in this section. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TAG = new PropertyDescriptor.Builder()
+ .name(ASANA_TAG_NAME)
+ .displayName("Tag")
+ .description("Fetch only objects having this tag. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TEAM_NAME = new PropertyDescriptor.Builder()
+ .name(ASANA_TEAM_NAME)
+ .displayName("Team")
+ .description("Team name. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TEAM_MEMBERS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OUTPUT_BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name(ASANA_OUTPUT_BATCH_SIZE)
+ .displayName("Output Batch Size")
+ .description("The number of items batched together in a single Flow File. If set to 1 (default), then each item is"
+ + " transferred in a separate Flow File and each will have an asana.gid attribute, to help identifying"
+ + " the fetched item on the server side, if needed. If the batch size is greater than 1, then the"
+ + " specified amount of items are batched together in a single Flow File as a Json array, and the"
+ + " Flow Files won't have the asana.gid attribute.")
+ .defaultValue("1")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+
+ protected static final List<PropertyDescriptor> DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ PROP_ASANA_CLIENT_SERVICE,
+ PROP_DISTRIBUTED_CACHE_SERVICE,
+ PROP_ASANA_OBJECT_TYPE,
+ PROP_ASANA_PROJECT,
+ PROP_ASANA_SECTION,
+ PROP_ASANA_TEAM_NAME,
+ PROP_ASANA_TAG,
+ PROP_ASANA_OUTPUT_BATCH_SIZE
+ ));
+
+ protected static final Relationship REL_NEW = new Relationship.Builder()
+ .name(REL_NAME_NEW)
+ .description("Newly collected objects are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_UPDATED = new Relationship.Builder()
+ .name(REL_NAME_UPDATED)
+ .description("Objects that have already been collected earlier, but were updated since, are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_REMOVED = new Relationship.Builder()
+ .name(REL_NAME_REMOVED)
+ .description("Notification about deleted objects are routed to this relationship. "
+ + "Flow files will not have any payload. IDs of the resources no longer exist "
+ + "are carried by the asana.gid attribute of the generated FlowFiles.")
+ .build();
+
+ protected static final Set<Relationship> RELATIONSHIPS = Collections.unmodifiableSet(new HashSet<>(Arrays.asList(
+ REL_NEW,
+ REL_UPDATED,
+ REL_REMOVED
+ )));
+ protected static final GenericObjectSerDe<String> STATE_MAP_KEY_SERIALIZER = new GenericObjectSerDe<>();
+ protected static final GenericObjectSerDe<Map<String, String>> STATE_MAP_VALUE_SERIALIZER = new GenericObjectSerDe<>();
+
+ private volatile AsanaObjectFetcher objectFetcher;
+ private volatile Integer batchSize;
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return RELATIONSHIPS;
+ }
+
+ @Override
+ public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return DESCRIPTORS;
+ }
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) throws InitializationException {
+ AsanaClientProviderService controllerService = context.getProperty(PROP_ASANA_CLIENT_SERVICE).asControllerService(AsanaClientProviderService.class);
+ AsanaClient client = controllerService.createClient();
+ batchSize = context.getProperty(PROP_ASANA_OUTPUT_BATCH_SIZE).asInteger();
+
+ try {
+ getLogger().debug("Initializing object fetcher...");
+ objectFetcher = createObjectFetcher(context, client);
+ } catch (Exception e) {
+ throw new InitializationException(e);
+ }
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ try {
+ Map<String, String> state = recoverState(context).orElse(Collections.emptyMap());
+ getLogger().debug("Attempting to load state: {}", state);
+ objectFetcher.loadState(state);
+ } catch (Exception e) {
+ getLogger().info("Failed to recover state. Falling back to clean start.");
+ objectFetcher.clearState();
+ }
Review Comment:
I think we should distinguish errors coming from `recoverState()` versus `loadState()`.
`recoverState()` can throw an exception if the external cache is temporarily not available but in this case we should not clear the state but rather log the error and do nothing until the cache issue is fixed.
`loadState()` errors seem to be unrecoverable errors and clean start is reasonable for this case.
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/resources/docs/org.apache.nifi.processors.asana.GetAsanaObject/additionalDetails.html:
##########
@@ -0,0 +1,100 @@
+<!DOCTYPE html>
+<html lang="en" xmlns="http://www.w3.org/1999/html">
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+<head>
+ <meta charset="utf-8"/>
+ <title>GetAsanaObject</title>
+ <link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"/>
+ <style>
+ h2 {margin-top: 4em}
+ h3 {margin-top: 3em}
+ td {text-align: left}
+ </style>
+</head>
+
+<body>
+
+<h1>GetAsanaObject</h1>
+
+<h3>Description</h3>
+<p>
+ This processor collects various objects (eg. tasks, comments, etc...) from Asana via the specified
+ <code>AsanaClientService</code>. When the processor started for the first time with a given configuration
+ it collects each of the objects matching the user specified criteria, and emits <code>FlowFile</code>s
+ of each on the <code>NEW</code> relationship. Then, it polls Asana in the frequency of the configured <em>Yield</em>
+ duration, and detects changes by comparing the object fingerprints. When there are updates, it emits them through
Review Comment:
```suggestion
of each on the <code>NEW</code> relationship. Then, it polls Asana in the frequency of the configured <em>Run Schedule</em>
and detects changes by comparing the object fingerprints. When there are updates, it emits them through
```
The base frequency is `Run Schedule`. `Yield Duration` takes effect only if there were no changed items in the previous run (and `Run Schedule < Yield Duration`).
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,405 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import static java.lang.String.format;
+import static java.lang.String.join;
+import static java.util.Collections.singletonMap;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_EVENTS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_MEMBERS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_STATUS_ATTACHMENTS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_STATUS_UPDATES;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_STORIES;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_TASKS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_TASK_ATTACHMENTS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_TEAM_MEMBERS;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientProviderService;
+import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaObjectState;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+@TriggerSerially
+@PrimaryNodeOnly
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.")
+@Tags({"asana", "source", "ingest"})
+@CapabilityDescription("This processor collects data from Asana")
+public class GetAsanaObject extends AbstractProcessor {
+
+ protected static final String ASANA_GID = "asana.gid";
+ protected static final String ASANA_CLIENT_SERVICE = "asana-controller-service";
+ protected static final String DISTRIBUTED_CACHE_SERVICE = "distributed-cache-service";
+ protected static final String ASANA_OBJECT_TYPE = "asana-object-type";
+ protected static final String ASANA_PROJECT_NAME = "asana-project-name";
+ protected static final String ASANA_SECTION_NAME = "asana-section-name";
+ protected static final String ASANA_TAG_NAME = "asana-tag-name";
+ protected static final String ASANA_TEAM_NAME = "asana-team-name";
+ protected static final String ASANA_OUTPUT_BATCH_SIZE = "asana-output-batch-size";
+ protected static final String REL_NAME_NEW = "new";
+ protected static final String REL_NAME_UPDATED = "updated";
+ protected static final String REL_NAME_REMOVED = "removed";
+
+ protected static final PropertyDescriptor PROP_ASANA_CLIENT_SERVICE = new PropertyDescriptor.Builder()
+ .name(ASANA_CLIENT_SERVICE)
+ .displayName("Asana Client Service")
+ .description("Specify which controller service to use for accessing Asana.")
+ .required(true)
+ .identifiesControllerService(AsanaClientProviderService.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_DISTRIBUTED_CACHE_SERVICE = new Builder()
+ .name(DISTRIBUTED_CACHE_SERVICE)
+ .displayName("Distributed Cache Service")
+ .description("Cache service to store fetched item fingerprints. These, from the last successful query"
+ + " are stored, in order to enable incremental loading and change detection.")
+ .required(true)
+ .identifiesControllerService(DistributedMapCacheClient.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OBJECT_TYPE = new PropertyDescriptor.Builder()
+ .name(ASANA_OBJECT_TYPE)
+ .displayName("Object Type")
+ .description("Specify what kind of objects to be collected from Asana")
+ .required(true)
+ .allowableValues(AsanaObjectType.class)
+ .defaultValue(AV_COLLECT_TASKS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_PROJECT = new PropertyDescriptor.Builder()
+ .name(ASANA_PROJECT_NAME)
+ .displayName("Project Name")
+ .description("Fetch only objects in this project. Case sensitive.")
+ .required(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(
+ PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_PROJECT_MEMBERS,
+ AV_COLLECT_STORIES,
+ AV_COLLECT_PROJECT_STATUS_UPDATES,
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ AV_COLLECT_PROJECT_EVENTS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_SECTION = new PropertyDescriptor.Builder()
+ .name(ASANA_SECTION_NAME)
+ .displayName("Section Name")
+ .description("Fetch only objects in this section. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TAG = new PropertyDescriptor.Builder()
+ .name(ASANA_TAG_NAME)
+ .displayName("Tag")
+ .description("Fetch only objects having this tag. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TEAM_NAME = new PropertyDescriptor.Builder()
+ .name(ASANA_TEAM_NAME)
+ .displayName("Team")
+ .description("Team name. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TEAM_MEMBERS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OUTPUT_BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name(ASANA_OUTPUT_BATCH_SIZE)
+ .displayName("Output Batch Size")
+ .description("The number of items batched together in a single Flow File. If set to 1 (default), then each item is"
+ + " transferred in a separate Flow File and each will have an asana.gid attribute, to help identifying"
+ + " the fetched item on the server side, if needed. If the batch size is greater than 1, then the"
+ + " specified amount of items are batched together in a single Flow File as a Json array, and the"
+ + " Flow Files won't have the asana.gid attribute.")
+ .defaultValue("1")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+
+ protected static final List<PropertyDescriptor> DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ PROP_ASANA_CLIENT_SERVICE,
+ PROP_DISTRIBUTED_CACHE_SERVICE,
+ PROP_ASANA_OBJECT_TYPE,
+ PROP_ASANA_PROJECT,
+ PROP_ASANA_SECTION,
+ PROP_ASANA_TEAM_NAME,
+ PROP_ASANA_TAG,
+ PROP_ASANA_OUTPUT_BATCH_SIZE
+ ));
+
+ protected static final Relationship REL_NEW = new Relationship.Builder()
+ .name(REL_NAME_NEW)
+ .description("Newly collected objects are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_UPDATED = new Relationship.Builder()
+ .name(REL_NAME_UPDATED)
+ .description("Objects that have already been collected earlier, but were updated since, are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_REMOVED = new Relationship.Builder()
+ .name(REL_NAME_REMOVED)
+ .description("Notification about deleted objects are routed to this relationship. "
+ + "Flow files will not have any payload. IDs of the resources no longer exist "
+ + "are carried by the asana.gid attribute of the generated FlowFiles.")
+ .build();
+
+ protected static final Set<Relationship> RELATIONSHIPS = Collections.unmodifiableSet(new HashSet<>(Arrays.asList(
+ REL_NEW,
+ REL_UPDATED,
+ REL_REMOVED
+ )));
+ protected static final GenericObjectSerDe<String> STATE_MAP_KEY_SERIALIZER = new GenericObjectSerDe<>();
+ protected static final GenericObjectSerDe<Map<String, String>> STATE_MAP_VALUE_SERIALIZER = new GenericObjectSerDe<>();
+
+ private volatile AsanaObjectFetcher objectFetcher;
+ private volatile Integer batchSize;
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return RELATIONSHIPS;
+ }
+
+ @Override
+ public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return DESCRIPTORS;
+ }
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) throws InitializationException {
+ AsanaClientProviderService controllerService = context.getProperty(PROP_ASANA_CLIENT_SERVICE).asControllerService(AsanaClientProviderService.class);
+ AsanaClient client = controllerService.createClient();
+ batchSize = context.getProperty(PROP_ASANA_OUTPUT_BATCH_SIZE).asInteger();
+
+ try {
+ getLogger().debug("Initializing object fetcher...");
+ objectFetcher = createObjectFetcher(context, client);
+ } catch (Exception e) {
+ throw new InitializationException(e);
+ }
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ try {
+ Map<String, String> state = recoverState(context).orElse(Collections.emptyMap());
+ getLogger().debug("Attempting to load state: {}", state);
+ objectFetcher.loadState(state);
Review Comment:
`AsanaObjectFetcher`'s state is kept in memory until the next `onTrigger()` call. It should be cleared at the end of `onTrigger()` in order to free up the resources in the fetcher object (like the `lastFingerprints` map). Or the fetcher object can be discarded and recreated in every `onTrigger`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r1035779685
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,393 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import static java.lang.String.format;
+import static java.util.Collections.singletonMap;
+import static java.util.stream.Collectors.groupingBy;
+import static java.util.stream.Collectors.joining;
+import static org.apache.commons.collections4.ListUtils.partition;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_EVENTS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_MEMBERS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_STATUS_ATTACHMENTS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_STATUS_UPDATES;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_STORIES;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_TASKS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_TASK_ATTACHMENTS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_TEAM_MEMBERS;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientProviderService;
+import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaObjectState;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+@TriggerSerially
+@PrimaryNodeOnly
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.")
+@Tags({"asana", "source", "ingest"})
+@CapabilityDescription("This processor collects data from Asana")
+public class GetAsanaObject extends AbstractProcessor {
+
+ protected static final String ASANA_GID = "asana.gid";
+ protected static final String ASANA_CLIENT_SERVICE = "asana-controller-service";
+ protected static final String DISTRIBUTED_CACHE_SERVICE = "distributed-cache-service";
+ protected static final String ASANA_OBJECT_TYPE = "asana-object-type";
+ protected static final String ASANA_PROJECT_NAME = "asana-project-name";
+ protected static final String ASANA_SECTION_NAME = "asana-section-name";
+ protected static final String ASANA_TAG_NAME = "asana-tag-name";
+ protected static final String ASANA_TEAM_NAME = "asana-team-name";
+ protected static final String ASANA_OUTPUT_BATCH_SIZE = "asana-output-batch-size";
+ protected static final String REL_NAME_NEW = "new";
+ protected static final String REL_NAME_UPDATED = "updated";
+ protected static final String REL_NAME_REMOVED = "removed";
+
+ protected static final PropertyDescriptor PROP_ASANA_CLIENT_SERVICE = new PropertyDescriptor.Builder()
+ .name(ASANA_CLIENT_SERVICE)
+ .displayName("Asana Client Service")
+ .description("Specify which controller service to use for accessing Asana.")
+ .required(true)
+ .identifiesControllerService(AsanaClientProviderService.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_DISTRIBUTED_CACHE_SERVICE = new Builder()
+ .name(DISTRIBUTED_CACHE_SERVICE)
+ .displayName("Distributed Cache Service")
+ .description("Cache service to store fetched item fingerprints. These, from the last successful query"
+ + " are stored, in order to enable incremental loading and change detection.")
+ .required(true)
+ .identifiesControllerService(DistributedMapCacheClient.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OBJECT_TYPE = new PropertyDescriptor.Builder()
+ .name(ASANA_OBJECT_TYPE)
+ .displayName("Object Type")
+ .description("Specify what kind of objects to be collected from Asana")
+ .required(true)
+ .allowableValues(AsanaObjectType.class)
+ .defaultValue(AV_COLLECT_TASKS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_PROJECT = new PropertyDescriptor.Builder()
+ .name(ASANA_PROJECT_NAME)
+ .displayName("Project Name")
+ .description("Fetch only objects in this project. Case sensitive.")
+ .required(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(
+ PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS.getValue(),
+ AV_COLLECT_TASK_ATTACHMENTS.getValue(),
+ AV_COLLECT_PROJECT_MEMBERS.getValue(),
+ AV_COLLECT_STORIES.getValue(),
+ AV_COLLECT_PROJECT_STATUS_UPDATES.getValue(),
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS.getValue(),
+ AV_COLLECT_PROJECT_EVENTS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_SECTION = new PropertyDescriptor.Builder()
+ .name(ASANA_SECTION_NAME)
+ .displayName("Section Name")
+ .description("Fetch only objects in this section. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS.getValue(),
+ AV_COLLECT_TASK_ATTACHMENTS.getValue(),
+ AV_COLLECT_STORIES.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TAG = new PropertyDescriptor.Builder()
+ .name(ASANA_TAG_NAME)
+ .displayName("Tag")
+ .description("Fetch only objects having this tag. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS.getValue(),
+ AV_COLLECT_TASK_ATTACHMENTS.getValue(),
+ AV_COLLECT_STORIES.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TEAM_NAME = new PropertyDescriptor.Builder()
+ .name(ASANA_TEAM_NAME)
+ .displayName("Team")
+ .description("Team name. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TEAM_MEMBERS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OUTPUT_BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name(ASANA_OUTPUT_BATCH_SIZE)
+ .displayName("Output Batch Size")
+ .description("The number of items batched together in a single Flow File. If set to 1 (default), then each item is"
+ + " transferred in a separate Flow File and each will have an asana.gid attribute, to help identifying"
+ + " the fetched item on the server side, if needed. If the batch size is greater than 1, then the"
+ + " specified amount of items are batched together in a single Flow File as a Json array, and the"
+ + " Flow Files won't have the asana.gid attribute.")
+ .defaultValue("1")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+
+ protected static final List<PropertyDescriptor> DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ PROP_ASANA_CLIENT_SERVICE,
+ PROP_DISTRIBUTED_CACHE_SERVICE,
+ PROP_ASANA_OBJECT_TYPE,
+ PROP_ASANA_PROJECT,
+ PROP_ASANA_SECTION,
+ PROP_ASANA_TEAM_NAME,
+ PROP_ASANA_TAG,
+ PROP_ASANA_OUTPUT_BATCH_SIZE
+ ));
+
+ protected static final Relationship REL_NEW = new Relationship.Builder()
+ .name(REL_NAME_NEW)
+ .description("Newly collected objects are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_UPDATED = new Relationship.Builder()
+ .name(REL_NAME_UPDATED)
+ .description("Objects that have already been collected earlier, but were updated since, are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_REMOVED = new Relationship.Builder()
+ .name(REL_NAME_REMOVED)
+ .description("Notification about deleted objects are routed to this relationship. "
+ + "Flow files will not have any payload. IDs of the resources no longer exist "
+ + "are carried by the asana.gid attribute of the generated FlowFiles.")
+ .build();
+
+ protected static final Set<Relationship> RELATIONSHIPS = Collections.unmodifiableSet(new HashSet<>(Arrays.asList(
+ REL_NEW,
+ REL_UPDATED,
+ REL_REMOVED
+ )));
+
+ private volatile AsanaObjectFetcher objectFetcher;
+ private volatile Integer batchSize;
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return RELATIONSHIPS;
+ }
+
+ @Override
+ public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return DESCRIPTORS;
+ }
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) throws InitializationException {
+ AsanaClientProviderService controllerService = context.getProperty(PROP_ASANA_CLIENT_SERVICE).asControllerService(AsanaClientProviderService.class);
+ AsanaClient client = controllerService.createClient();
+ batchSize = context.getProperty(PROP_ASANA_OUTPUT_BATCH_SIZE).asInteger();
+
+ try {
+ getLogger().debug("Initializing object fetcher...");
+ objectFetcher = createObjectFetcher(context, client);
+ } catch (Exception e) {
+ throw new InitializationException(e);
+ }
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ try {
+ Map<String, String> state = recoverState(context).orElse(Collections.emptyMap());
+ getLogger().debug("Attempting to load state: {}", state);
+ objectFetcher.loadState(state);
+ } catch (Exception e) {
+ getLogger().info("Failed to recover state. Falling back to clean start.");
+ objectFetcher.clearState();
+ }
+ getLogger().debug("Initial state: {}", objectFetcher.saveState());
+
+ Collection<FlowFile> newItems = new ArrayList<>();
+ Collection<FlowFile> updatedItems = new ArrayList<>();
+ Collection<FlowFile> removedItems = new ArrayList<>();
+ Map<AsanaObjectState, Collection<FlowFile>> flowFiles = new HashMap<>();
+ flowFiles.put(AsanaObjectState.NEW, newItems);
+ flowFiles.put(AsanaObjectState.UPDATED, updatedItems);
+ flowFiles.put(AsanaObjectState.REMOVED, removedItems);
+
+ List<AsanaObject> allObjects = new ArrayList<>();
+
+ AsanaObject nextObject;
+ while ((nextObject = objectFetcher.fetchNext()) != null) {
+ allObjects.add(nextObject);
+ }
+
+ Map<AsanaObjectState, List<AsanaObject>> allObjectsByState = allObjects.stream()
+ .collect(groupingBy(AsanaObject::getState));
+
+ if (batchSize == 1) {
+ allObjectsByState
+ .forEach((asanaObjectState, asanaObjects) -> asanaObjects.forEach(
+ asanaObject -> {
+ final Map<String, String> attributes = new HashMap<>(2);
+ attributes.put(CoreAttributes.MIME_TYPE.key(), ContentType.APPLICATION_JSON.getMimeType());
+ attributes.put(ASANA_GID, asanaObject.getGid());
+ FlowFile flowFile = createFlowFileWithStringPayload(session, asanaObject.getContent());
+ flowFile = session.putAllAttributes(flowFile, attributes);
+ flowFiles.get(asanaObject.getState()).add(flowFile);
+ }
+ ));
+ } else {
+ allObjectsByState
+ .forEach((asanaObjectState, asanaObjects) -> partition(asanaObjects, batchSize).forEach(
+ asanaObjectsInPartition -> {
+ FlowFile flowFile = createFlowFileWithStringPayload(session, format("[%s]",
+ asanaObjectsInPartition.stream().map(AsanaObject::getContent)
+ .collect(joining(","))));
+ flowFile = session.putAllAttributes(flowFile,
+ singletonMap(CoreAttributes.MIME_TYPE.key(),
+ ContentType.APPLICATION_JSON.getMimeType()));
+ flowFiles.get(asanaObjectState).add(flowFile);
+ }
+ ));
+ }
+
+ if (flowFiles.values().stream().allMatch(Collection::isEmpty)) {
+ context.yield();
+ getLogger().debug("Yielding, as there are no new FlowFiles.");
+ } else {
+ session.transfer(newItems, REL_NEW);
+ session.transfer(updatedItems, REL_UPDATED);
+ session.transfer(removedItems, REL_REMOVED);
+ }
+
+ Map<String, String> state = objectFetcher.saveState();
+ persistState(state, context);
+
+ getLogger().debug(
+ "New state after transferring {} new, {} updated, and {} removed items: {}",
+ newItems.size(), updatedItems.size(), removedItems.size(), state);
+
+ session.commitAsync();
+ }
+
+ protected AsanaObjectFetcher createObjectFetcher(final ProcessContext context, AsanaClient client) {
+ final String objectType = context.getProperty(PROP_ASANA_OBJECT_TYPE).getValue();
+ final String projectName = context.getProperty(PROP_ASANA_PROJECT).getValue();
+ final String sectionName = context.getProperty(PROP_ASANA_SECTION).getValue();
+ final String teamName = context.getProperty(PROP_ASANA_TEAM_NAME).getValue();
+ final String tagName = context.getProperty(PROP_ASANA_TAG).getValue();
+
+ switch (AsanaObjectType.fromValue(objectType)) {
+ case AV_COLLECT_TASKS:
+ return new AsanaTaskFetcher(client, projectName, sectionName, tagName);
+ case AV_COLLECT_PROJECTS:
+ return new AsanaProjectFetcher(client);
+ case AV_COLLECT_PROJECT_EVENTS:
+ return new AsanaProjectEventFetcher(client, projectName);
+ case AV_COLLECT_PROJECT_MEMBERS:
+ return new AsanaProjectMembershipFetcher(client, projectName);
+ case AV_COLLECT_PROJECT_STATUS_ATTACHMENTS:
+ return new AsanaProjectStatusAttachmentFetcher(client, projectName);
+ case AV_COLLECT_PROJECT_STATUS_UPDATES:
+ return new AsanaProjectStatusFetcher(client, projectName);
+ case AV_COLLECT_STORIES:
+ return new AsanaStoryFetcher(client, projectName, sectionName, tagName);
+ case AV_COLLECT_TAGS:
+ return new AsanaTagFetcher(client);
+ case AV_COLLECT_TASK_ATTACHMENTS:
+ return new AsanaTaskAttachmentFetcher(client, projectName, sectionName, tagName);
+ case AV_COLLECT_TEAMS:
+ return new AsanaTeamFetcher(client);
+ case AV_COLLECT_TEAM_MEMBERS:
+ return new AsanaTeamMemberFetcher(client, teamName);
+ case AV_COLLECT_USERS:
+ return new AsanaUserFetcher(client);
+ }
+
+ throw new ProcessException("Cannot fetch objects of type: " + objectType);
+ }
+
+ private Optional<Map<String, String>> recoverState(final ProcessContext context) {
+ final DistributedMapCacheClient client = getDistributedMapCacheClient(context);
+ try {
+ final Map<String, String> result = client.get(getIdentifier(), new GenericObjectSerDe<>(), new GenericObjectSerDe<>());
Review Comment:
Done in 29673c5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r1035793778
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-services-api-nar/src/main/resources/META-INF/NOTICE:
##########
@@ -0,0 +1,108 @@
+nifi-asana-services-api-nar
+Copyright 2015-2022 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+===========================================
+Apache Software License v2
+===========================================
+
+The following binary components are provided under the Apache Software License v2
+
+ (ASLv2) Apache Commons IO
+ The following NOTICE information applies:
+ Apache Commons IO
+ Copyright 2002-2022 The Apache Software Foundation
Review Comment:
Remaining part done in 16ebeba
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on PR #6504:
URL: https://github.com/apache/nifi/pull/6504#issuecomment-1330964547
@exceptionfactory , @turcsanyip I have implemented support for `DistributedMapCacheClient` in 3ddb7bb. Please check my PR again, when you have time for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] turcsanyip commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
turcsanyip commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r994985443
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientServiceApi;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@TriggerSerially
+@Stateful(scopes = {Scope.LOCAL}, description = "Fingerprints of items in the last successful query are stored in order to enable incremental loading and change detection.")
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.")
+@Tags({"asana", "source", "connector", "ingest"})
+@CapabilityDescription("This processor collects data from Asana")
+public class GetAsanaObject extends AbstractProcessor {
+
+ protected static final String ASANA_GID = "asana.gid";
+ protected static final String AV_NAME_COLLECT_TASKS = "asana-collect-tasks";
+ protected static final String AV_NAME_COLLECT_TASK_ATTACHMENTS = "asana-collect-task-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECTS = "asana-collect-projects";
+ protected static final String AV_NAME_COLLECT_TAGS = "asana-collect-tags";
+ protected static final String AV_NAME_COLLECT_USERS = "asana-collect-users";
+ protected static final String AV_NAME_COLLECT_PROJECT_MEMBERS = "asana-collect-project-members";
+ protected static final String AV_NAME_COLLECT_TEAMS = "asana-collect-teams";
+ protected static final String AV_NAME_COLLECT_TEAM_MEMBERS = "asana-collect-team-members";
+ protected static final String AV_NAME_COLLECT_STORIES = "asana-collect-stories";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_UPDATES = "asana-collect-project-status-updates";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS = "asana-collect-project-status-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECT_EVENTS = "asana-collect-project-events";
+ protected static final String ASANA_CONTROLLER_SERVICE = "asana-controller-service";
+ protected static final String ASANA_OBJECT_TYPE = "asana-object-type";
+ protected static final String ASANA_PROJECT_NAME = "asana-project-name";
+ protected static final String ASANA_SECTION_NAME = "asana-section-name";
+ protected static final String ASANA_TAG_NAME = "asana-tag-name";
+ protected static final String ASANA_TEAM_NAME = "asana-team-name";
+ protected static final String ASANA_OUTPUT_BATCH_SIZE = "asana-output-batch-size";
+ protected static final String REL_NAME_NEW = "new";
+ protected static final String REL_NAME_UPDATED = "updated";
+ protected static final String REL_NAME_REMOVED = "removed";
+
+ protected static final AllowableValue AV_COLLECT_TASKS = new AllowableValue(
+ AV_NAME_COLLECT_TASKS,
+ "Tasks",
+ "Collect tasks matching to the specified conditions.");
+
+ protected static final AllowableValue AV_COLLECT_TASK_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_TASK_ATTACHMENTS,
+ "Task attachments",
+ "Collect attached files of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECTS,
+ "Projects",
+ "Collect projects of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TAGS = new AllowableValue(
+ AV_NAME_COLLECT_TAGS,
+ "Tags",
+ "Collect tags of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_USERS = new AllowableValue(
+ AV_NAME_COLLECT_USERS,
+ "Users",
+ "Collect users assigned to the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_MEMBERS,
+ "Members of a project",
+ "Collect users assigned to the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAMS = new AllowableValue(
+ AV_NAME_COLLECT_TEAMS,
+ "Teams",
+ "Collect teams of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAM_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_TEAM_MEMBERS,
+ "Team members",
+ "Collect users assigned to the specified team."
+ );
+
+ protected static final AllowableValue AV_COLLECT_STORIES = new AllowableValue(
+ AV_NAME_COLLECT_STORIES,
+ "Stories of tasks",
+ "Collect stories (comments) of of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_UPDATES = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_UPDATES,
+ "Status updates of a project",
+ "Collect status updates of the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ "Attachments of status updates",
+ "Collect attached files of project status updates."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_EVENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_EVENTS,
+ "Events of a project",
+ "Collect various events happening on the specified project and on its' tasks."
+ );
+
+ protected static final PropertyDescriptor PROP_ASANA_CONTROLLER_SERVICE = new PropertyDescriptor.Builder()
+ .name(ASANA_CONTROLLER_SERVICE)
+ .displayName("Controller service")
+ .description("Specify which controller service to use for accessing Asana.")
+ .required(true)
+ .identifiesControllerService(AsanaClientServiceApi.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OBJECT_TYPE = new PropertyDescriptor.Builder()
+ .name(ASANA_OBJECT_TYPE)
+ .displayName("Object type to be collected")
+ .description("Specify what kind of objects to be collected from Asana")
+ .required(true)
+ .allowableValues(
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_PROJECTS,
+ AV_COLLECT_TAGS,
+ AV_COLLECT_USERS,
+ AV_COLLECT_PROJECT_MEMBERS,
+ AV_COLLECT_TEAMS,
+ AV_COLLECT_TEAM_MEMBERS,
+ AV_COLLECT_STORIES,
+ AV_COLLECT_PROJECT_STATUS_UPDATES,
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ AV_COLLECT_PROJECT_EVENTS)
+ .defaultValue(AV_COLLECT_TASKS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_PROJECT = new PropertyDescriptor.Builder()
+ .name(ASANA_PROJECT_NAME)
+ .displayName("Project name")
Review Comment:
Please use Title Case for property names: `Project Name`, `Section Name`, etc.
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-services-api/src/main/java/org/apache/nifi/controller/asana/AsanaClientServiceApi.java:
##########
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.controller.asana;
+
+import org.apache.nifi.controller.ControllerService;
+
+/**
+ * This interface represents an API to the controller service of Asana processors.
+ */
+public interface AsanaClientServiceApi extends ControllerService {
Review Comment:
The convention for the controller service interface / implementation class names (at least for the new ones):
`XXXService` / `StandardXXXService`
Also, we use "Provider" tag for this type of controller services that provide a client. So the names would be:
`AsanaClientProviderService` / `StandardAsanaClientProviderService`
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientServiceApi;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@TriggerSerially
+@Stateful(scopes = {Scope.LOCAL}, description = "Fingerprints of items in the last successful query are stored in order to enable incremental loading and change detection.")
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.")
+@Tags({"asana", "source", "connector", "ingest"})
+@CapabilityDescription("This processor collects data from Asana")
+public class GetAsanaObject extends AbstractProcessor {
+
+ protected static final String ASANA_GID = "asana.gid";
+ protected static final String AV_NAME_COLLECT_TASKS = "asana-collect-tasks";
+ protected static final String AV_NAME_COLLECT_TASK_ATTACHMENTS = "asana-collect-task-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECTS = "asana-collect-projects";
+ protected static final String AV_NAME_COLLECT_TAGS = "asana-collect-tags";
+ protected static final String AV_NAME_COLLECT_USERS = "asana-collect-users";
+ protected static final String AV_NAME_COLLECT_PROJECT_MEMBERS = "asana-collect-project-members";
+ protected static final String AV_NAME_COLLECT_TEAMS = "asana-collect-teams";
+ protected static final String AV_NAME_COLLECT_TEAM_MEMBERS = "asana-collect-team-members";
+ protected static final String AV_NAME_COLLECT_STORIES = "asana-collect-stories";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_UPDATES = "asana-collect-project-status-updates";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS = "asana-collect-project-status-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECT_EVENTS = "asana-collect-project-events";
+ protected static final String ASANA_CONTROLLER_SERVICE = "asana-controller-service";
+ protected static final String ASANA_OBJECT_TYPE = "asana-object-type";
+ protected static final String ASANA_PROJECT_NAME = "asana-project-name";
+ protected static final String ASANA_SECTION_NAME = "asana-section-name";
+ protected static final String ASANA_TAG_NAME = "asana-tag-name";
+ protected static final String ASANA_TEAM_NAME = "asana-team-name";
+ protected static final String ASANA_OUTPUT_BATCH_SIZE = "asana-output-batch-size";
+ protected static final String REL_NAME_NEW = "new";
+ protected static final String REL_NAME_UPDATED = "updated";
+ protected static final String REL_NAME_REMOVED = "removed";
+
+ protected static final AllowableValue AV_COLLECT_TASKS = new AllowableValue(
+ AV_NAME_COLLECT_TASKS,
+ "Tasks",
+ "Collect tasks matching to the specified conditions.");
+
+ protected static final AllowableValue AV_COLLECT_TASK_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_TASK_ATTACHMENTS,
+ "Task attachments",
+ "Collect attached files of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECTS,
+ "Projects",
+ "Collect projects of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TAGS = new AllowableValue(
+ AV_NAME_COLLECT_TAGS,
+ "Tags",
+ "Collect tags of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_USERS = new AllowableValue(
+ AV_NAME_COLLECT_USERS,
+ "Users",
+ "Collect users assigned to the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_MEMBERS,
+ "Members of a project",
+ "Collect users assigned to the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAMS = new AllowableValue(
+ AV_NAME_COLLECT_TEAMS,
+ "Teams",
+ "Collect teams of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAM_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_TEAM_MEMBERS,
+ "Team members",
+ "Collect users assigned to the specified team."
+ );
+
+ protected static final AllowableValue AV_COLLECT_STORIES = new AllowableValue(
+ AV_NAME_COLLECT_STORIES,
+ "Stories of tasks",
+ "Collect stories (comments) of of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_UPDATES = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_UPDATES,
+ "Status updates of a project",
+ "Collect status updates of the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ "Attachments of status updates",
+ "Collect attached files of project status updates."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_EVENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_EVENTS,
+ "Events of a project",
+ "Collect various events happening on the specified project and on its' tasks."
+ );
+
+ protected static final PropertyDescriptor PROP_ASANA_CONTROLLER_SERVICE = new PropertyDescriptor.Builder()
+ .name(ASANA_CONTROLLER_SERVICE)
+ .displayName("Controller service")
+ .description("Specify which controller service to use for accessing Asana.")
+ .required(true)
+ .identifiesControllerService(AsanaClientServiceApi.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OBJECT_TYPE = new PropertyDescriptor.Builder()
+ .name(ASANA_OBJECT_TYPE)
+ .displayName("Object type to be collected")
+ .description("Specify what kind of objects to be collected from Asana")
+ .required(true)
+ .allowableValues(
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_PROJECTS,
+ AV_COLLECT_TAGS,
+ AV_COLLECT_USERS,
+ AV_COLLECT_PROJECT_MEMBERS,
+ AV_COLLECT_TEAMS,
+ AV_COLLECT_TEAM_MEMBERS,
+ AV_COLLECT_STORIES,
+ AV_COLLECT_PROJECT_STATUS_UPDATES,
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ AV_COLLECT_PROJECT_EVENTS)
+ .defaultValue(AV_COLLECT_TASKS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_PROJECT = new PropertyDescriptor.Builder()
+ .name(ASANA_PROJECT_NAME)
+ .displayName("Project name")
+ .description("Fetch only objects in this project. Case sensitive.")
+ .required(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(
+ PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_PROJECT_MEMBERS,
+ AV_COLLECT_STORIES,
+ AV_COLLECT_PROJECT_STATUS_UPDATES,
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ AV_COLLECT_PROJECT_EVENTS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_SECTION = new PropertyDescriptor.Builder()
+ .name(ASANA_SECTION_NAME)
+ .displayName("Section name")
+ .description("Fetch only objects in this section. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TASKS, AV_COLLECT_TASK_ATTACHMENTS, AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TAG = new PropertyDescriptor.Builder()
+ .name(ASANA_TAG_NAME)
+ .displayName("Tag")
+ .description("Fetch only objects having this tag. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TASKS, AV_COLLECT_TASK_ATTACHMENTS, AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TEAM_NAME = new PropertyDescriptor.Builder()
+ .name(ASANA_TEAM_NAME)
+ .displayName("Team")
+ .description("Team name. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TEAM_MEMBERS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OUTPUT_BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name(ASANA_OUTPUT_BATCH_SIZE)
+ .displayName("Output Batch Size")
+ .description("The number of output FlowFiles to queue before committing the process session. When set to zero, the session will be committed when all result set rows "
+ + "have been processed and the output FlowFiles are ready for transfer to the downstream relationship. For large result sets, this can cause a large burst of FlowFiles "
+ + "to be transferred at the end of processor execution. If this property is set, then when the specified number of FlowFiles are ready for transfer, then the session will "
+ + "be committed, thus releasing the FlowFiles to the downstream relationship.")
+ .defaultValue("0")
+ .required(true)
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+
+ protected static final List<PropertyDescriptor> DESCRIPTORS = Collections.unmodifiableList(Lists.newArrayList(
+ PROP_ASANA_CONTROLLER_SERVICE,
+ PROP_ASANA_OBJECT_TYPE,
+ PROP_ASANA_PROJECT,
+ PROP_ASANA_SECTION,
+ PROP_ASANA_TEAM_NAME,
+ PROP_ASANA_TAG,
+ PROP_ASANA_OUTPUT_BATCH_SIZE
+ ));
+
+ protected static final Relationship REL_NEW = new Relationship.Builder()
+ .name(REL_NAME_NEW)
+ .description("Newly collected objects are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_UPDATED = new Relationship.Builder()
+ .name(REL_NAME_UPDATED)
+ .description("Objects that have already been collected earlier, but were updated since, are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_REMOVED = new Relationship.Builder()
+ .name(REL_NAME_REMOVED)
+ .description("Notification about deleted objects are routed to this relationship. "
+ + "Flow files will not have any payload. IDs of the resources no longer exist "
+ + "are carried by the asana.gid attribute of the generated FlowFiles.")
+ .build();
+
+ protected static final Set<Relationship> RELATIONSHIPS = Collections.unmodifiableSet(Sets.newHashSet(
+ REL_NEW,
+ REL_UPDATED,
+ REL_REMOVED
+ ));
+
+ final Scope STATE_STORAGE_SCOPE = Scope.LOCAL;
+
+ AsanaClientServiceApi controllerService;
+ AsanaObjectFetcher objectFetcher;
+ private Integer batchSize;
Review Comment:
```suggestion
private volatile AsanaObjectFetcher objectFetcher;
private volatile Integer batchSize;
```
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientServiceApi;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@TriggerSerially
+@Stateful(scopes = {Scope.LOCAL}, description = "Fingerprints of items in the last successful query are stored in order to enable incremental loading and change detection.")
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.")
+@Tags({"asana", "source", "connector", "ingest"})
Review Comment:
We do not use "connector" terminology for tagging.
```suggestion
@Tags({"asana", "source", "ingest"})
```
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientServiceApi;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@TriggerSerially
+@Stateful(scopes = {Scope.LOCAL}, description = "Fingerprints of items in the last successful query are stored in order to enable incremental loading and change detection.")
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.")
+@Tags({"asana", "source", "connector", "ingest"})
+@CapabilityDescription("This processor collects data from Asana")
+public class GetAsanaObject extends AbstractProcessor {
+
+ protected static final String ASANA_GID = "asana.gid";
+ protected static final String AV_NAME_COLLECT_TASKS = "asana-collect-tasks";
+ protected static final String AV_NAME_COLLECT_TASK_ATTACHMENTS = "asana-collect-task-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECTS = "asana-collect-projects";
+ protected static final String AV_NAME_COLLECT_TAGS = "asana-collect-tags";
+ protected static final String AV_NAME_COLLECT_USERS = "asana-collect-users";
+ protected static final String AV_NAME_COLLECT_PROJECT_MEMBERS = "asana-collect-project-members";
+ protected static final String AV_NAME_COLLECT_TEAMS = "asana-collect-teams";
+ protected static final String AV_NAME_COLLECT_TEAM_MEMBERS = "asana-collect-team-members";
+ protected static final String AV_NAME_COLLECT_STORIES = "asana-collect-stories";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_UPDATES = "asana-collect-project-status-updates";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS = "asana-collect-project-status-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECT_EVENTS = "asana-collect-project-events";
+ protected static final String ASANA_CONTROLLER_SERVICE = "asana-controller-service";
+ protected static final String ASANA_OBJECT_TYPE = "asana-object-type";
+ protected static final String ASANA_PROJECT_NAME = "asana-project-name";
+ protected static final String ASANA_SECTION_NAME = "asana-section-name";
+ protected static final String ASANA_TAG_NAME = "asana-tag-name";
+ protected static final String ASANA_TEAM_NAME = "asana-team-name";
+ protected static final String ASANA_OUTPUT_BATCH_SIZE = "asana-output-batch-size";
+ protected static final String REL_NAME_NEW = "new";
+ protected static final String REL_NAME_UPDATED = "updated";
+ protected static final String REL_NAME_REMOVED = "removed";
+
+ protected static final AllowableValue AV_COLLECT_TASKS = new AllowableValue(
+ AV_NAME_COLLECT_TASKS,
+ "Tasks",
+ "Collect tasks matching to the specified conditions.");
+
+ protected static final AllowableValue AV_COLLECT_TASK_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_TASK_ATTACHMENTS,
+ "Task attachments",
+ "Collect attached files of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECTS,
+ "Projects",
+ "Collect projects of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TAGS = new AllowableValue(
+ AV_NAME_COLLECT_TAGS,
+ "Tags",
+ "Collect tags of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_USERS = new AllowableValue(
+ AV_NAME_COLLECT_USERS,
+ "Users",
+ "Collect users assigned to the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_MEMBERS,
+ "Members of a project",
+ "Collect users assigned to the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAMS = new AllowableValue(
+ AV_NAME_COLLECT_TEAMS,
+ "Teams",
+ "Collect teams of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAM_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_TEAM_MEMBERS,
+ "Team members",
+ "Collect users assigned to the specified team."
+ );
+
+ protected static final AllowableValue AV_COLLECT_STORIES = new AllowableValue(
+ AV_NAME_COLLECT_STORIES,
+ "Stories of tasks",
+ "Collect stories (comments) of of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_UPDATES = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_UPDATES,
+ "Status updates of a project",
+ "Collect status updates of the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ "Attachments of status updates",
+ "Collect attached files of project status updates."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_EVENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_EVENTS,
+ "Events of a project",
+ "Collect various events happening on the specified project and on its' tasks."
+ );
+
+ protected static final PropertyDescriptor PROP_ASANA_CONTROLLER_SERVICE = new PropertyDescriptor.Builder()
+ .name(ASANA_CONTROLLER_SERVICE)
+ .displayName("Controller service")
+ .description("Specify which controller service to use for accessing Asana.")
+ .required(true)
+ .identifiesControllerService(AsanaClientServiceApi.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OBJECT_TYPE = new PropertyDescriptor.Builder()
+ .name(ASANA_OBJECT_TYPE)
+ .displayName("Object type to be collected")
Review Comment:
Please use short names: `Object Type`
##########
nifi-nar-bundles/nifi-asana-bundle/pom.xml:
##########
@@ -0,0 +1,65 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-nar-bundles</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-asana-bundle</artifactId>
+ <packaging>pom</packaging>
+
+ <modules>
+ <module>nifi-asana-processors</module>
+ <module>nifi-asana-services</module>
+ <module>nifi-asana-services-api</module>
+ <module>nifi-asana-nar</module>
Review Comment:
Each service/processor module should have its own nar:
- `nifi-asana-services-api-nar` (it contains the service interfaces and it is the parent of the other two nars)
- `nifi-asana-services-nar` (service implementations, the processors should not see it directly, only via the interfaces)
- `nifi-asana-processors-nar`
Example: `nifi-dropbox-bundle`
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientServiceApi;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@TriggerSerially
+@Stateful(scopes = {Scope.LOCAL}, description = "Fingerprints of items in the last successful query are stored in order to enable incremental loading and change detection.")
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.")
+@Tags({"asana", "source", "connector", "ingest"})
+@CapabilityDescription("This processor collects data from Asana")
+public class GetAsanaObject extends AbstractProcessor {
+
+ protected static final String ASANA_GID = "asana.gid";
+ protected static final String AV_NAME_COLLECT_TASKS = "asana-collect-tasks";
+ protected static final String AV_NAME_COLLECT_TASK_ATTACHMENTS = "asana-collect-task-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECTS = "asana-collect-projects";
+ protected static final String AV_NAME_COLLECT_TAGS = "asana-collect-tags";
+ protected static final String AV_NAME_COLLECT_USERS = "asana-collect-users";
+ protected static final String AV_NAME_COLLECT_PROJECT_MEMBERS = "asana-collect-project-members";
+ protected static final String AV_NAME_COLLECT_TEAMS = "asana-collect-teams";
+ protected static final String AV_NAME_COLLECT_TEAM_MEMBERS = "asana-collect-team-members";
+ protected static final String AV_NAME_COLLECT_STORIES = "asana-collect-stories";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_UPDATES = "asana-collect-project-status-updates";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS = "asana-collect-project-status-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECT_EVENTS = "asana-collect-project-events";
+ protected static final String ASANA_CONTROLLER_SERVICE = "asana-controller-service";
+ protected static final String ASANA_OBJECT_TYPE = "asana-object-type";
+ protected static final String ASANA_PROJECT_NAME = "asana-project-name";
+ protected static final String ASANA_SECTION_NAME = "asana-section-name";
+ protected static final String ASANA_TAG_NAME = "asana-tag-name";
+ protected static final String ASANA_TEAM_NAME = "asana-team-name";
+ protected static final String ASANA_OUTPUT_BATCH_SIZE = "asana-output-batch-size";
+ protected static final String REL_NAME_NEW = "new";
+ protected static final String REL_NAME_UPDATED = "updated";
+ protected static final String REL_NAME_REMOVED = "removed";
+
+ protected static final AllowableValue AV_COLLECT_TASKS = new AllowableValue(
+ AV_NAME_COLLECT_TASKS,
+ "Tasks",
+ "Collect tasks matching to the specified conditions.");
+
+ protected static final AllowableValue AV_COLLECT_TASK_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_TASK_ATTACHMENTS,
+ "Task attachments",
+ "Collect attached files of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECTS,
+ "Projects",
+ "Collect projects of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TAGS = new AllowableValue(
+ AV_NAME_COLLECT_TAGS,
+ "Tags",
+ "Collect tags of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_USERS = new AllowableValue(
+ AV_NAME_COLLECT_USERS,
+ "Users",
+ "Collect users assigned to the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_MEMBERS,
+ "Members of a project",
+ "Collect users assigned to the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAMS = new AllowableValue(
+ AV_NAME_COLLECT_TEAMS,
+ "Teams",
+ "Collect teams of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAM_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_TEAM_MEMBERS,
+ "Team members",
+ "Collect users assigned to the specified team."
+ );
+
+ protected static final AllowableValue AV_COLLECT_STORIES = new AllowableValue(
+ AV_NAME_COLLECT_STORIES,
+ "Stories of tasks",
+ "Collect stories (comments) of of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_UPDATES = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_UPDATES,
+ "Status updates of a project",
+ "Collect status updates of the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ "Attachments of status updates",
+ "Collect attached files of project status updates."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_EVENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_EVENTS,
+ "Events of a project",
+ "Collect various events happening on the specified project and on its' tasks."
+ );
+
+ protected static final PropertyDescriptor PROP_ASANA_CONTROLLER_SERVICE = new PropertyDescriptor.Builder()
+ .name(ASANA_CONTROLLER_SERVICE)
+ .displayName("Controller service")
Review Comment:
Please use a descriptive name for the controller service: `Asana Client Service`
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientServiceApi;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@TriggerSerially
Review Comment:
Only a single instance of the processor may run at a time because it loads data incrementally. `@TriggerSerially` works within a NiFi node / instance but it is not enough in a clustered environment. Please add `@PrimaryNodeOnly` as well which runs the processor only on the primary node.
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-services/src/main/java/org/apache/nifi/controller/asana/AsanaClientImpl.java:
##########
@@ -0,0 +1,303 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.controller.asana;
+
+import com.asana.Client;
+import com.asana.errors.InvalidTokenError;
+import com.asana.models.Attachment;
+import com.asana.models.Event;
+import com.asana.models.Project;
+import com.asana.models.ProjectMembership;
+import com.asana.models.ProjectStatus;
+import com.asana.models.Resource;
+import com.asana.models.ResultBodyCollection;
+import com.asana.models.Section;
+import com.asana.models.Story;
+import com.asana.models.Tag;
+import com.asana.models.Task;
+import com.asana.models.Team;
+import com.asana.models.User;
+import com.asana.models.Workspace;
+import com.asana.requests.CollectionRequest;
+import com.asana.requests.EventsRequest;
+import com.google.gson.annotations.SerializedName;
+
+import java.io.IOException;
+import java.lang.reflect.Field;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.StreamSupport;
+
+public class AsanaClientImpl implements AsanaClient {
Review Comment:
`StandardAsanaClient` may sound better than "Impl" (similar to the controller service interface / implementation).
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientServiceApi;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@TriggerSerially
+@Stateful(scopes = {Scope.LOCAL}, description = "Fingerprints of items in the last successful query are stored in order to enable incremental loading and change detection.")
Review Comment:
In a clustered environment the state needs to be shared between nodes in order to be able to continue the work when the primary node changes. Please use `Scope.CLUSTER`.
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-services/src/main/java/org/apache/nifi/controller/asana/AsanaClientService.java:
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.controller.asana;
+
+import com.asana.Client;
+import com.google.common.collect.Lists;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.resource.ResourceCardinality;
+import org.apache.nifi.components.resource.ResourceType;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.Collections;
+import java.util.List;
+
+import static org.apache.nifi.controller.asana.AsanaClientImpl.ASANA_CLIENT_OPTION_BASE_URL;
+
+@CapabilityDescription("Common service to authenticate with Asana, and to work on a specified workspace.")
+@Tags({"asana", "service", "authentication"})
+public class AsanaClientService extends AbstractControllerService implements AsanaClientServiceApi {
+
+ protected static final String ASANA_API_URL = "asana-api-url";
+ protected static final String ASANA_PERSONAL_ACCESS_TOKEN = "asana-personal-access-token";
+ protected static final String ASANA_WORKSPACE_NAME = "asana-workspace-name";
+
+ protected static final PropertyDescriptor PROP_ASANA_API_BASE_URL = new PropertyDescriptor.Builder()
+ .name(ASANA_API_URL)
+ .displayName("API URL")
+ .description("Base URL of Asana API. Leave it as default, unless you have your own Asana instance "
+ + "serving on a different URL. (typical for on-premise installations)")
+ .required(true)
+ .defaultValue(Client.DEFAULTS.get(ASANA_CLIENT_OPTION_BASE_URL).toString())
+ .identifiesExternalResource(ResourceCardinality.SINGLE, ResourceType.URL)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_PERSONAL_ACCESS_TOKEN = new PropertyDescriptor.Builder()
+ .name(ASANA_PERSONAL_ACCESS_TOKEN)
+ .displayName("Personal Access Token")
+ .description("Similarly to entering your username/password into a website, when you access "
+ + "your Asana data via the API you need to authenticate. Personal Access Token (PAT) "
+ + "is an authentication mechanism for accessing the API. You can generate a PAT from "
+ + "the Asana developer console. Refer to Asana Authentication Quick Start for detailed "
+ + "instructions on getting started.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_WORKSPACE_NAME = new PropertyDescriptor.Builder()
+ .name(ASANA_WORKSPACE_NAME)
+ .displayName("Workspace")
+ .description("Specify which Asana workspace to use. Case sensitive. "
+ + "A workspace is the highest-level organizational unit in Asana. All projects and tasks "
+ + "have an associated workspace. An organization is a special kind of workspace that "
+ + "represents a company. In an organization, you can group your projects into teams.")
+ .required(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .build();
+
+ protected static final List<PropertyDescriptor> DESCRIPTORS = Collections.unmodifiableList(Lists.newArrayList(
+ PROP_ASANA_API_BASE_URL,
+ PROP_ASANA_PERSONAL_ACCESS_TOKEN,
+ PROP_ASANA_WORKSPACE_NAME
+ ));
+
+ private String personalAccessToken;
+ private String workspaceName;
+ private String baseUrl;
Review Comment:
These fields are written and read by different threads. For this reason, please add `volatile` to ensure read consistency.
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientServiceApi;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@TriggerSerially
+@Stateful(scopes = {Scope.LOCAL}, description = "Fingerprints of items in the last successful query are stored in order to enable incremental loading and change detection.")
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.")
+@Tags({"asana", "source", "connector", "ingest"})
+@CapabilityDescription("This processor collects data from Asana")
+public class GetAsanaObject extends AbstractProcessor {
+
+ protected static final String ASANA_GID = "asana.gid";
+ protected static final String AV_NAME_COLLECT_TASKS = "asana-collect-tasks";
+ protected static final String AV_NAME_COLLECT_TASK_ATTACHMENTS = "asana-collect-task-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECTS = "asana-collect-projects";
+ protected static final String AV_NAME_COLLECT_TAGS = "asana-collect-tags";
+ protected static final String AV_NAME_COLLECT_USERS = "asana-collect-users";
+ protected static final String AV_NAME_COLLECT_PROJECT_MEMBERS = "asana-collect-project-members";
+ protected static final String AV_NAME_COLLECT_TEAMS = "asana-collect-teams";
+ protected static final String AV_NAME_COLLECT_TEAM_MEMBERS = "asana-collect-team-members";
+ protected static final String AV_NAME_COLLECT_STORIES = "asana-collect-stories";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_UPDATES = "asana-collect-project-status-updates";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS = "asana-collect-project-status-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECT_EVENTS = "asana-collect-project-events";
+ protected static final String ASANA_CONTROLLER_SERVICE = "asana-controller-service";
+ protected static final String ASANA_OBJECT_TYPE = "asana-object-type";
+ protected static final String ASANA_PROJECT_NAME = "asana-project-name";
+ protected static final String ASANA_SECTION_NAME = "asana-section-name";
+ protected static final String ASANA_TAG_NAME = "asana-tag-name";
+ protected static final String ASANA_TEAM_NAME = "asana-team-name";
+ protected static final String ASANA_OUTPUT_BATCH_SIZE = "asana-output-batch-size";
+ protected static final String REL_NAME_NEW = "new";
+ protected static final String REL_NAME_UPDATED = "updated";
+ protected static final String REL_NAME_REMOVED = "removed";
+
+ protected static final AllowableValue AV_COLLECT_TASKS = new AllowableValue(
+ AV_NAME_COLLECT_TASKS,
+ "Tasks",
+ "Collect tasks matching to the specified conditions.");
+
+ protected static final AllowableValue AV_COLLECT_TASK_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_TASK_ATTACHMENTS,
+ "Task attachments",
+ "Collect attached files of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECTS,
+ "Projects",
+ "Collect projects of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TAGS = new AllowableValue(
+ AV_NAME_COLLECT_TAGS,
+ "Tags",
+ "Collect tags of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_USERS = new AllowableValue(
+ AV_NAME_COLLECT_USERS,
+ "Users",
+ "Collect users assigned to the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_MEMBERS,
+ "Members of a project",
+ "Collect users assigned to the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAMS = new AllowableValue(
+ AV_NAME_COLLECT_TEAMS,
+ "Teams",
+ "Collect teams of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAM_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_TEAM_MEMBERS,
+ "Team members",
+ "Collect users assigned to the specified team."
+ );
+
+ protected static final AllowableValue AV_COLLECT_STORIES = new AllowableValue(
+ AV_NAME_COLLECT_STORIES,
+ "Stories of tasks",
+ "Collect stories (comments) of of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_UPDATES = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_UPDATES,
+ "Status updates of a project",
+ "Collect status updates of the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ "Attachments of status updates",
+ "Collect attached files of project status updates."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_EVENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_EVENTS,
+ "Events of a project",
+ "Collect various events happening on the specified project and on its' tasks."
+ );
+
+ protected static final PropertyDescriptor PROP_ASANA_CONTROLLER_SERVICE = new PropertyDescriptor.Builder()
+ .name(ASANA_CONTROLLER_SERVICE)
+ .displayName("Controller service")
+ .description("Specify which controller service to use for accessing Asana.")
+ .required(true)
+ .identifiesControllerService(AsanaClientServiceApi.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OBJECT_TYPE = new PropertyDescriptor.Builder()
+ .name(ASANA_OBJECT_TYPE)
+ .displayName("Object type to be collected")
+ .description("Specify what kind of objects to be collected from Asana")
+ .required(true)
+ .allowableValues(
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_PROJECTS,
+ AV_COLLECT_TAGS,
+ AV_COLLECT_USERS,
+ AV_COLLECT_PROJECT_MEMBERS,
+ AV_COLLECT_TEAMS,
+ AV_COLLECT_TEAM_MEMBERS,
+ AV_COLLECT_STORIES,
+ AV_COLLECT_PROJECT_STATUS_UPDATES,
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ AV_COLLECT_PROJECT_EVENTS)
+ .defaultValue(AV_COLLECT_TASKS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_PROJECT = new PropertyDescriptor.Builder()
+ .name(ASANA_PROJECT_NAME)
+ .displayName("Project name")
+ .description("Fetch only objects in this project. Case sensitive.")
+ .required(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(
+ PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_PROJECT_MEMBERS,
+ AV_COLLECT_STORIES,
+ AV_COLLECT_PROJECT_STATUS_UPDATES,
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ AV_COLLECT_PROJECT_EVENTS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_SECTION = new PropertyDescriptor.Builder()
+ .name(ASANA_SECTION_NAME)
+ .displayName("Section name")
+ .description("Fetch only objects in this section. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TASKS, AV_COLLECT_TASK_ATTACHMENTS, AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TAG = new PropertyDescriptor.Builder()
+ .name(ASANA_TAG_NAME)
+ .displayName("Tag")
+ .description("Fetch only objects having this tag. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TASKS, AV_COLLECT_TASK_ATTACHMENTS, AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TEAM_NAME = new PropertyDescriptor.Builder()
+ .name(ASANA_TEAM_NAME)
+ .displayName("Team")
+ .description("Team name. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TEAM_MEMBERS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OUTPUT_BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name(ASANA_OUTPUT_BATCH_SIZE)
+ .displayName("Output Batch Size")
+ .description("The number of output FlowFiles to queue before committing the process session. When set to zero, the session will be committed when all result set rows "
+ + "have been processed and the output FlowFiles are ready for transfer to the downstream relationship. For large result sets, this can cause a large burst of FlowFiles "
+ + "to be transferred at the end of processor execution. If this property is set, then when the specified number of FlowFiles are ready for transfer, then the session will "
+ + "be committed, thus releasing the FlowFiles to the downstream relationship.")
+ .defaultValue("0")
+ .required(true)
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+
+ protected static final List<PropertyDescriptor> DESCRIPTORS = Collections.unmodifiableList(Lists.newArrayList(
+ PROP_ASANA_CONTROLLER_SERVICE,
+ PROP_ASANA_OBJECT_TYPE,
+ PROP_ASANA_PROJECT,
+ PROP_ASANA_SECTION,
+ PROP_ASANA_TEAM_NAME,
+ PROP_ASANA_TAG,
+ PROP_ASANA_OUTPUT_BATCH_SIZE
+ ));
+
+ protected static final Relationship REL_NEW = new Relationship.Builder()
+ .name(REL_NAME_NEW)
+ .description("Newly collected objects are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_UPDATED = new Relationship.Builder()
+ .name(REL_NAME_UPDATED)
+ .description("Objects that have already been collected earlier, but were updated since, are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_REMOVED = new Relationship.Builder()
+ .name(REL_NAME_REMOVED)
+ .description("Notification about deleted objects are routed to this relationship. "
+ + "Flow files will not have any payload. IDs of the resources no longer exist "
+ + "are carried by the asana.gid attribute of the generated FlowFiles.")
+ .build();
+
+ protected static final Set<Relationship> RELATIONSHIPS = Collections.unmodifiableSet(Sets.newHashSet(
+ REL_NEW,
+ REL_UPDATED,
+ REL_REMOVED
+ ));
+
+ final Scope STATE_STORAGE_SCOPE = Scope.LOCAL;
+
+ AsanaClientServiceApi controllerService;
Review Comment:
It should be a local variable in `onScheduled()`.
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientServiceApi;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@TriggerSerially
+@Stateful(scopes = {Scope.LOCAL}, description = "Fingerprints of items in the last successful query are stored in order to enable incremental loading and change detection.")
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.")
+@Tags({"asana", "source", "connector", "ingest"})
+@CapabilityDescription("This processor collects data from Asana")
+public class GetAsanaObject extends AbstractProcessor {
+
+ protected static final String ASANA_GID = "asana.gid";
+ protected static final String AV_NAME_COLLECT_TASKS = "asana-collect-tasks";
+ protected static final String AV_NAME_COLLECT_TASK_ATTACHMENTS = "asana-collect-task-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECTS = "asana-collect-projects";
+ protected static final String AV_NAME_COLLECT_TAGS = "asana-collect-tags";
+ protected static final String AV_NAME_COLLECT_USERS = "asana-collect-users";
+ protected static final String AV_NAME_COLLECT_PROJECT_MEMBERS = "asana-collect-project-members";
+ protected static final String AV_NAME_COLLECT_TEAMS = "asana-collect-teams";
+ protected static final String AV_NAME_COLLECT_TEAM_MEMBERS = "asana-collect-team-members";
+ protected static final String AV_NAME_COLLECT_STORIES = "asana-collect-stories";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_UPDATES = "asana-collect-project-status-updates";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS = "asana-collect-project-status-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECT_EVENTS = "asana-collect-project-events";
+ protected static final String ASANA_CONTROLLER_SERVICE = "asana-controller-service";
+ protected static final String ASANA_OBJECT_TYPE = "asana-object-type";
+ protected static final String ASANA_PROJECT_NAME = "asana-project-name";
+ protected static final String ASANA_SECTION_NAME = "asana-section-name";
+ protected static final String ASANA_TAG_NAME = "asana-tag-name";
+ protected static final String ASANA_TEAM_NAME = "asana-team-name";
+ protected static final String ASANA_OUTPUT_BATCH_SIZE = "asana-output-batch-size";
+ protected static final String REL_NAME_NEW = "new";
+ protected static final String REL_NAME_UPDATED = "updated";
+ protected static final String REL_NAME_REMOVED = "removed";
+
+ protected static final AllowableValue AV_COLLECT_TASKS = new AllowableValue(
+ AV_NAME_COLLECT_TASKS,
+ "Tasks",
+ "Collect tasks matching to the specified conditions.");
+
+ protected static final AllowableValue AV_COLLECT_TASK_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_TASK_ATTACHMENTS,
+ "Task attachments",
+ "Collect attached files of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECTS,
+ "Projects",
+ "Collect projects of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TAGS = new AllowableValue(
+ AV_NAME_COLLECT_TAGS,
+ "Tags",
+ "Collect tags of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_USERS = new AllowableValue(
+ AV_NAME_COLLECT_USERS,
+ "Users",
+ "Collect users assigned to the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_MEMBERS,
+ "Members of a project",
+ "Collect users assigned to the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAMS = new AllowableValue(
+ AV_NAME_COLLECT_TEAMS,
+ "Teams",
+ "Collect teams of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAM_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_TEAM_MEMBERS,
+ "Team members",
+ "Collect users assigned to the specified team."
+ );
+
+ protected static final AllowableValue AV_COLLECT_STORIES = new AllowableValue(
+ AV_NAME_COLLECT_STORIES,
+ "Stories of tasks",
+ "Collect stories (comments) of of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_UPDATES = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_UPDATES,
+ "Status updates of a project",
+ "Collect status updates of the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ "Attachments of status updates",
+ "Collect attached files of project status updates."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_EVENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_EVENTS,
+ "Events of a project",
+ "Collect various events happening on the specified project and on its' tasks."
+ );
+
+ protected static final PropertyDescriptor PROP_ASANA_CONTROLLER_SERVICE = new PropertyDescriptor.Builder()
+ .name(ASANA_CONTROLLER_SERVICE)
+ .displayName("Controller service")
+ .description("Specify which controller service to use for accessing Asana.")
+ .required(true)
+ .identifiesControllerService(AsanaClientServiceApi.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OBJECT_TYPE = new PropertyDescriptor.Builder()
+ .name(ASANA_OBJECT_TYPE)
+ .displayName("Object type to be collected")
+ .description("Specify what kind of objects to be collected from Asana")
+ .required(true)
+ .allowableValues(
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_PROJECTS,
+ AV_COLLECT_TAGS,
+ AV_COLLECT_USERS,
+ AV_COLLECT_PROJECT_MEMBERS,
+ AV_COLLECT_TEAMS,
+ AV_COLLECT_TEAM_MEMBERS,
+ AV_COLLECT_STORIES,
+ AV_COLLECT_PROJECT_STATUS_UPDATES,
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ AV_COLLECT_PROJECT_EVENTS)
+ .defaultValue(AV_COLLECT_TASKS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_PROJECT = new PropertyDescriptor.Builder()
+ .name(ASANA_PROJECT_NAME)
+ .displayName("Project name")
+ .description("Fetch only objects in this project. Case sensitive.")
+ .required(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(
+ PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_PROJECT_MEMBERS,
+ AV_COLLECT_STORIES,
+ AV_COLLECT_PROJECT_STATUS_UPDATES,
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ AV_COLLECT_PROJECT_EVENTS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_SECTION = new PropertyDescriptor.Builder()
+ .name(ASANA_SECTION_NAME)
+ .displayName("Section name")
+ .description("Fetch only objects in this section. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TASKS, AV_COLLECT_TASK_ATTACHMENTS, AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TAG = new PropertyDescriptor.Builder()
+ .name(ASANA_TAG_NAME)
+ .displayName("Tag")
+ .description("Fetch only objects having this tag. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TASKS, AV_COLLECT_TASK_ATTACHMENTS, AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TEAM_NAME = new PropertyDescriptor.Builder()
+ .name(ASANA_TEAM_NAME)
+ .displayName("Team")
+ .description("Team name. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TEAM_MEMBERS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OUTPUT_BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name(ASANA_OUTPUT_BATCH_SIZE)
+ .displayName("Output Batch Size")
+ .description("The number of output FlowFiles to queue before committing the process session. When set to zero, the session will be committed when all result set rows "
+ + "have been processed and the output FlowFiles are ready for transfer to the downstream relationship. For large result sets, this can cause a large burst of FlowFiles "
+ + "to be transferred at the end of processor execution. If this property is set, then when the specified number of FlowFiles are ready for transfer, then the session will "
+ + "be committed, thus releasing the FlowFiles to the downstream relationship.")
+ .defaultValue("0")
+ .required(true)
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+
+ protected static final List<PropertyDescriptor> DESCRIPTORS = Collections.unmodifiableList(Lists.newArrayList(
+ PROP_ASANA_CONTROLLER_SERVICE,
+ PROP_ASANA_OBJECT_TYPE,
+ PROP_ASANA_PROJECT,
+ PROP_ASANA_SECTION,
+ PROP_ASANA_TEAM_NAME,
+ PROP_ASANA_TAG,
+ PROP_ASANA_OUTPUT_BATCH_SIZE
+ ));
+
+ protected static final Relationship REL_NEW = new Relationship.Builder()
+ .name(REL_NAME_NEW)
+ .description("Newly collected objects are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_UPDATED = new Relationship.Builder()
+ .name(REL_NAME_UPDATED)
+ .description("Objects that have already been collected earlier, but were updated since, are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_REMOVED = new Relationship.Builder()
+ .name(REL_NAME_REMOVED)
+ .description("Notification about deleted objects are routed to this relationship. "
+ + "Flow files will not have any payload. IDs of the resources no longer exist "
+ + "are carried by the asana.gid attribute of the generated FlowFiles.")
+ .build();
+
+ protected static final Set<Relationship> RELATIONSHIPS = Collections.unmodifiableSet(Sets.newHashSet(
+ REL_NEW,
+ REL_UPDATED,
+ REL_REMOVED
+ ));
+
+ final Scope STATE_STORAGE_SCOPE = Scope.LOCAL;
Review Comment:
```suggestion
private static final Scope STATE_STORAGE_SCOPE = Scope.LOCAL;
```
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientServiceApi;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@TriggerSerially
+@Stateful(scopes = {Scope.LOCAL}, description = "Fingerprints of items in the last successful query are stored in order to enable incremental loading and change detection.")
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.")
+@Tags({"asana", "source", "connector", "ingest"})
+@CapabilityDescription("This processor collects data from Asana")
+public class GetAsanaObject extends AbstractProcessor {
+
+ protected static final String ASANA_GID = "asana.gid";
+ protected static final String AV_NAME_COLLECT_TASKS = "asana-collect-tasks";
+ protected static final String AV_NAME_COLLECT_TASK_ATTACHMENTS = "asana-collect-task-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECTS = "asana-collect-projects";
+ protected static final String AV_NAME_COLLECT_TAGS = "asana-collect-tags";
+ protected static final String AV_NAME_COLLECT_USERS = "asana-collect-users";
+ protected static final String AV_NAME_COLLECT_PROJECT_MEMBERS = "asana-collect-project-members";
+ protected static final String AV_NAME_COLLECT_TEAMS = "asana-collect-teams";
+ protected static final String AV_NAME_COLLECT_TEAM_MEMBERS = "asana-collect-team-members";
+ protected static final String AV_NAME_COLLECT_STORIES = "asana-collect-stories";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_UPDATES = "asana-collect-project-status-updates";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS = "asana-collect-project-status-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECT_EVENTS = "asana-collect-project-events";
+ protected static final String ASANA_CONTROLLER_SERVICE = "asana-controller-service";
+ protected static final String ASANA_OBJECT_TYPE = "asana-object-type";
+ protected static final String ASANA_PROJECT_NAME = "asana-project-name";
+ protected static final String ASANA_SECTION_NAME = "asana-section-name";
+ protected static final String ASANA_TAG_NAME = "asana-tag-name";
+ protected static final String ASANA_TEAM_NAME = "asana-team-name";
+ protected static final String ASANA_OUTPUT_BATCH_SIZE = "asana-output-batch-size";
+ protected static final String REL_NAME_NEW = "new";
+ protected static final String REL_NAME_UPDATED = "updated";
+ protected static final String REL_NAME_REMOVED = "removed";
+
+ protected static final AllowableValue AV_COLLECT_TASKS = new AllowableValue(
+ AV_NAME_COLLECT_TASKS,
+ "Tasks",
+ "Collect tasks matching to the specified conditions.");
+
+ protected static final AllowableValue AV_COLLECT_TASK_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_TASK_ATTACHMENTS,
+ "Task attachments",
+ "Collect attached files of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECTS,
+ "Projects",
+ "Collect projects of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TAGS = new AllowableValue(
+ AV_NAME_COLLECT_TAGS,
+ "Tags",
+ "Collect tags of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_USERS = new AllowableValue(
+ AV_NAME_COLLECT_USERS,
+ "Users",
+ "Collect users assigned to the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_MEMBERS,
+ "Members of a project",
+ "Collect users assigned to the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAMS = new AllowableValue(
+ AV_NAME_COLLECT_TEAMS,
+ "Teams",
+ "Collect teams of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAM_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_TEAM_MEMBERS,
+ "Team members",
+ "Collect users assigned to the specified team."
+ );
+
+ protected static final AllowableValue AV_COLLECT_STORIES = new AllowableValue(
+ AV_NAME_COLLECT_STORIES,
+ "Stories of tasks",
+ "Collect stories (comments) of of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_UPDATES = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_UPDATES,
+ "Status updates of a project",
+ "Collect status updates of the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ "Attachments of status updates",
+ "Collect attached files of project status updates."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_EVENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_EVENTS,
+ "Events of a project",
+ "Collect various events happening on the specified project and on its' tasks."
+ );
+
+ protected static final PropertyDescriptor PROP_ASANA_CONTROLLER_SERVICE = new PropertyDescriptor.Builder()
+ .name(ASANA_CONTROLLER_SERVICE)
+ .displayName("Controller service")
+ .description("Specify which controller service to use for accessing Asana.")
+ .required(true)
+ .identifiesControllerService(AsanaClientServiceApi.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OBJECT_TYPE = new PropertyDescriptor.Builder()
+ .name(ASANA_OBJECT_TYPE)
+ .displayName("Object type to be collected")
+ .description("Specify what kind of objects to be collected from Asana")
+ .required(true)
+ .allowableValues(
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_PROJECTS,
+ AV_COLLECT_TAGS,
+ AV_COLLECT_USERS,
+ AV_COLLECT_PROJECT_MEMBERS,
+ AV_COLLECT_TEAMS,
+ AV_COLLECT_TEAM_MEMBERS,
+ AV_COLLECT_STORIES,
+ AV_COLLECT_PROJECT_STATUS_UPDATES,
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ AV_COLLECT_PROJECT_EVENTS)
+ .defaultValue(AV_COLLECT_TASKS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_PROJECT = new PropertyDescriptor.Builder()
+ .name(ASANA_PROJECT_NAME)
+ .displayName("Project name")
+ .description("Fetch only objects in this project. Case sensitive.")
+ .required(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(
+ PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_PROJECT_MEMBERS,
+ AV_COLLECT_STORIES,
+ AV_COLLECT_PROJECT_STATUS_UPDATES,
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ AV_COLLECT_PROJECT_EVENTS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_SECTION = new PropertyDescriptor.Builder()
+ .name(ASANA_SECTION_NAME)
+ .displayName("Section name")
+ .description("Fetch only objects in this section. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TASKS, AV_COLLECT_TASK_ATTACHMENTS, AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TAG = new PropertyDescriptor.Builder()
+ .name(ASANA_TAG_NAME)
+ .displayName("Tag")
+ .description("Fetch only objects having this tag. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TASKS, AV_COLLECT_TASK_ATTACHMENTS, AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TEAM_NAME = new PropertyDescriptor.Builder()
+ .name(ASANA_TEAM_NAME)
+ .displayName("Team")
+ .description("Team name. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TEAM_MEMBERS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OUTPUT_BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name(ASANA_OUTPUT_BATCH_SIZE)
+ .displayName("Output Batch Size")
+ .description("The number of output FlowFiles to queue before committing the process session. When set to zero, the session will be committed when all result set rows "
+ + "have been processed and the output FlowFiles are ready for transfer to the downstream relationship. For large result sets, this can cause a large burst of FlowFiles "
+ + "to be transferred at the end of processor execution. If this property is set, then when the specified number of FlowFiles are ready for transfer, then the session will "
+ + "be committed, thus releasing the FlowFiles to the downstream relationship.")
+ .defaultValue("0")
+ .required(true)
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+
+ protected static final List<PropertyDescriptor> DESCRIPTORS = Collections.unmodifiableList(Lists.newArrayList(
Review Comment:
We try to avoid using `guava` in our code, even if it comes transitively. Please use some alternative solution like `Arrays.asList()` for lists and `new HashSet<>(Arrays.asList())` for sets.
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientServiceApi;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@TriggerSerially
+@Stateful(scopes = {Scope.LOCAL}, description = "Fingerprints of items in the last successful query are stored in order to enable incremental loading and change detection.")
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.")
+@Tags({"asana", "source", "connector", "ingest"})
+@CapabilityDescription("This processor collects data from Asana")
+public class GetAsanaObject extends AbstractProcessor {
+
+ protected static final String ASANA_GID = "asana.gid";
+ protected static final String AV_NAME_COLLECT_TASKS = "asana-collect-tasks";
+ protected static final String AV_NAME_COLLECT_TASK_ATTACHMENTS = "asana-collect-task-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECTS = "asana-collect-projects";
+ protected static final String AV_NAME_COLLECT_TAGS = "asana-collect-tags";
+ protected static final String AV_NAME_COLLECT_USERS = "asana-collect-users";
+ protected static final String AV_NAME_COLLECT_PROJECT_MEMBERS = "asana-collect-project-members";
+ protected static final String AV_NAME_COLLECT_TEAMS = "asana-collect-teams";
+ protected static final String AV_NAME_COLLECT_TEAM_MEMBERS = "asana-collect-team-members";
+ protected static final String AV_NAME_COLLECT_STORIES = "asana-collect-stories";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_UPDATES = "asana-collect-project-status-updates";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS = "asana-collect-project-status-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECT_EVENTS = "asana-collect-project-events";
+ protected static final String ASANA_CONTROLLER_SERVICE = "asana-controller-service";
+ protected static final String ASANA_OBJECT_TYPE = "asana-object-type";
+ protected static final String ASANA_PROJECT_NAME = "asana-project-name";
+ protected static final String ASANA_SECTION_NAME = "asana-section-name";
+ protected static final String ASANA_TAG_NAME = "asana-tag-name";
+ protected static final String ASANA_TEAM_NAME = "asana-team-name";
+ protected static final String ASANA_OUTPUT_BATCH_SIZE = "asana-output-batch-size";
+ protected static final String REL_NAME_NEW = "new";
+ protected static final String REL_NAME_UPDATED = "updated";
+ protected static final String REL_NAME_REMOVED = "removed";
+
+ protected static final AllowableValue AV_COLLECT_TASKS = new AllowableValue(
+ AV_NAME_COLLECT_TASKS,
+ "Tasks",
+ "Collect tasks matching to the specified conditions.");
+
+ protected static final AllowableValue AV_COLLECT_TASK_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_TASK_ATTACHMENTS,
+ "Task attachments",
+ "Collect attached files of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECTS,
+ "Projects",
+ "Collect projects of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TAGS = new AllowableValue(
+ AV_NAME_COLLECT_TAGS,
+ "Tags",
+ "Collect tags of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_USERS = new AllowableValue(
+ AV_NAME_COLLECT_USERS,
+ "Users",
+ "Collect users assigned to the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_MEMBERS,
+ "Members of a project",
+ "Collect users assigned to the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAMS = new AllowableValue(
+ AV_NAME_COLLECT_TEAMS,
+ "Teams",
+ "Collect teams of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAM_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_TEAM_MEMBERS,
+ "Team members",
+ "Collect users assigned to the specified team."
+ );
+
+ protected static final AllowableValue AV_COLLECT_STORIES = new AllowableValue(
+ AV_NAME_COLLECT_STORIES,
+ "Stories of tasks",
+ "Collect stories (comments) of of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_UPDATES = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_UPDATES,
+ "Status updates of a project",
+ "Collect status updates of the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ "Attachments of status updates",
+ "Collect attached files of project status updates."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_EVENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_EVENTS,
+ "Events of a project",
+ "Collect various events happening on the specified project and on its' tasks."
+ );
+
+ protected static final PropertyDescriptor PROP_ASANA_CONTROLLER_SERVICE = new PropertyDescriptor.Builder()
+ .name(ASANA_CONTROLLER_SERVICE)
+ .displayName("Controller service")
+ .description("Specify which controller service to use for accessing Asana.")
+ .required(true)
+ .identifiesControllerService(AsanaClientServiceApi.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OBJECT_TYPE = new PropertyDescriptor.Builder()
+ .name(ASANA_OBJECT_TYPE)
+ .displayName("Object type to be collected")
+ .description("Specify what kind of objects to be collected from Asana")
+ .required(true)
+ .allowableValues(
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_PROJECTS,
+ AV_COLLECT_TAGS,
+ AV_COLLECT_USERS,
+ AV_COLLECT_PROJECT_MEMBERS,
+ AV_COLLECT_TEAMS,
+ AV_COLLECT_TEAM_MEMBERS,
+ AV_COLLECT_STORIES,
+ AV_COLLECT_PROJECT_STATUS_UPDATES,
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ AV_COLLECT_PROJECT_EVENTS)
+ .defaultValue(AV_COLLECT_TASKS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_PROJECT = new PropertyDescriptor.Builder()
+ .name(ASANA_PROJECT_NAME)
+ .displayName("Project name")
+ .description("Fetch only objects in this project. Case sensitive.")
+ .required(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(
+ PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_PROJECT_MEMBERS,
+ AV_COLLECT_STORIES,
+ AV_COLLECT_PROJECT_STATUS_UPDATES,
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ AV_COLLECT_PROJECT_EVENTS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_SECTION = new PropertyDescriptor.Builder()
+ .name(ASANA_SECTION_NAME)
+ .displayName("Section name")
+ .description("Fetch only objects in this section. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TASKS, AV_COLLECT_TASK_ATTACHMENTS, AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TAG = new PropertyDescriptor.Builder()
+ .name(ASANA_TAG_NAME)
+ .displayName("Tag")
+ .description("Fetch only objects having this tag. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TASKS, AV_COLLECT_TASK_ATTACHMENTS, AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TEAM_NAME = new PropertyDescriptor.Builder()
+ .name(ASANA_TEAM_NAME)
+ .displayName("Team")
+ .description("Team name. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TEAM_MEMBERS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OUTPUT_BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name(ASANA_OUTPUT_BATCH_SIZE)
+ .displayName("Output Batch Size")
+ .description("The number of output FlowFiles to queue before committing the process session. When set to zero, the session will be committed when all result set rows "
+ + "have been processed and the output FlowFiles are ready for transfer to the downstream relationship. For large result sets, this can cause a large burst of FlowFiles "
+ + "to be transferred at the end of processor execution. If this property is set, then when the specified number of FlowFiles are ready for transfer, then the session will "
+ + "be committed, thus releasing the FlowFiles to the downstream relationship.")
+ .defaultValue("0")
+ .required(true)
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+
+ protected static final List<PropertyDescriptor> DESCRIPTORS = Collections.unmodifiableList(Lists.newArrayList(
+ PROP_ASANA_CONTROLLER_SERVICE,
+ PROP_ASANA_OBJECT_TYPE,
+ PROP_ASANA_PROJECT,
+ PROP_ASANA_SECTION,
+ PROP_ASANA_TEAM_NAME,
+ PROP_ASANA_TAG,
+ PROP_ASANA_OUTPUT_BATCH_SIZE
+ ));
+
+ protected static final Relationship REL_NEW = new Relationship.Builder()
+ .name(REL_NAME_NEW)
+ .description("Newly collected objects are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_UPDATED = new Relationship.Builder()
+ .name(REL_NAME_UPDATED)
+ .description("Objects that have already been collected earlier, but were updated since, are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_REMOVED = new Relationship.Builder()
+ .name(REL_NAME_REMOVED)
+ .description("Notification about deleted objects are routed to this relationship. "
+ + "Flow files will not have any payload. IDs of the resources no longer exist "
+ + "are carried by the asana.gid attribute of the generated FlowFiles.")
+ .build();
+
+ protected static final Set<Relationship> RELATIONSHIPS = Collections.unmodifiableSet(Sets.newHashSet(
+ REL_NEW,
+ REL_UPDATED,
+ REL_REMOVED
+ ));
+
+ final Scope STATE_STORAGE_SCOPE = Scope.LOCAL;
+
+ AsanaClientServiceApi controllerService;
+ AsanaObjectFetcher objectFetcher;
+ private Integer batchSize;
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return RELATIONSHIPS;
+ }
+
+ @Override
+ public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return DESCRIPTORS;
+ }
+
+ @OnScheduled
+ public synchronized void onScheduled(final ProcessContext context) throws InitializationException {
+ controllerService = context.getProperty(PROP_ASANA_CONTROLLER_SERVICE).asControllerService(AsanaClientServiceApi.class);
+ AsanaClient client = controllerService.createClient();
+ batchSize = context.getProperty(PROP_ASANA_OUTPUT_BATCH_SIZE).asInteger();
+
+ try {
+ getLogger().debug("Initializing object fetcher...");
+ objectFetcher = createObjectFetcher(context, client);
+ } catch (Exception e) {
+ throw new InitializationException(e);
+ }
+ }
+
+ @Override
+ public synchronized void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ try {
+ Map<String, String> state = recoverState(context).orElse(Collections.emptyMap());
+ getLogger().debug("Attempting to load state: {}", state);
+ objectFetcher.loadState(state);
+ } catch (Exception e) {
+ getLogger().info("Failed to recover state. Falling back to clean start.");
+ objectFetcher.clearState();
+ }
+ getLogger().debug("Initial state: {}", objectFetcher.saveState());
+
+ Collection<FlowFile> newItems = new ArrayList<>();
+ Collection<FlowFile> updatedItems = new ArrayList<>();
+ Collection<FlowFile> removedItems = new ArrayList<>();
+
+ AsanaObject nextObject;
+ while ((batchSize == 0 || (batchSize > (newItems.size() + updatedItems.size() + removedItems.size()))) && (nextObject = objectFetcher.fetchNext()) != null) {
+ final Map<String, String> attributes = new HashMap<>(2);
+ attributes.put(CoreAttributes.MIME_TYPE.key(), ContentType.APPLICATION_JSON.getMimeType());
+ attributes.put(ASANA_GID, nextObject.getGid());
+ FlowFile flowFile = createFlowFileWithStringPayload(session, nextObject.getContent());
+ flowFile = session.putAllAttributes(flowFile, attributes);
+
+ switch (nextObject.getState()) {
+ case NEW:
+ newItems.add(flowFile);
+ break;
+ case REMOVED:
+ removedItems.add(flowFile);
+ break;
+ default:
+ updatedItems.add(flowFile);
+ }
+ }
+
+ if (newItems.isEmpty() && updatedItems.isEmpty() && removedItems.isEmpty()) {
+ context.yield();
+ getLogger().debug("Yielding, as there are no new FlowFiles.");
+ return;
+ }
+
+ session.transfer(newItems, REL_NEW);
+ session.transfer(updatedItems, REL_UPDATED);
+ session.transfer(removedItems, REL_REMOVED);
+
+ session.commitAsync();
+ Map<String, String> state = objectFetcher.saveState();
+ try {
+ persistState(state, context);
+ } catch (IOException e) {
+ throw new ProcessException(e);
+ }
+ getLogger().debug(
+ "New state after transferring {} new, {} updated, and {} removed items: {}",
+ newItems.size(), updatedItems.size(), removedItems.size(), state);
+ }
+
+ protected AsanaObjectFetcher createObjectFetcher(final ProcessContext context, AsanaClient client) {
+ final String objectType = context.getProperty(PROP_ASANA_OBJECT_TYPE).getValue();
+ final String projectName = context.getProperty(PROP_ASANA_PROJECT).getValue();
+ final String sectionName = context.getProperty(PROP_ASANA_SECTION).getValue();
+ final String teamName = context.getProperty(PROP_ASANA_TEAM_NAME).getValue();
+ final String tagName = context.getProperty(PROP_ASANA_TAG).getValue();
+
+ switch (objectType) {
+ case AV_NAME_COLLECT_TASKS:
+ return new AsanaTaskFetcher(client, projectName, sectionName, tagName);
+ case AV_NAME_COLLECT_PROJECTS:
+ return new AsanaProjectFetcher(client);
+ case AV_NAME_COLLECT_PROJECT_EVENTS:
+ return new AsanaProjectEventFetcher(client, projectName);
+ case AV_NAME_COLLECT_PROJECT_MEMBERS:
+ return new AsanaProjectMembershipFetcher(client, projectName);
+ case AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS:
+ return new AsanaProjectStatusAttachmentFetcher(client, projectName);
+ case AV_NAME_COLLECT_PROJECT_STATUS_UPDATES:
+ return new AsanaProjectStatusFetcher(client, projectName);
+ case AV_NAME_COLLECT_STORIES:
+ return new AsanaStoryFetcher(client, projectName, sectionName, tagName);
+ case AV_NAME_COLLECT_TAGS:
+ return new AsanaTagFetcher(client);
+ case AV_NAME_COLLECT_TASK_ATTACHMENTS:
+ return new AsanaTaskAttachmentFetcher(client, projectName, sectionName, tagName);
+ case AV_NAME_COLLECT_TEAMS:
+ return new AsanaTeamFetcher(client);
+ case AV_NAME_COLLECT_TEAM_MEMBERS:
+ return new AsanaTeamMemberFetcher(client, teamName);
+ case AV_NAME_COLLECT_USERS:
+ return new AsanaUserFetcher(client);
+ }
+
+ throw new RuntimeException("Cannot fetch objects of type: " + objectType);
Review Comment:
Please do not throw raw `RuntimeException`s, use NiFi's `ProcessException` instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on PR #6504:
URL: https://github.com/apache/nifi/pull/6504#issuecomment-1279006794
@turcsanyip Thank you for reviewing. I have attempted to address all the issues you pointed out. Please check again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
exceptionfactory commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r1030605452
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/AsanaObjectType.java:
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import java.util.Arrays;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.DescribedValue;
+
+public enum AsanaObjectType implements DescribedValue {
+ AV_COLLECT_TASKS(
+ "asana-collect-tasks",
+ "Tasks", ""
+ + "Collect tasks matching to the specified conditions."
+ ),
+ AV_COLLECT_TASK_ATTACHMENTS(
+ "asana-collect-task-attachments",
+ "Task Attachments",
+ "Collect attached files of tasks matching to the specified conditions."
+ ),
+ AV_COLLECT_PROJECTS(
+ "asana-collect-projects",
+ "Projects",
+ "Collect projects of the workspace."
+ ),
+ AV_COLLECT_TAGS(
+ "asana-collect-tags",
+ "Tags",
+ "Collect tags of the workspace."
+ ),
+ AV_COLLECT_USERS(
+ "asana-collect-users",
+ "Users",
+ "Collect users assigned to the workspace."
+ ),
+ AV_COLLECT_PROJECT_MEMBERS(
+ "asana-collect-project-members",
+ "Members of a Project",
+ "Collect users assigned to the specified project."
+ ),
+ AV_COLLECT_TEAMS(
+ "asana-collect-teams",
+ "Teams",
+ "Collect teams of the workspace."
+ ),
+ AV_COLLECT_TEAM_MEMBERS(
+ "asana-collect-team-members",
+ "Team Members",
+ "Collect users assigned to the specified team."
+ ),
+ AV_COLLECT_STORIES(
+ "asana-collect-stories",
+ "Stories of Tasks",
+ "Collect stories (comments) of of tasks matching to the specified conditions."
+ ),
+ AV_COLLECT_PROJECT_STATUS_UPDATES(
+ "asana-collect-project-status-updates",
+ "Status Updates of a Project",
+ "Collect status updates of the specified project."
+ ),
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS(
+ "asana-collect-project-status-attachments",
+ "Attachments of Status Updates",
+ "Collect attached files of project status updates."
+ ),
+ AV_COLLECT_PROJECT_EVENTS(
+ "asana-collect-project-events",
+ "Events of a Project",
+ "Collect various events happening on the specified project and on its' tasks."
+ );
+
+ private final String value;
+ private final String displayName;
+ private final String description;
+
+ AsanaObjectType(String value, String displayName, String description) {
+ this.value = value;
+ this.displayName = displayName;
+ this.description = description;
+ }
+
+ @Override
+ public String getValue() {
+ return value;
+ }
+
+ @Override
+ public String getDisplayName() {
+ return displayName;
+ }
+
+ @Override
+ public String getDescription() {
+ return description;
+ }
+
+ public AllowableValue createAllowableValue() {
+ return new AllowableValue(value, displayName, description);
+ }
Review Comment:
This method should be unnecessary, since `getValue()` can be used directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r1030394878
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-services-api/src/main/java/org/apache/nifi/controller/asana/AsanaClient.java:
##########
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.controller.asana;
+
+import com.asana.models.Attachment;
+import com.asana.models.Project;
+import com.asana.models.ProjectMembership;
+import com.asana.models.ProjectStatus;
+import com.asana.models.Section;
+import com.asana.models.Story;
+import com.asana.models.Tag;
+import com.asana.models.Task;
+import com.asana.models.Team;
+import com.asana.models.User;
+
+import java.util.Map;
+
+/**
+ * This interface represents a client to Asana REST server, with some basic filtering options built in.
+ */
+public interface AsanaClient {
+ /**
+ * Find & retrieve an Asana project by its name. If multiple projects match, returns the first.
+ * If there is no match, then {@link RuntimeException} is thrown. Note that constant ordering
Review Comment:
Done in 20cd9e9
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r1035802134
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,393 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import static java.lang.String.format;
+import static java.util.Collections.singletonMap;
+import static java.util.stream.Collectors.groupingBy;
+import static java.util.stream.Collectors.joining;
+import static org.apache.commons.collections4.ListUtils.partition;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_EVENTS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_MEMBERS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_STATUS_ATTACHMENTS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_STATUS_UPDATES;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_STORIES;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_TASKS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_TASK_ATTACHMENTS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_TEAM_MEMBERS;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientProviderService;
+import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaObjectState;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+@TriggerSerially
+@PrimaryNodeOnly
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.")
+@Tags({"asana", "source", "ingest"})
+@CapabilityDescription("This processor collects data from Asana")
+public class GetAsanaObject extends AbstractProcessor {
+
+ protected static final String ASANA_GID = "asana.gid";
+ protected static final String ASANA_CLIENT_SERVICE = "asana-controller-service";
+ protected static final String DISTRIBUTED_CACHE_SERVICE = "distributed-cache-service";
+ protected static final String ASANA_OBJECT_TYPE = "asana-object-type";
+ protected static final String ASANA_PROJECT_NAME = "asana-project-name";
+ protected static final String ASANA_SECTION_NAME = "asana-section-name";
+ protected static final String ASANA_TAG_NAME = "asana-tag-name";
+ protected static final String ASANA_TEAM_NAME = "asana-team-name";
+ protected static final String ASANA_OUTPUT_BATCH_SIZE = "asana-output-batch-size";
+ protected static final String REL_NAME_NEW = "new";
+ protected static final String REL_NAME_UPDATED = "updated";
+ protected static final String REL_NAME_REMOVED = "removed";
+
+ protected static final PropertyDescriptor PROP_ASANA_CLIENT_SERVICE = new PropertyDescriptor.Builder()
+ .name(ASANA_CLIENT_SERVICE)
+ .displayName("Asana Client Service")
+ .description("Specify which controller service to use for accessing Asana.")
+ .required(true)
+ .identifiesControllerService(AsanaClientProviderService.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_DISTRIBUTED_CACHE_SERVICE = new Builder()
+ .name(DISTRIBUTED_CACHE_SERVICE)
+ .displayName("Distributed Cache Service")
+ .description("Cache service to store fetched item fingerprints. These, from the last successful query"
+ + " are stored, in order to enable incremental loading and change detection.")
+ .required(true)
+ .identifiesControllerService(DistributedMapCacheClient.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OBJECT_TYPE = new PropertyDescriptor.Builder()
+ .name(ASANA_OBJECT_TYPE)
+ .displayName("Object Type")
+ .description("Specify what kind of objects to be collected from Asana")
+ .required(true)
+ .allowableValues(AsanaObjectType.class)
+ .defaultValue(AV_COLLECT_TASKS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_PROJECT = new PropertyDescriptor.Builder()
+ .name(ASANA_PROJECT_NAME)
+ .displayName("Project Name")
+ .description("Fetch only objects in this project. Case sensitive.")
+ .required(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(
+ PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS.getValue(),
+ AV_COLLECT_TASK_ATTACHMENTS.getValue(),
+ AV_COLLECT_PROJECT_MEMBERS.getValue(),
+ AV_COLLECT_STORIES.getValue(),
+ AV_COLLECT_PROJECT_STATUS_UPDATES.getValue(),
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS.getValue(),
+ AV_COLLECT_PROJECT_EVENTS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_SECTION = new PropertyDescriptor.Builder()
+ .name(ASANA_SECTION_NAME)
+ .displayName("Section Name")
+ .description("Fetch only objects in this section. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS.getValue(),
+ AV_COLLECT_TASK_ATTACHMENTS.getValue(),
+ AV_COLLECT_STORIES.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TAG = new PropertyDescriptor.Builder()
+ .name(ASANA_TAG_NAME)
+ .displayName("Tag")
+ .description("Fetch only objects having this tag. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS.getValue(),
+ AV_COLLECT_TASK_ATTACHMENTS.getValue(),
+ AV_COLLECT_STORIES.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TEAM_NAME = new PropertyDescriptor.Builder()
+ .name(ASANA_TEAM_NAME)
+ .displayName("Team")
+ .description("Team name. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TEAM_MEMBERS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OUTPUT_BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name(ASANA_OUTPUT_BATCH_SIZE)
+ .displayName("Output Batch Size")
+ .description("The number of items batched together in a single Flow File. If set to 1 (default), then each item is"
+ + " transferred in a separate Flow File and each will have an asana.gid attribute, to help identifying"
+ + " the fetched item on the server side, if needed. If the batch size is greater than 1, then the"
+ + " specified amount of items are batched together in a single Flow File as a Json array, and the"
+ + " Flow Files won't have the asana.gid attribute.")
+ .defaultValue("1")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+
+ protected static final List<PropertyDescriptor> DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ PROP_ASANA_CLIENT_SERVICE,
+ PROP_DISTRIBUTED_CACHE_SERVICE,
+ PROP_ASANA_OBJECT_TYPE,
+ PROP_ASANA_PROJECT,
+ PROP_ASANA_SECTION,
+ PROP_ASANA_TEAM_NAME,
+ PROP_ASANA_TAG,
+ PROP_ASANA_OUTPUT_BATCH_SIZE
+ ));
+
+ protected static final Relationship REL_NEW = new Relationship.Builder()
+ .name(REL_NAME_NEW)
+ .description("Newly collected objects are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_UPDATED = new Relationship.Builder()
+ .name(REL_NAME_UPDATED)
+ .description("Objects that have already been collected earlier, but were updated since, are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_REMOVED = new Relationship.Builder()
+ .name(REL_NAME_REMOVED)
+ .description("Notification about deleted objects are routed to this relationship. "
+ + "Flow files will not have any payload. IDs of the resources no longer exist "
+ + "are carried by the asana.gid attribute of the generated FlowFiles.")
+ .build();
+
+ protected static final Set<Relationship> RELATIONSHIPS = Collections.unmodifiableSet(new HashSet<>(Arrays.asList(
+ REL_NEW,
+ REL_UPDATED,
+ REL_REMOVED
+ )));
+
+ private volatile AsanaObjectFetcher objectFetcher;
+ private volatile Integer batchSize;
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return RELATIONSHIPS;
+ }
+
+ @Override
+ public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return DESCRIPTORS;
+ }
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) throws InitializationException {
+ AsanaClientProviderService controllerService = context.getProperty(PROP_ASANA_CLIENT_SERVICE).asControllerService(AsanaClientProviderService.class);
+ AsanaClient client = controllerService.createClient();
+ batchSize = context.getProperty(PROP_ASANA_OUTPUT_BATCH_SIZE).asInteger();
+
+ try {
+ getLogger().debug("Initializing object fetcher...");
+ objectFetcher = createObjectFetcher(context, client);
+ } catch (Exception e) {
+ throw new InitializationException(e);
+ }
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ try {
+ Map<String, String> state = recoverState(context).orElse(Collections.emptyMap());
+ getLogger().debug("Attempting to load state: {}", state);
+ objectFetcher.loadState(state);
+ } catch (Exception e) {
+ getLogger().info("Failed to recover state. Falling back to clean start.");
+ objectFetcher.clearState();
+ }
+ getLogger().debug("Initial state: {}", objectFetcher.saveState());
+
+ Collection<FlowFile> newItems = new ArrayList<>();
+ Collection<FlowFile> updatedItems = new ArrayList<>();
+ Collection<FlowFile> removedItems = new ArrayList<>();
+ Map<AsanaObjectState, Collection<FlowFile>> flowFiles = new HashMap<>();
+ flowFiles.put(AsanaObjectState.NEW, newItems);
+ flowFiles.put(AsanaObjectState.UPDATED, updatedItems);
+ flowFiles.put(AsanaObjectState.REMOVED, removedItems);
+
+ List<AsanaObject> allObjects = new ArrayList<>();
+
+ AsanaObject nextObject;
+ while ((nextObject = objectFetcher.fetchNext()) != null) {
+ allObjects.add(nextObject);
Review Comment:
@turcsanyip Good point. But this requires a bigger change in the architecture, so I need some time to implement it, and to update all the affected tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r1035777053
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,393 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import static java.lang.String.format;
+import static java.util.Collections.singletonMap;
+import static java.util.stream.Collectors.groupingBy;
+import static java.util.stream.Collectors.joining;
+import static org.apache.commons.collections4.ListUtils.partition;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_EVENTS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_MEMBERS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_STATUS_ATTACHMENTS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_STATUS_UPDATES;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_STORIES;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_TASKS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_TASK_ATTACHMENTS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_TEAM_MEMBERS;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientProviderService;
+import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaObjectState;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+@TriggerSerially
+@PrimaryNodeOnly
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.")
+@Tags({"asana", "source", "ingest"})
+@CapabilityDescription("This processor collects data from Asana")
+public class GetAsanaObject extends AbstractProcessor {
+
+ protected static final String ASANA_GID = "asana.gid";
+ protected static final String ASANA_CLIENT_SERVICE = "asana-controller-service";
+ protected static final String DISTRIBUTED_CACHE_SERVICE = "distributed-cache-service";
+ protected static final String ASANA_OBJECT_TYPE = "asana-object-type";
+ protected static final String ASANA_PROJECT_NAME = "asana-project-name";
+ protected static final String ASANA_SECTION_NAME = "asana-section-name";
+ protected static final String ASANA_TAG_NAME = "asana-tag-name";
+ protected static final String ASANA_TEAM_NAME = "asana-team-name";
+ protected static final String ASANA_OUTPUT_BATCH_SIZE = "asana-output-batch-size";
+ protected static final String REL_NAME_NEW = "new";
+ protected static final String REL_NAME_UPDATED = "updated";
+ protected static final String REL_NAME_REMOVED = "removed";
+
+ protected static final PropertyDescriptor PROP_ASANA_CLIENT_SERVICE = new PropertyDescriptor.Builder()
+ .name(ASANA_CLIENT_SERVICE)
+ .displayName("Asana Client Service")
+ .description("Specify which controller service to use for accessing Asana.")
+ .required(true)
+ .identifiesControllerService(AsanaClientProviderService.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_DISTRIBUTED_CACHE_SERVICE = new Builder()
+ .name(DISTRIBUTED_CACHE_SERVICE)
+ .displayName("Distributed Cache Service")
+ .description("Cache service to store fetched item fingerprints. These, from the last successful query"
+ + " are stored, in order to enable incremental loading and change detection.")
+ .required(true)
+ .identifiesControllerService(DistributedMapCacheClient.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OBJECT_TYPE = new PropertyDescriptor.Builder()
+ .name(ASANA_OBJECT_TYPE)
+ .displayName("Object Type")
+ .description("Specify what kind of objects to be collected from Asana")
+ .required(true)
+ .allowableValues(AsanaObjectType.class)
+ .defaultValue(AV_COLLECT_TASKS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_PROJECT = new PropertyDescriptor.Builder()
+ .name(ASANA_PROJECT_NAME)
+ .displayName("Project Name")
+ .description("Fetch only objects in this project. Case sensitive.")
+ .required(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(
+ PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS.getValue(),
+ AV_COLLECT_TASK_ATTACHMENTS.getValue(),
+ AV_COLLECT_PROJECT_MEMBERS.getValue(),
+ AV_COLLECT_STORIES.getValue(),
+ AV_COLLECT_PROJECT_STATUS_UPDATES.getValue(),
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS.getValue(),
+ AV_COLLECT_PROJECT_EVENTS.getValue())
Review Comment:
Done in f4070b2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r1030309384
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-services/pom.xml:
##########
@@ -0,0 +1,64 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-asana-bundle</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-asana-services</artifactId>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-api</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-mock</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>com.squareup.okhttp3</groupId>
+ <artifactId>mockwebserver</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>commons-io</groupId>
+ <artifactId>commons-io</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>com.asana</groupId>
+ <artifactId>asana</artifactId>
+ </dependency>
Review Comment:
Done in 10f4c6b
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r1031406580
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-services-api-nar/src/main/resources/META-INF/NOTICE:
##########
@@ -0,0 +1,51 @@
+nifi-asana-nar
+Copyright 2015-2022 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+===========================================
+Apache Software License v2
+===========================================
+
+The following binary components are provided under the Apache Software License v2
+
+ (ASLv2) Apache Commons IO
+ The following NOTICE information applies:
+ Apache Commons IO
+ Copyright 2002-2017 The Apache Software Foundation
+
+ (ASLv2) Jackson JSON processor
+ The following NOTICE information applies:
+ # Jackson JSON processor
+
+ Jackson is a high-performance, Free/Open Source JSON processing library.
+ It was originally written by Tatu Saloranta (tatu.saloranta@iki.fi), and has
+ been in development since 2007.
+ It is currently developed by a community of developers, as well as supported
+ commercially by FasterXML.com.
+
+ ## Licensing
+
+ Jackson core and extension components may licensed under different licenses.
+ To find the details that apply to this artifact see the accompanying LICENSE file.
+ For more information, including possible other licensing options, contact
+ FasterXML.com (http://fasterxml.com).
+
+ ## Credits
+
+ A list of contributors may be found from CREDITS file, which is included
+ in some artifacts (usually source distributions); but is always available
+ from the source code management (SCM) system project uses.
Review Comment:
Done in b90eeaf
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r1030314403
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,457 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import static java.lang.String.format;
+import static java.util.Collections.singletonMap;
+import static java.util.stream.Collectors.groupingBy;
+import static java.util.stream.Collectors.joining;
+import static org.apache.commons.collections4.ListUtils.partition;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientProviderService;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaObjectState;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+@TriggerSerially
+@PrimaryNodeOnly
+@Stateful(scopes = {Scope.CLUSTER}, description = "Fingerprints of items in the last successful query are stored in order to enable incremental loading and change detection.")
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.")
+@Tags({"asana", "source", "ingest"})
+@CapabilityDescription("This processor collects data from Asana")
+public class GetAsanaObject extends AbstractProcessor {
+
+ protected static final String ASANA_GID = "asana.gid";
+ protected static final String AV_NAME_COLLECT_TASKS = "asana-collect-tasks";
+ protected static final String AV_NAME_COLLECT_TASK_ATTACHMENTS = "asana-collect-task-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECTS = "asana-collect-projects";
+ protected static final String AV_NAME_COLLECT_TAGS = "asana-collect-tags";
+ protected static final String AV_NAME_COLLECT_USERS = "asana-collect-users";
+ protected static final String AV_NAME_COLLECT_PROJECT_MEMBERS = "asana-collect-project-members";
+ protected static final String AV_NAME_COLLECT_TEAMS = "asana-collect-teams";
+ protected static final String AV_NAME_COLLECT_TEAM_MEMBERS = "asana-collect-team-members";
+ protected static final String AV_NAME_COLLECT_STORIES = "asana-collect-stories";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_UPDATES = "asana-collect-project-status-updates";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS = "asana-collect-project-status-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECT_EVENTS = "asana-collect-project-events";
+ protected static final String ASANA_CONTROLLER_SERVICE = "asana-controller-service";
+ protected static final String ASANA_OBJECT_TYPE = "asana-object-type";
+ protected static final String ASANA_PROJECT_NAME = "asana-project-name";
+ protected static final String ASANA_SECTION_NAME = "asana-section-name";
+ protected static final String ASANA_TAG_NAME = "asana-tag-name";
+ protected static final String ASANA_TEAM_NAME = "asana-team-name";
+ protected static final String ASANA_OUTPUT_BATCH_SIZE = "asana-output-batch-size";
+ protected static final String REL_NAME_NEW = "new";
+ protected static final String REL_NAME_UPDATED = "updated";
+ protected static final String REL_NAME_REMOVED = "removed";
+
+ protected static final AllowableValue AV_COLLECT_TASKS = new AllowableValue(
+ AV_NAME_COLLECT_TASKS,
+ "Tasks",
+ "Collect tasks matching to the specified conditions.");
+
+ protected static final AllowableValue AV_COLLECT_TASK_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_TASK_ATTACHMENTS,
+ "Task Attachments",
+ "Collect attached files of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECTS,
+ "Projects",
+ "Collect projects of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TAGS = new AllowableValue(
+ AV_NAME_COLLECT_TAGS,
+ "Tags",
+ "Collect tags of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_USERS = new AllowableValue(
+ AV_NAME_COLLECT_USERS,
+ "Users",
+ "Collect users assigned to the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_MEMBERS,
+ "Members of a Project",
+ "Collect users assigned to the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAMS = new AllowableValue(
+ AV_NAME_COLLECT_TEAMS,
+ "Teams",
+ "Collect teams of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAM_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_TEAM_MEMBERS,
+ "Team Members",
+ "Collect users assigned to the specified team."
+ );
+
+ protected static final AllowableValue AV_COLLECT_STORIES = new AllowableValue(
+ AV_NAME_COLLECT_STORIES,
+ "Stories of Tasks",
+ "Collect stories (comments) of of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_UPDATES = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_UPDATES,
+ "Status Updates of a Project",
+ "Collect status updates of the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ "Attachments of Status Updates",
+ "Collect attached files of project status updates."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_EVENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_EVENTS,
+ "Events of a Project",
+ "Collect various events happening on the specified project and on its' tasks."
+ );
+
+ protected static final PropertyDescriptor PROP_ASANA_CONTROLLER_SERVICE = new PropertyDescriptor.Builder()
+ .name(ASANA_CONTROLLER_SERVICE)
+ .displayName("Asana Controller Service")
+ .description("Specify which controller service to use for accessing Asana.")
+ .required(true)
+ .identifiesControllerService(AsanaClientProviderService.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OBJECT_TYPE = new PropertyDescriptor.Builder()
+ .name(ASANA_OBJECT_TYPE)
+ .displayName("Object Type")
+ .description("Specify what kind of objects to be collected from Asana")
+ .required(true)
+ .allowableValues(
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_PROJECTS,
+ AV_COLLECT_TAGS,
+ AV_COLLECT_USERS,
+ AV_COLLECT_PROJECT_MEMBERS,
+ AV_COLLECT_TEAMS,
+ AV_COLLECT_TEAM_MEMBERS,
+ AV_COLLECT_STORIES,
+ AV_COLLECT_PROJECT_STATUS_UPDATES,
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ AV_COLLECT_PROJECT_EVENTS)
+ .defaultValue(AV_COLLECT_TASKS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_PROJECT = new PropertyDescriptor.Builder()
+ .name(ASANA_PROJECT_NAME)
+ .displayName("Project Name")
+ .description("Fetch only objects in this project. Case sensitive.")
+ .required(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(
+ PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_PROJECT_MEMBERS,
+ AV_COLLECT_STORIES,
+ AV_COLLECT_PROJECT_STATUS_UPDATES,
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ AV_COLLECT_PROJECT_EVENTS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_SECTION = new PropertyDescriptor.Builder()
+ .name(ASANA_SECTION_NAME)
+ .displayName("Section Name")
+ .description("Fetch only objects in this section. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TASKS, AV_COLLECT_TASK_ATTACHMENTS, AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TAG = new PropertyDescriptor.Builder()
+ .name(ASANA_TAG_NAME)
+ .displayName("Tag")
+ .description("Fetch only objects having this tag. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TASKS, AV_COLLECT_TASK_ATTACHMENTS, AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TEAM_NAME = new PropertyDescriptor.Builder()
+ .name(ASANA_TEAM_NAME)
+ .displayName("Team")
+ .description("Team name. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TEAM_MEMBERS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OUTPUT_BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name(ASANA_OUTPUT_BATCH_SIZE)
+ .displayName("Output Batch Size")
+ .description("The number of items batched together in a single Flow File. If set to 1 (default), then each item is"
+ + " transferred in a separate Flow File and each will have an asana.gid attribute, to help identifying"
+ + " the fetched item on the server side, if needed. If the batch size is greater than 1, then the"
+ + " specified amount of items are batched together in a single Flow File as a Json array, and the"
+ + " Flow Files won't have the asana.gid attribute.")
+ .defaultValue("1")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+
+ protected static final List<PropertyDescriptor> DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ PROP_ASANA_CONTROLLER_SERVICE,
+ PROP_ASANA_OBJECT_TYPE,
+ PROP_ASANA_PROJECT,
+ PROP_ASANA_SECTION,
+ PROP_ASANA_TEAM_NAME,
+ PROP_ASANA_TAG,
+ PROP_ASANA_OUTPUT_BATCH_SIZE
+ ));
+
+ protected static final Relationship REL_NEW = new Relationship.Builder()
+ .name(REL_NAME_NEW)
+ .description("Newly collected objects are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_UPDATED = new Relationship.Builder()
+ .name(REL_NAME_UPDATED)
+ .description("Objects that have already been collected earlier, but were updated since, are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_REMOVED = new Relationship.Builder()
+ .name(REL_NAME_REMOVED)
+ .description("Notification about deleted objects are routed to this relationship. "
+ + "Flow files will not have any payload. IDs of the resources no longer exist "
+ + "are carried by the asana.gid attribute of the generated FlowFiles.")
+ .build();
+
+ protected static final Set<Relationship> RELATIONSHIPS = Collections.unmodifiableSet(new HashSet<>(Arrays.asList(
+ REL_NEW,
+ REL_UPDATED,
+ REL_REMOVED
+ )));
+
+ private static final Scope STATE_STORAGE_SCOPE = Scope.CLUSTER;
+
+ private volatile AsanaObjectFetcher objectFetcher;
+ private volatile Integer batchSize;
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return RELATIONSHIPS;
+ }
+
+ @Override
+ public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return DESCRIPTORS;
+ }
+
+ @OnScheduled
+ public synchronized void onScheduled(final ProcessContext context) throws InitializationException {
+ AsanaClientProviderService controllerService = context.getProperty(PROP_ASANA_CONTROLLER_SERVICE).asControllerService(AsanaClientProviderService.class);
+ AsanaClient client = controllerService.createClient();
+ batchSize = context.getProperty(PROP_ASANA_OUTPUT_BATCH_SIZE).asInteger();
+
+ try {
+ getLogger().debug("Initializing object fetcher...");
+ objectFetcher = createObjectFetcher(context, client);
+ } catch (Exception e) {
+ throw new InitializationException(e);
+ }
+ }
+
+ @Override
+ public synchronized void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
Review Comment:
Addressed in 0111adc
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,457 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import static java.lang.String.format;
+import static java.util.Collections.singletonMap;
+import static java.util.stream.Collectors.groupingBy;
+import static java.util.stream.Collectors.joining;
+import static org.apache.commons.collections4.ListUtils.partition;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientProviderService;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaObjectState;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+@TriggerSerially
+@PrimaryNodeOnly
+@Stateful(scopes = {Scope.CLUSTER}, description = "Fingerprints of items in the last successful query are stored in order to enable incremental loading and change detection.")
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.")
+@Tags({"asana", "source", "ingest"})
+@CapabilityDescription("This processor collects data from Asana")
+public class GetAsanaObject extends AbstractProcessor {
+
+ protected static final String ASANA_GID = "asana.gid";
+ protected static final String AV_NAME_COLLECT_TASKS = "asana-collect-tasks";
+ protected static final String AV_NAME_COLLECT_TASK_ATTACHMENTS = "asana-collect-task-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECTS = "asana-collect-projects";
+ protected static final String AV_NAME_COLLECT_TAGS = "asana-collect-tags";
+ protected static final String AV_NAME_COLLECT_USERS = "asana-collect-users";
+ protected static final String AV_NAME_COLLECT_PROJECT_MEMBERS = "asana-collect-project-members";
+ protected static final String AV_NAME_COLLECT_TEAMS = "asana-collect-teams";
+ protected static final String AV_NAME_COLLECT_TEAM_MEMBERS = "asana-collect-team-members";
+ protected static final String AV_NAME_COLLECT_STORIES = "asana-collect-stories";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_UPDATES = "asana-collect-project-status-updates";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS = "asana-collect-project-status-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECT_EVENTS = "asana-collect-project-events";
+ protected static final String ASANA_CONTROLLER_SERVICE = "asana-controller-service";
+ protected static final String ASANA_OBJECT_TYPE = "asana-object-type";
+ protected static final String ASANA_PROJECT_NAME = "asana-project-name";
+ protected static final String ASANA_SECTION_NAME = "asana-section-name";
+ protected static final String ASANA_TAG_NAME = "asana-tag-name";
+ protected static final String ASANA_TEAM_NAME = "asana-team-name";
+ protected static final String ASANA_OUTPUT_BATCH_SIZE = "asana-output-batch-size";
+ protected static final String REL_NAME_NEW = "new";
+ protected static final String REL_NAME_UPDATED = "updated";
+ protected static final String REL_NAME_REMOVED = "removed";
+
+ protected static final AllowableValue AV_COLLECT_TASKS = new AllowableValue(
+ AV_NAME_COLLECT_TASKS,
+ "Tasks",
+ "Collect tasks matching to the specified conditions.");
+
+ protected static final AllowableValue AV_COLLECT_TASK_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_TASK_ATTACHMENTS,
+ "Task Attachments",
+ "Collect attached files of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECTS,
+ "Projects",
+ "Collect projects of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TAGS = new AllowableValue(
+ AV_NAME_COLLECT_TAGS,
+ "Tags",
+ "Collect tags of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_USERS = new AllowableValue(
+ AV_NAME_COLLECT_USERS,
+ "Users",
+ "Collect users assigned to the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_MEMBERS,
+ "Members of a Project",
+ "Collect users assigned to the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAMS = new AllowableValue(
+ AV_NAME_COLLECT_TEAMS,
+ "Teams",
+ "Collect teams of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAM_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_TEAM_MEMBERS,
+ "Team Members",
+ "Collect users assigned to the specified team."
+ );
+
+ protected static final AllowableValue AV_COLLECT_STORIES = new AllowableValue(
+ AV_NAME_COLLECT_STORIES,
+ "Stories of Tasks",
+ "Collect stories (comments) of of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_UPDATES = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_UPDATES,
+ "Status Updates of a Project",
+ "Collect status updates of the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ "Attachments of Status Updates",
+ "Collect attached files of project status updates."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_EVENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_EVENTS,
+ "Events of a Project",
+ "Collect various events happening on the specified project and on its' tasks."
+ );
+
+ protected static final PropertyDescriptor PROP_ASANA_CONTROLLER_SERVICE = new PropertyDescriptor.Builder()
+ .name(ASANA_CONTROLLER_SERVICE)
+ .displayName("Asana Controller Service")
+ .description("Specify which controller service to use for accessing Asana.")
+ .required(true)
+ .identifiesControllerService(AsanaClientProviderService.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OBJECT_TYPE = new PropertyDescriptor.Builder()
+ .name(ASANA_OBJECT_TYPE)
+ .displayName("Object Type")
+ .description("Specify what kind of objects to be collected from Asana")
+ .required(true)
+ .allowableValues(
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_PROJECTS,
+ AV_COLLECT_TAGS,
+ AV_COLLECT_USERS,
+ AV_COLLECT_PROJECT_MEMBERS,
+ AV_COLLECT_TEAMS,
+ AV_COLLECT_TEAM_MEMBERS,
+ AV_COLLECT_STORIES,
+ AV_COLLECT_PROJECT_STATUS_UPDATES,
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ AV_COLLECT_PROJECT_EVENTS)
+ .defaultValue(AV_COLLECT_TASKS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_PROJECT = new PropertyDescriptor.Builder()
+ .name(ASANA_PROJECT_NAME)
+ .displayName("Project Name")
+ .description("Fetch only objects in this project. Case sensitive.")
+ .required(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(
+ PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_PROJECT_MEMBERS,
+ AV_COLLECT_STORIES,
+ AV_COLLECT_PROJECT_STATUS_UPDATES,
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ AV_COLLECT_PROJECT_EVENTS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_SECTION = new PropertyDescriptor.Builder()
+ .name(ASANA_SECTION_NAME)
+ .displayName("Section Name")
+ .description("Fetch only objects in this section. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TASKS, AV_COLLECT_TASK_ATTACHMENTS, AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TAG = new PropertyDescriptor.Builder()
+ .name(ASANA_TAG_NAME)
+ .displayName("Tag")
+ .description("Fetch only objects having this tag. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TASKS, AV_COLLECT_TASK_ATTACHMENTS, AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TEAM_NAME = new PropertyDescriptor.Builder()
+ .name(ASANA_TEAM_NAME)
+ .displayName("Team")
+ .description("Team name. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TEAM_MEMBERS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OUTPUT_BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name(ASANA_OUTPUT_BATCH_SIZE)
+ .displayName("Output Batch Size")
+ .description("The number of items batched together in a single Flow File. If set to 1 (default), then each item is"
+ + " transferred in a separate Flow File and each will have an asana.gid attribute, to help identifying"
+ + " the fetched item on the server side, if needed. If the batch size is greater than 1, then the"
+ + " specified amount of items are batched together in a single Flow File as a Json array, and the"
+ + " Flow Files won't have the asana.gid attribute.")
+ .defaultValue("1")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+
+ protected static final List<PropertyDescriptor> DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ PROP_ASANA_CONTROLLER_SERVICE,
+ PROP_ASANA_OBJECT_TYPE,
+ PROP_ASANA_PROJECT,
+ PROP_ASANA_SECTION,
+ PROP_ASANA_TEAM_NAME,
+ PROP_ASANA_TAG,
+ PROP_ASANA_OUTPUT_BATCH_SIZE
+ ));
+
+ protected static final Relationship REL_NEW = new Relationship.Builder()
+ .name(REL_NAME_NEW)
+ .description("Newly collected objects are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_UPDATED = new Relationship.Builder()
+ .name(REL_NAME_UPDATED)
+ .description("Objects that have already been collected earlier, but were updated since, are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_REMOVED = new Relationship.Builder()
+ .name(REL_NAME_REMOVED)
+ .description("Notification about deleted objects are routed to this relationship. "
+ + "Flow files will not have any payload. IDs of the resources no longer exist "
+ + "are carried by the asana.gid attribute of the generated FlowFiles.")
+ .build();
+
+ protected static final Set<Relationship> RELATIONSHIPS = Collections.unmodifiableSet(new HashSet<>(Arrays.asList(
+ REL_NEW,
+ REL_UPDATED,
+ REL_REMOVED
+ )));
+
+ private static final Scope STATE_STORAGE_SCOPE = Scope.CLUSTER;
+
+ private volatile AsanaObjectFetcher objectFetcher;
+ private volatile Integer batchSize;
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return RELATIONSHIPS;
+ }
+
+ @Override
+ public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return DESCRIPTORS;
+ }
+
+ @OnScheduled
+ public synchronized void onScheduled(final ProcessContext context) throws InitializationException {
Review Comment:
Addressed in 0111adc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r1017172747
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,429 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientProviderService;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@TriggerSerially
+@PrimaryNodeOnly
+@Stateful(scopes = {Scope.CLUSTER}, description = "Fingerprints of items in the last successful query are stored in order to enable incremental loading and change detection.")
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.")
+@Tags({"asana", "source", "ingest"})
+@CapabilityDescription("This processor collects data from Asana")
+public class GetAsanaObject extends AbstractProcessor {
+
+ protected static final String ASANA_GID = "asana.gid";
+ protected static final String AV_NAME_COLLECT_TASKS = "asana-collect-tasks";
+ protected static final String AV_NAME_COLLECT_TASK_ATTACHMENTS = "asana-collect-task-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECTS = "asana-collect-projects";
+ protected static final String AV_NAME_COLLECT_TAGS = "asana-collect-tags";
+ protected static final String AV_NAME_COLLECT_USERS = "asana-collect-users";
+ protected static final String AV_NAME_COLLECT_PROJECT_MEMBERS = "asana-collect-project-members";
+ protected static final String AV_NAME_COLLECT_TEAMS = "asana-collect-teams";
+ protected static final String AV_NAME_COLLECT_TEAM_MEMBERS = "asana-collect-team-members";
+ protected static final String AV_NAME_COLLECT_STORIES = "asana-collect-stories";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_UPDATES = "asana-collect-project-status-updates";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS = "asana-collect-project-status-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECT_EVENTS = "asana-collect-project-events";
+ protected static final String ASANA_CONTROLLER_SERVICE = "asana-controller-service";
+ protected static final String ASANA_OBJECT_TYPE = "asana-object-type";
+ protected static final String ASANA_PROJECT_NAME = "asana-project-name";
+ protected static final String ASANA_SECTION_NAME = "asana-section-name";
+ protected static final String ASANA_TAG_NAME = "asana-tag-name";
+ protected static final String ASANA_TEAM_NAME = "asana-team-name";
+ protected static final String ASANA_OUTPUT_BATCH_SIZE = "asana-output-batch-size";
+ protected static final String REL_NAME_NEW = "new";
+ protected static final String REL_NAME_UPDATED = "updated";
+ protected static final String REL_NAME_REMOVED = "removed";
+
+ protected static final AllowableValue AV_COLLECT_TASKS = new AllowableValue(
+ AV_NAME_COLLECT_TASKS,
+ "Tasks",
+ "Collect tasks matching to the specified conditions.");
+
+ protected static final AllowableValue AV_COLLECT_TASK_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_TASK_ATTACHMENTS,
+ "Task Attachments",
+ "Collect attached files of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECTS,
+ "Projects",
+ "Collect projects of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TAGS = new AllowableValue(
+ AV_NAME_COLLECT_TAGS,
+ "Tags",
+ "Collect tags of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_USERS = new AllowableValue(
+ AV_NAME_COLLECT_USERS,
+ "Users",
+ "Collect users assigned to the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_MEMBERS,
+ "Members of a Project",
+ "Collect users assigned to the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAMS = new AllowableValue(
+ AV_NAME_COLLECT_TEAMS,
+ "Teams",
+ "Collect teams of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAM_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_TEAM_MEMBERS,
+ "Team Members",
+ "Collect users assigned to the specified team."
+ );
+
+ protected static final AllowableValue AV_COLLECT_STORIES = new AllowableValue(
+ AV_NAME_COLLECT_STORIES,
+ "Stories of Tasks",
+ "Collect stories (comments) of of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_UPDATES = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_UPDATES,
+ "Status Updates of a Project",
+ "Collect status updates of the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ "Attachments of Status Updates",
+ "Collect attached files of project status updates."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_EVENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_EVENTS,
+ "Events of a Project",
+ "Collect various events happening on the specified project and on its' tasks."
+ );
+
+ protected static final PropertyDescriptor PROP_ASANA_CONTROLLER_SERVICE = new PropertyDescriptor.Builder()
+ .name(ASANA_CONTROLLER_SERVICE)
+ .displayName("Asana Controller Service")
+ .description("Specify which controller service to use for accessing Asana.")
+ .required(true)
+ .identifiesControllerService(AsanaClientProviderService.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OBJECT_TYPE = new PropertyDescriptor.Builder()
+ .name(ASANA_OBJECT_TYPE)
+ .displayName("Object Type")
+ .description("Specify what kind of objects to be collected from Asana")
+ .required(true)
+ .allowableValues(
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_PROJECTS,
+ AV_COLLECT_TAGS,
+ AV_COLLECT_USERS,
+ AV_COLLECT_PROJECT_MEMBERS,
+ AV_COLLECT_TEAMS,
+ AV_COLLECT_TEAM_MEMBERS,
+ AV_COLLECT_STORIES,
+ AV_COLLECT_PROJECT_STATUS_UPDATES,
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ AV_COLLECT_PROJECT_EVENTS)
+ .defaultValue(AV_COLLECT_TASKS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_PROJECT = new PropertyDescriptor.Builder()
+ .name(ASANA_PROJECT_NAME)
+ .displayName("Project Name")
+ .description("Fetch only objects in this project. Case sensitive.")
+ .required(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(
+ PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_PROJECT_MEMBERS,
+ AV_COLLECT_STORIES,
+ AV_COLLECT_PROJECT_STATUS_UPDATES,
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ AV_COLLECT_PROJECT_EVENTS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_SECTION = new PropertyDescriptor.Builder()
+ .name(ASANA_SECTION_NAME)
+ .displayName("Section Name")
+ .description("Fetch only objects in this section. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TASKS, AV_COLLECT_TASK_ATTACHMENTS, AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TAG = new PropertyDescriptor.Builder()
+ .name(ASANA_TAG_NAME)
+ .displayName("Tag")
+ .description("Fetch only objects having this tag. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TASKS, AV_COLLECT_TASK_ATTACHMENTS, AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TEAM_NAME = new PropertyDescriptor.Builder()
+ .name(ASANA_TEAM_NAME)
+ .displayName("Team")
+ .description("Team name. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TEAM_MEMBERS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OUTPUT_BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name(ASANA_OUTPUT_BATCH_SIZE)
+ .displayName("Output Batch Size")
+ .description("The number of output FlowFiles to queue before committing the process session. When set to zero, the session will be committed when all result set rows "
+ + "have been processed and the output FlowFiles are ready for transfer to the downstream relationship. For large result sets, this can cause a large burst of FlowFiles "
+ + "to be transferred at the end of processor execution. If this property is set, then when the specified number of FlowFiles are ready for transfer, then the session will "
+ + "be committed, thus releasing the FlowFiles to the downstream relationship.")
+ .defaultValue("0")
+ .required(true)
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+
+ protected static final List<PropertyDescriptor> DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ PROP_ASANA_CONTROLLER_SERVICE,
+ PROP_ASANA_OBJECT_TYPE,
+ PROP_ASANA_PROJECT,
+ PROP_ASANA_SECTION,
+ PROP_ASANA_TEAM_NAME,
+ PROP_ASANA_TAG,
+ PROP_ASANA_OUTPUT_BATCH_SIZE
+ ));
+
+ protected static final Relationship REL_NEW = new Relationship.Builder()
+ .name(REL_NAME_NEW)
+ .description("Newly collected objects are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_UPDATED = new Relationship.Builder()
+ .name(REL_NAME_UPDATED)
+ .description("Objects that have already been collected earlier, but were updated since, are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_REMOVED = new Relationship.Builder()
+ .name(REL_NAME_REMOVED)
+ .description("Notification about deleted objects are routed to this relationship. "
+ + "Flow files will not have any payload. IDs of the resources no longer exist "
+ + "are carried by the asana.gid attribute of the generated FlowFiles.")
+ .build();
+
+ protected static final Set<Relationship> RELATIONSHIPS = Collections.unmodifiableSet(new HashSet<>(Arrays.asList(
+ REL_NEW,
+ REL_UPDATED,
+ REL_REMOVED
+ )));
+
+ private static final Scope STATE_STORAGE_SCOPE = Scope.CLUSTER;
+
+ private volatile AsanaObjectFetcher objectFetcher;
+ private volatile Integer batchSize;
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return RELATIONSHIPS;
+ }
+
+ @Override
+ public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return DESCRIPTORS;
+ }
+
+ @OnScheduled
+ public synchronized void onScheduled(final ProcessContext context) throws InitializationException {
+ AsanaClientProviderService controllerService = context.getProperty(PROP_ASANA_CONTROLLER_SERVICE).asControllerService(AsanaClientProviderService.class);
+ AsanaClient client = controllerService.createClient();
+ batchSize = context.getProperty(PROP_ASANA_OUTPUT_BATCH_SIZE).asInteger();
+
+ try {
+ getLogger().debug("Initializing object fetcher...");
+ objectFetcher = createObjectFetcher(context, client);
+ } catch (Exception e) {
+ throw new InitializationException(e);
+ }
+ }
+
+ @Override
+ public synchronized void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ try {
+ Map<String, String> state = recoverState(context).orElse(Collections.emptyMap());
+ getLogger().debug("Attempting to load state: {}", state);
+ objectFetcher.loadState(state);
+ } catch (Exception e) {
+ getLogger().info("Failed to recover state. Falling back to clean start.");
+ objectFetcher.clearState();
+ }
+ getLogger().debug("Initial state: {}", objectFetcher.saveState());
+
+ Collection<FlowFile> newItems = new ArrayList<>();
+ Collection<FlowFile> updatedItems = new ArrayList<>();
+ Collection<FlowFile> removedItems = new ArrayList<>();
+
+ AsanaObject nextObject;
+ while ((batchSize == 0 || (batchSize > (newItems.size() + updatedItems.size() + removedItems.size()))) && (nextObject = objectFetcher.fetchNext()) != null) {
+ final Map<String, String> attributes = new HashMap<>(2);
+ attributes.put(CoreAttributes.MIME_TYPE.key(), ContentType.APPLICATION_JSON.getMimeType());
+ attributes.put(ASANA_GID, nextObject.getGid());
+ FlowFile flowFile = createFlowFileWithStringPayload(session, nextObject.getContent());
Review Comment:
Addressed in [7d439f7](https://github.com/apache/nifi/pull/6504/commits/7d439f7f32d40768ddab998948563ef3f29dc012).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r1042719333
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,405 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import static java.lang.String.format;
+import static java.lang.String.join;
+import static java.util.Collections.singletonMap;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_EVENTS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_MEMBERS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_STATUS_ATTACHMENTS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_STATUS_UPDATES;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_STORIES;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_TASKS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_TASK_ATTACHMENTS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_TEAM_MEMBERS;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientProviderService;
+import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaObjectState;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+@TriggerSerially
+@PrimaryNodeOnly
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.")
+@Tags({"asana", "source", "ingest"})
+@CapabilityDescription("This processor collects data from Asana")
+public class GetAsanaObject extends AbstractProcessor {
+
+ protected static final String ASANA_GID = "asana.gid";
+ protected static final String ASANA_CLIENT_SERVICE = "asana-controller-service";
+ protected static final String DISTRIBUTED_CACHE_SERVICE = "distributed-cache-service";
+ protected static final String ASANA_OBJECT_TYPE = "asana-object-type";
+ protected static final String ASANA_PROJECT_NAME = "asana-project-name";
+ protected static final String ASANA_SECTION_NAME = "asana-section-name";
+ protected static final String ASANA_TAG_NAME = "asana-tag-name";
+ protected static final String ASANA_TEAM_NAME = "asana-team-name";
+ protected static final String ASANA_OUTPUT_BATCH_SIZE = "asana-output-batch-size";
+ protected static final String REL_NAME_NEW = "new";
+ protected static final String REL_NAME_UPDATED = "updated";
+ protected static final String REL_NAME_REMOVED = "removed";
+
+ protected static final PropertyDescriptor PROP_ASANA_CLIENT_SERVICE = new PropertyDescriptor.Builder()
+ .name(ASANA_CLIENT_SERVICE)
+ .displayName("Asana Client Service")
+ .description("Specify which controller service to use for accessing Asana.")
+ .required(true)
+ .identifiesControllerService(AsanaClientProviderService.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_DISTRIBUTED_CACHE_SERVICE = new Builder()
+ .name(DISTRIBUTED_CACHE_SERVICE)
+ .displayName("Distributed Cache Service")
+ .description("Cache service to store fetched item fingerprints. These, from the last successful query"
+ + " are stored, in order to enable incremental loading and change detection.")
+ .required(true)
+ .identifiesControllerService(DistributedMapCacheClient.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OBJECT_TYPE = new PropertyDescriptor.Builder()
+ .name(ASANA_OBJECT_TYPE)
+ .displayName("Object Type")
+ .description("Specify what kind of objects to be collected from Asana")
+ .required(true)
+ .allowableValues(AsanaObjectType.class)
+ .defaultValue(AV_COLLECT_TASKS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_PROJECT = new PropertyDescriptor.Builder()
+ .name(ASANA_PROJECT_NAME)
+ .displayName("Project Name")
+ .description("Fetch only objects in this project. Case sensitive.")
+ .required(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(
+ PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_PROJECT_MEMBERS,
+ AV_COLLECT_STORIES,
+ AV_COLLECT_PROJECT_STATUS_UPDATES,
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ AV_COLLECT_PROJECT_EVENTS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_SECTION = new PropertyDescriptor.Builder()
+ .name(ASANA_SECTION_NAME)
+ .displayName("Section Name")
+ .description("Fetch only objects in this section. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TAG = new PropertyDescriptor.Builder()
+ .name(ASANA_TAG_NAME)
+ .displayName("Tag")
+ .description("Fetch only objects having this tag. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TEAM_NAME = new PropertyDescriptor.Builder()
+ .name(ASANA_TEAM_NAME)
+ .displayName("Team")
+ .description("Team name. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TEAM_MEMBERS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OUTPUT_BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name(ASANA_OUTPUT_BATCH_SIZE)
+ .displayName("Output Batch Size")
+ .description("The number of items batched together in a single Flow File. If set to 1 (default), then each item is"
+ + " transferred in a separate Flow File and each will have an asana.gid attribute, to help identifying"
+ + " the fetched item on the server side, if needed. If the batch size is greater than 1, then the"
+ + " specified amount of items are batched together in a single Flow File as a Json array, and the"
+ + " Flow Files won't have the asana.gid attribute.")
+ .defaultValue("1")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+
+ protected static final List<PropertyDescriptor> DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ PROP_ASANA_CLIENT_SERVICE,
+ PROP_DISTRIBUTED_CACHE_SERVICE,
+ PROP_ASANA_OBJECT_TYPE,
+ PROP_ASANA_PROJECT,
+ PROP_ASANA_SECTION,
+ PROP_ASANA_TEAM_NAME,
+ PROP_ASANA_TAG,
+ PROP_ASANA_OUTPUT_BATCH_SIZE
+ ));
+
+ protected static final Relationship REL_NEW = new Relationship.Builder()
+ .name(REL_NAME_NEW)
+ .description("Newly collected objects are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_UPDATED = new Relationship.Builder()
+ .name(REL_NAME_UPDATED)
+ .description("Objects that have already been collected earlier, but were updated since, are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_REMOVED = new Relationship.Builder()
+ .name(REL_NAME_REMOVED)
+ .description("Notification about deleted objects are routed to this relationship. "
+ + "Flow files will not have any payload. IDs of the resources no longer exist "
+ + "are carried by the asana.gid attribute of the generated FlowFiles.")
+ .build();
+
+ protected static final Set<Relationship> RELATIONSHIPS = Collections.unmodifiableSet(new HashSet<>(Arrays.asList(
+ REL_NEW,
+ REL_UPDATED,
+ REL_REMOVED
+ )));
+ protected static final GenericObjectSerDe<String> STATE_MAP_KEY_SERIALIZER = new GenericObjectSerDe<>();
+ protected static final GenericObjectSerDe<Map<String, String>> STATE_MAP_VALUE_SERIALIZER = new GenericObjectSerDe<>();
+
+ private volatile AsanaObjectFetcher objectFetcher;
+ private volatile Integer batchSize;
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return RELATIONSHIPS;
+ }
+
+ @Override
+ public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return DESCRIPTORS;
+ }
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) throws InitializationException {
+ AsanaClientProviderService controllerService = context.getProperty(PROP_ASANA_CLIENT_SERVICE).asControllerService(AsanaClientProviderService.class);
+ AsanaClient client = controllerService.createClient();
+ batchSize = context.getProperty(PROP_ASANA_OUTPUT_BATCH_SIZE).asInteger();
+
+ try {
+ getLogger().debug("Initializing object fetcher...");
+ objectFetcher = createObjectFetcher(context, client);
+ } catch (Exception e) {
+ throw new InitializationException(e);
+ }
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ try {
+ Map<String, String> state = recoverState(context).orElse(Collections.emptyMap());
+ getLogger().debug("Attempting to load state: {}", state);
+ objectFetcher.loadState(state);
+ } catch (Exception e) {
+ getLogger().info("Failed to recover state. Falling back to clean start.");
+ objectFetcher.clearState();
+ }
Review Comment:
Done in 93ac273
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r995754281
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientServiceApi;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@TriggerSerially
+@Stateful(scopes = {Scope.LOCAL}, description = "Fingerprints of items in the last successful query are stored in order to enable incremental loading and change detection.")
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.")
+@Tags({"asana", "source", "connector", "ingest"})
+@CapabilityDescription("This processor collects data from Asana")
+public class GetAsanaObject extends AbstractProcessor {
+
+ protected static final String ASANA_GID = "asana.gid";
+ protected static final String AV_NAME_COLLECT_TASKS = "asana-collect-tasks";
+ protected static final String AV_NAME_COLLECT_TASK_ATTACHMENTS = "asana-collect-task-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECTS = "asana-collect-projects";
+ protected static final String AV_NAME_COLLECT_TAGS = "asana-collect-tags";
+ protected static final String AV_NAME_COLLECT_USERS = "asana-collect-users";
+ protected static final String AV_NAME_COLLECT_PROJECT_MEMBERS = "asana-collect-project-members";
+ protected static final String AV_NAME_COLLECT_TEAMS = "asana-collect-teams";
+ protected static final String AV_NAME_COLLECT_TEAM_MEMBERS = "asana-collect-team-members";
+ protected static final String AV_NAME_COLLECT_STORIES = "asana-collect-stories";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_UPDATES = "asana-collect-project-status-updates";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS = "asana-collect-project-status-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECT_EVENTS = "asana-collect-project-events";
+ protected static final String ASANA_CONTROLLER_SERVICE = "asana-controller-service";
+ protected static final String ASANA_OBJECT_TYPE = "asana-object-type";
+ protected static final String ASANA_PROJECT_NAME = "asana-project-name";
+ protected static final String ASANA_SECTION_NAME = "asana-section-name";
+ protected static final String ASANA_TAG_NAME = "asana-tag-name";
+ protected static final String ASANA_TEAM_NAME = "asana-team-name";
+ protected static final String ASANA_OUTPUT_BATCH_SIZE = "asana-output-batch-size";
+ protected static final String REL_NAME_NEW = "new";
+ protected static final String REL_NAME_UPDATED = "updated";
+ protected static final String REL_NAME_REMOVED = "removed";
+
+ protected static final AllowableValue AV_COLLECT_TASKS = new AllowableValue(
+ AV_NAME_COLLECT_TASKS,
+ "Tasks",
+ "Collect tasks matching to the specified conditions.");
+
+ protected static final AllowableValue AV_COLLECT_TASK_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_TASK_ATTACHMENTS,
+ "Task attachments",
+ "Collect attached files of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECTS,
+ "Projects",
+ "Collect projects of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TAGS = new AllowableValue(
+ AV_NAME_COLLECT_TAGS,
+ "Tags",
+ "Collect tags of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_USERS = new AllowableValue(
+ AV_NAME_COLLECT_USERS,
+ "Users",
+ "Collect users assigned to the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_MEMBERS,
+ "Members of a project",
+ "Collect users assigned to the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAMS = new AllowableValue(
+ AV_NAME_COLLECT_TEAMS,
+ "Teams",
+ "Collect teams of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAM_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_TEAM_MEMBERS,
+ "Team members",
+ "Collect users assigned to the specified team."
+ );
+
+ protected static final AllowableValue AV_COLLECT_STORIES = new AllowableValue(
+ AV_NAME_COLLECT_STORIES,
+ "Stories of tasks",
+ "Collect stories (comments) of of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_UPDATES = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_UPDATES,
+ "Status updates of a project",
+ "Collect status updates of the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ "Attachments of status updates",
+ "Collect attached files of project status updates."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_EVENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_EVENTS,
+ "Events of a project",
+ "Collect various events happening on the specified project and on its' tasks."
+ );
+
+ protected static final PropertyDescriptor PROP_ASANA_CONTROLLER_SERVICE = new PropertyDescriptor.Builder()
+ .name(ASANA_CONTROLLER_SERVICE)
+ .displayName("Controller service")
+ .description("Specify which controller service to use for accessing Asana.")
+ .required(true)
+ .identifiesControllerService(AsanaClientServiceApi.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OBJECT_TYPE = new PropertyDescriptor.Builder()
+ .name(ASANA_OBJECT_TYPE)
+ .displayName("Object type to be collected")
+ .description("Specify what kind of objects to be collected from Asana")
+ .required(true)
+ .allowableValues(
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_PROJECTS,
+ AV_COLLECT_TAGS,
+ AV_COLLECT_USERS,
+ AV_COLLECT_PROJECT_MEMBERS,
+ AV_COLLECT_TEAMS,
+ AV_COLLECT_TEAM_MEMBERS,
+ AV_COLLECT_STORIES,
+ AV_COLLECT_PROJECT_STATUS_UPDATES,
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ AV_COLLECT_PROJECT_EVENTS)
+ .defaultValue(AV_COLLECT_TASKS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_PROJECT = new PropertyDescriptor.Builder()
+ .name(ASANA_PROJECT_NAME)
+ .displayName("Project name")
Review Comment:
Done in [dc2a050](https://github.com/apache/nifi/pull/6504/commits/dc2a050548a389d89181670371748af741eefd61)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
exceptionfactory commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r1012402834
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,429 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientProviderService;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@TriggerSerially
+@PrimaryNodeOnly
+@Stateful(scopes = {Scope.CLUSTER}, description = "Fingerprints of items in the last successful query are stored in order to enable incremental loading and change detection.")
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.")
+@Tags({"asana", "source", "ingest"})
+@CapabilityDescription("This processor collects data from Asana")
+public class GetAsanaObject extends AbstractProcessor {
+
+ protected static final String ASANA_GID = "asana.gid";
+ protected static final String AV_NAME_COLLECT_TASKS = "asana-collect-tasks";
+ protected static final String AV_NAME_COLLECT_TASK_ATTACHMENTS = "asana-collect-task-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECTS = "asana-collect-projects";
+ protected static final String AV_NAME_COLLECT_TAGS = "asana-collect-tags";
+ protected static final String AV_NAME_COLLECT_USERS = "asana-collect-users";
+ protected static final String AV_NAME_COLLECT_PROJECT_MEMBERS = "asana-collect-project-members";
+ protected static final String AV_NAME_COLLECT_TEAMS = "asana-collect-teams";
+ protected static final String AV_NAME_COLLECT_TEAM_MEMBERS = "asana-collect-team-members";
+ protected static final String AV_NAME_COLLECT_STORIES = "asana-collect-stories";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_UPDATES = "asana-collect-project-status-updates";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS = "asana-collect-project-status-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECT_EVENTS = "asana-collect-project-events";
+ protected static final String ASANA_CONTROLLER_SERVICE = "asana-controller-service";
+ protected static final String ASANA_OBJECT_TYPE = "asana-object-type";
+ protected static final String ASANA_PROJECT_NAME = "asana-project-name";
+ protected static final String ASANA_SECTION_NAME = "asana-section-name";
+ protected static final String ASANA_TAG_NAME = "asana-tag-name";
+ protected static final String ASANA_TEAM_NAME = "asana-team-name";
+ protected static final String ASANA_OUTPUT_BATCH_SIZE = "asana-output-batch-size";
+ protected static final String REL_NAME_NEW = "new";
+ protected static final String REL_NAME_UPDATED = "updated";
+ protected static final String REL_NAME_REMOVED = "removed";
+
+ protected static final AllowableValue AV_COLLECT_TASKS = new AllowableValue(
+ AV_NAME_COLLECT_TASKS,
+ "Tasks",
+ "Collect tasks matching to the specified conditions.");
+
+ protected static final AllowableValue AV_COLLECT_TASK_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_TASK_ATTACHMENTS,
+ "Task Attachments",
+ "Collect attached files of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECTS,
+ "Projects",
+ "Collect projects of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TAGS = new AllowableValue(
+ AV_NAME_COLLECT_TAGS,
+ "Tags",
+ "Collect tags of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_USERS = new AllowableValue(
+ AV_NAME_COLLECT_USERS,
+ "Users",
+ "Collect users assigned to the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_MEMBERS,
+ "Members of a Project",
+ "Collect users assigned to the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAMS = new AllowableValue(
+ AV_NAME_COLLECT_TEAMS,
+ "Teams",
+ "Collect teams of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAM_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_TEAM_MEMBERS,
+ "Team Members",
+ "Collect users assigned to the specified team."
+ );
+
+ protected static final AllowableValue AV_COLLECT_STORIES = new AllowableValue(
+ AV_NAME_COLLECT_STORIES,
+ "Stories of Tasks",
+ "Collect stories (comments) of of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_UPDATES = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_UPDATES,
+ "Status Updates of a Project",
+ "Collect status updates of the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ "Attachments of Status Updates",
+ "Collect attached files of project status updates."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_EVENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_EVENTS,
+ "Events of a Project",
+ "Collect various events happening on the specified project and on its' tasks."
+ );
+
+ protected static final PropertyDescriptor PROP_ASANA_CONTROLLER_SERVICE = new PropertyDescriptor.Builder()
+ .name(ASANA_CONTROLLER_SERVICE)
+ .displayName("Asana Controller Service")
+ .description("Specify which controller service to use for accessing Asana.")
+ .required(true)
+ .identifiesControllerService(AsanaClientProviderService.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OBJECT_TYPE = new PropertyDescriptor.Builder()
+ .name(ASANA_OBJECT_TYPE)
+ .displayName("Object Type")
+ .description("Specify what kind of objects to be collected from Asana")
+ .required(true)
+ .allowableValues(
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_PROJECTS,
+ AV_COLLECT_TAGS,
+ AV_COLLECT_USERS,
+ AV_COLLECT_PROJECT_MEMBERS,
+ AV_COLLECT_TEAMS,
+ AV_COLLECT_TEAM_MEMBERS,
+ AV_COLLECT_STORIES,
+ AV_COLLECT_PROJECT_STATUS_UPDATES,
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ AV_COLLECT_PROJECT_EVENTS)
+ .defaultValue(AV_COLLECT_TASKS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_PROJECT = new PropertyDescriptor.Builder()
+ .name(ASANA_PROJECT_NAME)
+ .displayName("Project Name")
+ .description("Fetch only objects in this project. Case sensitive.")
+ .required(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(
+ PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_PROJECT_MEMBERS,
+ AV_COLLECT_STORIES,
+ AV_COLLECT_PROJECT_STATUS_UPDATES,
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ AV_COLLECT_PROJECT_EVENTS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_SECTION = new PropertyDescriptor.Builder()
+ .name(ASANA_SECTION_NAME)
+ .displayName("Section Name")
+ .description("Fetch only objects in this section. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TASKS, AV_COLLECT_TASK_ATTACHMENTS, AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TAG = new PropertyDescriptor.Builder()
+ .name(ASANA_TAG_NAME)
+ .displayName("Tag")
+ .description("Fetch only objects having this tag. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TASKS, AV_COLLECT_TASK_ATTACHMENTS, AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TEAM_NAME = new PropertyDescriptor.Builder()
+ .name(ASANA_TEAM_NAME)
+ .displayName("Team")
+ .description("Team name. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TEAM_MEMBERS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OUTPUT_BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name(ASANA_OUTPUT_BATCH_SIZE)
+ .displayName("Output Batch Size")
+ .description("The number of output FlowFiles to queue before committing the process session. When set to zero, the session will be committed when all result set rows "
+ + "have been processed and the output FlowFiles are ready for transfer to the downstream relationship. For large result sets, this can cause a large burst of FlowFiles "
+ + "to be transferred at the end of processor execution. If this property is set, then when the specified number of FlowFiles are ready for transfer, then the session will "
+ + "be committed, thus releasing the FlowFiles to the downstream relationship.")
+ .defaultValue("0")
+ .required(true)
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+
+ protected static final List<PropertyDescriptor> DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ PROP_ASANA_CONTROLLER_SERVICE,
+ PROP_ASANA_OBJECT_TYPE,
+ PROP_ASANA_PROJECT,
+ PROP_ASANA_SECTION,
+ PROP_ASANA_TEAM_NAME,
+ PROP_ASANA_TAG,
+ PROP_ASANA_OUTPUT_BATCH_SIZE
+ ));
+
+ protected static final Relationship REL_NEW = new Relationship.Builder()
+ .name(REL_NAME_NEW)
+ .description("Newly collected objects are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_UPDATED = new Relationship.Builder()
+ .name(REL_NAME_UPDATED)
+ .description("Objects that have already been collected earlier, but were updated since, are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_REMOVED = new Relationship.Builder()
+ .name(REL_NAME_REMOVED)
+ .description("Notification about deleted objects are routed to this relationship. "
+ + "Flow files will not have any payload. IDs of the resources no longer exist "
+ + "are carried by the asana.gid attribute of the generated FlowFiles.")
+ .build();
+
+ protected static final Set<Relationship> RELATIONSHIPS = Collections.unmodifiableSet(new HashSet<>(Arrays.asList(
+ REL_NEW,
+ REL_UPDATED,
+ REL_REMOVED
+ )));
+
+ private static final Scope STATE_STORAGE_SCOPE = Scope.CLUSTER;
+
+ private volatile AsanaObjectFetcher objectFetcher;
+ private volatile Integer batchSize;
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return RELATIONSHIPS;
+ }
+
+ @Override
+ public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return DESCRIPTORS;
+ }
+
+ @OnScheduled
+ public synchronized void onScheduled(final ProcessContext context) throws InitializationException {
+ AsanaClientProviderService controllerService = context.getProperty(PROP_ASANA_CONTROLLER_SERVICE).asControllerService(AsanaClientProviderService.class);
+ AsanaClient client = controllerService.createClient();
+ batchSize = context.getProperty(PROP_ASANA_OUTPUT_BATCH_SIZE).asInteger();
+
+ try {
+ getLogger().debug("Initializing object fetcher...");
+ objectFetcher = createObjectFetcher(context, client);
+ } catch (Exception e) {
+ throw new InitializationException(e);
+ }
+ }
+
+ @Override
+ public synchronized void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ try {
+ Map<String, String> state = recoverState(context).orElse(Collections.emptyMap());
+ getLogger().debug("Attempting to load state: {}", state);
+ objectFetcher.loadState(state);
+ } catch (Exception e) {
+ getLogger().info("Failed to recover state. Falling back to clean start.");
+ objectFetcher.clearState();
+ }
+ getLogger().debug("Initial state: {}", objectFetcher.saveState());
+
+ Collection<FlowFile> newItems = new ArrayList<>();
+ Collection<FlowFile> updatedItems = new ArrayList<>();
+ Collection<FlowFile> removedItems = new ArrayList<>();
+
+ AsanaObject nextObject;
+ while ((batchSize == 0 || (batchSize > (newItems.size() + updatedItems.size() + removedItems.size()))) && (nextObject = objectFetcher.fetchNext()) != null) {
+ final Map<String, String> attributes = new HashMap<>(2);
+ attributes.put(CoreAttributes.MIME_TYPE.key(), ContentType.APPLICATION_JSON.getMimeType());
+ attributes.put(ASANA_GID, nextObject.getGid());
+ FlowFile flowFile = createFlowFileWithStringPayload(session, nextObject.getContent());
Review Comment:
Creating a FlowFile per item is not very efficient and will result in poor performance for large volumes of data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r995754764
##########
nifi-nar-bundles/nifi-asana-bundle/pom.xml:
##########
@@ -0,0 +1,65 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-nar-bundles</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-asana-bundle</artifactId>
+ <packaging>pom</packaging>
+
+ <modules>
+ <module>nifi-asana-processors</module>
+ <module>nifi-asana-services</module>
+ <module>nifi-asana-services-api</module>
+ <module>nifi-asana-nar</module>
Review Comment:
Done in [dc2a050](https://github.com/apache/nifi/pull/6504/commits/dc2a050548a389d89181670371748af741eefd61)
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientServiceApi;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@TriggerSerially
Review Comment:
Done in [dc2a050](https://github.com/apache/nifi/pull/6504/commits/dc2a050548a389d89181670371748af741eefd61)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r1030319428
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/pom.xml:
##########
@@ -0,0 +1,69 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-asana-bundle</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-asana-processors</artifactId>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-api</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-mock</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>commons-io</groupId>
+ <artifactId>commons-io</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.commons</groupId>
+ <artifactId>commons-collections4</artifactId>
+ <version>4.4</version>
+ </dependency>
+ <dependency>
+ <groupId>com.asana</groupId>
+ <artifactId>asana</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-utils</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ <scope>compile</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-asana-services-api</artifactId>
+ </dependency>
Review Comment:
First two addressed in df3c96e & last one addressed in 10f4c6b + 960adba
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r1031401385
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-services/pom.xml:
##########
@@ -0,0 +1,64 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-asana-bundle</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-asana-services</artifactId>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-api</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-mock</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>com.squareup.okhttp3</groupId>
+ <artifactId>mockwebserver</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>commons-io</groupId>
+ <artifactId>commons-io</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>com.asana</groupId>
+ <artifactId>asana</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-utils</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ <scope>compile</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-asana-services-api</artifactId>
+ </dependency>
Review Comment:
Done in 4a5b807
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r1030593995
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,457 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import static java.lang.String.format;
+import static java.util.Collections.singletonMap;
+import static java.util.stream.Collectors.groupingBy;
+import static java.util.stream.Collectors.joining;
+import static org.apache.commons.collections4.ListUtils.partition;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientProviderService;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaObjectState;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+@TriggerSerially
+@PrimaryNodeOnly
+@Stateful(scopes = {Scope.CLUSTER}, description = "Fingerprints of items in the last successful query are stored in order to enable incremental loading and change detection.")
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.")
+@Tags({"asana", "source", "ingest"})
+@CapabilityDescription("This processor collects data from Asana")
+public class GetAsanaObject extends AbstractProcessor {
+
+ protected static final String ASANA_GID = "asana.gid";
+ protected static final String AV_NAME_COLLECT_TASKS = "asana-collect-tasks";
+ protected static final String AV_NAME_COLLECT_TASK_ATTACHMENTS = "asana-collect-task-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECTS = "asana-collect-projects";
+ protected static final String AV_NAME_COLLECT_TAGS = "asana-collect-tags";
+ protected static final String AV_NAME_COLLECT_USERS = "asana-collect-users";
+ protected static final String AV_NAME_COLLECT_PROJECT_MEMBERS = "asana-collect-project-members";
+ protected static final String AV_NAME_COLLECT_TEAMS = "asana-collect-teams";
+ protected static final String AV_NAME_COLLECT_TEAM_MEMBERS = "asana-collect-team-members";
+ protected static final String AV_NAME_COLLECT_STORIES = "asana-collect-stories";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_UPDATES = "asana-collect-project-status-updates";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS = "asana-collect-project-status-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECT_EVENTS = "asana-collect-project-events";
Review Comment:
Done in e46ac29
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r1042718987
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/resources/docs/org.apache.nifi.processors.asana.GetAsanaObject/additionalDetails.html:
##########
@@ -0,0 +1,100 @@
+<!DOCTYPE html>
+<html lang="en" xmlns="http://www.w3.org/1999/html">
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+<head>
+ <meta charset="utf-8"/>
+ <title>GetAsanaObject</title>
+ <link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"/>
+ <style>
+ h2 {margin-top: 4em}
+ h3 {margin-top: 3em}
+ td {text-align: left}
+ </style>
+</head>
+
+<body>
+
+<h1>GetAsanaObject</h1>
+
+<h3>Description</h3>
+<p>
+ This processor collects various objects (eg. tasks, comments, etc...) from Asana via the specified
+ <code>AsanaClientService</code>. When the processor started for the first time with a given configuration
+ it collects each of the objects matching the user specified criteria, and emits <code>FlowFile</code>s
+ of each on the <code>NEW</code> relationship. Then, it polls Asana in the frequency of the configured <em>Yield</em>
+ duration, and detects changes by comparing the object fingerprints. When there are updates, it emits them through
Review Comment:
Done in 7872fb2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on PR #6504:
URL: https://github.com/apache/nifi/pull/6504#issuecomment-1341645043
@turcsanyip I have attempted to address each of the discovered issues. Please check again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r1014225037
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-services/src/main/java/org/apache/nifi/controller/asana/StandardAsanaClient.java:
##########
@@ -0,0 +1,303 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.controller.asana;
+
+import com.asana.Client;
+import com.asana.errors.InvalidTokenError;
+import com.asana.models.Attachment;
+import com.asana.models.Event;
+import com.asana.models.Project;
+import com.asana.models.ProjectMembership;
+import com.asana.models.ProjectStatus;
+import com.asana.models.Resource;
+import com.asana.models.ResultBodyCollection;
+import com.asana.models.Section;
+import com.asana.models.Story;
+import com.asana.models.Tag;
+import com.asana.models.Task;
+import com.asana.models.Team;
+import com.asana.models.User;
+import com.asana.models.Workspace;
+import com.asana.requests.CollectionRequest;
+import com.asana.requests.EventsRequest;
+import com.google.gson.annotations.SerializedName;
+
+import java.io.IOException;
+import java.lang.reflect.Field;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.StreamSupport;
+
+public class StandardAsanaClient implements AsanaClient {
+
+ static final String ASANA_CLIENT_OPTION_BASE_URL = "base_url";
+
+ private final Client client;
+ private final Workspace workspace;
+
+ public StandardAsanaClient(String personalAccessToken, String workspaceName, String baseUrl) {
+ client = Client.accessToken(personalAccessToken);
+ if (baseUrl != null) {
+ client.options.put(ASANA_CLIENT_OPTION_BASE_URL, baseUrl);
+ }
+ workspace = getWorkspaceByName(workspaceName);
+ }
+
+ @Override
+ public Project getProjectByName(String projectName) {
+ return getProjects()
+ .values()
+ .stream()
+ .filter(p -> p.name.equals(projectName))
+ .findFirst()
+ .orElseThrow(() -> new RuntimeException("No such project: " + projectName));
+ }
+
+ @Override
+ public Map<String, Project> getProjects() {
+ try {
+ return collectionRequestToMap(
+ client.projects.getProjects(null, null, workspace.gid, null, null, getSerializedFieldNames(Project.class), false)
+ );
+ } catch (IOException e) {
+ throw new RuntimeException(e);
Review Comment:
Done in [c33ff0f](https://github.com/apache/nifi/pull/6504/commits/c33ff0fd9ab9cee55d4c921ebd54740883235618)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r1030295185
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/pom.xml:
##########
@@ -0,0 +1,69 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-asana-bundle</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-asana-processors</artifactId>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-api</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-mock</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>commons-io</groupId>
+ <artifactId>commons-io</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.commons</groupId>
+ <artifactId>commons-collections4</artifactId>
+ <version>4.4</version>
+ </dependency>
+ <dependency>
+ <groupId>com.asana</groupId>
+ <artifactId>asana</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-utils</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ <scope>compile</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-asana-services-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.mockito</groupId>
+ <artifactId>mockito-junit-jupiter</artifactId>
+ <scope>test</scope>
+ </dependency>
Review Comment:
Done in c72eeab
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r1030319428
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/pom.xml:
##########
@@ -0,0 +1,69 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-asana-bundle</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-asana-processors</artifactId>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-api</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-mock</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>commons-io</groupId>
+ <artifactId>commons-io</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.commons</groupId>
+ <artifactId>commons-collections4</artifactId>
+ <version>4.4</version>
+ </dependency>
+ <dependency>
+ <groupId>com.asana</groupId>
+ <artifactId>asana</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-utils</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ <scope>compile</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-asana-services-api</artifactId>
+ </dependency>
Review Comment:
Addressed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on PR #6504:
URL: https://github.com/apache/nifi/pull/6504#issuecomment-1353524643
> @takraj Thanks for the changes! Overall it looks good to me, except the last commit:
>
> > Fix constant IOException in onTrigger() when DistributedMapCache has no entries
>
> So there was an issue when `DistributedMapCache` was used (I tested it with Hazelcast cache and did not run into it). I debugged it a bit and the root cause seems to be that `GenericObjectSerDe.deserialize()` should handle empty byte arrays. Like in [CacheValueDeserializer](https://github.com/apache/nifi/blob/78be613a0f85b664695ea2cbfaf26163f9b8e454/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/FetchDistributedMapCache.java#L287-L296). Please fix it in the deserializer instead of cacthing the exception in `onTrigger()`.
>
> There are also other changes in the last commit whose purpose is unclear for me: `stateBackup` handling and moving the fetcher creation into `onTrigger()` instead of `onScheduled()`. If there is no good reason for them, I would suggest reverting the whole commit and adding the `GenericObjectSerDe` fix only.
@turcsanyip Done. Please re-check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on PR #6504:
URL: https://github.com/apache/nifi/pull/6504#issuecomment-1331352294
Rebased & squashed. Versions bumped to 1.20.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r1042718710
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,405 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import static java.lang.String.format;
+import static java.lang.String.join;
+import static java.util.Collections.singletonMap;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_EVENTS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_MEMBERS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_STATUS_ATTACHMENTS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_STATUS_UPDATES;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_STORIES;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_TASKS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_TASK_ATTACHMENTS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_TEAM_MEMBERS;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientProviderService;
+import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaObjectState;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+@TriggerSerially
+@PrimaryNodeOnly
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.")
+@Tags({"asana", "source", "ingest"})
+@CapabilityDescription("This processor collects data from Asana")
+public class GetAsanaObject extends AbstractProcessor {
Review Comment:
Done in 92a5f86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r1042720020
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/utils/PollableAsanaObjectFetcher.java:
##########
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana.utils;
+
+import static java.util.Collections.emptyIterator;
+
+import java.util.Iterator;
+
+public abstract class PollableAsanaObjectFetcher implements AsanaObjectFetcher {
+
+ private Iterator<AsanaObject> pending;
+
+ public PollableAsanaObjectFetcher() {
+ pending = emptyIterator();
+ }
+
+ @Override
+ public AsanaObject fetchNext() {
+ if (!pending.hasNext()) {
+ pending = poll();
+ }
+ return pending.hasNext() ? pending.next() : null;
+ }
+
+ protected abstract Iterator<AsanaObject> poll();
Review Comment:
Done in 6f241d7
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on PR #6504:
URL: https://github.com/apache/nifi/pull/6504#issuecomment-1321801319
Thank you for reviewing @turcsanyip ,
> Also noted that the processor stores fingerprints for all seen Asana entries in the state (if I understand correctly). We try to avoid storing a large number of entries or big items in the state. I see the processor stores the fingerprints merged in a json in a single entry. What is the expected size of this state item?
Yes, I believe you do see it correctly. There is no upper bound for the number of these items, and it scales with the number of items in Asana. The fingerprint of a single item is a fixed 64 bytes, encoded in Base64, which is about 37% larger than this, so 80-90 bytes per item is expected. The whole list is gzipped, but as the hashes are traditionally non-compressible, I wouldn't expect much size deflation in the case of a fingerprint list. (approx. 11-12k items would fit in 1 MB compressed state)
Where Asana exposes the modification time (Tasks), or where the items are immutable (Attachments), I used the modification date / creation date as fingerprint, which are just text-represented integers with relatively small differences, and thus well compressible. In this case 4-8 bytes per item is expected. (approx. 130k items would fit in 1 MB compressed state)
There is an alternative solution, where I could utilize the Asana event API, which only requires storing a relatively short ID (less than 100 bytes), and that would notify us when an item is added, updated, or removed. The drawback is that, in this case we would need to fetch the affected items one by one, which may cause rate-limitation problems if there are more than 1500 items to fetch per minute. Another drawback is that we could apply this mechanism only for items having a parent item. Teams, Users, Tags don't have such, therefore in those cases I still need to use the current method, with fingerprints.
Switching to Event API requires a medium-sized change in the code. And almost all Asana integration tests would needed to be changed, because of mocks.
What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r1035780536
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,393 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import static java.lang.String.format;
+import static java.util.Collections.singletonMap;
+import static java.util.stream.Collectors.groupingBy;
+import static java.util.stream.Collectors.joining;
+import static org.apache.commons.collections4.ListUtils.partition;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_EVENTS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_MEMBERS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_STATUS_ATTACHMENTS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_STATUS_UPDATES;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_STORIES;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_TASKS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_TASK_ATTACHMENTS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_TEAM_MEMBERS;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientProviderService;
+import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaObjectState;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+@TriggerSerially
+@PrimaryNodeOnly
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.")
+@Tags({"asana", "source", "ingest"})
+@CapabilityDescription("This processor collects data from Asana")
+public class GetAsanaObject extends AbstractProcessor {
+
+ protected static final String ASANA_GID = "asana.gid";
+ protected static final String ASANA_CLIENT_SERVICE = "asana-controller-service";
+ protected static final String DISTRIBUTED_CACHE_SERVICE = "distributed-cache-service";
+ protected static final String ASANA_OBJECT_TYPE = "asana-object-type";
+ protected static final String ASANA_PROJECT_NAME = "asana-project-name";
+ protected static final String ASANA_SECTION_NAME = "asana-section-name";
+ protected static final String ASANA_TAG_NAME = "asana-tag-name";
+ protected static final String ASANA_TEAM_NAME = "asana-team-name";
+ protected static final String ASANA_OUTPUT_BATCH_SIZE = "asana-output-batch-size";
+ protected static final String REL_NAME_NEW = "new";
+ protected static final String REL_NAME_UPDATED = "updated";
+ protected static final String REL_NAME_REMOVED = "removed";
+
+ protected static final PropertyDescriptor PROP_ASANA_CLIENT_SERVICE = new PropertyDescriptor.Builder()
+ .name(ASANA_CLIENT_SERVICE)
+ .displayName("Asana Client Service")
+ .description("Specify which controller service to use for accessing Asana.")
+ .required(true)
+ .identifiesControllerService(AsanaClientProviderService.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_DISTRIBUTED_CACHE_SERVICE = new Builder()
+ .name(DISTRIBUTED_CACHE_SERVICE)
+ .displayName("Distributed Cache Service")
+ .description("Cache service to store fetched item fingerprints. These, from the last successful query"
+ + " are stored, in order to enable incremental loading and change detection.")
+ .required(true)
+ .identifiesControllerService(DistributedMapCacheClient.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OBJECT_TYPE = new PropertyDescriptor.Builder()
+ .name(ASANA_OBJECT_TYPE)
+ .displayName("Object Type")
+ .description("Specify what kind of objects to be collected from Asana")
+ .required(true)
+ .allowableValues(AsanaObjectType.class)
+ .defaultValue(AV_COLLECT_TASKS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_PROJECT = new PropertyDescriptor.Builder()
+ .name(ASANA_PROJECT_NAME)
+ .displayName("Project Name")
+ .description("Fetch only objects in this project. Case sensitive.")
+ .required(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(
+ PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS.getValue(),
+ AV_COLLECT_TASK_ATTACHMENTS.getValue(),
+ AV_COLLECT_PROJECT_MEMBERS.getValue(),
+ AV_COLLECT_STORIES.getValue(),
+ AV_COLLECT_PROJECT_STATUS_UPDATES.getValue(),
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS.getValue(),
+ AV_COLLECT_PROJECT_EVENTS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_SECTION = new PropertyDescriptor.Builder()
+ .name(ASANA_SECTION_NAME)
+ .displayName("Section Name")
+ .description("Fetch only objects in this section. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS.getValue(),
+ AV_COLLECT_TASK_ATTACHMENTS.getValue(),
+ AV_COLLECT_STORIES.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TAG = new PropertyDescriptor.Builder()
+ .name(ASANA_TAG_NAME)
+ .displayName("Tag")
+ .description("Fetch only objects having this tag. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS.getValue(),
+ AV_COLLECT_TASK_ATTACHMENTS.getValue(),
+ AV_COLLECT_STORIES.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TEAM_NAME = new PropertyDescriptor.Builder()
+ .name(ASANA_TEAM_NAME)
+ .displayName("Team")
+ .description("Team name. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TEAM_MEMBERS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OUTPUT_BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name(ASANA_OUTPUT_BATCH_SIZE)
+ .displayName("Output Batch Size")
+ .description("The number of items batched together in a single Flow File. If set to 1 (default), then each item is"
+ + " transferred in a separate Flow File and each will have an asana.gid attribute, to help identifying"
+ + " the fetched item on the server side, if needed. If the batch size is greater than 1, then the"
+ + " specified amount of items are batched together in a single Flow File as a Json array, and the"
+ + " Flow Files won't have the asana.gid attribute.")
+ .defaultValue("1")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+
+ protected static final List<PropertyDescriptor> DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ PROP_ASANA_CLIENT_SERVICE,
+ PROP_DISTRIBUTED_CACHE_SERVICE,
+ PROP_ASANA_OBJECT_TYPE,
+ PROP_ASANA_PROJECT,
+ PROP_ASANA_SECTION,
+ PROP_ASANA_TEAM_NAME,
+ PROP_ASANA_TAG,
+ PROP_ASANA_OUTPUT_BATCH_SIZE
+ ));
+
+ protected static final Relationship REL_NEW = new Relationship.Builder()
+ .name(REL_NAME_NEW)
+ .description("Newly collected objects are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_UPDATED = new Relationship.Builder()
+ .name(REL_NAME_UPDATED)
+ .description("Objects that have already been collected earlier, but were updated since, are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_REMOVED = new Relationship.Builder()
+ .name(REL_NAME_REMOVED)
+ .description("Notification about deleted objects are routed to this relationship. "
+ + "Flow files will not have any payload. IDs of the resources no longer exist "
+ + "are carried by the asana.gid attribute of the generated FlowFiles.")
+ .build();
+
+ protected static final Set<Relationship> RELATIONSHIPS = Collections.unmodifiableSet(new HashSet<>(Arrays.asList(
+ REL_NEW,
+ REL_UPDATED,
+ REL_REMOVED
+ )));
+
+ private volatile AsanaObjectFetcher objectFetcher;
+ private volatile Integer batchSize;
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return RELATIONSHIPS;
+ }
+
+ @Override
+ public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return DESCRIPTORS;
+ }
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) throws InitializationException {
+ AsanaClientProviderService controllerService = context.getProperty(PROP_ASANA_CLIENT_SERVICE).asControllerService(AsanaClientProviderService.class);
+ AsanaClient client = controllerService.createClient();
+ batchSize = context.getProperty(PROP_ASANA_OUTPUT_BATCH_SIZE).asInteger();
+
+ try {
+ getLogger().debug("Initializing object fetcher...");
+ objectFetcher = createObjectFetcher(context, client);
+ } catch (Exception e) {
+ throw new InitializationException(e);
+ }
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ try {
+ Map<String, String> state = recoverState(context).orElse(Collections.emptyMap());
+ getLogger().debug("Attempting to load state: {}", state);
+ objectFetcher.loadState(state);
+ } catch (Exception e) {
+ getLogger().info("Failed to recover state. Falling back to clean start.");
+ objectFetcher.clearState();
+ }
+ getLogger().debug("Initial state: {}", objectFetcher.saveState());
Review Comment:
Done in 23d890d
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r1031404542
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors-nar/src/main/resources/META-INF/NOTICE:
##########
@@ -0,0 +1,51 @@
+nifi-asana-nar
Review Comment:
This one done in bc6c94a & others in b90eeaf
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r1031471213
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors-nar/src/main/resources/META-INF/NOTICE:
##########
@@ -0,0 +1,51 @@
+nifi-asana-nar
+Copyright 2015-2022 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+===========================================
+Apache Software License v2
+===========================================
+
+The following binary components are provided under the Apache Software License v2
+
+ (ASLv2) Apache Commons IO
+ The following NOTICE information applies:
+ Apache Commons IO
+ Copyright 2002-2017 The Apache Software Foundation
+
+ (ASLv2) Jackson JSON processor
+ The following NOTICE information applies:
+ # Jackson JSON processor
+
+ Jackson is a high-performance, Free/Open Source JSON processing library.
+ It was originally written by Tatu Saloranta (tatu.saloranta@iki.fi), and has
+ been in development since 2007.
+ It is currently developed by a community of developers, as well as supported
+ commercially by FasterXML.com.
+
+ ## Licensing
+
+ Jackson core and extension components may licensed under different licenses.
+ To find the details that apply to this artifact see the accompanying LICENSE file.
+ For more information, including possible other licensing options, contact
+ FasterXML.com (http://fasterxml.com).
+
+ ## Credits
+
+ A list of contributors may be found from CREDITS file, which is included
+ in some artifacts (usually source distributions); but is always available
+ from the source code management (SCM) system project uses.
+
+
+===========================================
+The MIT License
+===========================================
+
+The following binary components are provided under the MIT License
+
+ (MIT License) Java client library for the Asana API
+ The following NOTICE information applies:
+ Asana
+ Copyright (c) 2015
Review Comment:
Done in b90eeaf
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on PR #6504:
URL: https://github.com/apache/nifi/pull/6504#issuecomment-1324788246
@exceptionfactory Thanks for the suggestion about `DistributedMapCacheClient`, I'll check soon if that approach could work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r1030380888
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,457 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import static java.lang.String.format;
+import static java.util.Collections.singletonMap;
+import static java.util.stream.Collectors.groupingBy;
+import static java.util.stream.Collectors.joining;
+import static org.apache.commons.collections4.ListUtils.partition;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientProviderService;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaObjectState;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+@TriggerSerially
+@PrimaryNodeOnly
+@Stateful(scopes = {Scope.CLUSTER}, description = "Fingerprints of items in the last successful query are stored in order to enable incremental loading and change detection.")
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.")
+@Tags({"asana", "source", "ingest"})
+@CapabilityDescription("This processor collects data from Asana")
+public class GetAsanaObject extends AbstractProcessor {
+
+ protected static final String ASANA_GID = "asana.gid";
+ protected static final String AV_NAME_COLLECT_TASKS = "asana-collect-tasks";
+ protected static final String AV_NAME_COLLECT_TASK_ATTACHMENTS = "asana-collect-task-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECTS = "asana-collect-projects";
+ protected static final String AV_NAME_COLLECT_TAGS = "asana-collect-tags";
+ protected static final String AV_NAME_COLLECT_USERS = "asana-collect-users";
+ protected static final String AV_NAME_COLLECT_PROJECT_MEMBERS = "asana-collect-project-members";
+ protected static final String AV_NAME_COLLECT_TEAMS = "asana-collect-teams";
+ protected static final String AV_NAME_COLLECT_TEAM_MEMBERS = "asana-collect-team-members";
+ protected static final String AV_NAME_COLLECT_STORIES = "asana-collect-stories";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_UPDATES = "asana-collect-project-status-updates";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS = "asana-collect-project-status-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECT_EVENTS = "asana-collect-project-events";
+ protected static final String ASANA_CONTROLLER_SERVICE = "asana-controller-service";
+ protected static final String ASANA_OBJECT_TYPE = "asana-object-type";
+ protected static final String ASANA_PROJECT_NAME = "asana-project-name";
+ protected static final String ASANA_SECTION_NAME = "asana-section-name";
+ protected static final String ASANA_TAG_NAME = "asana-tag-name";
+ protected static final String ASANA_TEAM_NAME = "asana-team-name";
+ protected static final String ASANA_OUTPUT_BATCH_SIZE = "asana-output-batch-size";
+ protected static final String REL_NAME_NEW = "new";
+ protected static final String REL_NAME_UPDATED = "updated";
+ protected static final String REL_NAME_REMOVED = "removed";
+
+ protected static final AllowableValue AV_COLLECT_TASKS = new AllowableValue(
+ AV_NAME_COLLECT_TASKS,
+ "Tasks",
+ "Collect tasks matching to the specified conditions.");
+
+ protected static final AllowableValue AV_COLLECT_TASK_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_TASK_ATTACHMENTS,
+ "Task Attachments",
+ "Collect attached files of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECTS,
+ "Projects",
+ "Collect projects of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TAGS = new AllowableValue(
+ AV_NAME_COLLECT_TAGS,
+ "Tags",
+ "Collect tags of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_USERS = new AllowableValue(
+ AV_NAME_COLLECT_USERS,
+ "Users",
+ "Collect users assigned to the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_MEMBERS,
+ "Members of a Project",
+ "Collect users assigned to the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAMS = new AllowableValue(
+ AV_NAME_COLLECT_TEAMS,
+ "Teams",
+ "Collect teams of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAM_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_TEAM_MEMBERS,
+ "Team Members",
+ "Collect users assigned to the specified team."
+ );
+
+ protected static final AllowableValue AV_COLLECT_STORIES = new AllowableValue(
+ AV_NAME_COLLECT_STORIES,
+ "Stories of Tasks",
+ "Collect stories (comments) of of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_UPDATES = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_UPDATES,
+ "Status Updates of a Project",
+ "Collect status updates of the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ "Attachments of Status Updates",
+ "Collect attached files of project status updates."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_EVENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_EVENTS,
+ "Events of a Project",
+ "Collect various events happening on the specified project and on its' tasks."
+ );
+
+ protected static final PropertyDescriptor PROP_ASANA_CONTROLLER_SERVICE = new PropertyDescriptor.Builder()
+ .name(ASANA_CONTROLLER_SERVICE)
+ .displayName("Asana Controller Service")
+ .description("Specify which controller service to use for accessing Asana.")
+ .required(true)
+ .identifiesControllerService(AsanaClientProviderService.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OBJECT_TYPE = new PropertyDescriptor.Builder()
+ .name(ASANA_OBJECT_TYPE)
+ .displayName("Object Type")
+ .description("Specify what kind of objects to be collected from Asana")
+ .required(true)
+ .allowableValues(
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_PROJECTS,
+ AV_COLLECT_TAGS,
+ AV_COLLECT_USERS,
+ AV_COLLECT_PROJECT_MEMBERS,
+ AV_COLLECT_TEAMS,
+ AV_COLLECT_TEAM_MEMBERS,
+ AV_COLLECT_STORIES,
+ AV_COLLECT_PROJECT_STATUS_UPDATES,
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ AV_COLLECT_PROJECT_EVENTS)
+ .defaultValue(AV_COLLECT_TASKS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_PROJECT = new PropertyDescriptor.Builder()
+ .name(ASANA_PROJECT_NAME)
+ .displayName("Project Name")
+ .description("Fetch only objects in this project. Case sensitive.")
+ .required(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(
+ PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_PROJECT_MEMBERS,
+ AV_COLLECT_STORIES,
+ AV_COLLECT_PROJECT_STATUS_UPDATES,
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ AV_COLLECT_PROJECT_EVENTS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_SECTION = new PropertyDescriptor.Builder()
+ .name(ASANA_SECTION_NAME)
+ .displayName("Section Name")
+ .description("Fetch only objects in this section. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TASKS, AV_COLLECT_TASK_ATTACHMENTS, AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TAG = new PropertyDescriptor.Builder()
+ .name(ASANA_TAG_NAME)
+ .displayName("Tag")
+ .description("Fetch only objects having this tag. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TASKS, AV_COLLECT_TASK_ATTACHMENTS, AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TEAM_NAME = new PropertyDescriptor.Builder()
+ .name(ASANA_TEAM_NAME)
+ .displayName("Team")
+ .description("Team name. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TEAM_MEMBERS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OUTPUT_BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name(ASANA_OUTPUT_BATCH_SIZE)
+ .displayName("Output Batch Size")
+ .description("The number of items batched together in a single Flow File. If set to 1 (default), then each item is"
+ + " transferred in a separate Flow File and each will have an asana.gid attribute, to help identifying"
+ + " the fetched item on the server side, if needed. If the batch size is greater than 1, then the"
+ + " specified amount of items are batched together in a single Flow File as a Json array, and the"
+ + " Flow Files won't have the asana.gid attribute.")
+ .defaultValue("1")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+
+ protected static final List<PropertyDescriptor> DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ PROP_ASANA_CONTROLLER_SERVICE,
+ PROP_ASANA_OBJECT_TYPE,
+ PROP_ASANA_PROJECT,
+ PROP_ASANA_SECTION,
+ PROP_ASANA_TEAM_NAME,
+ PROP_ASANA_TAG,
+ PROP_ASANA_OUTPUT_BATCH_SIZE
+ ));
+
+ protected static final Relationship REL_NEW = new Relationship.Builder()
+ .name(REL_NAME_NEW)
+ .description("Newly collected objects are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_UPDATED = new Relationship.Builder()
+ .name(REL_NAME_UPDATED)
+ .description("Objects that have already been collected earlier, but were updated since, are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_REMOVED = new Relationship.Builder()
+ .name(REL_NAME_REMOVED)
+ .description("Notification about deleted objects are routed to this relationship. "
+ + "Flow files will not have any payload. IDs of the resources no longer exist "
+ + "are carried by the asana.gid attribute of the generated FlowFiles.")
+ .build();
+
+ protected static final Set<Relationship> RELATIONSHIPS = Collections.unmodifiableSet(new HashSet<>(Arrays.asList(
+ REL_NEW,
+ REL_UPDATED,
+ REL_REMOVED
+ )));
+
+ private static final Scope STATE_STORAGE_SCOPE = Scope.CLUSTER;
+
+ private volatile AsanaObjectFetcher objectFetcher;
+ private volatile Integer batchSize;
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return RELATIONSHIPS;
+ }
+
+ @Override
+ public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return DESCRIPTORS;
+ }
+
+ @OnScheduled
+ public synchronized void onScheduled(final ProcessContext context) throws InitializationException {
+ AsanaClientProviderService controllerService = context.getProperty(PROP_ASANA_CONTROLLER_SERVICE).asControllerService(AsanaClientProviderService.class);
+ AsanaClient client = controllerService.createClient();
+ batchSize = context.getProperty(PROP_ASANA_OUTPUT_BATCH_SIZE).asInteger();
+
+ try {
+ getLogger().debug("Initializing object fetcher...");
+ objectFetcher = createObjectFetcher(context, client);
+ } catch (Exception e) {
+ throw new InitializationException(e);
+ }
+ }
+
+ @Override
+ public synchronized void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ try {
+ Map<String, String> state = recoverState(context).orElse(Collections.emptyMap());
+ getLogger().debug("Attempting to load state: {}", state);
+ objectFetcher.loadState(state);
+ } catch (Exception e) {
+ getLogger().info("Failed to recover state. Falling back to clean start.");
+ objectFetcher.clearState();
+ }
+ getLogger().debug("Initial state: {}", objectFetcher.saveState());
+
+ Collection<FlowFile> newItems = new ArrayList<>();
+ Collection<FlowFile> updatedItems = new ArrayList<>();
+ Collection<FlowFile> removedItems = new ArrayList<>();
+ Map<AsanaObjectState, Collection<FlowFile>> flowFiles = new HashMap<>();
+ flowFiles.put(AsanaObjectState.NEW, newItems);
+ flowFiles.put(AsanaObjectState.UPDATED, updatedItems);
+ flowFiles.put(AsanaObjectState.REMOVED, removedItems);
+
+ List<AsanaObject> allObjects = new ArrayList<>();
+
+ AsanaObject nextObject;
+ while ((nextObject = objectFetcher.fetchNext()) != null) {
+ allObjects.add(nextObject);
+ }
+
+ Map<AsanaObjectState, List<AsanaObject>> allObjectsByState = allObjects.stream()
+ .collect(groupingBy(AsanaObject::getState));
+
+ if (batchSize == 1) {
+ allObjectsByState
+ .forEach((asanaObjectState, asanaObjects) -> asanaObjects.forEach(
+ asanaObject -> {
+ final Map<String, String> attributes = new HashMap<>(2);
+ attributes.put(CoreAttributes.MIME_TYPE.key(), ContentType.APPLICATION_JSON.getMimeType());
+ attributes.put(ASANA_GID, asanaObject.getGid());
+ FlowFile flowFile = createFlowFileWithStringPayload(session, asanaObject.getContent());
+ flowFile = session.putAllAttributes(flowFile, attributes);
+ flowFiles.get(asanaObject.getState()).add(flowFile);
+ }
+ ));
+ } else {
+ allObjectsByState
+ .forEach((asanaObjectState, asanaObjects) -> partition(asanaObjects, batchSize).forEach(
+ asanaObjectsInPartition -> {
+ FlowFile flowFile = createFlowFileWithStringPayload(session, format("[%s]",
+ asanaObjectsInPartition.stream().map(AsanaObject::getContent)
+ .collect(joining(","))));
+ flowFile = session.putAllAttributes(flowFile,
+ singletonMap(CoreAttributes.MIME_TYPE.key(),
+ ContentType.APPLICATION_JSON.getMimeType()));
+ flowFiles.get(asanaObjectState).add(flowFile);
+ }
+ ));
+ }
+
+ if (flowFiles.values().stream().allMatch(Collection::isEmpty)) {
+ context.yield();
+ getLogger().debug("Yielding, as there are no new FlowFiles.");
Review Comment:
No, we cannot. See my comment: https://github.com/apache/nifi/pull/6504#discussion_r1030379015
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r1030318410
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-services/pom.xml:
##########
@@ -0,0 +1,64 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-asana-bundle</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-asana-services</artifactId>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-api</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-mock</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>com.squareup.okhttp3</groupId>
+ <artifactId>mockwebserver</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>commons-io</groupId>
+ <artifactId>commons-io</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>com.asana</groupId>
+ <artifactId>asana</artifactId>
+ </dependency>
Review Comment:
Had to add f1a07fc in order to prevent `NoClassDefFoundError`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] turcsanyip commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
turcsanyip commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r1027996972
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientServiceApi;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@TriggerSerially
+@Stateful(scopes = {Scope.LOCAL}, description = "Fingerprints of items in the last successful query are stored in order to enable incremental loading and change detection.")
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.")
+@Tags({"asana", "source", "connector", "ingest"})
+@CapabilityDescription("This processor collects data from Asana")
+public class GetAsanaObject extends AbstractProcessor {
+
+ protected static final String ASANA_GID = "asana.gid";
+ protected static final String AV_NAME_COLLECT_TASKS = "asana-collect-tasks";
+ protected static final String AV_NAME_COLLECT_TASK_ATTACHMENTS = "asana-collect-task-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECTS = "asana-collect-projects";
+ protected static final String AV_NAME_COLLECT_TAGS = "asana-collect-tags";
+ protected static final String AV_NAME_COLLECT_USERS = "asana-collect-users";
+ protected static final String AV_NAME_COLLECT_PROJECT_MEMBERS = "asana-collect-project-members";
+ protected static final String AV_NAME_COLLECT_TEAMS = "asana-collect-teams";
+ protected static final String AV_NAME_COLLECT_TEAM_MEMBERS = "asana-collect-team-members";
+ protected static final String AV_NAME_COLLECT_STORIES = "asana-collect-stories";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_UPDATES = "asana-collect-project-status-updates";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS = "asana-collect-project-status-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECT_EVENTS = "asana-collect-project-events";
+ protected static final String ASANA_CONTROLLER_SERVICE = "asana-controller-service";
+ protected static final String ASANA_OBJECT_TYPE = "asana-object-type";
+ protected static final String ASANA_PROJECT_NAME = "asana-project-name";
+ protected static final String ASANA_SECTION_NAME = "asana-section-name";
+ protected static final String ASANA_TAG_NAME = "asana-tag-name";
+ protected static final String ASANA_TEAM_NAME = "asana-team-name";
+ protected static final String ASANA_OUTPUT_BATCH_SIZE = "asana-output-batch-size";
+ protected static final String REL_NAME_NEW = "new";
+ protected static final String REL_NAME_UPDATED = "updated";
+ protected static final String REL_NAME_REMOVED = "removed";
+
+ protected static final AllowableValue AV_COLLECT_TASKS = new AllowableValue(
+ AV_NAME_COLLECT_TASKS,
+ "Tasks",
+ "Collect tasks matching to the specified conditions.");
+
+ protected static final AllowableValue AV_COLLECT_TASK_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_TASK_ATTACHMENTS,
+ "Task attachments",
+ "Collect attached files of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECTS,
+ "Projects",
+ "Collect projects of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TAGS = new AllowableValue(
+ AV_NAME_COLLECT_TAGS,
+ "Tags",
+ "Collect tags of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_USERS = new AllowableValue(
+ AV_NAME_COLLECT_USERS,
+ "Users",
+ "Collect users assigned to the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_MEMBERS,
+ "Members of a project",
+ "Collect users assigned to the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAMS = new AllowableValue(
+ AV_NAME_COLLECT_TEAMS,
+ "Teams",
+ "Collect teams of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAM_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_TEAM_MEMBERS,
+ "Team members",
+ "Collect users assigned to the specified team."
+ );
+
+ protected static final AllowableValue AV_COLLECT_STORIES = new AllowableValue(
+ AV_NAME_COLLECT_STORIES,
+ "Stories of tasks",
+ "Collect stories (comments) of of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_UPDATES = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_UPDATES,
+ "Status updates of a project",
+ "Collect status updates of the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ "Attachments of status updates",
+ "Collect attached files of project status updates."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_EVENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_EVENTS,
+ "Events of a project",
+ "Collect various events happening on the specified project and on its' tasks."
+ );
+
+ protected static final PropertyDescriptor PROP_ASANA_CONTROLLER_SERVICE = new PropertyDescriptor.Builder()
+ .name(ASANA_CONTROLLER_SERVICE)
+ .displayName("Controller service")
Review Comment:
The property has been renamed to `Asana Controller Service`. Could you please use `Asana Client Service` instead as in the suggestion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r995755460
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-services/src/main/java/org/apache/nifi/controller/asana/AsanaClientImpl.java:
##########
@@ -0,0 +1,303 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.controller.asana;
+
+import com.asana.Client;
+import com.asana.errors.InvalidTokenError;
+import com.asana.models.Attachment;
+import com.asana.models.Event;
+import com.asana.models.Project;
+import com.asana.models.ProjectMembership;
+import com.asana.models.ProjectStatus;
+import com.asana.models.Resource;
+import com.asana.models.ResultBodyCollection;
+import com.asana.models.Section;
+import com.asana.models.Story;
+import com.asana.models.Tag;
+import com.asana.models.Task;
+import com.asana.models.Team;
+import com.asana.models.User;
+import com.asana.models.Workspace;
+import com.asana.requests.CollectionRequest;
+import com.asana.requests.EventsRequest;
+import com.google.gson.annotations.SerializedName;
+
+import java.io.IOException;
+import java.lang.reflect.Field;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.StreamSupport;
+
+public class AsanaClientImpl implements AsanaClient {
Review Comment:
Done in [dc2a050](https://github.com/apache/nifi/pull/6504/commits/dc2a050548a389d89181670371748af741eefd61)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r995755730
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-services/src/main/java/org/apache/nifi/controller/asana/AsanaClientService.java:
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.controller.asana;
+
+import com.asana.Client;
+import com.google.common.collect.Lists;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.resource.ResourceCardinality;
+import org.apache.nifi.components.resource.ResourceType;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.Collections;
+import java.util.List;
+
+import static org.apache.nifi.controller.asana.AsanaClientImpl.ASANA_CLIENT_OPTION_BASE_URL;
+
+@CapabilityDescription("Common service to authenticate with Asana, and to work on a specified workspace.")
+@Tags({"asana", "service", "authentication"})
+public class AsanaClientService extends AbstractControllerService implements AsanaClientServiceApi {
+
+ protected static final String ASANA_API_URL = "asana-api-url";
+ protected static final String ASANA_PERSONAL_ACCESS_TOKEN = "asana-personal-access-token";
+ protected static final String ASANA_WORKSPACE_NAME = "asana-workspace-name";
+
+ protected static final PropertyDescriptor PROP_ASANA_API_BASE_URL = new PropertyDescriptor.Builder()
+ .name(ASANA_API_URL)
+ .displayName("API URL")
+ .description("Base URL of Asana API. Leave it as default, unless you have your own Asana instance "
+ + "serving on a different URL. (typical for on-premise installations)")
+ .required(true)
+ .defaultValue(Client.DEFAULTS.get(ASANA_CLIENT_OPTION_BASE_URL).toString())
+ .identifiesExternalResource(ResourceCardinality.SINGLE, ResourceType.URL)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_PERSONAL_ACCESS_TOKEN = new PropertyDescriptor.Builder()
+ .name(ASANA_PERSONAL_ACCESS_TOKEN)
+ .displayName("Personal Access Token")
+ .description("Similarly to entering your username/password into a website, when you access "
+ + "your Asana data via the API you need to authenticate. Personal Access Token (PAT) "
+ + "is an authentication mechanism for accessing the API. You can generate a PAT from "
+ + "the Asana developer console. Refer to Asana Authentication Quick Start for detailed "
+ + "instructions on getting started.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_WORKSPACE_NAME = new PropertyDescriptor.Builder()
+ .name(ASANA_WORKSPACE_NAME)
+ .displayName("Workspace")
+ .description("Specify which Asana workspace to use. Case sensitive. "
+ + "A workspace is the highest-level organizational unit in Asana. All projects and tasks "
+ + "have an associated workspace. An organization is a special kind of workspace that "
+ + "represents a company. In an organization, you can group your projects into teams.")
+ .required(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .build();
+
+ protected static final List<PropertyDescriptor> DESCRIPTORS = Collections.unmodifiableList(Lists.newArrayList(
+ PROP_ASANA_API_BASE_URL,
+ PROP_ASANA_PERSONAL_ACCESS_TOKEN,
+ PROP_ASANA_WORKSPACE_NAME
+ ));
+
+ private String personalAccessToken;
+ private String workspaceName;
+ private String baseUrl;
Review Comment:
Done in [dc2a050](https://github.com/apache/nifi/pull/6504/commits/dc2a050548a389d89181670371748af741eefd61)
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientServiceApi;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@TriggerSerially
+@Stateful(scopes = {Scope.LOCAL}, description = "Fingerprints of items in the last successful query are stored in order to enable incremental loading and change detection.")
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.")
+@Tags({"asana", "source", "connector", "ingest"})
+@CapabilityDescription("This processor collects data from Asana")
+public class GetAsanaObject extends AbstractProcessor {
+
+ protected static final String ASANA_GID = "asana.gid";
+ protected static final String AV_NAME_COLLECT_TASKS = "asana-collect-tasks";
+ protected static final String AV_NAME_COLLECT_TASK_ATTACHMENTS = "asana-collect-task-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECTS = "asana-collect-projects";
+ protected static final String AV_NAME_COLLECT_TAGS = "asana-collect-tags";
+ protected static final String AV_NAME_COLLECT_USERS = "asana-collect-users";
+ protected static final String AV_NAME_COLLECT_PROJECT_MEMBERS = "asana-collect-project-members";
+ protected static final String AV_NAME_COLLECT_TEAMS = "asana-collect-teams";
+ protected static final String AV_NAME_COLLECT_TEAM_MEMBERS = "asana-collect-team-members";
+ protected static final String AV_NAME_COLLECT_STORIES = "asana-collect-stories";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_UPDATES = "asana-collect-project-status-updates";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS = "asana-collect-project-status-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECT_EVENTS = "asana-collect-project-events";
+ protected static final String ASANA_CONTROLLER_SERVICE = "asana-controller-service";
+ protected static final String ASANA_OBJECT_TYPE = "asana-object-type";
+ protected static final String ASANA_PROJECT_NAME = "asana-project-name";
+ protected static final String ASANA_SECTION_NAME = "asana-section-name";
+ protected static final String ASANA_TAG_NAME = "asana-tag-name";
+ protected static final String ASANA_TEAM_NAME = "asana-team-name";
+ protected static final String ASANA_OUTPUT_BATCH_SIZE = "asana-output-batch-size";
+ protected static final String REL_NAME_NEW = "new";
+ protected static final String REL_NAME_UPDATED = "updated";
+ protected static final String REL_NAME_REMOVED = "removed";
+
+ protected static final AllowableValue AV_COLLECT_TASKS = new AllowableValue(
+ AV_NAME_COLLECT_TASKS,
+ "Tasks",
+ "Collect tasks matching to the specified conditions.");
+
+ protected static final AllowableValue AV_COLLECT_TASK_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_TASK_ATTACHMENTS,
+ "Task attachments",
+ "Collect attached files of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECTS,
+ "Projects",
+ "Collect projects of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TAGS = new AllowableValue(
+ AV_NAME_COLLECT_TAGS,
+ "Tags",
+ "Collect tags of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_USERS = new AllowableValue(
+ AV_NAME_COLLECT_USERS,
+ "Users",
+ "Collect users assigned to the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_MEMBERS,
+ "Members of a project",
+ "Collect users assigned to the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAMS = new AllowableValue(
+ AV_NAME_COLLECT_TEAMS,
+ "Teams",
+ "Collect teams of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAM_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_TEAM_MEMBERS,
+ "Team members",
+ "Collect users assigned to the specified team."
+ );
+
+ protected static final AllowableValue AV_COLLECT_STORIES = new AllowableValue(
+ AV_NAME_COLLECT_STORIES,
+ "Stories of tasks",
+ "Collect stories (comments) of of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_UPDATES = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_UPDATES,
+ "Status updates of a project",
+ "Collect status updates of the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ "Attachments of status updates",
+ "Collect attached files of project status updates."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_EVENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_EVENTS,
+ "Events of a project",
+ "Collect various events happening on the specified project and on its' tasks."
+ );
+
+ protected static final PropertyDescriptor PROP_ASANA_CONTROLLER_SERVICE = new PropertyDescriptor.Builder()
+ .name(ASANA_CONTROLLER_SERVICE)
+ .displayName("Controller service")
+ .description("Specify which controller service to use for accessing Asana.")
+ .required(true)
+ .identifiesControllerService(AsanaClientServiceApi.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OBJECT_TYPE = new PropertyDescriptor.Builder()
+ .name(ASANA_OBJECT_TYPE)
+ .displayName("Object type to be collected")
+ .description("Specify what kind of objects to be collected from Asana")
+ .required(true)
+ .allowableValues(
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_PROJECTS,
+ AV_COLLECT_TAGS,
+ AV_COLLECT_USERS,
+ AV_COLLECT_PROJECT_MEMBERS,
+ AV_COLLECT_TEAMS,
+ AV_COLLECT_TEAM_MEMBERS,
+ AV_COLLECT_STORIES,
+ AV_COLLECT_PROJECT_STATUS_UPDATES,
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ AV_COLLECT_PROJECT_EVENTS)
+ .defaultValue(AV_COLLECT_TASKS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_PROJECT = new PropertyDescriptor.Builder()
+ .name(ASANA_PROJECT_NAME)
+ .displayName("Project name")
+ .description("Fetch only objects in this project. Case sensitive.")
+ .required(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(
+ PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_PROJECT_MEMBERS,
+ AV_COLLECT_STORIES,
+ AV_COLLECT_PROJECT_STATUS_UPDATES,
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ AV_COLLECT_PROJECT_EVENTS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_SECTION = new PropertyDescriptor.Builder()
+ .name(ASANA_SECTION_NAME)
+ .displayName("Section name")
+ .description("Fetch only objects in this section. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TASKS, AV_COLLECT_TASK_ATTACHMENTS, AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TAG = new PropertyDescriptor.Builder()
+ .name(ASANA_TAG_NAME)
+ .displayName("Tag")
+ .description("Fetch only objects having this tag. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TASKS, AV_COLLECT_TASK_ATTACHMENTS, AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TEAM_NAME = new PropertyDescriptor.Builder()
+ .name(ASANA_TEAM_NAME)
+ .displayName("Team")
+ .description("Team name. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TEAM_MEMBERS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OUTPUT_BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name(ASANA_OUTPUT_BATCH_SIZE)
+ .displayName("Output Batch Size")
+ .description("The number of output FlowFiles to queue before committing the process session. When set to zero, the session will be committed when all result set rows "
+ + "have been processed and the output FlowFiles are ready for transfer to the downstream relationship. For large result sets, this can cause a large burst of FlowFiles "
+ + "to be transferred at the end of processor execution. If this property is set, then when the specified number of FlowFiles are ready for transfer, then the session will "
+ + "be committed, thus releasing the FlowFiles to the downstream relationship.")
+ .defaultValue("0")
+ .required(true)
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+
+ protected static final List<PropertyDescriptor> DESCRIPTORS = Collections.unmodifiableList(Lists.newArrayList(
+ PROP_ASANA_CONTROLLER_SERVICE,
+ PROP_ASANA_OBJECT_TYPE,
+ PROP_ASANA_PROJECT,
+ PROP_ASANA_SECTION,
+ PROP_ASANA_TEAM_NAME,
+ PROP_ASANA_TAG,
+ PROP_ASANA_OUTPUT_BATCH_SIZE
+ ));
+
+ protected static final Relationship REL_NEW = new Relationship.Builder()
+ .name(REL_NAME_NEW)
+ .description("Newly collected objects are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_UPDATED = new Relationship.Builder()
+ .name(REL_NAME_UPDATED)
+ .description("Objects that have already been collected earlier, but were updated since, are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_REMOVED = new Relationship.Builder()
+ .name(REL_NAME_REMOVED)
+ .description("Notification about deleted objects are routed to this relationship. "
+ + "Flow files will not have any payload. IDs of the resources no longer exist "
+ + "are carried by the asana.gid attribute of the generated FlowFiles.")
+ .build();
+
+ protected static final Set<Relationship> RELATIONSHIPS = Collections.unmodifiableSet(Sets.newHashSet(
+ REL_NEW,
+ REL_UPDATED,
+ REL_REMOVED
+ ));
+
+ final Scope STATE_STORAGE_SCOPE = Scope.LOCAL;
Review Comment:
Done in [dc2a050](https://github.com/apache/nifi/pull/6504/commits/dc2a050548a389d89181670371748af741eefd61)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r1031393295
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientServiceApi;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@TriggerSerially
+@Stateful(scopes = {Scope.LOCAL}, description = "Fingerprints of items in the last successful query are stored in order to enable incremental loading and change detection.")
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.")
+@Tags({"asana", "source", "connector", "ingest"})
+@CapabilityDescription("This processor collects data from Asana")
+public class GetAsanaObject extends AbstractProcessor {
+
+ protected static final String ASANA_GID = "asana.gid";
+ protected static final String AV_NAME_COLLECT_TASKS = "asana-collect-tasks";
+ protected static final String AV_NAME_COLLECT_TASK_ATTACHMENTS = "asana-collect-task-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECTS = "asana-collect-projects";
+ protected static final String AV_NAME_COLLECT_TAGS = "asana-collect-tags";
+ protected static final String AV_NAME_COLLECT_USERS = "asana-collect-users";
+ protected static final String AV_NAME_COLLECT_PROJECT_MEMBERS = "asana-collect-project-members";
+ protected static final String AV_NAME_COLLECT_TEAMS = "asana-collect-teams";
+ protected static final String AV_NAME_COLLECT_TEAM_MEMBERS = "asana-collect-team-members";
+ protected static final String AV_NAME_COLLECT_STORIES = "asana-collect-stories";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_UPDATES = "asana-collect-project-status-updates";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS = "asana-collect-project-status-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECT_EVENTS = "asana-collect-project-events";
+ protected static final String ASANA_CONTROLLER_SERVICE = "asana-controller-service";
+ protected static final String ASANA_OBJECT_TYPE = "asana-object-type";
+ protected static final String ASANA_PROJECT_NAME = "asana-project-name";
+ protected static final String ASANA_SECTION_NAME = "asana-section-name";
+ protected static final String ASANA_TAG_NAME = "asana-tag-name";
+ protected static final String ASANA_TEAM_NAME = "asana-team-name";
+ protected static final String ASANA_OUTPUT_BATCH_SIZE = "asana-output-batch-size";
+ protected static final String REL_NAME_NEW = "new";
+ protected static final String REL_NAME_UPDATED = "updated";
+ protected static final String REL_NAME_REMOVED = "removed";
+
+ protected static final AllowableValue AV_COLLECT_TASKS = new AllowableValue(
+ AV_NAME_COLLECT_TASKS,
+ "Tasks",
+ "Collect tasks matching to the specified conditions.");
+
+ protected static final AllowableValue AV_COLLECT_TASK_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_TASK_ATTACHMENTS,
+ "Task attachments",
+ "Collect attached files of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECTS,
+ "Projects",
+ "Collect projects of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TAGS = new AllowableValue(
+ AV_NAME_COLLECT_TAGS,
+ "Tags",
+ "Collect tags of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_USERS = new AllowableValue(
+ AV_NAME_COLLECT_USERS,
+ "Users",
+ "Collect users assigned to the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_MEMBERS,
+ "Members of a project",
+ "Collect users assigned to the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAMS = new AllowableValue(
+ AV_NAME_COLLECT_TEAMS,
+ "Teams",
+ "Collect teams of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAM_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_TEAM_MEMBERS,
+ "Team members",
+ "Collect users assigned to the specified team."
+ );
+
+ protected static final AllowableValue AV_COLLECT_STORIES = new AllowableValue(
+ AV_NAME_COLLECT_STORIES,
+ "Stories of tasks",
+ "Collect stories (comments) of of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_UPDATES = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_UPDATES,
+ "Status updates of a project",
+ "Collect status updates of the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ "Attachments of status updates",
+ "Collect attached files of project status updates."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_EVENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_EVENTS,
+ "Events of a project",
+ "Collect various events happening on the specified project and on its' tasks."
+ );
+
+ protected static final PropertyDescriptor PROP_ASANA_CONTROLLER_SERVICE = new PropertyDescriptor.Builder()
+ .name(ASANA_CONTROLLER_SERVICE)
+ .displayName("Controller service")
Review Comment:
Addressed in df3c96ed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r1031400055
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-services/pom.xml:
##########
@@ -0,0 +1,64 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-asana-bundle</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-asana-services</artifactId>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-api</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-mock</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>com.squareup.okhttp3</groupId>
+ <artifactId>mockwebserver</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>commons-io</groupId>
+ <artifactId>commons-io</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>com.asana</groupId>
+ <artifactId>asana</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-utils</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ <scope>compile</scope>
+ </dependency>
Review Comment:
Done in d584219
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r1031397141
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-services-api/pom.xml:
##########
@@ -0,0 +1,39 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-asana-bundle</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-asana-services-api</artifactId>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-api</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ </dependency>
Review Comment:
Done in bc6c94a
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r1031561336
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/AsanaObjectType.java:
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import java.util.Arrays;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.DescribedValue;
+
+public enum AsanaObjectType implements DescribedValue {
+ AV_COLLECT_TASKS(
+ "asana-collect-tasks",
+ "Tasks", ""
+ + "Collect tasks matching to the specified conditions."
+ ),
+ AV_COLLECT_TASK_ATTACHMENTS(
+ "asana-collect-task-attachments",
+ "Task Attachments",
+ "Collect attached files of tasks matching to the specified conditions."
+ ),
+ AV_COLLECT_PROJECTS(
+ "asana-collect-projects",
+ "Projects",
+ "Collect projects of the workspace."
+ ),
+ AV_COLLECT_TAGS(
+ "asana-collect-tags",
+ "Tags",
+ "Collect tags of the workspace."
+ ),
+ AV_COLLECT_USERS(
+ "asana-collect-users",
+ "Users",
+ "Collect users assigned to the workspace."
+ ),
+ AV_COLLECT_PROJECT_MEMBERS(
+ "asana-collect-project-members",
+ "Members of a Project",
+ "Collect users assigned to the specified project."
+ ),
+ AV_COLLECT_TEAMS(
+ "asana-collect-teams",
+ "Teams",
+ "Collect teams of the workspace."
+ ),
+ AV_COLLECT_TEAM_MEMBERS(
+ "asana-collect-team-members",
+ "Team Members",
+ "Collect users assigned to the specified team."
+ ),
+ AV_COLLECT_STORIES(
+ "asana-collect-stories",
+ "Stories of Tasks",
+ "Collect stories (comments) of of tasks matching to the specified conditions."
+ ),
+ AV_COLLECT_PROJECT_STATUS_UPDATES(
+ "asana-collect-project-status-updates",
+ "Status Updates of a Project",
+ "Collect status updates of the specified project."
+ ),
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS(
+ "asana-collect-project-status-attachments",
+ "Attachments of Status Updates",
+ "Collect attached files of project status updates."
+ ),
+ AV_COLLECT_PROJECT_EVENTS(
+ "asana-collect-project-events",
+ "Events of a Project",
+ "Collect various events happening on the specified project and on its' tasks."
+ );
+
+ private final String value;
+ private final String displayName;
+ private final String description;
+
+ AsanaObjectType(String value, String displayName, String description) {
+ this.value = value;
+ this.displayName = displayName;
+ this.description = description;
+ }
+
+ @Override
+ public String getValue() {
+ return value;
+ }
+
+ @Override
+ public String getDisplayName() {
+ return displayName;
+ }
+
+ @Override
+ public String getDescription() {
+ return description;
+ }
+
+ public AllowableValue createAllowableValue() {
+ return new AllowableValue(value, displayName, description);
+ }
Review Comment:
Removed in b90f954
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r1030379015
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,457 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import static java.lang.String.format;
+import static java.util.Collections.singletonMap;
+import static java.util.stream.Collectors.groupingBy;
+import static java.util.stream.Collectors.joining;
+import static org.apache.commons.collections4.ListUtils.partition;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientProviderService;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaObjectState;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+@TriggerSerially
+@PrimaryNodeOnly
+@Stateful(scopes = {Scope.CLUSTER}, description = "Fingerprints of items in the last successful query are stored in order to enable incremental loading and change detection.")
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.")
+@Tags({"asana", "source", "ingest"})
+@CapabilityDescription("This processor collects data from Asana")
+public class GetAsanaObject extends AbstractProcessor {
+
+ protected static final String ASANA_GID = "asana.gid";
+ protected static final String AV_NAME_COLLECT_TASKS = "asana-collect-tasks";
+ protected static final String AV_NAME_COLLECT_TASK_ATTACHMENTS = "asana-collect-task-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECTS = "asana-collect-projects";
+ protected static final String AV_NAME_COLLECT_TAGS = "asana-collect-tags";
+ protected static final String AV_NAME_COLLECT_USERS = "asana-collect-users";
+ protected static final String AV_NAME_COLLECT_PROJECT_MEMBERS = "asana-collect-project-members";
+ protected static final String AV_NAME_COLLECT_TEAMS = "asana-collect-teams";
+ protected static final String AV_NAME_COLLECT_TEAM_MEMBERS = "asana-collect-team-members";
+ protected static final String AV_NAME_COLLECT_STORIES = "asana-collect-stories";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_UPDATES = "asana-collect-project-status-updates";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS = "asana-collect-project-status-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECT_EVENTS = "asana-collect-project-events";
+ protected static final String ASANA_CONTROLLER_SERVICE = "asana-controller-service";
+ protected static final String ASANA_OBJECT_TYPE = "asana-object-type";
+ protected static final String ASANA_PROJECT_NAME = "asana-project-name";
+ protected static final String ASANA_SECTION_NAME = "asana-section-name";
+ protected static final String ASANA_TAG_NAME = "asana-tag-name";
+ protected static final String ASANA_TEAM_NAME = "asana-team-name";
+ protected static final String ASANA_OUTPUT_BATCH_SIZE = "asana-output-batch-size";
+ protected static final String REL_NAME_NEW = "new";
+ protected static final String REL_NAME_UPDATED = "updated";
+ protected static final String REL_NAME_REMOVED = "removed";
+
+ protected static final AllowableValue AV_COLLECT_TASKS = new AllowableValue(
+ AV_NAME_COLLECT_TASKS,
+ "Tasks",
+ "Collect tasks matching to the specified conditions.");
+
+ protected static final AllowableValue AV_COLLECT_TASK_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_TASK_ATTACHMENTS,
+ "Task Attachments",
+ "Collect attached files of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECTS,
+ "Projects",
+ "Collect projects of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TAGS = new AllowableValue(
+ AV_NAME_COLLECT_TAGS,
+ "Tags",
+ "Collect tags of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_USERS = new AllowableValue(
+ AV_NAME_COLLECT_USERS,
+ "Users",
+ "Collect users assigned to the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_MEMBERS,
+ "Members of a Project",
+ "Collect users assigned to the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAMS = new AllowableValue(
+ AV_NAME_COLLECT_TEAMS,
+ "Teams",
+ "Collect teams of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAM_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_TEAM_MEMBERS,
+ "Team Members",
+ "Collect users assigned to the specified team."
+ );
+
+ protected static final AllowableValue AV_COLLECT_STORIES = new AllowableValue(
+ AV_NAME_COLLECT_STORIES,
+ "Stories of Tasks",
+ "Collect stories (comments) of of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_UPDATES = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_UPDATES,
+ "Status Updates of a Project",
+ "Collect status updates of the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ "Attachments of Status Updates",
+ "Collect attached files of project status updates."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_EVENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_EVENTS,
+ "Events of a Project",
+ "Collect various events happening on the specified project and on its' tasks."
+ );
+
+ protected static final PropertyDescriptor PROP_ASANA_CONTROLLER_SERVICE = new PropertyDescriptor.Builder()
+ .name(ASANA_CONTROLLER_SERVICE)
+ .displayName("Asana Controller Service")
+ .description("Specify which controller service to use for accessing Asana.")
+ .required(true)
+ .identifiesControllerService(AsanaClientProviderService.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OBJECT_TYPE = new PropertyDescriptor.Builder()
+ .name(ASANA_OBJECT_TYPE)
+ .displayName("Object Type")
+ .description("Specify what kind of objects to be collected from Asana")
+ .required(true)
+ .allowableValues(
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_PROJECTS,
+ AV_COLLECT_TAGS,
+ AV_COLLECT_USERS,
+ AV_COLLECT_PROJECT_MEMBERS,
+ AV_COLLECT_TEAMS,
+ AV_COLLECT_TEAM_MEMBERS,
+ AV_COLLECT_STORIES,
+ AV_COLLECT_PROJECT_STATUS_UPDATES,
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ AV_COLLECT_PROJECT_EVENTS)
+ .defaultValue(AV_COLLECT_TASKS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_PROJECT = new PropertyDescriptor.Builder()
+ .name(ASANA_PROJECT_NAME)
+ .displayName("Project Name")
+ .description("Fetch only objects in this project. Case sensitive.")
+ .required(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(
+ PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_PROJECT_MEMBERS,
+ AV_COLLECT_STORIES,
+ AV_COLLECT_PROJECT_STATUS_UPDATES,
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ AV_COLLECT_PROJECT_EVENTS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_SECTION = new PropertyDescriptor.Builder()
+ .name(ASANA_SECTION_NAME)
+ .displayName("Section Name")
+ .description("Fetch only objects in this section. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TASKS, AV_COLLECT_TASK_ATTACHMENTS, AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TAG = new PropertyDescriptor.Builder()
+ .name(ASANA_TAG_NAME)
+ .displayName("Tag")
+ .description("Fetch only objects having this tag. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TASKS, AV_COLLECT_TASK_ATTACHMENTS, AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TEAM_NAME = new PropertyDescriptor.Builder()
+ .name(ASANA_TEAM_NAME)
+ .displayName("Team")
+ .description("Team name. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TEAM_MEMBERS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OUTPUT_BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name(ASANA_OUTPUT_BATCH_SIZE)
+ .displayName("Output Batch Size")
+ .description("The number of items batched together in a single Flow File. If set to 1 (default), then each item is"
+ + " transferred in a separate Flow File and each will have an asana.gid attribute, to help identifying"
+ + " the fetched item on the server side, if needed. If the batch size is greater than 1, then the"
+ + " specified amount of items are batched together in a single Flow File as a Json array, and the"
+ + " Flow Files won't have the asana.gid attribute.")
+ .defaultValue("1")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+
+ protected static final List<PropertyDescriptor> DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ PROP_ASANA_CONTROLLER_SERVICE,
+ PROP_ASANA_OBJECT_TYPE,
+ PROP_ASANA_PROJECT,
+ PROP_ASANA_SECTION,
+ PROP_ASANA_TEAM_NAME,
+ PROP_ASANA_TAG,
+ PROP_ASANA_OUTPUT_BATCH_SIZE
+ ));
+
+ protected static final Relationship REL_NEW = new Relationship.Builder()
+ .name(REL_NAME_NEW)
+ .description("Newly collected objects are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_UPDATED = new Relationship.Builder()
+ .name(REL_NAME_UPDATED)
+ .description("Objects that have already been collected earlier, but were updated since, are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_REMOVED = new Relationship.Builder()
+ .name(REL_NAME_REMOVED)
+ .description("Notification about deleted objects are routed to this relationship. "
+ + "Flow files will not have any payload. IDs of the resources no longer exist "
+ + "are carried by the asana.gid attribute of the generated FlowFiles.")
+ .build();
+
+ protected static final Set<Relationship> RELATIONSHIPS = Collections.unmodifiableSet(new HashSet<>(Arrays.asList(
+ REL_NEW,
+ REL_UPDATED,
+ REL_REMOVED
+ )));
+
+ private static final Scope STATE_STORAGE_SCOPE = Scope.CLUSTER;
+
+ private volatile AsanaObjectFetcher objectFetcher;
+ private volatile Integer batchSize;
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return RELATIONSHIPS;
+ }
+
+ @Override
+ public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return DESCRIPTORS;
+ }
+
+ @OnScheduled
+ public synchronized void onScheduled(final ProcessContext context) throws InitializationException {
+ AsanaClientProviderService controllerService = context.getProperty(PROP_ASANA_CONTROLLER_SERVICE).asControllerService(AsanaClientProviderService.class);
+ AsanaClient client = controllerService.createClient();
+ batchSize = context.getProperty(PROP_ASANA_OUTPUT_BATCH_SIZE).asInteger();
+
+ try {
+ getLogger().debug("Initializing object fetcher...");
+ objectFetcher = createObjectFetcher(context, client);
+ } catch (Exception e) {
+ throw new InitializationException(e);
+ }
+ }
+
+ @Override
+ public synchronized void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ try {
+ Map<String, String> state = recoverState(context).orElse(Collections.emptyMap());
+ getLogger().debug("Attempting to load state: {}", state);
+ objectFetcher.loadState(state);
+ } catch (Exception e) {
+ getLogger().info("Failed to recover state. Falling back to clean start.");
+ objectFetcher.clearState();
+ }
+ getLogger().debug("Initial state: {}", objectFetcher.saveState());
+
+ Collection<FlowFile> newItems = new ArrayList<>();
+ Collection<FlowFile> updatedItems = new ArrayList<>();
+ Collection<FlowFile> removedItems = new ArrayList<>();
+ Map<AsanaObjectState, Collection<FlowFile>> flowFiles = new HashMap<>();
+ flowFiles.put(AsanaObjectState.NEW, newItems);
+ flowFiles.put(AsanaObjectState.UPDATED, updatedItems);
+ flowFiles.put(AsanaObjectState.REMOVED, removedItems);
+
+ List<AsanaObject> allObjects = new ArrayList<>();
+
+ AsanaObject nextObject;
+ while ((nextObject = objectFetcher.fetchNext()) != null) {
+ allObjects.add(nextObject);
+ }
+
+ Map<AsanaObjectState, List<AsanaObject>> allObjectsByState = allObjects.stream()
+ .collect(groupingBy(AsanaObject::getState));
+
+ if (batchSize == 1) {
+ allObjectsByState
+ .forEach((asanaObjectState, asanaObjects) -> asanaObjects.forEach(
+ asanaObject -> {
+ final Map<String, String> attributes = new HashMap<>(2);
+ attributes.put(CoreAttributes.MIME_TYPE.key(), ContentType.APPLICATION_JSON.getMimeType());
+ attributes.put(ASANA_GID, asanaObject.getGid());
+ FlowFile flowFile = createFlowFileWithStringPayload(session, asanaObject.getContent());
+ flowFile = session.putAllAttributes(flowFile, attributes);
+ flowFiles.get(asanaObject.getState()).add(flowFile);
+ }
+ ));
+ } else {
+ allObjectsByState
+ .forEach((asanaObjectState, asanaObjects) -> partition(asanaObjects, batchSize).forEach(
+ asanaObjectsInPartition -> {
+ FlowFile flowFile = createFlowFileWithStringPayload(session, format("[%s]",
+ asanaObjectsInPartition.stream().map(AsanaObject::getContent)
+ .collect(joining(","))));
+ flowFile = session.putAllAttributes(flowFile,
+ singletonMap(CoreAttributes.MIME_TYPE.key(),
+ ContentType.APPLICATION_JSON.getMimeType()));
+ flowFiles.get(asanaObjectState).add(flowFile);
+ }
+ ));
+ }
+
+ if (flowFiles.values().stream().allMatch(Collection::isEmpty)) {
+ context.yield();
+ getLogger().debug("Yielding, as there are no new FlowFiles.");
+ } else {
+ session.transfer(newItems, REL_NEW);
+ session.transfer(updatedItems, REL_UPDATED);
+ session.transfer(removedItems, REL_REMOVED);
+ session.commitAsync();
+ }
+ Map<String, String> state = objectFetcher.saveState();
+ try {
+ persistState(state, context);
+ } catch (IOException e) {
+ throw new ProcessException(e);
+ }
Review Comment:
Tried to apply this suggestion in df3c96e . but since [`AsanaProjectEventFetcher`](https://github.com/apache/nifi/blob/126b1b5aaf2158e6c1f68854bb1058f0971d3578/nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/utils/AsanaProjectEventFetcher.java) generates new state even if there are no new `FlowFiles` are generated, it would break, because the new state would not be persisted.
Added 126b1b5 to address this issue. Only parts of your suggestion have been applied.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r1031407472
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-services-nar/src/main/resources/META-INF/NOTICE:
##########
@@ -0,0 +1,51 @@
+nifi-asana-nar
+Copyright 2015-2022 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+===========================================
+Apache Software License v2
+===========================================
+
+The following binary components are provided under the Apache Software License v2
+
+ (ASLv2) Apache Commons IO
+ The following NOTICE information applies:
+ Apache Commons IO
+ Copyright 2002-2017 The Apache Software Foundation
+
+ (ASLv2) Jackson JSON processor
+ The following NOTICE information applies:
+ # Jackson JSON processor
+
+ Jackson is a high-performance, Free/Open Source JSON processing library.
+ It was originally written by Tatu Saloranta (tatu.saloranta@iki.fi), and has
+ been in development since 2007.
+ It is currently developed by a community of developers, as well as supported
+ commercially by FasterXML.com.
+
+ ## Licensing
+
+ Jackson core and extension components may licensed under different licenses.
+ To find the details that apply to this artifact see the accompanying LICENSE file.
+ For more information, including possible other licensing options, contact
+ FasterXML.com (http://fasterxml.com).
+
+ ## Credits
+
+ A list of contributors may be found from CREDITS file, which is included
+ in some artifacts (usually source distributions); but is always available
+ from the source code management (SCM) system project uses.
+
+
+===========================================
+The MIT License
+===========================================
+
+The following binary components are provided under the MIT License
+
+ (MIT License) Java client library for the Asana API
+ The following NOTICE information applies:
+ Asana
+ Copyright (c) 2015
Review Comment:
Done in b90eeaf
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-services-nar/src/main/resources/META-INF/NOTICE:
##########
@@ -0,0 +1,51 @@
+nifi-asana-nar
+Copyright 2015-2022 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+===========================================
+Apache Software License v2
+===========================================
+
+The following binary components are provided under the Apache Software License v2
+
+ (ASLv2) Apache Commons IO
+ The following NOTICE information applies:
+ Apache Commons IO
+ Copyright 2002-2017 The Apache Software Foundation
+
+ (ASLv2) Jackson JSON processor
+ The following NOTICE information applies:
+ # Jackson JSON processor
+
+ Jackson is a high-performance, Free/Open Source JSON processing library.
+ It was originally written by Tatu Saloranta (tatu.saloranta@iki.fi), and has
+ been in development since 2007.
+ It is currently developed by a community of developers, as well as supported
+ commercially by FasterXML.com.
+
+ ## Licensing
+
+ Jackson core and extension components may licensed under different licenses.
+ To find the details that apply to this artifact see the accompanying LICENSE file.
+ For more information, including possible other licensing options, contact
+ FasterXML.com (http://fasterxml.com).
+
+ ## Credits
+
+ A list of contributors may be found from CREDITS file, which is included
+ in some artifacts (usually source distributions); but is always available
+ from the source code management (SCM) system project uses.
Review Comment:
Done in b90eeaf
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] turcsanyip commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
turcsanyip commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r1035214769
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-services-api/src/main/java/org/apache/nifi/controller/asana/AsanaClient.java:
##########
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.controller.asana;
+
+import com.asana.models.Attachment;
+import com.asana.models.Project;
+import com.asana.models.ProjectMembership;
+import com.asana.models.ProjectStatus;
+import com.asana.models.Section;
+import com.asana.models.Story;
+import com.asana.models.Tag;
+import com.asana.models.Task;
+import com.asana.models.Team;
+import com.asana.models.User;
+
+import java.util.Map;
+
+/**
+ * This interface represents a client to Asana REST server, with some basic filtering options built in.
+ */
+public interface AsanaClient {
+ /**
+ * Find & retrieve an Asana project by its name. If multiple projects match, returns the first.
+ * If there is no match, then {@link RuntimeException} is thrown. Note that constant ordering
Review Comment:
```suggestion
* If there is no match, then {@link AsanaClientException} is thrown. Note that constant ordering
```
Also in the javadoc of `getTeamByName()` and `getSectionByName()`.
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-services-api-nar/src/main/resources/META-INF/NOTICE:
##########
@@ -0,0 +1,108 @@
+nifi-asana-services-api-nar
+Copyright 2015-2022 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+===========================================
+Apache Software License v2
+===========================================
+
+The following binary components are provided under the Apache Software License v2
+
+ (ASLv2) Apache Commons IO
+ The following NOTICE information applies:
+ Apache Commons IO
+ Copyright 2002-2022 The Apache Software Foundation
Review Comment:
Thanks for updating the `NOTICE` file, overall it looks good!
`Commons IO` is not present in the services-api-nar so it should be removed.
I forgot to mention earlier but the entries need to be added in the assembly nar as well (`nifi-assembly/NOTICE`). This "global" NOTICE file is kind of a union of all entries in the nar.
Could you please copy those items to that NOTICE file which are not present there yet?
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,393 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import static java.lang.String.format;
+import static java.util.Collections.singletonMap;
+import static java.util.stream.Collectors.groupingBy;
+import static java.util.stream.Collectors.joining;
+import static org.apache.commons.collections4.ListUtils.partition;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_EVENTS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_MEMBERS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_STATUS_ATTACHMENTS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_STATUS_UPDATES;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_STORIES;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_TASKS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_TASK_ATTACHMENTS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_TEAM_MEMBERS;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientProviderService;
+import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaObjectState;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+@TriggerSerially
+@PrimaryNodeOnly
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.")
+@Tags({"asana", "source", "ingest"})
+@CapabilityDescription("This processor collects data from Asana")
+public class GetAsanaObject extends AbstractProcessor {
+
+ protected static final String ASANA_GID = "asana.gid";
+ protected static final String ASANA_CLIENT_SERVICE = "asana-controller-service";
+ protected static final String DISTRIBUTED_CACHE_SERVICE = "distributed-cache-service";
+ protected static final String ASANA_OBJECT_TYPE = "asana-object-type";
+ protected static final String ASANA_PROJECT_NAME = "asana-project-name";
+ protected static final String ASANA_SECTION_NAME = "asana-section-name";
+ protected static final String ASANA_TAG_NAME = "asana-tag-name";
+ protected static final String ASANA_TEAM_NAME = "asana-team-name";
+ protected static final String ASANA_OUTPUT_BATCH_SIZE = "asana-output-batch-size";
+ protected static final String REL_NAME_NEW = "new";
+ protected static final String REL_NAME_UPDATED = "updated";
+ protected static final String REL_NAME_REMOVED = "removed";
+
+ protected static final PropertyDescriptor PROP_ASANA_CLIENT_SERVICE = new PropertyDescriptor.Builder()
+ .name(ASANA_CLIENT_SERVICE)
+ .displayName("Asana Client Service")
+ .description("Specify which controller service to use for accessing Asana.")
+ .required(true)
+ .identifiesControllerService(AsanaClientProviderService.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_DISTRIBUTED_CACHE_SERVICE = new Builder()
+ .name(DISTRIBUTED_CACHE_SERVICE)
+ .displayName("Distributed Cache Service")
+ .description("Cache service to store fetched item fingerprints. These, from the last successful query"
+ + " are stored, in order to enable incremental loading and change detection.")
+ .required(true)
+ .identifiesControllerService(DistributedMapCacheClient.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OBJECT_TYPE = new PropertyDescriptor.Builder()
+ .name(ASANA_OBJECT_TYPE)
+ .displayName("Object Type")
+ .description("Specify what kind of objects to be collected from Asana")
+ .required(true)
+ .allowableValues(AsanaObjectType.class)
+ .defaultValue(AV_COLLECT_TASKS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_PROJECT = new PropertyDescriptor.Builder()
+ .name(ASANA_PROJECT_NAME)
+ .displayName("Project Name")
+ .description("Fetch only objects in this project. Case sensitive.")
+ .required(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(
+ PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS.getValue(),
+ AV_COLLECT_TASK_ATTACHMENTS.getValue(),
+ AV_COLLECT_PROJECT_MEMBERS.getValue(),
+ AV_COLLECT_STORIES.getValue(),
+ AV_COLLECT_PROJECT_STATUS_UPDATES.getValue(),
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS.getValue(),
+ AV_COLLECT_PROJECT_EVENTS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_SECTION = new PropertyDescriptor.Builder()
+ .name(ASANA_SECTION_NAME)
+ .displayName("Section Name")
+ .description("Fetch only objects in this section. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS.getValue(),
+ AV_COLLECT_TASK_ATTACHMENTS.getValue(),
+ AV_COLLECT_STORIES.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TAG = new PropertyDescriptor.Builder()
+ .name(ASANA_TAG_NAME)
+ .displayName("Tag")
+ .description("Fetch only objects having this tag. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS.getValue(),
+ AV_COLLECT_TASK_ATTACHMENTS.getValue(),
+ AV_COLLECT_STORIES.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TEAM_NAME = new PropertyDescriptor.Builder()
+ .name(ASANA_TEAM_NAME)
+ .displayName("Team")
+ .description("Team name. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TEAM_MEMBERS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OUTPUT_BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name(ASANA_OUTPUT_BATCH_SIZE)
+ .displayName("Output Batch Size")
+ .description("The number of items batched together in a single Flow File. If set to 1 (default), then each item is"
+ + " transferred in a separate Flow File and each will have an asana.gid attribute, to help identifying"
+ + " the fetched item on the server side, if needed. If the batch size is greater than 1, then the"
+ + " specified amount of items are batched together in a single Flow File as a Json array, and the"
+ + " Flow Files won't have the asana.gid attribute.")
+ .defaultValue("1")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+
+ protected static final List<PropertyDescriptor> DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ PROP_ASANA_CLIENT_SERVICE,
+ PROP_DISTRIBUTED_CACHE_SERVICE,
+ PROP_ASANA_OBJECT_TYPE,
+ PROP_ASANA_PROJECT,
+ PROP_ASANA_SECTION,
+ PROP_ASANA_TEAM_NAME,
+ PROP_ASANA_TAG,
+ PROP_ASANA_OUTPUT_BATCH_SIZE
+ ));
+
+ protected static final Relationship REL_NEW = new Relationship.Builder()
+ .name(REL_NAME_NEW)
+ .description("Newly collected objects are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_UPDATED = new Relationship.Builder()
+ .name(REL_NAME_UPDATED)
+ .description("Objects that have already been collected earlier, but were updated since, are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_REMOVED = new Relationship.Builder()
+ .name(REL_NAME_REMOVED)
+ .description("Notification about deleted objects are routed to this relationship. "
+ + "Flow files will not have any payload. IDs of the resources no longer exist "
+ + "are carried by the asana.gid attribute of the generated FlowFiles.")
+ .build();
+
+ protected static final Set<Relationship> RELATIONSHIPS = Collections.unmodifiableSet(new HashSet<>(Arrays.asList(
+ REL_NEW,
+ REL_UPDATED,
+ REL_REMOVED
+ )));
+
+ private volatile AsanaObjectFetcher objectFetcher;
+ private volatile Integer batchSize;
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return RELATIONSHIPS;
+ }
+
+ @Override
+ public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return DESCRIPTORS;
+ }
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) throws InitializationException {
+ AsanaClientProviderService controllerService = context.getProperty(PROP_ASANA_CLIENT_SERVICE).asControllerService(AsanaClientProviderService.class);
+ AsanaClient client = controllerService.createClient();
+ batchSize = context.getProperty(PROP_ASANA_OUTPUT_BATCH_SIZE).asInteger();
+
+ try {
+ getLogger().debug("Initializing object fetcher...");
+ objectFetcher = createObjectFetcher(context, client);
+ } catch (Exception e) {
+ throw new InitializationException(e);
+ }
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ try {
+ Map<String, String> state = recoverState(context).orElse(Collections.emptyMap());
+ getLogger().debug("Attempting to load state: {}", state);
+ objectFetcher.loadState(state);
+ } catch (Exception e) {
+ getLogger().info("Failed to recover state. Falling back to clean start.");
+ objectFetcher.clearState();
+ }
+ getLogger().debug("Initial state: {}", objectFetcher.saveState());
Review Comment:
I would put it into `if (getLogger().isDebugEnabled()) { ... }` in order to prevent the unnecessary map conversions in `objectFetcher.saveState()`.
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,393 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import static java.lang.String.format;
+import static java.util.Collections.singletonMap;
+import static java.util.stream.Collectors.groupingBy;
+import static java.util.stream.Collectors.joining;
+import static org.apache.commons.collections4.ListUtils.partition;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_EVENTS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_MEMBERS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_STATUS_ATTACHMENTS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_STATUS_UPDATES;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_STORIES;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_TASKS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_TASK_ATTACHMENTS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_TEAM_MEMBERS;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientProviderService;
+import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaObjectState;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+@TriggerSerially
+@PrimaryNodeOnly
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.")
+@Tags({"asana", "source", "ingest"})
+@CapabilityDescription("This processor collects data from Asana")
+public class GetAsanaObject extends AbstractProcessor {
+
+ protected static final String ASANA_GID = "asana.gid";
+ protected static final String ASANA_CLIENT_SERVICE = "asana-controller-service";
+ protected static final String DISTRIBUTED_CACHE_SERVICE = "distributed-cache-service";
+ protected static final String ASANA_OBJECT_TYPE = "asana-object-type";
+ protected static final String ASANA_PROJECT_NAME = "asana-project-name";
+ protected static final String ASANA_SECTION_NAME = "asana-section-name";
+ protected static final String ASANA_TAG_NAME = "asana-tag-name";
+ protected static final String ASANA_TEAM_NAME = "asana-team-name";
+ protected static final String ASANA_OUTPUT_BATCH_SIZE = "asana-output-batch-size";
+ protected static final String REL_NAME_NEW = "new";
+ protected static final String REL_NAME_UPDATED = "updated";
+ protected static final String REL_NAME_REMOVED = "removed";
+
+ protected static final PropertyDescriptor PROP_ASANA_CLIENT_SERVICE = new PropertyDescriptor.Builder()
+ .name(ASANA_CLIENT_SERVICE)
+ .displayName("Asana Client Service")
+ .description("Specify which controller service to use for accessing Asana.")
+ .required(true)
+ .identifiesControllerService(AsanaClientProviderService.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_DISTRIBUTED_CACHE_SERVICE = new Builder()
+ .name(DISTRIBUTED_CACHE_SERVICE)
+ .displayName("Distributed Cache Service")
+ .description("Cache service to store fetched item fingerprints. These, from the last successful query"
+ + " are stored, in order to enable incremental loading and change detection.")
+ .required(true)
+ .identifiesControllerService(DistributedMapCacheClient.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OBJECT_TYPE = new PropertyDescriptor.Builder()
+ .name(ASANA_OBJECT_TYPE)
+ .displayName("Object Type")
+ .description("Specify what kind of objects to be collected from Asana")
+ .required(true)
+ .allowableValues(AsanaObjectType.class)
+ .defaultValue(AV_COLLECT_TASKS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_PROJECT = new PropertyDescriptor.Builder()
+ .name(ASANA_PROJECT_NAME)
+ .displayName("Project Name")
+ .description("Fetch only objects in this project. Case sensitive.")
+ .required(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(
+ PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS.getValue(),
+ AV_COLLECT_TASK_ATTACHMENTS.getValue(),
+ AV_COLLECT_PROJECT_MEMBERS.getValue(),
+ AV_COLLECT_STORIES.getValue(),
+ AV_COLLECT_PROJECT_STATUS_UPDATES.getValue(),
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS.getValue(),
+ AV_COLLECT_PROJECT_EVENTS.getValue())
Review Comment:
Since [NIFI-10775](https://issues.apache.org/jira/browse/NIFI-10775) (merged recently), `getValue()` is not needed. `DescribedValue` can be used directly.
```suggestion
.dependsOn(
PROP_ASANA_OBJECT_TYPE,
AV_COLLECT_TASKS,
AV_COLLECT_TASK_ATTACHMENTS,
AV_COLLECT_PROJECT_MEMBERS,
AV_COLLECT_STORIES,
AV_COLLECT_PROJECT_STATUS_UPDATES,
AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
AV_COLLECT_PROJECT_EVENTS)
```
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,393 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import static java.lang.String.format;
+import static java.util.Collections.singletonMap;
+import static java.util.stream.Collectors.groupingBy;
+import static java.util.stream.Collectors.joining;
+import static org.apache.commons.collections4.ListUtils.partition;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_EVENTS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_MEMBERS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_STATUS_ATTACHMENTS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_STATUS_UPDATES;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_STORIES;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_TASKS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_TASK_ATTACHMENTS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_TEAM_MEMBERS;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientProviderService;
+import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaObjectState;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+@TriggerSerially
+@PrimaryNodeOnly
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.")
+@Tags({"asana", "source", "ingest"})
+@CapabilityDescription("This processor collects data from Asana")
+public class GetAsanaObject extends AbstractProcessor {
+
+ protected static final String ASANA_GID = "asana.gid";
+ protected static final String ASANA_CLIENT_SERVICE = "asana-controller-service";
+ protected static final String DISTRIBUTED_CACHE_SERVICE = "distributed-cache-service";
+ protected static final String ASANA_OBJECT_TYPE = "asana-object-type";
+ protected static final String ASANA_PROJECT_NAME = "asana-project-name";
+ protected static final String ASANA_SECTION_NAME = "asana-section-name";
+ protected static final String ASANA_TAG_NAME = "asana-tag-name";
+ protected static final String ASANA_TEAM_NAME = "asana-team-name";
+ protected static final String ASANA_OUTPUT_BATCH_SIZE = "asana-output-batch-size";
+ protected static final String REL_NAME_NEW = "new";
+ protected static final String REL_NAME_UPDATED = "updated";
+ protected static final String REL_NAME_REMOVED = "removed";
+
+ protected static final PropertyDescriptor PROP_ASANA_CLIENT_SERVICE = new PropertyDescriptor.Builder()
+ .name(ASANA_CLIENT_SERVICE)
+ .displayName("Asana Client Service")
+ .description("Specify which controller service to use for accessing Asana.")
+ .required(true)
+ .identifiesControllerService(AsanaClientProviderService.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_DISTRIBUTED_CACHE_SERVICE = new Builder()
+ .name(DISTRIBUTED_CACHE_SERVICE)
+ .displayName("Distributed Cache Service")
+ .description("Cache service to store fetched item fingerprints. These, from the last successful query"
+ + " are stored, in order to enable incremental loading and change detection.")
+ .required(true)
+ .identifiesControllerService(DistributedMapCacheClient.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OBJECT_TYPE = new PropertyDescriptor.Builder()
+ .name(ASANA_OBJECT_TYPE)
+ .displayName("Object Type")
+ .description("Specify what kind of objects to be collected from Asana")
+ .required(true)
+ .allowableValues(AsanaObjectType.class)
+ .defaultValue(AV_COLLECT_TASKS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_PROJECT = new PropertyDescriptor.Builder()
+ .name(ASANA_PROJECT_NAME)
+ .displayName("Project Name")
+ .description("Fetch only objects in this project. Case sensitive.")
+ .required(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(
+ PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS.getValue(),
+ AV_COLLECT_TASK_ATTACHMENTS.getValue(),
+ AV_COLLECT_PROJECT_MEMBERS.getValue(),
+ AV_COLLECT_STORIES.getValue(),
+ AV_COLLECT_PROJECT_STATUS_UPDATES.getValue(),
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS.getValue(),
+ AV_COLLECT_PROJECT_EVENTS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_SECTION = new PropertyDescriptor.Builder()
+ .name(ASANA_SECTION_NAME)
+ .displayName("Section Name")
+ .description("Fetch only objects in this section. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS.getValue(),
+ AV_COLLECT_TASK_ATTACHMENTS.getValue(),
+ AV_COLLECT_STORIES.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TAG = new PropertyDescriptor.Builder()
+ .name(ASANA_TAG_NAME)
+ .displayName("Tag")
+ .description("Fetch only objects having this tag. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS.getValue(),
+ AV_COLLECT_TASK_ATTACHMENTS.getValue(),
+ AV_COLLECT_STORIES.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TEAM_NAME = new PropertyDescriptor.Builder()
+ .name(ASANA_TEAM_NAME)
+ .displayName("Team")
+ .description("Team name. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TEAM_MEMBERS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OUTPUT_BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name(ASANA_OUTPUT_BATCH_SIZE)
+ .displayName("Output Batch Size")
+ .description("The number of items batched together in a single Flow File. If set to 1 (default), then each item is"
+ + " transferred in a separate Flow File and each will have an asana.gid attribute, to help identifying"
+ + " the fetched item on the server side, if needed. If the batch size is greater than 1, then the"
+ + " specified amount of items are batched together in a single Flow File as a Json array, and the"
+ + " Flow Files won't have the asana.gid attribute.")
+ .defaultValue("1")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+
+ protected static final List<PropertyDescriptor> DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ PROP_ASANA_CLIENT_SERVICE,
+ PROP_DISTRIBUTED_CACHE_SERVICE,
+ PROP_ASANA_OBJECT_TYPE,
+ PROP_ASANA_PROJECT,
+ PROP_ASANA_SECTION,
+ PROP_ASANA_TEAM_NAME,
+ PROP_ASANA_TAG,
+ PROP_ASANA_OUTPUT_BATCH_SIZE
+ ));
+
+ protected static final Relationship REL_NEW = new Relationship.Builder()
+ .name(REL_NAME_NEW)
+ .description("Newly collected objects are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_UPDATED = new Relationship.Builder()
+ .name(REL_NAME_UPDATED)
+ .description("Objects that have already been collected earlier, but were updated since, are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_REMOVED = new Relationship.Builder()
+ .name(REL_NAME_REMOVED)
+ .description("Notification about deleted objects are routed to this relationship. "
+ + "Flow files will not have any payload. IDs of the resources no longer exist "
+ + "are carried by the asana.gid attribute of the generated FlowFiles.")
+ .build();
+
+ protected static final Set<Relationship> RELATIONSHIPS = Collections.unmodifiableSet(new HashSet<>(Arrays.asList(
+ REL_NEW,
+ REL_UPDATED,
+ REL_REMOVED
+ )));
+
+ private volatile AsanaObjectFetcher objectFetcher;
+ private volatile Integer batchSize;
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return RELATIONSHIPS;
+ }
+
+ @Override
+ public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return DESCRIPTORS;
+ }
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) throws InitializationException {
+ AsanaClientProviderService controllerService = context.getProperty(PROP_ASANA_CLIENT_SERVICE).asControllerService(AsanaClientProviderService.class);
+ AsanaClient client = controllerService.createClient();
+ batchSize = context.getProperty(PROP_ASANA_OUTPUT_BATCH_SIZE).asInteger();
+
+ try {
+ getLogger().debug("Initializing object fetcher...");
+ objectFetcher = createObjectFetcher(context, client);
+ } catch (Exception e) {
+ throw new InitializationException(e);
+ }
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ try {
+ Map<String, String> state = recoverState(context).orElse(Collections.emptyMap());
+ getLogger().debug("Attempting to load state: {}", state);
+ objectFetcher.loadState(state);
+ } catch (Exception e) {
+ getLogger().info("Failed to recover state. Falling back to clean start.");
+ objectFetcher.clearState();
+ }
+ getLogger().debug("Initial state: {}", objectFetcher.saveState());
+
+ Collection<FlowFile> newItems = new ArrayList<>();
+ Collection<FlowFile> updatedItems = new ArrayList<>();
+ Collection<FlowFile> removedItems = new ArrayList<>();
+ Map<AsanaObjectState, Collection<FlowFile>> flowFiles = new HashMap<>();
+ flowFiles.put(AsanaObjectState.NEW, newItems);
+ flowFiles.put(AsanaObjectState.UPDATED, updatedItems);
+ flowFiles.put(AsanaObjectState.REMOVED, removedItems);
+
+ List<AsanaObject> allObjects = new ArrayList<>();
+
+ AsanaObject nextObject;
+ while ((nextObject = objectFetcher.fetchNext()) != null) {
+ allObjects.add(nextObject);
+ }
+
+ Map<AsanaObjectState, List<AsanaObject>> allObjectsByState = allObjects.stream()
+ .collect(groupingBy(AsanaObject::getState));
+
+ if (batchSize == 1) {
+ allObjectsByState
+ .forEach((asanaObjectState, asanaObjects) -> asanaObjects.forEach(
+ asanaObject -> {
+ final Map<String, String> attributes = new HashMap<>(2);
+ attributes.put(CoreAttributes.MIME_TYPE.key(), ContentType.APPLICATION_JSON.getMimeType());
+ attributes.put(ASANA_GID, asanaObject.getGid());
+ FlowFile flowFile = createFlowFileWithStringPayload(session, asanaObject.getContent());
+ flowFile = session.putAllAttributes(flowFile, attributes);
+ flowFiles.get(asanaObject.getState()).add(flowFile);
+ }
+ ));
+ } else {
+ allObjectsByState
+ .forEach((asanaObjectState, asanaObjects) -> partition(asanaObjects, batchSize).forEach(
+ asanaObjectsInPartition -> {
+ FlowFile flowFile = createFlowFileWithStringPayload(session, format("[%s]",
+ asanaObjectsInPartition.stream().map(AsanaObject::getContent)
+ .collect(joining(","))));
+ flowFile = session.putAllAttributes(flowFile,
+ singletonMap(CoreAttributes.MIME_TYPE.key(),
+ ContentType.APPLICATION_JSON.getMimeType()));
+ flowFiles.get(asanaObjectState).add(flowFile);
+ }
+ ));
+ }
+
+ if (flowFiles.values().stream().allMatch(Collection::isEmpty)) {
+ context.yield();
+ getLogger().debug("Yielding, as there are no new FlowFiles.");
+ } else {
+ session.transfer(newItems, REL_NEW);
+ session.transfer(updatedItems, REL_UPDATED);
+ session.transfer(removedItems, REL_REMOVED);
+ }
+
+ Map<String, String> state = objectFetcher.saveState();
+ persistState(state, context);
+
+ getLogger().debug(
+ "New state after transferring {} new, {} updated, and {} removed items: {}",
+ newItems.size(), updatedItems.size(), removedItems.size(), state);
+
+ session.commitAsync();
Review Comment:
Unlike session state, the MapCache operations are not part of the session's transaction. For this reason, the session should be committed first, in order to guarantee at-least-once delivery.
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,393 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import static java.lang.String.format;
+import static java.util.Collections.singletonMap;
+import static java.util.stream.Collectors.groupingBy;
+import static java.util.stream.Collectors.joining;
+import static org.apache.commons.collections4.ListUtils.partition;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_EVENTS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_MEMBERS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_STATUS_ATTACHMENTS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_STATUS_UPDATES;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_STORIES;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_TASKS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_TASK_ATTACHMENTS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_TEAM_MEMBERS;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientProviderService;
+import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaObjectState;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+@TriggerSerially
+@PrimaryNodeOnly
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.")
+@Tags({"asana", "source", "ingest"})
+@CapabilityDescription("This processor collects data from Asana")
+public class GetAsanaObject extends AbstractProcessor {
+
+ protected static final String ASANA_GID = "asana.gid";
+ protected static final String ASANA_CLIENT_SERVICE = "asana-controller-service";
+ protected static final String DISTRIBUTED_CACHE_SERVICE = "distributed-cache-service";
+ protected static final String ASANA_OBJECT_TYPE = "asana-object-type";
+ protected static final String ASANA_PROJECT_NAME = "asana-project-name";
+ protected static final String ASANA_SECTION_NAME = "asana-section-name";
+ protected static final String ASANA_TAG_NAME = "asana-tag-name";
+ protected static final String ASANA_TEAM_NAME = "asana-team-name";
+ protected static final String ASANA_OUTPUT_BATCH_SIZE = "asana-output-batch-size";
+ protected static final String REL_NAME_NEW = "new";
+ protected static final String REL_NAME_UPDATED = "updated";
+ protected static final String REL_NAME_REMOVED = "removed";
+
+ protected static final PropertyDescriptor PROP_ASANA_CLIENT_SERVICE = new PropertyDescriptor.Builder()
+ .name(ASANA_CLIENT_SERVICE)
+ .displayName("Asana Client Service")
+ .description("Specify which controller service to use for accessing Asana.")
+ .required(true)
+ .identifiesControllerService(AsanaClientProviderService.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_DISTRIBUTED_CACHE_SERVICE = new Builder()
+ .name(DISTRIBUTED_CACHE_SERVICE)
+ .displayName("Distributed Cache Service")
+ .description("Cache service to store fetched item fingerprints. These, from the last successful query"
+ + " are stored, in order to enable incremental loading and change detection.")
+ .required(true)
+ .identifiesControllerService(DistributedMapCacheClient.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OBJECT_TYPE = new PropertyDescriptor.Builder()
+ .name(ASANA_OBJECT_TYPE)
+ .displayName("Object Type")
+ .description("Specify what kind of objects to be collected from Asana")
+ .required(true)
+ .allowableValues(AsanaObjectType.class)
+ .defaultValue(AV_COLLECT_TASKS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_PROJECT = new PropertyDescriptor.Builder()
+ .name(ASANA_PROJECT_NAME)
+ .displayName("Project Name")
+ .description("Fetch only objects in this project. Case sensitive.")
+ .required(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(
+ PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS.getValue(),
+ AV_COLLECT_TASK_ATTACHMENTS.getValue(),
+ AV_COLLECT_PROJECT_MEMBERS.getValue(),
+ AV_COLLECT_STORIES.getValue(),
+ AV_COLLECT_PROJECT_STATUS_UPDATES.getValue(),
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS.getValue(),
+ AV_COLLECT_PROJECT_EVENTS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_SECTION = new PropertyDescriptor.Builder()
+ .name(ASANA_SECTION_NAME)
+ .displayName("Section Name")
+ .description("Fetch only objects in this section. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS.getValue(),
+ AV_COLLECT_TASK_ATTACHMENTS.getValue(),
+ AV_COLLECT_STORIES.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TAG = new PropertyDescriptor.Builder()
+ .name(ASANA_TAG_NAME)
+ .displayName("Tag")
+ .description("Fetch only objects having this tag. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS.getValue(),
+ AV_COLLECT_TASK_ATTACHMENTS.getValue(),
+ AV_COLLECT_STORIES.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TEAM_NAME = new PropertyDescriptor.Builder()
+ .name(ASANA_TEAM_NAME)
+ .displayName("Team")
+ .description("Team name. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TEAM_MEMBERS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OUTPUT_BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name(ASANA_OUTPUT_BATCH_SIZE)
+ .displayName("Output Batch Size")
+ .description("The number of items batched together in a single Flow File. If set to 1 (default), then each item is"
+ + " transferred in a separate Flow File and each will have an asana.gid attribute, to help identifying"
+ + " the fetched item on the server side, if needed. If the batch size is greater than 1, then the"
+ + " specified amount of items are batched together in a single Flow File as a Json array, and the"
+ + " Flow Files won't have the asana.gid attribute.")
+ .defaultValue("1")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+
+ protected static final List<PropertyDescriptor> DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ PROP_ASANA_CLIENT_SERVICE,
+ PROP_DISTRIBUTED_CACHE_SERVICE,
+ PROP_ASANA_OBJECT_TYPE,
+ PROP_ASANA_PROJECT,
+ PROP_ASANA_SECTION,
+ PROP_ASANA_TEAM_NAME,
+ PROP_ASANA_TAG,
+ PROP_ASANA_OUTPUT_BATCH_SIZE
+ ));
+
+ protected static final Relationship REL_NEW = new Relationship.Builder()
+ .name(REL_NAME_NEW)
+ .description("Newly collected objects are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_UPDATED = new Relationship.Builder()
+ .name(REL_NAME_UPDATED)
+ .description("Objects that have already been collected earlier, but were updated since, are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_REMOVED = new Relationship.Builder()
+ .name(REL_NAME_REMOVED)
+ .description("Notification about deleted objects are routed to this relationship. "
+ + "Flow files will not have any payload. IDs of the resources no longer exist "
+ + "are carried by the asana.gid attribute of the generated FlowFiles.")
+ .build();
+
+ protected static final Set<Relationship> RELATIONSHIPS = Collections.unmodifiableSet(new HashSet<>(Arrays.asList(
+ REL_NEW,
+ REL_UPDATED,
+ REL_REMOVED
+ )));
+
+ private volatile AsanaObjectFetcher objectFetcher;
+ private volatile Integer batchSize;
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return RELATIONSHIPS;
+ }
+
+ @Override
+ public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return DESCRIPTORS;
+ }
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) throws InitializationException {
+ AsanaClientProviderService controllerService = context.getProperty(PROP_ASANA_CLIENT_SERVICE).asControllerService(AsanaClientProviderService.class);
+ AsanaClient client = controllerService.createClient();
+ batchSize = context.getProperty(PROP_ASANA_OUTPUT_BATCH_SIZE).asInteger();
+
+ try {
+ getLogger().debug("Initializing object fetcher...");
+ objectFetcher = createObjectFetcher(context, client);
+ } catch (Exception e) {
+ throw new InitializationException(e);
+ }
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ try {
+ Map<String, String> state = recoverState(context).orElse(Collections.emptyMap());
+ getLogger().debug("Attempting to load state: {}", state);
+ objectFetcher.loadState(state);
+ } catch (Exception e) {
+ getLogger().info("Failed to recover state. Falling back to clean start.");
+ objectFetcher.clearState();
+ }
+ getLogger().debug("Initial state: {}", objectFetcher.saveState());
+
+ Collection<FlowFile> newItems = new ArrayList<>();
+ Collection<FlowFile> updatedItems = new ArrayList<>();
+ Collection<FlowFile> removedItems = new ArrayList<>();
+ Map<AsanaObjectState, Collection<FlowFile>> flowFiles = new HashMap<>();
+ flowFiles.put(AsanaObjectState.NEW, newItems);
+ flowFiles.put(AsanaObjectState.UPDATED, updatedItems);
+ flowFiles.put(AsanaObjectState.REMOVED, removedItems);
+
+ List<AsanaObject> allObjects = new ArrayList<>();
+
+ AsanaObject nextObject;
+ while ((nextObject = objectFetcher.fetchNext()) != null) {
+ allObjects.add(nextObject);
+ }
+
+ Map<AsanaObjectState, List<AsanaObject>> allObjectsByState = allObjects.stream()
+ .collect(groupingBy(AsanaObject::getState));
+
+ if (batchSize == 1) {
+ allObjectsByState
+ .forEach((asanaObjectState, asanaObjects) -> asanaObjects.forEach(
+ asanaObject -> {
+ final Map<String, String> attributes = new HashMap<>(2);
+ attributes.put(CoreAttributes.MIME_TYPE.key(), ContentType.APPLICATION_JSON.getMimeType());
+ attributes.put(ASANA_GID, asanaObject.getGid());
+ FlowFile flowFile = createFlowFileWithStringPayload(session, asanaObject.getContent());
+ flowFile = session.putAllAttributes(flowFile, attributes);
+ flowFiles.get(asanaObject.getState()).add(flowFile);
+ }
+ ));
+ } else {
+ allObjectsByState
+ .forEach((asanaObjectState, asanaObjects) -> partition(asanaObjects, batchSize).forEach(
+ asanaObjectsInPartition -> {
+ FlowFile flowFile = createFlowFileWithStringPayload(session, format("[%s]",
+ asanaObjectsInPartition.stream().map(AsanaObject::getContent)
+ .collect(joining(","))));
+ flowFile = session.putAllAttributes(flowFile,
+ singletonMap(CoreAttributes.MIME_TYPE.key(),
+ ContentType.APPLICATION_JSON.getMimeType()));
+ flowFiles.get(asanaObjectState).add(flowFile);
+ }
+ ));
+ }
+
+ if (flowFiles.values().stream().allMatch(Collection::isEmpty)) {
+ context.yield();
+ getLogger().debug("Yielding, as there are no new FlowFiles.");
+ } else {
+ session.transfer(newItems, REL_NEW);
+ session.transfer(updatedItems, REL_UPDATED);
+ session.transfer(removedItems, REL_REMOVED);
+ }
+
+ Map<String, String> state = objectFetcher.saveState();
+ persistState(state, context);
+
+ getLogger().debug(
+ "New state after transferring {} new, {} updated, and {} removed items: {}",
+ newItems.size(), updatedItems.size(), removedItems.size(), state);
+
+ session.commitAsync();
+ }
+
+ protected AsanaObjectFetcher createObjectFetcher(final ProcessContext context, AsanaClient client) {
+ final String objectType = context.getProperty(PROP_ASANA_OBJECT_TYPE).getValue();
+ final String projectName = context.getProperty(PROP_ASANA_PROJECT).getValue();
+ final String sectionName = context.getProperty(PROP_ASANA_SECTION).getValue();
+ final String teamName = context.getProperty(PROP_ASANA_TEAM_NAME).getValue();
+ final String tagName = context.getProperty(PROP_ASANA_TAG).getValue();
+
+ switch (AsanaObjectType.fromValue(objectType)) {
+ case AV_COLLECT_TASKS:
+ return new AsanaTaskFetcher(client, projectName, sectionName, tagName);
+ case AV_COLLECT_PROJECTS:
+ return new AsanaProjectFetcher(client);
+ case AV_COLLECT_PROJECT_EVENTS:
+ return new AsanaProjectEventFetcher(client, projectName);
+ case AV_COLLECT_PROJECT_MEMBERS:
+ return new AsanaProjectMembershipFetcher(client, projectName);
+ case AV_COLLECT_PROJECT_STATUS_ATTACHMENTS:
+ return new AsanaProjectStatusAttachmentFetcher(client, projectName);
+ case AV_COLLECT_PROJECT_STATUS_UPDATES:
+ return new AsanaProjectStatusFetcher(client, projectName);
+ case AV_COLLECT_STORIES:
+ return new AsanaStoryFetcher(client, projectName, sectionName, tagName);
+ case AV_COLLECT_TAGS:
+ return new AsanaTagFetcher(client);
+ case AV_COLLECT_TASK_ATTACHMENTS:
+ return new AsanaTaskAttachmentFetcher(client, projectName, sectionName, tagName);
+ case AV_COLLECT_TEAMS:
+ return new AsanaTeamFetcher(client);
+ case AV_COLLECT_TEAM_MEMBERS:
+ return new AsanaTeamMemberFetcher(client, teamName);
+ case AV_COLLECT_USERS:
+ return new AsanaUserFetcher(client);
+ }
+
+ throw new ProcessException("Cannot fetch objects of type: " + objectType);
+ }
+
+ private Optional<Map<String, String>> recoverState(final ProcessContext context) {
+ final DistributedMapCacheClient client = getDistributedMapCacheClient(context);
+ try {
+ final Map<String, String> result = client.get(getIdentifier(), new GenericObjectSerDe<>(), new GenericObjectSerDe<>());
Review Comment:
The same instance could be used for the key/value serializers. Also, it could be a singleton stored in a static field.
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/test/java/org/apache/nifi/processors/asana/AsanaTaskFetcherTest.java:
##########
@@ -0,0 +1,436 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import com.asana.models.Project;
+import com.asana.models.Section;
+import com.asana.models.Tag;
+import com.asana.models.Task;
+import com.google.api.client.util.DateTime;
+import org.apache.groovy.util.Maps;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaObjectState;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.mockito.Mock;
+import org.mockito.junit.jupiter.MockitoExtension;
+import org.mockito.junit.jupiter.MockitoSettings;
+import org.mockito.quality.Strictness;
+
+import static java.util.Collections.emptyMap;
+import static java.util.Collections.singletonMap;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.atLeastOnce;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.verifyNoMoreInteractions;
+import static org.mockito.Mockito.when;
+
+@ExtendWith(MockitoExtension.class)
+@MockitoSettings(strictness = Strictness.LENIENT)
+public class AsanaTaskFetcherTest {
+
+ @Mock
+ private AsanaClient client;
+ private Project project;
+ private Section section;
+ private Tag tag;
+
+ @BeforeEach
+ public void init() {
+ project = new Project();
+ project.gid = "123";
+ project.modifiedAt = new DateTime(123456789);
+ project.name = "My Project";
+
+ when(client.getProjectByName(project.name)).thenReturn(project);
+
+ section = new Section();
+ section.gid = "456";
+ section.project = project;
+ section.name = "Some section";
+ section.createdAt = new DateTime(123456789);
+
+ when(client.getSections(project)).thenReturn(singletonMap(section.gid, section));
+ when(client.getSectionByName(project, section.name)).thenReturn(section);
+
+ tag = new Tag();
+ tag.gid = "9876";
+ tag.name = "Foo";
+ tag.createdAt = new DateTime(123456789);
+
+ when(client.getTags()).thenReturn(singletonMap(tag.gid, tag));
+ }
+
+ @Test
+ public void testNoObjectsFetchedWhenNoTasksReturned() {
+ when(client.getTasks(any(Project.class))).thenReturn(emptyMap());
+
+ final AsanaObjectFetcher fetcher = new AsanaTaskFetcher(client, project.name, null, null);
+ assertNull(fetcher.fetchNext());
+
+ verify(client, atLeastOnce()).getProjectByName(project.name);
+ verify(client, times(1)).getTasks(project);
+ verifyNoMoreInteractions(client);
+ }
+
+ @Test
+ public void testNoObjectsFetchedWhenNoTasksReturnedBySection() {
+ when(client.getTasks(any(Section.class))).thenReturn(emptyMap());
+
+ final AsanaObjectFetcher fetcher = new AsanaTaskFetcher(client, project.name, section.name, null);
+ assertNull(fetcher.fetchNext());
+
+ verify(client, atLeastOnce()).getProjectByName(project.name);
+ verify(client, atLeastOnce()).getSectionByName(project, section.name);
+ verify(client, times(1)).getTasks(section);
+ verifyNoMoreInteractions(client);
+ }
+
+ @Test
+ public void testNoObjectsFetchedWhenNoTasksReturnedByTag() {
+ when(client.getTasks(any(Project.class))).thenReturn(emptyMap());
+
+ final AsanaObjectFetcher fetcher = new AsanaTaskFetcher(client, project.name, null, tag.name);
+ assertNull(fetcher.fetchNext());
+
+ verify(client, atLeastOnce()).getProjectByName(project.name);
+ verify(client, atLeastOnce()).getTags();
+ verify(client, times(1)).getTasks(project);
+ verify(client, times(1)).getTasks(tag);
+ verifyNoMoreInteractions(client);
+ }
+
+ @Test
+ public void testSingleTaskFetched() {
+ final Task task = new Task();
+ task.gid = "1234";
+ task.name = "My first task";
+ task.modifiedAt = new DateTime(123456789);
+
+ when(client.getTasks(any(Project.class))).thenReturn(singletonMap(task.gid, task));
+
+ final AsanaObjectFetcher fetcher = new AsanaTaskFetcher(client, project.name, null, null);
+ final AsanaObject object = fetcher.fetchNext();
+
+ assertEquals(AsanaObjectState.NEW, object.getState());
+ assertEquals(task.gid, object.getGid());
+
+ verify(client, atLeastOnce()).getProjectByName(project.name);
+ verify(client, times(1)).getTasks(project);
+ verifyNoMoreInteractions(client);
+ }
+
+ @Test
+ public void testSingleTaskFetchedBySection() {
+ final Task task = new Task();
+ task.gid = "1234";
+ task.name = "My first task";
+ task.modifiedAt = new DateTime(123456789);
+
+ when(client.getTasks(any(Section.class))).thenReturn(singletonMap(task.gid, task));
+
+ final AsanaObjectFetcher fetcher = new AsanaTaskFetcher(client, project.name, section.name, null);
+ final AsanaObject object = fetcher.fetchNext();
+
+ assertEquals(AsanaObjectState.NEW, object.getState());
+ assertEquals(task.gid, object.getGid());
+
+ verify(client, atLeastOnce()).getProjectByName(project.name);
+ verify(client, atLeastOnce()).getSectionByName(project, section.name);
+ verify(client, times(1)).getTasks(section);
+ verifyNoMoreInteractions(client);
+ }
+
+ @Test
+ public void testSingleTaskFetchedByTag() {
+ final Task task = new Task();
+ task.gid = "1234";
+ task.name = "My first task";
+ task.modifiedAt = new DateTime(123456789);
+
+ when(client.getTasks(any(Project.class))).thenReturn(singletonMap(task.gid, task));
+ when(client.getTasks(any(Tag.class))).thenReturn(singletonMap(task.gid, task));
+
+ final AsanaObjectFetcher fetcher = new AsanaTaskFetcher(client, project.name, null, tag.name);
+ final AsanaObject object = fetcher.fetchNext();
+
+ assertEquals(AsanaObjectState.NEW, object.getState());
+ assertEquals(task.gid, object.getGid());
+
+ verify(client, atLeastOnce()).getProjectByName(project.name);
+ verify(client, atLeastOnce()).getTags();
+ verify(client, times(1)).getTasks(project);
+ verify(client, times(1)).getTasks(tag);
+ verifyNoMoreInteractions(client);
+ }
+
+ @Test
+ public void testNoTaskFetchedByNonMatchingTag() {
+ final Task task1 = new Task();
+ task1.gid = "1234";
+ task1.name = "My first task";
+ task1.modifiedAt = new DateTime(123456789);
+
+ final Task task2 = new Task();
+ task2.gid = "5678";
+ task2.name = "My other task";
+ task2.modifiedAt = new DateTime(123456789);
+
+ when(client.getTasks(any(Project.class))).thenReturn(singletonMap(task1.gid, task1));
+ when(client.getTasks(any(Tag.class))).thenReturn(singletonMap(task2.gid, task2));
+
+ final AsanaObjectFetcher fetcher = new AsanaTaskFetcher(client, project.name, null, tag.name);
+ assertNull(fetcher.fetchNext());
+
+ verify(client, atLeastOnce()).getProjectByName(project.name);
+ verify(client, atLeastOnce()).getTags();
+ verify(client, times(1)).getTasks(project);
+ verify(client, times(1)).getTasks(tag);
+ verifyNoMoreInteractions(client);
+ }
+
+ @Test
+ public void testTaskRemovedFromSection() {
+ final Task task = new Task();
+ task.gid = "1234";
+ task.name = "My first task";
+ task.modifiedAt = new DateTime(123456789);
+
+ when(client.getTasks(any(Section.class))).thenReturn(singletonMap(task.gid, task));
+
+ final AsanaObjectFetcher fetcher = new AsanaTaskFetcher(client, project.name, section.name, null);
+ assertNotNull(fetcher.fetchNext());
+
+ when(client.getTasks(any(Section.class))).thenReturn(emptyMap());
+
+ final AsanaObject object = fetcher.fetchNext();
+ assertEquals(AsanaObjectState.REMOVED, object.getState());
+ assertEquals(task.gid, object.getGid());
+
+ verify(client, atLeastOnce()).getProjectByName(project.name);
+ verify(client, atLeastOnce()).getSectionByName(project, section.name);
+ verify(client, times(2)).getTasks(section);
+ verifyNoMoreInteractions(client);
+ }
+
+ @Test
+ public void testTaskUntagged() {
+ final Task task = new Task();
+ task.gid = "1234";
+ task.name = "My first task";
+ task.modifiedAt = new DateTime(123456789);
+
+ when(client.getTasks(any(Project.class))).thenReturn(singletonMap(task.gid, task));
+ when(client.getTasks(any(Tag.class))).thenReturn(singletonMap(task.gid, task));
+
+ final AsanaObjectFetcher fetcher = new AsanaTaskFetcher(client, project.name, null, tag.name);
+ assertNotNull(fetcher.fetchNext());
+
+ when(client.getTasks(any(Tag.class))).thenReturn(emptyMap());
+
+ final AsanaObject object = fetcher.fetchNext();
+ assertEquals(AsanaObjectState.REMOVED, object.getState());
+ assertEquals(task.gid, object.getGid());
+
+ verify(client, atLeastOnce()).getProjectByName(project.name);
+ verify(client, atLeastOnce()).getTags();
+ verify(client, times(2)).getTasks(project);
+ verify(client, times(2)).getTasks(tag);
+ verifyNoMoreInteractions(client);
+ }
+
+ @Test
+ public void testCollectMultipleTasksWithSameTagAndFilterOutDuplicates() {
+ final Tag anotherTagWithSameName = new Tag();
+ anotherTagWithSameName.gid = "555";
+ anotherTagWithSameName.name = tag.name;
+
+ when(client.getTags()).thenReturn(Maps.of(tag.gid, tag, anotherTagWithSameName.gid, anotherTagWithSameName));
+
+ final Task task1 = new Task();
+ task1.gid = "1234";
+ task1.name = "My first task";
+ task1.modifiedAt = new DateTime(123456789);
+
+ final Task task2 = new Task();
+ task2.gid = "1212";
+ task2.name = "My other task";
+ task2.modifiedAt = new DateTime(234567891);
+
+ final Task task3 = new Task();
+ task3.gid = "333";
+ task3.name = "My third task";
+ task3.modifiedAt = new DateTime(345678912);
+
+ final Task task4 = new Task();
+ task4.gid = "444";
+ task4.name = "A task without tag";
+ task4.modifiedAt = new DateTime(456789123);
+
+ when(client.getTasks(any(Project.class))).thenReturn(Maps.of(task1.gid, task1, task2.gid, task2, task3.gid, task3, task4.gid, task4));
+ when(client.getTasks(tag)).thenReturn(singletonMap(task1.gid, task1));
+ when(client.getTasks(anotherTagWithSameName)).thenReturn(Maps.of(task1.gid, task1, task2.gid, task2, task3.gid, task3));
+
+ final AsanaObjectFetcher fetcher = new AsanaTaskFetcher(client, project.name, null, tag.name);
+
+ final AsanaObject object1 = fetcher.fetchNext();
+ assertEquals(AsanaObjectState.NEW, object1.getState());
+
+ final AsanaObject object2 = fetcher.fetchNext();
+ assertEquals(AsanaObjectState.NEW, object2.getState());
+ assertNotEquals(object1, object2);
+
+ final AsanaObject object3 = fetcher.fetchNext();
+ assertEquals(AsanaObjectState.NEW, object3.getState());
+ assertNotEquals(object1, object3);
+ assertNotEquals(object2, object3);
+
+ assertNull(fetcher.fetchNext());
+
+ verify(client, atLeastOnce()).getProjectByName(project.name);
+ verify(client, atLeastOnce()).getTags();
+ verify(client, times(2)).getTasks(project);
+ verify(client, times(2)).getTasks(tag);
+ verify(client, times(2)).getTasks(anotherTagWithSameName);
+ verifyNoMoreInteractions(client);
+ }
+
+ @Test
+ public void testTaskUpdatedOnlyWhenModificationDateChanges() {
+ final Task task = new Task();
+ task.gid = "1234";
+ task.name = "My first task";
+ task.modifiedAt = new DateTime(123456789);
+
+ when(client.getTasks(any(Project.class))).thenReturn(singletonMap(task.gid, task));
+
+ final AsanaObjectFetcher fetcher = new AsanaTaskFetcher(client, project.name, null, null);
+ assertNotNull(fetcher.fetchNext());
+ assertNull(fetcher.fetchNext());
+
+ task.name = "Update my task";
+ assertNull(fetcher.fetchNext());
+
+ task.modifiedAt = new DateTime(234567891);
+ final AsanaObject object = fetcher.fetchNext();
+
+ assertEquals(AsanaObjectState.UPDATED, object.getState());
+ assertEquals(task.gid, object.getGid());
+
+ verify(client, atLeastOnce()).getProjectByName(project.name);
+ verify(client, times(4)).getTasks(project);
+ verifyNoMoreInteractions(client);
+ }
+
+ @Test
+ public void testRestoreStateAndContinue() {
+ final Task task = new Task();
+ task.gid = "1234";
+ task.name = "My first task";
+ task.modifiedAt = new DateTime(123456789);
+
+ when(client.getTasks(any(Project.class))).thenReturn(singletonMap(task.gid, task));
+
+ final AsanaObjectFetcher fetcher1 = new AsanaTaskFetcher(client, project.name, null, null);
+ assertNotNull(fetcher1.fetchNext());
+
+ final AsanaObjectFetcher fetcher2 = new AsanaTaskFetcher(client, project.name, null, null);
+ fetcher2.loadState(fetcher1.saveState());
+
+ task.modifiedAt = new DateTime(234567891);
+ final AsanaObject object = fetcher2.fetchNext();
+
+ assertEquals(AsanaObjectState.UPDATED, object.getState());
+ assertEquals(task.gid, object.getGid());
+
+ verify(client, atLeastOnce()).getProjectByName(project.name);
+ verify(client, times(2)).getTasks(project);
+ verifyNoMoreInteractions(client);
+ }
+
+ @Test
+ public void testClearState() {
+ final Task task = new Task();
+ task.gid = "1234";
+ task.name = "My first task";
+ task.modifiedAt = new DateTime(123456789);
+
+ when(client.getTasks(any(Project.class))).thenReturn(singletonMap(task.gid, task));
+
+ final AsanaObjectFetcher fetcher = new AsanaTaskFetcher(client, project.name, null, null);
+ assertNotNull(fetcher.fetchNext());
+
+ fetcher.clearState();
+
+ task.modifiedAt = new DateTime(234567891);
+ final AsanaObject object = fetcher.fetchNext();
+
+ assertEquals(AsanaObjectState.NEW, object.getState());
+ assertEquals(task.gid, object.getGid());
+
+ verify(client, atLeastOnce()).getProjectByName(project.name);
+ verify(client, times(2)).getTasks(project);
+ verifyNoMoreInteractions(client);
+ }
+
+ @Test
+ public void testWrongStateForConfigurationThrows() {
+ final Project otherProject = new Project();
+ otherProject.gid = "999";
+ otherProject.name = "Other Project";
+
+ final Section otherSection = new Section();
+ otherSection.gid = "888";
+ otherSection.name = "Other Section";
+
+ final Tag otherTag = new Tag();
+ otherTag.gid = "777";
+ otherTag.name = "Other Tag";
+
+ when(client.getProjectByName(otherProject.name)).thenReturn(otherProject);
+ when(client.getSectionByName(project, otherSection.name)).thenReturn(otherSection);
+
+ final AsanaObjectFetcher fetcher1 = new AsanaTaskFetcher(client, project.name, null, null);
+ final AsanaObjectFetcher fetcher2 = new AsanaTaskFetcher(client, otherProject.name, null, null);
+ assertThrows(RuntimeException.class, () -> fetcher2.loadState(fetcher1.saveState()));
Review Comment:
Minor: `AsanaObjectFetcherException.class` could be used in the tests (or the given specific exception class).
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,393 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import static java.lang.String.format;
+import static java.util.Collections.singletonMap;
+import static java.util.stream.Collectors.groupingBy;
+import static java.util.stream.Collectors.joining;
+import static org.apache.commons.collections4.ListUtils.partition;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_EVENTS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_MEMBERS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_STATUS_ATTACHMENTS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_STATUS_UPDATES;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_STORIES;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_TASKS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_TASK_ATTACHMENTS;
+import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_TEAM_MEMBERS;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyDescriptor.Builder;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientProviderService;
+import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaObjectState;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+@TriggerSerially
+@PrimaryNodeOnly
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.")
+@Tags({"asana", "source", "ingest"})
+@CapabilityDescription("This processor collects data from Asana")
+public class GetAsanaObject extends AbstractProcessor {
+
+ protected static final String ASANA_GID = "asana.gid";
+ protected static final String ASANA_CLIENT_SERVICE = "asana-controller-service";
+ protected static final String DISTRIBUTED_CACHE_SERVICE = "distributed-cache-service";
+ protected static final String ASANA_OBJECT_TYPE = "asana-object-type";
+ protected static final String ASANA_PROJECT_NAME = "asana-project-name";
+ protected static final String ASANA_SECTION_NAME = "asana-section-name";
+ protected static final String ASANA_TAG_NAME = "asana-tag-name";
+ protected static final String ASANA_TEAM_NAME = "asana-team-name";
+ protected static final String ASANA_OUTPUT_BATCH_SIZE = "asana-output-batch-size";
+ protected static final String REL_NAME_NEW = "new";
+ protected static final String REL_NAME_UPDATED = "updated";
+ protected static final String REL_NAME_REMOVED = "removed";
+
+ protected static final PropertyDescriptor PROP_ASANA_CLIENT_SERVICE = new PropertyDescriptor.Builder()
+ .name(ASANA_CLIENT_SERVICE)
+ .displayName("Asana Client Service")
+ .description("Specify which controller service to use for accessing Asana.")
+ .required(true)
+ .identifiesControllerService(AsanaClientProviderService.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_DISTRIBUTED_CACHE_SERVICE = new Builder()
+ .name(DISTRIBUTED_CACHE_SERVICE)
+ .displayName("Distributed Cache Service")
+ .description("Cache service to store fetched item fingerprints. These, from the last successful query"
+ + " are stored, in order to enable incremental loading and change detection.")
+ .required(true)
+ .identifiesControllerService(DistributedMapCacheClient.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OBJECT_TYPE = new PropertyDescriptor.Builder()
+ .name(ASANA_OBJECT_TYPE)
+ .displayName("Object Type")
+ .description("Specify what kind of objects to be collected from Asana")
+ .required(true)
+ .allowableValues(AsanaObjectType.class)
+ .defaultValue(AV_COLLECT_TASKS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_PROJECT = new PropertyDescriptor.Builder()
+ .name(ASANA_PROJECT_NAME)
+ .displayName("Project Name")
+ .description("Fetch only objects in this project. Case sensitive.")
+ .required(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(
+ PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS.getValue(),
+ AV_COLLECT_TASK_ATTACHMENTS.getValue(),
+ AV_COLLECT_PROJECT_MEMBERS.getValue(),
+ AV_COLLECT_STORIES.getValue(),
+ AV_COLLECT_PROJECT_STATUS_UPDATES.getValue(),
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS.getValue(),
+ AV_COLLECT_PROJECT_EVENTS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_SECTION = new PropertyDescriptor.Builder()
+ .name(ASANA_SECTION_NAME)
+ .displayName("Section Name")
+ .description("Fetch only objects in this section. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS.getValue(),
+ AV_COLLECT_TASK_ATTACHMENTS.getValue(),
+ AV_COLLECT_STORIES.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TAG = new PropertyDescriptor.Builder()
+ .name(ASANA_TAG_NAME)
+ .displayName("Tag")
+ .description("Fetch only objects having this tag. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS.getValue(),
+ AV_COLLECT_TASK_ATTACHMENTS.getValue(),
+ AV_COLLECT_STORIES.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TEAM_NAME = new PropertyDescriptor.Builder()
+ .name(ASANA_TEAM_NAME)
+ .displayName("Team")
+ .description("Team name. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TEAM_MEMBERS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OUTPUT_BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name(ASANA_OUTPUT_BATCH_SIZE)
+ .displayName("Output Batch Size")
+ .description("The number of items batched together in a single Flow File. If set to 1 (default), then each item is"
+ + " transferred in a separate Flow File and each will have an asana.gid attribute, to help identifying"
+ + " the fetched item on the server side, if needed. If the batch size is greater than 1, then the"
+ + " specified amount of items are batched together in a single Flow File as a Json array, and the"
+ + " Flow Files won't have the asana.gid attribute.")
+ .defaultValue("1")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .build();
+
+ protected static final List<PropertyDescriptor> DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ PROP_ASANA_CLIENT_SERVICE,
+ PROP_DISTRIBUTED_CACHE_SERVICE,
+ PROP_ASANA_OBJECT_TYPE,
+ PROP_ASANA_PROJECT,
+ PROP_ASANA_SECTION,
+ PROP_ASANA_TEAM_NAME,
+ PROP_ASANA_TAG,
+ PROP_ASANA_OUTPUT_BATCH_SIZE
+ ));
+
+ protected static final Relationship REL_NEW = new Relationship.Builder()
+ .name(REL_NAME_NEW)
+ .description("Newly collected objects are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_UPDATED = new Relationship.Builder()
+ .name(REL_NAME_UPDATED)
+ .description("Objects that have already been collected earlier, but were updated since, are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_REMOVED = new Relationship.Builder()
+ .name(REL_NAME_REMOVED)
+ .description("Notification about deleted objects are routed to this relationship. "
+ + "Flow files will not have any payload. IDs of the resources no longer exist "
+ + "are carried by the asana.gid attribute of the generated FlowFiles.")
+ .build();
+
+ protected static final Set<Relationship> RELATIONSHIPS = Collections.unmodifiableSet(new HashSet<>(Arrays.asList(
+ REL_NEW,
+ REL_UPDATED,
+ REL_REMOVED
+ )));
+
+ private volatile AsanaObjectFetcher objectFetcher;
+ private volatile Integer batchSize;
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return RELATIONSHIPS;
+ }
+
+ @Override
+ public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return DESCRIPTORS;
+ }
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) throws InitializationException {
+ AsanaClientProviderService controllerService = context.getProperty(PROP_ASANA_CLIENT_SERVICE).asControllerService(AsanaClientProviderService.class);
+ AsanaClient client = controllerService.createClient();
+ batchSize = context.getProperty(PROP_ASANA_OUTPUT_BATCH_SIZE).asInteger();
+
+ try {
+ getLogger().debug("Initializing object fetcher...");
+ objectFetcher = createObjectFetcher(context, client);
+ } catch (Exception e) {
+ throw new InitializationException(e);
+ }
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ try {
+ Map<String, String> state = recoverState(context).orElse(Collections.emptyMap());
+ getLogger().debug("Attempting to load state: {}", state);
+ objectFetcher.loadState(state);
+ } catch (Exception e) {
+ getLogger().info("Failed to recover state. Falling back to clean start.");
+ objectFetcher.clearState();
+ }
+ getLogger().debug("Initial state: {}", objectFetcher.saveState());
+
+ Collection<FlowFile> newItems = new ArrayList<>();
+ Collection<FlowFile> updatedItems = new ArrayList<>();
+ Collection<FlowFile> removedItems = new ArrayList<>();
+ Map<AsanaObjectState, Collection<FlowFile>> flowFiles = new HashMap<>();
+ flowFiles.put(AsanaObjectState.NEW, newItems);
+ flowFiles.put(AsanaObjectState.UPDATED, updatedItems);
+ flowFiles.put(AsanaObjectState.REMOVED, removedItems);
+
+ List<AsanaObject> allObjects = new ArrayList<>();
+
+ AsanaObject nextObject;
+ while ((nextObject = objectFetcher.fetchNext()) != null) {
+ allObjects.add(nextObject);
Review Comment:
All fetched entities are collected in memory which can lead memory issues. Could not we iterate over the entities and process them on the fly?
Like: fetchNext => create FF => transfer
In case of `batchSize > 1` we need some buffers though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r1014225297
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-services/src/main/java/org/apache/nifi/controller/asana/StandardAsanaClientProviderService.java:
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.controller.asana;
+
+import com.asana.Client;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.resource.ResourceCardinality;
+import org.apache.nifi.components.resource.ResourceType;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+
+import static org.apache.nifi.controller.asana.StandardAsanaClient.ASANA_CLIENT_OPTION_BASE_URL;
+
+@CapabilityDescription("Common service to authenticate with Asana, and to work on a specified workspace.")
+@Tags({"asana", "service", "authentication"})
+public class StandardAsanaClientProviderService extends AbstractControllerService implements AsanaClientProviderService {
+
+ protected static final String ASANA_API_URL = "asana-api-url";
+ protected static final String ASANA_PERSONAL_ACCESS_TOKEN = "asana-personal-access-token";
+ protected static final String ASANA_WORKSPACE_NAME = "asana-workspace-name";
+
+ protected static final PropertyDescriptor PROP_ASANA_API_BASE_URL = new PropertyDescriptor.Builder()
+ .name(ASANA_API_URL)
+ .displayName("API URL")
+ .description("Base URL of Asana API. Leave it as default, unless you have your own Asana instance "
+ + "serving on a different URL. (typical for on-premise installations)")
+ .required(true)
+ .defaultValue(Client.DEFAULTS.get(ASANA_CLIENT_OPTION_BASE_URL).toString())
+ .identifiesExternalResource(ResourceCardinality.SINGLE, ResourceType.URL)
Review Comment:
Done in [fcaf466](https://github.com/apache/nifi/pull/6504/commits/fcaf466a7bf7ed07cde8d0d6b2da34c8352fd74d)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] takraj commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
Posted by GitBox <gi...@apache.org>.
takraj commented on code in PR #6504:
URL: https://github.com/apache/nifi/pull/6504#discussion_r995755926
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientServiceApi;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@TriggerSerially
+@Stateful(scopes = {Scope.LOCAL}, description = "Fingerprints of items in the last successful query are stored in order to enable incremental loading and change detection.")
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.")
+@Tags({"asana", "source", "connector", "ingest"})
+@CapabilityDescription("This processor collects data from Asana")
+public class GetAsanaObject extends AbstractProcessor {
+
+ protected static final String ASANA_GID = "asana.gid";
+ protected static final String AV_NAME_COLLECT_TASKS = "asana-collect-tasks";
+ protected static final String AV_NAME_COLLECT_TASK_ATTACHMENTS = "asana-collect-task-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECTS = "asana-collect-projects";
+ protected static final String AV_NAME_COLLECT_TAGS = "asana-collect-tags";
+ protected static final String AV_NAME_COLLECT_USERS = "asana-collect-users";
+ protected static final String AV_NAME_COLLECT_PROJECT_MEMBERS = "asana-collect-project-members";
+ protected static final String AV_NAME_COLLECT_TEAMS = "asana-collect-teams";
+ protected static final String AV_NAME_COLLECT_TEAM_MEMBERS = "asana-collect-team-members";
+ protected static final String AV_NAME_COLLECT_STORIES = "asana-collect-stories";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_UPDATES = "asana-collect-project-status-updates";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS = "asana-collect-project-status-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECT_EVENTS = "asana-collect-project-events";
+ protected static final String ASANA_CONTROLLER_SERVICE = "asana-controller-service";
+ protected static final String ASANA_OBJECT_TYPE = "asana-object-type";
+ protected static final String ASANA_PROJECT_NAME = "asana-project-name";
+ protected static final String ASANA_SECTION_NAME = "asana-section-name";
+ protected static final String ASANA_TAG_NAME = "asana-tag-name";
+ protected static final String ASANA_TEAM_NAME = "asana-team-name";
+ protected static final String ASANA_OUTPUT_BATCH_SIZE = "asana-output-batch-size";
+ protected static final String REL_NAME_NEW = "new";
+ protected static final String REL_NAME_UPDATED = "updated";
+ protected static final String REL_NAME_REMOVED = "removed";
+
+ protected static final AllowableValue AV_COLLECT_TASKS = new AllowableValue(
+ AV_NAME_COLLECT_TASKS,
+ "Tasks",
+ "Collect tasks matching to the specified conditions.");
+
+ protected static final AllowableValue AV_COLLECT_TASK_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_TASK_ATTACHMENTS,
+ "Task attachments",
+ "Collect attached files of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECTS,
+ "Projects",
+ "Collect projects of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TAGS = new AllowableValue(
+ AV_NAME_COLLECT_TAGS,
+ "Tags",
+ "Collect tags of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_USERS = new AllowableValue(
+ AV_NAME_COLLECT_USERS,
+ "Users",
+ "Collect users assigned to the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_MEMBERS,
+ "Members of a project",
+ "Collect users assigned to the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAMS = new AllowableValue(
+ AV_NAME_COLLECT_TEAMS,
+ "Teams",
+ "Collect teams of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAM_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_TEAM_MEMBERS,
+ "Team members",
+ "Collect users assigned to the specified team."
+ );
+
+ protected static final AllowableValue AV_COLLECT_STORIES = new AllowableValue(
+ AV_NAME_COLLECT_STORIES,
+ "Stories of tasks",
+ "Collect stories (comments) of of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_UPDATES = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_UPDATES,
+ "Status updates of a project",
+ "Collect status updates of the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ "Attachments of status updates",
+ "Collect attached files of project status updates."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_EVENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_EVENTS,
+ "Events of a project",
+ "Collect various events happening on the specified project and on its' tasks."
+ );
+
+ protected static final PropertyDescriptor PROP_ASANA_CONTROLLER_SERVICE = new PropertyDescriptor.Builder()
+ .name(ASANA_CONTROLLER_SERVICE)
+ .displayName("Controller service")
+ .description("Specify which controller service to use for accessing Asana.")
+ .required(true)
+ .identifiesControllerService(AsanaClientServiceApi.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OBJECT_TYPE = new PropertyDescriptor.Builder()
+ .name(ASANA_OBJECT_TYPE)
+ .displayName("Object type to be collected")
+ .description("Specify what kind of objects to be collected from Asana")
+ .required(true)
+ .allowableValues(
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_PROJECTS,
+ AV_COLLECT_TAGS,
+ AV_COLLECT_USERS,
+ AV_COLLECT_PROJECT_MEMBERS,
+ AV_COLLECT_TEAMS,
+ AV_COLLECT_TEAM_MEMBERS,
+ AV_COLLECT_STORIES,
+ AV_COLLECT_PROJECT_STATUS_UPDATES,
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ AV_COLLECT_PROJECT_EVENTS)
+ .defaultValue(AV_COLLECT_TASKS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_PROJECT = new PropertyDescriptor.Builder()
+ .name(ASANA_PROJECT_NAME)
+ .displayName("Project name")
+ .description("Fetch only objects in this project. Case sensitive.")
+ .required(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(
+ PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_PROJECT_MEMBERS,
+ AV_COLLECT_STORIES,
+ AV_COLLECT_PROJECT_STATUS_UPDATES,
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ AV_COLLECT_PROJECT_EVENTS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_SECTION = new PropertyDescriptor.Builder()
+ .name(ASANA_SECTION_NAME)
+ .displayName("Section name")
+ .description("Fetch only objects in this section. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TASKS, AV_COLLECT_TASK_ATTACHMENTS, AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TAG = new PropertyDescriptor.Builder()
+ .name(ASANA_TAG_NAME)
+ .displayName("Tag")
+ .description("Fetch only objects having this tag. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TASKS, AV_COLLECT_TASK_ATTACHMENTS, AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TEAM_NAME = new PropertyDescriptor.Builder()
+ .name(ASANA_TEAM_NAME)
+ .displayName("Team")
+ .description("Team name. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TEAM_MEMBERS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OUTPUT_BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name(ASANA_OUTPUT_BATCH_SIZE)
+ .displayName("Output Batch Size")
+ .description("The number of output FlowFiles to queue before committing the process session. When set to zero, the session will be committed when all result set rows "
+ + "have been processed and the output FlowFiles are ready for transfer to the downstream relationship. For large result sets, this can cause a large burst of FlowFiles "
+ + "to be transferred at the end of processor execution. If this property is set, then when the specified number of FlowFiles are ready for transfer, then the session will "
+ + "be committed, thus releasing the FlowFiles to the downstream relationship.")
+ .defaultValue("0")
+ .required(true)
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+
+ protected static final List<PropertyDescriptor> DESCRIPTORS = Collections.unmodifiableList(Lists.newArrayList(
+ PROP_ASANA_CONTROLLER_SERVICE,
+ PROP_ASANA_OBJECT_TYPE,
+ PROP_ASANA_PROJECT,
+ PROP_ASANA_SECTION,
+ PROP_ASANA_TEAM_NAME,
+ PROP_ASANA_TAG,
+ PROP_ASANA_OUTPUT_BATCH_SIZE
+ ));
+
+ protected static final Relationship REL_NEW = new Relationship.Builder()
+ .name(REL_NAME_NEW)
+ .description("Newly collected objects are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_UPDATED = new Relationship.Builder()
+ .name(REL_NAME_UPDATED)
+ .description("Objects that have already been collected earlier, but were updated since, are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_REMOVED = new Relationship.Builder()
+ .name(REL_NAME_REMOVED)
+ .description("Notification about deleted objects are routed to this relationship. "
+ + "Flow files will not have any payload. IDs of the resources no longer exist "
+ + "are carried by the asana.gid attribute of the generated FlowFiles.")
+ .build();
+
+ protected static final Set<Relationship> RELATIONSHIPS = Collections.unmodifiableSet(Sets.newHashSet(
+ REL_NEW,
+ REL_UPDATED,
+ REL_REMOVED
+ ));
+
+ final Scope STATE_STORAGE_SCOPE = Scope.LOCAL;
+
+ AsanaClientServiceApi controllerService;
Review Comment:
Done in [dc2a050](https://github.com/apache/nifi/pull/6504/commits/dc2a050548a389d89181670371748af741eefd61)
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientServiceApi;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@TriggerSerially
+@Stateful(scopes = {Scope.LOCAL}, description = "Fingerprints of items in the last successful query are stored in order to enable incremental loading and change detection.")
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.")
+@Tags({"asana", "source", "connector", "ingest"})
+@CapabilityDescription("This processor collects data from Asana")
+public class GetAsanaObject extends AbstractProcessor {
+
+ protected static final String ASANA_GID = "asana.gid";
+ protected static final String AV_NAME_COLLECT_TASKS = "asana-collect-tasks";
+ protected static final String AV_NAME_COLLECT_TASK_ATTACHMENTS = "asana-collect-task-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECTS = "asana-collect-projects";
+ protected static final String AV_NAME_COLLECT_TAGS = "asana-collect-tags";
+ protected static final String AV_NAME_COLLECT_USERS = "asana-collect-users";
+ protected static final String AV_NAME_COLLECT_PROJECT_MEMBERS = "asana-collect-project-members";
+ protected static final String AV_NAME_COLLECT_TEAMS = "asana-collect-teams";
+ protected static final String AV_NAME_COLLECT_TEAM_MEMBERS = "asana-collect-team-members";
+ protected static final String AV_NAME_COLLECT_STORIES = "asana-collect-stories";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_UPDATES = "asana-collect-project-status-updates";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS = "asana-collect-project-status-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECT_EVENTS = "asana-collect-project-events";
+ protected static final String ASANA_CONTROLLER_SERVICE = "asana-controller-service";
+ protected static final String ASANA_OBJECT_TYPE = "asana-object-type";
+ protected static final String ASANA_PROJECT_NAME = "asana-project-name";
+ protected static final String ASANA_SECTION_NAME = "asana-section-name";
+ protected static final String ASANA_TAG_NAME = "asana-tag-name";
+ protected static final String ASANA_TEAM_NAME = "asana-team-name";
+ protected static final String ASANA_OUTPUT_BATCH_SIZE = "asana-output-batch-size";
+ protected static final String REL_NAME_NEW = "new";
+ protected static final String REL_NAME_UPDATED = "updated";
+ protected static final String REL_NAME_REMOVED = "removed";
+
+ protected static final AllowableValue AV_COLLECT_TASKS = new AllowableValue(
+ AV_NAME_COLLECT_TASKS,
+ "Tasks",
+ "Collect tasks matching to the specified conditions.");
+
+ protected static final AllowableValue AV_COLLECT_TASK_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_TASK_ATTACHMENTS,
+ "Task attachments",
+ "Collect attached files of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECTS,
+ "Projects",
+ "Collect projects of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TAGS = new AllowableValue(
+ AV_NAME_COLLECT_TAGS,
+ "Tags",
+ "Collect tags of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_USERS = new AllowableValue(
+ AV_NAME_COLLECT_USERS,
+ "Users",
+ "Collect users assigned to the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_MEMBERS,
+ "Members of a project",
+ "Collect users assigned to the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAMS = new AllowableValue(
+ AV_NAME_COLLECT_TEAMS,
+ "Teams",
+ "Collect teams of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAM_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_TEAM_MEMBERS,
+ "Team members",
+ "Collect users assigned to the specified team."
+ );
+
+ protected static final AllowableValue AV_COLLECT_STORIES = new AllowableValue(
+ AV_NAME_COLLECT_STORIES,
+ "Stories of tasks",
+ "Collect stories (comments) of of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_UPDATES = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_UPDATES,
+ "Status updates of a project",
+ "Collect status updates of the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ "Attachments of status updates",
+ "Collect attached files of project status updates."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_EVENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_EVENTS,
+ "Events of a project",
+ "Collect various events happening on the specified project and on its' tasks."
+ );
+
+ protected static final PropertyDescriptor PROP_ASANA_CONTROLLER_SERVICE = new PropertyDescriptor.Builder()
+ .name(ASANA_CONTROLLER_SERVICE)
+ .displayName("Controller service")
+ .description("Specify which controller service to use for accessing Asana.")
+ .required(true)
+ .identifiesControllerService(AsanaClientServiceApi.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OBJECT_TYPE = new PropertyDescriptor.Builder()
+ .name(ASANA_OBJECT_TYPE)
+ .displayName("Object type to be collected")
+ .description("Specify what kind of objects to be collected from Asana")
+ .required(true)
+ .allowableValues(
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_PROJECTS,
+ AV_COLLECT_TAGS,
+ AV_COLLECT_USERS,
+ AV_COLLECT_PROJECT_MEMBERS,
+ AV_COLLECT_TEAMS,
+ AV_COLLECT_TEAM_MEMBERS,
+ AV_COLLECT_STORIES,
+ AV_COLLECT_PROJECT_STATUS_UPDATES,
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ AV_COLLECT_PROJECT_EVENTS)
+ .defaultValue(AV_COLLECT_TASKS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_PROJECT = new PropertyDescriptor.Builder()
+ .name(ASANA_PROJECT_NAME)
+ .displayName("Project name")
+ .description("Fetch only objects in this project. Case sensitive.")
+ .required(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(
+ PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_PROJECT_MEMBERS,
+ AV_COLLECT_STORIES,
+ AV_COLLECT_PROJECT_STATUS_UPDATES,
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ AV_COLLECT_PROJECT_EVENTS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_SECTION = new PropertyDescriptor.Builder()
+ .name(ASANA_SECTION_NAME)
+ .displayName("Section name")
+ .description("Fetch only objects in this section. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TASKS, AV_COLLECT_TASK_ATTACHMENTS, AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TAG = new PropertyDescriptor.Builder()
+ .name(ASANA_TAG_NAME)
+ .displayName("Tag")
+ .description("Fetch only objects having this tag. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TASKS, AV_COLLECT_TASK_ATTACHMENTS, AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TEAM_NAME = new PropertyDescriptor.Builder()
+ .name(ASANA_TEAM_NAME)
+ .displayName("Team")
+ .description("Team name. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TEAM_MEMBERS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OUTPUT_BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name(ASANA_OUTPUT_BATCH_SIZE)
+ .displayName("Output Batch Size")
+ .description("The number of output FlowFiles to queue before committing the process session. When set to zero, the session will be committed when all result set rows "
+ + "have been processed and the output FlowFiles are ready for transfer to the downstream relationship. For large result sets, this can cause a large burst of FlowFiles "
+ + "to be transferred at the end of processor execution. If this property is set, then when the specified number of FlowFiles are ready for transfer, then the session will "
+ + "be committed, thus releasing the FlowFiles to the downstream relationship.")
+ .defaultValue("0")
+ .required(true)
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+
+ protected static final List<PropertyDescriptor> DESCRIPTORS = Collections.unmodifiableList(Lists.newArrayList(
+ PROP_ASANA_CONTROLLER_SERVICE,
+ PROP_ASANA_OBJECT_TYPE,
+ PROP_ASANA_PROJECT,
+ PROP_ASANA_SECTION,
+ PROP_ASANA_TEAM_NAME,
+ PROP_ASANA_TAG,
+ PROP_ASANA_OUTPUT_BATCH_SIZE
+ ));
+
+ protected static final Relationship REL_NEW = new Relationship.Builder()
+ .name(REL_NAME_NEW)
+ .description("Newly collected objects are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_UPDATED = new Relationship.Builder()
+ .name(REL_NAME_UPDATED)
+ .description("Objects that have already been collected earlier, but were updated since, are routed to this relationship.")
+ .build();
+
+ protected static final Relationship REL_REMOVED = new Relationship.Builder()
+ .name(REL_NAME_REMOVED)
+ .description("Notification about deleted objects are routed to this relationship. "
+ + "Flow files will not have any payload. IDs of the resources no longer exist "
+ + "are carried by the asana.gid attribute of the generated FlowFiles.")
+ .build();
+
+ protected static final Set<Relationship> RELATIONSHIPS = Collections.unmodifiableSet(Sets.newHashSet(
+ REL_NEW,
+ REL_UPDATED,
+ REL_REMOVED
+ ));
+
+ final Scope STATE_STORAGE_SCOPE = Scope.LOCAL;
+
+ AsanaClientServiceApi controllerService;
+ AsanaObjectFetcher objectFetcher;
+ private Integer batchSize;
Review Comment:
Done in [dc2a050](https://github.com/apache/nifi/pull/6504/commits/dc2a050548a389d89181670371748af741eefd61)
##########
nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java:
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.asana;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+import org.apache.http.entity.ContentType;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.components.state.StateMap;
+import org.apache.nifi.controller.asana.AsanaClient;
+import org.apache.nifi.controller.asana.AsanaClientServiceApi;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.asana.utils.AsanaObject;
+import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTagFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher;
+import org.apache.nifi.processors.asana.utils.AsanaUserFetcher;
+import org.apache.nifi.reporting.InitializationException;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@TriggerSerially
+@Stateful(scopes = {Scope.LOCAL}, description = "Fingerprints of items in the last successful query are stored in order to enable incremental loading and change detection.")
+@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
+@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.")
+@Tags({"asana", "source", "connector", "ingest"})
+@CapabilityDescription("This processor collects data from Asana")
+public class GetAsanaObject extends AbstractProcessor {
+
+ protected static final String ASANA_GID = "asana.gid";
+ protected static final String AV_NAME_COLLECT_TASKS = "asana-collect-tasks";
+ protected static final String AV_NAME_COLLECT_TASK_ATTACHMENTS = "asana-collect-task-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECTS = "asana-collect-projects";
+ protected static final String AV_NAME_COLLECT_TAGS = "asana-collect-tags";
+ protected static final String AV_NAME_COLLECT_USERS = "asana-collect-users";
+ protected static final String AV_NAME_COLLECT_PROJECT_MEMBERS = "asana-collect-project-members";
+ protected static final String AV_NAME_COLLECT_TEAMS = "asana-collect-teams";
+ protected static final String AV_NAME_COLLECT_TEAM_MEMBERS = "asana-collect-team-members";
+ protected static final String AV_NAME_COLLECT_STORIES = "asana-collect-stories";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_UPDATES = "asana-collect-project-status-updates";
+ protected static final String AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS = "asana-collect-project-status-attachments";
+ protected static final String AV_NAME_COLLECT_PROJECT_EVENTS = "asana-collect-project-events";
+ protected static final String ASANA_CONTROLLER_SERVICE = "asana-controller-service";
+ protected static final String ASANA_OBJECT_TYPE = "asana-object-type";
+ protected static final String ASANA_PROJECT_NAME = "asana-project-name";
+ protected static final String ASANA_SECTION_NAME = "asana-section-name";
+ protected static final String ASANA_TAG_NAME = "asana-tag-name";
+ protected static final String ASANA_TEAM_NAME = "asana-team-name";
+ protected static final String ASANA_OUTPUT_BATCH_SIZE = "asana-output-batch-size";
+ protected static final String REL_NAME_NEW = "new";
+ protected static final String REL_NAME_UPDATED = "updated";
+ protected static final String REL_NAME_REMOVED = "removed";
+
+ protected static final AllowableValue AV_COLLECT_TASKS = new AllowableValue(
+ AV_NAME_COLLECT_TASKS,
+ "Tasks",
+ "Collect tasks matching to the specified conditions.");
+
+ protected static final AllowableValue AV_COLLECT_TASK_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_TASK_ATTACHMENTS,
+ "Task attachments",
+ "Collect attached files of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECTS,
+ "Projects",
+ "Collect projects of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TAGS = new AllowableValue(
+ AV_NAME_COLLECT_TAGS,
+ "Tags",
+ "Collect tags of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_USERS = new AllowableValue(
+ AV_NAME_COLLECT_USERS,
+ "Users",
+ "Collect users assigned to the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_MEMBERS,
+ "Members of a project",
+ "Collect users assigned to the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAMS = new AllowableValue(
+ AV_NAME_COLLECT_TEAMS,
+ "Teams",
+ "Collect teams of the workspace."
+ );
+
+ protected static final AllowableValue AV_COLLECT_TEAM_MEMBERS = new AllowableValue(
+ AV_NAME_COLLECT_TEAM_MEMBERS,
+ "Team members",
+ "Collect users assigned to the specified team."
+ );
+
+ protected static final AllowableValue AV_COLLECT_STORIES = new AllowableValue(
+ AV_NAME_COLLECT_STORIES,
+ "Stories of tasks",
+ "Collect stories (comments) of of tasks matching to the specified conditions."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_UPDATES = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_UPDATES,
+ "Status updates of a project",
+ "Collect status updates of the specified project."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_STATUS_ATTACHMENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ "Attachments of status updates",
+ "Collect attached files of project status updates."
+ );
+
+ protected static final AllowableValue AV_COLLECT_PROJECT_EVENTS = new AllowableValue(
+ AV_NAME_COLLECT_PROJECT_EVENTS,
+ "Events of a project",
+ "Collect various events happening on the specified project and on its' tasks."
+ );
+
+ protected static final PropertyDescriptor PROP_ASANA_CONTROLLER_SERVICE = new PropertyDescriptor.Builder()
+ .name(ASANA_CONTROLLER_SERVICE)
+ .displayName("Controller service")
+ .description("Specify which controller service to use for accessing Asana.")
+ .required(true)
+ .identifiesControllerService(AsanaClientServiceApi.class)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OBJECT_TYPE = new PropertyDescriptor.Builder()
+ .name(ASANA_OBJECT_TYPE)
+ .displayName("Object type to be collected")
+ .description("Specify what kind of objects to be collected from Asana")
+ .required(true)
+ .allowableValues(
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_PROJECTS,
+ AV_COLLECT_TAGS,
+ AV_COLLECT_USERS,
+ AV_COLLECT_PROJECT_MEMBERS,
+ AV_COLLECT_TEAMS,
+ AV_COLLECT_TEAM_MEMBERS,
+ AV_COLLECT_STORIES,
+ AV_COLLECT_PROJECT_STATUS_UPDATES,
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ AV_COLLECT_PROJECT_EVENTS)
+ .defaultValue(AV_COLLECT_TASKS.getValue())
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_PROJECT = new PropertyDescriptor.Builder()
+ .name(ASANA_PROJECT_NAME)
+ .displayName("Project name")
+ .description("Fetch only objects in this project. Case sensitive.")
+ .required(true)
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(
+ PROP_ASANA_OBJECT_TYPE,
+ AV_COLLECT_TASKS,
+ AV_COLLECT_TASK_ATTACHMENTS,
+ AV_COLLECT_PROJECT_MEMBERS,
+ AV_COLLECT_STORIES,
+ AV_COLLECT_PROJECT_STATUS_UPDATES,
+ AV_COLLECT_PROJECT_STATUS_ATTACHMENTS,
+ AV_COLLECT_PROJECT_EVENTS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_SECTION = new PropertyDescriptor.Builder()
+ .name(ASANA_SECTION_NAME)
+ .displayName("Section name")
+ .description("Fetch only objects in this section. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TASKS, AV_COLLECT_TASK_ATTACHMENTS, AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TAG = new PropertyDescriptor.Builder()
+ .name(ASANA_TAG_NAME)
+ .displayName("Tag")
+ .description("Fetch only objects having this tag. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TASKS, AV_COLLECT_TASK_ATTACHMENTS, AV_COLLECT_STORIES)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_TEAM_NAME = new PropertyDescriptor.Builder()
+ .name(ASANA_TEAM_NAME)
+ .displayName("Team")
+ .description("Team name. Case sensitive.")
+ .addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+ .dependsOn(PROP_ASANA_OBJECT_TYPE, AV_COLLECT_TEAM_MEMBERS)
+ .build();
+
+ protected static final PropertyDescriptor PROP_ASANA_OUTPUT_BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name(ASANA_OUTPUT_BATCH_SIZE)
+ .displayName("Output Batch Size")
+ .description("The number of output FlowFiles to queue before committing the process session. When set to zero, the session will be committed when all result set rows "
+ + "have been processed and the output FlowFiles are ready for transfer to the downstream relationship. For large result sets, this can cause a large burst of FlowFiles "
+ + "to be transferred at the end of processor execution. If this property is set, then when the specified number of FlowFiles are ready for transfer, then the session will "
+ + "be committed, thus releasing the FlowFiles to the downstream relationship.")
+ .defaultValue("0")
+ .required(true)
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+
+ protected static final List<PropertyDescriptor> DESCRIPTORS = Collections.unmodifiableList(Lists.newArrayList(
Review Comment:
Done in [dc2a050](https://github.com/apache/nifi/pull/6504/commits/dc2a050548a389d89181670371748af741eefd61)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org