You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by GitBox <gi...@apache.org> on 2022/04/08 06:39:58 UTC
[GitHub] [nifi] emiliosetiadarma opened a new pull request, #5947: NIFI-9755: Implement ConsumeTwitter
emiliosetiadarma opened a new pull request, #5947:
URL: https://github.com/apache/nifi/pull/5947
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
#### Description of PR
Implemented ConsumeTwitter with the new Twitter API SDK (https://github.com/twitterdev/twitter-api-java-sdk). This processor also supports batching of many tweets into one FlowFile.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [x] Does your PR title start with **NIFI-XXXX** where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target branch (typically `main`)?
- [] Is your initial contribution a single, squashed commit? _Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not `squash` or use `--force` when pushing to allow for clean monitoring of changes._
### For code changes:
- [x] Have you ensured that the full suite of tests is executed via `mvn -Pcontrib-check clean install` at the root `nifi` folder?
- [x] Have you written or updated unit tests to verify your changes?
- [x] Have you verified that the full build is successful on JDK 8?
- [x] Have you verified that the full build is successful on JDK 11?
- [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main `LICENSE` file under `nifi-assembly`?
- [x] If applicable, have you updated the `NOTICE` file, including the main `NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to .name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for build issues and submit an update to your PR as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
exceptionfactory commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r863882243
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/pom.xml:
##########
@@ -35,6 +35,21 @@
<artifactId>nifi-utils</artifactId>
<version>1.17.0-SNAPSHOT</version>
</dependency>
+ <dependency>
+ <groupId>com.twitter</groupId>
+ <artifactId>twitter-api-java-sdk</artifactId>
+ <version>1.1.2</version>
Review Comment:
This version can be incremented to `1.1.4`.
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,366 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.UUID;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue(StreamEndpoint.SAMPLE_ENDPOINT.getEndpointName(),
+ "Sample Stream",
+ "Streams about one percent of all Tweets. " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/volume-streams/api-reference/get-tweets-sample-stream");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue(StreamEndpoint.SEARCH_ENDPOINT.getEndpointName(),
+ "Search Stream",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty. " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/get-tweets-search-stream");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("stream-endpoint")
+ .displayName("Stream Endpoint")
+ .description("Specifies which endpoint tweets should be streamed. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
Review Comment:
This can be simplified since the individual values describe what is necessary for each option.
```suggestion
.description("The source from which the processor will consume Tweets.")
```
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TweetStreamService.java:
##########
@@ -0,0 +1,222 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiClient;
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TweetStreamService {
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ScheduledExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Long backoffMultiplier;
+ private int backoffAttempts;
+ private int attemptCounter;
+ private Long backoffTime;
+ private Long maximumBackoff;
+
+ private StreamEndpoint endpoint;
+
+ public TweetStreamService(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ Objects.requireNonNull(context);
+ Objects.requireNonNull(queue);
+ Objects.requireNonNull(logger);
+
+ this.queue = queue;
+ this.logger = logger;
+
+ this.tweetFields = parseCommaSeparatedPropreties(context, ConsumeTwitter.TWEET_FIELDS);
+ this.userFields = parseCommaSeparatedPropreties(context, ConsumeTwitter.USER_FIELDS);
+ this.mediaFields = parseCommaSeparatedPropreties(context, ConsumeTwitter.MEDIA_FIELDS);
+ this.pollFields = parseCommaSeparatedPropreties(context, ConsumeTwitter.POLL_FIELDS);
+ this.placeFields = parseCommaSeparatedPropreties(context, ConsumeTwitter.PLACE_FIELDS);
+ this.expansions = parseCommaSeparatedPropreties(context, ConsumeTwitter.EXPANSIONS);
+ this.backfillMinutes = context.getProperty(ConsumeTwitter.BACKFILL_MINUTES).asInteger();
+
+ this.backoffMultiplier = 1L;
+ this.backoffAttempts = context.getProperty(ConsumeTwitter.BACKOFF_ATTEMPTS).asInteger();
+ this.attemptCounter = 0;
+ this.backoffTime = context.getProperty(ConsumeTwitter.BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+ this.maximumBackoff = context.getProperty(ConsumeTwitter.MAXIMUM_BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+
+ ApiClient client = new ApiClient();
+ final int connectTimeout = context.getProperty(ConsumeTwitter.CONNECT_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ final int readTimeout = context.getProperty(ConsumeTwitter.READ_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ client = client.setConnectTimeout(connectTimeout);
+ client = client.setReadTimeout(readTimeout);
+ api = new TwitterApi(client);
+
+ TwitterCredentialsBearer creds = new TwitterCredentialsBearer(context.getProperty(ConsumeTwitter.BEARER_TOKEN).getValue());
+ api.setTwitterCredentials(creds);
+
+ this.executorService = Executors.newSingleThreadScheduledExecutor();
+ }
+
+ public String getBasePath() {
+ return api.getApiClient().getBasePath();
+ }
+
+ public void setBasePath(final String path) {
+ api.getApiClient().setBasePath(path);
+ }
Review Comment:
By refactoring resolution of the base path property to the constructor, both of these methods can be removed.
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/pom.xml:
##########
@@ -35,6 +35,21 @@
<artifactId>nifi-utils</artifactId>
<version>1.17.0-SNAPSHOT</version>
</dependency>
+ <dependency>
+ <groupId>com.twitter</groupId>
+ <artifactId>twitter-api-java-sdk</artifactId>
+ <version>1.1.2</version>
+ <exclusions>
+ <exclusion>
+ <groupId>org.json</groupId>
+ <artifactId>json</artifactId>
+ </exclusion>
Review Comment:
This dependency does not appear to included, so the exclusion can be removed.
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,366 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.UUID;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue(StreamEndpoint.SAMPLE_ENDPOINT.getEndpointName(),
+ "Sample Stream",
+ "Streams about one percent of all Tweets. " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/volume-streams/api-reference/get-tweets-sample-stream");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue(StreamEndpoint.SEARCH_ENDPOINT.getEndpointName(),
+ "Search Stream",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty. " +
Review Comment:
Recommend minor adjustments:
```suggestion
"The search stream produces Tweets that match filtering rules configured on Twitter services. At least one filtering rule must be configured." +
```
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,366 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.UUID;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
Review Comment:
This line can be removed, it is probably not worth noting for the purposes of the processor documentation.
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,366 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.UUID;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue(StreamEndpoint.SAMPLE_ENDPOINT.getEndpointName(),
+ "Sample Stream",
+ "Streams about one percent of all Tweets. " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/volume-streams/api-reference/get-tweets-sample-stream");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue(StreamEndpoint.SEARCH_ENDPOINT.getEndpointName(),
+ "Search Stream",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty. " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/get-tweets-search-stream");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("stream-endpoint")
+ .displayName("Stream Endpoint")
+ .description("Specifies which endpoint tweets should be streamed. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("Maximum size of internal queue for streamed messages")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor BACKOFF_ATTEMPTS = new PropertyDescriptor.Builder()
+ .name("backoff-attempts")
+ .displayName("Backoff Attempts")
+ .description("The number of reconnection tries the processor will attempt in the event of " +
+ "a disconnection of the stream for any reason, before throwing an exception. To start a stream after " +
+ "this exception occur and the connection is fixed, please stop and restart the processor. If the value" +
+ "of this property is 0, then backoff will never occur and the processor will always need to be restarted" +
+ "if the stream fails.")
+ .required(true)
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .defaultValue("5")
+ .build();
+ public static final PropertyDescriptor BACKOFF_TIME = new PropertyDescriptor.Builder()
+ .name("backoff-time")
+ .displayName("Backoff Time")
+ .description("The duration to backoff to start attempting a new stream if" +
+ "the current one fails for any reason. Will increase by factor of 2 every time a restart fails")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("1 mins")
+ .build();
+ public static final PropertyDescriptor MAXIMUM_BACKOFF_TIME = new PropertyDescriptor.Builder()
+ .name("maximum-backoff-time")
+ .displayName("Maximum Backoff Time")
+ .description("The maximum duration to backoff to start attempting a new stream." +
+ "It is recommended that this number be much higher than the 'Backoff Time' property")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("5 mins")
+ .build();
+ public static final PropertyDescriptor CONNECT_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("connect-timeout")
+ .displayName("Connect Timeout")
+ .description("The maximum time in which client should establish a connection with the " +
+ "Twitter API before a time out. A value of 0 will mean no timeout at all")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("10 secs")
+ .build();
+ public static final PropertyDescriptor READ_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("read-timeout")
+ .displayName("Read Timeout")
+ .description("The maximum time of inactivity between receiving tweets from Twitter through " +
+ "the API before a timeout. A value of 0 will mean no timeout at all")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("10 secs")
+ .build();
+ public static final PropertyDescriptor BACKFILL_MINUTES = new PropertyDescriptor.Builder()
+ .name("backfill-minutes")
+ .displayName("Backfill Minutes")
+ .description("The number of minutes (up to 5 minutes) of streaming data to be requested after a " +
+ "disconnect. Only available for project with academic research access. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/integrate/" +
+ "recovery-and-redundancy-features")
+ .required(true)
+ .defaultValue("0")
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("tweet-fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values include: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("user-fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user " +
+ "for proper usage. Possible field values include: " +
+ "created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, " +
+ "protected, public_metrics, url, username, verified, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor MEDIA_FIELDS = new PropertyDescriptor.Builder()
+ .name("media-fields")
+ .displayName("Media Fields")
+ .description("A comma-separated list of media fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/media " +
+ "for proper usage. Possible field values include: " +
+ "alt_text, duration_ms, height, media_key, non_public_metrics, organic_metrics, preview_image_url, " +
+ "promoted_metrics, public_metrics, type, url, width")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor POLL_FIELDS = new PropertyDescriptor.Builder()
+ .name("poll-fields")
+ .displayName("Poll Fields")
+ .description("A comma-separated list of poll fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/poll " +
+ "for proper usage. Possible field values include: " +
+ "duration_minutes, end_datetime, id, options, voting_status")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PLACE_FIELDS = new PropertyDescriptor.Builder()
+ .name("place-fields")
+ .displayName("Place Fields")
+ .description("A comma-separated list of place fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/place " +
+ "for proper usage. Possible field values include: " +
+ "contained_within, country, country_code, full_name, geo, id, name, place_type")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor EXPANSIONS = new PropertyDescriptor.Builder()
+ .name("expansions")
+ .displayName("Expansions")
+ .description("A comma-separated list of expansions for objects in the returned tweet. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/expansions " +
+ "for proper usage. Possible field values include: " +
+ "author_id, referenced_tweets.id, referenced_tweets.id.author_id, entities.mentions.username, " +
+ "attachments.poll_ids, attachments.media_keys ,in_reply_to_user_id, geo.place_id")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("FlowFiles containing an array of one or more Tweets")
+ .build();
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+
+ private TweetStreamService tweetStreamService;
+
+ private volatile BlockingQueue<String> messageQueue;
+
+ @Override
+ protected void init(ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptors = new ArrayList<>();
+ descriptors.add(ENDPOINT);
+ descriptors.add(BASE_PATH);
+ descriptors.add(BEARER_TOKEN);
+ descriptors.add(QUEUE_SIZE);
+ descriptors.add(BATCH_SIZE);
+ descriptors.add(BACKOFF_ATTEMPTS);
+ descriptors.add(BACKOFF_TIME);
+ descriptors.add(MAXIMUM_BACKOFF_TIME);
+ descriptors.add(CONNECT_TIMEOUT);
+ descriptors.add(READ_TIMEOUT);
+ descriptors.add(BACKFILL_MINUTES);
+ descriptors.add(TWEET_FIELDS);
+ descriptors.add(USER_FIELDS);
+ descriptors.add(MEDIA_FIELDS);
+ descriptors.add(POLL_FIELDS);
+ descriptors.add(PLACE_FIELDS);
+ descriptors.add(EXPANSIONS);
+
+ this.descriptors = Collections.unmodifiableList(descriptors);
+
+ final Set<Relationship> relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ this.relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return this.descriptors;
+ }
+
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ messageQueue = new LinkedBlockingQueue<>(context.getProperty(QUEUE_SIZE).asInteger());
+
+ tweetStreamService = new TweetStreamService(context, messageQueue, getLogger());
+ tweetStreamService.setBasePath(context.getProperty(BASE_PATH).getValue());
+ final String endpointName = context.getProperty(ENDPOINT).getValue();
+ if (ENDPOINT_SAMPLE.getValue().equals(endpointName)) {
+ tweetStreamService.start(StreamEndpoint.SAMPLE_ENDPOINT);
+ } else {
+ tweetStreamService.start(StreamEndpoint.SEARCH_ENDPOINT);
+ }
+ }
+
+ @Override
+ public void onTrigger(ProcessContext context, ProcessSession session) throws ProcessException {
+ if (messageQueue.isEmpty()) {
+ context.yield();
+ return;
+ }
+
+ FlowFile flowFile = session.create();
+ flowFile = session.write(flowFile, new OutputStreamCallback() {
+ @Override
+ public void process(OutputStream out) throws IOException {
+
+ final int batchSize = context.getProperty(BATCH_SIZE).asInteger();
+ String tweet = messageQueue.poll();
Review Comment:
The `poll()` method can return `null` if the processor is configured with multiple concurrent tasks. For this reason, the `messageQueue.poll()` call should be pulled out of the callback, and used as a conditional to decide whether to create a new FlowFile.
Although `messageQueue.isEmpty()` accomplishes a similar check, the potential for multiple concurrent tasks running this method could create a problem. Instead, having the first line of the `onTrigger()` method call `messageQueue.poll()` should work. Something along the following lines:
```
final String firstTweet = messageQueue.poll();
if (firstTweet == null) {
context.yield();
} else {
FlowFile flowFile = session.create();
...
}
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TweetStreamService.java:
##########
@@ -0,0 +1,222 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiClient;
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TweetStreamService {
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ScheduledExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Long backoffMultiplier;
+ private int backoffAttempts;
+ private int attemptCounter;
+ private Long backoffTime;
+ private Long maximumBackoff;
+
+ private StreamEndpoint endpoint;
+
+ public TweetStreamService(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ Objects.requireNonNull(context);
+ Objects.requireNonNull(queue);
+ Objects.requireNonNull(logger);
+
+ this.queue = queue;
+ this.logger = logger;
+
+ this.tweetFields = parseCommaSeparatedPropreties(context, ConsumeTwitter.TWEET_FIELDS);
+ this.userFields = parseCommaSeparatedPropreties(context, ConsumeTwitter.USER_FIELDS);
+ this.mediaFields = parseCommaSeparatedPropreties(context, ConsumeTwitter.MEDIA_FIELDS);
+ this.pollFields = parseCommaSeparatedPropreties(context, ConsumeTwitter.POLL_FIELDS);
+ this.placeFields = parseCommaSeparatedPropreties(context, ConsumeTwitter.PLACE_FIELDS);
+ this.expansions = parseCommaSeparatedPropreties(context, ConsumeTwitter.EXPANSIONS);
+ this.backfillMinutes = context.getProperty(ConsumeTwitter.BACKFILL_MINUTES).asInteger();
+
+ this.backoffMultiplier = 1L;
+ this.backoffAttempts = context.getProperty(ConsumeTwitter.BACKOFF_ATTEMPTS).asInteger();
+ this.attemptCounter = 0;
+ this.backoffTime = context.getProperty(ConsumeTwitter.BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+ this.maximumBackoff = context.getProperty(ConsumeTwitter.MAXIMUM_BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+
+ ApiClient client = new ApiClient();
+ final int connectTimeout = context.getProperty(ConsumeTwitter.CONNECT_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ final int readTimeout = context.getProperty(ConsumeTwitter.READ_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ client = client.setConnectTimeout(connectTimeout);
+ client = client.setReadTimeout(readTimeout);
+ api = new TwitterApi(client);
+
+ TwitterCredentialsBearer creds = new TwitterCredentialsBearer(context.getProperty(ConsumeTwitter.BEARER_TOKEN).getValue());
+ api.setTwitterCredentials(creds);
+
+ this.executorService = Executors.newSingleThreadScheduledExecutor();
+ }
+
+ public String getBasePath() {
+ return api.getApiClient().getBasePath();
+ }
+
+ public void setBasePath(final String path) {
+ api.getApiClient().setBasePath(path);
+ }
+
+ public String getTransitUri(final String endpoint) {
+ if (endpoint.equals(StreamEndpoint.SAMPLE_ENDPOINT.getEndpointName())) {
+ return api.getApiClient().getBasePath() + StreamEndpoint.SAMPLE_ENDPOINT.getPath();
+ } else if (endpoint.equals(StreamEndpoint.SEARCH_ENDPOINT.getEndpointName())) {
+ return api.getApiClient().getBasePath() + StreamEndpoint.SEARCH_ENDPOINT.getPath();
+ } else {
+ logger.warn("Unrecognized endpoint in getTransitUri. Returning basePath");
+ return api.getApiClient().getBasePath();
+ }
+ }
+
+ /**
+ * This method would be called when we would like the stream to get started. This method will spin off a thread that
+ * will continue to queue tweets on to the given queue passed in the constructor. The thread will continue
+ * to run until {@code stop} is called.
+ * @param endpoint {@code TwitterStreamAPI.SAMPLE_ENDPOINT} or {@code TwitterStreamAPI.SEARCH_ENDPOINT}
Review Comment:
```suggestion
```
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,366 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.UUID;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue(StreamEndpoint.SAMPLE_ENDPOINT.getEndpointName(),
+ "Sample Stream",
+ "Streams about one percent of all Tweets. " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/volume-streams/api-reference/get-tweets-sample-stream");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue(StreamEndpoint.SEARCH_ENDPOINT.getEndpointName(),
+ "Search Stream",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty. " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/get-tweets-search-stream");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("stream-endpoint")
+ .displayName("Stream Endpoint")
+ .description("Specifies which endpoint tweets should be streamed. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("Maximum size of internal queue for streamed messages")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor BACKOFF_ATTEMPTS = new PropertyDescriptor.Builder()
+ .name("backoff-attempts")
+ .displayName("Backoff Attempts")
+ .description("The number of reconnection tries the processor will attempt in the event of " +
+ "a disconnection of the stream for any reason, before throwing an exception. To start a stream after " +
+ "this exception occur and the connection is fixed, please stop and restart the processor. If the value" +
+ "of this property is 0, then backoff will never occur and the processor will always need to be restarted" +
+ "if the stream fails.")
+ .required(true)
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .defaultValue("5")
+ .build();
+ public static final PropertyDescriptor BACKOFF_TIME = new PropertyDescriptor.Builder()
+ .name("backoff-time")
+ .displayName("Backoff Time")
+ .description("The duration to backoff to start attempting a new stream if" +
+ "the current one fails for any reason. Will increase by factor of 2 every time a restart fails")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("1 mins")
+ .build();
+ public static final PropertyDescriptor MAXIMUM_BACKOFF_TIME = new PropertyDescriptor.Builder()
+ .name("maximum-backoff-time")
+ .displayName("Maximum Backoff Time")
+ .description("The maximum duration to backoff to start attempting a new stream." +
+ "It is recommended that this number be much higher than the 'Backoff Time' property")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("5 mins")
+ .build();
+ public static final PropertyDescriptor CONNECT_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("connect-timeout")
+ .displayName("Connect Timeout")
+ .description("The maximum time in which client should establish a connection with the " +
+ "Twitter API before a time out. A value of 0 will mean no timeout at all")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("10 secs")
+ .build();
+ public static final PropertyDescriptor READ_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("read-timeout")
+ .displayName("Read Timeout")
+ .description("The maximum time of inactivity between receiving tweets from Twitter through " +
+ "the API before a timeout. A value of 0 will mean no timeout at all")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("10 secs")
+ .build();
+ public static final PropertyDescriptor BACKFILL_MINUTES = new PropertyDescriptor.Builder()
+ .name("backfill-minutes")
+ .displayName("Backfill Minutes")
+ .description("The number of minutes (up to 5 minutes) of streaming data to be requested after a " +
+ "disconnect. Only available for project with academic research access. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/integrate/" +
+ "recovery-and-redundancy-features")
+ .required(true)
+ .defaultValue("0")
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("tweet-fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values include: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("user-fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user " +
+ "for proper usage. Possible field values include: " +
+ "created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, " +
+ "protected, public_metrics, url, username, verified, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor MEDIA_FIELDS = new PropertyDescriptor.Builder()
+ .name("media-fields")
+ .displayName("Media Fields")
+ .description("A comma-separated list of media fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/media " +
+ "for proper usage. Possible field values include: " +
+ "alt_text, duration_ms, height, media_key, non_public_metrics, organic_metrics, preview_image_url, " +
+ "promoted_metrics, public_metrics, type, url, width")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor POLL_FIELDS = new PropertyDescriptor.Builder()
+ .name("poll-fields")
+ .displayName("Poll Fields")
+ .description("A comma-separated list of poll fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/poll " +
+ "for proper usage. Possible field values include: " +
+ "duration_minutes, end_datetime, id, options, voting_status")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PLACE_FIELDS = new PropertyDescriptor.Builder()
+ .name("place-fields")
+ .displayName("Place Fields")
+ .description("A comma-separated list of place fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/place " +
+ "for proper usage. Possible field values include: " +
+ "contained_within, country, country_code, full_name, geo, id, name, place_type")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor EXPANSIONS = new PropertyDescriptor.Builder()
+ .name("expansions")
+ .displayName("Expansions")
+ .description("A comma-separated list of expansions for objects in the returned tweet. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/expansions " +
+ "for proper usage. Possible field values include: " +
+ "author_id, referenced_tweets.id, referenced_tweets.id.author_id, entities.mentions.username, " +
+ "attachments.poll_ids, attachments.media_keys ,in_reply_to_user_id, geo.place_id")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("FlowFiles containing an array of one or more Tweets")
+ .build();
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+
+ private TweetStreamService tweetStreamService;
+
+ private volatile BlockingQueue<String> messageQueue;
+
+ @Override
+ protected void init(ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptors = new ArrayList<>();
+ descriptors.add(ENDPOINT);
+ descriptors.add(BASE_PATH);
+ descriptors.add(BEARER_TOKEN);
+ descriptors.add(QUEUE_SIZE);
+ descriptors.add(BATCH_SIZE);
+ descriptors.add(BACKOFF_ATTEMPTS);
+ descriptors.add(BACKOFF_TIME);
+ descriptors.add(MAXIMUM_BACKOFF_TIME);
+ descriptors.add(CONNECT_TIMEOUT);
+ descriptors.add(READ_TIMEOUT);
+ descriptors.add(BACKFILL_MINUTES);
+ descriptors.add(TWEET_FIELDS);
+ descriptors.add(USER_FIELDS);
+ descriptors.add(MEDIA_FIELDS);
+ descriptors.add(POLL_FIELDS);
+ descriptors.add(PLACE_FIELDS);
+ descriptors.add(EXPANSIONS);
+
+ this.descriptors = Collections.unmodifiableList(descriptors);
+
+ final Set<Relationship> relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ this.relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return this.descriptors;
+ }
+
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ messageQueue = new LinkedBlockingQueue<>(context.getProperty(QUEUE_SIZE).asInteger());
+
+ tweetStreamService = new TweetStreamService(context, messageQueue, getLogger());
+ tweetStreamService.setBasePath(context.getProperty(BASE_PATH).getValue());
+ final String endpointName = context.getProperty(ENDPOINT).getValue();
+ if (ENDPOINT_SAMPLE.getValue().equals(endpointName)) {
+ tweetStreamService.start(StreamEndpoint.SAMPLE_ENDPOINT);
+ } else {
+ tweetStreamService.start(StreamEndpoint.SEARCH_ENDPOINT);
+ }
+ }
+
+ @Override
+ public void onTrigger(ProcessContext context, ProcessSession session) throws ProcessException {
+ if (messageQueue.isEmpty()) {
+ context.yield();
+ return;
+ }
+
+ FlowFile flowFile = session.create();
+ flowFile = session.write(flowFile, new OutputStreamCallback() {
+ @Override
+ public void process(OutputStream out) throws IOException {
+
+ final int batchSize = context.getProperty(BATCH_SIZE).asInteger();
+ String tweet = messageQueue.poll();
+ int tweetCount = 1;
Review Comment:
The `tweetCount` variable is not used. It is useful information, so recommend setting a FlowFile attribute named `tweets` with the number. That may involve the need to create an AtomicInteger outside of the callback, which can be updated and referenced after the callback completes.
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TweetStreamService.java:
##########
@@ -0,0 +1,222 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiClient;
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TweetStreamService {
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ScheduledExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Long backoffMultiplier;
+ private int backoffAttempts;
+ private int attemptCounter;
+ private Long backoffTime;
+ private Long maximumBackoff;
+
+ private StreamEndpoint endpoint;
+
+ public TweetStreamService(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ Objects.requireNonNull(context);
+ Objects.requireNonNull(queue);
+ Objects.requireNonNull(logger);
+
+ this.queue = queue;
+ this.logger = logger;
+
+ this.tweetFields = parseCommaSeparatedPropreties(context, ConsumeTwitter.TWEET_FIELDS);
+ this.userFields = parseCommaSeparatedPropreties(context, ConsumeTwitter.USER_FIELDS);
+ this.mediaFields = parseCommaSeparatedPropreties(context, ConsumeTwitter.MEDIA_FIELDS);
+ this.pollFields = parseCommaSeparatedPropreties(context, ConsumeTwitter.POLL_FIELDS);
+ this.placeFields = parseCommaSeparatedPropreties(context, ConsumeTwitter.PLACE_FIELDS);
+ this.expansions = parseCommaSeparatedPropreties(context, ConsumeTwitter.EXPANSIONS);
+ this.backfillMinutes = context.getProperty(ConsumeTwitter.BACKFILL_MINUTES).asInteger();
+
+ this.backoffMultiplier = 1L;
+ this.backoffAttempts = context.getProperty(ConsumeTwitter.BACKOFF_ATTEMPTS).asInteger();
+ this.attemptCounter = 0;
+ this.backoffTime = context.getProperty(ConsumeTwitter.BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+ this.maximumBackoff = context.getProperty(ConsumeTwitter.MAXIMUM_BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+
+ ApiClient client = new ApiClient();
+ final int connectTimeout = context.getProperty(ConsumeTwitter.CONNECT_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ final int readTimeout = context.getProperty(ConsumeTwitter.READ_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ client = client.setConnectTimeout(connectTimeout);
+ client = client.setReadTimeout(readTimeout);
+ api = new TwitterApi(client);
+
+ TwitterCredentialsBearer creds = new TwitterCredentialsBearer(context.getProperty(ConsumeTwitter.BEARER_TOKEN).getValue());
+ api.setTwitterCredentials(creds);
+
+ this.executorService = Executors.newSingleThreadScheduledExecutor();
+ }
+
+ public String getBasePath() {
+ return api.getApiClient().getBasePath();
+ }
+
+ public void setBasePath(final String path) {
+ api.getApiClient().setBasePath(path);
+ }
+
+ public String getTransitUri(final String endpoint) {
+ if (endpoint.equals(StreamEndpoint.SAMPLE_ENDPOINT.getEndpointName())) {
+ return api.getApiClient().getBasePath() + StreamEndpoint.SAMPLE_ENDPOINT.getPath();
+ } else if (endpoint.equals(StreamEndpoint.SEARCH_ENDPOINT.getEndpointName())) {
+ return api.getApiClient().getBasePath() + StreamEndpoint.SEARCH_ENDPOINT.getPath();
+ } else {
+ logger.warn("Unrecognized endpoint in getTransitUri. Returning basePath");
+ return api.getApiClient().getBasePath();
+ }
+ }
+
+ /**
+ * This method would be called when we would like the stream to get started. This method will spin off a thread that
+ * will continue to queue tweets on to the given queue passed in the constructor. The thread will continue
+ * to run until {@code stop} is called.
+ * @param endpoint {@code TwitterStreamAPI.SAMPLE_ENDPOINT} or {@code TwitterStreamAPI.SEARCH_ENDPOINT}
+ */
+ public void start(final StreamEndpoint endpoint) {
+ this.endpoint = endpoint;
+ executorService.execute(new TweetStreamStarter());
+ }
Review Comment:
As mentioned elsewhere, recommend refactoring to determine the endpoint in the constructor, and simplify this method:
```suggestion
public void start() {
executorService.execute(new TweetStreamStarter());
}
```
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,366 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.UUID;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue(StreamEndpoint.SAMPLE_ENDPOINT.getEndpointName(),
+ "Sample Stream",
+ "Streams about one percent of all Tweets. " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/volume-streams/api-reference/get-tweets-sample-stream");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue(StreamEndpoint.SEARCH_ENDPOINT.getEndpointName(),
+ "Search Stream",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty. " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/get-tweets-search-stream");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("stream-endpoint")
+ .displayName("Stream Endpoint")
+ .description("Specifies which endpoint tweets should be streamed. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("Maximum size of internal queue for streamed messages")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
Review Comment:
Recommend changing the default value to `1000`. Although most users may want to change this up or down, having a higher value is a better guideline in general.
```suggestion
.defaultValue("1000")
```
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TweetStreamService.java:
##########
@@ -0,0 +1,222 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiClient;
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TweetStreamService {
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ScheduledExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Long backoffMultiplier;
+ private int backoffAttempts;
+ private int attemptCounter;
+ private Long backoffTime;
+ private Long maximumBackoff;
+
+ private StreamEndpoint endpoint;
+
+ public TweetStreamService(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ Objects.requireNonNull(context);
+ Objects.requireNonNull(queue);
+ Objects.requireNonNull(logger);
+
+ this.queue = queue;
+ this.logger = logger;
+
+ this.tweetFields = parseCommaSeparatedPropreties(context, ConsumeTwitter.TWEET_FIELDS);
+ this.userFields = parseCommaSeparatedPropreties(context, ConsumeTwitter.USER_FIELDS);
+ this.mediaFields = parseCommaSeparatedPropreties(context, ConsumeTwitter.MEDIA_FIELDS);
+ this.pollFields = parseCommaSeparatedPropreties(context, ConsumeTwitter.POLL_FIELDS);
+ this.placeFields = parseCommaSeparatedPropreties(context, ConsumeTwitter.PLACE_FIELDS);
+ this.expansions = parseCommaSeparatedPropreties(context, ConsumeTwitter.EXPANSIONS);
+ this.backfillMinutes = context.getProperty(ConsumeTwitter.BACKFILL_MINUTES).asInteger();
+
+ this.backoffMultiplier = 1L;
+ this.backoffAttempts = context.getProperty(ConsumeTwitter.BACKOFF_ATTEMPTS).asInteger();
+ this.attemptCounter = 0;
+ this.backoffTime = context.getProperty(ConsumeTwitter.BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+ this.maximumBackoff = context.getProperty(ConsumeTwitter.MAXIMUM_BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+
+ ApiClient client = new ApiClient();
+ final int connectTimeout = context.getProperty(ConsumeTwitter.CONNECT_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ final int readTimeout = context.getProperty(ConsumeTwitter.READ_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ client = client.setConnectTimeout(connectTimeout);
+ client = client.setReadTimeout(readTimeout);
+ api = new TwitterApi(client);
+
+ TwitterCredentialsBearer creds = new TwitterCredentialsBearer(context.getProperty(ConsumeTwitter.BEARER_TOKEN).getValue());
+ api.setTwitterCredentials(creds);
+
+ this.executorService = Executors.newSingleThreadScheduledExecutor();
+ }
+
+ public String getBasePath() {
+ return api.getApiClient().getBasePath();
+ }
+
+ public void setBasePath(final String path) {
+ api.getApiClient().setBasePath(path);
+ }
+
+ public String getTransitUri(final String endpoint) {
+ if (endpoint.equals(StreamEndpoint.SAMPLE_ENDPOINT.getEndpointName())) {
+ return api.getApiClient().getBasePath() + StreamEndpoint.SAMPLE_ENDPOINT.getPath();
+ } else if (endpoint.equals(StreamEndpoint.SEARCH_ENDPOINT.getEndpointName())) {
+ return api.getApiClient().getBasePath() + StreamEndpoint.SEARCH_ENDPOINT.getPath();
+ } else {
+ logger.warn("Unrecognized endpoint in getTransitUri. Returning basePath");
+ return api.getApiClient().getBasePath();
+ }
+ }
+
+ /**
+ * This method would be called when we would like the stream to get started. This method will spin off a thread that
+ * will continue to queue tweets on to the given queue passed in the constructor. The thread will continue
+ * to run until {@code stop} is called.
+ * @param endpoint {@code TwitterStreamAPI.SAMPLE_ENDPOINT} or {@code TwitterStreamAPI.SEARCH_ENDPOINT}
+ */
+ public void start(final StreamEndpoint endpoint) {
+ this.endpoint = endpoint;
+ executorService.execute(new TweetStreamStarter());
+ }
+
+ /**
+ * This method would be called when we would like the stream to get stopped. The stream will be closed and the
+ * executorService will be shut down.
+ */
+ public void stop() {
+ if (stream != null) {
+ try {
+ stream.close();
+ } catch (IOException e) {
+ logger.error("Closing response stream failed", e);
+ }
+ }
+
+ executorService.shutdownNow();
+ }
+
+ private Long calculateBackoffDelay() {
+ long backoff = backoffMultiplier * backoffTime;
+ long delay = (backoff > maximumBackoff) ? maximumBackoff : backoff;
+ return delay;
+ }
+
+ private void scheduleStartStreamWithBackoff() {
+ // use exponential(by factor of 2) backoff in scheduling the next TweetStreamStarter
+ if (attemptCounter >= backoffAttempts) {
+ throw new ProcessException(String.format("Connection failed after maximum attempts [%d]", attemptCounter));
+ }
+ attemptCounter += 1;
+ long delay = calculateBackoffDelay();
+ backoffMultiplier *= 2;
+ logger.info("Scheduling new stream connection after delay [{} s]", delay);
+ executorService.schedule(new TweetStreamStarter(), delay, TimeUnit.SECONDS);
+ }
+
+ private void resetBackoff() {
+ attemptCounter = 0;
+ backoffMultiplier = 1L;
+ }
+
+ private class TweetStreamStarter implements Runnable {
+ @Override
+ public void run() {
+ try {
+ if (endpoint.equals(StreamEndpoint.SAMPLE_ENDPOINT)) {
+ stream = api.tweets().sampleStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else {
+ stream = api.tweets().searchStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ }
+ executorService.execute(new TweetStreamHandler());
+ } catch (ApiException e) {
+ stream = null;
+ logger.warn("Twitter Stream [{}] API connection failed: HTTP {}", endpoint.getEndpointName(), e.getCode(), e);
+ scheduleStartStreamWithBackoff();
+ } catch (Exception e) {
+ stream = null;
+ logger.warn("Twitter Stream [{}] connection failed", endpoint.getEndpointName(), e);
+ scheduleStartStreamWithBackoff();
+ }
+ }
+ }
+
+ private class TweetStreamHandler implements Runnable {
+ @Override
+ public void run() {
+ try (BufferedReader reader = new BufferedReader(new InputStreamReader(stream))) {
+ String tweetRecord = reader.readLine();
+ while (tweetRecord != null) {
+ queue.offer(tweetRecord);
Review Comment:
The `offer()` method can return `false`, indicating that the queue is full, which result in missing records. The `put()` method blocks, which seems like the best behavior for now, in order to avoid reading more from the stream until space is available in the queue. The `put()` method can throw an `InterruptedException`, signalling that processing is being terminated. Recommend adding a second catch block for that condition, and logging that adding Tweet to the queue failed.
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,366 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.UUID;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue(StreamEndpoint.SAMPLE_ENDPOINT.getEndpointName(),
+ "Sample Stream",
+ "Streams about one percent of all Tweets. " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/volume-streams/api-reference/get-tweets-sample-stream");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue(StreamEndpoint.SEARCH_ENDPOINT.getEndpointName(),
+ "Search Stream",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty. " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/get-tweets-search-stream");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("stream-endpoint")
+ .displayName("Stream Endpoint")
+ .description("Specifies which endpoint tweets should be streamed. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("Maximum size of internal queue for streamed messages")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor BACKOFF_ATTEMPTS = new PropertyDescriptor.Builder()
+ .name("backoff-attempts")
+ .displayName("Backoff Attempts")
+ .description("The number of reconnection tries the processor will attempt in the event of " +
+ "a disconnection of the stream for any reason, before throwing an exception. To start a stream after " +
+ "this exception occur and the connection is fixed, please stop and restart the processor. If the value" +
+ "of this property is 0, then backoff will never occur and the processor will always need to be restarted" +
+ "if the stream fails.")
+ .required(true)
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .defaultValue("5")
+ .build();
+ public static final PropertyDescriptor BACKOFF_TIME = new PropertyDescriptor.Builder()
+ .name("backoff-time")
+ .displayName("Backoff Time")
+ .description("The duration to backoff to start attempting a new stream if" +
+ "the current one fails for any reason. Will increase by factor of 2 every time a restart fails")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("1 mins")
+ .build();
+ public static final PropertyDescriptor MAXIMUM_BACKOFF_TIME = new PropertyDescriptor.Builder()
+ .name("maximum-backoff-time")
+ .displayName("Maximum Backoff Time")
+ .description("The maximum duration to backoff to start attempting a new stream." +
+ "It is recommended that this number be much higher than the 'Backoff Time' property")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("5 mins")
+ .build();
+ public static final PropertyDescriptor CONNECT_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("connect-timeout")
+ .displayName("Connect Timeout")
+ .description("The maximum time in which client should establish a connection with the " +
+ "Twitter API before a time out. A value of 0 will mean no timeout at all")
Review Comment:
```suggestion
"Twitter API before a time out. Setting the value to 0 disables connection timeouts.")
```
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,366 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.UUID;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue(StreamEndpoint.SAMPLE_ENDPOINT.getEndpointName(),
+ "Sample Stream",
+ "Streams about one percent of all Tweets. " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/volume-streams/api-reference/get-tweets-sample-stream");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue(StreamEndpoint.SEARCH_ENDPOINT.getEndpointName(),
+ "Search Stream",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty. " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/get-tweets-search-stream");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("stream-endpoint")
+ .displayName("Stream Endpoint")
+ .description("Specifies which endpoint tweets should be streamed. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("Maximum size of internal queue for streamed messages")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor BACKOFF_ATTEMPTS = new PropertyDescriptor.Builder()
+ .name("backoff-attempts")
+ .displayName("Backoff Attempts")
+ .description("The number of reconnection tries the processor will attempt in the event of " +
+ "a disconnection of the stream for any reason, before throwing an exception. To start a stream after " +
+ "this exception occur and the connection is fixed, please stop and restart the processor. If the value" +
+ "of this property is 0, then backoff will never occur and the processor will always need to be restarted" +
+ "if the stream fails.")
+ .required(true)
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .defaultValue("5")
+ .build();
+ public static final PropertyDescriptor BACKOFF_TIME = new PropertyDescriptor.Builder()
+ .name("backoff-time")
+ .displayName("Backoff Time")
+ .description("The duration to backoff to start attempting a new stream if" +
+ "the current one fails for any reason. Will increase by factor of 2 every time a restart fails")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("1 mins")
+ .build();
+ public static final PropertyDescriptor MAXIMUM_BACKOFF_TIME = new PropertyDescriptor.Builder()
+ .name("maximum-backoff-time")
+ .displayName("Maximum Backoff Time")
+ .description("The maximum duration to backoff to start attempting a new stream." +
+ "It is recommended that this number be much higher than the 'Backoff Time' property")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("5 mins")
+ .build();
+ public static final PropertyDescriptor CONNECT_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("connect-timeout")
+ .displayName("Connect Timeout")
+ .description("The maximum time in which client should establish a connection with the " +
+ "Twitter API before a time out. A value of 0 will mean no timeout at all")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("10 secs")
+ .build();
+ public static final PropertyDescriptor READ_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("read-timeout")
+ .displayName("Read Timeout")
+ .description("The maximum time of inactivity between receiving tweets from Twitter through " +
+ "the API before a timeout. A value of 0 will mean no timeout at all")
Review Comment:
```suggestion
"the API before a timeout. Setting the value to 0 disables read timeouts.")
```
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TweetStreamService.java:
##########
@@ -0,0 +1,222 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiClient;
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TweetStreamService {
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ScheduledExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Long backoffMultiplier;
+ private int backoffAttempts;
+ private int attemptCounter;
+ private Long backoffTime;
+ private Long maximumBackoff;
+
+ private StreamEndpoint endpoint;
+
+ public TweetStreamService(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ Objects.requireNonNull(context);
+ Objects.requireNonNull(queue);
+ Objects.requireNonNull(logger);
+
+ this.queue = queue;
+ this.logger = logger;
+
+ this.tweetFields = parseCommaSeparatedPropreties(context, ConsumeTwitter.TWEET_FIELDS);
+ this.userFields = parseCommaSeparatedPropreties(context, ConsumeTwitter.USER_FIELDS);
+ this.mediaFields = parseCommaSeparatedPropreties(context, ConsumeTwitter.MEDIA_FIELDS);
+ this.pollFields = parseCommaSeparatedPropreties(context, ConsumeTwitter.POLL_FIELDS);
+ this.placeFields = parseCommaSeparatedPropreties(context, ConsumeTwitter.PLACE_FIELDS);
+ this.expansions = parseCommaSeparatedPropreties(context, ConsumeTwitter.EXPANSIONS);
+ this.backfillMinutes = context.getProperty(ConsumeTwitter.BACKFILL_MINUTES).asInteger();
+
+ this.backoffMultiplier = 1L;
+ this.backoffAttempts = context.getProperty(ConsumeTwitter.BACKOFF_ATTEMPTS).asInteger();
+ this.attemptCounter = 0;
+ this.backoffTime = context.getProperty(ConsumeTwitter.BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+ this.maximumBackoff = context.getProperty(ConsumeTwitter.MAXIMUM_BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+
+ ApiClient client = new ApiClient();
+ final int connectTimeout = context.getProperty(ConsumeTwitter.CONNECT_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ final int readTimeout = context.getProperty(ConsumeTwitter.READ_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ client = client.setConnectTimeout(connectTimeout);
+ client = client.setReadTimeout(readTimeout);
+ api = new TwitterApi(client);
+
+ TwitterCredentialsBearer creds = new TwitterCredentialsBearer(context.getProperty(ConsumeTwitter.BEARER_TOKEN).getValue());
+ api.setTwitterCredentials(creds);
+
+ this.executorService = Executors.newSingleThreadScheduledExecutor();
+ }
+
+ public String getBasePath() {
+ return api.getApiClient().getBasePath();
+ }
+
+ public void setBasePath(final String path) {
+ api.getApiClient().setBasePath(path);
+ }
+
+ public String getTransitUri(final String endpoint) {
+ if (endpoint.equals(StreamEndpoint.SAMPLE_ENDPOINT.getEndpointName())) {
+ return api.getApiClient().getBasePath() + StreamEndpoint.SAMPLE_ENDPOINT.getPath();
+ } else if (endpoint.equals(StreamEndpoint.SEARCH_ENDPOINT.getEndpointName())) {
+ return api.getApiClient().getBasePath() + StreamEndpoint.SEARCH_ENDPOINT.getPath();
+ } else {
+ logger.warn("Unrecognized endpoint in getTransitUri. Returning basePath");
+ return api.getApiClient().getBasePath();
+ }
+ }
+
+ /**
+ * This method would be called when we would like the stream to get started. This method will spin off a thread that
+ * will continue to queue tweets on to the given queue passed in the constructor. The thread will continue
+ * to run until {@code stop} is called.
+ * @param endpoint {@code TwitterStreamAPI.SAMPLE_ENDPOINT} or {@code TwitterStreamAPI.SEARCH_ENDPOINT}
+ */
+ public void start(final StreamEndpoint endpoint) {
+ this.endpoint = endpoint;
+ executorService.execute(new TweetStreamStarter());
+ }
+
+ /**
+ * This method would be called when we would like the stream to get stopped. The stream will be closed and the
+ * executorService will be shut down.
+ */
+ public void stop() {
+ if (stream != null) {
+ try {
+ stream.close();
+ } catch (IOException e) {
+ logger.error("Closing response stream failed", e);
+ }
+ }
+
+ executorService.shutdownNow();
+ }
+
+ private Long calculateBackoffDelay() {
+ long backoff = backoffMultiplier * backoffTime;
+ long delay = (backoff > maximumBackoff) ? maximumBackoff : backoff;
+ return delay;
+ }
+
+ private void scheduleStartStreamWithBackoff() {
+ // use exponential(by factor of 2) backoff in scheduling the next TweetStreamStarter
+ if (attemptCounter >= backoffAttempts) {
+ throw new ProcessException(String.format("Connection failed after maximum attempts [%d]", attemptCounter));
+ }
+ attemptCounter += 1;
+ long delay = calculateBackoffDelay();
+ backoffMultiplier *= 2;
+ logger.info("Scheduling new stream connection after delay [{} s]", delay);
+ executorService.schedule(new TweetStreamStarter(), delay, TimeUnit.SECONDS);
+ }
+
+ private void resetBackoff() {
+ attemptCounter = 0;
+ backoffMultiplier = 1L;
+ }
+
+ private class TweetStreamStarter implements Runnable {
+ @Override
+ public void run() {
+ try {
+ if (endpoint.equals(StreamEndpoint.SAMPLE_ENDPOINT)) {
+ stream = api.tweets().sampleStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else {
+ stream = api.tweets().searchStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ }
+ executorService.execute(new TweetStreamHandler());
+ } catch (ApiException e) {
+ stream = null;
+ logger.warn("Twitter Stream [{}] API connection failed: HTTP {}", endpoint.getEndpointName(), e.getCode(), e);
+ scheduleStartStreamWithBackoff();
+ } catch (Exception e) {
+ stream = null;
+ logger.warn("Twitter Stream [{}] connection failed", endpoint.getEndpointName(), e);
+ scheduleStartStreamWithBackoff();
+ }
+ }
+ }
+
+ private class TweetStreamHandler implements Runnable {
+ @Override
+ public void run() {
+ try (BufferedReader reader = new BufferedReader(new InputStreamReader(stream))) {
+ String tweetRecord = reader.readLine();
+ while (tweetRecord != null) {
+ queue.offer(tweetRecord);
+
+ // reset backoff multiplier upon successful receipt of a tweet
+ resetBackoff();
+
+ tweetRecord = reader.readLine();
+ }
+ } catch (IOException e) {
+ logger.info("Stream is closed or has stopped", e);
+ }
+ logger.info("Tweet stream processing completed");
+ scheduleStartStreamWithBackoff();
+ }
+ }
+
+ private Set<String> parseCommaSeparatedPropreties(final ProcessContext context, final PropertyDescriptor property) {
Review Comment:
Minor wording correction:
```suggestion
private Set<String> parseCommaSeparatedProperties(final ProcessContext context, final PropertyDescriptor property) {
```
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TweetStreamService.java:
##########
@@ -0,0 +1,222 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiClient;
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TweetStreamService {
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ScheduledExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Long backoffMultiplier;
+ private int backoffAttempts;
+ private int attemptCounter;
+ private Long backoffTime;
+ private Long maximumBackoff;
+
+ private StreamEndpoint endpoint;
+
+ public TweetStreamService(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ Objects.requireNonNull(context);
+ Objects.requireNonNull(queue);
+ Objects.requireNonNull(logger);
+
+ this.queue = queue;
+ this.logger = logger;
+
+ this.tweetFields = parseCommaSeparatedPropreties(context, ConsumeTwitter.TWEET_FIELDS);
+ this.userFields = parseCommaSeparatedPropreties(context, ConsumeTwitter.USER_FIELDS);
+ this.mediaFields = parseCommaSeparatedPropreties(context, ConsumeTwitter.MEDIA_FIELDS);
+ this.pollFields = parseCommaSeparatedPropreties(context, ConsumeTwitter.POLL_FIELDS);
+ this.placeFields = parseCommaSeparatedPropreties(context, ConsumeTwitter.PLACE_FIELDS);
+ this.expansions = parseCommaSeparatedPropreties(context, ConsumeTwitter.EXPANSIONS);
+ this.backfillMinutes = context.getProperty(ConsumeTwitter.BACKFILL_MINUTES).asInteger();
+
+ this.backoffMultiplier = 1L;
+ this.backoffAttempts = context.getProperty(ConsumeTwitter.BACKOFF_ATTEMPTS).asInteger();
+ this.attemptCounter = 0;
+ this.backoffTime = context.getProperty(ConsumeTwitter.BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+ this.maximumBackoff = context.getProperty(ConsumeTwitter.MAXIMUM_BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+
+ ApiClient client = new ApiClient();
+ final int connectTimeout = context.getProperty(ConsumeTwitter.CONNECT_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ final int readTimeout = context.getProperty(ConsumeTwitter.READ_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ client = client.setConnectTimeout(connectTimeout);
+ client = client.setReadTimeout(readTimeout);
+ api = new TwitterApi(client);
+
+ TwitterCredentialsBearer creds = new TwitterCredentialsBearer(context.getProperty(ConsumeTwitter.BEARER_TOKEN).getValue());
+ api.setTwitterCredentials(creds);
+
+ this.executorService = Executors.newSingleThreadScheduledExecutor();
+ }
+
+ public String getBasePath() {
+ return api.getApiClient().getBasePath();
+ }
+
+ public void setBasePath(final String path) {
+ api.getApiClient().setBasePath(path);
+ }
+
+ public String getTransitUri(final String endpoint) {
+ if (endpoint.equals(StreamEndpoint.SAMPLE_ENDPOINT.getEndpointName())) {
+ return api.getApiClient().getBasePath() + StreamEndpoint.SAMPLE_ENDPOINT.getPath();
+ } else if (endpoint.equals(StreamEndpoint.SEARCH_ENDPOINT.getEndpointName())) {
+ return api.getApiClient().getBasePath() + StreamEndpoint.SEARCH_ENDPOINT.getPath();
+ } else {
+ logger.warn("Unrecognized endpoint in getTransitUri. Returning basePath");
+ return api.getApiClient().getBasePath();
+ }
+ }
+
+ /**
+ * This method would be called when we would like the stream to get started. This method will spin off a thread that
+ * will continue to queue tweets on to the given queue passed in the constructor. The thread will continue
+ * to run until {@code stop} is called.
+ * @param endpoint {@code TwitterStreamAPI.SAMPLE_ENDPOINT} or {@code TwitterStreamAPI.SEARCH_ENDPOINT}
+ */
+ public void start(final StreamEndpoint endpoint) {
+ this.endpoint = endpoint;
+ executorService.execute(new TweetStreamStarter());
+ }
+
+ /**
+ * This method would be called when we would like the stream to get stopped. The stream will be closed and the
+ * executorService will be shut down.
+ */
+ public void stop() {
+ if (stream != null) {
+ try {
+ stream.close();
+ } catch (IOException e) {
+ logger.error("Closing response stream failed", e);
+ }
+ }
+
+ executorService.shutdownNow();
+ }
+
+ private Long calculateBackoffDelay() {
+ long backoff = backoffMultiplier * backoffTime;
+ long delay = (backoff > maximumBackoff) ? maximumBackoff : backoff;
+ return delay;
+ }
+
+ private void scheduleStartStreamWithBackoff() {
+ // use exponential(by factor of 2) backoff in scheduling the next TweetStreamStarter
+ if (attemptCounter >= backoffAttempts) {
+ throw new ProcessException(String.format("Connection failed after maximum attempts [%d]", attemptCounter));
+ }
+ attemptCounter += 1;
+ long delay = calculateBackoffDelay();
+ backoffMultiplier *= 2;
+ logger.info("Scheduling new stream connection after delay [{} s]", delay);
+ executorService.schedule(new TweetStreamStarter(), delay, TimeUnit.SECONDS);
+ }
+
+ private void resetBackoff() {
+ attemptCounter = 0;
+ backoffMultiplier = 1L;
+ }
+
+ private class TweetStreamStarter implements Runnable {
+ @Override
+ public void run() {
+ try {
+ if (endpoint.equals(StreamEndpoint.SAMPLE_ENDPOINT)) {
+ stream = api.tweets().sampleStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else {
+ stream = api.tweets().searchStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ }
+ executorService.execute(new TweetStreamHandler());
+ } catch (ApiException e) {
+ stream = null;
+ logger.warn("Twitter Stream [{}] API connection failed: HTTP {}", endpoint.getEndpointName(), e.getCode(), e);
Review Comment:
Unfortunately the `ApiException` does not include the response body as part of the standard message, so adjusting the log to include it would be helpful:
```suggestion
logger.warn("Twitter Stream [{}] API connection failed: HTTP {} {}", endpoint.getEndpointName(), e.getCode(), e.getResponseBody(), e);
```
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,366 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.UUID;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue(StreamEndpoint.SAMPLE_ENDPOINT.getEndpointName(),
+ "Sample Stream",
+ "Streams about one percent of all Tweets. " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/volume-streams/api-reference/get-tweets-sample-stream");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue(StreamEndpoint.SEARCH_ENDPOINT.getEndpointName(),
+ "Search Stream",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty. " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/get-tweets-search-stream");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("stream-endpoint")
+ .displayName("Stream Endpoint")
+ .description("Specifies which endpoint tweets should be streamed. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
Review Comment:
```suggestion
.description("The base path that the processor will use for making HTTP requests. " +
"The default value should be sufficient for most use cases.")
```
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,366 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.UUID;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue(StreamEndpoint.SAMPLE_ENDPOINT.getEndpointName(),
+ "Sample Stream",
+ "Streams about one percent of all Tweets. " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/volume-streams/api-reference/get-tweets-sample-stream");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue(StreamEndpoint.SEARCH_ENDPOINT.getEndpointName(),
+ "Search Stream",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty. " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/get-tweets-search-stream");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("stream-endpoint")
+ .displayName("Stream Endpoint")
+ .description("Specifies which endpoint tweets should be streamed. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("Maximum size of internal queue for streamed messages")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor BACKOFF_ATTEMPTS = new PropertyDescriptor.Builder()
+ .name("backoff-attempts")
+ .displayName("Backoff Attempts")
+ .description("The number of reconnection tries the processor will attempt in the event of " +
+ "a disconnection of the stream for any reason, before throwing an exception. To start a stream after " +
+ "this exception occur and the connection is fixed, please stop and restart the processor. If the value" +
+ "of this property is 0, then backoff will never occur and the processor will always need to be restarted" +
+ "if the stream fails.")
+ .required(true)
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .defaultValue("5")
+ .build();
+ public static final PropertyDescriptor BACKOFF_TIME = new PropertyDescriptor.Builder()
+ .name("backoff-time")
+ .displayName("Backoff Time")
+ .description("The duration to backoff to start attempting a new stream if" +
+ "the current one fails for any reason. Will increase by factor of 2 every time a restart fails")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("1 mins")
+ .build();
+ public static final PropertyDescriptor MAXIMUM_BACKOFF_TIME = new PropertyDescriptor.Builder()
+ .name("maximum-backoff-time")
+ .displayName("Maximum Backoff Time")
+ .description("The maximum duration to backoff to start attempting a new stream." +
+ "It is recommended that this number be much higher than the 'Backoff Time' property")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("5 mins")
+ .build();
+ public static final PropertyDescriptor CONNECT_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("connect-timeout")
+ .displayName("Connect Timeout")
+ .description("The maximum time in which client should establish a connection with the " +
+ "Twitter API before a time out. A value of 0 will mean no timeout at all")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("10 secs")
+ .build();
+ public static final PropertyDescriptor READ_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("read-timeout")
+ .displayName("Read Timeout")
+ .description("The maximum time of inactivity between receiving tweets from Twitter through " +
+ "the API before a timeout. A value of 0 will mean no timeout at all")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("10 secs")
+ .build();
+ public static final PropertyDescriptor BACKFILL_MINUTES = new PropertyDescriptor.Builder()
+ .name("backfill-minutes")
+ .displayName("Backfill Minutes")
+ .description("The number of minutes (up to 5 minutes) of streaming data to be requested after a " +
+ "disconnect. Only available for project with academic research access. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/integrate/" +
+ "recovery-and-redundancy-features")
+ .required(true)
+ .defaultValue("0")
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("tweet-fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values include: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("user-fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user " +
+ "for proper usage. Possible field values include: " +
+ "created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, " +
+ "protected, public_metrics, url, username, verified, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor MEDIA_FIELDS = new PropertyDescriptor.Builder()
+ .name("media-fields")
+ .displayName("Media Fields")
+ .description("A comma-separated list of media fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/media " +
+ "for proper usage. Possible field values include: " +
+ "alt_text, duration_ms, height, media_key, non_public_metrics, organic_metrics, preview_image_url, " +
+ "promoted_metrics, public_metrics, type, url, width")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor POLL_FIELDS = new PropertyDescriptor.Builder()
+ .name("poll-fields")
+ .displayName("Poll Fields")
+ .description("A comma-separated list of poll fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/poll " +
+ "for proper usage. Possible field values include: " +
+ "duration_minutes, end_datetime, id, options, voting_status")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PLACE_FIELDS = new PropertyDescriptor.Builder()
+ .name("place-fields")
+ .displayName("Place Fields")
+ .description("A comma-separated list of place fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/place " +
+ "for proper usage. Possible field values include: " +
+ "contained_within, country, country_code, full_name, geo, id, name, place_type")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor EXPANSIONS = new PropertyDescriptor.Builder()
+ .name("expansions")
+ .displayName("Expansions")
+ .description("A comma-separated list of expansions for objects in the returned tweet. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/expansions " +
+ "for proper usage. Possible field values include: " +
+ "author_id, referenced_tweets.id, referenced_tweets.id.author_id, entities.mentions.username, " +
+ "attachments.poll_ids, attachments.media_keys ,in_reply_to_user_id, geo.place_id")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("FlowFiles containing an array of one or more Tweets")
+ .build();
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+
+ private TweetStreamService tweetStreamService;
+
+ private volatile BlockingQueue<String> messageQueue;
+
+ @Override
+ protected void init(ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptors = new ArrayList<>();
+ descriptors.add(ENDPOINT);
+ descriptors.add(BASE_PATH);
+ descriptors.add(BEARER_TOKEN);
+ descriptors.add(QUEUE_SIZE);
+ descriptors.add(BATCH_SIZE);
+ descriptors.add(BACKOFF_ATTEMPTS);
+ descriptors.add(BACKOFF_TIME);
+ descriptors.add(MAXIMUM_BACKOFF_TIME);
+ descriptors.add(CONNECT_TIMEOUT);
+ descriptors.add(READ_TIMEOUT);
+ descriptors.add(BACKFILL_MINUTES);
+ descriptors.add(TWEET_FIELDS);
+ descriptors.add(USER_FIELDS);
+ descriptors.add(MEDIA_FIELDS);
+ descriptors.add(POLL_FIELDS);
+ descriptors.add(PLACE_FIELDS);
+ descriptors.add(EXPANSIONS);
+
+ this.descriptors = Collections.unmodifiableList(descriptors);
+
+ final Set<Relationship> relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ this.relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return this.descriptors;
+ }
+
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ messageQueue = new LinkedBlockingQueue<>(context.getProperty(QUEUE_SIZE).asInteger());
+
+ tweetStreamService = new TweetStreamService(context, messageQueue, getLogger());
+ tweetStreamService.setBasePath(context.getProperty(BASE_PATH).getValue());
+ final String endpointName = context.getProperty(ENDPOINT).getValue();
Review Comment:
The base path can be set in the constructor instead of a separate `setBasePath()` method.
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,366 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.UUID;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue(StreamEndpoint.SAMPLE_ENDPOINT.getEndpointName(),
+ "Sample Stream",
+ "Streams about one percent of all Tweets. " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/volume-streams/api-reference/get-tweets-sample-stream");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue(StreamEndpoint.SEARCH_ENDPOINT.getEndpointName(),
+ "Search Stream",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty. " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/get-tweets-search-stream");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("stream-endpoint")
+ .displayName("Stream Endpoint")
+ .description("Specifies which endpoint tweets should be streamed. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("Maximum size of internal queue for streamed messages")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor BACKOFF_ATTEMPTS = new PropertyDescriptor.Builder()
+ .name("backoff-attempts")
+ .displayName("Backoff Attempts")
+ .description("The number of reconnection tries the processor will attempt in the event of " +
+ "a disconnection of the stream for any reason, before throwing an exception. To start a stream after " +
+ "this exception occur and the connection is fixed, please stop and restart the processor. If the value" +
+ "of this property is 0, then backoff will never occur and the processor will always need to be restarted" +
+ "if the stream fails.")
+ .required(true)
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .defaultValue("5")
+ .build();
+ public static final PropertyDescriptor BACKOFF_TIME = new PropertyDescriptor.Builder()
+ .name("backoff-time")
+ .displayName("Backoff Time")
+ .description("The duration to backoff to start attempting a new stream if" +
Review Comment:
Additional space needed:
```suggestion
.description("The duration to backoff before requesting a new stream if " +
```
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,366 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.UUID;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue(StreamEndpoint.SAMPLE_ENDPOINT.getEndpointName(),
+ "Sample Stream",
+ "Streams about one percent of all Tweets. " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/volume-streams/api-reference/get-tweets-sample-stream");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue(StreamEndpoint.SEARCH_ENDPOINT.getEndpointName(),
+ "Search Stream",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty. " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/get-tweets-search-stream");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("stream-endpoint")
+ .displayName("Stream Endpoint")
+ .description("Specifies which endpoint tweets should be streamed. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("Maximum size of internal queue for streamed messages")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor BACKOFF_ATTEMPTS = new PropertyDescriptor.Builder()
+ .name("backoff-attempts")
+ .displayName("Backoff Attempts")
+ .description("The number of reconnection tries the processor will attempt in the event of " +
+ "a disconnection of the stream for any reason, before throwing an exception. To start a stream after " +
+ "this exception occur and the connection is fixed, please stop and restart the processor. If the value" +
+ "of this property is 0, then backoff will never occur and the processor will always need to be restarted" +
+ "if the stream fails.")
+ .required(true)
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .defaultValue("5")
+ .build();
+ public static final PropertyDescriptor BACKOFF_TIME = new PropertyDescriptor.Builder()
+ .name("backoff-time")
+ .displayName("Backoff Time")
+ .description("The duration to backoff to start attempting a new stream if" +
+ "the current one fails for any reason. Will increase by factor of 2 every time a restart fails")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("1 mins")
+ .build();
+ public static final PropertyDescriptor MAXIMUM_BACKOFF_TIME = new PropertyDescriptor.Builder()
+ .name("maximum-backoff-time")
+ .displayName("Maximum Backoff Time")
+ .description("The maximum duration to backoff to start attempting a new stream." +
+ "It is recommended that this number be much higher than the 'Backoff Time' property")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("5 mins")
+ .build();
+ public static final PropertyDescriptor CONNECT_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("connect-timeout")
+ .displayName("Connect Timeout")
+ .description("The maximum time in which client should establish a connection with the " +
+ "Twitter API before a time out. A value of 0 will mean no timeout at all")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("10 secs")
+ .build();
+ public static final PropertyDescriptor READ_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("read-timeout")
+ .displayName("Read Timeout")
+ .description("The maximum time of inactivity between receiving tweets from Twitter through " +
+ "the API before a timeout. A value of 0 will mean no timeout at all")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("10 secs")
+ .build();
+ public static final PropertyDescriptor BACKFILL_MINUTES = new PropertyDescriptor.Builder()
+ .name("backfill-minutes")
+ .displayName("Backfill Minutes")
+ .description("The number of minutes (up to 5 minutes) of streaming data to be requested after a " +
+ "disconnect. Only available for project with academic research access. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/integrate/" +
+ "recovery-and-redundancy-features")
+ .required(true)
+ .defaultValue("0")
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("tweet-fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values include: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("user-fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user " +
+ "for proper usage. Possible field values include: " +
+ "created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, " +
+ "protected, public_metrics, url, username, verified, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor MEDIA_FIELDS = new PropertyDescriptor.Builder()
+ .name("media-fields")
+ .displayName("Media Fields")
+ .description("A comma-separated list of media fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/media " +
+ "for proper usage. Possible field values include: " +
+ "alt_text, duration_ms, height, media_key, non_public_metrics, organic_metrics, preview_image_url, " +
+ "promoted_metrics, public_metrics, type, url, width")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor POLL_FIELDS = new PropertyDescriptor.Builder()
+ .name("poll-fields")
+ .displayName("Poll Fields")
+ .description("A comma-separated list of poll fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/poll " +
+ "for proper usage. Possible field values include: " +
+ "duration_minutes, end_datetime, id, options, voting_status")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PLACE_FIELDS = new PropertyDescriptor.Builder()
+ .name("place-fields")
+ .displayName("Place Fields")
+ .description("A comma-separated list of place fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/place " +
+ "for proper usage. Possible field values include: " +
+ "contained_within, country, country_code, full_name, geo, id, name, place_type")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor EXPANSIONS = new PropertyDescriptor.Builder()
+ .name("expansions")
+ .displayName("Expansions")
+ .description("A comma-separated list of expansions for objects in the returned tweet. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/expansions " +
+ "for proper usage. Possible field values include: " +
+ "author_id, referenced_tweets.id, referenced_tweets.id.author_id, entities.mentions.username, " +
+ "attachments.poll_ids, attachments.media_keys ,in_reply_to_user_id, geo.place_id")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("FlowFiles containing an array of one or more Tweets")
+ .build();
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+
+ private TweetStreamService tweetStreamService;
+
+ private volatile BlockingQueue<String> messageQueue;
+
+ @Override
+ protected void init(ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptors = new ArrayList<>();
+ descriptors.add(ENDPOINT);
+ descriptors.add(BASE_PATH);
+ descriptors.add(BEARER_TOKEN);
+ descriptors.add(QUEUE_SIZE);
+ descriptors.add(BATCH_SIZE);
+ descriptors.add(BACKOFF_ATTEMPTS);
+ descriptors.add(BACKOFF_TIME);
+ descriptors.add(MAXIMUM_BACKOFF_TIME);
+ descriptors.add(CONNECT_TIMEOUT);
+ descriptors.add(READ_TIMEOUT);
+ descriptors.add(BACKFILL_MINUTES);
+ descriptors.add(TWEET_FIELDS);
+ descriptors.add(USER_FIELDS);
+ descriptors.add(MEDIA_FIELDS);
+ descriptors.add(POLL_FIELDS);
+ descriptors.add(PLACE_FIELDS);
+ descriptors.add(EXPANSIONS);
+
+ this.descriptors = Collections.unmodifiableList(descriptors);
+
+ final Set<Relationship> relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ this.relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return this.descriptors;
+ }
+
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ messageQueue = new LinkedBlockingQueue<>(context.getProperty(QUEUE_SIZE).asInteger());
+
+ tweetStreamService = new TweetStreamService(context, messageQueue, getLogger());
+ tweetStreamService.setBasePath(context.getProperty(BASE_PATH).getValue());
+ final String endpointName = context.getProperty(ENDPOINT).getValue();
+ if (ENDPOINT_SAMPLE.getValue().equals(endpointName)) {
+ tweetStreamService.start(StreamEndpoint.SAMPLE_ENDPOINT);
+ } else {
+ tweetStreamService.start(StreamEndpoint.SEARCH_ENDPOINT);
+ }
Review Comment:
Instead of passing the endpoint value to `start()`, the property value can be evaluated and set in the `TweetStreamService` constructor, which will simplify this block:
```suggestion
tweetStreamService.start();
```
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,366 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.UUID;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue(StreamEndpoint.SAMPLE_ENDPOINT.getEndpointName(),
+ "Sample Stream",
+ "Streams about one percent of all Tweets. " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/volume-streams/api-reference/get-tweets-sample-stream");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue(StreamEndpoint.SEARCH_ENDPOINT.getEndpointName(),
+ "Search Stream",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty. " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/get-tweets-search-stream");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("stream-endpoint")
+ .displayName("Stream Endpoint")
+ .description("Specifies which endpoint tweets should be streamed. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("Maximum size of internal queue for streamed messages")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
Review Comment:
Additional space needed:
```suggestion
.description("The maximum size of the number of Tweets to be written to a single FlowFile. " +
```
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/test/java/org/apache/nifi/processors/twitter/TestConsumeTwitter.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import okhttp3.mockwebserver.MockResponse;
+import okhttp3.mockwebserver.MockWebServer;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+
+public class TestConsumeTwitter {
+ private MockWebServer mockWebServer;
+
+ private TestRunner runner;
+
+ private final String SAMPLE_TWEET = "{\"data\":{\"id\":\"123\",\"text\":\"This is a sample tweet and is not real!\"}}";
+ private final String EXPECTED_TWEET = "[{\"data\":{\"id\":\"123\",\"text\":\"This is a sample tweet and is not real!\"}}]";
+
+ @BeforeEach
+ public void setRunnerAndAPI() {
+ mockWebServer = new MockWebServer();
+
+ runner = TestRunners.newTestRunner(ConsumeTwitter.class);
+
+ runner.setProperty(ConsumeTwitter.BEARER_TOKEN, "BEARER_TOKEN");
+ final String basePath = mockWebServer.url("").toString();
+ runner.setProperty(ConsumeTwitter.BASE_PATH, basePath);
+ }
+
+ @AfterEach
+ public void shutdownServerAndAPI() throws IOException {
+ mockWebServer.shutdown();
+ }
+
+ @Test
+ public void testReceiveSingleTweetInStream() throws InterruptedException {
+ MockResponse response = new MockResponse()
+ .setResponseCode(200)
+ .setBody(SAMPLE_TWEET)
+ .addHeader("Content-Type", "application/json");
+ mockWebServer.enqueue(response);
+
+
+ runner.setProperty(ConsumeTwitter.ENDPOINT, ConsumeTwitter.ENDPOINT_SAMPLE);
+ runner.setProperty(ConsumeTwitter.QUEUE_SIZE, "10000");
+ runner.setProperty(ConsumeTwitter.BATCH_SIZE, "10");
+ runner.setProperty(ConsumeTwitter.BACKFILL_MINUTES, "0");
+
+ runner.assertValid();
+
+ // the TwitterStreamAPI class spins up another thread and might not be done queueing tweets in one run of the processor
+ final int maxTries = 3;
+ final long runSchedule = 250;
+ runner.setRunSchedule(runSchedule);
+ runner.run(maxTries, false, true);
+ runner.stop();
+
+ // there should only be a single FlowFile containing a tweet
+ runner.assertTransferCount(ConsumeTwitter.REL_SUCCESS, 1);
+ MockFlowFile flowFile = runner.getFlowFilesForRelationship(ConsumeTwitter.REL_SUCCESS).get(0);
+ flowFile.assertContentEquals(EXPECTED_TWEET);
+ flowFile.assertAttributeEquals(CoreAttributes.MIME_TYPE.key(), "application/json");
Review Comment:
After adding a `tweets` attribute, recommend adding a check to assert the expected number of Tweets returned.
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,366 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.UUID;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue(StreamEndpoint.SAMPLE_ENDPOINT.getEndpointName(),
+ "Sample Stream",
+ "Streams about one percent of all Tweets. " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/volume-streams/api-reference/get-tweets-sample-stream");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue(StreamEndpoint.SEARCH_ENDPOINT.getEndpointName(),
+ "Search Stream",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty. " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/get-tweets-search-stream");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("stream-endpoint")
+ .displayName("Stream Endpoint")
+ .description("Specifies which endpoint tweets should be streamed. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("Maximum size of internal queue for streamed messages")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
Review Comment:
```suggestion
"Will write fewer Tweets based on the number available in the queue at the time of processor invocation.")
```
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,366 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.UUID;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
Review Comment:
The newlines should be removed.
```suggestion
+ "Tweet object model."
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r848780791
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TwitterStreamAPI.java:
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+public class TwitterStreamAPI {
+ public static final String SEARCH_ENDPOINT = "Search Endpoint";
+ public static final String SAMPLE_ENDPOINT = "Sample Endpoint";
+ public static final String SAMPLE_PATH = "/2/tweets/sample/stream";
+ public static final String SEARCH_PATH = "/2/tweets/search/stream";
+
+ private static final String BEARER_TOKEN_PROPERTY_NAME = "bearer-token";
+ private static final String TWEET_FIELDS_PROPERTY_NAME = "tweet-fields";
+ private static final String USER_FIELDS_PROPERTY_NAME = "user-fields";
+ private static final String MEDIA_FIELDS_PROPERTY_NAME = "media-fields";
+ private static final String POLL_FIELDS_PROPERTY_NAME = "poll-fields";
+ private static final String PLACE_FIELDS_PROPERTY_NAME = "place-fields";
+ private static final String EXPANSIONS_PROPERTY_NAME = "expansions";
+ private static final String BACKFILL_MINUTES_PROPERTY_NAME = "backfill-minutes";
+
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Set<String> parseCommaSeparatedListPropreties(final ProcessContext context, final String propertyName) {
+ Set<String> fields = null;
+ if (context.getProperty(propertyName).isSet()) {
+ fields = new HashSet<>();
+ final String fieldsString = context.getProperty(propertyName).getValue();
+ for (final String field: fieldsString.split(",")) {
+ fields.add(field.trim());
+ }
+ }
+ return fields;
+ }
+
+ public TwitterStreamAPI(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ assert context != null;
+ assert queue != null;
+ assert logger != null;
+
+ this.queue = queue;
+ this.logger = logger;
+
+ this.tweetFields = parseCommaSeparatedListPropreties(context, TWEET_FIELDS_PROPERTY_NAME);
+ this.userFields = parseCommaSeparatedListPropreties(context, USER_FIELDS_PROPERTY_NAME);
+ this.mediaFields = parseCommaSeparatedListPropreties(context, MEDIA_FIELDS_PROPERTY_NAME);
+ this.pollFields = parseCommaSeparatedListPropreties(context, POLL_FIELDS_PROPERTY_NAME);
+ this.placeFields = parseCommaSeparatedListPropreties(context, PLACE_FIELDS_PROPERTY_NAME);
+ this.expansions = parseCommaSeparatedListPropreties(context, EXPANSIONS_PROPERTY_NAME);
+ this.backfillMinutes = context.getProperty(BACKFILL_MINUTES_PROPERTY_NAME).asInteger();
+
+ TwitterCredentialsBearer creds = new TwitterCredentialsBearer(context.getProperty(BEARER_TOKEN_PROPERTY_NAME).getValue());
+ api = new TwitterApi();
+ api.setTwitterCredentials(creds);
+
+ this.executorService = Executors.newSingleThreadExecutor();
+ }
+
+ public String getBasePath() {
+ return api.getApiClient().getBasePath();
+ }
+
+ public void setBasePath(final String path) {
+ api.getApiClient().setBasePath(path);
+ }
+
+ /**
+ * This method would be called when we would like the stream to get started. This method will spin off a thread that
+ * will continue to queue tweets on to the given queue passed in the constructor. The thread will continue
+ * to run until {@code stop} is called.
+ * @param endpoint {@code TwitterStreamAPI.SAMPLE_ENDPOINT} or {@code TwitterStreamAPI.SEARCH_ENDPOINT}
+ */
+ public void start(final String endpoint) {
+ try {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ stream = api.tweets().sampleStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else if (endpoint.equals(SEARCH_ENDPOINT)) {
+ stream = api.tweets().searchStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else {
+ throw new ProcessException("Endpoint was invalid value: " + endpoint);
+ }
+ } catch (ApiException e) {
+ logger.error("Received error {}: {}", e.getCode(), e.getMessage());
+ }
+
+ executorService.execute(new TweetQueuer());
+ }
+
+ /**
+ * This method would be called when we would like the stream to get stopped. The stream will be closed and the
+ * executorService will be shut down. If it fails to shutdown, then it will be forcefully terminated.
+ */
+ public void stop() {
+ try {
+ stream.close();
+ } catch (IOException e) {
+ logger.error("IOException occurred while closing stream: {}", e.getMessage());
+ }
+
+ executorService.shutdown();
+ try {
+ if (!executorService.awaitTermination(3, TimeUnit.SECONDS)) {
+ executorService.shutdownNow();
+ }
+ } catch (InterruptedException e) {
+ executorService.shutdownNow();
+ logger.error("InterruptedException occurred while waiting for executor service to shut down: {}", e.getMessage());
+ }
+ }
+
+ private class TweetQueuer implements Runnable {
+ @Override
+ public void run() {
+ try (BufferedReader reader = new BufferedReader(new InputStreamReader(stream))) {
+ String tweetLine = reader.readLine();
+ while (tweetLine != null) {
+ queue.offer(tweetLine);
+ try {
+ tweetLine = reader.readLine();
+ } catch (IOException e) {
+ logger.info("Stream closed, no more tweets to read, exiting Queuer");
Review Comment:
Making the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on PR #5947:
URL: https://github.com/apache/nifi/pull/5947#issuecomment-1093743848
Awaiting further discussion on `TestConsumeTwitter` before making this PR ready for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r861126942
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.UUID;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
Review Comment:
Making the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r861143658
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TweetStreamService.java:
##########
@@ -0,0 +1,242 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiClient;
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TweetStreamService {
+ public static final String SEARCH_ENDPOINT = "Search Endpoint";
+ public static final String SAMPLE_ENDPOINT = "Sample Endpoint";
+ private static final String SAMPLE_PATH = "/2/tweets/sample/stream";
+ private static final String SEARCH_PATH = "/2/tweets/search/stream";
+
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ScheduledExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Long backoffMultiplier;
+ private int backoffAttempts;
+ private int attemptCounter;
+ private Long backoffTime;
+ private Long maximumBackoff;
+
+ private String endpoint;
+
+ private Set<String> parseCommaSeparatedListPropreties(final ProcessContext context, final PropertyDescriptor property) {
Review Comment:
Making the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r864035627
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,366 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.UUID;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue(StreamEndpoint.SAMPLE_ENDPOINT.getEndpointName(),
+ "Sample Stream",
+ "Streams about one percent of all Tweets. " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/volume-streams/api-reference/get-tweets-sample-stream");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue(StreamEndpoint.SEARCH_ENDPOINT.getEndpointName(),
+ "Search Stream",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty. " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/get-tweets-search-stream");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("stream-endpoint")
+ .displayName("Stream Endpoint")
+ .description("Specifies which endpoint tweets should be streamed. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("Maximum size of internal queue for streamed messages")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor BACKOFF_ATTEMPTS = new PropertyDescriptor.Builder()
+ .name("backoff-attempts")
+ .displayName("Backoff Attempts")
+ .description("The number of reconnection tries the processor will attempt in the event of " +
+ "a disconnection of the stream for any reason, before throwing an exception. To start a stream after " +
+ "this exception occur and the connection is fixed, please stop and restart the processor. If the value" +
+ "of this property is 0, then backoff will never occur and the processor will always need to be restarted" +
+ "if the stream fails.")
+ .required(true)
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .defaultValue("5")
+ .build();
+ public static final PropertyDescriptor BACKOFF_TIME = new PropertyDescriptor.Builder()
+ .name("backoff-time")
+ .displayName("Backoff Time")
+ .description("The duration to backoff to start attempting a new stream if" +
+ "the current one fails for any reason. Will increase by factor of 2 every time a restart fails")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("1 mins")
+ .build();
+ public static final PropertyDescriptor MAXIMUM_BACKOFF_TIME = new PropertyDescriptor.Builder()
+ .name("maximum-backoff-time")
+ .displayName("Maximum Backoff Time")
+ .description("The maximum duration to backoff to start attempting a new stream." +
+ "It is recommended that this number be much higher than the 'Backoff Time' property")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("5 mins")
+ .build();
+ public static final PropertyDescriptor CONNECT_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("connect-timeout")
+ .displayName("Connect Timeout")
+ .description("The maximum time in which client should establish a connection with the " +
+ "Twitter API before a time out. A value of 0 will mean no timeout at all")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("10 secs")
+ .build();
+ public static final PropertyDescriptor READ_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("read-timeout")
+ .displayName("Read Timeout")
+ .description("The maximum time of inactivity between receiving tweets from Twitter through " +
+ "the API before a timeout. A value of 0 will mean no timeout at all")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("10 secs")
+ .build();
+ public static final PropertyDescriptor BACKFILL_MINUTES = new PropertyDescriptor.Builder()
+ .name("backfill-minutes")
+ .displayName("Backfill Minutes")
+ .description("The number of minutes (up to 5 minutes) of streaming data to be requested after a " +
+ "disconnect. Only available for project with academic research access. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/integrate/" +
+ "recovery-and-redundancy-features")
+ .required(true)
+ .defaultValue("0")
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("tweet-fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values include: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("user-fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user " +
+ "for proper usage. Possible field values include: " +
+ "created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, " +
+ "protected, public_metrics, url, username, verified, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor MEDIA_FIELDS = new PropertyDescriptor.Builder()
+ .name("media-fields")
+ .displayName("Media Fields")
+ .description("A comma-separated list of media fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/media " +
+ "for proper usage. Possible field values include: " +
+ "alt_text, duration_ms, height, media_key, non_public_metrics, organic_metrics, preview_image_url, " +
+ "promoted_metrics, public_metrics, type, url, width")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor POLL_FIELDS = new PropertyDescriptor.Builder()
+ .name("poll-fields")
+ .displayName("Poll Fields")
+ .description("A comma-separated list of poll fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/poll " +
+ "for proper usage. Possible field values include: " +
+ "duration_minutes, end_datetime, id, options, voting_status")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PLACE_FIELDS = new PropertyDescriptor.Builder()
+ .name("place-fields")
+ .displayName("Place Fields")
+ .description("A comma-separated list of place fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/place " +
+ "for proper usage. Possible field values include: " +
+ "contained_within, country, country_code, full_name, geo, id, name, place_type")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor EXPANSIONS = new PropertyDescriptor.Builder()
+ .name("expansions")
+ .displayName("Expansions")
+ .description("A comma-separated list of expansions for objects in the returned tweet. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/expansions " +
+ "for proper usage. Possible field values include: " +
+ "author_id, referenced_tweets.id, referenced_tweets.id.author_id, entities.mentions.username, " +
+ "attachments.poll_ids, attachments.media_keys ,in_reply_to_user_id, geo.place_id")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("FlowFiles containing an array of one or more Tweets")
+ .build();
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+
+ private TweetStreamService tweetStreamService;
+
+ private volatile BlockingQueue<String> messageQueue;
+
+ @Override
+ protected void init(ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptors = new ArrayList<>();
+ descriptors.add(ENDPOINT);
+ descriptors.add(BASE_PATH);
+ descriptors.add(BEARER_TOKEN);
+ descriptors.add(QUEUE_SIZE);
+ descriptors.add(BATCH_SIZE);
+ descriptors.add(BACKOFF_ATTEMPTS);
+ descriptors.add(BACKOFF_TIME);
+ descriptors.add(MAXIMUM_BACKOFF_TIME);
+ descriptors.add(CONNECT_TIMEOUT);
+ descriptors.add(READ_TIMEOUT);
+ descriptors.add(BACKFILL_MINUTES);
+ descriptors.add(TWEET_FIELDS);
+ descriptors.add(USER_FIELDS);
+ descriptors.add(MEDIA_FIELDS);
+ descriptors.add(POLL_FIELDS);
+ descriptors.add(PLACE_FIELDS);
+ descriptors.add(EXPANSIONS);
+
+ this.descriptors = Collections.unmodifiableList(descriptors);
+
+ final Set<Relationship> relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ this.relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return this.descriptors;
+ }
+
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ messageQueue = new LinkedBlockingQueue<>(context.getProperty(QUEUE_SIZE).asInteger());
+
+ tweetStreamService = new TweetStreamService(context, messageQueue, getLogger());
+ tweetStreamService.setBasePath(context.getProperty(BASE_PATH).getValue());
+ final String endpointName = context.getProperty(ENDPOINT).getValue();
+ if (ENDPOINT_SAMPLE.getValue().equals(endpointName)) {
+ tweetStreamService.start(StreamEndpoint.SAMPLE_ENDPOINT);
+ } else {
+ tweetStreamService.start(StreamEndpoint.SEARCH_ENDPOINT);
+ }
Review Comment:
Making the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r864033469
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,366 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.UUID;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue(StreamEndpoint.SAMPLE_ENDPOINT.getEndpointName(),
+ "Sample Stream",
+ "Streams about one percent of all Tweets. " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/volume-streams/api-reference/get-tweets-sample-stream");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue(StreamEndpoint.SEARCH_ENDPOINT.getEndpointName(),
+ "Search Stream",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty. " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/get-tweets-search-stream");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("stream-endpoint")
+ .displayName("Stream Endpoint")
+ .description("Specifies which endpoint tweets should be streamed. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("Maximum size of internal queue for streamed messages")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor BACKOFF_ATTEMPTS = new PropertyDescriptor.Builder()
+ .name("backoff-attempts")
+ .displayName("Backoff Attempts")
+ .description("The number of reconnection tries the processor will attempt in the event of " +
+ "a disconnection of the stream for any reason, before throwing an exception. To start a stream after " +
+ "this exception occur and the connection is fixed, please stop and restart the processor. If the value" +
+ "of this property is 0, then backoff will never occur and the processor will always need to be restarted" +
+ "if the stream fails.")
+ .required(true)
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .defaultValue("5")
+ .build();
+ public static final PropertyDescriptor BACKOFF_TIME = new PropertyDescriptor.Builder()
+ .name("backoff-time")
+ .displayName("Backoff Time")
+ .description("The duration to backoff to start attempting a new stream if" +
+ "the current one fails for any reason. Will increase by factor of 2 every time a restart fails")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("1 mins")
+ .build();
+ public static final PropertyDescriptor MAXIMUM_BACKOFF_TIME = new PropertyDescriptor.Builder()
+ .name("maximum-backoff-time")
+ .displayName("Maximum Backoff Time")
+ .description("The maximum duration to backoff to start attempting a new stream." +
+ "It is recommended that this number be much higher than the 'Backoff Time' property")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("5 mins")
+ .build();
+ public static final PropertyDescriptor CONNECT_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("connect-timeout")
+ .displayName("Connect Timeout")
+ .description("The maximum time in which client should establish a connection with the " +
+ "Twitter API before a time out. A value of 0 will mean no timeout at all")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("10 secs")
+ .build();
+ public static final PropertyDescriptor READ_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("read-timeout")
+ .displayName("Read Timeout")
+ .description("The maximum time of inactivity between receiving tweets from Twitter through " +
+ "the API before a timeout. A value of 0 will mean no timeout at all")
Review Comment:
Making the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
exceptionfactory commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r847282030
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,316 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("Twitter Endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("Base Path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("Bearer Token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("Queue Size")
+ .displayName("Queue Size")
+ .description("The size of the queue used to queue tweets")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("Batch Size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("Tweet Fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values are: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("User Fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user " +
+ "for proper usage. Possible field values are: " +
+ "created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, " +
+ "protected, public_metrics, url, username, verified, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor MEDIA_FIELDS = new PropertyDescriptor.Builder()
+ .name("Media Fields")
Review Comment:
The Twitter [Fields](https://developer.twitter.com/en/docs/twitter-api/fields) documentation lists a large number of fields for each type, and the default value of empty string aligns with their REST API. Including all of the fields by default seems difficult to maintain, as the fields could change over time.
It seems best to provide links to the Twitter object model documentation, where each of the fields are documented, such as the following:
- [Tweets](https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet)
- [Users](https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r848782906
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/pom.xml:
##########
@@ -69,5 +80,17 @@
<version>1.17.0-SNAPSHOT</version>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>com.squareup.okhttp3</groupId>
+ <artifactId>okhttp</artifactId>
+ <version>4.9.3</version>
+ <scope>test</scope>
Review Comment:
Making the changes
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,318 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("consume-twitter-endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("The size of the queue used to queue tweets")
Review Comment:
Making the changes
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,318 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("consume-twitter-endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("The size of the queue used to queue tweets")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("tweet-fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values are: " +
Review Comment:
Making the changes
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,318 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("consume-twitter-endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("The size of the queue used to queue tweets")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("tweet-fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values are: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("user-fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user " +
+ "for proper usage. Possible field values are: " +
Review Comment:
Making the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r848775744
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TwitterStreamAPI.java:
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+public class TwitterStreamAPI {
+ public static final String SEARCH_ENDPOINT = "Search Endpoint";
+ public static final String SAMPLE_ENDPOINT = "Sample Endpoint";
+ public static final String SAMPLE_PATH = "/2/tweets/sample/stream";
+ public static final String SEARCH_PATH = "/2/tweets/search/stream";
+
+ private static final String BEARER_TOKEN_PROPERTY_NAME = "bearer-token";
+ private static final String TWEET_FIELDS_PROPERTY_NAME = "tweet-fields";
+ private static final String USER_FIELDS_PROPERTY_NAME = "user-fields";
+ private static final String MEDIA_FIELDS_PROPERTY_NAME = "media-fields";
+ private static final String POLL_FIELDS_PROPERTY_NAME = "poll-fields";
+ private static final String PLACE_FIELDS_PROPERTY_NAME = "place-fields";
+ private static final String EXPANSIONS_PROPERTY_NAME = "expansions";
+ private static final String BACKFILL_MINUTES_PROPERTY_NAME = "backfill-minutes";
+
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Set<String> parseCommaSeparatedListPropreties(final ProcessContext context, final String propertyName) {
+ Set<String> fields = null;
+ if (context.getProperty(propertyName).isSet()) {
+ fields = new HashSet<>();
+ final String fieldsString = context.getProperty(propertyName).getValue();
+ for (final String field: fieldsString.split(",")) {
+ fields.add(field.trim());
+ }
+ }
+ return fields;
+ }
+
+ public TwitterStreamAPI(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ assert context != null;
+ assert queue != null;
+ assert logger != null;
+
+ this.queue = queue;
+ this.logger = logger;
+
+ this.tweetFields = parseCommaSeparatedListPropreties(context, TWEET_FIELDS_PROPERTY_NAME);
+ this.userFields = parseCommaSeparatedListPropreties(context, USER_FIELDS_PROPERTY_NAME);
+ this.mediaFields = parseCommaSeparatedListPropreties(context, MEDIA_FIELDS_PROPERTY_NAME);
+ this.pollFields = parseCommaSeparatedListPropreties(context, POLL_FIELDS_PROPERTY_NAME);
+ this.placeFields = parseCommaSeparatedListPropreties(context, PLACE_FIELDS_PROPERTY_NAME);
+ this.expansions = parseCommaSeparatedListPropreties(context, EXPANSIONS_PROPERTY_NAME);
+ this.backfillMinutes = context.getProperty(BACKFILL_MINUTES_PROPERTY_NAME).asInteger();
+
+ TwitterCredentialsBearer creds = new TwitterCredentialsBearer(context.getProperty(BEARER_TOKEN_PROPERTY_NAME).getValue());
+ api = new TwitterApi();
+ api.setTwitterCredentials(creds);
+
+ this.executorService = Executors.newSingleThreadExecutor();
+ }
+
+ public String getBasePath() {
+ return api.getApiClient().getBasePath();
+ }
+
+ public void setBasePath(final String path) {
+ api.getApiClient().setBasePath(path);
+ }
+
+ /**
+ * This method would be called when we would like the stream to get started. This method will spin off a thread that
+ * will continue to queue tweets on to the given queue passed in the constructor. The thread will continue
+ * to run until {@code stop} is called.
+ * @param endpoint {@code TwitterStreamAPI.SAMPLE_ENDPOINT} or {@code TwitterStreamAPI.SEARCH_ENDPOINT}
+ */
+ public void start(final String endpoint) {
+ try {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ stream = api.tweets().sampleStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else if (endpoint.equals(SEARCH_ENDPOINT)) {
+ stream = api.tweets().searchStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else {
+ throw new ProcessException("Endpoint was invalid value: " + endpoint);
+ }
+ } catch (ApiException e) {
+ logger.error("Received error {}: {}", e.getCode(), e.getMessage());
+ }
+
+ executorService.execute(new TweetQueuer());
+ }
+
+ /**
+ * This method would be called when we would like the stream to get stopped. The stream will be closed and the
+ * executorService will be shut down. If it fails to shutdown, then it will be forcefully terminated.
+ */
+ public void stop() {
+ try {
+ stream.close();
+ } catch (IOException e) {
+ logger.error("IOException occurred while closing stream: {}", e.getMessage());
+ }
+
+ executorService.shutdown();
+ try {
+ if (!executorService.awaitTermination(3, TimeUnit.SECONDS)) {
+ executorService.shutdownNow();
+ }
+ } catch (InterruptedException e) {
+ executorService.shutdownNow();
+ logger.error("InterruptedException occurred while waiting for executor service to shut down: {}", e.getMessage());
+ }
+ }
+
+ private class TweetQueuer implements Runnable {
Review Comment:
That makes more sense, queuer is not an actual word, so TweetStreamHandler seems better. Thanks for the suggestions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r848775434
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TwitterStreamAPI.java:
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+public class TwitterStreamAPI {
Review Comment:
Making the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r864033314
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,366 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.UUID;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue(StreamEndpoint.SAMPLE_ENDPOINT.getEndpointName(),
+ "Sample Stream",
+ "Streams about one percent of all Tweets. " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/volume-streams/api-reference/get-tweets-sample-stream");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue(StreamEndpoint.SEARCH_ENDPOINT.getEndpointName(),
+ "Search Stream",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty. " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/get-tweets-search-stream");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("stream-endpoint")
+ .displayName("Stream Endpoint")
+ .description("Specifies which endpoint tweets should be streamed. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("Maximum size of internal queue for streamed messages")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor BACKOFF_ATTEMPTS = new PropertyDescriptor.Builder()
+ .name("backoff-attempts")
+ .displayName("Backoff Attempts")
+ .description("The number of reconnection tries the processor will attempt in the event of " +
+ "a disconnection of the stream for any reason, before throwing an exception. To start a stream after " +
+ "this exception occur and the connection is fixed, please stop and restart the processor. If the value" +
+ "of this property is 0, then backoff will never occur and the processor will always need to be restarted" +
+ "if the stream fails.")
+ .required(true)
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .defaultValue("5")
+ .build();
+ public static final PropertyDescriptor BACKOFF_TIME = new PropertyDescriptor.Builder()
+ .name("backoff-time")
+ .displayName("Backoff Time")
+ .description("The duration to backoff to start attempting a new stream if" +
+ "the current one fails for any reason. Will increase by factor of 2 every time a restart fails")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("1 mins")
+ .build();
+ public static final PropertyDescriptor MAXIMUM_BACKOFF_TIME = new PropertyDescriptor.Builder()
+ .name("maximum-backoff-time")
+ .displayName("Maximum Backoff Time")
+ .description("The maximum duration to backoff to start attempting a new stream." +
+ "It is recommended that this number be much higher than the 'Backoff Time' property")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("5 mins")
+ .build();
+ public static final PropertyDescriptor CONNECT_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("connect-timeout")
+ .displayName("Connect Timeout")
+ .description("The maximum time in which client should establish a connection with the " +
+ "Twitter API before a time out. A value of 0 will mean no timeout at all")
Review Comment:
Making the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r864029920
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,366 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.UUID;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue(StreamEndpoint.SAMPLE_ENDPOINT.getEndpointName(),
+ "Sample Stream",
+ "Streams about one percent of all Tweets. " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/volume-streams/api-reference/get-tweets-sample-stream");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue(StreamEndpoint.SEARCH_ENDPOINT.getEndpointName(),
+ "Search Stream",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty. " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/get-tweets-search-stream");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("stream-endpoint")
+ .displayName("Stream Endpoint")
+ .description("Specifies which endpoint tweets should be streamed. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
Review Comment:
Makes sense, making the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r864028516
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,366 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.UUID;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue(StreamEndpoint.SAMPLE_ENDPOINT.getEndpointName(),
+ "Sample Stream",
+ "Streams about one percent of all Tweets. " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/volume-streams/api-reference/get-tweets-sample-stream");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue(StreamEndpoint.SEARCH_ENDPOINT.getEndpointName(),
+ "Search Stream",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty. " +
Review Comment:
Making the changes and changing "one filtering rule" to "one well-formed filtering rule"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r864058917
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TweetStreamService.java:
##########
@@ -0,0 +1,222 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiClient;
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TweetStreamService {
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ScheduledExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Long backoffMultiplier;
+ private int backoffAttempts;
+ private int attemptCounter;
+ private Long backoffTime;
+ private Long maximumBackoff;
+
+ private StreamEndpoint endpoint;
+
+ public TweetStreamService(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ Objects.requireNonNull(context);
+ Objects.requireNonNull(queue);
+ Objects.requireNonNull(logger);
+
+ this.queue = queue;
+ this.logger = logger;
+
+ this.tweetFields = parseCommaSeparatedPropreties(context, ConsumeTwitter.TWEET_FIELDS);
+ this.userFields = parseCommaSeparatedPropreties(context, ConsumeTwitter.USER_FIELDS);
+ this.mediaFields = parseCommaSeparatedPropreties(context, ConsumeTwitter.MEDIA_FIELDS);
+ this.pollFields = parseCommaSeparatedPropreties(context, ConsumeTwitter.POLL_FIELDS);
+ this.placeFields = parseCommaSeparatedPropreties(context, ConsumeTwitter.PLACE_FIELDS);
+ this.expansions = parseCommaSeparatedPropreties(context, ConsumeTwitter.EXPANSIONS);
+ this.backfillMinutes = context.getProperty(ConsumeTwitter.BACKFILL_MINUTES).asInteger();
+
+ this.backoffMultiplier = 1L;
+ this.backoffAttempts = context.getProperty(ConsumeTwitter.BACKOFF_ATTEMPTS).asInteger();
+ this.attemptCounter = 0;
+ this.backoffTime = context.getProperty(ConsumeTwitter.BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+ this.maximumBackoff = context.getProperty(ConsumeTwitter.MAXIMUM_BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+
+ ApiClient client = new ApiClient();
+ final int connectTimeout = context.getProperty(ConsumeTwitter.CONNECT_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ final int readTimeout = context.getProperty(ConsumeTwitter.READ_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ client = client.setConnectTimeout(connectTimeout);
+ client = client.setReadTimeout(readTimeout);
+ api = new TwitterApi(client);
+
+ TwitterCredentialsBearer creds = new TwitterCredentialsBearer(context.getProperty(ConsumeTwitter.BEARER_TOKEN).getValue());
+ api.setTwitterCredentials(creds);
+
+ this.executorService = Executors.newSingleThreadScheduledExecutor();
+ }
+
+ public String getBasePath() {
+ return api.getApiClient().getBasePath();
+ }
+
+ public void setBasePath(final String path) {
+ api.getApiClient().setBasePath(path);
+ }
Review Comment:
Making the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r846715738
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,316 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("Twitter Endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("Base Path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("Bearer Token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("Queue Size")
+ .displayName("Queue Size")
+ .description("The size of the queue used to queue tweets")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("Batch Size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("Tweet Fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values are: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("User Fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user " +
+ "for proper usage. Possible field values are: " +
+ "created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, " +
+ "protected, public_metrics, url, username, verified, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor MEDIA_FIELDS = new PropertyDescriptor.Builder()
+ .name("Media Fields")
+ .displayName("Media Fields")
+ .description("A comma-separated list of media fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/media " +
+ "for proper usage. Possible field values are: " +
+ "alt_text, duration_ms, height, media_key, non_public_metrics, organic_metrics, preview_image_url, " +
+ "promoted_metrics, public_metrics, type, url, width")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor POLL_FIELDS = new PropertyDescriptor.Builder()
+ .name("Poll Fields")
+ .displayName("Poll Fields")
+ .description("A comma-separated list of poll fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/poll " +
+ "for proper usage. Possible field values are: " +
+ "duration_minutes, end_datetime, id, options, voting_status")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PLACE_FIELDS = new PropertyDescriptor.Builder()
+ .name("Place Fields")
+ .displayName("Place Fields")
+ .description("A comma-separated list of place fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/place " +
+ "for proper usage. Possible field values are: " +
+ "contained_within, country, country_code, full_name, geo, id, name, place_type")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor EXPANSIONS = new PropertyDescriptor.Builder()
+ .name("Expansions")
+ .displayName("Expansions")
+ .description("A comma-separated list of expansions for objects in the returned tweet. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/expansions " +
+ "for proper usage. Possible field values are: " +
+ "author_id, referenced_tweets.id, referenced_tweets.id.author_id, entities.mentions.username, " +
+ "attachments.poll_ids, attachments.media_keys ,in_reply_to_user_id, geo.place_id")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor BACKFILL_MINUTES = new PropertyDescriptor.Builder()
+ .name("Backfill Minutes")
+ .displayName("Backfill Minutes")
+ .description("The number of minutes (up to 5 minutes) of streaming data to be requested after a " +
+ "disconnect. Only available for project with academic research access. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/integrate/" +
+ "recovery-and-redundancy-features")
+ .required(true)
+ .defaultValue("0")
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("All tweets will be routed to this relationship.")
+ .build();
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+
+ private TwitterStreamAPI api;
+
+ private volatile BlockingQueue<String> messageQueue;
+
+ private void emptyQueue() {
+ while (!messageQueue.isEmpty()) {
+ messageQueue.poll();
+ }
+ }
+
+ @Override
+ protected void init(ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptors = new ArrayList<>();
+ descriptors.add(ENDPOINT);
+ descriptors.add(BASE_PATH);
+ descriptors.add(BEARER_TOKEN);
+ descriptors.add(QUEUE_SIZE);
+ descriptors.add(BATCH_SIZE);
+ descriptors.add(TWEET_FIELDS);
+ descriptors.add(USER_FIELDS);
+ descriptors.add(MEDIA_FIELDS);
+ descriptors.add(POLL_FIELDS);
+ descriptors.add(PLACE_FIELDS);
+ descriptors.add(EXPANSIONS);
+ descriptors.add(BACKFILL_MINUTES);
+
+ this.descriptors = Collections.unmodifiableList(descriptors);
+
+ final Set<Relationship> relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ this.relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return this.descriptors;
+ }
+
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ messageQueue = new LinkedBlockingQueue<>(context.getProperty(QUEUE_SIZE).asInteger());
+
+ api = new TwitterStreamAPI(context, messageQueue, getLogger());
+ api.setBasePath(context.getProperty(BASE_PATH).getValue());
+ final String endpointName = context.getProperty(ENDPOINT).getValue();
+ if (ENDPOINT_SAMPLE.getValue().equals(endpointName)) {
+ api.start(TwitterStreamAPI.SAMPLE_ENDPOINT);
+ } else if (ENDPOINT_SEARCH.getValue().equals(endpointName)) {
+ api.start(TwitterStreamAPI.SEARCH_ENDPOINT);
+ } else {
+ throw new AssertionError("Endpoint was invalid value: " + endpointName);
Review Comment:
Will change all the assertion errors into ProcessExceptions, thanks for the comment!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] MikeThomsen commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
MikeThomsen commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r847174193
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,316 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
Review Comment:
I think this should have `@PrimaryNodeOnly` on it because we don't want this processor running equally on all nodes on the cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r848783150
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,318 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("consume-twitter-endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("The size of the queue used to queue tweets")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("tweet-fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values are: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("user-fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user " +
+ "for proper usage. Possible field values are: " +
+ "created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, " +
+ "protected, public_metrics, url, username, verified, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor MEDIA_FIELDS = new PropertyDescriptor.Builder()
+ .name("media-fields")
+ .displayName("Media Fields")
+ .description("A comma-separated list of media fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/media " +
+ "for proper usage. Possible field values are: " +
Review Comment:
Making the changes
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,318 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("consume-twitter-endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("The size of the queue used to queue tweets")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("tweet-fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values are: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("user-fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user " +
+ "for proper usage. Possible field values are: " +
+ "created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, " +
+ "protected, public_metrics, url, username, verified, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor MEDIA_FIELDS = new PropertyDescriptor.Builder()
+ .name("media-fields")
+ .displayName("Media Fields")
+ .description("A comma-separated list of media fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/media " +
+ "for proper usage. Possible field values are: " +
+ "alt_text, duration_ms, height, media_key, non_public_metrics, organic_metrics, preview_image_url, " +
+ "promoted_metrics, public_metrics, type, url, width")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor POLL_FIELDS = new PropertyDescriptor.Builder()
+ .name("poll-fields")
+ .displayName("Poll Fields")
+ .description("A comma-separated list of poll fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/poll " +
+ "for proper usage. Possible field values are: " +
Review Comment:
Making the changes
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,318 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("consume-twitter-endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("The size of the queue used to queue tweets")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("tweet-fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values are: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("user-fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user " +
+ "for proper usage. Possible field values are: " +
+ "created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, " +
+ "protected, public_metrics, url, username, verified, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor MEDIA_FIELDS = new PropertyDescriptor.Builder()
+ .name("media-fields")
+ .displayName("Media Fields")
+ .description("A comma-separated list of media fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/media " +
+ "for proper usage. Possible field values are: " +
+ "alt_text, duration_ms, height, media_key, non_public_metrics, organic_metrics, preview_image_url, " +
+ "promoted_metrics, public_metrics, type, url, width")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor POLL_FIELDS = new PropertyDescriptor.Builder()
+ .name("poll-fields")
+ .displayName("Poll Fields")
+ .description("A comma-separated list of poll fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/poll " +
+ "for proper usage. Possible field values are: " +
+ "duration_minutes, end_datetime, id, options, voting_status")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PLACE_FIELDS = new PropertyDescriptor.Builder()
+ .name("place-fields")
+ .displayName("Place Fields")
+ .description("A comma-separated list of place fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/place " +
+ "for proper usage. Possible field values are: " +
Review Comment:
Making the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r848777967
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TwitterStreamAPI.java:
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+public class TwitterStreamAPI {
+ public static final String SEARCH_ENDPOINT = "Search Endpoint";
+ public static final String SAMPLE_ENDPOINT = "Sample Endpoint";
+ public static final String SAMPLE_PATH = "/2/tweets/sample/stream";
+ public static final String SEARCH_PATH = "/2/tweets/search/stream";
+
+ private static final String BEARER_TOKEN_PROPERTY_NAME = "bearer-token";
+ private static final String TWEET_FIELDS_PROPERTY_NAME = "tweet-fields";
+ private static final String USER_FIELDS_PROPERTY_NAME = "user-fields";
+ private static final String MEDIA_FIELDS_PROPERTY_NAME = "media-fields";
+ private static final String POLL_FIELDS_PROPERTY_NAME = "poll-fields";
+ private static final String PLACE_FIELDS_PROPERTY_NAME = "place-fields";
+ private static final String EXPANSIONS_PROPERTY_NAME = "expansions";
+ private static final String BACKFILL_MINUTES_PROPERTY_NAME = "backfill-minutes";
+
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Set<String> parseCommaSeparatedListPropreties(final ProcessContext context, final String propertyName) {
+ Set<String> fields = null;
+ if (context.getProperty(propertyName).isSet()) {
+ fields = new HashSet<>();
+ final String fieldsString = context.getProperty(propertyName).getValue();
+ for (final String field: fieldsString.split(",")) {
+ fields.add(field.trim());
+ }
+ }
+ return fields;
+ }
+
+ public TwitterStreamAPI(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ assert context != null;
+ assert queue != null;
+ assert logger != null;
+
+ this.queue = queue;
+ this.logger = logger;
+
+ this.tweetFields = parseCommaSeparatedListPropreties(context, TWEET_FIELDS_PROPERTY_NAME);
+ this.userFields = parseCommaSeparatedListPropreties(context, USER_FIELDS_PROPERTY_NAME);
+ this.mediaFields = parseCommaSeparatedListPropreties(context, MEDIA_FIELDS_PROPERTY_NAME);
+ this.pollFields = parseCommaSeparatedListPropreties(context, POLL_FIELDS_PROPERTY_NAME);
+ this.placeFields = parseCommaSeparatedListPropreties(context, PLACE_FIELDS_PROPERTY_NAME);
+ this.expansions = parseCommaSeparatedListPropreties(context, EXPANSIONS_PROPERTY_NAME);
+ this.backfillMinutes = context.getProperty(BACKFILL_MINUTES_PROPERTY_NAME).asInteger();
+
+ TwitterCredentialsBearer creds = new TwitterCredentialsBearer(context.getProperty(BEARER_TOKEN_PROPERTY_NAME).getValue());
+ api = new TwitterApi();
+ api.setTwitterCredentials(creds);
+
+ this.executorService = Executors.newSingleThreadExecutor();
+ }
+
+ public String getBasePath() {
+ return api.getApiClient().getBasePath();
+ }
+
+ public void setBasePath(final String path) {
+ api.getApiClient().setBasePath(path);
+ }
+
+ /**
+ * This method would be called when we would like the stream to get started. This method will spin off a thread that
+ * will continue to queue tweets on to the given queue passed in the constructor. The thread will continue
+ * to run until {@code stop} is called.
+ * @param endpoint {@code TwitterStreamAPI.SAMPLE_ENDPOINT} or {@code TwitterStreamAPI.SEARCH_ENDPOINT}
+ */
+ public void start(final String endpoint) {
+ try {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ stream = api.tweets().sampleStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else if (endpoint.equals(SEARCH_ENDPOINT)) {
+ stream = api.tweets().searchStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else {
+ throw new ProcessException("Endpoint was invalid value: " + endpoint);
+ }
+ } catch (ApiException e) {
+ logger.error("Received error {}: {}", e.getCode(), e.getMessage());
Review Comment:
Will make the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r848774732
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,318 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("consume-twitter-endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("The size of the queue used to queue tweets")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("tweet-fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values are: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("user-fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user " +
+ "for proper usage. Possible field values are: " +
+ "created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, " +
+ "protected, public_metrics, url, username, verified, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor MEDIA_FIELDS = new PropertyDescriptor.Builder()
+ .name("media-fields")
+ .displayName("Media Fields")
+ .description("A comma-separated list of media fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/media " +
+ "for proper usage. Possible field values are: " +
+ "alt_text, duration_ms, height, media_key, non_public_metrics, organic_metrics, preview_image_url, " +
+ "promoted_metrics, public_metrics, type, url, width")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor POLL_FIELDS = new PropertyDescriptor.Builder()
+ .name("poll-fields")
+ .displayName("Poll Fields")
+ .description("A comma-separated list of poll fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/poll " +
+ "for proper usage. Possible field values are: " +
+ "duration_minutes, end_datetime, id, options, voting_status")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PLACE_FIELDS = new PropertyDescriptor.Builder()
+ .name("place-fields")
+ .displayName("Place Fields")
+ .description("A comma-separated list of place fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/place " +
+ "for proper usage. Possible field values are: " +
+ "contained_within, country, country_code, full_name, geo, id, name, place_type")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor EXPANSIONS = new PropertyDescriptor.Builder()
+ .name("expansions")
+ .displayName("Expansions")
+ .description("A comma-separated list of expansions for objects in the returned tweet. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/expansions " +
+ "for proper usage. Possible field values are: " +
+ "author_id, referenced_tweets.id, referenced_tweets.id.author_id, entities.mentions.username, " +
+ "attachments.poll_ids, attachments.media_keys ,in_reply_to_user_id, geo.place_id")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor BACKFILL_MINUTES = new PropertyDescriptor.Builder()
+ .name("backfill-minutes")
+ .displayName("Backfill Minutes")
+ .description("The number of minutes (up to 5 minutes) of streaming data to be requested after a " +
+ "disconnect. Only available for project with academic research access. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/integrate/" +
+ "recovery-and-redundancy-features")
+ .required(true)
+ .defaultValue("0")
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("All tweets will be routed to this relationship.")
+ .build();
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+
+ private TwitterStreamAPI api;
+
+ private volatile BlockingQueue<String> messageQueue;
+
+ private void emptyQueue() {
+ while (!messageQueue.isEmpty()) {
+ messageQueue.poll();
+ }
+ }
Review Comment:
Making the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r861122760
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/pom.xml:
##########
@@ -69,5 +84,13 @@
<version>1.17.0-SNAPSHOT</version>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>com.squareup.okhttp3</groupId>
+ <artifactId>okhttp</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>com.squareup.okhttp3</groupId>
+ <artifactId>mockwebserver</artifactId>
Review Comment:
Making the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r861169507
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TweetStreamService.java:
##########
@@ -0,0 +1,242 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiClient;
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TweetStreamService {
+ public static final String SEARCH_ENDPOINT = "Search Endpoint";
+ public static final String SAMPLE_ENDPOINT = "Sample Endpoint";
+ private static final String SAMPLE_PATH = "/2/tweets/sample/stream";
+ private static final String SEARCH_PATH = "/2/tweets/search/stream";
+
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ScheduledExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Long backoffMultiplier;
+ private int backoffAttempts;
+ private int attemptCounter;
+ private Long backoffTime;
+ private Long maximumBackoff;
+
+ private String endpoint;
+
+ private Set<String> parseCommaSeparatedListPropreties(final ProcessContext context, final PropertyDescriptor property) {
+ Set<String> fields = null;
+ if (context.getProperty(property).isSet()) {
+ fields = new HashSet<>();
+ final String fieldsString = context.getProperty(property).getValue();
+ for (final String field: fieldsString.split(",")) {
+ fields.add(field.trim());
+ }
+ }
+ return fields;
+ }
+
+ public TweetStreamService(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ assert context != null;
+ assert queue != null;
+ assert logger != null;
+
+ this.queue = queue;
+ this.logger = logger;
+
+ this.tweetFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.TWEET_FIELDS);
+ this.userFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.USER_FIELDS);
+ this.mediaFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.MEDIA_FIELDS);
+ this.pollFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.POLL_FIELDS);
+ this.placeFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.PLACE_FIELDS);
+ this.expansions = parseCommaSeparatedListPropreties(context, ConsumeTwitter.EXPANSIONS);
+ this.backfillMinutes = context.getProperty(ConsumeTwitter.BACKFILL_MINUTES).asInteger();
+
+ this.backoffMultiplier = 1L;
+ this.backoffAttempts = context.getProperty(ConsumeTwitter.BACKOFF_ATTEMPTS).asInteger();
+ this.attemptCounter = 0;
+ this.backoffTime = context.getProperty(ConsumeTwitter.BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+ this.maximumBackoff = context.getProperty(ConsumeTwitter.MAXIMUM_BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+
+ ApiClient client = new ApiClient();
+ final int connectTimeout = context.getProperty(ConsumeTwitter.CONNECT_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ final int readTimeout = context.getProperty(ConsumeTwitter.READ_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ client = client.setConnectTimeout(connectTimeout);
+ client = client.setReadTimeout(readTimeout);
+ api = new TwitterApi(client);
+
+ TwitterCredentialsBearer creds = new TwitterCredentialsBearer(context.getProperty(ConsumeTwitter.BEARER_TOKEN).getValue());
+ api.setTwitterCredentials(creds);
+
+ this.executorService = Executors.newSingleThreadScheduledExecutor();
+ }
+
+ public String getBasePath() {
+ return api.getApiClient().getBasePath();
+ }
+
+ public void setBasePath(final String path) {
+ api.getApiClient().setBasePath(path);
+ }
+
+ public String getTransitUri(final String endpoint) {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ return api.getApiClient().getBasePath() + SAMPLE_PATH;
+ } else if (endpoint.equals(SEARCH_ENDPOINT)) {
+ return api.getApiClient().getBasePath() + SEARCH_PATH;
+ } else {
+ logger.warn("Unrecognized endpoint in getTransitUri. Returning basePath");
+ return api.getApiClient().getBasePath();
+ }
+ }
+
+ /**
+ * This method would be called when we would like the stream to get started. This method will spin off a thread that
+ * will continue to queue tweets on to the given queue passed in the constructor. The thread will continue
+ * to run until {@code stop} is called.
+ * @param endpoint {@code TwitterStreamAPI.SAMPLE_ENDPOINT} or {@code TwitterStreamAPI.SEARCH_ENDPOINT}
+ */
+ public void start(final String endpoint) {
+ this.endpoint = endpoint;
+ executorService.execute(new TweetStreamStarter());
+ }
+
+ /**
+ * This method would be called when we would like the stream to get stopped. The stream will be closed and the
+ * executorService will be shut down.
+ */
+ public void stop() {
+ if (stream != null) {
+ try {
+ stream.close();
+ } catch (IOException e) {
+ logger.error("Closing response stream failed", e);
+ }
+ }
+
+ executorService.shutdownNow();
+ }
+
+ private Long calculateBackoffDelay() {
+ long backoff = backoffMultiplier * backoffTime;
+ long delay = (backoff > maximumBackoff) ? maximumBackoff : backoff;
+ return delay;
+ }
+
+ private void scheduleStartStreamWithBackoff() {
+ // use exponential(by factor of 2) backoff in scheduling the next TweetStreamStarter
+ if (attemptCounter >= backoffAttempts) {
+ final String msg = "Reached maximum attempts to backoff";
+ logger.error(msg);
+ throw new ProcessException(msg);
+ }
+ attemptCounter += 1;
+ long delay = calculateBackoffDelay();
+ backoffMultiplier *= 2;
+ logger.info("Scheduling new TweetStreamStarter after {}s delay", delay);
+ executorService.schedule(new TweetStreamStarter(), delay, TimeUnit.SECONDS);
+ }
+
+ private void resetBackoff() {
+ attemptCounter = 0;
+ backoffMultiplier = 1L;
+ }
+
+ private class TweetStreamStarter implements Runnable {
+ @Override
+ public void run() {
+ try {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ stream = api.tweets().sampleStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else {
+ stream = api.tweets().searchStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ }
+ } catch (ApiException e) {
+ stream = null;
+ logger.warn(String.format("Received ApiException, error %d: %s", e.getCode(), e.getMessage()));
+ scheduleStartStreamWithBackoff();
+ return;
+ } catch (Exception e) {
+ stream = null;
+ logger.warn(String.format("Received Exception %s: %s", e.getClass(), e.getMessage()));
+ scheduleStartStreamWithBackoff();
+ return;
+ }
+
+ if (stream == null) {
+ logger.warn("Stream is null, could not make a connection to the Twitter API");
+ scheduleStartStreamWithBackoff();
+ return;
+ } else {
+ executorService.execute(new TweetStreamHandler());
Review Comment:
That makes sense, making the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r861127790
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.UUID;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("consume-twitter-endpoint")
+ .displayName("Twitter Endpoint")
Review Comment:
Making the changes
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.UUID;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("consume-twitter-endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
Review Comment:
Making the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r861146773
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TweetStreamService.java:
##########
@@ -0,0 +1,242 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiClient;
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TweetStreamService {
+ public static final String SEARCH_ENDPOINT = "Search Endpoint";
+ public static final String SAMPLE_ENDPOINT = "Sample Endpoint";
+ private static final String SAMPLE_PATH = "/2/tweets/sample/stream";
+ private static final String SEARCH_PATH = "/2/tweets/search/stream";
+
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ScheduledExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Long backoffMultiplier;
+ private int backoffAttempts;
+ private int attemptCounter;
+ private Long backoffTime;
+ private Long maximumBackoff;
+
+ private String endpoint;
+
+ private Set<String> parseCommaSeparatedListPropreties(final ProcessContext context, final PropertyDescriptor property) {
+ Set<String> fields = null;
+ if (context.getProperty(property).isSet()) {
+ fields = new HashSet<>();
+ final String fieldsString = context.getProperty(property).getValue();
+ for (final String field: fieldsString.split(",")) {
+ fields.add(field.trim());
+ }
+ }
+ return fields;
+ }
+
+ public TweetStreamService(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ assert context != null;
+ assert queue != null;
+ assert logger != null;
+
+ this.queue = queue;
+ this.logger = logger;
+
+ this.tweetFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.TWEET_FIELDS);
+ this.userFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.USER_FIELDS);
+ this.mediaFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.MEDIA_FIELDS);
+ this.pollFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.POLL_FIELDS);
+ this.placeFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.PLACE_FIELDS);
+ this.expansions = parseCommaSeparatedListPropreties(context, ConsumeTwitter.EXPANSIONS);
+ this.backfillMinutes = context.getProperty(ConsumeTwitter.BACKFILL_MINUTES).asInteger();
+
+ this.backoffMultiplier = 1L;
+ this.backoffAttempts = context.getProperty(ConsumeTwitter.BACKOFF_ATTEMPTS).asInteger();
+ this.attemptCounter = 0;
+ this.backoffTime = context.getProperty(ConsumeTwitter.BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+ this.maximumBackoff = context.getProperty(ConsumeTwitter.MAXIMUM_BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+
+ ApiClient client = new ApiClient();
+ final int connectTimeout = context.getProperty(ConsumeTwitter.CONNECT_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ final int readTimeout = context.getProperty(ConsumeTwitter.READ_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ client = client.setConnectTimeout(connectTimeout);
+ client = client.setReadTimeout(readTimeout);
+ api = new TwitterApi(client);
+
+ TwitterCredentialsBearer creds = new TwitterCredentialsBearer(context.getProperty(ConsumeTwitter.BEARER_TOKEN).getValue());
+ api.setTwitterCredentials(creds);
+
+ this.executorService = Executors.newSingleThreadScheduledExecutor();
+ }
+
+ public String getBasePath() {
+ return api.getApiClient().getBasePath();
+ }
+
+ public void setBasePath(final String path) {
+ api.getApiClient().setBasePath(path);
+ }
+
+ public String getTransitUri(final String endpoint) {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ return api.getApiClient().getBasePath() + SAMPLE_PATH;
+ } else if (endpoint.equals(SEARCH_ENDPOINT)) {
+ return api.getApiClient().getBasePath() + SEARCH_PATH;
+ } else {
+ logger.warn("Unrecognized endpoint in getTransitUri. Returning basePath");
+ return api.getApiClient().getBasePath();
+ }
+ }
+
+ /**
+ * This method would be called when we would like the stream to get started. This method will spin off a thread that
+ * will continue to queue tweets on to the given queue passed in the constructor. The thread will continue
+ * to run until {@code stop} is called.
+ * @param endpoint {@code TwitterStreamAPI.SAMPLE_ENDPOINT} or {@code TwitterStreamAPI.SEARCH_ENDPOINT}
+ */
+ public void start(final String endpoint) {
+ this.endpoint = endpoint;
+ executorService.execute(new TweetStreamStarter());
+ }
+
+ /**
+ * This method would be called when we would like the stream to get stopped. The stream will be closed and the
+ * executorService will be shut down.
+ */
+ public void stop() {
+ if (stream != null) {
+ try {
+ stream.close();
+ } catch (IOException e) {
+ logger.error("Closing response stream failed", e);
+ }
+ }
+
+ executorService.shutdownNow();
+ }
+
+ private Long calculateBackoffDelay() {
+ long backoff = backoffMultiplier * backoffTime;
+ long delay = (backoff > maximumBackoff) ? maximumBackoff : backoff;
+ return delay;
+ }
+
+ private void scheduleStartStreamWithBackoff() {
+ // use exponential(by factor of 2) backoff in scheduling the next TweetStreamStarter
+ if (attemptCounter >= backoffAttempts) {
+ final String msg = "Reached maximum attempts to backoff";
+ logger.error(msg);
+ throw new ProcessException(msg);
Review Comment:
Making the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r846716681
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,316 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("Twitter Endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("Base Path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("Bearer Token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("Queue Size")
+ .displayName("Queue Size")
+ .description("The size of the queue used to queue tweets")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("Batch Size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("Tweet Fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values are: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("User Fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user " +
+ "for proper usage. Possible field values are: " +
+ "created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, " +
+ "protected, public_metrics, url, username, verified, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor MEDIA_FIELDS = new PropertyDescriptor.Builder()
+ .name("Media Fields")
Review Comment:
Thanks for the comment. Although I have some thoughts. These are non-mandatory fields and serve as query parameters when the Twitter API class issues a GET request. A default value (and making this a required property) that makes sense for a user experience perspective would be
- an empty string, or
- a comma separated list of every possible field that could be put in there
Any thoughts and recommendations? Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] MikeThomsen commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
MikeThomsen commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r848462300
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TwitterStreamAPI.java:
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+public class TwitterStreamAPI {
+ public static final String SEARCH_ENDPOINT = "Search Endpoint";
+ public static final String SAMPLE_ENDPOINT = "Sample Endpoint";
+ public static final String SAMPLE_PATH = "/2/tweets/sample/stream";
+ public static final String SEARCH_PATH = "/2/tweets/search/stream";
+
+ private static final String BEARER_TOKEN_PROPERTY_NAME = "bearer-token";
+ private static final String TWEET_FIELDS_PROPERTY_NAME = "tweet-fields";
+ private static final String USER_FIELDS_PROPERTY_NAME = "user-fields";
+ private static final String MEDIA_FIELDS_PROPERTY_NAME = "media-fields";
+ private static final String POLL_FIELDS_PROPERTY_NAME = "poll-fields";
+ private static final String PLACE_FIELDS_PROPERTY_NAME = "place-fields";
+ private static final String EXPANSIONS_PROPERTY_NAME = "expansions";
+ private static final String BACKFILL_MINUTES_PROPERTY_NAME = "backfill-minutes";
+
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Set<String> parseCommaSeparatedListPropreties(final ProcessContext context, final String propertyName) {
+ Set<String> fields = null;
+ if (context.getProperty(propertyName).isSet()) {
+ fields = new HashSet<>();
+ final String fieldsString = context.getProperty(propertyName).getValue();
+ for (final String field: fieldsString.split(",")) {
+ fields.add(field.trim());
+ }
+ }
+ return fields;
+ }
+
+ public TwitterStreamAPI(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ assert context != null;
+ assert queue != null;
+ assert logger != null;
+
+ this.queue = queue;
+ this.logger = logger;
+
+ this.tweetFields = parseCommaSeparatedListPropreties(context, TWEET_FIELDS_PROPERTY_NAME);
+ this.userFields = parseCommaSeparatedListPropreties(context, USER_FIELDS_PROPERTY_NAME);
+ this.mediaFields = parseCommaSeparatedListPropreties(context, MEDIA_FIELDS_PROPERTY_NAME);
+ this.pollFields = parseCommaSeparatedListPropreties(context, POLL_FIELDS_PROPERTY_NAME);
+ this.placeFields = parseCommaSeparatedListPropreties(context, PLACE_FIELDS_PROPERTY_NAME);
+ this.expansions = parseCommaSeparatedListPropreties(context, EXPANSIONS_PROPERTY_NAME);
+ this.backfillMinutes = context.getProperty(BACKFILL_MINUTES_PROPERTY_NAME).asInteger();
+
+ TwitterCredentialsBearer creds = new TwitterCredentialsBearer(context.getProperty(BEARER_TOKEN_PROPERTY_NAME).getValue());
+ api = new TwitterApi();
+ api.setTwitterCredentials(creds);
+
+ this.executorService = Executors.newSingleThreadExecutor();
+ }
+
+ public String getBasePath() {
+ return api.getApiClient().getBasePath();
+ }
+
+ public void setBasePath(final String path) {
+ api.getApiClient().setBasePath(path);
+ }
+
+ /**
+ * This method would be called when we would like the stream to get started. This method will spin off a thread that
+ * will continue to queue tweets on to the given queue passed in the constructor. The thread will continue
+ * to run until {@code stop} is called.
+ * @param endpoint {@code TwitterStreamAPI.SAMPLE_ENDPOINT} or {@code TwitterStreamAPI.SEARCH_ENDPOINT}
+ */
+ public void start(final String endpoint) {
+ try {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ stream = api.tweets().sampleStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else if (endpoint.equals(SEARCH_ENDPOINT)) {
+ stream = api.tweets().searchStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else {
+ throw new ProcessException("Endpoint was invalid value: " + endpoint);
+ }
+ } catch (ApiException e) {
+ logger.error("Received error {}: {}", e.getCode(), e.getMessage());
+ }
+
+ executorService.execute(new TweetQueuer());
Review Comment:
Tested this locally with the blocking my corporate env is doing. In the JVM debugger, `stream` was null here in this env.
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TwitterStreamAPI.java:
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+public class TwitterStreamAPI {
+ public static final String SEARCH_ENDPOINT = "Search Endpoint";
+ public static final String SAMPLE_ENDPOINT = "Sample Endpoint";
+ public static final String SAMPLE_PATH = "/2/tweets/sample/stream";
+ public static final String SEARCH_PATH = "/2/tweets/search/stream";
+
+ private static final String BEARER_TOKEN_PROPERTY_NAME = "bearer-token";
+ private static final String TWEET_FIELDS_PROPERTY_NAME = "tweet-fields";
+ private static final String USER_FIELDS_PROPERTY_NAME = "user-fields";
+ private static final String MEDIA_FIELDS_PROPERTY_NAME = "media-fields";
+ private static final String POLL_FIELDS_PROPERTY_NAME = "poll-fields";
+ private static final String PLACE_FIELDS_PROPERTY_NAME = "place-fields";
+ private static final String EXPANSIONS_PROPERTY_NAME = "expansions";
+ private static final String BACKFILL_MINUTES_PROPERTY_NAME = "backfill-minutes";
+
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Set<String> parseCommaSeparatedListPropreties(final ProcessContext context, final String propertyName) {
+ Set<String> fields = null;
+ if (context.getProperty(propertyName).isSet()) {
+ fields = new HashSet<>();
+ final String fieldsString = context.getProperty(propertyName).getValue();
+ for (final String field: fieldsString.split(",")) {
+ fields.add(field.trim());
+ }
+ }
+ return fields;
+ }
+
+ public TwitterStreamAPI(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ assert context != null;
+ assert queue != null;
+ assert logger != null;
+
+ this.queue = queue;
+ this.logger = logger;
+
+ this.tweetFields = parseCommaSeparatedListPropreties(context, TWEET_FIELDS_PROPERTY_NAME);
+ this.userFields = parseCommaSeparatedListPropreties(context, USER_FIELDS_PROPERTY_NAME);
+ this.mediaFields = parseCommaSeparatedListPropreties(context, MEDIA_FIELDS_PROPERTY_NAME);
+ this.pollFields = parseCommaSeparatedListPropreties(context, POLL_FIELDS_PROPERTY_NAME);
+ this.placeFields = parseCommaSeparatedListPropreties(context, PLACE_FIELDS_PROPERTY_NAME);
+ this.expansions = parseCommaSeparatedListPropreties(context, EXPANSIONS_PROPERTY_NAME);
+ this.backfillMinutes = context.getProperty(BACKFILL_MINUTES_PROPERTY_NAME).asInteger();
+
+ TwitterCredentialsBearer creds = new TwitterCredentialsBearer(context.getProperty(BEARER_TOKEN_PROPERTY_NAME).getValue());
+ api = new TwitterApi();
+ api.setTwitterCredentials(creds);
+
+ this.executorService = Executors.newSingleThreadExecutor();
+ }
+
+ public String getBasePath() {
+ return api.getApiClient().getBasePath();
+ }
+
+ public void setBasePath(final String path) {
+ api.getApiClient().setBasePath(path);
+ }
+
+ /**
+ * This method would be called when we would like the stream to get started. This method will spin off a thread that
+ * will continue to queue tweets on to the given queue passed in the constructor. The thread will continue
+ * to run until {@code stop} is called.
+ * @param endpoint {@code TwitterStreamAPI.SAMPLE_ENDPOINT} or {@code TwitterStreamAPI.SEARCH_ENDPOINT}
+ */
+ public void start(final String endpoint) {
+ try {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ stream = api.tweets().sampleStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else if (endpoint.equals(SEARCH_ENDPOINT)) {
+ stream = api.tweets().searchStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else {
+ throw new ProcessException("Endpoint was invalid value: " + endpoint);
+ }
+ } catch (ApiException e) {
+ logger.error("Received error {}: {}", e.getCode(), e.getMessage());
+ }
+
+ executorService.execute(new TweetQueuer());
+ }
+
+ /**
+ * This method would be called when we would like the stream to get stopped. The stream will be closed and the
+ * executorService will be shut down. If it fails to shutdown, then it will be forcefully terminated.
+ */
+ public void stop() {
+ try {
+ stream.close();
+ } catch (IOException e) {
+ logger.error("IOException occurred while closing stream: {}", e.getMessage());
+ }
+
+ executorService.shutdown();
+ try {
+ if (!executorService.awaitTermination(3, TimeUnit.SECONDS)) {
+ executorService.shutdownNow();
+ }
+ } catch (InterruptedException e) {
+ executorService.shutdownNow();
+ logger.error("InterruptedException occurred while waiting for executor service to shut down: {}", e.getMessage());
+ }
+ }
+
+ private class TweetQueuer implements Runnable {
+ @Override
+ public void run() {
+ try (BufferedReader reader = new BufferedReader(new InputStreamReader(stream))) {
+ String tweetLine = reader.readLine();
+ while (tweetLine != null) {
Review Comment:
Granted, there are a lot of Twitter users, but doesn't this carry some intrinsic risk that the processor will silently stop consuming if it catches a < 1sec lull in Twitter activity?
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TwitterStreamAPI.java:
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+public class TwitterStreamAPI {
+ public static final String SEARCH_ENDPOINT = "Search Endpoint";
+ public static final String SAMPLE_ENDPOINT = "Sample Endpoint";
+ public static final String SAMPLE_PATH = "/2/tweets/sample/stream";
+ public static final String SEARCH_PATH = "/2/tweets/search/stream";
+
+ private static final String BEARER_TOKEN_PROPERTY_NAME = "bearer-token";
+ private static final String TWEET_FIELDS_PROPERTY_NAME = "tweet-fields";
+ private static final String USER_FIELDS_PROPERTY_NAME = "user-fields";
+ private static final String MEDIA_FIELDS_PROPERTY_NAME = "media-fields";
+ private static final String POLL_FIELDS_PROPERTY_NAME = "poll-fields";
+ private static final String PLACE_FIELDS_PROPERTY_NAME = "place-fields";
+ private static final String EXPANSIONS_PROPERTY_NAME = "expansions";
+ private static final String BACKFILL_MINUTES_PROPERTY_NAME = "backfill-minutes";
+
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Set<String> parseCommaSeparatedListPropreties(final ProcessContext context, final String propertyName) {
+ Set<String> fields = null;
+ if (context.getProperty(propertyName).isSet()) {
+ fields = new HashSet<>();
+ final String fieldsString = context.getProperty(propertyName).getValue();
+ for (final String field: fieldsString.split(",")) {
+ fields.add(field.trim());
+ }
+ }
+ return fields;
+ }
+
+ public TwitterStreamAPI(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ assert context != null;
+ assert queue != null;
+ assert logger != null;
+
+ this.queue = queue;
+ this.logger = logger;
+
+ this.tweetFields = parseCommaSeparatedListPropreties(context, TWEET_FIELDS_PROPERTY_NAME);
+ this.userFields = parseCommaSeparatedListPropreties(context, USER_FIELDS_PROPERTY_NAME);
+ this.mediaFields = parseCommaSeparatedListPropreties(context, MEDIA_FIELDS_PROPERTY_NAME);
+ this.pollFields = parseCommaSeparatedListPropreties(context, POLL_FIELDS_PROPERTY_NAME);
+ this.placeFields = parseCommaSeparatedListPropreties(context, PLACE_FIELDS_PROPERTY_NAME);
+ this.expansions = parseCommaSeparatedListPropreties(context, EXPANSIONS_PROPERTY_NAME);
+ this.backfillMinutes = context.getProperty(BACKFILL_MINUTES_PROPERTY_NAME).asInteger();
+
+ TwitterCredentialsBearer creds = new TwitterCredentialsBearer(context.getProperty(BEARER_TOKEN_PROPERTY_NAME).getValue());
+ api = new TwitterApi();
+ api.setTwitterCredentials(creds);
+
+ this.executorService = Executors.newSingleThreadExecutor();
+ }
+
+ public String getBasePath() {
+ return api.getApiClient().getBasePath();
+ }
+
+ public void setBasePath(final String path) {
+ api.getApiClient().setBasePath(path);
+ }
+
+ /**
+ * This method would be called when we would like the stream to get started. This method will spin off a thread that
+ * will continue to queue tweets on to the given queue passed in the constructor. The thread will continue
+ * to run until {@code stop} is called.
+ * @param endpoint {@code TwitterStreamAPI.SAMPLE_ENDPOINT} or {@code TwitterStreamAPI.SEARCH_ENDPOINT}
+ */
+ public void start(final String endpoint) {
+ try {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ stream = api.tweets().sampleStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else if (endpoint.equals(SEARCH_ENDPOINT)) {
+ stream = api.tweets().searchStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else {
+ throw new ProcessException("Endpoint was invalid value: " + endpoint);
+ }
+ } catch (ApiException e) {
+ logger.error("Received error {}: {}", e.getCode(), e.getMessage());
+ }
+
+ executorService.execute(new TweetQueuer());
Review Comment:
Checking for a null `stream` and throwing a `ProcessException` that says something like "could not make a connection to the Twitter API" would help people debug what's going on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
exceptionfactory commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r847285589
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,316 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("Twitter Endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("Base Path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("Bearer Token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("Queue Size")
+ .displayName("Queue Size")
+ .description("The size of the queue used to queue tweets")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("Batch Size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("Tweet Fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values are: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("User Fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user " +
+ "for proper usage. Possible field values are: " +
+ "created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, " +
+ "protected, public_metrics, url, username, verified, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor MEDIA_FIELDS = new PropertyDescriptor.Builder()
+ .name("Media Fields")
+ .displayName("Media Fields")
+ .description("A comma-separated list of media fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/media " +
+ "for proper usage. Possible field values are: " +
+ "alt_text, duration_ms, height, media_key, non_public_metrics, organic_metrics, preview_image_url, " +
+ "promoted_metrics, public_metrics, type, url, width")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor POLL_FIELDS = new PropertyDescriptor.Builder()
+ .name("Poll Fields")
+ .displayName("Poll Fields")
+ .description("A comma-separated list of poll fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/poll " +
+ "for proper usage. Possible field values are: " +
+ "duration_minutes, end_datetime, id, options, voting_status")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PLACE_FIELDS = new PropertyDescriptor.Builder()
+ .name("Place Fields")
+ .displayName("Place Fields")
+ .description("A comma-separated list of place fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/place " +
+ "for proper usage. Possible field values are: " +
+ "contained_within, country, country_code, full_name, geo, id, name, place_type")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor EXPANSIONS = new PropertyDescriptor.Builder()
+ .name("Expansions")
+ .displayName("Expansions")
+ .description("A comma-separated list of expansions for objects in the returned tweet. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/expansions " +
+ "for proper usage. Possible field values are: " +
+ "author_id, referenced_tweets.id, referenced_tweets.id.author_id, entities.mentions.username, " +
+ "attachments.poll_ids, attachments.media_keys ,in_reply_to_user_id, geo.place_id")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor BACKFILL_MINUTES = new PropertyDescriptor.Builder()
+ .name("Backfill Minutes")
+ .displayName("Backfill Minutes")
+ .description("The number of minutes (up to 5 minutes) of streaming data to be requested after a " +
+ "disconnect. Only available for project with academic research access. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/integrate/" +
+ "recovery-and-redundancy-features")
+ .required(true)
+ .defaultValue("0")
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("All tweets will be routed to this relationship.")
+ .build();
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+
+ private TwitterStreamAPI api;
+
+ private volatile BlockingQueue<String> messageQueue;
+
+ private void emptyQueue() {
+ while (!messageQueue.isEmpty()) {
+ messageQueue.poll();
+ }
+ }
+
+ @Override
+ protected void init(ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptors = new ArrayList<>();
+ descriptors.add(ENDPOINT);
+ descriptors.add(BASE_PATH);
+ descriptors.add(BEARER_TOKEN);
+ descriptors.add(QUEUE_SIZE);
+ descriptors.add(BATCH_SIZE);
+ descriptors.add(TWEET_FIELDS);
+ descriptors.add(USER_FIELDS);
+ descriptors.add(MEDIA_FIELDS);
+ descriptors.add(POLL_FIELDS);
+ descriptors.add(PLACE_FIELDS);
+ descriptors.add(EXPANSIONS);
+ descriptors.add(BACKFILL_MINUTES);
+
+ this.descriptors = Collections.unmodifiableList(descriptors);
+
+ final Set<Relationship> relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ this.relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return this.descriptors;
+ }
+
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ messageQueue = new LinkedBlockingQueue<>(context.getProperty(QUEUE_SIZE).asInteger());
+
+ api = new TwitterStreamAPI(context, messageQueue, getLogger());
+ api.setBasePath(context.getProperty(BASE_PATH).getValue());
+ final String endpointName = context.getProperty(ENDPOINT).getValue();
+ if (ENDPOINT_SAMPLE.getValue().equals(endpointName)) {
+ api.start(TwitterStreamAPI.SAMPLE_ENDPOINT);
+ } else if (ENDPOINT_SEARCH.getValue().equals(endpointName)) {
+ api.start(TwitterStreamAPI.SEARCH_ENDPOINT);
+ } else {
+ throw new AssertionError("Endpoint was invalid value: " + endpointName);
+ }
+ }
+
+ @Override
+ public void onTrigger(ProcessContext context, ProcessSession session) throws ProcessException {
+ if (messageQueue.isEmpty()) {
+ context.yield();
+ return;
+ }
+
+ FlowFile flowFile = session.create();
+ flowFile = session.write(flowFile, new OutputStreamCallback() {
Review Comment:
Concurring with @emiliosetiadarma, writing the strings directly from the Twitter API response stream seems like the optimal approach. The Twitter SDK follows the same strategy, unfortunately it doesn't provide a clean way to control the lifecycle of the thread responsible for reading the stream. Although this approach requires expecting each string to contain a well-formed object, this seems acceptable given the performance considerations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r847672866
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,316 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
Review Comment:
Made the changes! Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r848782099
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,318 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("consume-twitter-endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("The size of the queue used to queue tweets")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("tweet-fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values are: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("user-fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user " +
+ "for proper usage. Possible field values are: " +
+ "created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, " +
+ "protected, public_metrics, url, username, verified, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor MEDIA_FIELDS = new PropertyDescriptor.Builder()
+ .name("media-fields")
+ .displayName("Media Fields")
+ .description("A comma-separated list of media fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/media " +
+ "for proper usage. Possible field values are: " +
+ "alt_text, duration_ms, height, media_key, non_public_metrics, organic_metrics, preview_image_url, " +
+ "promoted_metrics, public_metrics, type, url, width")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor POLL_FIELDS = new PropertyDescriptor.Builder()
+ .name("poll-fields")
+ .displayName("Poll Fields")
+ .description("A comma-separated list of poll fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/poll " +
+ "for proper usage. Possible field values are: " +
+ "duration_minutes, end_datetime, id, options, voting_status")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PLACE_FIELDS = new PropertyDescriptor.Builder()
+ .name("place-fields")
+ .displayName("Place Fields")
+ .description("A comma-separated list of place fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/place " +
+ "for proper usage. Possible field values are: " +
+ "contained_within, country, country_code, full_name, geo, id, name, place_type")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor EXPANSIONS = new PropertyDescriptor.Builder()
+ .name("expansions")
+ .displayName("Expansions")
+ .description("A comma-separated list of expansions for objects in the returned tweet. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/expansions " +
+ "for proper usage. Possible field values are: " +
Review Comment:
Making the changes
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,318 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("consume-twitter-endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("The size of the queue used to queue tweets")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("tweet-fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values are: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("user-fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user " +
+ "for proper usage. Possible field values are: " +
+ "created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, " +
+ "protected, public_metrics, url, username, verified, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor MEDIA_FIELDS = new PropertyDescriptor.Builder()
+ .name("media-fields")
+ .displayName("Media Fields")
+ .description("A comma-separated list of media fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/media " +
+ "for proper usage. Possible field values are: " +
+ "alt_text, duration_ms, height, media_key, non_public_metrics, organic_metrics, preview_image_url, " +
+ "promoted_metrics, public_metrics, type, url, width")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor POLL_FIELDS = new PropertyDescriptor.Builder()
+ .name("poll-fields")
+ .displayName("Poll Fields")
+ .description("A comma-separated list of poll fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/poll " +
+ "for proper usage. Possible field values are: " +
+ "duration_minutes, end_datetime, id, options, voting_status")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PLACE_FIELDS = new PropertyDescriptor.Builder()
+ .name("place-fields")
+ .displayName("Place Fields")
+ .description("A comma-separated list of place fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/place " +
+ "for proper usage. Possible field values are: " +
+ "contained_within, country, country_code, full_name, geo, id, name, place_type")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor EXPANSIONS = new PropertyDescriptor.Builder()
+ .name("expansions")
+ .displayName("Expansions")
+ .description("A comma-separated list of expansions for objects in the returned tweet. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/expansions " +
+ "for proper usage. Possible field values are: " +
+ "author_id, referenced_tweets.id, referenced_tweets.id.author_id, entities.mentions.username, " +
+ "attachments.poll_ids, attachments.media_keys ,in_reply_to_user_id, geo.place_id")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor BACKFILL_MINUTES = new PropertyDescriptor.Builder()
+ .name("backfill-minutes")
+ .displayName("Backfill Minutes")
+ .description("The number of minutes (up to 5 minutes) of streaming data to be requested after a " +
+ "disconnect. Only available for project with academic research access. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/integrate/" +
+ "recovery-and-redundancy-features")
+ .required(true)
+ .defaultValue("0")
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("All tweets will be routed to this relationship.")
Review Comment:
Making the changes
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,318 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("consume-twitter-endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("The size of the queue used to queue tweets")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("tweet-fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values are: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("user-fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user " +
+ "for proper usage. Possible field values are: " +
+ "created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, " +
+ "protected, public_metrics, url, username, verified, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor MEDIA_FIELDS = new PropertyDescriptor.Builder()
+ .name("media-fields")
+ .displayName("Media Fields")
+ .description("A comma-separated list of media fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/media " +
+ "for proper usage. Possible field values are: " +
+ "alt_text, duration_ms, height, media_key, non_public_metrics, organic_metrics, preview_image_url, " +
+ "promoted_metrics, public_metrics, type, url, width")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor POLL_FIELDS = new PropertyDescriptor.Builder()
+ .name("poll-fields")
+ .displayName("Poll Fields")
+ .description("A comma-separated list of poll fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/poll " +
+ "for proper usage. Possible field values are: " +
+ "duration_minutes, end_datetime, id, options, voting_status")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PLACE_FIELDS = new PropertyDescriptor.Builder()
+ .name("place-fields")
+ .displayName("Place Fields")
+ .description("A comma-separated list of place fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/place " +
+ "for proper usage. Possible field values are: " +
+ "contained_within, country, country_code, full_name, geo, id, name, place_type")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor EXPANSIONS = new PropertyDescriptor.Builder()
+ .name("expansions")
+ .displayName("Expansions")
+ .description("A comma-separated list of expansions for objects in the returned tweet. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/expansions " +
+ "for proper usage. Possible field values are: " +
+ "author_id, referenced_tweets.id, referenced_tweets.id.author_id, entities.mentions.username, " +
+ "attachments.poll_ids, attachments.media_keys ,in_reply_to_user_id, geo.place_id")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor BACKFILL_MINUTES = new PropertyDescriptor.Builder()
+ .name("backfill-minutes")
+ .displayName("Backfill Minutes")
+ .description("The number of minutes (up to 5 minutes) of streaming data to be requested after a " +
+ "disconnect. Only available for project with academic research access. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/integrate/" +
+ "recovery-and-redundancy-features")
+ .required(true)
+ .defaultValue("0")
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("All tweets will be routed to this relationship.")
+ .build();
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+
+ private TwitterStreamAPI api;
+
+ private volatile BlockingQueue<String> messageQueue;
+
+ private void emptyQueue() {
+ while (!messageQueue.isEmpty()) {
+ messageQueue.poll();
+ }
+ }
+
+ @Override
+ protected void init(ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptors = new ArrayList<>();
+ descriptors.add(ENDPOINT);
+ descriptors.add(BASE_PATH);
+ descriptors.add(BEARER_TOKEN);
+ descriptors.add(QUEUE_SIZE);
+ descriptors.add(BATCH_SIZE);
+ descriptors.add(TWEET_FIELDS);
+ descriptors.add(USER_FIELDS);
+ descriptors.add(MEDIA_FIELDS);
+ descriptors.add(POLL_FIELDS);
+ descriptors.add(PLACE_FIELDS);
+ descriptors.add(EXPANSIONS);
+ descriptors.add(BACKFILL_MINUTES);
+
+ this.descriptors = Collections.unmodifiableList(descriptors);
+
+ final Set<Relationship> relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ this.relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return this.descriptors;
+ }
+
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ messageQueue = new LinkedBlockingQueue<>(context.getProperty(QUEUE_SIZE).asInteger());
+
+ api = new TwitterStreamAPI(context, messageQueue, getLogger());
+ api.setBasePath(context.getProperty(BASE_PATH).getValue());
+ final String endpointName = context.getProperty(ENDPOINT).getValue();
+ if (ENDPOINT_SAMPLE.getValue().equals(endpointName)) {
+ api.start(TwitterStreamAPI.SAMPLE_ENDPOINT);
+ } else if (ENDPOINT_SEARCH.getValue().equals(endpointName)) {
+ api.start(TwitterStreamAPI.SEARCH_ENDPOINT);
+ } else {
+ throw new ProcessException("Endpoint was invalid value: " + endpointName);
+ }
+ }
+
+ @Override
+ public void onTrigger(ProcessContext context, ProcessSession session) throws ProcessException {
+ if (messageQueue.isEmpty()) {
+ context.yield();
+ return;
+ }
+
+ FlowFile flowFile = session.create();
+ flowFile = session.write(flowFile, new OutputStreamCallback() {
+ @Override
+ public void process(OutputStream out) throws IOException {
+ out.write("[".getBytes(StandardCharsets.UTF_8));
+ final int batchSize = context.getProperty(BATCH_SIZE).asInteger();
+ String tweet = messageQueue.poll();
+ out.write(tweet.getBytes(StandardCharsets.UTF_8));
+ int tweetCount = 1;
+ while (tweetCount < batchSize && !messageQueue.isEmpty()) {
+ out.write(",".getBytes(StandardCharsets.UTF_8));
+ tweet = messageQueue.poll();
+ out.write(tweet.getBytes(StandardCharsets.UTF_8));
+ tweetCount++;
+ }
+ out.write("]".getBytes(StandardCharsets.UTF_8));
+ }
+ });
+
+ final Map<String, String> attributes = new HashMap<>();
+ attributes.put(CoreAttributes.MIME_TYPE.key(), "application/json");
+ attributes.put(CoreAttributes.FILENAME.key(), flowFile.getAttribute(CoreAttributes.FILENAME.key()) + ".json");
Review Comment:
Making the changes
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,318 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("consume-twitter-endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("The size of the queue used to queue tweets")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("tweet-fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values are: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("user-fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user " +
+ "for proper usage. Possible field values are: " +
+ "created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, " +
+ "protected, public_metrics, url, username, verified, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor MEDIA_FIELDS = new PropertyDescriptor.Builder()
+ .name("media-fields")
+ .displayName("Media Fields")
+ .description("A comma-separated list of media fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/media " +
+ "for proper usage. Possible field values are: " +
+ "alt_text, duration_ms, height, media_key, non_public_metrics, organic_metrics, preview_image_url, " +
+ "promoted_metrics, public_metrics, type, url, width")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor POLL_FIELDS = new PropertyDescriptor.Builder()
+ .name("poll-fields")
+ .displayName("Poll Fields")
+ .description("A comma-separated list of poll fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/poll " +
+ "for proper usage. Possible field values are: " +
+ "duration_minutes, end_datetime, id, options, voting_status")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PLACE_FIELDS = new PropertyDescriptor.Builder()
+ .name("place-fields")
+ .displayName("Place Fields")
+ .description("A comma-separated list of place fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/place " +
+ "for proper usage. Possible field values are: " +
+ "contained_within, country, country_code, full_name, geo, id, name, place_type")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor EXPANSIONS = new PropertyDescriptor.Builder()
+ .name("expansions")
+ .displayName("Expansions")
+ .description("A comma-separated list of expansions for objects in the returned tweet. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/expansions " +
+ "for proper usage. Possible field values are: " +
+ "author_id, referenced_tweets.id, referenced_tweets.id.author_id, entities.mentions.username, " +
+ "attachments.poll_ids, attachments.media_keys ,in_reply_to_user_id, geo.place_id")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor BACKFILL_MINUTES = new PropertyDescriptor.Builder()
+ .name("backfill-minutes")
+ .displayName("Backfill Minutes")
+ .description("The number of minutes (up to 5 minutes) of streaming data to be requested after a " +
+ "disconnect. Only available for project with academic research access. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/integrate/" +
+ "recovery-and-redundancy-features")
+ .required(true)
+ .defaultValue("0")
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("All tweets will be routed to this relationship.")
+ .build();
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+
+ private TwitterStreamAPI api;
+
+ private volatile BlockingQueue<String> messageQueue;
+
+ private void emptyQueue() {
+ while (!messageQueue.isEmpty()) {
+ messageQueue.poll();
+ }
+ }
+
+ @Override
+ protected void init(ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptors = new ArrayList<>();
+ descriptors.add(ENDPOINT);
+ descriptors.add(BASE_PATH);
+ descriptors.add(BEARER_TOKEN);
+ descriptors.add(QUEUE_SIZE);
+ descriptors.add(BATCH_SIZE);
+ descriptors.add(TWEET_FIELDS);
+ descriptors.add(USER_FIELDS);
+ descriptors.add(MEDIA_FIELDS);
+ descriptors.add(POLL_FIELDS);
+ descriptors.add(PLACE_FIELDS);
+ descriptors.add(EXPANSIONS);
+ descriptors.add(BACKFILL_MINUTES);
+
+ this.descriptors = Collections.unmodifiableList(descriptors);
+
+ final Set<Relationship> relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ this.relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return this.descriptors;
+ }
+
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ messageQueue = new LinkedBlockingQueue<>(context.getProperty(QUEUE_SIZE).asInteger());
+
+ api = new TwitterStreamAPI(context, messageQueue, getLogger());
+ api.setBasePath(context.getProperty(BASE_PATH).getValue());
+ final String endpointName = context.getProperty(ENDPOINT).getValue();
+ if (ENDPOINT_SAMPLE.getValue().equals(endpointName)) {
+ api.start(TwitterStreamAPI.SAMPLE_ENDPOINT);
+ } else if (ENDPOINT_SEARCH.getValue().equals(endpointName)) {
+ api.start(TwitterStreamAPI.SEARCH_ENDPOINT);
+ } else {
+ throw new ProcessException("Endpoint was invalid value: " + endpointName);
+ }
+ }
+
+ @Override
+ public void onTrigger(ProcessContext context, ProcessSession session) throws ProcessException {
+ if (messageQueue.isEmpty()) {
+ context.yield();
+ return;
+ }
+
+ FlowFile flowFile = session.create();
+ flowFile = session.write(flowFile, new OutputStreamCallback() {
+ @Override
+ public void process(OutputStream out) throws IOException {
+ out.write("[".getBytes(StandardCharsets.UTF_8));
Review Comment:
Making the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r864057479
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,366 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.UUID;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue(StreamEndpoint.SAMPLE_ENDPOINT.getEndpointName(),
+ "Sample Stream",
+ "Streams about one percent of all Tweets. " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/volume-streams/api-reference/get-tweets-sample-stream");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue(StreamEndpoint.SEARCH_ENDPOINT.getEndpointName(),
+ "Search Stream",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty. " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/get-tweets-search-stream");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("stream-endpoint")
+ .displayName("Stream Endpoint")
+ .description("Specifies which endpoint tweets should be streamed. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("Maximum size of internal queue for streamed messages")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor BACKOFF_ATTEMPTS = new PropertyDescriptor.Builder()
+ .name("backoff-attempts")
+ .displayName("Backoff Attempts")
+ .description("The number of reconnection tries the processor will attempt in the event of " +
+ "a disconnection of the stream for any reason, before throwing an exception. To start a stream after " +
+ "this exception occur and the connection is fixed, please stop and restart the processor. If the value" +
+ "of this property is 0, then backoff will never occur and the processor will always need to be restarted" +
+ "if the stream fails.")
+ .required(true)
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .defaultValue("5")
+ .build();
+ public static final PropertyDescriptor BACKOFF_TIME = new PropertyDescriptor.Builder()
+ .name("backoff-time")
+ .displayName("Backoff Time")
+ .description("The duration to backoff to start attempting a new stream if" +
+ "the current one fails for any reason. Will increase by factor of 2 every time a restart fails")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("1 mins")
+ .build();
+ public static final PropertyDescriptor MAXIMUM_BACKOFF_TIME = new PropertyDescriptor.Builder()
+ .name("maximum-backoff-time")
+ .displayName("Maximum Backoff Time")
+ .description("The maximum duration to backoff to start attempting a new stream." +
+ "It is recommended that this number be much higher than the 'Backoff Time' property")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("5 mins")
+ .build();
+ public static final PropertyDescriptor CONNECT_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("connect-timeout")
+ .displayName("Connect Timeout")
+ .description("The maximum time in which client should establish a connection with the " +
+ "Twitter API before a time out. A value of 0 will mean no timeout at all")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("10 secs")
+ .build();
+ public static final PropertyDescriptor READ_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("read-timeout")
+ .displayName("Read Timeout")
+ .description("The maximum time of inactivity between receiving tweets from Twitter through " +
+ "the API before a timeout. A value of 0 will mean no timeout at all")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("10 secs")
+ .build();
+ public static final PropertyDescriptor BACKFILL_MINUTES = new PropertyDescriptor.Builder()
+ .name("backfill-minutes")
+ .displayName("Backfill Minutes")
+ .description("The number of minutes (up to 5 minutes) of streaming data to be requested after a " +
+ "disconnect. Only available for project with academic research access. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/integrate/" +
+ "recovery-and-redundancy-features")
+ .required(true)
+ .defaultValue("0")
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("tweet-fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values include: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("user-fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user " +
+ "for proper usage. Possible field values include: " +
+ "created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, " +
+ "protected, public_metrics, url, username, verified, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor MEDIA_FIELDS = new PropertyDescriptor.Builder()
+ .name("media-fields")
+ .displayName("Media Fields")
+ .description("A comma-separated list of media fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/media " +
+ "for proper usage. Possible field values include: " +
+ "alt_text, duration_ms, height, media_key, non_public_metrics, organic_metrics, preview_image_url, " +
+ "promoted_metrics, public_metrics, type, url, width")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor POLL_FIELDS = new PropertyDescriptor.Builder()
+ .name("poll-fields")
+ .displayName("Poll Fields")
+ .description("A comma-separated list of poll fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/poll " +
+ "for proper usage. Possible field values include: " +
+ "duration_minutes, end_datetime, id, options, voting_status")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PLACE_FIELDS = new PropertyDescriptor.Builder()
+ .name("place-fields")
+ .displayName("Place Fields")
+ .description("A comma-separated list of place fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/place " +
+ "for proper usage. Possible field values include: " +
+ "contained_within, country, country_code, full_name, geo, id, name, place_type")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor EXPANSIONS = new PropertyDescriptor.Builder()
+ .name("expansions")
+ .displayName("Expansions")
+ .description("A comma-separated list of expansions for objects in the returned tweet. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/expansions " +
+ "for proper usage. Possible field values include: " +
+ "author_id, referenced_tweets.id, referenced_tweets.id.author_id, entities.mentions.username, " +
+ "attachments.poll_ids, attachments.media_keys ,in_reply_to_user_id, geo.place_id")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("FlowFiles containing an array of one or more Tweets")
+ .build();
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+
+ private TweetStreamService tweetStreamService;
+
+ private volatile BlockingQueue<String> messageQueue;
+
+ @Override
+ protected void init(ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptors = new ArrayList<>();
+ descriptors.add(ENDPOINT);
+ descriptors.add(BASE_PATH);
+ descriptors.add(BEARER_TOKEN);
+ descriptors.add(QUEUE_SIZE);
+ descriptors.add(BATCH_SIZE);
+ descriptors.add(BACKOFF_ATTEMPTS);
+ descriptors.add(BACKOFF_TIME);
+ descriptors.add(MAXIMUM_BACKOFF_TIME);
+ descriptors.add(CONNECT_TIMEOUT);
+ descriptors.add(READ_TIMEOUT);
+ descriptors.add(BACKFILL_MINUTES);
+ descriptors.add(TWEET_FIELDS);
+ descriptors.add(USER_FIELDS);
+ descriptors.add(MEDIA_FIELDS);
+ descriptors.add(POLL_FIELDS);
+ descriptors.add(PLACE_FIELDS);
+ descriptors.add(EXPANSIONS);
+
+ this.descriptors = Collections.unmodifiableList(descriptors);
+
+ final Set<Relationship> relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ this.relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return this.descriptors;
+ }
+
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ messageQueue = new LinkedBlockingQueue<>(context.getProperty(QUEUE_SIZE).asInteger());
+
+ tweetStreamService = new TweetStreamService(context, messageQueue, getLogger());
+ tweetStreamService.setBasePath(context.getProperty(BASE_PATH).getValue());
+ final String endpointName = context.getProperty(ENDPOINT).getValue();
+ if (ENDPOINT_SAMPLE.getValue().equals(endpointName)) {
+ tweetStreamService.start(StreamEndpoint.SAMPLE_ENDPOINT);
+ } else {
+ tweetStreamService.start(StreamEndpoint.SEARCH_ENDPOINT);
+ }
+ }
+
+ @Override
+ public void onTrigger(ProcessContext context, ProcessSession session) throws ProcessException {
+ if (messageQueue.isEmpty()) {
+ context.yield();
+ return;
+ }
+
+ FlowFile flowFile = session.create();
+ flowFile = session.write(flowFile, new OutputStreamCallback() {
+ @Override
+ public void process(OutputStream out) throws IOException {
+
+ final int batchSize = context.getProperty(BATCH_SIZE).asInteger();
+ String tweet = messageQueue.poll();
+ int tweetCount = 1;
Review Comment:
Making the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r864042109
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,366 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.UUID;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue(StreamEndpoint.SAMPLE_ENDPOINT.getEndpointName(),
+ "Sample Stream",
+ "Streams about one percent of all Tweets. " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/volume-streams/api-reference/get-tweets-sample-stream");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue(StreamEndpoint.SEARCH_ENDPOINT.getEndpointName(),
+ "Search Stream",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty. " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/get-tweets-search-stream");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("stream-endpoint")
+ .displayName("Stream Endpoint")
+ .description("Specifies which endpoint tweets should be streamed. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("Maximum size of internal queue for streamed messages")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor BACKOFF_ATTEMPTS = new PropertyDescriptor.Builder()
+ .name("backoff-attempts")
+ .displayName("Backoff Attempts")
+ .description("The number of reconnection tries the processor will attempt in the event of " +
+ "a disconnection of the stream for any reason, before throwing an exception. To start a stream after " +
+ "this exception occur and the connection is fixed, please stop and restart the processor. If the value" +
+ "of this property is 0, then backoff will never occur and the processor will always need to be restarted" +
+ "if the stream fails.")
+ .required(true)
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .defaultValue("5")
+ .build();
+ public static final PropertyDescriptor BACKOFF_TIME = new PropertyDescriptor.Builder()
+ .name("backoff-time")
+ .displayName("Backoff Time")
+ .description("The duration to backoff to start attempting a new stream if" +
+ "the current one fails for any reason. Will increase by factor of 2 every time a restart fails")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("1 mins")
+ .build();
+ public static final PropertyDescriptor MAXIMUM_BACKOFF_TIME = new PropertyDescriptor.Builder()
+ .name("maximum-backoff-time")
+ .displayName("Maximum Backoff Time")
+ .description("The maximum duration to backoff to start attempting a new stream." +
+ "It is recommended that this number be much higher than the 'Backoff Time' property")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("5 mins")
+ .build();
+ public static final PropertyDescriptor CONNECT_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("connect-timeout")
+ .displayName("Connect Timeout")
+ .description("The maximum time in which client should establish a connection with the " +
+ "Twitter API before a time out. A value of 0 will mean no timeout at all")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("10 secs")
+ .build();
+ public static final PropertyDescriptor READ_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("read-timeout")
+ .displayName("Read Timeout")
+ .description("The maximum time of inactivity between receiving tweets from Twitter through " +
+ "the API before a timeout. A value of 0 will mean no timeout at all")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("10 secs")
+ .build();
+ public static final PropertyDescriptor BACKFILL_MINUTES = new PropertyDescriptor.Builder()
+ .name("backfill-minutes")
+ .displayName("Backfill Minutes")
+ .description("The number of minutes (up to 5 minutes) of streaming data to be requested after a " +
+ "disconnect. Only available for project with academic research access. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/integrate/" +
+ "recovery-and-redundancy-features")
+ .required(true)
+ .defaultValue("0")
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("tweet-fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values include: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("user-fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user " +
+ "for proper usage. Possible field values include: " +
+ "created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, " +
+ "protected, public_metrics, url, username, verified, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor MEDIA_FIELDS = new PropertyDescriptor.Builder()
+ .name("media-fields")
+ .displayName("Media Fields")
+ .description("A comma-separated list of media fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/media " +
+ "for proper usage. Possible field values include: " +
+ "alt_text, duration_ms, height, media_key, non_public_metrics, organic_metrics, preview_image_url, " +
+ "promoted_metrics, public_metrics, type, url, width")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor POLL_FIELDS = new PropertyDescriptor.Builder()
+ .name("poll-fields")
+ .displayName("Poll Fields")
+ .description("A comma-separated list of poll fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/poll " +
+ "for proper usage. Possible field values include: " +
+ "duration_minutes, end_datetime, id, options, voting_status")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PLACE_FIELDS = new PropertyDescriptor.Builder()
+ .name("place-fields")
+ .displayName("Place Fields")
+ .description("A comma-separated list of place fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/place " +
+ "for proper usage. Possible field values include: " +
+ "contained_within, country, country_code, full_name, geo, id, name, place_type")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor EXPANSIONS = new PropertyDescriptor.Builder()
+ .name("expansions")
+ .displayName("Expansions")
+ .description("A comma-separated list of expansions for objects in the returned tweet. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/expansions " +
+ "for proper usage. Possible field values include: " +
+ "author_id, referenced_tweets.id, referenced_tweets.id.author_id, entities.mentions.username, " +
+ "attachments.poll_ids, attachments.media_keys ,in_reply_to_user_id, geo.place_id")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("FlowFiles containing an array of one or more Tweets")
+ .build();
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+
+ private TweetStreamService tweetStreamService;
+
+ private volatile BlockingQueue<String> messageQueue;
+
+ @Override
+ protected void init(ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptors = new ArrayList<>();
+ descriptors.add(ENDPOINT);
+ descriptors.add(BASE_PATH);
+ descriptors.add(BEARER_TOKEN);
+ descriptors.add(QUEUE_SIZE);
+ descriptors.add(BATCH_SIZE);
+ descriptors.add(BACKOFF_ATTEMPTS);
+ descriptors.add(BACKOFF_TIME);
+ descriptors.add(MAXIMUM_BACKOFF_TIME);
+ descriptors.add(CONNECT_TIMEOUT);
+ descriptors.add(READ_TIMEOUT);
+ descriptors.add(BACKFILL_MINUTES);
+ descriptors.add(TWEET_FIELDS);
+ descriptors.add(USER_FIELDS);
+ descriptors.add(MEDIA_FIELDS);
+ descriptors.add(POLL_FIELDS);
+ descriptors.add(PLACE_FIELDS);
+ descriptors.add(EXPANSIONS);
+
+ this.descriptors = Collections.unmodifiableList(descriptors);
+
+ final Set<Relationship> relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ this.relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return this.descriptors;
+ }
+
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ messageQueue = new LinkedBlockingQueue<>(context.getProperty(QUEUE_SIZE).asInteger());
+
+ tweetStreamService = new TweetStreamService(context, messageQueue, getLogger());
+ tweetStreamService.setBasePath(context.getProperty(BASE_PATH).getValue());
+ final String endpointName = context.getProperty(ENDPOINT).getValue();
Review Comment:
Making the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r864030625
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,366 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.UUID;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue(StreamEndpoint.SAMPLE_ENDPOINT.getEndpointName(),
+ "Sample Stream",
+ "Streams about one percent of all Tweets. " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/volume-streams/api-reference/get-tweets-sample-stream");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue(StreamEndpoint.SEARCH_ENDPOINT.getEndpointName(),
+ "Search Stream",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty. " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/get-tweets-search-stream");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("stream-endpoint")
+ .displayName("Stream Endpoint")
+ .description("Specifies which endpoint tweets should be streamed. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("Maximum size of internal queue for streamed messages")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
Review Comment:
Making the changes
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,366 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.UUID;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue(StreamEndpoint.SAMPLE_ENDPOINT.getEndpointName(),
+ "Sample Stream",
+ "Streams about one percent of all Tweets. " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/volume-streams/api-reference/get-tweets-sample-stream");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue(StreamEndpoint.SEARCH_ENDPOINT.getEndpointName(),
+ "Search Stream",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty. " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/get-tweets-search-stream");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("stream-endpoint")
+ .displayName("Stream Endpoint")
+ .description("Specifies which endpoint tweets should be streamed. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
Review Comment:
Making the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r846717146
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,316 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("Twitter Endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("Base Path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("Bearer Token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("Queue Size")
+ .displayName("Queue Size")
+ .description("The size of the queue used to queue tweets")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("Batch Size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("Tweet Fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values are: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("User Fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user " +
+ "for proper usage. Possible field values are: " +
+ "created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, " +
+ "protected, public_metrics, url, username, verified, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor MEDIA_FIELDS = new PropertyDescriptor.Builder()
+ .name("Media Fields")
+ .displayName("Media Fields")
+ .description("A comma-separated list of media fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/media " +
+ "for proper usage. Possible field values are: " +
+ "alt_text, duration_ms, height, media_key, non_public_metrics, organic_metrics, preview_image_url, " +
+ "promoted_metrics, public_metrics, type, url, width")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor POLL_FIELDS = new PropertyDescriptor.Builder()
+ .name("Poll Fields")
+ .displayName("Poll Fields")
+ .description("A comma-separated list of poll fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/poll " +
+ "for proper usage. Possible field values are: " +
+ "duration_minutes, end_datetime, id, options, voting_status")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PLACE_FIELDS = new PropertyDescriptor.Builder()
+ .name("Place Fields")
+ .displayName("Place Fields")
+ .description("A comma-separated list of place fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/place " +
+ "for proper usage. Possible field values are: " +
+ "contained_within, country, country_code, full_name, geo, id, name, place_type")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor EXPANSIONS = new PropertyDescriptor.Builder()
+ .name("Expansions")
+ .displayName("Expansions")
+ .description("A comma-separated list of expansions for objects in the returned tweet. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/expansions " +
+ "for proper usage. Possible field values are: " +
+ "author_id, referenced_tweets.id, referenced_tweets.id.author_id, entities.mentions.username, " +
+ "attachments.poll_ids, attachments.media_keys ,in_reply_to_user_id, geo.place_id")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor BACKFILL_MINUTES = new PropertyDescriptor.Builder()
+ .name("Backfill Minutes")
+ .displayName("Backfill Minutes")
+ .description("The number of minutes (up to 5 minutes) of streaming data to be requested after a " +
+ "disconnect. Only available for project with academic research access. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/integrate/" +
+ "recovery-and-redundancy-features")
+ .required(true)
+ .defaultValue("0")
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("All tweets will be routed to this relationship.")
+ .build();
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+
+ private TwitterStreamAPI api;
+
+ private volatile BlockingQueue<String> messageQueue;
+
+ private void emptyQueue() {
+ while (!messageQueue.isEmpty()) {
+ messageQueue.poll();
+ }
+ }
+
+ @Override
+ protected void init(ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptors = new ArrayList<>();
+ descriptors.add(ENDPOINT);
+ descriptors.add(BASE_PATH);
+ descriptors.add(BEARER_TOKEN);
+ descriptors.add(QUEUE_SIZE);
+ descriptors.add(BATCH_SIZE);
+ descriptors.add(TWEET_FIELDS);
+ descriptors.add(USER_FIELDS);
+ descriptors.add(MEDIA_FIELDS);
+ descriptors.add(POLL_FIELDS);
+ descriptors.add(PLACE_FIELDS);
+ descriptors.add(EXPANSIONS);
+ descriptors.add(BACKFILL_MINUTES);
+
+ this.descriptors = Collections.unmodifiableList(descriptors);
+
+ final Set<Relationship> relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ this.relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return this.descriptors;
+ }
+
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ messageQueue = new LinkedBlockingQueue<>(context.getProperty(QUEUE_SIZE).asInteger());
+
+ api = new TwitterStreamAPI(context, messageQueue, getLogger());
+ api.setBasePath(context.getProperty(BASE_PATH).getValue());
+ final String endpointName = context.getProperty(ENDPOINT).getValue();
+ if (ENDPOINT_SAMPLE.getValue().equals(endpointName)) {
+ api.start(TwitterStreamAPI.SAMPLE_ENDPOINT);
+ } else if (ENDPOINT_SEARCH.getValue().equals(endpointName)) {
+ api.start(TwitterStreamAPI.SEARCH_ENDPOINT);
+ } else {
+ throw new AssertionError("Endpoint was invalid value: " + endpointName);
+ }
+ }
+
+ @Override
+ public void onTrigger(ProcessContext context, ProcessSession session) throws ProcessException {
+ if (messageQueue.isEmpty()) {
+ context.yield();
+ return;
+ }
+
+ FlowFile flowFile = session.create();
+ flowFile = session.write(flowFile, new OutputStreamCallback() {
Review Comment:
Thanks for the comment. A few thoughts. Would that consume too much memory though? Especially if the batch size is small and the maximum queue size is large, the `BlockingQueue` that the thread uses to store the tweets could be larger as opposed to when the contents of the queue were just a string.
Also, as of current writing, I'm sure enough that we don't have to worry about malformed tweets as part of the stream that is returned by the API. If the GET request for the stream is invalid, the underlying API provided by the Twitter Java SDK throws an `ApiException`. Since that's the case writing strings into the `OutputStream` directly seems efficient.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r846715607
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,316 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("Twitter Endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("Base Path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
Review Comment:
Twitter provides the bearer tokens when you sign up for the developer account as of this writing. They also offer a way to invalidate tokens through their API. See https://developer.twitter.com/en/docs/authentication/oauth-2-0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] MikeThomsen commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
MikeThomsen commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r847171433
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,316 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("Twitter Endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("Base Path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("Bearer Token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("Queue Size")
+ .displayName("Queue Size")
+ .description("The size of the queue used to queue tweets")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("Batch Size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("Tweet Fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values are: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("User Fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user " +
+ "for proper usage. Possible field values are: " +
+ "created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, " +
+ "protected, public_metrics, url, username, verified, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor MEDIA_FIELDS = new PropertyDescriptor.Builder()
+ .name("Media Fields")
+ .displayName("Media Fields")
+ .description("A comma-separated list of media fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/media " +
+ "for proper usage. Possible field values are: " +
+ "alt_text, duration_ms, height, media_key, non_public_metrics, organic_metrics, preview_image_url, " +
+ "promoted_metrics, public_metrics, type, url, width")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor POLL_FIELDS = new PropertyDescriptor.Builder()
+ .name("Poll Fields")
+ .displayName("Poll Fields")
+ .description("A comma-separated list of poll fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/poll " +
+ "for proper usage. Possible field values are: " +
+ "duration_minutes, end_datetime, id, options, voting_status")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PLACE_FIELDS = new PropertyDescriptor.Builder()
+ .name("Place Fields")
+ .displayName("Place Fields")
+ .description("A comma-separated list of place fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/place " +
+ "for proper usage. Possible field values are: " +
+ "contained_within, country, country_code, full_name, geo, id, name, place_type")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor EXPANSIONS = new PropertyDescriptor.Builder()
+ .name("Expansions")
+ .displayName("Expansions")
+ .description("A comma-separated list of expansions for objects in the returned tweet. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/expansions " +
+ "for proper usage. Possible field values are: " +
+ "author_id, referenced_tweets.id, referenced_tweets.id.author_id, entities.mentions.username, " +
+ "attachments.poll_ids, attachments.media_keys ,in_reply_to_user_id, geo.place_id")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor BACKFILL_MINUTES = new PropertyDescriptor.Builder()
+ .name("Backfill Minutes")
+ .displayName("Backfill Minutes")
+ .description("The number of minutes (up to 5 minutes) of streaming data to be requested after a " +
+ "disconnect. Only available for project with academic research access. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/integrate/" +
+ "recovery-and-redundancy-features")
+ .required(true)
+ .defaultValue("0")
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("All tweets will be routed to this relationship.")
+ .build();
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+
+ private TwitterStreamAPI api;
+
+ private volatile BlockingQueue<String> messageQueue;
+
+ private void emptyQueue() {
+ while (!messageQueue.isEmpty()) {
+ messageQueue.poll();
+ }
+ }
+
+ @Override
+ protected void init(ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptors = new ArrayList<>();
+ descriptors.add(ENDPOINT);
+ descriptors.add(BASE_PATH);
+ descriptors.add(BEARER_TOKEN);
+ descriptors.add(QUEUE_SIZE);
+ descriptors.add(BATCH_SIZE);
+ descriptors.add(TWEET_FIELDS);
+ descriptors.add(USER_FIELDS);
+ descriptors.add(MEDIA_FIELDS);
+ descriptors.add(POLL_FIELDS);
+ descriptors.add(PLACE_FIELDS);
+ descriptors.add(EXPANSIONS);
+ descriptors.add(BACKFILL_MINUTES);
+
+ this.descriptors = Collections.unmodifiableList(descriptors);
+
+ final Set<Relationship> relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ this.relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return this.descriptors;
+ }
+
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ messageQueue = new LinkedBlockingQueue<>(context.getProperty(QUEUE_SIZE).asInteger());
+
+ api = new TwitterStreamAPI(context, messageQueue, getLogger());
+ api.setBasePath(context.getProperty(BASE_PATH).getValue());
+ final String endpointName = context.getProperty(ENDPOINT).getValue();
+ if (ENDPOINT_SAMPLE.getValue().equals(endpointName)) {
+ api.start(TwitterStreamAPI.SAMPLE_ENDPOINT);
+ } else if (ENDPOINT_SEARCH.getValue().equals(endpointName)) {
+ api.start(TwitterStreamAPI.SEARCH_ENDPOINT);
+ } else {
+ throw new AssertionError("Endpoint was invalid value: " + endpointName);
+ }
+ }
+
+ @Override
+ public void onTrigger(ProcessContext context, ProcessSession session) throws ProcessException {
+ if (messageQueue.isEmpty()) {
+ context.yield();
+ return;
+ }
+
+ FlowFile flowFile = session.create();
+ flowFile = session.write(flowFile, new OutputStreamCallback() {
Review Comment:
Fair enough.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r848782320
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,318 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("consume-twitter-endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("The size of the queue used to queue tweets")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("tweet-fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values are: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("user-fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user " +
+ "for proper usage. Possible field values are: " +
+ "created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, " +
+ "protected, public_metrics, url, username, verified, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor MEDIA_FIELDS = new PropertyDescriptor.Builder()
+ .name("media-fields")
+ .displayName("Media Fields")
+ .description("A comma-separated list of media fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/media " +
+ "for proper usage. Possible field values are: " +
+ "alt_text, duration_ms, height, media_key, non_public_metrics, organic_metrics, preview_image_url, " +
+ "promoted_metrics, public_metrics, type, url, width")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor POLL_FIELDS = new PropertyDescriptor.Builder()
+ .name("poll-fields")
+ .displayName("Poll Fields")
+ .description("A comma-separated list of poll fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/poll " +
+ "for proper usage. Possible field values are: " +
+ "duration_minutes, end_datetime, id, options, voting_status")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PLACE_FIELDS = new PropertyDescriptor.Builder()
+ .name("place-fields")
+ .displayName("Place Fields")
+ .description("A comma-separated list of place fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/place " +
+ "for proper usage. Possible field values are: " +
+ "contained_within, country, country_code, full_name, geo, id, name, place_type")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor EXPANSIONS = new PropertyDescriptor.Builder()
+ .name("expansions")
+ .displayName("Expansions")
+ .description("A comma-separated list of expansions for objects in the returned tweet. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/expansions " +
+ "for proper usage. Possible field values are: " +
+ "author_id, referenced_tweets.id, referenced_tweets.id.author_id, entities.mentions.username, " +
+ "attachments.poll_ids, attachments.media_keys ,in_reply_to_user_id, geo.place_id")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor BACKFILL_MINUTES = new PropertyDescriptor.Builder()
+ .name("backfill-minutes")
+ .displayName("Backfill Minutes")
+ .description("The number of minutes (up to 5 minutes) of streaming data to be requested after a " +
+ "disconnect. Only available for project with academic research access. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/integrate/" +
+ "recovery-and-redundancy-features")
+ .required(true)
+ .defaultValue("0")
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("All tweets will be routed to this relationship.")
+ .build();
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+
+ private TwitterStreamAPI api;
+
+ private volatile BlockingQueue<String> messageQueue;
+
+ private void emptyQueue() {
+ while (!messageQueue.isEmpty()) {
+ messageQueue.poll();
+ }
+ }
+
+ @Override
+ protected void init(ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptors = new ArrayList<>();
+ descriptors.add(ENDPOINT);
+ descriptors.add(BASE_PATH);
+ descriptors.add(BEARER_TOKEN);
+ descriptors.add(QUEUE_SIZE);
+ descriptors.add(BATCH_SIZE);
+ descriptors.add(TWEET_FIELDS);
+ descriptors.add(USER_FIELDS);
+ descriptors.add(MEDIA_FIELDS);
+ descriptors.add(POLL_FIELDS);
+ descriptors.add(PLACE_FIELDS);
+ descriptors.add(EXPANSIONS);
+ descriptors.add(BACKFILL_MINUTES);
+
+ this.descriptors = Collections.unmodifiableList(descriptors);
+
+ final Set<Relationship> relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ this.relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return this.descriptors;
+ }
+
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ messageQueue = new LinkedBlockingQueue<>(context.getProperty(QUEUE_SIZE).asInteger());
+
+ api = new TwitterStreamAPI(context, messageQueue, getLogger());
+ api.setBasePath(context.getProperty(BASE_PATH).getValue());
+ final String endpointName = context.getProperty(ENDPOINT).getValue();
+ if (ENDPOINT_SAMPLE.getValue().equals(endpointName)) {
+ api.start(TwitterStreamAPI.SAMPLE_ENDPOINT);
+ } else if (ENDPOINT_SEARCH.getValue().equals(endpointName)) {
+ api.start(TwitterStreamAPI.SEARCH_ENDPOINT);
+ } else {
+ throw new ProcessException("Endpoint was invalid value: " + endpointName);
+ }
+ }
+
+ @Override
+ public void onTrigger(ProcessContext context, ProcessSession session) throws ProcessException {
+ if (messageQueue.isEmpty()) {
+ context.yield();
+ return;
+ }
+
+ FlowFile flowFile = session.create();
+ flowFile = session.write(flowFile, new OutputStreamCallback() {
+ @Override
+ public void process(OutputStream out) throws IOException {
+ out.write("[".getBytes(StandardCharsets.UTF_8));
+ final int batchSize = context.getProperty(BATCH_SIZE).asInteger();
+ String tweet = messageQueue.poll();
+ out.write(tweet.getBytes(StandardCharsets.UTF_8));
+ int tweetCount = 1;
+ while (tweetCount < batchSize && !messageQueue.isEmpty()) {
+ out.write(",".getBytes(StandardCharsets.UTF_8));
+ tweet = messageQueue.poll();
+ out.write(tweet.getBytes(StandardCharsets.UTF_8));
+ tweetCount++;
+ }
+ out.write("]".getBytes(StandardCharsets.UTF_8));
Review Comment:
Making the changes
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,318 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("consume-twitter-endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("The size of the queue used to queue tweets")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("tweet-fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values are: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("user-fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user " +
+ "for proper usage. Possible field values are: " +
+ "created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, " +
+ "protected, public_metrics, url, username, verified, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor MEDIA_FIELDS = new PropertyDescriptor.Builder()
+ .name("media-fields")
+ .displayName("Media Fields")
+ .description("A comma-separated list of media fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/media " +
+ "for proper usage. Possible field values are: " +
+ "alt_text, duration_ms, height, media_key, non_public_metrics, organic_metrics, preview_image_url, " +
+ "promoted_metrics, public_metrics, type, url, width")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor POLL_FIELDS = new PropertyDescriptor.Builder()
+ .name("poll-fields")
+ .displayName("Poll Fields")
+ .description("A comma-separated list of poll fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/poll " +
+ "for proper usage. Possible field values are: " +
+ "duration_minutes, end_datetime, id, options, voting_status")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PLACE_FIELDS = new PropertyDescriptor.Builder()
+ .name("place-fields")
+ .displayName("Place Fields")
+ .description("A comma-separated list of place fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/place " +
+ "for proper usage. Possible field values are: " +
+ "contained_within, country, country_code, full_name, geo, id, name, place_type")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor EXPANSIONS = new PropertyDescriptor.Builder()
+ .name("expansions")
+ .displayName("Expansions")
+ .description("A comma-separated list of expansions for objects in the returned tweet. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/expansions " +
+ "for proper usage. Possible field values are: " +
+ "author_id, referenced_tweets.id, referenced_tweets.id.author_id, entities.mentions.username, " +
+ "attachments.poll_ids, attachments.media_keys ,in_reply_to_user_id, geo.place_id")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor BACKFILL_MINUTES = new PropertyDescriptor.Builder()
+ .name("backfill-minutes")
+ .displayName("Backfill Minutes")
+ .description("The number of minutes (up to 5 minutes) of streaming data to be requested after a " +
+ "disconnect. Only available for project with academic research access. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/integrate/" +
+ "recovery-and-redundancy-features")
+ .required(true)
+ .defaultValue("0")
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("All tweets will be routed to this relationship.")
+ .build();
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+
+ private TwitterStreamAPI api;
+
+ private volatile BlockingQueue<String> messageQueue;
+
+ private void emptyQueue() {
+ while (!messageQueue.isEmpty()) {
+ messageQueue.poll();
+ }
+ }
+
+ @Override
+ protected void init(ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptors = new ArrayList<>();
+ descriptors.add(ENDPOINT);
+ descriptors.add(BASE_PATH);
+ descriptors.add(BEARER_TOKEN);
+ descriptors.add(QUEUE_SIZE);
+ descriptors.add(BATCH_SIZE);
+ descriptors.add(TWEET_FIELDS);
+ descriptors.add(USER_FIELDS);
+ descriptors.add(MEDIA_FIELDS);
+ descriptors.add(POLL_FIELDS);
+ descriptors.add(PLACE_FIELDS);
+ descriptors.add(EXPANSIONS);
+ descriptors.add(BACKFILL_MINUTES);
+
+ this.descriptors = Collections.unmodifiableList(descriptors);
+
+ final Set<Relationship> relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ this.relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return this.descriptors;
+ }
+
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ messageQueue = new LinkedBlockingQueue<>(context.getProperty(QUEUE_SIZE).asInteger());
+
+ api = new TwitterStreamAPI(context, messageQueue, getLogger());
+ api.setBasePath(context.getProperty(BASE_PATH).getValue());
+ final String endpointName = context.getProperty(ENDPOINT).getValue();
+ if (ENDPOINT_SAMPLE.getValue().equals(endpointName)) {
+ api.start(TwitterStreamAPI.SAMPLE_ENDPOINT);
+ } else if (ENDPOINT_SEARCH.getValue().equals(endpointName)) {
+ api.start(TwitterStreamAPI.SEARCH_ENDPOINT);
+ } else {
+ throw new ProcessException("Endpoint was invalid value: " + endpointName);
+ }
+ }
+
+ @Override
+ public void onTrigger(ProcessContext context, ProcessSession session) throws ProcessException {
+ if (messageQueue.isEmpty()) {
+ context.yield();
+ return;
+ }
+
+ FlowFile flowFile = session.create();
+ flowFile = session.write(flowFile, new OutputStreamCallback() {
+ @Override
+ public void process(OutputStream out) throws IOException {
+ out.write("[".getBytes(StandardCharsets.UTF_8));
+ final int batchSize = context.getProperty(BATCH_SIZE).asInteger();
+ String tweet = messageQueue.poll();
+ out.write(tweet.getBytes(StandardCharsets.UTF_8));
+ int tweetCount = 1;
+ while (tweetCount < batchSize && !messageQueue.isEmpty()) {
+ out.write(",".getBytes(StandardCharsets.UTF_8));
Review Comment:
Making the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r848786178
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/test/java/org/apache/nifi/processors/twitter/TestConsumeTwitter.java:
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import okhttp3.mockwebserver.MockResponse;
+import okhttp3.mockwebserver.MockWebServer;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+
+public class TestConsumeTwitter {
+ private MockWebServer mockWebServer;
+
+ private TestRunner runner;
+
+ private final String SAMPLE_TWEET = "{\"data\":{\"id\":\"123\",\"text\":\"This is a sample tweet and is not real!\"}}";
+
+ @BeforeEach
+ public void setRunnerAndAPI() {
+ mockWebServer = new MockWebServer();
+
+ runner = TestRunners.newTestRunner(ConsumeTwitter.class);
+
+ runner.setProperty(ConsumeTwitter.BEARER_TOKEN, "BEARER_TOKEN");
+ final String basePath = mockWebServer.url("").toString();
+ runner.setProperty(ConsumeTwitter.BASE_PATH, basePath);
+ }
+
+ @AfterEach
+ public void shutdownServerAndAPI() throws IOException {
+ mockWebServer.shutdown();
+ }
+
+ @Test
+ public void testReceiveSingleTweetInStream() {
+ MockResponse response = new MockResponse()
+ .setResponseCode(200)
+ .setBody(SAMPLE_TWEET)
+ .addHeader("Content-Type", "application/json");
+ mockWebServer.enqueue(response);
+
+
+ runner.setProperty(ConsumeTwitter.ENDPOINT, ConsumeTwitter.ENDPOINT_SAMPLE);
+ runner.setProperty(ConsumeTwitter.QUEUE_SIZE, "10000");
+ runner.setProperty(ConsumeTwitter.BATCH_SIZE, "10");
+ runner.setProperty(ConsumeTwitter.BACKFILL_MINUTES, "0");
+
+ runner.assertValid();
+
+ // the TwitterStreamAPI class spins up another thread and might not be done queueing tweets in one run of the processor
+ final int maxTries = 100;
+ int tries = 0;
+ while (tries < maxTries && runner.getFlowFilesForRelationship(ConsumeTwitter.REL_SUCCESS).size() == 0) {
+ runner.run();
+ tries++;
+ }
+
+ // there should only be a single FlowFile containing a tweet
+ runner.assertTransferCount(ConsumeTwitter.REL_SUCCESS, 1);
+ MockFlowFile flowFile = runner.getFlowFilesForRelationship(ConsumeTwitter.REL_SUCCESS).get(0);
+ flowFile.assertContentEquals("[" + SAMPLE_TWEET + "]");
+ flowFile.assertAttributeEquals("mime.type", "application/json");
Review Comment:
Making the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
exceptionfactory commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r860870113
##########
nifi-assembly/NOTICE:
##########
@@ -964,6 +964,11 @@ The following binary components are provided under the Apache Software License v
The following NOTICE information applies:
Hosebird Client (hbc)
Copyright 2013 Twitter, Inc.
+
+ (ASLv2) Twitter API Client Library for Java
+ The following NOTICE information applies:
+ Twitter API Client Library for Java
+ Copyright 2022 Twitter, Inc.
Review Comment:
Thanks for making the other license adjustments. Since the Twitter Java SDK does not include a specific NOTICE file, this additional notice section is not necessary and can be removed.
```suggestion
```
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/pom.xml:
##########
@@ -69,5 +84,13 @@
<version>1.17.0-SNAPSHOT</version>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>com.squareup.okhttp3</groupId>
+ <artifactId>okhttp</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>com.squareup.okhttp3</groupId>
+ <artifactId>mockwebserver</artifactId>
Review Comment:
This dependency should be marked with the `test` scope.
```suggestion
<artifactId>mockwebserver</artifactId>
<scope>test</scope>
```
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-social-media-nar/src/main/resources/META-INF/NOTICE:
##########
@@ -44,3 +44,21 @@ The following binary components are provided under the Apache Software License v
The following NOTICE information applies:
Hosebird Client (hbc)
Copyright 2013 Twitter, Inc.
+
+ (ASLv2) Twitter API Client Library for Java
+ The following NOTICE information applies:
+ Twitter API Client Library for Java
+ Copyright 2022 Twitter, Inc.
Review Comment:
This section of the NOTICE file changes can be removed.
```suggestion
```
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.UUID;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
Review Comment:
Recommend adjusting the name and referencing the Twitter API documentation:
```suggestion
static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Stream",
"Sample Stream",
"Streams about one percent of all Tweets. https://developer.twitter.com/en/docs/twitter-api/tweets/volume-streams/api-reference/get-tweets-sample-stream");
```
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.UUID;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
Review Comment:
Recommend adjusting the name:
```suggestion
static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Stream",
"Search Stream",
"Stream Tweets that match configured stream filtering rules. " +
"If rules are not configured, then the stream will be empty. " +
"https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/get-tweets-search-stream");
```
```suggestion
static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
"Search Endpoint",
"The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
"If rules are not configured, then the stream will be empty");
```
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.UUID;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("consume-twitter-endpoint")
+ .displayName("Twitter Endpoint")
Review Comment:
Recommend the following naming adjustments:
```suggestion
.name("stream-endpoint")
.displayName("Stream Endpoint")
```
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.UUID;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("consume-twitter-endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
Review Comment:
```suggestion
.description("Specifies which endpoint from which Tweets should be streamed. " +
``
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.UUID;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("consume-twitter-endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("Maximum size of internal queue for streamed messages")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor BACKOFF_ATTEMPTS = new PropertyDescriptor.Builder()
+ .name("backoff-attempts")
+ .displayName("Backoff Attempts")
+ .description("The number of reconnection tries the processor will attempt in the event of " +
+ "a disconnection of the stream for any reason, before throwing an exception. To start a stream after " +
+ "this exception occur and the connection is fixed, please stop and restart the processor. If the value" +
+ "of this property is 0, then backoff will never occur and the processor will always need to be restarted" +
+ "if the stream fails.")
+ .required(true)
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .defaultValue("5")
+ .build();
+ public static final PropertyDescriptor BACKOFF_TIME = new PropertyDescriptor.Builder()
+ .name("backoff-time")
+ .displayName("Backoff Time")
+ .description("The duration to backoff to start attempting a new stream if" +
+ "the current one fails for any reason. Will increase by factor of 2 every time a restart fails")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("1 mins")
+ .build();
+ public static final PropertyDescriptor MAXIMUM_BACKOFF_TIME = new PropertyDescriptor.Builder()
+ .name("maximum-backoff-time")
+ .displayName("Maximum Backoff Time")
+ .description("The maximum duration to backoff to start attempting a new stream." +
+ "It is recommended that this number be much higher than the 'Backoff Time' property")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("5 mins")
+ .build();
+ public static final PropertyDescriptor CONNECT_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("connect-timeout")
+ .displayName("Connect Timeout")
+ .description("The maximum time in which client should establish a connection with the " +
+ "Twitter API before a time out. A value of 0 will mean no timeout at all")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("10 secs")
+ .build();
+ public static final PropertyDescriptor READ_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("read-timeout")
+ .displayName("Read Timeout")
+ .description("The maximum time of inactivity between receiving tweets from Twitter through " +
+ "the API before a timeout. A value of 0 will mean no timeout at all")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("10 secs")
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("tweet-fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values include: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("user-fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user " +
+ "for proper usage. Possible field values include: " +
+ "created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, " +
+ "protected, public_metrics, url, username, verified, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor MEDIA_FIELDS = new PropertyDescriptor.Builder()
+ .name("media-fields")
+ .displayName("Media Fields")
+ .description("A comma-separated list of media fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/media " +
+ "for proper usage. Possible field values include: " +
+ "alt_text, duration_ms, height, media_key, non_public_metrics, organic_metrics, preview_image_url, " +
+ "promoted_metrics, public_metrics, type, url, width")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor POLL_FIELDS = new PropertyDescriptor.Builder()
+ .name("poll-fields")
+ .displayName("Poll Fields")
+ .description("A comma-separated list of poll fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/poll " +
+ "for proper usage. Possible field values include: " +
+ "duration_minutes, end_datetime, id, options, voting_status")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PLACE_FIELDS = new PropertyDescriptor.Builder()
+ .name("place-fields")
+ .displayName("Place Fields")
+ .description("A comma-separated list of place fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/place " +
+ "for proper usage. Possible field values include: " +
+ "contained_within, country, country_code, full_name, geo, id, name, place_type")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor EXPANSIONS = new PropertyDescriptor.Builder()
+ .name("expansions")
+ .displayName("Expansions")
+ .description("A comma-separated list of expansions for objects in the returned tweet. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/expansions " +
+ "for proper usage. Possible field values include: " +
+ "author_id, referenced_tweets.id, referenced_tweets.id.author_id, entities.mentions.username, " +
+ "attachments.poll_ids, attachments.media_keys ,in_reply_to_user_id, geo.place_id")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor BACKFILL_MINUTES = new PropertyDescriptor.Builder()
+ .name("backfill-minutes")
+ .displayName("Backfill Minutes")
+ .description("The number of minutes (up to 5 minutes) of streaming data to be requested after a " +
+ "disconnect. Only available for project with academic research access. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/integrate/" +
+ "recovery-and-redundancy-features")
+ .required(true)
+ .defaultValue("0")
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("FlowFiles containing an array of one or more Tweets")
+ .build();
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+
+ private TweetStreamService tweetStreamService;
+
+ private volatile BlockingQueue<String> messageQueue;
+
+ @Override
+ protected void init(ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptors = new ArrayList<>();
+ descriptors.add(ENDPOINT);
+ descriptors.add(BASE_PATH);
+ descriptors.add(BEARER_TOKEN);
+ descriptors.add(QUEUE_SIZE);
+ descriptors.add(BATCH_SIZE);
+ descriptors.add(BACKOFF_ATTEMPTS);
+ descriptors.add(BACKOFF_TIME);
+ descriptors.add(MAXIMUM_BACKOFF_TIME);
+ descriptors.add(CONNECT_TIMEOUT);
+ descriptors.add(READ_TIMEOUT);
+ descriptors.add(TWEET_FIELDS);
+ descriptors.add(USER_FIELDS);
+ descriptors.add(MEDIA_FIELDS);
+ descriptors.add(POLL_FIELDS);
+ descriptors.add(PLACE_FIELDS);
+ descriptors.add(EXPANSIONS);
+ descriptors.add(BACKFILL_MINUTES);
+
+ this.descriptors = Collections.unmodifiableList(descriptors);
+
+ final Set<Relationship> relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ this.relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return this.descriptors;
+ }
+
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ messageQueue = new LinkedBlockingQueue<>(context.getProperty(QUEUE_SIZE).asInteger());
+
+ tweetStreamService = new TweetStreamService(context, messageQueue, getLogger());
+ tweetStreamService.setBasePath(context.getProperty(BASE_PATH).getValue());
+ final String endpointName = context.getProperty(ENDPOINT).getValue();
+ if (ENDPOINT_SAMPLE.getValue().equals(endpointName)) {
+ tweetStreamService.start(TweetStreamService.SAMPLE_ENDPOINT);
+ } else {
+ tweetStreamService.start(TweetStreamService.SEARCH_ENDPOINT);
+ }
+ }
+
+ @Override
+ public void onTrigger(ProcessContext context, ProcessSession session) throws ProcessException {
+ if (messageQueue.isEmpty()) {
+ context.yield();
+ return;
+ }
+
+ FlowFile flowFile = session.create();
+ flowFile = session.write(flowFile, new OutputStreamCallback() {
+ @Override
+ public void process(OutputStream out) throws IOException {
+ out.write('[');
+ final int batchSize = context.getProperty(BATCH_SIZE).asInteger();
+ String tweet = messageQueue.poll();
+ out.write(tweet.getBytes(StandardCharsets.UTF_8));
+ int tweetCount = 1;
Review Comment:
Recommend adjusting the ordering of these lines to poll before writing:
```suggestion
final int batchSize = context.getProperty(BATCH_SIZE).asInteger();
String tweet = messageQueue.poll();
int tweetCount = 1;
out.write('[');
out.write(tweet.getBytes(StandardCharsets.UTF_8));
```
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TweetStreamService.java:
##########
@@ -0,0 +1,242 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiClient;
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TweetStreamService {
+ public static final String SEARCH_ENDPOINT = "Search Endpoint";
+ public static final String SAMPLE_ENDPOINT = "Sample Endpoint";
+ private static final String SAMPLE_PATH = "/2/tweets/sample/stream";
+ private static final String SEARCH_PATH = "/2/tweets/search/stream";
Review Comment:
Add a `StreamEndpoint` enum would also allow defining a `path` property where these values could be associated.
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TweetStreamService.java:
##########
@@ -0,0 +1,242 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiClient;
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TweetStreamService {
+ public static final String SEARCH_ENDPOINT = "Search Endpoint";
+ public static final String SAMPLE_ENDPOINT = "Sample Endpoint";
Review Comment:
Rather than redefining accepted values here and in the Allowable Values, recommend creating an `enum` named `StreamEndpoint` with `SEARCH` and `SAMPLE` values. That way the enum can be used for both the property descriptor and the argument to this service.
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TweetStreamService.java:
##########
@@ -0,0 +1,242 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiClient;
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TweetStreamService {
+ public static final String SEARCH_ENDPOINT = "Search Endpoint";
+ public static final String SAMPLE_ENDPOINT = "Sample Endpoint";
+ private static final String SAMPLE_PATH = "/2/tweets/sample/stream";
+ private static final String SEARCH_PATH = "/2/tweets/search/stream";
+
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ScheduledExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Long backoffMultiplier;
+ private int backoffAttempts;
+ private int attemptCounter;
+ private Long backoffTime;
+ private Long maximumBackoff;
+
+ private String endpoint;
+
+ private Set<String> parseCommaSeparatedListPropreties(final ProcessContext context, final PropertyDescriptor property) {
+ Set<String> fields = null;
+ if (context.getProperty(property).isSet()) {
+ fields = new HashSet<>();
+ final String fieldsString = context.getProperty(property).getValue();
+ for (final String field: fieldsString.split(",")) {
+ fields.add(field.trim());
+ }
+ }
+ return fields;
+ }
+
+ public TweetStreamService(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ assert context != null;
+ assert queue != null;
+ assert logger != null;
Review Comment:
The `assert` keyword should not be used. Instead `Objects.requireNonNull()` should work.
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.UUID;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("consume-twitter-endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("Maximum size of internal queue for streamed messages")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor BACKOFF_ATTEMPTS = new PropertyDescriptor.Builder()
+ .name("backoff-attempts")
+ .displayName("Backoff Attempts")
+ .description("The number of reconnection tries the processor will attempt in the event of " +
+ "a disconnection of the stream for any reason, before throwing an exception. To start a stream after " +
+ "this exception occur and the connection is fixed, please stop and restart the processor. If the value" +
+ "of this property is 0, then backoff will never occur and the processor will always need to be restarted" +
+ "if the stream fails.")
+ .required(true)
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .defaultValue("5")
+ .build();
+ public static final PropertyDescriptor BACKOFF_TIME = new PropertyDescriptor.Builder()
+ .name("backoff-time")
+ .displayName("Backoff Time")
+ .description("The duration to backoff to start attempting a new stream if" +
+ "the current one fails for any reason. Will increase by factor of 2 every time a restart fails")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("1 mins")
+ .build();
+ public static final PropertyDescriptor MAXIMUM_BACKOFF_TIME = new PropertyDescriptor.Builder()
+ .name("maximum-backoff-time")
+ .displayName("Maximum Backoff Time")
+ .description("The maximum duration to backoff to start attempting a new stream." +
+ "It is recommended that this number be much higher than the 'Backoff Time' property")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("5 mins")
+ .build();
+ public static final PropertyDescriptor CONNECT_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("connect-timeout")
+ .displayName("Connect Timeout")
+ .description("The maximum time in which client should establish a connection with the " +
+ "Twitter API before a time out. A value of 0 will mean no timeout at all")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("10 secs")
+ .build();
+ public static final PropertyDescriptor READ_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("read-timeout")
+ .displayName("Read Timeout")
+ .description("The maximum time of inactivity between receiving tweets from Twitter through " +
+ "the API before a timeout. A value of 0 will mean no timeout at all")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("10 secs")
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("tweet-fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values include: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("user-fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user " +
+ "for proper usage. Possible field values include: " +
+ "created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, " +
+ "protected, public_metrics, url, username, verified, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor MEDIA_FIELDS = new PropertyDescriptor.Builder()
+ .name("media-fields")
+ .displayName("Media Fields")
+ .description("A comma-separated list of media fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/media " +
+ "for proper usage. Possible field values include: " +
+ "alt_text, duration_ms, height, media_key, non_public_metrics, organic_metrics, preview_image_url, " +
+ "promoted_metrics, public_metrics, type, url, width")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor POLL_FIELDS = new PropertyDescriptor.Builder()
+ .name("poll-fields")
+ .displayName("Poll Fields")
+ .description("A comma-separated list of poll fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/poll " +
+ "for proper usage. Possible field values include: " +
+ "duration_minutes, end_datetime, id, options, voting_status")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PLACE_FIELDS = new PropertyDescriptor.Builder()
+ .name("place-fields")
+ .displayName("Place Fields")
+ .description("A comma-separated list of place fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/place " +
+ "for proper usage. Possible field values include: " +
+ "contained_within, country, country_code, full_name, geo, id, name, place_type")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor EXPANSIONS = new PropertyDescriptor.Builder()
+ .name("expansions")
+ .displayName("Expansions")
+ .description("A comma-separated list of expansions for objects in the returned tweet. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/expansions " +
+ "for proper usage. Possible field values include: " +
+ "author_id, referenced_tweets.id, referenced_tweets.id.author_id, entities.mentions.username, " +
+ "attachments.poll_ids, attachments.media_keys ,in_reply_to_user_id, geo.place_id")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor BACKFILL_MINUTES = new PropertyDescriptor.Builder()
+ .name("backfill-minutes")
+ .displayName("Backfill Minutes")
+ .description("The number of minutes (up to 5 minutes) of streaming data to be requested after a " +
+ "disconnect. Only available for project with academic research access. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/integrate/" +
+ "recovery-and-redundancy-features")
+ .required(true)
+ .defaultValue("0")
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
Review Comment:
As a required property, recommend moving this after the Read Timeout property.
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TweetStreamService.java:
##########
@@ -0,0 +1,242 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiClient;
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TweetStreamService {
+ public static final String SEARCH_ENDPOINT = "Search Endpoint";
+ public static final String SAMPLE_ENDPOINT = "Sample Endpoint";
+ private static final String SAMPLE_PATH = "/2/tweets/sample/stream";
+ private static final String SEARCH_PATH = "/2/tweets/search/stream";
+
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ScheduledExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Long backoffMultiplier;
+ private int backoffAttempts;
+ private int attemptCounter;
+ private Long backoffTime;
+ private Long maximumBackoff;
+
+ private String endpoint;
+
+ private Set<String> parseCommaSeparatedListPropreties(final ProcessContext context, final PropertyDescriptor property) {
+ Set<String> fields = null;
+ if (context.getProperty(property).isSet()) {
+ fields = new HashSet<>();
+ final String fieldsString = context.getProperty(property).getValue();
+ for (final String field: fieldsString.split(",")) {
+ fields.add(field.trim());
+ }
+ }
+ return fields;
+ }
+
+ public TweetStreamService(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ assert context != null;
+ assert queue != null;
+ assert logger != null;
+
+ this.queue = queue;
+ this.logger = logger;
+
+ this.tweetFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.TWEET_FIELDS);
+ this.userFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.USER_FIELDS);
+ this.mediaFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.MEDIA_FIELDS);
+ this.pollFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.POLL_FIELDS);
+ this.placeFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.PLACE_FIELDS);
+ this.expansions = parseCommaSeparatedListPropreties(context, ConsumeTwitter.EXPANSIONS);
+ this.backfillMinutes = context.getProperty(ConsumeTwitter.BACKFILL_MINUTES).asInteger();
+
+ this.backoffMultiplier = 1L;
+ this.backoffAttempts = context.getProperty(ConsumeTwitter.BACKOFF_ATTEMPTS).asInteger();
+ this.attemptCounter = 0;
+ this.backoffTime = context.getProperty(ConsumeTwitter.BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+ this.maximumBackoff = context.getProperty(ConsumeTwitter.MAXIMUM_BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+
+ ApiClient client = new ApiClient();
+ final int connectTimeout = context.getProperty(ConsumeTwitter.CONNECT_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ final int readTimeout = context.getProperty(ConsumeTwitter.READ_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ client = client.setConnectTimeout(connectTimeout);
+ client = client.setReadTimeout(readTimeout);
+ api = new TwitterApi(client);
+
+ TwitterCredentialsBearer creds = new TwitterCredentialsBearer(context.getProperty(ConsumeTwitter.BEARER_TOKEN).getValue());
+ api.setTwitterCredentials(creds);
+
+ this.executorService = Executors.newSingleThreadScheduledExecutor();
+ }
+
+ public String getBasePath() {
+ return api.getApiClient().getBasePath();
+ }
+
+ public void setBasePath(final String path) {
+ api.getApiClient().setBasePath(path);
+ }
+
+ public String getTransitUri(final String endpoint) {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ return api.getApiClient().getBasePath() + SAMPLE_PATH;
+ } else if (endpoint.equals(SEARCH_ENDPOINT)) {
+ return api.getApiClient().getBasePath() + SEARCH_PATH;
+ } else {
+ logger.warn("Unrecognized endpoint in getTransitUri. Returning basePath");
+ return api.getApiClient().getBasePath();
+ }
+ }
+
+ /**
+ * This method would be called when we would like the stream to get started. This method will spin off a thread that
+ * will continue to queue tweets on to the given queue passed in the constructor. The thread will continue
+ * to run until {@code stop} is called.
+ * @param endpoint {@code TwitterStreamAPI.SAMPLE_ENDPOINT} or {@code TwitterStreamAPI.SEARCH_ENDPOINT}
+ */
+ public void start(final String endpoint) {
Review Comment:
This method could be changed to accept the `StreamEndpoint` enum.
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TweetStreamService.java:
##########
@@ -0,0 +1,242 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiClient;
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TweetStreamService {
+ public static final String SEARCH_ENDPOINT = "Search Endpoint";
+ public static final String SAMPLE_ENDPOINT = "Sample Endpoint";
+ private static final String SAMPLE_PATH = "/2/tweets/sample/stream";
+ private static final String SEARCH_PATH = "/2/tweets/search/stream";
+
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ScheduledExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Long backoffMultiplier;
+ private int backoffAttempts;
+ private int attemptCounter;
+ private Long backoffTime;
+ private Long maximumBackoff;
+
+ private String endpoint;
+
+ private Set<String> parseCommaSeparatedListPropreties(final ProcessContext context, final PropertyDescriptor property) {
+ Set<String> fields = null;
+ if (context.getProperty(property).isSet()) {
+ fields = new HashSet<>();
+ final String fieldsString = context.getProperty(property).getValue();
+ for (final String field: fieldsString.split(",")) {
+ fields.add(field.trim());
+ }
+ }
+ return fields;
+ }
Review Comment:
Private methods should be organized after public methods and constructors, recommend moving this to the end of the class.
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TweetStreamService.java:
##########
@@ -0,0 +1,242 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiClient;
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TweetStreamService {
+ public static final String SEARCH_ENDPOINT = "Search Endpoint";
+ public static final String SAMPLE_ENDPOINT = "Sample Endpoint";
+ private static final String SAMPLE_PATH = "/2/tweets/sample/stream";
+ private static final String SEARCH_PATH = "/2/tweets/search/stream";
+
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ScheduledExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Long backoffMultiplier;
+ private int backoffAttempts;
+ private int attemptCounter;
+ private Long backoffTime;
+ private Long maximumBackoff;
+
+ private String endpoint;
+
+ private Set<String> parseCommaSeparatedListPropreties(final ProcessContext context, final PropertyDescriptor property) {
Review Comment:
```suggestion
private Set<String> parseCommaSeparatedListPropreties(final ProcessContext context, final PropertyDescriptor property) {
```
```suggestion
private Set<String> parseCommaSeparatedProperties(final ProcessContext context, final PropertyDescriptor property) {
```
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TweetStreamService.java:
##########
@@ -0,0 +1,242 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiClient;
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TweetStreamService {
+ public static final String SEARCH_ENDPOINT = "Search Endpoint";
+ public static final String SAMPLE_ENDPOINT = "Sample Endpoint";
+ private static final String SAMPLE_PATH = "/2/tweets/sample/stream";
+ private static final String SEARCH_PATH = "/2/tweets/search/stream";
+
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ScheduledExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Long backoffMultiplier;
+ private int backoffAttempts;
+ private int attemptCounter;
+ private Long backoffTime;
+ private Long maximumBackoff;
+
+ private String endpoint;
+
+ private Set<String> parseCommaSeparatedListPropreties(final ProcessContext context, final PropertyDescriptor property) {
+ Set<String> fields = null;
+ if (context.getProperty(property).isSet()) {
+ fields = new HashSet<>();
+ final String fieldsString = context.getProperty(property).getValue();
+ for (final String field: fieldsString.split(",")) {
+ fields.add(field.trim());
+ }
+ }
+ return fields;
+ }
+
+ public TweetStreamService(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ assert context != null;
+ assert queue != null;
+ assert logger != null;
+
+ this.queue = queue;
+ this.logger = logger;
+
+ this.tweetFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.TWEET_FIELDS);
+ this.userFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.USER_FIELDS);
+ this.mediaFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.MEDIA_FIELDS);
+ this.pollFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.POLL_FIELDS);
+ this.placeFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.PLACE_FIELDS);
+ this.expansions = parseCommaSeparatedListPropreties(context, ConsumeTwitter.EXPANSIONS);
+ this.backfillMinutes = context.getProperty(ConsumeTwitter.BACKFILL_MINUTES).asInteger();
+
+ this.backoffMultiplier = 1L;
+ this.backoffAttempts = context.getProperty(ConsumeTwitter.BACKOFF_ATTEMPTS).asInteger();
+ this.attemptCounter = 0;
+ this.backoffTime = context.getProperty(ConsumeTwitter.BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+ this.maximumBackoff = context.getProperty(ConsumeTwitter.MAXIMUM_BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+
+ ApiClient client = new ApiClient();
+ final int connectTimeout = context.getProperty(ConsumeTwitter.CONNECT_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ final int readTimeout = context.getProperty(ConsumeTwitter.READ_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ client = client.setConnectTimeout(connectTimeout);
+ client = client.setReadTimeout(readTimeout);
+ api = new TwitterApi(client);
+
+ TwitterCredentialsBearer creds = new TwitterCredentialsBearer(context.getProperty(ConsumeTwitter.BEARER_TOKEN).getValue());
+ api.setTwitterCredentials(creds);
+
+ this.executorService = Executors.newSingleThreadScheduledExecutor();
+ }
+
+ public String getBasePath() {
+ return api.getApiClient().getBasePath();
+ }
+
+ public void setBasePath(final String path) {
+ api.getApiClient().setBasePath(path);
+ }
+
+ public String getTransitUri(final String endpoint) {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ return api.getApiClient().getBasePath() + SAMPLE_PATH;
+ } else if (endpoint.equals(SEARCH_ENDPOINT)) {
+ return api.getApiClient().getBasePath() + SEARCH_PATH;
+ } else {
+ logger.warn("Unrecognized endpoint in getTransitUri. Returning basePath");
+ return api.getApiClient().getBasePath();
+ }
+ }
+
+ /**
+ * This method would be called when we would like the stream to get started. This method will spin off a thread that
+ * will continue to queue tweets on to the given queue passed in the constructor. The thread will continue
+ * to run until {@code stop} is called.
+ * @param endpoint {@code TwitterStreamAPI.SAMPLE_ENDPOINT} or {@code TwitterStreamAPI.SEARCH_ENDPOINT}
+ */
+ public void start(final String endpoint) {
+ this.endpoint = endpoint;
+ executorService.execute(new TweetStreamStarter());
+ }
+
+ /**
+ * This method would be called when we would like the stream to get stopped. The stream will be closed and the
+ * executorService will be shut down.
+ */
+ public void stop() {
+ if (stream != null) {
+ try {
+ stream.close();
+ } catch (IOException e) {
+ logger.error("Closing response stream failed", e);
+ }
+ }
+
+ executorService.shutdownNow();
+ }
+
+ private Long calculateBackoffDelay() {
+ long backoff = backoffMultiplier * backoffTime;
+ long delay = (backoff > maximumBackoff) ? maximumBackoff : backoff;
+ return delay;
+ }
+
+ private void scheduleStartStreamWithBackoff() {
+ // use exponential(by factor of 2) backoff in scheduling the next TweetStreamStarter
+ if (attemptCounter >= backoffAttempts) {
+ final String msg = "Reached maximum attempts to backoff";
+ logger.error(msg);
+ throw new ProcessException(msg);
Review Comment:
Logging the error is not necessary if also throwing the exception.
```suggestion
final String msg = "Reached maximum attempts to backoff";
logger.error(msg);
throw new ProcessException(msg);
```
```suggestion
throw new ProcessException(String.format("Connection failed after maximum attempts [%d]", attemptCounter));
```
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.UUID;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("consume-twitter-endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("Maximum size of internal queue for streamed messages")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor BACKOFF_ATTEMPTS = new PropertyDescriptor.Builder()
+ .name("backoff-attempts")
+ .displayName("Backoff Attempts")
+ .description("The number of reconnection tries the processor will attempt in the event of " +
+ "a disconnection of the stream for any reason, before throwing an exception. To start a stream after " +
+ "this exception occur and the connection is fixed, please stop and restart the processor. If the value" +
+ "of this property is 0, then backoff will never occur and the processor will always need to be restarted" +
+ "if the stream fails.")
+ .required(true)
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .defaultValue("5")
+ .build();
+ public static final PropertyDescriptor BACKOFF_TIME = new PropertyDescriptor.Builder()
+ .name("backoff-time")
+ .displayName("Backoff Time")
+ .description("The duration to backoff to start attempting a new stream if" +
+ "the current one fails for any reason. Will increase by factor of 2 every time a restart fails")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("1 mins")
+ .build();
+ public static final PropertyDescriptor MAXIMUM_BACKOFF_TIME = new PropertyDescriptor.Builder()
+ .name("maximum-backoff-time")
+ .displayName("Maximum Backoff Time")
+ .description("The maximum duration to backoff to start attempting a new stream." +
+ "It is recommended that this number be much higher than the 'Backoff Time' property")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("5 mins")
+ .build();
+ public static final PropertyDescriptor CONNECT_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("connect-timeout")
+ .displayName("Connect Timeout")
+ .description("The maximum time in which client should establish a connection with the " +
+ "Twitter API before a time out. A value of 0 will mean no timeout at all")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("10 secs")
+ .build();
+ public static final PropertyDescriptor READ_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("read-timeout")
+ .displayName("Read Timeout")
+ .description("The maximum time of inactivity between receiving tweets from Twitter through " +
+ "the API before a timeout. A value of 0 will mean no timeout at all")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("10 secs")
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("tweet-fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values include: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("user-fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user " +
+ "for proper usage. Possible field values include: " +
+ "created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, " +
+ "protected, public_metrics, url, username, verified, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor MEDIA_FIELDS = new PropertyDescriptor.Builder()
+ .name("media-fields")
+ .displayName("Media Fields")
+ .description("A comma-separated list of media fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/media " +
+ "for proper usage. Possible field values include: " +
+ "alt_text, duration_ms, height, media_key, non_public_metrics, organic_metrics, preview_image_url, " +
+ "promoted_metrics, public_metrics, type, url, width")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor POLL_FIELDS = new PropertyDescriptor.Builder()
+ .name("poll-fields")
+ .displayName("Poll Fields")
+ .description("A comma-separated list of poll fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/poll " +
+ "for proper usage. Possible field values include: " +
+ "duration_minutes, end_datetime, id, options, voting_status")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PLACE_FIELDS = new PropertyDescriptor.Builder()
+ .name("place-fields")
+ .displayName("Place Fields")
+ .description("A comma-separated list of place fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/place " +
+ "for proper usage. Possible field values include: " +
+ "contained_within, country, country_code, full_name, geo, id, name, place_type")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor EXPANSIONS = new PropertyDescriptor.Builder()
+ .name("expansions")
+ .displayName("Expansions")
+ .description("A comma-separated list of expansions for objects in the returned tweet. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/expansions " +
+ "for proper usage. Possible field values include: " +
+ "author_id, referenced_tweets.id, referenced_tweets.id.author_id, entities.mentions.username, " +
+ "attachments.poll_ids, attachments.media_keys ,in_reply_to_user_id, geo.place_id")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor BACKFILL_MINUTES = new PropertyDescriptor.Builder()
+ .name("backfill-minutes")
+ .displayName("Backfill Minutes")
+ .description("The number of minutes (up to 5 minutes) of streaming data to be requested after a " +
+ "disconnect. Only available for project with academic research access. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/integrate/" +
+ "recovery-and-redundancy-features")
+ .required(true)
+ .defaultValue("0")
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("FlowFiles containing an array of one or more Tweets")
+ .build();
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+
+ private TweetStreamService tweetStreamService;
+
+ private volatile BlockingQueue<String> messageQueue;
+
+ @Override
+ protected void init(ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptors = new ArrayList<>();
+ descriptors.add(ENDPOINT);
+ descriptors.add(BASE_PATH);
+ descriptors.add(BEARER_TOKEN);
+ descriptors.add(QUEUE_SIZE);
+ descriptors.add(BATCH_SIZE);
+ descriptors.add(BACKOFF_ATTEMPTS);
+ descriptors.add(BACKOFF_TIME);
+ descriptors.add(MAXIMUM_BACKOFF_TIME);
+ descriptors.add(CONNECT_TIMEOUT);
+ descriptors.add(READ_TIMEOUT);
+ descriptors.add(TWEET_FIELDS);
+ descriptors.add(USER_FIELDS);
+ descriptors.add(MEDIA_FIELDS);
+ descriptors.add(POLL_FIELDS);
+ descriptors.add(PLACE_FIELDS);
+ descriptors.add(EXPANSIONS);
+ descriptors.add(BACKFILL_MINUTES);
Review Comment:
Recommend adjusting the order to place Backfill Minutes after the last required property, keeping the optional fields after all required properties.
```suggestion
descriptors.add(ENDPOINT);
descriptors.add(BASE_PATH);
descriptors.add(BEARER_TOKEN);
descriptors.add(QUEUE_SIZE);
descriptors.add(BATCH_SIZE);
descriptors.add(BACKOFF_ATTEMPTS);
descriptors.add(BACKOFF_TIME);
descriptors.add(MAXIMUM_BACKOFF_TIME);
descriptors.add(CONNECT_TIMEOUT);
descriptors.add(READ_TIMEOUT);
descriptors.add(BACKFILL_MINUTES);
descriptors.add(TWEET_FIELDS);
descriptors.add(USER_FIELDS);
descriptors.add(MEDIA_FIELDS);
descriptors.add(POLL_FIELDS);
descriptors.add(PLACE_FIELDS);
descriptors.add(EXPANSIONS);
```
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TweetStreamService.java:
##########
@@ -0,0 +1,242 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiClient;
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TweetStreamService {
+ public static final String SEARCH_ENDPOINT = "Search Endpoint";
+ public static final String SAMPLE_ENDPOINT = "Sample Endpoint";
+ private static final String SAMPLE_PATH = "/2/tweets/sample/stream";
+ private static final String SEARCH_PATH = "/2/tweets/search/stream";
+
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ScheduledExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Long backoffMultiplier;
+ private int backoffAttempts;
+ private int attemptCounter;
+ private Long backoffTime;
+ private Long maximumBackoff;
+
+ private String endpoint;
+
+ private Set<String> parseCommaSeparatedListPropreties(final ProcessContext context, final PropertyDescriptor property) {
+ Set<String> fields = null;
+ if (context.getProperty(property).isSet()) {
+ fields = new HashSet<>();
+ final String fieldsString = context.getProperty(property).getValue();
+ for (final String field: fieldsString.split(",")) {
+ fields.add(field.trim());
+ }
+ }
+ return fields;
+ }
+
+ public TweetStreamService(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ assert context != null;
+ assert queue != null;
+ assert logger != null;
+
+ this.queue = queue;
+ this.logger = logger;
+
+ this.tweetFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.TWEET_FIELDS);
+ this.userFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.USER_FIELDS);
+ this.mediaFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.MEDIA_FIELDS);
+ this.pollFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.POLL_FIELDS);
+ this.placeFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.PLACE_FIELDS);
+ this.expansions = parseCommaSeparatedListPropreties(context, ConsumeTwitter.EXPANSIONS);
+ this.backfillMinutes = context.getProperty(ConsumeTwitter.BACKFILL_MINUTES).asInteger();
+
+ this.backoffMultiplier = 1L;
+ this.backoffAttempts = context.getProperty(ConsumeTwitter.BACKOFF_ATTEMPTS).asInteger();
+ this.attemptCounter = 0;
+ this.backoffTime = context.getProperty(ConsumeTwitter.BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+ this.maximumBackoff = context.getProperty(ConsumeTwitter.MAXIMUM_BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+
+ ApiClient client = new ApiClient();
+ final int connectTimeout = context.getProperty(ConsumeTwitter.CONNECT_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ final int readTimeout = context.getProperty(ConsumeTwitter.READ_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ client = client.setConnectTimeout(connectTimeout);
+ client = client.setReadTimeout(readTimeout);
+ api = new TwitterApi(client);
+
+ TwitterCredentialsBearer creds = new TwitterCredentialsBearer(context.getProperty(ConsumeTwitter.BEARER_TOKEN).getValue());
+ api.setTwitterCredentials(creds);
+
+ this.executorService = Executors.newSingleThreadScheduledExecutor();
+ }
+
+ public String getBasePath() {
+ return api.getApiClient().getBasePath();
+ }
+
+ public void setBasePath(final String path) {
+ api.getApiClient().setBasePath(path);
+ }
+
+ public String getTransitUri(final String endpoint) {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ return api.getApiClient().getBasePath() + SAMPLE_PATH;
+ } else if (endpoint.equals(SEARCH_ENDPOINT)) {
+ return api.getApiClient().getBasePath() + SEARCH_PATH;
+ } else {
+ logger.warn("Unrecognized endpoint in getTransitUri. Returning basePath");
+ return api.getApiClient().getBasePath();
+ }
+ }
+
+ /**
+ * This method would be called when we would like the stream to get started. This method will spin off a thread that
+ * will continue to queue tweets on to the given queue passed in the constructor. The thread will continue
+ * to run until {@code stop} is called.
+ * @param endpoint {@code TwitterStreamAPI.SAMPLE_ENDPOINT} or {@code TwitterStreamAPI.SEARCH_ENDPOINT}
+ */
+ public void start(final String endpoint) {
+ this.endpoint = endpoint;
+ executorService.execute(new TweetStreamStarter());
+ }
+
+ /**
+ * This method would be called when we would like the stream to get stopped. The stream will be closed and the
+ * executorService will be shut down.
+ */
+ public void stop() {
+ if (stream != null) {
+ try {
+ stream.close();
+ } catch (IOException e) {
+ logger.error("Closing response stream failed", e);
+ }
+ }
+
+ executorService.shutdownNow();
+ }
+
+ private Long calculateBackoffDelay() {
+ long backoff = backoffMultiplier * backoffTime;
+ long delay = (backoff > maximumBackoff) ? maximumBackoff : backoff;
+ return delay;
+ }
+
+ private void scheduleStartStreamWithBackoff() {
+ // use exponential(by factor of 2) backoff in scheduling the next TweetStreamStarter
+ if (attemptCounter >= backoffAttempts) {
+ final String msg = "Reached maximum attempts to backoff";
+ logger.error(msg);
+ throw new ProcessException(msg);
+ }
+ attemptCounter += 1;
+ long delay = calculateBackoffDelay();
+ backoffMultiplier *= 2;
+ logger.info("Scheduling new TweetStreamStarter after {}s delay", delay);
+ executorService.schedule(new TweetStreamStarter(), delay, TimeUnit.SECONDS);
+ }
+
+ private void resetBackoff() {
+ attemptCounter = 0;
+ backoffMultiplier = 1L;
+ }
+
+ private class TweetStreamStarter implements Runnable {
+ @Override
+ public void run() {
+ try {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ stream = api.tweets().sampleStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else {
+ stream = api.tweets().searchStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ }
+ } catch (ApiException e) {
+ stream = null;
+ logger.warn(String.format("Received ApiException, error %d: %s", e.getCode(), e.getMessage()));
+ scheduleStartStreamWithBackoff();
+ return;
+ } catch (Exception e) {
+ stream = null;
+ logger.warn(String.format("Received Exception %s: %s", e.getClass(), e.getMessage()));
+ scheduleStartStreamWithBackoff();
+ return;
+ }
+
+ if (stream == null) {
+ logger.warn("Stream is null, could not make a connection to the Twitter API");
+ scheduleStartStreamWithBackoff();
+ return;
Review Comment:
Is this condition possible? It seems like it should not happen from the above code.
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TweetStreamService.java:
##########
@@ -0,0 +1,242 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiClient;
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TweetStreamService {
+ public static final String SEARCH_ENDPOINT = "Search Endpoint";
+ public static final String SAMPLE_ENDPOINT = "Sample Endpoint";
+ private static final String SAMPLE_PATH = "/2/tweets/sample/stream";
+ private static final String SEARCH_PATH = "/2/tweets/search/stream";
+
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ScheduledExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Long backoffMultiplier;
+ private int backoffAttempts;
+ private int attemptCounter;
+ private Long backoffTime;
+ private Long maximumBackoff;
+
+ private String endpoint;
+
+ private Set<String> parseCommaSeparatedListPropreties(final ProcessContext context, final PropertyDescriptor property) {
+ Set<String> fields = null;
+ if (context.getProperty(property).isSet()) {
+ fields = new HashSet<>();
+ final String fieldsString = context.getProperty(property).getValue();
+ for (final String field: fieldsString.split(",")) {
+ fields.add(field.trim());
+ }
+ }
+ return fields;
+ }
+
+ public TweetStreamService(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ assert context != null;
+ assert queue != null;
+ assert logger != null;
+
+ this.queue = queue;
+ this.logger = logger;
+
+ this.tweetFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.TWEET_FIELDS);
+ this.userFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.USER_FIELDS);
+ this.mediaFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.MEDIA_FIELDS);
+ this.pollFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.POLL_FIELDS);
+ this.placeFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.PLACE_FIELDS);
+ this.expansions = parseCommaSeparatedListPropreties(context, ConsumeTwitter.EXPANSIONS);
+ this.backfillMinutes = context.getProperty(ConsumeTwitter.BACKFILL_MINUTES).asInteger();
+
+ this.backoffMultiplier = 1L;
+ this.backoffAttempts = context.getProperty(ConsumeTwitter.BACKOFF_ATTEMPTS).asInteger();
+ this.attemptCounter = 0;
+ this.backoffTime = context.getProperty(ConsumeTwitter.BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+ this.maximumBackoff = context.getProperty(ConsumeTwitter.MAXIMUM_BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+
+ ApiClient client = new ApiClient();
+ final int connectTimeout = context.getProperty(ConsumeTwitter.CONNECT_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ final int readTimeout = context.getProperty(ConsumeTwitter.READ_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ client = client.setConnectTimeout(connectTimeout);
+ client = client.setReadTimeout(readTimeout);
+ api = new TwitterApi(client);
+
+ TwitterCredentialsBearer creds = new TwitterCredentialsBearer(context.getProperty(ConsumeTwitter.BEARER_TOKEN).getValue());
+ api.setTwitterCredentials(creds);
+
+ this.executorService = Executors.newSingleThreadScheduledExecutor();
+ }
+
+ public String getBasePath() {
+ return api.getApiClient().getBasePath();
+ }
+
+ public void setBasePath(final String path) {
+ api.getApiClient().setBasePath(path);
+ }
+
+ public String getTransitUri(final String endpoint) {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ return api.getApiClient().getBasePath() + SAMPLE_PATH;
+ } else if (endpoint.equals(SEARCH_ENDPOINT)) {
+ return api.getApiClient().getBasePath() + SEARCH_PATH;
+ } else {
+ logger.warn("Unrecognized endpoint in getTransitUri. Returning basePath");
+ return api.getApiClient().getBasePath();
+ }
+ }
+
+ /**
+ * This method would be called when we would like the stream to get started. This method will spin off a thread that
+ * will continue to queue tweets on to the given queue passed in the constructor. The thread will continue
+ * to run until {@code stop} is called.
+ * @param endpoint {@code TwitterStreamAPI.SAMPLE_ENDPOINT} or {@code TwitterStreamAPI.SEARCH_ENDPOINT}
+ */
+ public void start(final String endpoint) {
+ this.endpoint = endpoint;
+ executorService.execute(new TweetStreamStarter());
+ }
+
+ /**
+ * This method would be called when we would like the stream to get stopped. The stream will be closed and the
+ * executorService will be shut down.
+ */
+ public void stop() {
+ if (stream != null) {
+ try {
+ stream.close();
+ } catch (IOException e) {
+ logger.error("Closing response stream failed", e);
+ }
+ }
+
+ executorService.shutdownNow();
+ }
+
+ private Long calculateBackoffDelay() {
+ long backoff = backoffMultiplier * backoffTime;
+ long delay = (backoff > maximumBackoff) ? maximumBackoff : backoff;
+ return delay;
+ }
+
+ private void scheduleStartStreamWithBackoff() {
+ // use exponential(by factor of 2) backoff in scheduling the next TweetStreamStarter
+ if (attemptCounter >= backoffAttempts) {
+ final String msg = "Reached maximum attempts to backoff";
+ logger.error(msg);
+ throw new ProcessException(msg);
+ }
+ attemptCounter += 1;
+ long delay = calculateBackoffDelay();
+ backoffMultiplier *= 2;
+ logger.info("Scheduling new TweetStreamStarter after {}s delay", delay);
+ executorService.schedule(new TweetStreamStarter(), delay, TimeUnit.SECONDS);
+ }
+
+ private void resetBackoff() {
+ attemptCounter = 0;
+ backoffMultiplier = 1L;
+ }
+
+ private class TweetStreamStarter implements Runnable {
+ @Override
+ public void run() {
+ try {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ stream = api.tweets().sampleStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else {
+ stream = api.tweets().searchStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ }
+ } catch (ApiException e) {
+ stream = null;
+ logger.warn(String.format("Received ApiException, error %d: %s", e.getCode(), e.getMessage()));
Review Comment:
The exception itself should be included to provide a full stack trace in the logs:
```suggestion
logger.warn("Twitter Stream [{}] API connection failed: HTTP {}", endpoint, e.getCode(), e);
```
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TweetStreamService.java:
##########
@@ -0,0 +1,242 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiClient;
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TweetStreamService {
+ public static final String SEARCH_ENDPOINT = "Search Endpoint";
+ public static final String SAMPLE_ENDPOINT = "Sample Endpoint";
+ private static final String SAMPLE_PATH = "/2/tweets/sample/stream";
+ private static final String SEARCH_PATH = "/2/tweets/search/stream";
+
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ScheduledExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Long backoffMultiplier;
+ private int backoffAttempts;
+ private int attemptCounter;
+ private Long backoffTime;
+ private Long maximumBackoff;
+
+ private String endpoint;
+
+ private Set<String> parseCommaSeparatedListPropreties(final ProcessContext context, final PropertyDescriptor property) {
+ Set<String> fields = null;
+ if (context.getProperty(property).isSet()) {
+ fields = new HashSet<>();
+ final String fieldsString = context.getProperty(property).getValue();
+ for (final String field: fieldsString.split(",")) {
+ fields.add(field.trim());
+ }
+ }
+ return fields;
+ }
+
+ public TweetStreamService(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ assert context != null;
+ assert queue != null;
+ assert logger != null;
+
+ this.queue = queue;
+ this.logger = logger;
+
+ this.tweetFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.TWEET_FIELDS);
+ this.userFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.USER_FIELDS);
+ this.mediaFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.MEDIA_FIELDS);
+ this.pollFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.POLL_FIELDS);
+ this.placeFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.PLACE_FIELDS);
+ this.expansions = parseCommaSeparatedListPropreties(context, ConsumeTwitter.EXPANSIONS);
+ this.backfillMinutes = context.getProperty(ConsumeTwitter.BACKFILL_MINUTES).asInteger();
+
+ this.backoffMultiplier = 1L;
+ this.backoffAttempts = context.getProperty(ConsumeTwitter.BACKOFF_ATTEMPTS).asInteger();
+ this.attemptCounter = 0;
+ this.backoffTime = context.getProperty(ConsumeTwitter.BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+ this.maximumBackoff = context.getProperty(ConsumeTwitter.MAXIMUM_BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+
+ ApiClient client = new ApiClient();
+ final int connectTimeout = context.getProperty(ConsumeTwitter.CONNECT_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ final int readTimeout = context.getProperty(ConsumeTwitter.READ_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ client = client.setConnectTimeout(connectTimeout);
+ client = client.setReadTimeout(readTimeout);
+ api = new TwitterApi(client);
+
+ TwitterCredentialsBearer creds = new TwitterCredentialsBearer(context.getProperty(ConsumeTwitter.BEARER_TOKEN).getValue());
+ api.setTwitterCredentials(creds);
+
+ this.executorService = Executors.newSingleThreadScheduledExecutor();
+ }
+
+ public String getBasePath() {
+ return api.getApiClient().getBasePath();
+ }
+
+ public void setBasePath(final String path) {
+ api.getApiClient().setBasePath(path);
+ }
+
+ public String getTransitUri(final String endpoint) {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ return api.getApiClient().getBasePath() + SAMPLE_PATH;
+ } else if (endpoint.equals(SEARCH_ENDPOINT)) {
+ return api.getApiClient().getBasePath() + SEARCH_PATH;
+ } else {
+ logger.warn("Unrecognized endpoint in getTransitUri. Returning basePath");
+ return api.getApiClient().getBasePath();
+ }
+ }
+
+ /**
+ * This method would be called when we would like the stream to get started. This method will spin off a thread that
+ * will continue to queue tweets on to the given queue passed in the constructor. The thread will continue
+ * to run until {@code stop} is called.
+ * @param endpoint {@code TwitterStreamAPI.SAMPLE_ENDPOINT} or {@code TwitterStreamAPI.SEARCH_ENDPOINT}
+ */
+ public void start(final String endpoint) {
+ this.endpoint = endpoint;
+ executorService.execute(new TweetStreamStarter());
+ }
+
+ /**
+ * This method would be called when we would like the stream to get stopped. The stream will be closed and the
+ * executorService will be shut down.
+ */
+ public void stop() {
+ if (stream != null) {
+ try {
+ stream.close();
+ } catch (IOException e) {
+ logger.error("Closing response stream failed", e);
+ }
+ }
+
+ executorService.shutdownNow();
+ }
+
+ private Long calculateBackoffDelay() {
+ long backoff = backoffMultiplier * backoffTime;
+ long delay = (backoff > maximumBackoff) ? maximumBackoff : backoff;
+ return delay;
+ }
+
+ private void scheduleStartStreamWithBackoff() {
+ // use exponential(by factor of 2) backoff in scheduling the next TweetStreamStarter
+ if (attemptCounter >= backoffAttempts) {
+ final String msg = "Reached maximum attempts to backoff";
+ logger.error(msg);
+ throw new ProcessException(msg);
+ }
+ attemptCounter += 1;
+ long delay = calculateBackoffDelay();
+ backoffMultiplier *= 2;
+ logger.info("Scheduling new TweetStreamStarter after {}s delay", delay);
Review Comment:
Recommend rewording:
```suggestion
logger.info("Scheduling new stream connection after delay [{} s]", delay);
```
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TweetStreamService.java:
##########
@@ -0,0 +1,242 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiClient;
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TweetStreamService {
+ public static final String SEARCH_ENDPOINT = "Search Endpoint";
+ public static final String SAMPLE_ENDPOINT = "Sample Endpoint";
+ private static final String SAMPLE_PATH = "/2/tweets/sample/stream";
+ private static final String SEARCH_PATH = "/2/tweets/search/stream";
+
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ScheduledExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Long backoffMultiplier;
+ private int backoffAttempts;
+ private int attemptCounter;
+ private Long backoffTime;
+ private Long maximumBackoff;
+
+ private String endpoint;
+
+ private Set<String> parseCommaSeparatedListPropreties(final ProcessContext context, final PropertyDescriptor property) {
+ Set<String> fields = null;
+ if (context.getProperty(property).isSet()) {
+ fields = new HashSet<>();
+ final String fieldsString = context.getProperty(property).getValue();
+ for (final String field: fieldsString.split(",")) {
+ fields.add(field.trim());
+ }
+ }
+ return fields;
+ }
+
+ public TweetStreamService(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ assert context != null;
+ assert queue != null;
+ assert logger != null;
+
+ this.queue = queue;
+ this.logger = logger;
+
+ this.tweetFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.TWEET_FIELDS);
+ this.userFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.USER_FIELDS);
+ this.mediaFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.MEDIA_FIELDS);
+ this.pollFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.POLL_FIELDS);
+ this.placeFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.PLACE_FIELDS);
+ this.expansions = parseCommaSeparatedListPropreties(context, ConsumeTwitter.EXPANSIONS);
+ this.backfillMinutes = context.getProperty(ConsumeTwitter.BACKFILL_MINUTES).asInteger();
+
+ this.backoffMultiplier = 1L;
+ this.backoffAttempts = context.getProperty(ConsumeTwitter.BACKOFF_ATTEMPTS).asInteger();
+ this.attemptCounter = 0;
+ this.backoffTime = context.getProperty(ConsumeTwitter.BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+ this.maximumBackoff = context.getProperty(ConsumeTwitter.MAXIMUM_BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+
+ ApiClient client = new ApiClient();
+ final int connectTimeout = context.getProperty(ConsumeTwitter.CONNECT_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ final int readTimeout = context.getProperty(ConsumeTwitter.READ_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ client = client.setConnectTimeout(connectTimeout);
+ client = client.setReadTimeout(readTimeout);
+ api = new TwitterApi(client);
+
+ TwitterCredentialsBearer creds = new TwitterCredentialsBearer(context.getProperty(ConsumeTwitter.BEARER_TOKEN).getValue());
+ api.setTwitterCredentials(creds);
+
+ this.executorService = Executors.newSingleThreadScheduledExecutor();
+ }
+
+ public String getBasePath() {
+ return api.getApiClient().getBasePath();
+ }
+
+ public void setBasePath(final String path) {
+ api.getApiClient().setBasePath(path);
+ }
+
+ public String getTransitUri(final String endpoint) {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ return api.getApiClient().getBasePath() + SAMPLE_PATH;
+ } else if (endpoint.equals(SEARCH_ENDPOINT)) {
+ return api.getApiClient().getBasePath() + SEARCH_PATH;
+ } else {
+ logger.warn("Unrecognized endpoint in getTransitUri. Returning basePath");
+ return api.getApiClient().getBasePath();
+ }
+ }
+
+ /**
+ * This method would be called when we would like the stream to get started. This method will spin off a thread that
+ * will continue to queue tweets on to the given queue passed in the constructor. The thread will continue
+ * to run until {@code stop} is called.
+ * @param endpoint {@code TwitterStreamAPI.SAMPLE_ENDPOINT} or {@code TwitterStreamAPI.SEARCH_ENDPOINT}
+ */
+ public void start(final String endpoint) {
+ this.endpoint = endpoint;
+ executorService.execute(new TweetStreamStarter());
+ }
+
+ /**
+ * This method would be called when we would like the stream to get stopped. The stream will be closed and the
+ * executorService will be shut down.
+ */
+ public void stop() {
+ if (stream != null) {
+ try {
+ stream.close();
+ } catch (IOException e) {
+ logger.error("Closing response stream failed", e);
+ }
+ }
+
+ executorService.shutdownNow();
+ }
+
+ private Long calculateBackoffDelay() {
+ long backoff = backoffMultiplier * backoffTime;
+ long delay = (backoff > maximumBackoff) ? maximumBackoff : backoff;
+ return delay;
+ }
+
+ private void scheduleStartStreamWithBackoff() {
+ // use exponential(by factor of 2) backoff in scheduling the next TweetStreamStarter
+ if (attemptCounter >= backoffAttempts) {
+ final String msg = "Reached maximum attempts to backoff";
+ logger.error(msg);
+ throw new ProcessException(msg);
+ }
+ attemptCounter += 1;
+ long delay = calculateBackoffDelay();
+ backoffMultiplier *= 2;
+ logger.info("Scheduling new TweetStreamStarter after {}s delay", delay);
+ executorService.schedule(new TweetStreamStarter(), delay, TimeUnit.SECONDS);
+ }
+
+ private void resetBackoff() {
+ attemptCounter = 0;
+ backoffMultiplier = 1L;
+ }
+
+ private class TweetStreamStarter implements Runnable {
+ @Override
+ public void run() {
+ try {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ stream = api.tweets().sampleStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else {
+ stream = api.tweets().searchStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ }
+ } catch (ApiException e) {
+ stream = null;
+ logger.warn(String.format("Received ApiException, error %d: %s", e.getCode(), e.getMessage()));
+ scheduleStartStreamWithBackoff();
+ return;
+ } catch (Exception e) {
+ stream = null;
+ logger.warn(String.format("Received Exception %s: %s", e.getClass(), e.getMessage()));
+ scheduleStartStreamWithBackoff();
+ return;
+ }
+
+ if (stream == null) {
+ logger.warn("Stream is null, could not make a connection to the Twitter API");
+ scheduleStartStreamWithBackoff();
+ return;
+ } else {
+ executorService.execute(new TweetStreamHandler());
+ }
+ }
+ }
+
+ private class TweetStreamHandler implements Runnable {
+ @Override
+ public void run() {
+ try (BufferedReader reader = new BufferedReader(new InputStreamReader(stream))) {
+ String tweetRecord = reader.readLine();
+ while (tweetRecord != null) {
+ queue.offer(tweetRecord);
+
+ // reset backoff multiplier upon successful receipt of a tweet
+ resetBackoff();
+ try {
+ tweetRecord = reader.readLine();
+ } catch (IOException e) {
Review Comment:
Is this separate catch necessary as opposed to letting the outer catch `IOException` handle the failure?
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TweetStreamService.java:
##########
@@ -0,0 +1,242 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiClient;
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TweetStreamService {
+ public static final String SEARCH_ENDPOINT = "Search Endpoint";
+ public static final String SAMPLE_ENDPOINT = "Sample Endpoint";
+ private static final String SAMPLE_PATH = "/2/tweets/sample/stream";
+ private static final String SEARCH_PATH = "/2/tweets/search/stream";
+
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ScheduledExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Long backoffMultiplier;
+ private int backoffAttempts;
+ private int attemptCounter;
+ private Long backoffTime;
+ private Long maximumBackoff;
+
+ private String endpoint;
+
+ private Set<String> parseCommaSeparatedListPropreties(final ProcessContext context, final PropertyDescriptor property) {
+ Set<String> fields = null;
+ if (context.getProperty(property).isSet()) {
+ fields = new HashSet<>();
+ final String fieldsString = context.getProperty(property).getValue();
+ for (final String field: fieldsString.split(",")) {
+ fields.add(field.trim());
+ }
+ }
+ return fields;
+ }
+
+ public TweetStreamService(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ assert context != null;
+ assert queue != null;
+ assert logger != null;
+
+ this.queue = queue;
+ this.logger = logger;
+
+ this.tweetFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.TWEET_FIELDS);
+ this.userFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.USER_FIELDS);
+ this.mediaFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.MEDIA_FIELDS);
+ this.pollFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.POLL_FIELDS);
+ this.placeFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.PLACE_FIELDS);
+ this.expansions = parseCommaSeparatedListPropreties(context, ConsumeTwitter.EXPANSIONS);
+ this.backfillMinutes = context.getProperty(ConsumeTwitter.BACKFILL_MINUTES).asInteger();
+
+ this.backoffMultiplier = 1L;
+ this.backoffAttempts = context.getProperty(ConsumeTwitter.BACKOFF_ATTEMPTS).asInteger();
+ this.attemptCounter = 0;
+ this.backoffTime = context.getProperty(ConsumeTwitter.BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+ this.maximumBackoff = context.getProperty(ConsumeTwitter.MAXIMUM_BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+
+ ApiClient client = new ApiClient();
+ final int connectTimeout = context.getProperty(ConsumeTwitter.CONNECT_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ final int readTimeout = context.getProperty(ConsumeTwitter.READ_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ client = client.setConnectTimeout(connectTimeout);
+ client = client.setReadTimeout(readTimeout);
+ api = new TwitterApi(client);
+
+ TwitterCredentialsBearer creds = new TwitterCredentialsBearer(context.getProperty(ConsumeTwitter.BEARER_TOKEN).getValue());
+ api.setTwitterCredentials(creds);
+
+ this.executorService = Executors.newSingleThreadScheduledExecutor();
+ }
+
+ public String getBasePath() {
+ return api.getApiClient().getBasePath();
+ }
+
+ public void setBasePath(final String path) {
+ api.getApiClient().setBasePath(path);
+ }
+
+ public String getTransitUri(final String endpoint) {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ return api.getApiClient().getBasePath() + SAMPLE_PATH;
+ } else if (endpoint.equals(SEARCH_ENDPOINT)) {
+ return api.getApiClient().getBasePath() + SEARCH_PATH;
+ } else {
+ logger.warn("Unrecognized endpoint in getTransitUri. Returning basePath");
+ return api.getApiClient().getBasePath();
+ }
+ }
+
+ /**
+ * This method would be called when we would like the stream to get started. This method will spin off a thread that
+ * will continue to queue tweets on to the given queue passed in the constructor. The thread will continue
+ * to run until {@code stop} is called.
+ * @param endpoint {@code TwitterStreamAPI.SAMPLE_ENDPOINT} or {@code TwitterStreamAPI.SEARCH_ENDPOINT}
+ */
+ public void start(final String endpoint) {
+ this.endpoint = endpoint;
+ executorService.execute(new TweetStreamStarter());
+ }
+
+ /**
+ * This method would be called when we would like the stream to get stopped. The stream will be closed and the
+ * executorService will be shut down.
+ */
+ public void stop() {
+ if (stream != null) {
+ try {
+ stream.close();
+ } catch (IOException e) {
+ logger.error("Closing response stream failed", e);
+ }
+ }
+
+ executorService.shutdownNow();
+ }
+
+ private Long calculateBackoffDelay() {
+ long backoff = backoffMultiplier * backoffTime;
+ long delay = (backoff > maximumBackoff) ? maximumBackoff : backoff;
+ return delay;
+ }
+
+ private void scheduleStartStreamWithBackoff() {
+ // use exponential(by factor of 2) backoff in scheduling the next TweetStreamStarter
+ if (attemptCounter >= backoffAttempts) {
+ final String msg = "Reached maximum attempts to backoff";
+ logger.error(msg);
+ throw new ProcessException(msg);
+ }
+ attemptCounter += 1;
+ long delay = calculateBackoffDelay();
+ backoffMultiplier *= 2;
+ logger.info("Scheduling new TweetStreamStarter after {}s delay", delay);
+ executorService.schedule(new TweetStreamStarter(), delay, TimeUnit.SECONDS);
+ }
+
+ private void resetBackoff() {
+ attemptCounter = 0;
+ backoffMultiplier = 1L;
+ }
+
+ private class TweetStreamStarter implements Runnable {
+ @Override
+ public void run() {
+ try {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ stream = api.tweets().sampleStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else {
+ stream = api.tweets().searchStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ }
+ } catch (ApiException e) {
+ stream = null;
+ logger.warn(String.format("Received ApiException, error %d: %s", e.getCode(), e.getMessage()));
+ scheduleStartStreamWithBackoff();
+ return;
+ } catch (Exception e) {
+ stream = null;
+ logger.warn(String.format("Received Exception %s: %s", e.getClass(), e.getMessage()));
Review Comment:
```suggestion
logger.warn("Twitter Stream [{}] connection failed", endpoint, e);
```
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TweetStreamService.java:
##########
@@ -0,0 +1,242 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiClient;
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TweetStreamService {
+ public static final String SEARCH_ENDPOINT = "Search Endpoint";
+ public static final String SAMPLE_ENDPOINT = "Sample Endpoint";
+ private static final String SAMPLE_PATH = "/2/tweets/sample/stream";
+ private static final String SEARCH_PATH = "/2/tweets/search/stream";
+
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ScheduledExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Long backoffMultiplier;
+ private int backoffAttempts;
+ private int attemptCounter;
+ private Long backoffTime;
+ private Long maximumBackoff;
+
+ private String endpoint;
+
+ private Set<String> parseCommaSeparatedListPropreties(final ProcessContext context, final PropertyDescriptor property) {
+ Set<String> fields = null;
+ if (context.getProperty(property).isSet()) {
+ fields = new HashSet<>();
+ final String fieldsString = context.getProperty(property).getValue();
+ for (final String field: fieldsString.split(",")) {
+ fields.add(field.trim());
+ }
+ }
+ return fields;
+ }
+
+ public TweetStreamService(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ assert context != null;
+ assert queue != null;
+ assert logger != null;
+
+ this.queue = queue;
+ this.logger = logger;
+
+ this.tweetFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.TWEET_FIELDS);
+ this.userFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.USER_FIELDS);
+ this.mediaFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.MEDIA_FIELDS);
+ this.pollFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.POLL_FIELDS);
+ this.placeFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.PLACE_FIELDS);
+ this.expansions = parseCommaSeparatedListPropreties(context, ConsumeTwitter.EXPANSIONS);
+ this.backfillMinutes = context.getProperty(ConsumeTwitter.BACKFILL_MINUTES).asInteger();
+
+ this.backoffMultiplier = 1L;
+ this.backoffAttempts = context.getProperty(ConsumeTwitter.BACKOFF_ATTEMPTS).asInteger();
+ this.attemptCounter = 0;
+ this.backoffTime = context.getProperty(ConsumeTwitter.BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+ this.maximumBackoff = context.getProperty(ConsumeTwitter.MAXIMUM_BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+
+ ApiClient client = new ApiClient();
+ final int connectTimeout = context.getProperty(ConsumeTwitter.CONNECT_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ final int readTimeout = context.getProperty(ConsumeTwitter.READ_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ client = client.setConnectTimeout(connectTimeout);
+ client = client.setReadTimeout(readTimeout);
+ api = new TwitterApi(client);
+
+ TwitterCredentialsBearer creds = new TwitterCredentialsBearer(context.getProperty(ConsumeTwitter.BEARER_TOKEN).getValue());
+ api.setTwitterCredentials(creds);
+
+ this.executorService = Executors.newSingleThreadScheduledExecutor();
+ }
+
+ public String getBasePath() {
+ return api.getApiClient().getBasePath();
+ }
+
+ public void setBasePath(final String path) {
+ api.getApiClient().setBasePath(path);
+ }
+
+ public String getTransitUri(final String endpoint) {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ return api.getApiClient().getBasePath() + SAMPLE_PATH;
+ } else if (endpoint.equals(SEARCH_ENDPOINT)) {
+ return api.getApiClient().getBasePath() + SEARCH_PATH;
+ } else {
+ logger.warn("Unrecognized endpoint in getTransitUri. Returning basePath");
+ return api.getApiClient().getBasePath();
+ }
+ }
+
+ /**
+ * This method would be called when we would like the stream to get started. This method will spin off a thread that
+ * will continue to queue tweets on to the given queue passed in the constructor. The thread will continue
+ * to run until {@code stop} is called.
+ * @param endpoint {@code TwitterStreamAPI.SAMPLE_ENDPOINT} or {@code TwitterStreamAPI.SEARCH_ENDPOINT}
+ */
+ public void start(final String endpoint) {
+ this.endpoint = endpoint;
+ executorService.execute(new TweetStreamStarter());
+ }
+
+ /**
+ * This method would be called when we would like the stream to get stopped. The stream will be closed and the
+ * executorService will be shut down.
+ */
+ public void stop() {
+ if (stream != null) {
+ try {
+ stream.close();
+ } catch (IOException e) {
+ logger.error("Closing response stream failed", e);
+ }
+ }
+
+ executorService.shutdownNow();
+ }
+
+ private Long calculateBackoffDelay() {
+ long backoff = backoffMultiplier * backoffTime;
+ long delay = (backoff > maximumBackoff) ? maximumBackoff : backoff;
+ return delay;
+ }
+
+ private void scheduleStartStreamWithBackoff() {
+ // use exponential(by factor of 2) backoff in scheduling the next TweetStreamStarter
+ if (attemptCounter >= backoffAttempts) {
+ final String msg = "Reached maximum attempts to backoff";
+ logger.error(msg);
+ throw new ProcessException(msg);
+ }
+ attemptCounter += 1;
+ long delay = calculateBackoffDelay();
+ backoffMultiplier *= 2;
+ logger.info("Scheduling new TweetStreamStarter after {}s delay", delay);
+ executorService.schedule(new TweetStreamStarter(), delay, TimeUnit.SECONDS);
+ }
+
+ private void resetBackoff() {
+ attemptCounter = 0;
+ backoffMultiplier = 1L;
+ }
+
+ private class TweetStreamStarter implements Runnable {
+ @Override
+ public void run() {
+ try {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ stream = api.tweets().sampleStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else {
+ stream = api.tweets().searchStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ }
+ } catch (ApiException e) {
+ stream = null;
+ logger.warn(String.format("Received ApiException, error %d: %s", e.getCode(), e.getMessage()));
+ scheduleStartStreamWithBackoff();
+ return;
+ } catch (Exception e) {
+ stream = null;
+ logger.warn(String.format("Received Exception %s: %s", e.getClass(), e.getMessage()));
+ scheduleStartStreamWithBackoff();
+ return;
+ }
+
+ if (stream == null) {
+ logger.warn("Stream is null, could not make a connection to the Twitter API");
+ scheduleStartStreamWithBackoff();
+ return;
+ } else {
+ executorService.execute(new TweetStreamHandler());
Review Comment:
Rather than including this here, it might be cleaner to keep it inside the `try` block, which would avoid the need for `return` statements in the `catch` blocks.
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/test/java/org/apache/nifi/processors/twitter/TestConsumeTwitter.java:
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import okhttp3.mockwebserver.MockResponse;
+import okhttp3.mockwebserver.MockWebServer;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+
+public class TestConsumeTwitter {
+ private MockWebServer mockWebServer;
+
+ private TestRunner runner;
+
+ private final String SAMPLE_TWEET = "{\"data\":{\"id\":\"123\",\"text\":\"This is a sample tweet and is not real!\"}}";
+
+ @BeforeEach
+ public void setRunnerAndAPI() {
+ mockWebServer = new MockWebServer();
+
+ runner = TestRunners.newTestRunner(ConsumeTwitter.class);
+
+ runner.setProperty(ConsumeTwitter.BEARER_TOKEN, "BEARER_TOKEN");
+ final String basePath = mockWebServer.url("").toString();
+ runner.setProperty(ConsumeTwitter.BASE_PATH, basePath);
+ }
+
+ @AfterEach
+ public void shutdownServerAndAPI() throws IOException {
+ mockWebServer.shutdown();
+ }
+
+ @Test
+ public void testReceiveSingleTweetInStream() throws InterruptedException {
+ MockResponse response = new MockResponse()
+ .setResponseCode(200)
+ .setBody(SAMPLE_TWEET)
+ .addHeader("Content-Type", "application/json");
+ mockWebServer.enqueue(response);
+
+
+ runner.setProperty(ConsumeTwitter.ENDPOINT, ConsumeTwitter.ENDPOINT_SAMPLE);
+ runner.setProperty(ConsumeTwitter.QUEUE_SIZE, "10000");
+ runner.setProperty(ConsumeTwitter.BATCH_SIZE, "10");
+ runner.setProperty(ConsumeTwitter.BACKFILL_MINUTES, "0");
+
+ runner.assertValid();
+
+ // the TwitterStreamAPI class spins up another thread and might not be done queueing tweets in one run of the processor
+ final int maxTries = 3;
+ final long runSchedule = 250;
+ runner.setRunSchedule(runSchedule);
+ runner.run(maxTries, false, true);
+ runner.stop();
+
+ // there should only be a single FlowFile containing a tweet
+ runner.assertTransferCount(ConsumeTwitter.REL_SUCCESS, 1);
+ MockFlowFile flowFile = runner.getFlowFilesForRelationship(ConsumeTwitter.REL_SUCCESS).get(0);
+ flowFile.assertContentEquals("[" + SAMPLE_TWEET + "]");
Review Comment:
Recommend including the square brackets on the expected variable:
```suggestion
flowFile.assertContentEquals(EXPECTED_TWEETS);
```
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TweetStreamService.java:
##########
@@ -0,0 +1,242 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiClient;
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TweetStreamService {
+ public static final String SEARCH_ENDPOINT = "Search Endpoint";
+ public static final String SAMPLE_ENDPOINT = "Sample Endpoint";
+ private static final String SAMPLE_PATH = "/2/tweets/sample/stream";
+ private static final String SEARCH_PATH = "/2/tweets/search/stream";
+
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ScheduledExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Long backoffMultiplier;
+ private int backoffAttempts;
+ private int attemptCounter;
+ private Long backoffTime;
+ private Long maximumBackoff;
+
+ private String endpoint;
+
+ private Set<String> parseCommaSeparatedListPropreties(final ProcessContext context, final PropertyDescriptor property) {
+ Set<String> fields = null;
+ if (context.getProperty(property).isSet()) {
+ fields = new HashSet<>();
+ final String fieldsString = context.getProperty(property).getValue();
+ for (final String field: fieldsString.split(",")) {
+ fields.add(field.trim());
+ }
+ }
+ return fields;
+ }
+
+ public TweetStreamService(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ assert context != null;
+ assert queue != null;
+ assert logger != null;
+
+ this.queue = queue;
+ this.logger = logger;
+
+ this.tweetFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.TWEET_FIELDS);
+ this.userFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.USER_FIELDS);
+ this.mediaFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.MEDIA_FIELDS);
+ this.pollFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.POLL_FIELDS);
+ this.placeFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.PLACE_FIELDS);
+ this.expansions = parseCommaSeparatedListPropreties(context, ConsumeTwitter.EXPANSIONS);
+ this.backfillMinutes = context.getProperty(ConsumeTwitter.BACKFILL_MINUTES).asInteger();
+
+ this.backoffMultiplier = 1L;
+ this.backoffAttempts = context.getProperty(ConsumeTwitter.BACKOFF_ATTEMPTS).asInteger();
+ this.attemptCounter = 0;
+ this.backoffTime = context.getProperty(ConsumeTwitter.BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+ this.maximumBackoff = context.getProperty(ConsumeTwitter.MAXIMUM_BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+
+ ApiClient client = new ApiClient();
+ final int connectTimeout = context.getProperty(ConsumeTwitter.CONNECT_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ final int readTimeout = context.getProperty(ConsumeTwitter.READ_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ client = client.setConnectTimeout(connectTimeout);
+ client = client.setReadTimeout(readTimeout);
+ api = new TwitterApi(client);
+
+ TwitterCredentialsBearer creds = new TwitterCredentialsBearer(context.getProperty(ConsumeTwitter.BEARER_TOKEN).getValue());
+ api.setTwitterCredentials(creds);
+
+ this.executorService = Executors.newSingleThreadScheduledExecutor();
+ }
+
+ public String getBasePath() {
+ return api.getApiClient().getBasePath();
+ }
+
+ public void setBasePath(final String path) {
+ api.getApiClient().setBasePath(path);
+ }
+
+ public String getTransitUri(final String endpoint) {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ return api.getApiClient().getBasePath() + SAMPLE_PATH;
+ } else if (endpoint.equals(SEARCH_ENDPOINT)) {
+ return api.getApiClient().getBasePath() + SEARCH_PATH;
+ } else {
+ logger.warn("Unrecognized endpoint in getTransitUri. Returning basePath");
+ return api.getApiClient().getBasePath();
+ }
+ }
+
+ /**
+ * This method would be called when we would like the stream to get started. This method will spin off a thread that
+ * will continue to queue tweets on to the given queue passed in the constructor. The thread will continue
+ * to run until {@code stop} is called.
+ * @param endpoint {@code TwitterStreamAPI.SAMPLE_ENDPOINT} or {@code TwitterStreamAPI.SEARCH_ENDPOINT}
+ */
+ public void start(final String endpoint) {
+ this.endpoint = endpoint;
+ executorService.execute(new TweetStreamStarter());
+ }
+
+ /**
+ * This method would be called when we would like the stream to get stopped. The stream will be closed and the
+ * executorService will be shut down.
+ */
+ public void stop() {
+ if (stream != null) {
+ try {
+ stream.close();
+ } catch (IOException e) {
+ logger.error("Closing response stream failed", e);
+ }
+ }
+
+ executorService.shutdownNow();
+ }
+
+ private Long calculateBackoffDelay() {
+ long backoff = backoffMultiplier * backoffTime;
+ long delay = (backoff > maximumBackoff) ? maximumBackoff : backoff;
+ return delay;
+ }
+
+ private void scheduleStartStreamWithBackoff() {
+ // use exponential(by factor of 2) backoff in scheduling the next TweetStreamStarter
+ if (attemptCounter >= backoffAttempts) {
+ final String msg = "Reached maximum attempts to backoff";
+ logger.error(msg);
+ throw new ProcessException(msg);
+ }
+ attemptCounter += 1;
+ long delay = calculateBackoffDelay();
+ backoffMultiplier *= 2;
+ logger.info("Scheduling new TweetStreamStarter after {}s delay", delay);
+ executorService.schedule(new TweetStreamStarter(), delay, TimeUnit.SECONDS);
+ }
+
+ private void resetBackoff() {
+ attemptCounter = 0;
+ backoffMultiplier = 1L;
+ }
+
+ private class TweetStreamStarter implements Runnable {
+ @Override
+ public void run() {
+ try {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ stream = api.tweets().sampleStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else {
+ stream = api.tweets().searchStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ }
+ } catch (ApiException e) {
+ stream = null;
+ logger.warn(String.format("Received ApiException, error %d: %s", e.getCode(), e.getMessage()));
+ scheduleStartStreamWithBackoff();
+ return;
+ } catch (Exception e) {
+ stream = null;
+ logger.warn(String.format("Received Exception %s: %s", e.getClass(), e.getMessage()));
+ scheduleStartStreamWithBackoff();
+ return;
+ }
+
+ if (stream == null) {
+ logger.warn("Stream is null, could not make a connection to the Twitter API");
+ scheduleStartStreamWithBackoff();
+ return;
+ } else {
+ executorService.execute(new TweetStreamHandler());
+ }
+ }
+ }
+
+ private class TweetStreamHandler implements Runnable {
+ @Override
+ public void run() {
+ try (BufferedReader reader = new BufferedReader(new InputStreamReader(stream))) {
+ String tweetRecord = reader.readLine();
+ while (tweetRecord != null) {
+ queue.offer(tweetRecord);
+
+ // reset backoff multiplier upon successful receipt of a tweet
+ resetBackoff();
+ try {
+ tweetRecord = reader.readLine();
+ } catch (IOException e) {
+ logger.info("Read Tweet failed: Stream processing completed");
+ break;
+ }
+ }
+ } catch (IOException e) {
+ logger.warn("Stream processing failed", e);
+ }
+ logger.info("TweetStreamHandler terminating");
Review Comment:
```suggestion
logger.info("Tweet Stream processing completed");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r861147382
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TweetStreamService.java:
##########
@@ -0,0 +1,242 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiClient;
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TweetStreamService {
+ public static final String SEARCH_ENDPOINT = "Search Endpoint";
+ public static final String SAMPLE_ENDPOINT = "Sample Endpoint";
+ private static final String SAMPLE_PATH = "/2/tweets/sample/stream";
+ private static final String SEARCH_PATH = "/2/tweets/search/stream";
+
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ScheduledExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Long backoffMultiplier;
+ private int backoffAttempts;
+ private int attemptCounter;
+ private Long backoffTime;
+ private Long maximumBackoff;
+
+ private String endpoint;
+
+ private Set<String> parseCommaSeparatedListPropreties(final ProcessContext context, final PropertyDescriptor property) {
+ Set<String> fields = null;
+ if (context.getProperty(property).isSet()) {
+ fields = new HashSet<>();
+ final String fieldsString = context.getProperty(property).getValue();
+ for (final String field: fieldsString.split(",")) {
+ fields.add(field.trim());
+ }
+ }
+ return fields;
+ }
+
+ public TweetStreamService(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ assert context != null;
+ assert queue != null;
+ assert logger != null;
+
+ this.queue = queue;
+ this.logger = logger;
+
+ this.tweetFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.TWEET_FIELDS);
+ this.userFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.USER_FIELDS);
+ this.mediaFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.MEDIA_FIELDS);
+ this.pollFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.POLL_FIELDS);
+ this.placeFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.PLACE_FIELDS);
+ this.expansions = parseCommaSeparatedListPropreties(context, ConsumeTwitter.EXPANSIONS);
+ this.backfillMinutes = context.getProperty(ConsumeTwitter.BACKFILL_MINUTES).asInteger();
+
+ this.backoffMultiplier = 1L;
+ this.backoffAttempts = context.getProperty(ConsumeTwitter.BACKOFF_ATTEMPTS).asInteger();
+ this.attemptCounter = 0;
+ this.backoffTime = context.getProperty(ConsumeTwitter.BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+ this.maximumBackoff = context.getProperty(ConsumeTwitter.MAXIMUM_BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+
+ ApiClient client = new ApiClient();
+ final int connectTimeout = context.getProperty(ConsumeTwitter.CONNECT_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ final int readTimeout = context.getProperty(ConsumeTwitter.READ_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ client = client.setConnectTimeout(connectTimeout);
+ client = client.setReadTimeout(readTimeout);
+ api = new TwitterApi(client);
+
+ TwitterCredentialsBearer creds = new TwitterCredentialsBearer(context.getProperty(ConsumeTwitter.BEARER_TOKEN).getValue());
+ api.setTwitterCredentials(creds);
+
+ this.executorService = Executors.newSingleThreadScheduledExecutor();
+ }
+
+ public String getBasePath() {
+ return api.getApiClient().getBasePath();
+ }
+
+ public void setBasePath(final String path) {
+ api.getApiClient().setBasePath(path);
+ }
+
+ public String getTransitUri(final String endpoint) {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ return api.getApiClient().getBasePath() + SAMPLE_PATH;
+ } else if (endpoint.equals(SEARCH_ENDPOINT)) {
+ return api.getApiClient().getBasePath() + SEARCH_PATH;
+ } else {
+ logger.warn("Unrecognized endpoint in getTransitUri. Returning basePath");
+ return api.getApiClient().getBasePath();
+ }
+ }
+
+ /**
+ * This method would be called when we would like the stream to get started. This method will spin off a thread that
+ * will continue to queue tweets on to the given queue passed in the constructor. The thread will continue
+ * to run until {@code stop} is called.
+ * @param endpoint {@code TwitterStreamAPI.SAMPLE_ENDPOINT} or {@code TwitterStreamAPI.SEARCH_ENDPOINT}
+ */
+ public void start(final String endpoint) {
+ this.endpoint = endpoint;
+ executorService.execute(new TweetStreamStarter());
+ }
+
+ /**
+ * This method would be called when we would like the stream to get stopped. The stream will be closed and the
+ * executorService will be shut down.
+ */
+ public void stop() {
+ if (stream != null) {
+ try {
+ stream.close();
+ } catch (IOException e) {
+ logger.error("Closing response stream failed", e);
+ }
+ }
+
+ executorService.shutdownNow();
+ }
+
+ private Long calculateBackoffDelay() {
+ long backoff = backoffMultiplier * backoffTime;
+ long delay = (backoff > maximumBackoff) ? maximumBackoff : backoff;
+ return delay;
+ }
+
+ private void scheduleStartStreamWithBackoff() {
+ // use exponential(by factor of 2) backoff in scheduling the next TweetStreamStarter
+ if (attemptCounter >= backoffAttempts) {
+ final String msg = "Reached maximum attempts to backoff";
+ logger.error(msg);
+ throw new ProcessException(msg);
+ }
+ attemptCounter += 1;
+ long delay = calculateBackoffDelay();
+ backoffMultiplier *= 2;
+ logger.info("Scheduling new TweetStreamStarter after {}s delay", delay);
Review Comment:
Making the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r861145141
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TweetStreamService.java:
##########
@@ -0,0 +1,242 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiClient;
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TweetStreamService {
+ public static final String SEARCH_ENDPOINT = "Search Endpoint";
+ public static final String SAMPLE_ENDPOINT = "Sample Endpoint";
+ private static final String SAMPLE_PATH = "/2/tweets/sample/stream";
+ private static final String SEARCH_PATH = "/2/tweets/search/stream";
+
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ScheduledExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Long backoffMultiplier;
+ private int backoffAttempts;
+ private int attemptCounter;
+ private Long backoffTime;
+ private Long maximumBackoff;
+
+ private String endpoint;
+
+ private Set<String> parseCommaSeparatedListPropreties(final ProcessContext context, final PropertyDescriptor property) {
+ Set<String> fields = null;
+ if (context.getProperty(property).isSet()) {
+ fields = new HashSet<>();
+ final String fieldsString = context.getProperty(property).getValue();
+ for (final String field: fieldsString.split(",")) {
+ fields.add(field.trim());
+ }
+ }
+ return fields;
+ }
+
+ public TweetStreamService(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ assert context != null;
+ assert queue != null;
+ assert logger != null;
Review Comment:
Making the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r861143413
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TweetStreamService.java:
##########
@@ -0,0 +1,242 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiClient;
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TweetStreamService {
+ public static final String SEARCH_ENDPOINT = "Search Endpoint";
+ public static final String SAMPLE_ENDPOINT = "Sample Endpoint";
+ private static final String SAMPLE_PATH = "/2/tweets/sample/stream";
+ private static final String SEARCH_PATH = "/2/tweets/search/stream";
+
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ScheduledExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Long backoffMultiplier;
+ private int backoffAttempts;
+ private int attemptCounter;
+ private Long backoffTime;
+ private Long maximumBackoff;
+
+ private String endpoint;
+
+ private Set<String> parseCommaSeparatedListPropreties(final ProcessContext context, final PropertyDescriptor property) {
+ Set<String> fields = null;
+ if (context.getProperty(property).isSet()) {
+ fields = new HashSet<>();
+ final String fieldsString = context.getProperty(property).getValue();
+ for (final String field: fieldsString.split(",")) {
+ fields.add(field.trim());
+ }
+ }
+ return fields;
+ }
Review Comment:
Making the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r846716705
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,316 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("Twitter Endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("Base Path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("Bearer Token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("Queue Size")
+ .displayName("Queue Size")
+ .description("The size of the queue used to queue tweets")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("Batch Size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("Tweet Fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values are: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("User Fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user " +
+ "for proper usage. Possible field values are: " +
+ "created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, " +
+ "protected, public_metrics, url, username, verified, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor MEDIA_FIELDS = new PropertyDescriptor.Builder()
+ .name("Media Fields")
+ .displayName("Media Fields")
+ .description("A comma-separated list of media fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/media " +
+ "for proper usage. Possible field values are: " +
+ "alt_text, duration_ms, height, media_key, non_public_metrics, organic_metrics, preview_image_url, " +
+ "promoted_metrics, public_metrics, type, url, width")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor POLL_FIELDS = new PropertyDescriptor.Builder()
+ .name("Poll Fields")
+ .displayName("Poll Fields")
+ .description("A comma-separated list of poll fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/poll " +
+ "for proper usage. Possible field values are: " +
+ "duration_minutes, end_datetime, id, options, voting_status")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PLACE_FIELDS = new PropertyDescriptor.Builder()
+ .name("Place Fields")
+ .displayName("Place Fields")
+ .description("A comma-separated list of place fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/place " +
+ "for proper usage. Possible field values are: " +
+ "contained_within, country, country_code, full_name, geo, id, name, place_type")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor EXPANSIONS = new PropertyDescriptor.Builder()
+ .name("Expansions")
+ .displayName("Expansions")
+ .description("A comma-separated list of expansions for objects in the returned tweet. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/expansions " +
+ "for proper usage. Possible field values are: " +
+ "author_id, referenced_tweets.id, referenced_tweets.id.author_id, entities.mentions.username, " +
+ "attachments.poll_ids, attachments.media_keys ,in_reply_to_user_id, geo.place_id")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor BACKFILL_MINUTES = new PropertyDescriptor.Builder()
+ .name("Backfill Minutes")
+ .displayName("Backfill Minutes")
+ .description("The number of minutes (up to 5 minutes) of streaming data to be requested after a " +
+ "disconnect. Only available for project with academic research access. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/integrate/" +
+ "recovery-and-redundancy-features")
+ .required(true)
+ .defaultValue("0")
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("All tweets will be routed to this relationship.")
+ .build();
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+
+ private TwitterStreamAPI api;
+
+ private volatile BlockingQueue<String> messageQueue;
+
+ private void emptyQueue() {
+ while (!messageQueue.isEmpty()) {
+ messageQueue.poll();
+ }
+ }
+
+ @Override
+ protected void init(ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptors = new ArrayList<>();
+ descriptors.add(ENDPOINT);
+ descriptors.add(BASE_PATH);
+ descriptors.add(BEARER_TOKEN);
+ descriptors.add(QUEUE_SIZE);
+ descriptors.add(BATCH_SIZE);
+ descriptors.add(TWEET_FIELDS);
+ descriptors.add(USER_FIELDS);
+ descriptors.add(MEDIA_FIELDS);
+ descriptors.add(POLL_FIELDS);
+ descriptors.add(PLACE_FIELDS);
+ descriptors.add(EXPANSIONS);
+ descriptors.add(BACKFILL_MINUTES);
+
+ this.descriptors = Collections.unmodifiableList(descriptors);
+
+ final Set<Relationship> relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ this.relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return this.descriptors;
+ }
+
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ messageQueue = new LinkedBlockingQueue<>(context.getProperty(QUEUE_SIZE).asInteger());
+
+ api = new TwitterStreamAPI(context, messageQueue, getLogger());
+ api.setBasePath(context.getProperty(BASE_PATH).getValue());
+ final String endpointName = context.getProperty(ENDPOINT).getValue();
+ if (ENDPOINT_SAMPLE.getValue().equals(endpointName)) {
+ api.start(TwitterStreamAPI.SAMPLE_ENDPOINT);
+ } else if (ENDPOINT_SEARCH.getValue().equals(endpointName)) {
+ api.start(TwitterStreamAPI.SEARCH_ENDPOINT);
+ } else {
+ throw new AssertionError("Endpoint was invalid value: " + endpointName);
+ }
+ }
+
+ @Override
+ public void onTrigger(ProcessContext context, ProcessSession session) throws ProcessException {
+ if (messageQueue.isEmpty()) {
+ context.yield();
+ return;
+ }
+
+ FlowFile flowFile = session.create();
+ flowFile = session.write(flowFile, new OutputStreamCallback() {
+ @Override
+ public void process(OutputStream out) throws IOException {
+ out.write("[".getBytes(StandardCharsets.UTF_8));
+ final int batchSize = context.getProperty(BATCH_SIZE).asInteger();
+ String tweet = messageQueue.poll();
+ out.write(tweet.getBytes(StandardCharsets.UTF_8));
+ int tweetCount = 1;
+ while (tweetCount < batchSize && !messageQueue.isEmpty()) {
+ out.write(",".getBytes(StandardCharsets.UTF_8));
+ tweet = messageQueue.poll();
+ out.write(tweet.getBytes(StandardCharsets.UTF_8));
+ tweetCount++;
+ }
+ out.write("]".getBytes(StandardCharsets.UTF_8));
+ }
+ });
+
+ final Map<String, String> attributes = new HashMap<>();
+ attributes.put(CoreAttributes.MIME_TYPE.key(), "application/json");
+ attributes.put(CoreAttributes.FILENAME.key(), flowFile.getAttribute(CoreAttributes.FILENAME.key()) + ".json");
+ flowFile = session.putAllAttributes(flowFile, attributes);
+
+ session.transfer(flowFile, REL_SUCCESS);
+
+ String transitUri = api.getBasePath();
+ final String endpointName = context.getProperty(ENDPOINT).getValue();
+ if (ENDPOINT_SAMPLE.getValue().equals(endpointName)) {
+ transitUri += TwitterStreamAPI.SAMPLE_PATH;
+ } else if (ENDPOINT_SEARCH.getValue().equals(endpointName)) {
+ transitUri += TwitterStreamAPI.SEARCH_PATH;
+ } else {
+ throw new AssertionError("Endpoint was invalid value: " + endpointName);
Review Comment:
Resolved in comment below
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TwitterStreamAPI.java:
##########
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+public class TwitterStreamAPI {
+ public static final String SEARCH_ENDPOINT = "Search Endpoint";
+ public static final String SAMPLE_ENDPOINT = "Sample Endpoint";
+ public static final String SAMPLE_PATH = "/2/tweets/sample/stream";
+ public static final String SEARCH_PATH = "/2/tweets/search/stream";
+
+ private static final String BEARER_TOKEN_PROPERTY_NAME = "Bearer Token";
+ private static final String TWEET_FIELDS_PROPERTY_NAME = "Tweet Fields";
+ private static final String USER_FIELDS_PROPERTY_NAME = "User Fields";
+ private static final String MEDIA_FIELDS_PROPERTY_NAME = "Media Fields";
+ private static final String POLL_FIELDS_PROPERTY_NAME = "Poll Fields";
+ private static final String PLACE_FIELDS_PROPERTY_NAME = "Place Fields";
+ private static final String EXPANSIONS_PROPERTY_NAME = "Expansions";
+ private static final String BACKFILL_MINUTES_PROPERTY_NAME = "Backfill Minutes";
+
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Set<String> parseCommaSeparatedListPropreties(final ProcessContext context, final String propertyName) {
+ Set<String> fields = null;
+ if (context.getProperty(propertyName).isSet()) {
+ fields = new HashSet<>();
+ final String fieldsString = context.getProperty(propertyName).getValue();
+ for (final String field: fieldsString.split(",")) {
+ fields.add(field.trim());
+ }
+ }
+ return fields;
+ }
+
+ public TwitterStreamAPI(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ assert context != null;
+ assert queue != null;
+ assert logger != null;
+
+ this.queue = queue;
+ this.logger = logger;
+
+ this.tweetFields = parseCommaSeparatedListPropreties(context, TWEET_FIELDS_PROPERTY_NAME);
+ this.userFields = parseCommaSeparatedListPropreties(context, USER_FIELDS_PROPERTY_NAME);
+ this.mediaFields = parseCommaSeparatedListPropreties(context, MEDIA_FIELDS_PROPERTY_NAME);
+ this.pollFields = parseCommaSeparatedListPropreties(context, POLL_FIELDS_PROPERTY_NAME);
+ this.placeFields = parseCommaSeparatedListPropreties(context, PLACE_FIELDS_PROPERTY_NAME);
+ this.expansions = parseCommaSeparatedListPropreties(context, EXPANSIONS_PROPERTY_NAME);
+ this.backfillMinutes = context.getProperty(BACKFILL_MINUTES_PROPERTY_NAME).asInteger();
+
+ TwitterCredentialsBearer creds = new TwitterCredentialsBearer(context.getProperty(BEARER_TOKEN_PROPERTY_NAME).getValue());
+ api = new TwitterApi();
+ api.setTwitterCredentials(creds);
+
+ this.executorService = Executors.newSingleThreadExecutor();
+ }
+
+ public String getBasePath() {
+ return api.getApiClient().getBasePath();
+ }
+
+ public void setBasePath(final String path) {
+ api.getApiClient().setBasePath(path);
+ }
+
+ /**
+ * This method would be called when we would like the stream to get started. This method will spin off a thread that
+ * will continue to queue tweets on to the given queue passed in the constructor. The thread will continue
+ * to run until {@code stop} is called.
+ * @param endpoint {@code TwitterStreamAPI.SAMPLE_ENDPOINT} or {@code TwitterStreamAPI.SEARCH_ENDPOINT}
+ */
+ public void start(final String endpoint) {
+ try {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ stream = api.tweets().sampleStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else if (endpoint.equals(SEARCH_ENDPOINT)) {
+ stream = api.tweets().searchStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else {
+ throw new AssertionError("Endpoint was invalid value: " + endpoint);
Review Comment:
Resolved in comment below
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r848773189
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,318 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("consume-twitter-endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("The size of the queue used to queue tweets")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("tweet-fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values are: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("user-fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user " +
+ "for proper usage. Possible field values are: " +
+ "created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, " +
+ "protected, public_metrics, url, username, verified, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor MEDIA_FIELDS = new PropertyDescriptor.Builder()
+ .name("media-fields")
+ .displayName("Media Fields")
+ .description("A comma-separated list of media fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/media " +
+ "for proper usage. Possible field values are: " +
+ "alt_text, duration_ms, height, media_key, non_public_metrics, organic_metrics, preview_image_url, " +
+ "promoted_metrics, public_metrics, type, url, width")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor POLL_FIELDS = new PropertyDescriptor.Builder()
+ .name("poll-fields")
+ .displayName("Poll Fields")
+ .description("A comma-separated list of poll fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/poll " +
+ "for proper usage. Possible field values are: " +
+ "duration_minutes, end_datetime, id, options, voting_status")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PLACE_FIELDS = new PropertyDescriptor.Builder()
+ .name("place-fields")
+ .displayName("Place Fields")
+ .description("A comma-separated list of place fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/place " +
+ "for proper usage. Possible field values are: " +
+ "contained_within, country, country_code, full_name, geo, id, name, place_type")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor EXPANSIONS = new PropertyDescriptor.Builder()
+ .name("expansions")
+ .displayName("Expansions")
+ .description("A comma-separated list of expansions for objects in the returned tweet. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/expansions " +
+ "for proper usage. Possible field values are: " +
+ "author_id, referenced_tweets.id, referenced_tweets.id.author_id, entities.mentions.username, " +
+ "attachments.poll_ids, attachments.media_keys ,in_reply_to_user_id, geo.place_id")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor BACKFILL_MINUTES = new PropertyDescriptor.Builder()
+ .name("backfill-minutes")
+ .displayName("Backfill Minutes")
+ .description("The number of minutes (up to 5 minutes) of streaming data to be requested after a " +
+ "disconnect. Only available for project with academic research access. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/integrate/" +
+ "recovery-and-redundancy-features")
+ .required(true)
+ .defaultValue("0")
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("All tweets will be routed to this relationship.")
+ .build();
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+
+ private TwitterStreamAPI api;
+
+ private volatile BlockingQueue<String> messageQueue;
+
+ private void emptyQueue() {
+ while (!messageQueue.isEmpty()) {
+ messageQueue.poll();
+ }
+ }
+
+ @Override
+ protected void init(ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptors = new ArrayList<>();
+ descriptors.add(ENDPOINT);
+ descriptors.add(BASE_PATH);
+ descriptors.add(BEARER_TOKEN);
+ descriptors.add(QUEUE_SIZE);
+ descriptors.add(BATCH_SIZE);
+ descriptors.add(TWEET_FIELDS);
+ descriptors.add(USER_FIELDS);
+ descriptors.add(MEDIA_FIELDS);
+ descriptors.add(POLL_FIELDS);
+ descriptors.add(PLACE_FIELDS);
+ descriptors.add(EXPANSIONS);
+ descriptors.add(BACKFILL_MINUTES);
+
+ this.descriptors = Collections.unmodifiableList(descriptors);
+
+ final Set<Relationship> relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ this.relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return this.descriptors;
+ }
+
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ messageQueue = new LinkedBlockingQueue<>(context.getProperty(QUEUE_SIZE).asInteger());
+
+ api = new TwitterStreamAPI(context, messageQueue, getLogger());
+ api.setBasePath(context.getProperty(BASE_PATH).getValue());
+ final String endpointName = context.getProperty(ENDPOINT).getValue();
+ if (ENDPOINT_SAMPLE.getValue().equals(endpointName)) {
+ api.start(TwitterStreamAPI.SAMPLE_ENDPOINT);
+ } else if (ENDPOINT_SEARCH.getValue().equals(endpointName)) {
+ api.start(TwitterStreamAPI.SEARCH_ENDPOINT);
+ } else {
+ throw new ProcessException("Endpoint was invalid value: " + endpointName);
+ }
+ }
+
+ @Override
+ public void onTrigger(ProcessContext context, ProcessSession session) throws ProcessException {
+ if (messageQueue.isEmpty()) {
+ context.yield();
+ return;
+ }
+
+ FlowFile flowFile = session.create();
+ flowFile = session.write(flowFile, new OutputStreamCallback() {
+ @Override
+ public void process(OutputStream out) throws IOException {
+ out.write("[".getBytes(StandardCharsets.UTF_8));
+ final int batchSize = context.getProperty(BATCH_SIZE).asInteger();
+ String tweet = messageQueue.poll();
+ out.write(tweet.getBytes(StandardCharsets.UTF_8));
+ int tweetCount = 1;
+ while (tweetCount < batchSize && !messageQueue.isEmpty()) {
+ out.write(",".getBytes(StandardCharsets.UTF_8));
+ tweet = messageQueue.poll();
+ out.write(tweet.getBytes(StandardCharsets.UTF_8));
+ tweetCount++;
+ }
+ out.write("]".getBytes(StandardCharsets.UTF_8));
+ }
+ });
+
+ final Map<String, String> attributes = new HashMap<>();
+ attributes.put(CoreAttributes.MIME_TYPE.key(), "application/json");
+ attributes.put(CoreAttributes.FILENAME.key(), flowFile.getAttribute(CoreAttributes.FILENAME.key()) + ".json");
+ flowFile = session.putAllAttributes(flowFile, attributes);
+
+ session.transfer(flowFile, REL_SUCCESS);
+
+ String transitUri = api.getBasePath();
+ final String endpointName = context.getProperty(ENDPOINT).getValue();
+ if (ENDPOINT_SAMPLE.getValue().equals(endpointName)) {
+ transitUri += TwitterStreamAPI.SAMPLE_PATH;
+ } else if (ENDPOINT_SEARCH.getValue().equals(endpointName)) {
+ transitUri += TwitterStreamAPI.SEARCH_PATH;
+ } else {
+ throw new ProcessException("Endpoint was invalid value: " + endpointName);
+ }
Review Comment:
Implementing the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r848781151
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TwitterStreamAPI.java:
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+public class TwitterStreamAPI {
+ public static final String SEARCH_ENDPOINT = "Search Endpoint";
+ public static final String SAMPLE_ENDPOINT = "Sample Endpoint";
+ public static final String SAMPLE_PATH = "/2/tweets/sample/stream";
+ public static final String SEARCH_PATH = "/2/tweets/search/stream";
+
+ private static final String BEARER_TOKEN_PROPERTY_NAME = "bearer-token";
+ private static final String TWEET_FIELDS_PROPERTY_NAME = "tweet-fields";
+ private static final String USER_FIELDS_PROPERTY_NAME = "user-fields";
+ private static final String MEDIA_FIELDS_PROPERTY_NAME = "media-fields";
+ private static final String POLL_FIELDS_PROPERTY_NAME = "poll-fields";
+ private static final String PLACE_FIELDS_PROPERTY_NAME = "place-fields";
+ private static final String EXPANSIONS_PROPERTY_NAME = "expansions";
+ private static final String BACKFILL_MINUTES_PROPERTY_NAME = "backfill-minutes";
+
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Set<String> parseCommaSeparatedListPropreties(final ProcessContext context, final String propertyName) {
+ Set<String> fields = null;
+ if (context.getProperty(propertyName).isSet()) {
+ fields = new HashSet<>();
+ final String fieldsString = context.getProperty(propertyName).getValue();
+ for (final String field: fieldsString.split(",")) {
+ fields.add(field.trim());
+ }
+ }
+ return fields;
+ }
+
+ public TwitterStreamAPI(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ assert context != null;
+ assert queue != null;
+ assert logger != null;
+
+ this.queue = queue;
+ this.logger = logger;
+
+ this.tweetFields = parseCommaSeparatedListPropreties(context, TWEET_FIELDS_PROPERTY_NAME);
+ this.userFields = parseCommaSeparatedListPropreties(context, USER_FIELDS_PROPERTY_NAME);
+ this.mediaFields = parseCommaSeparatedListPropreties(context, MEDIA_FIELDS_PROPERTY_NAME);
+ this.pollFields = parseCommaSeparatedListPropreties(context, POLL_FIELDS_PROPERTY_NAME);
+ this.placeFields = parseCommaSeparatedListPropreties(context, PLACE_FIELDS_PROPERTY_NAME);
+ this.expansions = parseCommaSeparatedListPropreties(context, EXPANSIONS_PROPERTY_NAME);
+ this.backfillMinutes = context.getProperty(BACKFILL_MINUTES_PROPERTY_NAME).asInteger();
+
+ TwitterCredentialsBearer creds = new TwitterCredentialsBearer(context.getProperty(BEARER_TOKEN_PROPERTY_NAME).getValue());
+ api = new TwitterApi();
+ api.setTwitterCredentials(creds);
+
+ this.executorService = Executors.newSingleThreadExecutor();
+ }
+
+ public String getBasePath() {
+ return api.getApiClient().getBasePath();
+ }
+
+ public void setBasePath(final String path) {
+ api.getApiClient().setBasePath(path);
+ }
+
+ /**
+ * This method would be called when we would like the stream to get started. This method will spin off a thread that
+ * will continue to queue tweets on to the given queue passed in the constructor. The thread will continue
+ * to run until {@code stop} is called.
+ * @param endpoint {@code TwitterStreamAPI.SAMPLE_ENDPOINT} or {@code TwitterStreamAPI.SEARCH_ENDPOINT}
+ */
+ public void start(final String endpoint) {
+ try {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ stream = api.tweets().sampleStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else if (endpoint.equals(SEARCH_ENDPOINT)) {
+ stream = api.tweets().searchStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else {
+ throw new ProcessException("Endpoint was invalid value: " + endpoint);
+ }
+ } catch (ApiException e) {
+ logger.error("Received error {}: {}", e.getCode(), e.getMessage());
+ }
+
+ executorService.execute(new TweetQueuer());
+ }
+
+ /**
+ * This method would be called when we would like the stream to get stopped. The stream will be closed and the
+ * executorService will be shut down. If it fails to shutdown, then it will be forcefully terminated.
+ */
+ public void stop() {
+ try {
+ stream.close();
+ } catch (IOException e) {
+ logger.error("IOException occurred while closing stream: {}", e.getMessage());
+ }
+
+ executorService.shutdown();
+ try {
+ if (!executorService.awaitTermination(3, TimeUnit.SECONDS)) {
+ executorService.shutdownNow();
+ }
+ } catch (InterruptedException e) {
+ executorService.shutdownNow();
+ logger.error("InterruptedException occurred while waiting for executor service to shut down: {}", e.getMessage());
+ }
+ }
+
+ private class TweetQueuer implements Runnable {
+ @Override
+ public void run() {
+ try (BufferedReader reader = new BufferedReader(new InputStreamReader(stream))) {
+ String tweetLine = reader.readLine();
+ while (tweetLine != null) {
+ queue.offer(tweetLine);
+ try {
+ tweetLine = reader.readLine();
+ } catch (IOException e) {
+ logger.info("Stream closed, no more tweets to read, exiting Queuer");
+ break;
+ }
+ }
+ } catch (IOException e) {
+ logger.warn("IOException occurred in TweetQueuer: {}", e.getMessage());
Review Comment:
Making the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r848782578
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/pom.xml:
##########
@@ -69,5 +80,17 @@
<version>1.17.0-SNAPSHOT</version>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>com.squareup.okhttp3</groupId>
+ <artifactId>okhttp</artifactId>
+ <version>4.9.3</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>com.squareup.okhttp3</groupId>
+ <artifactId>mockwebserver</artifactId>
+ <version>4.9.3</version>
Review Comment:
Making the changes
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/pom.xml:
##########
@@ -69,5 +80,17 @@
<version>1.17.0-SNAPSHOT</version>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>com.squareup.okhttp3</groupId>
+ <artifactId>okhttp</artifactId>
+ <version>4.9.3</version>
Review Comment:
Making the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r864060229
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TweetStreamService.java:
##########
@@ -0,0 +1,222 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiClient;
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TweetStreamService {
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ScheduledExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Long backoffMultiplier;
+ private int backoffAttempts;
+ private int attemptCounter;
+ private Long backoffTime;
+ private Long maximumBackoff;
+
+ private StreamEndpoint endpoint;
+
+ public TweetStreamService(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ Objects.requireNonNull(context);
+ Objects.requireNonNull(queue);
+ Objects.requireNonNull(logger);
+
+ this.queue = queue;
+ this.logger = logger;
+
+ this.tweetFields = parseCommaSeparatedPropreties(context, ConsumeTwitter.TWEET_FIELDS);
+ this.userFields = parseCommaSeparatedPropreties(context, ConsumeTwitter.USER_FIELDS);
+ this.mediaFields = parseCommaSeparatedPropreties(context, ConsumeTwitter.MEDIA_FIELDS);
+ this.pollFields = parseCommaSeparatedPropreties(context, ConsumeTwitter.POLL_FIELDS);
+ this.placeFields = parseCommaSeparatedPropreties(context, ConsumeTwitter.PLACE_FIELDS);
+ this.expansions = parseCommaSeparatedPropreties(context, ConsumeTwitter.EXPANSIONS);
+ this.backfillMinutes = context.getProperty(ConsumeTwitter.BACKFILL_MINUTES).asInteger();
+
+ this.backoffMultiplier = 1L;
+ this.backoffAttempts = context.getProperty(ConsumeTwitter.BACKOFF_ATTEMPTS).asInteger();
+ this.attemptCounter = 0;
+ this.backoffTime = context.getProperty(ConsumeTwitter.BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+ this.maximumBackoff = context.getProperty(ConsumeTwitter.MAXIMUM_BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+
+ ApiClient client = new ApiClient();
+ final int connectTimeout = context.getProperty(ConsumeTwitter.CONNECT_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ final int readTimeout = context.getProperty(ConsumeTwitter.READ_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ client = client.setConnectTimeout(connectTimeout);
+ client = client.setReadTimeout(readTimeout);
+ api = new TwitterApi(client);
+
+ TwitterCredentialsBearer creds = new TwitterCredentialsBearer(context.getProperty(ConsumeTwitter.BEARER_TOKEN).getValue());
+ api.setTwitterCredentials(creds);
+
+ this.executorService = Executors.newSingleThreadScheduledExecutor();
+ }
+
+ public String getBasePath() {
+ return api.getApiClient().getBasePath();
+ }
+
+ public void setBasePath(final String path) {
+ api.getApiClient().setBasePath(path);
+ }
+
+ public String getTransitUri(final String endpoint) {
+ if (endpoint.equals(StreamEndpoint.SAMPLE_ENDPOINT.getEndpointName())) {
+ return api.getApiClient().getBasePath() + StreamEndpoint.SAMPLE_ENDPOINT.getPath();
+ } else if (endpoint.equals(StreamEndpoint.SEARCH_ENDPOINT.getEndpointName())) {
+ return api.getApiClient().getBasePath() + StreamEndpoint.SEARCH_ENDPOINT.getPath();
+ } else {
+ logger.warn("Unrecognized endpoint in getTransitUri. Returning basePath");
+ return api.getApiClient().getBasePath();
+ }
+ }
+
+ /**
+ * This method would be called when we would like the stream to get started. This method will spin off a thread that
+ * will continue to queue tweets on to the given queue passed in the constructor. The thread will continue
+ * to run until {@code stop} is called.
+ * @param endpoint {@code TwitterStreamAPI.SAMPLE_ENDPOINT} or {@code TwitterStreamAPI.SEARCH_ENDPOINT}
+ */
+ public void start(final StreamEndpoint endpoint) {
+ this.endpoint = endpoint;
+ executorService.execute(new TweetStreamStarter());
+ }
Review Comment:
Making the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r861177679
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/test/java/org/apache/nifi/processors/twitter/TestConsumeTwitter.java:
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import okhttp3.mockwebserver.MockResponse;
+import okhttp3.mockwebserver.MockWebServer;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+
+public class TestConsumeTwitter {
+ private MockWebServer mockWebServer;
+
+ private TestRunner runner;
+
+ private final String SAMPLE_TWEET = "{\"data\":{\"id\":\"123\",\"text\":\"This is a sample tweet and is not real!\"}}";
+
+ @BeforeEach
+ public void setRunnerAndAPI() {
+ mockWebServer = new MockWebServer();
+
+ runner = TestRunners.newTestRunner(ConsumeTwitter.class);
+
+ runner.setProperty(ConsumeTwitter.BEARER_TOKEN, "BEARER_TOKEN");
+ final String basePath = mockWebServer.url("").toString();
+ runner.setProperty(ConsumeTwitter.BASE_PATH, basePath);
+ }
+
+ @AfterEach
+ public void shutdownServerAndAPI() throws IOException {
+ mockWebServer.shutdown();
+ }
+
+ @Test
+ public void testReceiveSingleTweetInStream() throws InterruptedException {
+ MockResponse response = new MockResponse()
+ .setResponseCode(200)
+ .setBody(SAMPLE_TWEET)
+ .addHeader("Content-Type", "application/json");
+ mockWebServer.enqueue(response);
+
+
+ runner.setProperty(ConsumeTwitter.ENDPOINT, ConsumeTwitter.ENDPOINT_SAMPLE);
+ runner.setProperty(ConsumeTwitter.QUEUE_SIZE, "10000");
+ runner.setProperty(ConsumeTwitter.BATCH_SIZE, "10");
+ runner.setProperty(ConsumeTwitter.BACKFILL_MINUTES, "0");
+
+ runner.assertValid();
+
+ // the TwitterStreamAPI class spins up another thread and might not be done queueing tweets in one run of the processor
+ final int maxTries = 3;
+ final long runSchedule = 250;
+ runner.setRunSchedule(runSchedule);
+ runner.run(maxTries, false, true);
+ runner.stop();
+
+ // there should only be a single FlowFile containing a tweet
+ runner.assertTransferCount(ConsumeTwitter.REL_SUCCESS, 1);
+ MockFlowFile flowFile = runner.getFlowFilesForRelationship(ConsumeTwitter.REL_SUCCESS).get(0);
+ flowFile.assertContentEquals("[" + SAMPLE_TWEET + "]");
Review Comment:
Making the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r864073312
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TweetStreamService.java:
##########
@@ -0,0 +1,222 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiClient;
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TweetStreamService {
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ScheduledExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Long backoffMultiplier;
+ private int backoffAttempts;
+ private int attemptCounter;
+ private Long backoffTime;
+ private Long maximumBackoff;
+
+ private StreamEndpoint endpoint;
+
+ public TweetStreamService(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ Objects.requireNonNull(context);
+ Objects.requireNonNull(queue);
+ Objects.requireNonNull(logger);
+
+ this.queue = queue;
+ this.logger = logger;
+
+ this.tweetFields = parseCommaSeparatedPropreties(context, ConsumeTwitter.TWEET_FIELDS);
+ this.userFields = parseCommaSeparatedPropreties(context, ConsumeTwitter.USER_FIELDS);
+ this.mediaFields = parseCommaSeparatedPropreties(context, ConsumeTwitter.MEDIA_FIELDS);
+ this.pollFields = parseCommaSeparatedPropreties(context, ConsumeTwitter.POLL_FIELDS);
+ this.placeFields = parseCommaSeparatedPropreties(context, ConsumeTwitter.PLACE_FIELDS);
+ this.expansions = parseCommaSeparatedPropreties(context, ConsumeTwitter.EXPANSIONS);
+ this.backfillMinutes = context.getProperty(ConsumeTwitter.BACKFILL_MINUTES).asInteger();
+
+ this.backoffMultiplier = 1L;
+ this.backoffAttempts = context.getProperty(ConsumeTwitter.BACKOFF_ATTEMPTS).asInteger();
+ this.attemptCounter = 0;
+ this.backoffTime = context.getProperty(ConsumeTwitter.BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+ this.maximumBackoff = context.getProperty(ConsumeTwitter.MAXIMUM_BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+
+ ApiClient client = new ApiClient();
+ final int connectTimeout = context.getProperty(ConsumeTwitter.CONNECT_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ final int readTimeout = context.getProperty(ConsumeTwitter.READ_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ client = client.setConnectTimeout(connectTimeout);
+ client = client.setReadTimeout(readTimeout);
+ api = new TwitterApi(client);
+
+ TwitterCredentialsBearer creds = new TwitterCredentialsBearer(context.getProperty(ConsumeTwitter.BEARER_TOKEN).getValue());
+ api.setTwitterCredentials(creds);
+
+ this.executorService = Executors.newSingleThreadScheduledExecutor();
+ }
+
+ public String getBasePath() {
+ return api.getApiClient().getBasePath();
+ }
+
+ public void setBasePath(final String path) {
+ api.getApiClient().setBasePath(path);
+ }
+
+ public String getTransitUri(final String endpoint) {
+ if (endpoint.equals(StreamEndpoint.SAMPLE_ENDPOINT.getEndpointName())) {
+ return api.getApiClient().getBasePath() + StreamEndpoint.SAMPLE_ENDPOINT.getPath();
+ } else if (endpoint.equals(StreamEndpoint.SEARCH_ENDPOINT.getEndpointName())) {
+ return api.getApiClient().getBasePath() + StreamEndpoint.SEARCH_ENDPOINT.getPath();
+ } else {
+ logger.warn("Unrecognized endpoint in getTransitUri. Returning basePath");
+ return api.getApiClient().getBasePath();
+ }
+ }
+
+ /**
+ * This method would be called when we would like the stream to get started. This method will spin off a thread that
+ * will continue to queue tweets on to the given queue passed in the constructor. The thread will continue
+ * to run until {@code stop} is called.
+ * @param endpoint {@code TwitterStreamAPI.SAMPLE_ENDPOINT} or {@code TwitterStreamAPI.SEARCH_ENDPOINT}
+ */
+ public void start(final StreamEndpoint endpoint) {
+ this.endpoint = endpoint;
+ executorService.execute(new TweetStreamStarter());
+ }
+
+ /**
+ * This method would be called when we would like the stream to get stopped. The stream will be closed and the
+ * executorService will be shut down.
+ */
+ public void stop() {
+ if (stream != null) {
+ try {
+ stream.close();
+ } catch (IOException e) {
+ logger.error("Closing response stream failed", e);
+ }
+ }
+
+ executorService.shutdownNow();
+ }
+
+ private Long calculateBackoffDelay() {
+ long backoff = backoffMultiplier * backoffTime;
+ long delay = (backoff > maximumBackoff) ? maximumBackoff : backoff;
+ return delay;
+ }
+
+ private void scheduleStartStreamWithBackoff() {
+ // use exponential(by factor of 2) backoff in scheduling the next TweetStreamStarter
+ if (attemptCounter >= backoffAttempts) {
+ throw new ProcessException(String.format("Connection failed after maximum attempts [%d]", attemptCounter));
+ }
+ attemptCounter += 1;
+ long delay = calculateBackoffDelay();
+ backoffMultiplier *= 2;
+ logger.info("Scheduling new stream connection after delay [{} s]", delay);
+ executorService.schedule(new TweetStreamStarter(), delay, TimeUnit.SECONDS);
+ }
+
+ private void resetBackoff() {
+ attemptCounter = 0;
+ backoffMultiplier = 1L;
+ }
+
+ private class TweetStreamStarter implements Runnable {
+ @Override
+ public void run() {
+ try {
+ if (endpoint.equals(StreamEndpoint.SAMPLE_ENDPOINT)) {
+ stream = api.tweets().sampleStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else {
+ stream = api.tweets().searchStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ }
+ executorService.execute(new TweetStreamHandler());
+ } catch (ApiException e) {
+ stream = null;
+ logger.warn("Twitter Stream [{}] API connection failed: HTTP {}", endpoint.getEndpointName(), e.getCode(), e);
+ scheduleStartStreamWithBackoff();
+ } catch (Exception e) {
+ stream = null;
+ logger.warn("Twitter Stream [{}] connection failed", endpoint.getEndpointName(), e);
+ scheduleStartStreamWithBackoff();
+ }
+ }
+ }
+
+ private class TweetStreamHandler implements Runnable {
+ @Override
+ public void run() {
+ try (BufferedReader reader = new BufferedReader(new InputStreamReader(stream))) {
+ String tweetRecord = reader.readLine();
+ while (tweetRecord != null) {
+ queue.offer(tweetRecord);
+
+ // reset backoff multiplier upon successful receipt of a tweet
+ resetBackoff();
+
+ tweetRecord = reader.readLine();
+ }
+ } catch (IOException e) {
+ logger.info("Stream is closed or has stopped", e);
+ }
+ logger.info("Tweet stream processing completed");
+ scheduleStartStreamWithBackoff();
+ }
+ }
+
+ private Set<String> parseCommaSeparatedPropreties(final ProcessContext context, final PropertyDescriptor property) {
Review Comment:
Thanks for the catch, making the change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r864073912
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/test/java/org/apache/nifi/processors/twitter/TestConsumeTwitter.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import okhttp3.mockwebserver.MockResponse;
+import okhttp3.mockwebserver.MockWebServer;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+
+public class TestConsumeTwitter {
+ private MockWebServer mockWebServer;
+
+ private TestRunner runner;
+
+ private final String SAMPLE_TWEET = "{\"data\":{\"id\":\"123\",\"text\":\"This is a sample tweet and is not real!\"}}";
+ private final String EXPECTED_TWEET = "[{\"data\":{\"id\":\"123\",\"text\":\"This is a sample tweet and is not real!\"}}]";
+
+ @BeforeEach
+ public void setRunnerAndAPI() {
+ mockWebServer = new MockWebServer();
+
+ runner = TestRunners.newTestRunner(ConsumeTwitter.class);
+
+ runner.setProperty(ConsumeTwitter.BEARER_TOKEN, "BEARER_TOKEN");
+ final String basePath = mockWebServer.url("").toString();
+ runner.setProperty(ConsumeTwitter.BASE_PATH, basePath);
+ }
+
+ @AfterEach
+ public void shutdownServerAndAPI() throws IOException {
+ mockWebServer.shutdown();
+ }
+
+ @Test
+ public void testReceiveSingleTweetInStream() throws InterruptedException {
+ MockResponse response = new MockResponse()
+ .setResponseCode(200)
+ .setBody(SAMPLE_TWEET)
+ .addHeader("Content-Type", "application/json");
+ mockWebServer.enqueue(response);
+
+
+ runner.setProperty(ConsumeTwitter.ENDPOINT, ConsumeTwitter.ENDPOINT_SAMPLE);
+ runner.setProperty(ConsumeTwitter.QUEUE_SIZE, "10000");
+ runner.setProperty(ConsumeTwitter.BATCH_SIZE, "10");
+ runner.setProperty(ConsumeTwitter.BACKFILL_MINUTES, "0");
+
+ runner.assertValid();
+
+ // the TwitterStreamAPI class spins up another thread and might not be done queueing tweets in one run of the processor
+ final int maxTries = 3;
+ final long runSchedule = 250;
+ runner.setRunSchedule(runSchedule);
+ runner.run(maxTries, false, true);
+ runner.stop();
+
+ // there should only be a single FlowFile containing a tweet
+ runner.assertTransferCount(ConsumeTwitter.REL_SUCCESS, 1);
+ MockFlowFile flowFile = runner.getFlowFilesForRelationship(ConsumeTwitter.REL_SUCCESS).get(0);
+ flowFile.assertContentEquals(EXPECTED_TWEET);
+ flowFile.assertAttributeEquals(CoreAttributes.MIME_TYPE.key(), "application/json");
Review Comment:
Making the change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r861124727
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.UUID;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("consume-twitter-endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("Maximum size of internal queue for streamed messages")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor BACKOFF_ATTEMPTS = new PropertyDescriptor.Builder()
+ .name("backoff-attempts")
+ .displayName("Backoff Attempts")
+ .description("The number of reconnection tries the processor will attempt in the event of " +
+ "a disconnection of the stream for any reason, before throwing an exception. To start a stream after " +
+ "this exception occur and the connection is fixed, please stop and restart the processor. If the value" +
+ "of this property is 0, then backoff will never occur and the processor will always need to be restarted" +
+ "if the stream fails.")
+ .required(true)
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .defaultValue("5")
+ .build();
+ public static final PropertyDescriptor BACKOFF_TIME = new PropertyDescriptor.Builder()
+ .name("backoff-time")
+ .displayName("Backoff Time")
+ .description("The duration to backoff to start attempting a new stream if" +
+ "the current one fails for any reason. Will increase by factor of 2 every time a restart fails")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("1 mins")
+ .build();
+ public static final PropertyDescriptor MAXIMUM_BACKOFF_TIME = new PropertyDescriptor.Builder()
+ .name("maximum-backoff-time")
+ .displayName("Maximum Backoff Time")
+ .description("The maximum duration to backoff to start attempting a new stream." +
+ "It is recommended that this number be much higher than the 'Backoff Time' property")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("5 mins")
+ .build();
+ public static final PropertyDescriptor CONNECT_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("connect-timeout")
+ .displayName("Connect Timeout")
+ .description("The maximum time in which client should establish a connection with the " +
+ "Twitter API before a time out. A value of 0 will mean no timeout at all")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("10 secs")
+ .build();
+ public static final PropertyDescriptor READ_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("read-timeout")
+ .displayName("Read Timeout")
+ .description("The maximum time of inactivity between receiving tweets from Twitter through " +
+ "the API before a timeout. A value of 0 will mean no timeout at all")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("10 secs")
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("tweet-fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values include: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("user-fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user " +
+ "for proper usage. Possible field values include: " +
+ "created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, " +
+ "protected, public_metrics, url, username, verified, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor MEDIA_FIELDS = new PropertyDescriptor.Builder()
+ .name("media-fields")
+ .displayName("Media Fields")
+ .description("A comma-separated list of media fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/media " +
+ "for proper usage. Possible field values include: " +
+ "alt_text, duration_ms, height, media_key, non_public_metrics, organic_metrics, preview_image_url, " +
+ "promoted_metrics, public_metrics, type, url, width")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor POLL_FIELDS = new PropertyDescriptor.Builder()
+ .name("poll-fields")
+ .displayName("Poll Fields")
+ .description("A comma-separated list of poll fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/poll " +
+ "for proper usage. Possible field values include: " +
+ "duration_minutes, end_datetime, id, options, voting_status")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PLACE_FIELDS = new PropertyDescriptor.Builder()
+ .name("place-fields")
+ .displayName("Place Fields")
+ .description("A comma-separated list of place fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/place " +
+ "for proper usage. Possible field values include: " +
+ "contained_within, country, country_code, full_name, geo, id, name, place_type")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor EXPANSIONS = new PropertyDescriptor.Builder()
+ .name("expansions")
+ .displayName("Expansions")
+ .description("A comma-separated list of expansions for objects in the returned tweet. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/expansions " +
+ "for proper usage. Possible field values include: " +
+ "author_id, referenced_tweets.id, referenced_tweets.id.author_id, entities.mentions.username, " +
+ "attachments.poll_ids, attachments.media_keys ,in_reply_to_user_id, geo.place_id")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor BACKFILL_MINUTES = new PropertyDescriptor.Builder()
+ .name("backfill-minutes")
+ .displayName("Backfill Minutes")
+ .description("The number of minutes (up to 5 minutes) of streaming data to be requested after a " +
+ "disconnect. Only available for project with academic research access. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/integrate/" +
+ "recovery-and-redundancy-features")
+ .required(true)
+ .defaultValue("0")
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("FlowFiles containing an array of one or more Tweets")
+ .build();
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+
+ private TweetStreamService tweetStreamService;
+
+ private volatile BlockingQueue<String> messageQueue;
+
+ @Override
+ protected void init(ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptors = new ArrayList<>();
+ descriptors.add(ENDPOINT);
+ descriptors.add(BASE_PATH);
+ descriptors.add(BEARER_TOKEN);
+ descriptors.add(QUEUE_SIZE);
+ descriptors.add(BATCH_SIZE);
+ descriptors.add(BACKOFF_ATTEMPTS);
+ descriptors.add(BACKOFF_TIME);
+ descriptors.add(MAXIMUM_BACKOFF_TIME);
+ descriptors.add(CONNECT_TIMEOUT);
+ descriptors.add(READ_TIMEOUT);
+ descriptors.add(TWEET_FIELDS);
+ descriptors.add(USER_FIELDS);
+ descriptors.add(MEDIA_FIELDS);
+ descriptors.add(POLL_FIELDS);
+ descriptors.add(PLACE_FIELDS);
+ descriptors.add(EXPANSIONS);
+ descriptors.add(BACKFILL_MINUTES);
+
+ this.descriptors = Collections.unmodifiableList(descriptors);
+
+ final Set<Relationship> relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ this.relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return this.descriptors;
+ }
+
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ messageQueue = new LinkedBlockingQueue<>(context.getProperty(QUEUE_SIZE).asInteger());
+
+ tweetStreamService = new TweetStreamService(context, messageQueue, getLogger());
+ tweetStreamService.setBasePath(context.getProperty(BASE_PATH).getValue());
+ final String endpointName = context.getProperty(ENDPOINT).getValue();
+ if (ENDPOINT_SAMPLE.getValue().equals(endpointName)) {
+ tweetStreamService.start(TweetStreamService.SAMPLE_ENDPOINT);
+ } else {
+ tweetStreamService.start(TweetStreamService.SEARCH_ENDPOINT);
+ }
+ }
+
+ @Override
+ public void onTrigger(ProcessContext context, ProcessSession session) throws ProcessException {
+ if (messageQueue.isEmpty()) {
+ context.yield();
+ return;
+ }
+
+ FlowFile flowFile = session.create();
+ flowFile = session.write(flowFile, new OutputStreamCallback() {
+ @Override
+ public void process(OutputStream out) throws IOException {
+ out.write('[');
+ final int batchSize = context.getProperty(BATCH_SIZE).asInteger();
+ String tweet = messageQueue.poll();
+ out.write(tweet.getBytes(StandardCharsets.UTF_8));
+ int tweetCount = 1;
Review Comment:
Thanks for the suggestion, making the changes!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r861125458
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.UUID;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
Review Comment:
Making the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r861122587
##########
nifi-assembly/NOTICE:
##########
@@ -964,6 +964,11 @@ The following binary components are provided under the Apache Software License v
The following NOTICE information applies:
Hosebird Client (hbc)
Copyright 2013 Twitter, Inc.
+
+ (ASLv2) Twitter API Client Library for Java
+ The following NOTICE information applies:
+ Twitter API Client Library for Java
+ Copyright 2022 Twitter, Inc.
Review Comment:
Making the changes
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-social-media-nar/src/main/resources/META-INF/NOTICE:
##########
@@ -44,3 +44,21 @@ The following binary components are provided under the Apache Software License v
The following NOTICE information applies:
Hosebird Client (hbc)
Copyright 2013 Twitter, Inc.
+
+ (ASLv2) Twitter API Client Library for Java
+ The following NOTICE information applies:
+ Twitter API Client Library for Java
+ Copyright 2022 Twitter, Inc.
Review Comment:
Making the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r846715635
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,316 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("Twitter Endpoint")
Review Comment:
Noted, thanks for the comments, will implement this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r850994796
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TwitterStreamAPI.java:
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+public class TwitterStreamAPI {
+ public static final String SEARCH_ENDPOINT = "Search Endpoint";
+ public static final String SAMPLE_ENDPOINT = "Sample Endpoint";
+ public static final String SAMPLE_PATH = "/2/tweets/sample/stream";
+ public static final String SEARCH_PATH = "/2/tweets/search/stream";
+
+ private static final String BEARER_TOKEN_PROPERTY_NAME = "bearer-token";
+ private static final String TWEET_FIELDS_PROPERTY_NAME = "tweet-fields";
+ private static final String USER_FIELDS_PROPERTY_NAME = "user-fields";
+ private static final String MEDIA_FIELDS_PROPERTY_NAME = "media-fields";
+ private static final String POLL_FIELDS_PROPERTY_NAME = "poll-fields";
+ private static final String PLACE_FIELDS_PROPERTY_NAME = "place-fields";
+ private static final String EXPANSIONS_PROPERTY_NAME = "expansions";
+ private static final String BACKFILL_MINUTES_PROPERTY_NAME = "backfill-minutes";
+
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Set<String> parseCommaSeparatedListPropreties(final ProcessContext context, final String propertyName) {
+ Set<String> fields = null;
+ if (context.getProperty(propertyName).isSet()) {
+ fields = new HashSet<>();
+ final String fieldsString = context.getProperty(propertyName).getValue();
+ for (final String field: fieldsString.split(",")) {
+ fields.add(field.trim());
+ }
+ }
+ return fields;
+ }
+
+ public TwitterStreamAPI(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ assert context != null;
+ assert queue != null;
+ assert logger != null;
+
+ this.queue = queue;
+ this.logger = logger;
+
+ this.tweetFields = parseCommaSeparatedListPropreties(context, TWEET_FIELDS_PROPERTY_NAME);
+ this.userFields = parseCommaSeparatedListPropreties(context, USER_FIELDS_PROPERTY_NAME);
+ this.mediaFields = parseCommaSeparatedListPropreties(context, MEDIA_FIELDS_PROPERTY_NAME);
+ this.pollFields = parseCommaSeparatedListPropreties(context, POLL_FIELDS_PROPERTY_NAME);
+ this.placeFields = parseCommaSeparatedListPropreties(context, PLACE_FIELDS_PROPERTY_NAME);
+ this.expansions = parseCommaSeparatedListPropreties(context, EXPANSIONS_PROPERTY_NAME);
+ this.backfillMinutes = context.getProperty(BACKFILL_MINUTES_PROPERTY_NAME).asInteger();
+
+ TwitterCredentialsBearer creds = new TwitterCredentialsBearer(context.getProperty(BEARER_TOKEN_PROPERTY_NAME).getValue());
+ api = new TwitterApi();
+ api.setTwitterCredentials(creds);
+
+ this.executorService = Executors.newSingleThreadExecutor();
+ }
+
+ public String getBasePath() {
+ return api.getApiClient().getBasePath();
+ }
+
+ public void setBasePath(final String path) {
+ api.getApiClient().setBasePath(path);
+ }
+
+ /**
+ * This method would be called when we would like the stream to get started. This method will spin off a thread that
+ * will continue to queue tweets on to the given queue passed in the constructor. The thread will continue
+ * to run until {@code stop} is called.
+ * @param endpoint {@code TwitterStreamAPI.SAMPLE_ENDPOINT} or {@code TwitterStreamAPI.SEARCH_ENDPOINT}
+ */
+ public void start(final String endpoint) {
+ try {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ stream = api.tweets().sampleStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else if (endpoint.equals(SEARCH_ENDPOINT)) {
+ stream = api.tweets().searchStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else {
+ throw new ProcessException("Endpoint was invalid value: " + endpoint);
+ }
+ } catch (ApiException e) {
+ logger.error("Received error {}: {}", e.getCode(), e.getMessage());
+ }
+
+ executorService.execute(new TweetQueuer());
+ }
+
+ /**
+ * This method would be called when we would like the stream to get stopped. The stream will be closed and the
+ * executorService will be shut down. If it fails to shutdown, then it will be forcefully terminated.
+ */
+ public void stop() {
+ try {
+ stream.close();
+ } catch (IOException e) {
+ logger.error("IOException occurred while closing stream: {}", e.getMessage());
+ }
+
+ executorService.shutdown();
+ try {
+ if (!executorService.awaitTermination(3, TimeUnit.SECONDS)) {
+ executorService.shutdownNow();
+ }
+ } catch (InterruptedException e) {
+ executorService.shutdownNow();
+ logger.error("InterruptedException occurred while waiting for executor service to shut down: {}", e.getMessage());
+ }
+ }
+
+ private class TweetQueuer implements Runnable {
+ @Override
+ public void run() {
+ try (BufferedReader reader = new BufferedReader(new InputStreamReader(stream))) {
+ String tweetLine = reader.readLine();
+ while (tweetLine != null) {
Review Comment:
Implementing and testing a backoff-based stream "restarter" if the current tweet stream closes or returns nothing for any reason. Stream may also return nothing if we do a `searchStream` with rules that are weird.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r861150207
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TweetStreamService.java:
##########
@@ -0,0 +1,242 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiClient;
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TweetStreamService {
+ public static final String SEARCH_ENDPOINT = "Search Endpoint";
+ public static final String SAMPLE_ENDPOINT = "Sample Endpoint";
+ private static final String SAMPLE_PATH = "/2/tweets/sample/stream";
+ private static final String SEARCH_PATH = "/2/tweets/search/stream";
+
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ScheduledExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Long backoffMultiplier;
+ private int backoffAttempts;
+ private int attemptCounter;
+ private Long backoffTime;
+ private Long maximumBackoff;
+
+ private String endpoint;
+
+ private Set<String> parseCommaSeparatedListPropreties(final ProcessContext context, final PropertyDescriptor property) {
+ Set<String> fields = null;
+ if (context.getProperty(property).isSet()) {
+ fields = new HashSet<>();
+ final String fieldsString = context.getProperty(property).getValue();
+ for (final String field: fieldsString.split(",")) {
+ fields.add(field.trim());
+ }
+ }
+ return fields;
+ }
+
+ public TweetStreamService(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ assert context != null;
+ assert queue != null;
+ assert logger != null;
+
+ this.queue = queue;
+ this.logger = logger;
+
+ this.tweetFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.TWEET_FIELDS);
+ this.userFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.USER_FIELDS);
+ this.mediaFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.MEDIA_FIELDS);
+ this.pollFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.POLL_FIELDS);
+ this.placeFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.PLACE_FIELDS);
+ this.expansions = parseCommaSeparatedListPropreties(context, ConsumeTwitter.EXPANSIONS);
+ this.backfillMinutes = context.getProperty(ConsumeTwitter.BACKFILL_MINUTES).asInteger();
+
+ this.backoffMultiplier = 1L;
+ this.backoffAttempts = context.getProperty(ConsumeTwitter.BACKOFF_ATTEMPTS).asInteger();
+ this.attemptCounter = 0;
+ this.backoffTime = context.getProperty(ConsumeTwitter.BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+ this.maximumBackoff = context.getProperty(ConsumeTwitter.MAXIMUM_BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+
+ ApiClient client = new ApiClient();
+ final int connectTimeout = context.getProperty(ConsumeTwitter.CONNECT_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ final int readTimeout = context.getProperty(ConsumeTwitter.READ_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ client = client.setConnectTimeout(connectTimeout);
+ client = client.setReadTimeout(readTimeout);
+ api = new TwitterApi(client);
+
+ TwitterCredentialsBearer creds = new TwitterCredentialsBearer(context.getProperty(ConsumeTwitter.BEARER_TOKEN).getValue());
+ api.setTwitterCredentials(creds);
+
+ this.executorService = Executors.newSingleThreadScheduledExecutor();
+ }
+
+ public String getBasePath() {
+ return api.getApiClient().getBasePath();
+ }
+
+ public void setBasePath(final String path) {
+ api.getApiClient().setBasePath(path);
+ }
+
+ public String getTransitUri(final String endpoint) {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ return api.getApiClient().getBasePath() + SAMPLE_PATH;
+ } else if (endpoint.equals(SEARCH_ENDPOINT)) {
+ return api.getApiClient().getBasePath() + SEARCH_PATH;
+ } else {
+ logger.warn("Unrecognized endpoint in getTransitUri. Returning basePath");
+ return api.getApiClient().getBasePath();
+ }
+ }
+
+ /**
+ * This method would be called when we would like the stream to get started. This method will spin off a thread that
+ * will continue to queue tweets on to the given queue passed in the constructor. The thread will continue
+ * to run until {@code stop} is called.
+ * @param endpoint {@code TwitterStreamAPI.SAMPLE_ENDPOINT} or {@code TwitterStreamAPI.SEARCH_ENDPOINT}
+ */
+ public void start(final String endpoint) {
+ this.endpoint = endpoint;
+ executorService.execute(new TweetStreamStarter());
+ }
+
+ /**
+ * This method would be called when we would like the stream to get stopped. The stream will be closed and the
+ * executorService will be shut down.
+ */
+ public void stop() {
+ if (stream != null) {
+ try {
+ stream.close();
+ } catch (IOException e) {
+ logger.error("Closing response stream failed", e);
+ }
+ }
+
+ executorService.shutdownNow();
+ }
+
+ private Long calculateBackoffDelay() {
+ long backoff = backoffMultiplier * backoffTime;
+ long delay = (backoff > maximumBackoff) ? maximumBackoff : backoff;
+ return delay;
+ }
+
+ private void scheduleStartStreamWithBackoff() {
+ // use exponential(by factor of 2) backoff in scheduling the next TweetStreamStarter
+ if (attemptCounter >= backoffAttempts) {
+ final String msg = "Reached maximum attempts to backoff";
+ logger.error(msg);
+ throw new ProcessException(msg);
+ }
+ attemptCounter += 1;
+ long delay = calculateBackoffDelay();
+ backoffMultiplier *= 2;
+ logger.info("Scheduling new TweetStreamStarter after {}s delay", delay);
+ executorService.schedule(new TweetStreamStarter(), delay, TimeUnit.SECONDS);
+ }
+
+ private void resetBackoff() {
+ attemptCounter = 0;
+ backoffMultiplier = 1L;
+ }
+
+ private class TweetStreamStarter implements Runnable {
+ @Override
+ public void run() {
+ try {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ stream = api.tweets().sampleStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else {
+ stream = api.tweets().searchStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ }
+ } catch (ApiException e) {
+ stream = null;
+ logger.warn(String.format("Received ApiException, error %d: %s", e.getCode(), e.getMessage()));
Review Comment:
Making the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] MikeThomsen commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
MikeThomsen commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r848413880
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TwitterStreamAPI.java:
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+public class TwitterStreamAPI {
+ public static final String SEARCH_ENDPOINT = "Search Endpoint";
+ public static final String SAMPLE_ENDPOINT = "Sample Endpoint";
+ public static final String SAMPLE_PATH = "/2/tweets/sample/stream";
+ public static final String SEARCH_PATH = "/2/tweets/search/stream";
+
+ private static final String BEARER_TOKEN_PROPERTY_NAME = "bearer-token";
+ private static final String TWEET_FIELDS_PROPERTY_NAME = "tweet-fields";
+ private static final String USER_FIELDS_PROPERTY_NAME = "user-fields";
+ private static final String MEDIA_FIELDS_PROPERTY_NAME = "media-fields";
+ private static final String POLL_FIELDS_PROPERTY_NAME = "poll-fields";
+ private static final String PLACE_FIELDS_PROPERTY_NAME = "place-fields";
+ private static final String EXPANSIONS_PROPERTY_NAME = "expansions";
+ private static final String BACKFILL_MINUTES_PROPERTY_NAME = "backfill-minutes";
+
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Set<String> parseCommaSeparatedListPropreties(final ProcessContext context, final String propertyName) {
+ Set<String> fields = null;
+ if (context.getProperty(propertyName).isSet()) {
+ fields = new HashSet<>();
+ final String fieldsString = context.getProperty(propertyName).getValue();
+ for (final String field: fieldsString.split(",")) {
+ fields.add(field.trim());
+ }
+ }
+ return fields;
+ }
+
+ public TwitterStreamAPI(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ assert context != null;
+ assert queue != null;
+ assert logger != null;
+
+ this.queue = queue;
+ this.logger = logger;
+
+ this.tweetFields = parseCommaSeparatedListPropreties(context, TWEET_FIELDS_PROPERTY_NAME);
+ this.userFields = parseCommaSeparatedListPropreties(context, USER_FIELDS_PROPERTY_NAME);
+ this.mediaFields = parseCommaSeparatedListPropreties(context, MEDIA_FIELDS_PROPERTY_NAME);
+ this.pollFields = parseCommaSeparatedListPropreties(context, POLL_FIELDS_PROPERTY_NAME);
+ this.placeFields = parseCommaSeparatedListPropreties(context, PLACE_FIELDS_PROPERTY_NAME);
+ this.expansions = parseCommaSeparatedListPropreties(context, EXPANSIONS_PROPERTY_NAME);
+ this.backfillMinutes = context.getProperty(BACKFILL_MINUTES_PROPERTY_NAME).asInteger();
+
+ TwitterCredentialsBearer creds = new TwitterCredentialsBearer(context.getProperty(BEARER_TOKEN_PROPERTY_NAME).getValue());
+ api = new TwitterApi();
+ api.setTwitterCredentials(creds);
+
+ this.executorService = Executors.newSingleThreadExecutor();
+ }
+
+ public String getBasePath() {
+ return api.getApiClient().getBasePath();
+ }
+
+ public void setBasePath(final String path) {
+ api.getApiClient().setBasePath(path);
+ }
+
+ /**
+ * This method would be called when we would like the stream to get started. This method will spin off a thread that
+ * will continue to queue tweets on to the given queue passed in the constructor. The thread will continue
+ * to run until {@code stop} is called.
+ * @param endpoint {@code TwitterStreamAPI.SAMPLE_ENDPOINT} or {@code TwitterStreamAPI.SEARCH_ENDPOINT}
+ */
+ public void start(final String endpoint) {
+ try {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ stream = api.tweets().sampleStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else if (endpoint.equals(SEARCH_ENDPOINT)) {
+ stream = api.tweets().searchStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else {
+ throw new ProcessException("Endpoint was invalid value: " + endpoint);
+ }
+ } catch (ApiException e) {
+ logger.error("Received error {}: {}", e.getCode(), e.getMessage());
+ }
+
+ executorService.execute(new TweetQueuer());
+ }
+
+ /**
+ * This method would be called when we would like the stream to get stopped. The stream will be closed and the
+ * executorService will be shut down. If it fails to shutdown, then it will be forcefully terminated.
+ */
+ public void stop() {
+ try {
+ stream.close();
+ } catch (IOException e) {
+ logger.error("IOException occurred while closing stream: {}", e.getMessage());
+ }
+
+ executorService.shutdown();
+ try {
+ if (!executorService.awaitTermination(3, TimeUnit.SECONDS)) {
+ executorService.shutdownNow();
+ }
+ } catch (InterruptedException e) {
+ executorService.shutdownNow();
+ logger.error("InterruptedException occurred while waiting for executor service to shut down: {}", e.getMessage());
+ }
+ }
+
+ private class TweetQueuer implements Runnable {
+ @Override
+ public void run() {
+ try (BufferedReader reader = new BufferedReader(new InputStreamReader(stream))) {
Review Comment:
Ran into a SSL handshake exception here while trying out on my work laptop. Happed in two places, so I'm confident it was a MitM attack by corporate security, so NBD. That said, I think expanding this catch to an Exception and wrapping/rethrowing as a `ProcessException` is the right call.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
exceptionfactory commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r848443321
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/pom.xml:
##########
@@ -69,5 +80,17 @@
<version>1.17.0-SNAPSHOT</version>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>com.squareup.okhttp3</groupId>
+ <artifactId>okhttp</artifactId>
+ <version>4.9.3</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>com.squareup.okhttp3</groupId>
+ <artifactId>mockwebserver</artifactId>
+ <version>4.9.3</version>
Review Comment:
This version number can be removed since the root Maven configuration specifies the version.
##########
nifi-assembly/NOTICE:
##########
@@ -964,6 +964,11 @@ The following binary components are provided under the Apache Software License v
The following NOTICE information applies:
Hosebird Client (hbc)
Copyright 2013 Twitter, Inc.
+
+ (ASLv2) Twitter API Client Library for Java
Review Comment:
It looks like the Twitter API Client Library includes a number of transitive dependencies, some of which are new to NiFi, so it looks like there are a few additions necessary. In particular the Scribe APIs and Apache Oltu libraries are new. A thorough check of the transitive dependencies is necessary to confirm whether there are any other changes necessary to the license and notice files.
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,318 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("consume-twitter-endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("The size of the queue used to queue tweets")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("tweet-fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values are: " +
Review Comment:
Recommend adjusting the wording slightly since this could change, but it is helpful to have these examples:
```suggestion
"for proper usage. Possible field values include: " +
```
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/pom.xml:
##########
@@ -69,5 +80,17 @@
<version>1.17.0-SNAPSHOT</version>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>com.squareup.okhttp3</groupId>
+ <artifactId>okhttp</artifactId>
+ <version>4.9.3</version>
Review Comment:
This version number can be removed since the root Maven configuration specifies the version.
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,318 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("consume-twitter-endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("The size of the queue used to queue tweets")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("tweet-fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values are: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("user-fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user " +
+ "for proper usage. Possible field values are: " +
Review Comment:
```suggestion
"for proper usage. Possible field values include: " +
```
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,318 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("consume-twitter-endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("The size of the queue used to queue tweets")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("tweet-fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values are: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("user-fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user " +
+ "for proper usage. Possible field values are: " +
+ "created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, " +
+ "protected, public_metrics, url, username, verified, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor MEDIA_FIELDS = new PropertyDescriptor.Builder()
+ .name("media-fields")
+ .displayName("Media Fields")
+ .description("A comma-separated list of media fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/media " +
+ "for proper usage. Possible field values are: " +
Review Comment:
```suggestion
"for proper usage. Possible field values include: " +
```
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,318 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("consume-twitter-endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("The size of the queue used to queue tweets")
Review Comment:
Recommend expanding the description for better clarity:
```suggestion
.description("Maximum size of internal queue for streamed messages")
```
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,318 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("consume-twitter-endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("The size of the queue used to queue tweets")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("tweet-fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values are: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("user-fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user " +
+ "for proper usage. Possible field values are: " +
+ "created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, " +
+ "protected, public_metrics, url, username, verified, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor MEDIA_FIELDS = new PropertyDescriptor.Builder()
+ .name("media-fields")
+ .displayName("Media Fields")
+ .description("A comma-separated list of media fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/media " +
+ "for proper usage. Possible field values are: " +
+ "alt_text, duration_ms, height, media_key, non_public_metrics, organic_metrics, preview_image_url, " +
+ "promoted_metrics, public_metrics, type, url, width")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor POLL_FIELDS = new PropertyDescriptor.Builder()
+ .name("poll-fields")
+ .displayName("Poll Fields")
+ .description("A comma-separated list of poll fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/poll " +
+ "for proper usage. Possible field values are: " +
+ "duration_minutes, end_datetime, id, options, voting_status")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PLACE_FIELDS = new PropertyDescriptor.Builder()
+ .name("place-fields")
+ .displayName("Place Fields")
+ .description("A comma-separated list of place fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/place " +
+ "for proper usage. Possible field values are: " +
+ "contained_within, country, country_code, full_name, geo, id, name, place_type")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor EXPANSIONS = new PropertyDescriptor.Builder()
+ .name("expansions")
+ .displayName("Expansions")
+ .description("A comma-separated list of expansions for objects in the returned tweet. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/expansions " +
+ "for proper usage. Possible field values are: " +
+ "author_id, referenced_tweets.id, referenced_tweets.id.author_id, entities.mentions.username, " +
+ "attachments.poll_ids, attachments.media_keys ,in_reply_to_user_id, geo.place_id")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor BACKFILL_MINUTES = new PropertyDescriptor.Builder()
+ .name("backfill-minutes")
+ .displayName("Backfill Minutes")
+ .description("The number of minutes (up to 5 minutes) of streaming data to be requested after a " +
+ "disconnect. Only available for project with academic research access. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/integrate/" +
+ "recovery-and-redundancy-features")
+ .required(true)
+ .defaultValue("0")
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("All tweets will be routed to this relationship.")
Review Comment:
```suggestion
.description("FlowFiles containing an array of one or more Tweets")
```
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,318 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("consume-twitter-endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("The size of the queue used to queue tweets")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("tweet-fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values are: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("user-fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user " +
+ "for proper usage. Possible field values are: " +
+ "created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, " +
+ "protected, public_metrics, url, username, verified, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor MEDIA_FIELDS = new PropertyDescriptor.Builder()
+ .name("media-fields")
+ .displayName("Media Fields")
+ .description("A comma-separated list of media fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/media " +
+ "for proper usage. Possible field values are: " +
+ "alt_text, duration_ms, height, media_key, non_public_metrics, organic_metrics, preview_image_url, " +
+ "promoted_metrics, public_metrics, type, url, width")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor POLL_FIELDS = new PropertyDescriptor.Builder()
+ .name("poll-fields")
+ .displayName("Poll Fields")
+ .description("A comma-separated list of poll fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/poll " +
+ "for proper usage. Possible field values are: " +
+ "duration_minutes, end_datetime, id, options, voting_status")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PLACE_FIELDS = new PropertyDescriptor.Builder()
+ .name("place-fields")
+ .displayName("Place Fields")
+ .description("A comma-separated list of place fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/place " +
+ "for proper usage. Possible field values are: " +
+ "contained_within, country, country_code, full_name, geo, id, name, place_type")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor EXPANSIONS = new PropertyDescriptor.Builder()
+ .name("expansions")
+ .displayName("Expansions")
+ .description("A comma-separated list of expansions for objects in the returned tweet. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/expansions " +
+ "for proper usage. Possible field values are: " +
+ "author_id, referenced_tweets.id, referenced_tweets.id.author_id, entities.mentions.username, " +
+ "attachments.poll_ids, attachments.media_keys ,in_reply_to_user_id, geo.place_id")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor BACKFILL_MINUTES = new PropertyDescriptor.Builder()
+ .name("backfill-minutes")
+ .displayName("Backfill Minutes")
+ .description("The number of minutes (up to 5 minutes) of streaming data to be requested after a " +
+ "disconnect. Only available for project with academic research access. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/integrate/" +
+ "recovery-and-redundancy-features")
+ .required(true)
+ .defaultValue("0")
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("All tweets will be routed to this relationship.")
+ .build();
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+
+ private TwitterStreamAPI api;
+
+ private volatile BlockingQueue<String> messageQueue;
+
+ private void emptyQueue() {
+ while (!messageQueue.isEmpty()) {
+ messageQueue.poll();
+ }
+ }
+
+ @Override
+ protected void init(ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptors = new ArrayList<>();
+ descriptors.add(ENDPOINT);
+ descriptors.add(BASE_PATH);
+ descriptors.add(BEARER_TOKEN);
+ descriptors.add(QUEUE_SIZE);
+ descriptors.add(BATCH_SIZE);
+ descriptors.add(TWEET_FIELDS);
+ descriptors.add(USER_FIELDS);
+ descriptors.add(MEDIA_FIELDS);
+ descriptors.add(POLL_FIELDS);
+ descriptors.add(PLACE_FIELDS);
+ descriptors.add(EXPANSIONS);
+ descriptors.add(BACKFILL_MINUTES);
+
+ this.descriptors = Collections.unmodifiableList(descriptors);
+
+ final Set<Relationship> relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ this.relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return this.descriptors;
+ }
+
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ messageQueue = new LinkedBlockingQueue<>(context.getProperty(QUEUE_SIZE).asInteger());
+
+ api = new TwitterStreamAPI(context, messageQueue, getLogger());
+ api.setBasePath(context.getProperty(BASE_PATH).getValue());
+ final String endpointName = context.getProperty(ENDPOINT).getValue();
+ if (ENDPOINT_SAMPLE.getValue().equals(endpointName)) {
+ api.start(TwitterStreamAPI.SAMPLE_ENDPOINT);
+ } else if (ENDPOINT_SEARCH.getValue().equals(endpointName)) {
+ api.start(TwitterStreamAPI.SEARCH_ENDPOINT);
+ } else {
+ throw new ProcessException("Endpoint was invalid value: " + endpointName);
+ }
+ }
+
+ @Override
+ public void onTrigger(ProcessContext context, ProcessSession session) throws ProcessException {
+ if (messageQueue.isEmpty()) {
+ context.yield();
+ return;
+ }
+
+ FlowFile flowFile = session.create();
+ flowFile = session.write(flowFile, new OutputStreamCallback() {
+ @Override
+ public void process(OutputStream out) throws IOException {
+ out.write("[".getBytes(StandardCharsets.UTF_8));
Review Comment:
This should be optimized to use the character directly:
```suggestion
out.write('[');
```
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,318 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("consume-twitter-endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("The size of the queue used to queue tweets")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("tweet-fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values are: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("user-fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user " +
+ "for proper usage. Possible field values are: " +
+ "created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, " +
+ "protected, public_metrics, url, username, verified, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor MEDIA_FIELDS = new PropertyDescriptor.Builder()
+ .name("media-fields")
+ .displayName("Media Fields")
+ .description("A comma-separated list of media fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/media " +
+ "for proper usage. Possible field values are: " +
+ "alt_text, duration_ms, height, media_key, non_public_metrics, organic_metrics, preview_image_url, " +
+ "promoted_metrics, public_metrics, type, url, width")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor POLL_FIELDS = new PropertyDescriptor.Builder()
+ .name("poll-fields")
+ .displayName("Poll Fields")
+ .description("A comma-separated list of poll fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/poll " +
+ "for proper usage. Possible field values are: " +
+ "duration_minutes, end_datetime, id, options, voting_status")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PLACE_FIELDS = new PropertyDescriptor.Builder()
+ .name("place-fields")
+ .displayName("Place Fields")
+ .description("A comma-separated list of place fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/place " +
+ "for proper usage. Possible field values are: " +
+ "contained_within, country, country_code, full_name, geo, id, name, place_type")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor EXPANSIONS = new PropertyDescriptor.Builder()
+ .name("expansions")
+ .displayName("Expansions")
+ .description("A comma-separated list of expansions for objects in the returned tweet. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/expansions " +
+ "for proper usage. Possible field values are: " +
+ "author_id, referenced_tweets.id, referenced_tweets.id.author_id, entities.mentions.username, " +
+ "attachments.poll_ids, attachments.media_keys ,in_reply_to_user_id, geo.place_id")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor BACKFILL_MINUTES = new PropertyDescriptor.Builder()
+ .name("backfill-minutes")
+ .displayName("Backfill Minutes")
+ .description("The number of minutes (up to 5 minutes) of streaming data to be requested after a " +
+ "disconnect. Only available for project with academic research access. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/integrate/" +
+ "recovery-and-redundancy-features")
+ .required(true)
+ .defaultValue("0")
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("All tweets will be routed to this relationship.")
+ .build();
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+
+ private TwitterStreamAPI api;
+
+ private volatile BlockingQueue<String> messageQueue;
+
+ private void emptyQueue() {
+ while (!messageQueue.isEmpty()) {
+ messageQueue.poll();
+ }
+ }
+
+ @Override
+ protected void init(ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptors = new ArrayList<>();
+ descriptors.add(ENDPOINT);
+ descriptors.add(BASE_PATH);
+ descriptors.add(BEARER_TOKEN);
+ descriptors.add(QUEUE_SIZE);
+ descriptors.add(BATCH_SIZE);
+ descriptors.add(TWEET_FIELDS);
+ descriptors.add(USER_FIELDS);
+ descriptors.add(MEDIA_FIELDS);
+ descriptors.add(POLL_FIELDS);
+ descriptors.add(PLACE_FIELDS);
+ descriptors.add(EXPANSIONS);
+ descriptors.add(BACKFILL_MINUTES);
+
+ this.descriptors = Collections.unmodifiableList(descriptors);
+
+ final Set<Relationship> relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ this.relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return this.descriptors;
+ }
+
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ messageQueue = new LinkedBlockingQueue<>(context.getProperty(QUEUE_SIZE).asInteger());
+
+ api = new TwitterStreamAPI(context, messageQueue, getLogger());
+ api.setBasePath(context.getProperty(BASE_PATH).getValue());
+ final String endpointName = context.getProperty(ENDPOINT).getValue();
+ if (ENDPOINT_SAMPLE.getValue().equals(endpointName)) {
+ api.start(TwitterStreamAPI.SAMPLE_ENDPOINT);
+ } else if (ENDPOINT_SEARCH.getValue().equals(endpointName)) {
+ api.start(TwitterStreamAPI.SEARCH_ENDPOINT);
+ } else {
+ throw new ProcessException("Endpoint was invalid value: " + endpointName);
+ }
+ }
+
+ @Override
+ public void onTrigger(ProcessContext context, ProcessSession session) throws ProcessException {
+ if (messageQueue.isEmpty()) {
+ context.yield();
+ return;
+ }
+
+ FlowFile flowFile = session.create();
+ flowFile = session.write(flowFile, new OutputStreamCallback() {
+ @Override
+ public void process(OutputStream out) throws IOException {
+ out.write("[".getBytes(StandardCharsets.UTF_8));
+ final int batchSize = context.getProperty(BATCH_SIZE).asInteger();
+ String tweet = messageQueue.poll();
+ out.write(tweet.getBytes(StandardCharsets.UTF_8));
+ int tweetCount = 1;
+ while (tweetCount < batchSize && !messageQueue.isEmpty()) {
+ out.write(",".getBytes(StandardCharsets.UTF_8));
+ tweet = messageQueue.poll();
+ out.write(tweet.getBytes(StandardCharsets.UTF_8));
+ tweetCount++;
+ }
+ out.write("]".getBytes(StandardCharsets.UTF_8));
+ }
+ });
+
+ final Map<String, String> attributes = new HashMap<>();
+ attributes.put(CoreAttributes.MIME_TYPE.key(), "application/json");
+ attributes.put(CoreAttributes.FILENAME.key(), flowFile.getAttribute(CoreAttributes.FILENAME.key()) + ".json");
Review Comment:
Recommend changing this to use a random UUID directly, instead of relying on the generated filename:
```suggestion
attributes.put(CoreAttributes.FILENAME.key(), String.format("%s.json", UUID.randomUUI()));
```
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,318 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("consume-twitter-endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("The size of the queue used to queue tweets")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("tweet-fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values are: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("user-fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user " +
+ "for proper usage. Possible field values are: " +
+ "created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, " +
+ "protected, public_metrics, url, username, verified, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor MEDIA_FIELDS = new PropertyDescriptor.Builder()
+ .name("media-fields")
+ .displayName("Media Fields")
+ .description("A comma-separated list of media fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/media " +
+ "for proper usage. Possible field values are: " +
+ "alt_text, duration_ms, height, media_key, non_public_metrics, organic_metrics, preview_image_url, " +
+ "promoted_metrics, public_metrics, type, url, width")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor POLL_FIELDS = new PropertyDescriptor.Builder()
+ .name("poll-fields")
+ .displayName("Poll Fields")
+ .description("A comma-separated list of poll fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/poll " +
+ "for proper usage. Possible field values are: " +
Review Comment:
```suggestion
"for proper usage. Possible field values include: " +
```
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TwitterStreamAPI.java:
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+public class TwitterStreamAPI {
+ public static final String SEARCH_ENDPOINT = "Search Endpoint";
+ public static final String SAMPLE_ENDPOINT = "Sample Endpoint";
+ public static final String SAMPLE_PATH = "/2/tweets/sample/stream";
+ public static final String SEARCH_PATH = "/2/tweets/search/stream";
+
+ private static final String BEARER_TOKEN_PROPERTY_NAME = "bearer-token";
+ private static final String TWEET_FIELDS_PROPERTY_NAME = "tweet-fields";
+ private static final String USER_FIELDS_PROPERTY_NAME = "user-fields";
+ private static final String MEDIA_FIELDS_PROPERTY_NAME = "media-fields";
+ private static final String POLL_FIELDS_PROPERTY_NAME = "poll-fields";
+ private static final String PLACE_FIELDS_PROPERTY_NAME = "place-fields";
+ private static final String EXPANSIONS_PROPERTY_NAME = "expansions";
+ private static final String BACKFILL_MINUTES_PROPERTY_NAME = "backfill-minutes";
+
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Set<String> parseCommaSeparatedListPropreties(final ProcessContext context, final String propertyName) {
+ Set<String> fields = null;
+ if (context.getProperty(propertyName).isSet()) {
+ fields = new HashSet<>();
+ final String fieldsString = context.getProperty(propertyName).getValue();
+ for (final String field: fieldsString.split(",")) {
+ fields.add(field.trim());
+ }
+ }
+ return fields;
+ }
+
+ public TwitterStreamAPI(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ assert context != null;
+ assert queue != null;
+ assert logger != null;
+
+ this.queue = queue;
+ this.logger = logger;
+
+ this.tweetFields = parseCommaSeparatedListPropreties(context, TWEET_FIELDS_PROPERTY_NAME);
+ this.userFields = parseCommaSeparatedListPropreties(context, USER_FIELDS_PROPERTY_NAME);
+ this.mediaFields = parseCommaSeparatedListPropreties(context, MEDIA_FIELDS_PROPERTY_NAME);
+ this.pollFields = parseCommaSeparatedListPropreties(context, POLL_FIELDS_PROPERTY_NAME);
+ this.placeFields = parseCommaSeparatedListPropreties(context, PLACE_FIELDS_PROPERTY_NAME);
+ this.expansions = parseCommaSeparatedListPropreties(context, EXPANSIONS_PROPERTY_NAME);
+ this.backfillMinutes = context.getProperty(BACKFILL_MINUTES_PROPERTY_NAME).asInteger();
+
+ TwitterCredentialsBearer creds = new TwitterCredentialsBearer(context.getProperty(BEARER_TOKEN_PROPERTY_NAME).getValue());
+ api = new TwitterApi();
+ api.setTwitterCredentials(creds);
+
+ this.executorService = Executors.newSingleThreadExecutor();
+ }
+
+ public String getBasePath() {
+ return api.getApiClient().getBasePath();
+ }
+
+ public void setBasePath(final String path) {
+ api.getApiClient().setBasePath(path);
+ }
+
+ /**
+ * This method would be called when we would like the stream to get started. This method will spin off a thread that
+ * will continue to queue tweets on to the given queue passed in the constructor. The thread will continue
+ * to run until {@code stop} is called.
+ * @param endpoint {@code TwitterStreamAPI.SAMPLE_ENDPOINT} or {@code TwitterStreamAPI.SEARCH_ENDPOINT}
+ */
+ public void start(final String endpoint) {
+ try {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ stream = api.tweets().sampleStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else if (endpoint.equals(SEARCH_ENDPOINT)) {
+ stream = api.tweets().searchStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else {
+ throw new ProcessException("Endpoint was invalid value: " + endpoint);
+ }
+ } catch (ApiException e) {
+ logger.error("Received error {}: {}", e.getCode(), e.getMessage());
+ }
+
+ executorService.execute(new TweetQueuer());
+ }
+
+ /**
+ * This method would be called when we would like the stream to get stopped. The stream will be closed and the
+ * executorService will be shut down. If it fails to shutdown, then it will be forcefully terminated.
+ */
+ public void stop() {
+ try {
+ stream.close();
+ } catch (IOException e) {
+ logger.error("IOException occurred while closing stream: {}", e.getMessage());
+ }
+
+ executorService.shutdown();
+ try {
+ if (!executorService.awaitTermination(3, TimeUnit.SECONDS)) {
+ executorService.shutdownNow();
+ }
+ } catch (InterruptedException e) {
+ executorService.shutdownNow();
+ logger.error("InterruptedException occurred while waiting for executor service to shut down: {}", e.getMessage());
+ }
+ }
+
+ private class TweetQueuer implements Runnable {
Review Comment:
This name is serviceable, but what do you think about renaming it to TweetStreamHandler?
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TwitterStreamAPI.java:
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+public class TwitterStreamAPI {
+ public static final String SEARCH_ENDPOINT = "Search Endpoint";
+ public static final String SAMPLE_ENDPOINT = "Sample Endpoint";
+ public static final String SAMPLE_PATH = "/2/tweets/sample/stream";
+ public static final String SEARCH_PATH = "/2/tweets/search/stream";
+
+ private static final String BEARER_TOKEN_PROPERTY_NAME = "bearer-token";
+ private static final String TWEET_FIELDS_PROPERTY_NAME = "tweet-fields";
+ private static final String USER_FIELDS_PROPERTY_NAME = "user-fields";
+ private static final String MEDIA_FIELDS_PROPERTY_NAME = "media-fields";
+ private static final String POLL_FIELDS_PROPERTY_NAME = "poll-fields";
+ private static final String PLACE_FIELDS_PROPERTY_NAME = "place-fields";
+ private static final String EXPANSIONS_PROPERTY_NAME = "expansions";
+ private static final String BACKFILL_MINUTES_PROPERTY_NAME = "backfill-minutes";
+
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Set<String> parseCommaSeparatedListPropreties(final ProcessContext context, final String propertyName) {
+ Set<String> fields = null;
+ if (context.getProperty(propertyName).isSet()) {
+ fields = new HashSet<>();
+ final String fieldsString = context.getProperty(propertyName).getValue();
+ for (final String field: fieldsString.split(",")) {
+ fields.add(field.trim());
+ }
+ }
+ return fields;
+ }
+
+ public TwitterStreamAPI(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ assert context != null;
+ assert queue != null;
+ assert logger != null;
+
+ this.queue = queue;
+ this.logger = logger;
+
+ this.tweetFields = parseCommaSeparatedListPropreties(context, TWEET_FIELDS_PROPERTY_NAME);
+ this.userFields = parseCommaSeparatedListPropreties(context, USER_FIELDS_PROPERTY_NAME);
+ this.mediaFields = parseCommaSeparatedListPropreties(context, MEDIA_FIELDS_PROPERTY_NAME);
+ this.pollFields = parseCommaSeparatedListPropreties(context, POLL_FIELDS_PROPERTY_NAME);
+ this.placeFields = parseCommaSeparatedListPropreties(context, PLACE_FIELDS_PROPERTY_NAME);
+ this.expansions = parseCommaSeparatedListPropreties(context, EXPANSIONS_PROPERTY_NAME);
+ this.backfillMinutes = context.getProperty(BACKFILL_MINUTES_PROPERTY_NAME).asInteger();
+
+ TwitterCredentialsBearer creds = new TwitterCredentialsBearer(context.getProperty(BEARER_TOKEN_PROPERTY_NAME).getValue());
+ api = new TwitterApi();
+ api.setTwitterCredentials(creds);
+
+ this.executorService = Executors.newSingleThreadExecutor();
+ }
+
+ public String getBasePath() {
+ return api.getApiClient().getBasePath();
+ }
+
+ public void setBasePath(final String path) {
+ api.getApiClient().setBasePath(path);
+ }
+
+ /**
+ * This method would be called when we would like the stream to get started. This method will spin off a thread that
+ * will continue to queue tweets on to the given queue passed in the constructor. The thread will continue
+ * to run until {@code stop} is called.
+ * @param endpoint {@code TwitterStreamAPI.SAMPLE_ENDPOINT} or {@code TwitterStreamAPI.SEARCH_ENDPOINT}
+ */
+ public void start(final String endpoint) {
+ try {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ stream = api.tweets().sampleStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else if (endpoint.equals(SEARCH_ENDPOINT)) {
+ stream = api.tweets().searchStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else {
+ throw new ProcessException("Endpoint was invalid value: " + endpoint);
+ }
+ } catch (ApiException e) {
+ logger.error("Received error {}: {}", e.getCode(), e.getMessage());
+ }
+
+ executorService.execute(new TweetQueuer());
+ }
+
+ /**
+ * This method would be called when we would like the stream to get stopped. The stream will be closed and the
+ * executorService will be shut down. If it fails to shutdown, then it will be forcefully terminated.
+ */
+ public void stop() {
+ try {
+ stream.close();
+ } catch (IOException e) {
+ logger.error("IOException occurred while closing stream: {}", e.getMessage());
+ }
+
+ executorService.shutdown();
+ try {
+ if (!executorService.awaitTermination(3, TimeUnit.SECONDS)) {
+ executorService.shutdownNow();
+ }
+ } catch (InterruptedException e) {
+ executorService.shutdownNow();
+ logger.error("InterruptedException occurred while waiting for executor service to shut down: {}", e.getMessage());
+ }
+ }
+
+ private class TweetQueuer implements Runnable {
+ @Override
+ public void run() {
+ try (BufferedReader reader = new BufferedReader(new InputStreamReader(stream))) {
+ String tweetLine = reader.readLine();
Review Comment:
Minor naming recommendation, adjusting the variable name and adding a comment helps communicate the fact that Twitter returns a complete JSON record in each line:
```suggestion
// Twitter returns one JSON object in each line of the stream
String tweetRecord = reader.readLine();
```
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TwitterStreamAPI.java:
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+public class TwitterStreamAPI {
+ public static final String SEARCH_ENDPOINT = "Search Endpoint";
+ public static final String SAMPLE_ENDPOINT = "Sample Endpoint";
+ public static final String SAMPLE_PATH = "/2/tweets/sample/stream";
+ public static final String SEARCH_PATH = "/2/tweets/search/stream";
+
+ private static final String BEARER_TOKEN_PROPERTY_NAME = "bearer-token";
+ private static final String TWEET_FIELDS_PROPERTY_NAME = "tweet-fields";
+ private static final String USER_FIELDS_PROPERTY_NAME = "user-fields";
+ private static final String MEDIA_FIELDS_PROPERTY_NAME = "media-fields";
+ private static final String POLL_FIELDS_PROPERTY_NAME = "poll-fields";
+ private static final String PLACE_FIELDS_PROPERTY_NAME = "place-fields";
+ private static final String EXPANSIONS_PROPERTY_NAME = "expansions";
+ private static final String BACKFILL_MINUTES_PROPERTY_NAME = "backfill-minutes";
+
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Set<String> parseCommaSeparatedListPropreties(final ProcessContext context, final String propertyName) {
+ Set<String> fields = null;
+ if (context.getProperty(propertyName).isSet()) {
+ fields = new HashSet<>();
+ final String fieldsString = context.getProperty(propertyName).getValue();
+ for (final String field: fieldsString.split(",")) {
+ fields.add(field.trim());
+ }
+ }
+ return fields;
+ }
+
+ public TwitterStreamAPI(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ assert context != null;
+ assert queue != null;
+ assert logger != null;
+
+ this.queue = queue;
+ this.logger = logger;
+
+ this.tweetFields = parseCommaSeparatedListPropreties(context, TWEET_FIELDS_PROPERTY_NAME);
+ this.userFields = parseCommaSeparatedListPropreties(context, USER_FIELDS_PROPERTY_NAME);
+ this.mediaFields = parseCommaSeparatedListPropreties(context, MEDIA_FIELDS_PROPERTY_NAME);
+ this.pollFields = parseCommaSeparatedListPropreties(context, POLL_FIELDS_PROPERTY_NAME);
+ this.placeFields = parseCommaSeparatedListPropreties(context, PLACE_FIELDS_PROPERTY_NAME);
+ this.expansions = parseCommaSeparatedListPropreties(context, EXPANSIONS_PROPERTY_NAME);
+ this.backfillMinutes = context.getProperty(BACKFILL_MINUTES_PROPERTY_NAME).asInteger();
+
+ TwitterCredentialsBearer creds = new TwitterCredentialsBearer(context.getProperty(BEARER_TOKEN_PROPERTY_NAME).getValue());
+ api = new TwitterApi();
+ api.setTwitterCredentials(creds);
+
+ this.executorService = Executors.newSingleThreadExecutor();
+ }
+
+ public String getBasePath() {
+ return api.getApiClient().getBasePath();
+ }
+
+ public void setBasePath(final String path) {
+ api.getApiClient().setBasePath(path);
+ }
+
+ /**
+ * This method would be called when we would like the stream to get started. This method will spin off a thread that
+ * will continue to queue tweets on to the given queue passed in the constructor. The thread will continue
+ * to run until {@code stop} is called.
+ * @param endpoint {@code TwitterStreamAPI.SAMPLE_ENDPOINT} or {@code TwitterStreamAPI.SEARCH_ENDPOINT}
+ */
+ public void start(final String endpoint) {
+ try {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ stream = api.tweets().sampleStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else if (endpoint.equals(SEARCH_ENDPOINT)) {
+ stream = api.tweets().searchStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else {
+ throw new ProcessException("Endpoint was invalid value: " + endpoint);
+ }
Review Comment:
Instead of checking each value and throwing what should be an `IllegalStateException` in the `else` case, recommend having an conditional check for the `SAMPLE_ENDPOINT`, and defaulting the the search endpoint in the else case.
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TwitterStreamAPI.java:
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+public class TwitterStreamAPI {
+ public static final String SEARCH_ENDPOINT = "Search Endpoint";
+ public static final String SAMPLE_ENDPOINT = "Sample Endpoint";
+ public static final String SAMPLE_PATH = "/2/tweets/sample/stream";
+ public static final String SEARCH_PATH = "/2/tweets/search/stream";
+
+ private static final String BEARER_TOKEN_PROPERTY_NAME = "bearer-token";
+ private static final String TWEET_FIELDS_PROPERTY_NAME = "tweet-fields";
+ private static final String USER_FIELDS_PROPERTY_NAME = "user-fields";
+ private static final String MEDIA_FIELDS_PROPERTY_NAME = "media-fields";
+ private static final String POLL_FIELDS_PROPERTY_NAME = "poll-fields";
+ private static final String PLACE_FIELDS_PROPERTY_NAME = "place-fields";
+ private static final String EXPANSIONS_PROPERTY_NAME = "expansions";
+ private static final String BACKFILL_MINUTES_PROPERTY_NAME = "backfill-minutes";
+
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Set<String> parseCommaSeparatedListPropreties(final ProcessContext context, final String propertyName) {
+ Set<String> fields = null;
+ if (context.getProperty(propertyName).isSet()) {
+ fields = new HashSet<>();
+ final String fieldsString = context.getProperty(propertyName).getValue();
+ for (final String field: fieldsString.split(",")) {
+ fields.add(field.trim());
+ }
+ }
+ return fields;
+ }
+
+ public TwitterStreamAPI(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ assert context != null;
+ assert queue != null;
+ assert logger != null;
+
+ this.queue = queue;
+ this.logger = logger;
+
+ this.tweetFields = parseCommaSeparatedListPropreties(context, TWEET_FIELDS_PROPERTY_NAME);
+ this.userFields = parseCommaSeparatedListPropreties(context, USER_FIELDS_PROPERTY_NAME);
+ this.mediaFields = parseCommaSeparatedListPropreties(context, MEDIA_FIELDS_PROPERTY_NAME);
+ this.pollFields = parseCommaSeparatedListPropreties(context, POLL_FIELDS_PROPERTY_NAME);
+ this.placeFields = parseCommaSeparatedListPropreties(context, PLACE_FIELDS_PROPERTY_NAME);
+ this.expansions = parseCommaSeparatedListPropreties(context, EXPANSIONS_PROPERTY_NAME);
+ this.backfillMinutes = context.getProperty(BACKFILL_MINUTES_PROPERTY_NAME).asInteger();
+
+ TwitterCredentialsBearer creds = new TwitterCredentialsBearer(context.getProperty(BEARER_TOKEN_PROPERTY_NAME).getValue());
+ api = new TwitterApi();
+ api.setTwitterCredentials(creds);
+
+ this.executorService = Executors.newSingleThreadExecutor();
+ }
+
+ public String getBasePath() {
+ return api.getApiClient().getBasePath();
+ }
+
+ public void setBasePath(final String path) {
+ api.getApiClient().setBasePath(path);
+ }
+
+ /**
+ * This method would be called when we would like the stream to get started. This method will spin off a thread that
+ * will continue to queue tweets on to the given queue passed in the constructor. The thread will continue
+ * to run until {@code stop} is called.
+ * @param endpoint {@code TwitterStreamAPI.SAMPLE_ENDPOINT} or {@code TwitterStreamAPI.SEARCH_ENDPOINT}
+ */
+ public void start(final String endpoint) {
+ try {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ stream = api.tweets().sampleStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else if (endpoint.equals(SEARCH_ENDPOINT)) {
+ stream = api.tweets().searchStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else {
+ throw new ProcessException("Endpoint was invalid value: " + endpoint);
+ }
+ } catch (ApiException e) {
+ logger.error("Received error {}: {}", e.getCode(), e.getMessage());
+ }
+
+ executorService.execute(new TweetQueuer());
+ }
+
+ /**
+ * This method would be called when we would like the stream to get stopped. The stream will be closed and the
+ * executorService will be shut down. If it fails to shutdown, then it will be forcefully terminated.
+ */
+ public void stop() {
+ try {
+ stream.close();
+ } catch (IOException e) {
+ logger.error("IOException occurred while closing stream: {}", e.getMessage());
+ }
+
+ executorService.shutdown();
+ try {
+ if (!executorService.awaitTermination(3, TimeUnit.SECONDS)) {
+ executorService.shutdownNow();
+ }
+ } catch (InterruptedException e) {
+ executorService.shutdownNow();
+ logger.error("InterruptedException occurred while waiting for executor service to shut down: {}", e.getMessage());
+ }
Review Comment:
Since the method already closes the response stream, a more graceful shutdown does not seem necessary. Recommend changing this to just call `shutdownNow()`:
```suggestion
executorService.shutdownNow();
```
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/pom.xml:
##########
@@ -69,5 +80,17 @@
<version>1.17.0-SNAPSHOT</version>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>com.squareup.okhttp3</groupId>
+ <artifactId>okhttp</artifactId>
+ <version>4.9.3</version>
+ <scope>test</scope>
Review Comment:
This is also a runtime dependency, so the `test` scope should be removed.
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,318 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("consume-twitter-endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("The size of the queue used to queue tweets")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("tweet-fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values are: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("user-fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user " +
+ "for proper usage. Possible field values are: " +
+ "created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, " +
+ "protected, public_metrics, url, username, verified, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor MEDIA_FIELDS = new PropertyDescriptor.Builder()
+ .name("media-fields")
+ .displayName("Media Fields")
+ .description("A comma-separated list of media fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/media " +
+ "for proper usage. Possible field values are: " +
+ "alt_text, duration_ms, height, media_key, non_public_metrics, organic_metrics, preview_image_url, " +
+ "promoted_metrics, public_metrics, type, url, width")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor POLL_FIELDS = new PropertyDescriptor.Builder()
+ .name("poll-fields")
+ .displayName("Poll Fields")
+ .description("A comma-separated list of poll fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/poll " +
+ "for proper usage. Possible field values are: " +
+ "duration_minutes, end_datetime, id, options, voting_status")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PLACE_FIELDS = new PropertyDescriptor.Builder()
+ .name("place-fields")
+ .displayName("Place Fields")
+ .description("A comma-separated list of place fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/place " +
+ "for proper usage. Possible field values are: " +
+ "contained_within, country, country_code, full_name, geo, id, name, place_type")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor EXPANSIONS = new PropertyDescriptor.Builder()
+ .name("expansions")
+ .displayName("Expansions")
+ .description("A comma-separated list of expansions for objects in the returned tweet. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/expansions " +
+ "for proper usage. Possible field values are: " +
Review Comment:
```suggestion
"for proper usage. Possible field values include: " +
```
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,318 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("consume-twitter-endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("The size of the queue used to queue tweets")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("tweet-fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values are: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("user-fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user " +
+ "for proper usage. Possible field values are: " +
+ "created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, " +
+ "protected, public_metrics, url, username, verified, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor MEDIA_FIELDS = new PropertyDescriptor.Builder()
+ .name("media-fields")
+ .displayName("Media Fields")
+ .description("A comma-separated list of media fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/media " +
+ "for proper usage. Possible field values are: " +
+ "alt_text, duration_ms, height, media_key, non_public_metrics, organic_metrics, preview_image_url, " +
+ "promoted_metrics, public_metrics, type, url, width")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor POLL_FIELDS = new PropertyDescriptor.Builder()
+ .name("poll-fields")
+ .displayName("Poll Fields")
+ .description("A comma-separated list of poll fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/poll " +
+ "for proper usage. Possible field values are: " +
+ "duration_minutes, end_datetime, id, options, voting_status")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PLACE_FIELDS = new PropertyDescriptor.Builder()
+ .name("place-fields")
+ .displayName("Place Fields")
+ .description("A comma-separated list of place fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/place " +
+ "for proper usage. Possible field values are: " +
Review Comment:
```suggestion
"for proper usage. Possible field values include: " +
```
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TwitterStreamAPI.java:
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+public class TwitterStreamAPI {
+ public static final String SEARCH_ENDPOINT = "Search Endpoint";
+ public static final String SAMPLE_ENDPOINT = "Sample Endpoint";
+ public static final String SAMPLE_PATH = "/2/tweets/sample/stream";
+ public static final String SEARCH_PATH = "/2/tweets/search/stream";
+
+ private static final String BEARER_TOKEN_PROPERTY_NAME = "bearer-token";
+ private static final String TWEET_FIELDS_PROPERTY_NAME = "tweet-fields";
+ private static final String USER_FIELDS_PROPERTY_NAME = "user-fields";
+ private static final String MEDIA_FIELDS_PROPERTY_NAME = "media-fields";
+ private static final String POLL_FIELDS_PROPERTY_NAME = "poll-fields";
+ private static final String PLACE_FIELDS_PROPERTY_NAME = "place-fields";
+ private static final String EXPANSIONS_PROPERTY_NAME = "expansions";
+ private static final String BACKFILL_MINUTES_PROPERTY_NAME = "backfill-minutes";
+
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Set<String> parseCommaSeparatedListPropreties(final ProcessContext context, final String propertyName) {
+ Set<String> fields = null;
+ if (context.getProperty(propertyName).isSet()) {
+ fields = new HashSet<>();
+ final String fieldsString = context.getProperty(propertyName).getValue();
+ for (final String field: fieldsString.split(",")) {
+ fields.add(field.trim());
+ }
+ }
+ return fields;
+ }
+
+ public TwitterStreamAPI(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ assert context != null;
+ assert queue != null;
+ assert logger != null;
+
+ this.queue = queue;
+ this.logger = logger;
+
+ this.tweetFields = parseCommaSeparatedListPropreties(context, TWEET_FIELDS_PROPERTY_NAME);
+ this.userFields = parseCommaSeparatedListPropreties(context, USER_FIELDS_PROPERTY_NAME);
+ this.mediaFields = parseCommaSeparatedListPropreties(context, MEDIA_FIELDS_PROPERTY_NAME);
+ this.pollFields = parseCommaSeparatedListPropreties(context, POLL_FIELDS_PROPERTY_NAME);
+ this.placeFields = parseCommaSeparatedListPropreties(context, PLACE_FIELDS_PROPERTY_NAME);
+ this.expansions = parseCommaSeparatedListPropreties(context, EXPANSIONS_PROPERTY_NAME);
+ this.backfillMinutes = context.getProperty(BACKFILL_MINUTES_PROPERTY_NAME).asInteger();
+
+ TwitterCredentialsBearer creds = new TwitterCredentialsBearer(context.getProperty(BEARER_TOKEN_PROPERTY_NAME).getValue());
+ api = new TwitterApi();
+ api.setTwitterCredentials(creds);
+
+ this.executorService = Executors.newSingleThreadExecutor();
+ }
+
+ public String getBasePath() {
+ return api.getApiClient().getBasePath();
+ }
+
+ public void setBasePath(final String path) {
+ api.getApiClient().setBasePath(path);
+ }
+
+ /**
+ * This method would be called when we would like the stream to get started. This method will spin off a thread that
+ * will continue to queue tweets on to the given queue passed in the constructor. The thread will continue
+ * to run until {@code stop} is called.
+ * @param endpoint {@code TwitterStreamAPI.SAMPLE_ENDPOINT} or {@code TwitterStreamAPI.SEARCH_ENDPOINT}
+ */
+ public void start(final String endpoint) {
+ try {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ stream = api.tweets().sampleStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else if (endpoint.equals(SEARCH_ENDPOINT)) {
+ stream = api.tweets().searchStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else {
+ throw new ProcessException("Endpoint was invalid value: " + endpoint);
+ }
+ } catch (ApiException e) {
+ logger.error("Received error {}: {}", e.getCode(), e.getMessage());
+ }
+
+ executorService.execute(new TweetQueuer());
+ }
+
+ /**
+ * This method would be called when we would like the stream to get stopped. The stream will be closed and the
+ * executorService will be shut down. If it fails to shutdown, then it will be forcefully terminated.
+ */
+ public void stop() {
+ try {
+ stream.close();
+ } catch (IOException e) {
+ logger.error("IOException occurred while closing stream: {}", e.getMessage());
+ }
+
+ executorService.shutdown();
+ try {
+ if (!executorService.awaitTermination(3, TimeUnit.SECONDS)) {
+ executorService.shutdownNow();
+ }
+ } catch (InterruptedException e) {
+ executorService.shutdownNow();
+ logger.error("InterruptedException occurred while waiting for executor service to shut down: {}", e.getMessage());
+ }
+ }
+
+ private class TweetQueuer implements Runnable {
+ @Override
+ public void run() {
+ try (BufferedReader reader = new BufferedReader(new InputStreamReader(stream))) {
+ String tweetLine = reader.readLine();
+ while (tweetLine != null) {
+ queue.offer(tweetLine);
+ try {
+ tweetLine = reader.readLine();
+ } catch (IOException e) {
+ logger.info("Stream closed, no more tweets to read, exiting Queuer");
+ break;
+ }
+ }
+ } catch (IOException e) {
+ logger.warn("IOException occurred in TweetQueuer: {}", e.getMessage());
Review Comment:
The exception should be passed as an argument to the logger for troubleshooting:
```suggestion
logger.warn("Stream processing failed", e);
```
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,318 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("consume-twitter-endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("The size of the queue used to queue tweets")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("tweet-fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values are: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("user-fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user " +
+ "for proper usage. Possible field values are: " +
+ "created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, " +
+ "protected, public_metrics, url, username, verified, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor MEDIA_FIELDS = new PropertyDescriptor.Builder()
+ .name("media-fields")
+ .displayName("Media Fields")
+ .description("A comma-separated list of media fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/media " +
+ "for proper usage. Possible field values are: " +
+ "alt_text, duration_ms, height, media_key, non_public_metrics, organic_metrics, preview_image_url, " +
+ "promoted_metrics, public_metrics, type, url, width")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor POLL_FIELDS = new PropertyDescriptor.Builder()
+ .name("poll-fields")
+ .displayName("Poll Fields")
+ .description("A comma-separated list of poll fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/poll " +
+ "for proper usage. Possible field values are: " +
+ "duration_minutes, end_datetime, id, options, voting_status")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PLACE_FIELDS = new PropertyDescriptor.Builder()
+ .name("place-fields")
+ .displayName("Place Fields")
+ .description("A comma-separated list of place fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/place " +
+ "for proper usage. Possible field values are: " +
+ "contained_within, country, country_code, full_name, geo, id, name, place_type")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor EXPANSIONS = new PropertyDescriptor.Builder()
+ .name("expansions")
+ .displayName("Expansions")
+ .description("A comma-separated list of expansions for objects in the returned tweet. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/expansions " +
+ "for proper usage. Possible field values are: " +
+ "author_id, referenced_tweets.id, referenced_tweets.id.author_id, entities.mentions.username, " +
+ "attachments.poll_ids, attachments.media_keys ,in_reply_to_user_id, geo.place_id")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor BACKFILL_MINUTES = new PropertyDescriptor.Builder()
+ .name("backfill-minutes")
+ .displayName("Backfill Minutes")
+ .description("The number of minutes (up to 5 minutes) of streaming data to be requested after a " +
+ "disconnect. Only available for project with academic research access. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/integrate/" +
+ "recovery-and-redundancy-features")
+ .required(true)
+ .defaultValue("0")
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("All tweets will be routed to this relationship.")
+ .build();
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+
+ private TwitterStreamAPI api;
+
+ private volatile BlockingQueue<String> messageQueue;
+
+ private void emptyQueue() {
+ while (!messageQueue.isEmpty()) {
+ messageQueue.poll();
+ }
+ }
+
+ @Override
+ protected void init(ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptors = new ArrayList<>();
+ descriptors.add(ENDPOINT);
+ descriptors.add(BASE_PATH);
+ descriptors.add(BEARER_TOKEN);
+ descriptors.add(QUEUE_SIZE);
+ descriptors.add(BATCH_SIZE);
+ descriptors.add(TWEET_FIELDS);
+ descriptors.add(USER_FIELDS);
+ descriptors.add(MEDIA_FIELDS);
+ descriptors.add(POLL_FIELDS);
+ descriptors.add(PLACE_FIELDS);
+ descriptors.add(EXPANSIONS);
+ descriptors.add(BACKFILL_MINUTES);
+
+ this.descriptors = Collections.unmodifiableList(descriptors);
+
+ final Set<Relationship> relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ this.relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return this.descriptors;
+ }
+
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ messageQueue = new LinkedBlockingQueue<>(context.getProperty(QUEUE_SIZE).asInteger());
+
+ api = new TwitterStreamAPI(context, messageQueue, getLogger());
+ api.setBasePath(context.getProperty(BASE_PATH).getValue());
+ final String endpointName = context.getProperty(ENDPOINT).getValue();
+ if (ENDPOINT_SAMPLE.getValue().equals(endpointName)) {
+ api.start(TwitterStreamAPI.SAMPLE_ENDPOINT);
+ } else if (ENDPOINT_SEARCH.getValue().equals(endpointName)) {
+ api.start(TwitterStreamAPI.SEARCH_ENDPOINT);
+ } else {
+ throw new ProcessException("Endpoint was invalid value: " + endpointName);
+ }
+ }
+
+ @Override
+ public void onTrigger(ProcessContext context, ProcessSession session) throws ProcessException {
+ if (messageQueue.isEmpty()) {
+ context.yield();
+ return;
+ }
+
+ FlowFile flowFile = session.create();
+ flowFile = session.write(flowFile, new OutputStreamCallback() {
+ @Override
+ public void process(OutputStream out) throws IOException {
+ out.write("[".getBytes(StandardCharsets.UTF_8));
+ final int batchSize = context.getProperty(BATCH_SIZE).asInteger();
+ String tweet = messageQueue.poll();
+ out.write(tweet.getBytes(StandardCharsets.UTF_8));
+ int tweetCount = 1;
+ while (tweetCount < batchSize && !messageQueue.isEmpty()) {
+ out.write(",".getBytes(StandardCharsets.UTF_8));
Review Comment:
```suggestion
out.write(',');
```
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TwitterStreamAPI.java:
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+public class TwitterStreamAPI {
+ public static final String SEARCH_ENDPOINT = "Search Endpoint";
+ public static final String SAMPLE_ENDPOINT = "Sample Endpoint";
+ public static final String SAMPLE_PATH = "/2/tweets/sample/stream";
+ public static final String SEARCH_PATH = "/2/tweets/search/stream";
+
+ private static final String BEARER_TOKEN_PROPERTY_NAME = "bearer-token";
+ private static final String TWEET_FIELDS_PROPERTY_NAME = "tweet-fields";
+ private static final String USER_FIELDS_PROPERTY_NAME = "user-fields";
+ private static final String MEDIA_FIELDS_PROPERTY_NAME = "media-fields";
+ private static final String POLL_FIELDS_PROPERTY_NAME = "poll-fields";
+ private static final String PLACE_FIELDS_PROPERTY_NAME = "place-fields";
+ private static final String EXPANSIONS_PROPERTY_NAME = "expansions";
+ private static final String BACKFILL_MINUTES_PROPERTY_NAME = "backfill-minutes";
+
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Set<String> parseCommaSeparatedListPropreties(final ProcessContext context, final String propertyName) {
+ Set<String> fields = null;
+ if (context.getProperty(propertyName).isSet()) {
+ fields = new HashSet<>();
+ final String fieldsString = context.getProperty(propertyName).getValue();
+ for (final String field: fieldsString.split(",")) {
+ fields.add(field.trim());
+ }
+ }
+ return fields;
+ }
+
+ public TwitterStreamAPI(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ assert context != null;
+ assert queue != null;
+ assert logger != null;
+
+ this.queue = queue;
+ this.logger = logger;
+
+ this.tweetFields = parseCommaSeparatedListPropreties(context, TWEET_FIELDS_PROPERTY_NAME);
+ this.userFields = parseCommaSeparatedListPropreties(context, USER_FIELDS_PROPERTY_NAME);
+ this.mediaFields = parseCommaSeparatedListPropreties(context, MEDIA_FIELDS_PROPERTY_NAME);
+ this.pollFields = parseCommaSeparatedListPropreties(context, POLL_FIELDS_PROPERTY_NAME);
+ this.placeFields = parseCommaSeparatedListPropreties(context, PLACE_FIELDS_PROPERTY_NAME);
+ this.expansions = parseCommaSeparatedListPropreties(context, EXPANSIONS_PROPERTY_NAME);
+ this.backfillMinutes = context.getProperty(BACKFILL_MINUTES_PROPERTY_NAME).asInteger();
+
+ TwitterCredentialsBearer creds = new TwitterCredentialsBearer(context.getProperty(BEARER_TOKEN_PROPERTY_NAME).getValue());
+ api = new TwitterApi();
+ api.setTwitterCredentials(creds);
+
+ this.executorService = Executors.newSingleThreadExecutor();
+ }
+
+ public String getBasePath() {
+ return api.getApiClient().getBasePath();
+ }
+
+ public void setBasePath(final String path) {
+ api.getApiClient().setBasePath(path);
+ }
+
+ /**
+ * This method would be called when we would like the stream to get started. This method will spin off a thread that
+ * will continue to queue tweets on to the given queue passed in the constructor. The thread will continue
+ * to run until {@code stop} is called.
+ * @param endpoint {@code TwitterStreamAPI.SAMPLE_ENDPOINT} or {@code TwitterStreamAPI.SEARCH_ENDPOINT}
+ */
+ public void start(final String endpoint) {
+ try {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ stream = api.tweets().sampleStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else if (endpoint.equals(SEARCH_ENDPOINT)) {
+ stream = api.tweets().searchStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else {
+ throw new ProcessException("Endpoint was invalid value: " + endpoint);
+ }
+ } catch (ApiException e) {
+ logger.error("Received error {}: {}", e.getCode(), e.getMessage());
Review Comment:
Instead of logging an error, the details should be wrapped and thrown in a new `ProcessException`, since processing cannot continue if the initial configuration fails.
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,318 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("consume-twitter-endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("The size of the queue used to queue tweets")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("tweet-fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values are: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("user-fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user " +
+ "for proper usage. Possible field values are: " +
+ "created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, " +
+ "protected, public_metrics, url, username, verified, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor MEDIA_FIELDS = new PropertyDescriptor.Builder()
+ .name("media-fields")
+ .displayName("Media Fields")
+ .description("A comma-separated list of media fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/media " +
+ "for proper usage. Possible field values are: " +
+ "alt_text, duration_ms, height, media_key, non_public_metrics, organic_metrics, preview_image_url, " +
+ "promoted_metrics, public_metrics, type, url, width")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor POLL_FIELDS = new PropertyDescriptor.Builder()
+ .name("poll-fields")
+ .displayName("Poll Fields")
+ .description("A comma-separated list of poll fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/poll " +
+ "for proper usage. Possible field values are: " +
+ "duration_minutes, end_datetime, id, options, voting_status")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PLACE_FIELDS = new PropertyDescriptor.Builder()
+ .name("place-fields")
+ .displayName("Place Fields")
+ .description("A comma-separated list of place fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/place " +
+ "for proper usage. Possible field values are: " +
+ "contained_within, country, country_code, full_name, geo, id, name, place_type")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor EXPANSIONS = new PropertyDescriptor.Builder()
+ .name("expansions")
+ .displayName("Expansions")
+ .description("A comma-separated list of expansions for objects in the returned tweet. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/expansions " +
+ "for proper usage. Possible field values are: " +
+ "author_id, referenced_tweets.id, referenced_tweets.id.author_id, entities.mentions.username, " +
+ "attachments.poll_ids, attachments.media_keys ,in_reply_to_user_id, geo.place_id")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor BACKFILL_MINUTES = new PropertyDescriptor.Builder()
+ .name("backfill-minutes")
+ .displayName("Backfill Minutes")
+ .description("The number of minutes (up to 5 minutes) of streaming data to be requested after a " +
+ "disconnect. Only available for project with academic research access. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/integrate/" +
+ "recovery-and-redundancy-features")
+ .required(true)
+ .defaultValue("0")
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("All tweets will be routed to this relationship.")
+ .build();
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+
+ private TwitterStreamAPI api;
+
+ private volatile BlockingQueue<String> messageQueue;
+
+ private void emptyQueue() {
+ while (!messageQueue.isEmpty()) {
+ messageQueue.poll();
+ }
+ }
+
+ @Override
+ protected void init(ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptors = new ArrayList<>();
+ descriptors.add(ENDPOINT);
+ descriptors.add(BASE_PATH);
+ descriptors.add(BEARER_TOKEN);
+ descriptors.add(QUEUE_SIZE);
+ descriptors.add(BATCH_SIZE);
+ descriptors.add(TWEET_FIELDS);
+ descriptors.add(USER_FIELDS);
+ descriptors.add(MEDIA_FIELDS);
+ descriptors.add(POLL_FIELDS);
+ descriptors.add(PLACE_FIELDS);
+ descriptors.add(EXPANSIONS);
+ descriptors.add(BACKFILL_MINUTES);
+
+ this.descriptors = Collections.unmodifiableList(descriptors);
+
+ final Set<Relationship> relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ this.relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return this.descriptors;
+ }
+
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ messageQueue = new LinkedBlockingQueue<>(context.getProperty(QUEUE_SIZE).asInteger());
+
+ api = new TwitterStreamAPI(context, messageQueue, getLogger());
+ api.setBasePath(context.getProperty(BASE_PATH).getValue());
+ final String endpointName = context.getProperty(ENDPOINT).getValue();
+ if (ENDPOINT_SAMPLE.getValue().equals(endpointName)) {
+ api.start(TwitterStreamAPI.SAMPLE_ENDPOINT);
+ } else if (ENDPOINT_SEARCH.getValue().equals(endpointName)) {
+ api.start(TwitterStreamAPI.SEARCH_ENDPOINT);
+ } else {
+ throw new ProcessException("Endpoint was invalid value: " + endpointName);
+ }
+ }
+
+ @Override
+ public void onTrigger(ProcessContext context, ProcessSession session) throws ProcessException {
+ if (messageQueue.isEmpty()) {
+ context.yield();
+ return;
+ }
+
+ FlowFile flowFile = session.create();
+ flowFile = session.write(flowFile, new OutputStreamCallback() {
+ @Override
+ public void process(OutputStream out) throws IOException {
+ out.write("[".getBytes(StandardCharsets.UTF_8));
+ final int batchSize = context.getProperty(BATCH_SIZE).asInteger();
+ String tweet = messageQueue.poll();
+ out.write(tweet.getBytes(StandardCharsets.UTF_8));
+ int tweetCount = 1;
+ while (tweetCount < batchSize && !messageQueue.isEmpty()) {
+ out.write(",".getBytes(StandardCharsets.UTF_8));
+ tweet = messageQueue.poll();
+ out.write(tweet.getBytes(StandardCharsets.UTF_8));
+ tweetCount++;
+ }
+ out.write("]".getBytes(StandardCharsets.UTF_8));
Review Comment:
```suggestion
out.write(']');
```
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,318 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("consume-twitter-endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("The size of the queue used to queue tweets")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("tweet-fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values are: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("user-fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user " +
+ "for proper usage. Possible field values are: " +
+ "created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, " +
+ "protected, public_metrics, url, username, verified, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor MEDIA_FIELDS = new PropertyDescriptor.Builder()
+ .name("media-fields")
+ .displayName("Media Fields")
+ .description("A comma-separated list of media fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/media " +
+ "for proper usage. Possible field values are: " +
+ "alt_text, duration_ms, height, media_key, non_public_metrics, organic_metrics, preview_image_url, " +
+ "promoted_metrics, public_metrics, type, url, width")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor POLL_FIELDS = new PropertyDescriptor.Builder()
+ .name("poll-fields")
+ .displayName("Poll Fields")
+ .description("A comma-separated list of poll fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/poll " +
+ "for proper usage. Possible field values are: " +
+ "duration_minutes, end_datetime, id, options, voting_status")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PLACE_FIELDS = new PropertyDescriptor.Builder()
+ .name("place-fields")
+ .displayName("Place Fields")
+ .description("A comma-separated list of place fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/place " +
+ "for proper usage. Possible field values are: " +
+ "contained_within, country, country_code, full_name, geo, id, name, place_type")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor EXPANSIONS = new PropertyDescriptor.Builder()
+ .name("expansions")
+ .displayName("Expansions")
+ .description("A comma-separated list of expansions for objects in the returned tweet. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/expansions " +
+ "for proper usage. Possible field values are: " +
+ "author_id, referenced_tweets.id, referenced_tweets.id.author_id, entities.mentions.username, " +
+ "attachments.poll_ids, attachments.media_keys ,in_reply_to_user_id, geo.place_id")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor BACKFILL_MINUTES = new PropertyDescriptor.Builder()
+ .name("backfill-minutes")
+ .displayName("Backfill Minutes")
+ .description("The number of minutes (up to 5 minutes) of streaming data to be requested after a " +
+ "disconnect. Only available for project with academic research access. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/integrate/" +
+ "recovery-and-redundancy-features")
+ .required(true)
+ .defaultValue("0")
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("All tweets will be routed to this relationship.")
+ .build();
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+
+ private TwitterStreamAPI api;
+
+ private volatile BlockingQueue<String> messageQueue;
+
+ private void emptyQueue() {
+ while (!messageQueue.isEmpty()) {
+ messageQueue.poll();
+ }
+ }
+
+ @Override
+ protected void init(ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptors = new ArrayList<>();
+ descriptors.add(ENDPOINT);
+ descriptors.add(BASE_PATH);
+ descriptors.add(BEARER_TOKEN);
+ descriptors.add(QUEUE_SIZE);
+ descriptors.add(BATCH_SIZE);
+ descriptors.add(TWEET_FIELDS);
+ descriptors.add(USER_FIELDS);
+ descriptors.add(MEDIA_FIELDS);
+ descriptors.add(POLL_FIELDS);
+ descriptors.add(PLACE_FIELDS);
+ descriptors.add(EXPANSIONS);
+ descriptors.add(BACKFILL_MINUTES);
+
+ this.descriptors = Collections.unmodifiableList(descriptors);
+
+ final Set<Relationship> relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ this.relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return this.descriptors;
+ }
+
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ messageQueue = new LinkedBlockingQueue<>(context.getProperty(QUEUE_SIZE).asInteger());
+
+ api = new TwitterStreamAPI(context, messageQueue, getLogger());
+ api.setBasePath(context.getProperty(BASE_PATH).getValue());
+ final String endpointName = context.getProperty(ENDPOINT).getValue();
+ if (ENDPOINT_SAMPLE.getValue().equals(endpointName)) {
+ api.start(TwitterStreamAPI.SAMPLE_ENDPOINT);
+ } else if (ENDPOINT_SEARCH.getValue().equals(endpointName)) {
+ api.start(TwitterStreamAPI.SEARCH_ENDPOINT);
+ } else {
+ throw new ProcessException("Endpoint was invalid value: " + endpointName);
+ }
+ }
+
+ @Override
+ public void onTrigger(ProcessContext context, ProcessSession session) throws ProcessException {
+ if (messageQueue.isEmpty()) {
+ context.yield();
+ return;
+ }
+
+ FlowFile flowFile = session.create();
+ flowFile = session.write(flowFile, new OutputStreamCallback() {
+ @Override
+ public void process(OutputStream out) throws IOException {
+ out.write("[".getBytes(StandardCharsets.UTF_8));
+ final int batchSize = context.getProperty(BATCH_SIZE).asInteger();
+ String tweet = messageQueue.poll();
+ out.write(tweet.getBytes(StandardCharsets.UTF_8));
+ int tweetCount = 1;
+ while (tweetCount < batchSize && !messageQueue.isEmpty()) {
+ out.write(",".getBytes(StandardCharsets.UTF_8));
+ tweet = messageQueue.poll();
+ out.write(tweet.getBytes(StandardCharsets.UTF_8));
+ tweetCount++;
+ }
+ out.write("]".getBytes(StandardCharsets.UTF_8));
+ }
+ });
+
+ final Map<String, String> attributes = new HashMap<>();
+ attributes.put(CoreAttributes.MIME_TYPE.key(), "application/json");
+ attributes.put(CoreAttributes.FILENAME.key(), flowFile.getAttribute(CoreAttributes.FILENAME.key()) + ".json");
+ flowFile = session.putAllAttributes(flowFile, attributes);
+
+ session.transfer(flowFile, REL_SUCCESS);
+
+ String transitUri = api.getBasePath();
+ final String endpointName = context.getProperty(ENDPOINT).getValue();
+ if (ENDPOINT_SAMPLE.getValue().equals(endpointName)) {
+ transitUri += TwitterStreamAPI.SAMPLE_PATH;
+ } else if (ENDPOINT_SEARCH.getValue().equals(endpointName)) {
+ transitUri += TwitterStreamAPI.SEARCH_PATH;
+ } else {
+ throw new ProcessException("Endpoint was invalid value: " + endpointName);
+ }
Review Comment:
It seems like it would be simpler to create a new method on `TwitterStreamAPI` named `getTransitUri()`, which would centralize the logic.
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TwitterStreamAPI.java:
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+public class TwitterStreamAPI {
+ public static final String SEARCH_ENDPOINT = "Search Endpoint";
+ public static final String SAMPLE_ENDPOINT = "Sample Endpoint";
+ public static final String SAMPLE_PATH = "/2/tweets/sample/stream";
+ public static final String SEARCH_PATH = "/2/tweets/search/stream";
+
+ private static final String BEARER_TOKEN_PROPERTY_NAME = "bearer-token";
+ private static final String TWEET_FIELDS_PROPERTY_NAME = "tweet-fields";
+ private static final String USER_FIELDS_PROPERTY_NAME = "user-fields";
+ private static final String MEDIA_FIELDS_PROPERTY_NAME = "media-fields";
+ private static final String POLL_FIELDS_PROPERTY_NAME = "poll-fields";
+ private static final String PLACE_FIELDS_PROPERTY_NAME = "place-fields";
+ private static final String EXPANSIONS_PROPERTY_NAME = "expansions";
+ private static final String BACKFILL_MINUTES_PROPERTY_NAME = "backfill-minutes";
+
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Set<String> parseCommaSeparatedListPropreties(final ProcessContext context, final String propertyName) {
+ Set<String> fields = null;
+ if (context.getProperty(propertyName).isSet()) {
+ fields = new HashSet<>();
+ final String fieldsString = context.getProperty(propertyName).getValue();
+ for (final String field: fieldsString.split(",")) {
+ fields.add(field.trim());
+ }
+ }
+ return fields;
+ }
+
+ public TwitterStreamAPI(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ assert context != null;
+ assert queue != null;
+ assert logger != null;
+
+ this.queue = queue;
+ this.logger = logger;
+
+ this.tweetFields = parseCommaSeparatedListPropreties(context, TWEET_FIELDS_PROPERTY_NAME);
+ this.userFields = parseCommaSeparatedListPropreties(context, USER_FIELDS_PROPERTY_NAME);
+ this.mediaFields = parseCommaSeparatedListPropreties(context, MEDIA_FIELDS_PROPERTY_NAME);
+ this.pollFields = parseCommaSeparatedListPropreties(context, POLL_FIELDS_PROPERTY_NAME);
+ this.placeFields = parseCommaSeparatedListPropreties(context, PLACE_FIELDS_PROPERTY_NAME);
+ this.expansions = parseCommaSeparatedListPropreties(context, EXPANSIONS_PROPERTY_NAME);
+ this.backfillMinutes = context.getProperty(BACKFILL_MINUTES_PROPERTY_NAME).asInteger();
+
+ TwitterCredentialsBearer creds = new TwitterCredentialsBearer(context.getProperty(BEARER_TOKEN_PROPERTY_NAME).getValue());
+ api = new TwitterApi();
+ api.setTwitterCredentials(creds);
+
+ this.executorService = Executors.newSingleThreadExecutor();
+ }
+
+ public String getBasePath() {
+ return api.getApiClient().getBasePath();
+ }
+
+ public void setBasePath(final String path) {
+ api.getApiClient().setBasePath(path);
+ }
+
+ /**
+ * This method would be called when we would like the stream to get started. This method will spin off a thread that
+ * will continue to queue tweets on to the given queue passed in the constructor. The thread will continue
+ * to run until {@code stop} is called.
+ * @param endpoint {@code TwitterStreamAPI.SAMPLE_ENDPOINT} or {@code TwitterStreamAPI.SEARCH_ENDPOINT}
+ */
+ public void start(final String endpoint) {
+ try {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ stream = api.tweets().sampleStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else if (endpoint.equals(SEARCH_ENDPOINT)) {
+ stream = api.tweets().searchStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else {
+ throw new ProcessException("Endpoint was invalid value: " + endpoint);
+ }
+ } catch (ApiException e) {
+ logger.error("Received error {}: {}", e.getCode(), e.getMessage());
+ }
+
+ executorService.execute(new TweetQueuer());
+ }
+
+ /**
+ * This method would be called when we would like the stream to get stopped. The stream will be closed and the
+ * executorService will be shut down. If it fails to shutdown, then it will be forcefully terminated.
+ */
+ public void stop() {
+ try {
+ stream.close();
+ } catch (IOException e) {
+ logger.error("IOException occurred while closing stream: {}", e.getMessage());
Review Comment:
Instead of naming the exception itself, recommend simplifying the message and include the exception to provide the full stack trace:
```suggestion
logger.error("Closing response stream failed", e);
```
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,318 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("consume-twitter-endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("The size of the queue used to queue tweets")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("tweet-fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values are: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("user-fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user " +
+ "for proper usage. Possible field values are: " +
+ "created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, " +
+ "protected, public_metrics, url, username, verified, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor MEDIA_FIELDS = new PropertyDescriptor.Builder()
+ .name("media-fields")
+ .displayName("Media Fields")
+ .description("A comma-separated list of media fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/media " +
+ "for proper usage. Possible field values are: " +
+ "alt_text, duration_ms, height, media_key, non_public_metrics, organic_metrics, preview_image_url, " +
+ "promoted_metrics, public_metrics, type, url, width")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor POLL_FIELDS = new PropertyDescriptor.Builder()
+ .name("poll-fields")
+ .displayName("Poll Fields")
+ .description("A comma-separated list of poll fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/poll " +
+ "for proper usage. Possible field values are: " +
+ "duration_minutes, end_datetime, id, options, voting_status")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PLACE_FIELDS = new PropertyDescriptor.Builder()
+ .name("place-fields")
+ .displayName("Place Fields")
+ .description("A comma-separated list of place fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/place " +
+ "for proper usage. Possible field values are: " +
+ "contained_within, country, country_code, full_name, geo, id, name, place_type")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor EXPANSIONS = new PropertyDescriptor.Builder()
+ .name("expansions")
+ .displayName("Expansions")
+ .description("A comma-separated list of expansions for objects in the returned tweet. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/expansions " +
+ "for proper usage. Possible field values are: " +
+ "author_id, referenced_tweets.id, referenced_tweets.id.author_id, entities.mentions.username, " +
+ "attachments.poll_ids, attachments.media_keys ,in_reply_to_user_id, geo.place_id")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor BACKFILL_MINUTES = new PropertyDescriptor.Builder()
+ .name("backfill-minutes")
+ .displayName("Backfill Minutes")
+ .description("The number of minutes (up to 5 minutes) of streaming data to be requested after a " +
+ "disconnect. Only available for project with academic research access. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/integrate/" +
+ "recovery-and-redundancy-features")
+ .required(true)
+ .defaultValue("0")
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("All tweets will be routed to this relationship.")
+ .build();
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+
+ private TwitterStreamAPI api;
+
+ private volatile BlockingQueue<String> messageQueue;
+
+ private void emptyQueue() {
+ while (!messageQueue.isEmpty()) {
+ messageQueue.poll();
+ }
+ }
Review Comment:
Recommend moving this method to the end of the class following standard ordering of public before private methods.
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,318 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("consume-twitter-endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("The size of the queue used to queue tweets")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("tweet-fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values are: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("user-fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user " +
+ "for proper usage. Possible field values are: " +
+ "created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, " +
+ "protected, public_metrics, url, username, verified, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor MEDIA_FIELDS = new PropertyDescriptor.Builder()
+ .name("media-fields")
+ .displayName("Media Fields")
+ .description("A comma-separated list of media fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/media " +
+ "for proper usage. Possible field values are: " +
+ "alt_text, duration_ms, height, media_key, non_public_metrics, organic_metrics, preview_image_url, " +
+ "promoted_metrics, public_metrics, type, url, width")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor POLL_FIELDS = new PropertyDescriptor.Builder()
+ .name("poll-fields")
+ .displayName("Poll Fields")
+ .description("A comma-separated list of poll fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/poll " +
+ "for proper usage. Possible field values are: " +
+ "duration_minutes, end_datetime, id, options, voting_status")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PLACE_FIELDS = new PropertyDescriptor.Builder()
+ .name("place-fields")
+ .displayName("Place Fields")
+ .description("A comma-separated list of place fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/place " +
+ "for proper usage. Possible field values are: " +
+ "contained_within, country, country_code, full_name, geo, id, name, place_type")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor EXPANSIONS = new PropertyDescriptor.Builder()
+ .name("expansions")
+ .displayName("Expansions")
+ .description("A comma-separated list of expansions for objects in the returned tweet. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/expansions " +
+ "for proper usage. Possible field values are: " +
+ "author_id, referenced_tweets.id, referenced_tweets.id.author_id, entities.mentions.username, " +
+ "attachments.poll_ids, attachments.media_keys ,in_reply_to_user_id, geo.place_id")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor BACKFILL_MINUTES = new PropertyDescriptor.Builder()
+ .name("backfill-minutes")
+ .displayName("Backfill Minutes")
+ .description("The number of minutes (up to 5 minutes) of streaming data to be requested after a " +
+ "disconnect. Only available for project with academic research access. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/integrate/" +
+ "recovery-and-redundancy-features")
+ .required(true)
+ .defaultValue("0")
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("All tweets will be routed to this relationship.")
+ .build();
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+
+ private TwitterStreamAPI api;
+
+ private volatile BlockingQueue<String> messageQueue;
+
+ private void emptyQueue() {
+ while (!messageQueue.isEmpty()) {
+ messageQueue.poll();
+ }
+ }
+
+ @Override
+ protected void init(ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptors = new ArrayList<>();
+ descriptors.add(ENDPOINT);
+ descriptors.add(BASE_PATH);
+ descriptors.add(BEARER_TOKEN);
+ descriptors.add(QUEUE_SIZE);
+ descriptors.add(BATCH_SIZE);
+ descriptors.add(TWEET_FIELDS);
+ descriptors.add(USER_FIELDS);
+ descriptors.add(MEDIA_FIELDS);
+ descriptors.add(POLL_FIELDS);
+ descriptors.add(PLACE_FIELDS);
+ descriptors.add(EXPANSIONS);
+ descriptors.add(BACKFILL_MINUTES);
+
+ this.descriptors = Collections.unmodifiableList(descriptors);
+
+ final Set<Relationship> relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ this.relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return this.descriptors;
+ }
+
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ messageQueue = new LinkedBlockingQueue<>(context.getProperty(QUEUE_SIZE).asInteger());
+
+ api = new TwitterStreamAPI(context, messageQueue, getLogger());
+ api.setBasePath(context.getProperty(BASE_PATH).getValue());
+ final String endpointName = context.getProperty(ENDPOINT).getValue();
+ if (ENDPOINT_SAMPLE.getValue().equals(endpointName)) {
+ api.start(TwitterStreamAPI.SAMPLE_ENDPOINT);
+ } else if (ENDPOINT_SEARCH.getValue().equals(endpointName)) {
+ api.start(TwitterStreamAPI.SEARCH_ENDPOINT);
+ } else {
+ throw new ProcessException("Endpoint was invalid value: " + endpointName);
+ }
+ }
+
+ @Override
+ public void onTrigger(ProcessContext context, ProcessSession session) throws ProcessException {
+ if (messageQueue.isEmpty()) {
+ context.yield();
+ return;
+ }
+
+ FlowFile flowFile = session.create();
+ flowFile = session.write(flowFile, new OutputStreamCallback() {
+ @Override
+ public void process(OutputStream out) throws IOException {
+ out.write("[".getBytes(StandardCharsets.UTF_8));
+ final int batchSize = context.getProperty(BATCH_SIZE).asInteger();
+ String tweet = messageQueue.poll();
+ out.write(tweet.getBytes(StandardCharsets.UTF_8));
+ int tweetCount = 1;
+ while (tweetCount < batchSize && !messageQueue.isEmpty()) {
+ out.write(",".getBytes(StandardCharsets.UTF_8));
+ tweet = messageQueue.poll();
Review Comment:
`Queue.poll()` can return `null` when the queue is empty, so the logic in this loop should be adjusted. Checking `tweet` for `null` instead of `messageQueue.isEmpty()` seems like one option.
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/test/java/org/apache/nifi/processors/twitter/TestConsumeTwitter.java:
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import okhttp3.mockwebserver.MockResponse;
+import okhttp3.mockwebserver.MockWebServer;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+
+public class TestConsumeTwitter {
+ private MockWebServer mockWebServer;
+
+ private TestRunner runner;
+
+ private final String SAMPLE_TWEET = "{\"data\":{\"id\":\"123\",\"text\":\"This is a sample tweet and is not real!\"}}";
+
+ @BeforeEach
+ public void setRunnerAndAPI() {
+ mockWebServer = new MockWebServer();
+
+ runner = TestRunners.newTestRunner(ConsumeTwitter.class);
+
+ runner.setProperty(ConsumeTwitter.BEARER_TOKEN, "BEARER_TOKEN");
+ final String basePath = mockWebServer.url("").toString();
+ runner.setProperty(ConsumeTwitter.BASE_PATH, basePath);
+ }
+
+ @AfterEach
+ public void shutdownServerAndAPI() throws IOException {
+ mockWebServer.shutdown();
+ }
+
+ @Test
+ public void testReceiveSingleTweetInStream() {
+ MockResponse response = new MockResponse()
+ .setResponseCode(200)
+ .setBody(SAMPLE_TWEET)
+ .addHeader("Content-Type", "application/json");
+ mockWebServer.enqueue(response);
+
+
+ runner.setProperty(ConsumeTwitter.ENDPOINT, ConsumeTwitter.ENDPOINT_SAMPLE);
+ runner.setProperty(ConsumeTwitter.QUEUE_SIZE, "10000");
+ runner.setProperty(ConsumeTwitter.BATCH_SIZE, "10");
+ runner.setProperty(ConsumeTwitter.BACKFILL_MINUTES, "0");
+
+ runner.assertValid();
+
+ // the TwitterStreamAPI class spins up another thread and might not be done queueing tweets in one run of the processor
+ final int maxTries = 100;
+ int tries = 0;
+ while (tries < maxTries && runner.getFlowFilesForRelationship(ConsumeTwitter.REL_SUCCESS).size() == 0) {
+ runner.run();
+ tries++;
+ }
+
+ // there should only be a single FlowFile containing a tweet
+ runner.assertTransferCount(ConsumeTwitter.REL_SUCCESS, 1);
+ MockFlowFile flowFile = runner.getFlowFilesForRelationship(ConsumeTwitter.REL_SUCCESS).get(0);
+ flowFile.assertContentEquals("[" + SAMPLE_TWEET + "]");
+ flowFile.assertAttributeEquals("mime.type", "application/json");
Review Comment:
The `mime.type` String can be replaced with `CoreAttributes`.
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/test/java/org/apache/nifi/processors/twitter/TestConsumeTwitter.java:
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import okhttp3.mockwebserver.MockResponse;
+import okhttp3.mockwebserver.MockWebServer;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+
+public class TestConsumeTwitter {
+ private MockWebServer mockWebServer;
+
+ private TestRunner runner;
+
+ private final String SAMPLE_TWEET = "{\"data\":{\"id\":\"123\",\"text\":\"This is a sample tweet and is not real!\"}}";
+
+ @BeforeEach
+ public void setRunnerAndAPI() {
+ mockWebServer = new MockWebServer();
+
+ runner = TestRunners.newTestRunner(ConsumeTwitter.class);
+
+ runner.setProperty(ConsumeTwitter.BEARER_TOKEN, "BEARER_TOKEN");
+ final String basePath = mockWebServer.url("").toString();
+ runner.setProperty(ConsumeTwitter.BASE_PATH, basePath);
+ }
+
+ @AfterEach
+ public void shutdownServerAndAPI() throws IOException {
+ mockWebServer.shutdown();
+ }
+
+ @Test
+ public void testReceiveSingleTweetInStream() {
+ MockResponse response = new MockResponse()
+ .setResponseCode(200)
+ .setBody(SAMPLE_TWEET)
+ .addHeader("Content-Type", "application/json");
+ mockWebServer.enqueue(response);
+
+
+ runner.setProperty(ConsumeTwitter.ENDPOINT, ConsumeTwitter.ENDPOINT_SAMPLE);
+ runner.setProperty(ConsumeTwitter.QUEUE_SIZE, "10000");
+ runner.setProperty(ConsumeTwitter.BATCH_SIZE, "10");
+ runner.setProperty(ConsumeTwitter.BACKFILL_MINUTES, "0");
+
+ runner.assertValid();
+
+ // the TwitterStreamAPI class spins up another thread and might not be done queueing tweets in one run of the processor
+ final int maxTries = 100;
+ int tries = 0;
+ while (tries < maxTries && runner.getFlowFilesForRelationship(ConsumeTwitter.REL_SUCCESS).size() == 0) {
+ runner.run();
+ tries++;
+ }
Review Comment:
Instead of introducing this loop, it should be possible to call `MockWebServer.takeRequest()`, which blocks until it receives something. At least that should be a starting point to adjust the approach.
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TwitterStreamAPI.java:
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+public class TwitterStreamAPI {
Review Comment:
Recommend renaming this class to something like `TweetStreamService`, since it is more narrowly focused on Tweets, as opposed to other Twitter capabilities.
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TwitterStreamAPI.java:
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+public class TwitterStreamAPI {
+ public static final String SEARCH_ENDPOINT = "Search Endpoint";
+ public static final String SAMPLE_ENDPOINT = "Sample Endpoint";
+ public static final String SAMPLE_PATH = "/2/tweets/sample/stream";
+ public static final String SEARCH_PATH = "/2/tweets/search/stream";
+
+ private static final String BEARER_TOKEN_PROPERTY_NAME = "bearer-token";
+ private static final String TWEET_FIELDS_PROPERTY_NAME = "tweet-fields";
+ private static final String USER_FIELDS_PROPERTY_NAME = "user-fields";
+ private static final String MEDIA_FIELDS_PROPERTY_NAME = "media-fields";
+ private static final String POLL_FIELDS_PROPERTY_NAME = "poll-fields";
+ private static final String PLACE_FIELDS_PROPERTY_NAME = "place-fields";
+ private static final String EXPANSIONS_PROPERTY_NAME = "expansions";
+ private static final String BACKFILL_MINUTES_PROPERTY_NAME = "backfill-minutes";
+
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Set<String> parseCommaSeparatedListPropreties(final ProcessContext context, final String propertyName) {
+ Set<String> fields = null;
+ if (context.getProperty(propertyName).isSet()) {
+ fields = new HashSet<>();
+ final String fieldsString = context.getProperty(propertyName).getValue();
+ for (final String field: fieldsString.split(",")) {
+ fields.add(field.trim());
+ }
+ }
+ return fields;
+ }
+
+ public TwitterStreamAPI(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ assert context != null;
+ assert queue != null;
+ assert logger != null;
+
+ this.queue = queue;
+ this.logger = logger;
+
+ this.tweetFields = parseCommaSeparatedListPropreties(context, TWEET_FIELDS_PROPERTY_NAME);
+ this.userFields = parseCommaSeparatedListPropreties(context, USER_FIELDS_PROPERTY_NAME);
+ this.mediaFields = parseCommaSeparatedListPropreties(context, MEDIA_FIELDS_PROPERTY_NAME);
+ this.pollFields = parseCommaSeparatedListPropreties(context, POLL_FIELDS_PROPERTY_NAME);
+ this.placeFields = parseCommaSeparatedListPropreties(context, PLACE_FIELDS_PROPERTY_NAME);
+ this.expansions = parseCommaSeparatedListPropreties(context, EXPANSIONS_PROPERTY_NAME);
+ this.backfillMinutes = context.getProperty(BACKFILL_MINUTES_PROPERTY_NAME).asInteger();
+
+ TwitterCredentialsBearer creds = new TwitterCredentialsBearer(context.getProperty(BEARER_TOKEN_PROPERTY_NAME).getValue());
+ api = new TwitterApi();
+ api.setTwitterCredentials(creds);
+
+ this.executorService = Executors.newSingleThreadExecutor();
+ }
+
+ public String getBasePath() {
+ return api.getApiClient().getBasePath();
+ }
+
+ public void setBasePath(final String path) {
+ api.getApiClient().setBasePath(path);
+ }
+
+ /**
+ * This method would be called when we would like the stream to get started. This method will spin off a thread that
+ * will continue to queue tweets on to the given queue passed in the constructor. The thread will continue
+ * to run until {@code stop} is called.
+ * @param endpoint {@code TwitterStreamAPI.SAMPLE_ENDPOINT} or {@code TwitterStreamAPI.SEARCH_ENDPOINT}
+ */
+ public void start(final String endpoint) {
+ try {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ stream = api.tweets().sampleStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else if (endpoint.equals(SEARCH_ENDPOINT)) {
+ stream = api.tweets().searchStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else {
+ throw new ProcessException("Endpoint was invalid value: " + endpoint);
+ }
+ } catch (ApiException e) {
+ logger.error("Received error {}: {}", e.getCode(), e.getMessage());
+ }
+
+ executorService.execute(new TweetQueuer());
+ }
+
+ /**
+ * This method would be called when we would like the stream to get stopped. The stream will be closed and the
+ * executorService will be shut down. If it fails to shutdown, then it will be forcefully terminated.
+ */
+ public void stop() {
+ try {
+ stream.close();
+ } catch (IOException e) {
+ logger.error("IOException occurred while closing stream: {}", e.getMessage());
+ }
+
+ executorService.shutdown();
+ try {
+ if (!executorService.awaitTermination(3, TimeUnit.SECONDS)) {
+ executorService.shutdownNow();
+ }
+ } catch (InterruptedException e) {
+ executorService.shutdownNow();
+ logger.error("InterruptedException occurred while waiting for executor service to shut down: {}", e.getMessage());
+ }
+ }
+
+ private class TweetQueuer implements Runnable {
+ @Override
+ public void run() {
+ try (BufferedReader reader = new BufferedReader(new InputStreamReader(stream))) {
+ String tweetLine = reader.readLine();
+ while (tweetLine != null) {
+ queue.offer(tweetLine);
+ try {
+ tweetLine = reader.readLine();
+ } catch (IOException e) {
+ logger.info("Stream closed, no more tweets to read, exiting Queuer");
Review Comment:
Recommend changing the log to debug and including the exception:
```suggestion
logger.debug("Read Tweet failed: Stream processing completed", e);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r848770776
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,318 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("consume-twitter-endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("The size of the queue used to queue tweets")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("tweet-fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values are: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("user-fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user " +
+ "for proper usage. Possible field values are: " +
+ "created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, " +
+ "protected, public_metrics, url, username, verified, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor MEDIA_FIELDS = new PropertyDescriptor.Builder()
+ .name("media-fields")
+ .displayName("Media Fields")
+ .description("A comma-separated list of media fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/media " +
+ "for proper usage. Possible field values are: " +
+ "alt_text, duration_ms, height, media_key, non_public_metrics, organic_metrics, preview_image_url, " +
+ "promoted_metrics, public_metrics, type, url, width")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor POLL_FIELDS = new PropertyDescriptor.Builder()
+ .name("poll-fields")
+ .displayName("Poll Fields")
+ .description("A comma-separated list of poll fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/poll " +
+ "for proper usage. Possible field values are: " +
+ "duration_minutes, end_datetime, id, options, voting_status")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PLACE_FIELDS = new PropertyDescriptor.Builder()
+ .name("place-fields")
+ .displayName("Place Fields")
+ .description("A comma-separated list of place fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/place " +
+ "for proper usage. Possible field values are: " +
+ "contained_within, country, country_code, full_name, geo, id, name, place_type")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor EXPANSIONS = new PropertyDescriptor.Builder()
+ .name("expansions")
+ .displayName("Expansions")
+ .description("A comma-separated list of expansions for objects in the returned tweet. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/expansions " +
+ "for proper usage. Possible field values are: " +
+ "author_id, referenced_tweets.id, referenced_tweets.id.author_id, entities.mentions.username, " +
+ "attachments.poll_ids, attachments.media_keys ,in_reply_to_user_id, geo.place_id")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor BACKFILL_MINUTES = new PropertyDescriptor.Builder()
+ .name("backfill-minutes")
+ .displayName("Backfill Minutes")
+ .description("The number of minutes (up to 5 minutes) of streaming data to be requested after a " +
+ "disconnect. Only available for project with academic research access. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/integrate/" +
+ "recovery-and-redundancy-features")
+ .required(true)
+ .defaultValue("0")
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("All tweets will be routed to this relationship.")
+ .build();
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+
+ private TwitterStreamAPI api;
+
+ private volatile BlockingQueue<String> messageQueue;
+
+ private void emptyQueue() {
+ while (!messageQueue.isEmpty()) {
+ messageQueue.poll();
+ }
+ }
+
+ @Override
+ protected void init(ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptors = new ArrayList<>();
+ descriptors.add(ENDPOINT);
+ descriptors.add(BASE_PATH);
+ descriptors.add(BEARER_TOKEN);
+ descriptors.add(QUEUE_SIZE);
+ descriptors.add(BATCH_SIZE);
+ descriptors.add(TWEET_FIELDS);
+ descriptors.add(USER_FIELDS);
+ descriptors.add(MEDIA_FIELDS);
+ descriptors.add(POLL_FIELDS);
+ descriptors.add(PLACE_FIELDS);
+ descriptors.add(EXPANSIONS);
+ descriptors.add(BACKFILL_MINUTES);
+
+ this.descriptors = Collections.unmodifiableList(descriptors);
+
+ final Set<Relationship> relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ this.relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return this.descriptors;
+ }
+
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ messageQueue = new LinkedBlockingQueue<>(context.getProperty(QUEUE_SIZE).asInteger());
+
+ api = new TwitterStreamAPI(context, messageQueue, getLogger());
+ api.setBasePath(context.getProperty(BASE_PATH).getValue());
+ final String endpointName = context.getProperty(ENDPOINT).getValue();
+ if (ENDPOINT_SAMPLE.getValue().equals(endpointName)) {
+ api.start(TwitterStreamAPI.SAMPLE_ENDPOINT);
+ } else if (ENDPOINT_SEARCH.getValue().equals(endpointName)) {
+ api.start(TwitterStreamAPI.SEARCH_ENDPOINT);
+ } else {
+ throw new ProcessException("Endpoint was invalid value: " + endpointName);
+ }
+ }
+
+ @Override
+ public void onTrigger(ProcessContext context, ProcessSession session) throws ProcessException {
+ if (messageQueue.isEmpty()) {
+ context.yield();
+ return;
+ }
+
+ FlowFile flowFile = session.create();
+ flowFile = session.write(flowFile, new OutputStreamCallback() {
+ @Override
+ public void process(OutputStream out) throws IOException {
+ out.write("[".getBytes(StandardCharsets.UTF_8));
+ final int batchSize = context.getProperty(BATCH_SIZE).asInteger();
+ String tweet = messageQueue.poll();
+ out.write(tweet.getBytes(StandardCharsets.UTF_8));
+ int tweetCount = 1;
+ while (tweetCount < batchSize && !messageQueue.isEmpty()) {
+ out.write(",".getBytes(StandardCharsets.UTF_8));
+ tweet = messageQueue.poll();
Review Comment:
I can implement that if I change the `isEmpty` condition to `(tweet = messageQueue.poll()) != null`. Making those changes!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r848779955
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TwitterStreamAPI.java:
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+public class TwitterStreamAPI {
+ public static final String SEARCH_ENDPOINT = "Search Endpoint";
+ public static final String SAMPLE_ENDPOINT = "Sample Endpoint";
+ public static final String SAMPLE_PATH = "/2/tweets/sample/stream";
+ public static final String SEARCH_PATH = "/2/tweets/search/stream";
+
+ private static final String BEARER_TOKEN_PROPERTY_NAME = "bearer-token";
+ private static final String TWEET_FIELDS_PROPERTY_NAME = "tweet-fields";
+ private static final String USER_FIELDS_PROPERTY_NAME = "user-fields";
+ private static final String MEDIA_FIELDS_PROPERTY_NAME = "media-fields";
+ private static final String POLL_FIELDS_PROPERTY_NAME = "poll-fields";
+ private static final String PLACE_FIELDS_PROPERTY_NAME = "place-fields";
+ private static final String EXPANSIONS_PROPERTY_NAME = "expansions";
+ private static final String BACKFILL_MINUTES_PROPERTY_NAME = "backfill-minutes";
+
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Set<String> parseCommaSeparatedListPropreties(final ProcessContext context, final String propertyName) {
+ Set<String> fields = null;
+ if (context.getProperty(propertyName).isSet()) {
+ fields = new HashSet<>();
+ final String fieldsString = context.getProperty(propertyName).getValue();
+ for (final String field: fieldsString.split(",")) {
+ fields.add(field.trim());
+ }
+ }
+ return fields;
+ }
+
+ public TwitterStreamAPI(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ assert context != null;
+ assert queue != null;
+ assert logger != null;
+
+ this.queue = queue;
+ this.logger = logger;
+
+ this.tweetFields = parseCommaSeparatedListPropreties(context, TWEET_FIELDS_PROPERTY_NAME);
+ this.userFields = parseCommaSeparatedListPropreties(context, USER_FIELDS_PROPERTY_NAME);
+ this.mediaFields = parseCommaSeparatedListPropreties(context, MEDIA_FIELDS_PROPERTY_NAME);
+ this.pollFields = parseCommaSeparatedListPropreties(context, POLL_FIELDS_PROPERTY_NAME);
+ this.placeFields = parseCommaSeparatedListPropreties(context, PLACE_FIELDS_PROPERTY_NAME);
+ this.expansions = parseCommaSeparatedListPropreties(context, EXPANSIONS_PROPERTY_NAME);
+ this.backfillMinutes = context.getProperty(BACKFILL_MINUTES_PROPERTY_NAME).asInteger();
+
+ TwitterCredentialsBearer creds = new TwitterCredentialsBearer(context.getProperty(BEARER_TOKEN_PROPERTY_NAME).getValue());
+ api = new TwitterApi();
+ api.setTwitterCredentials(creds);
+
+ this.executorService = Executors.newSingleThreadExecutor();
+ }
+
+ public String getBasePath() {
+ return api.getApiClient().getBasePath();
+ }
+
+ public void setBasePath(final String path) {
+ api.getApiClient().setBasePath(path);
+ }
+
+ /**
+ * This method would be called when we would like the stream to get started. This method will spin off a thread that
+ * will continue to queue tweets on to the given queue passed in the constructor. The thread will continue
+ * to run until {@code stop} is called.
+ * @param endpoint {@code TwitterStreamAPI.SAMPLE_ENDPOINT} or {@code TwitterStreamAPI.SEARCH_ENDPOINT}
+ */
+ public void start(final String endpoint) {
+ try {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ stream = api.tweets().sampleStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else if (endpoint.equals(SEARCH_ENDPOINT)) {
+ stream = api.tweets().searchStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else {
+ throw new ProcessException("Endpoint was invalid value: " + endpoint);
+ }
+ } catch (ApiException e) {
+ logger.error("Received error {}: {}", e.getCode(), e.getMessage());
+ }
+
+ executorService.execute(new TweetQueuer());
+ }
+
+ /**
+ * This method would be called when we would like the stream to get stopped. The stream will be closed and the
+ * executorService will be shut down. If it fails to shutdown, then it will be forcefully terminated.
+ */
+ public void stop() {
+ try {
+ stream.close();
+ } catch (IOException e) {
+ logger.error("IOException occurred while closing stream: {}", e.getMessage());
+ }
+
+ executorService.shutdown();
+ try {
+ if (!executorService.awaitTermination(3, TimeUnit.SECONDS)) {
+ executorService.shutdownNow();
+ }
+ } catch (InterruptedException e) {
+ executorService.shutdownNow();
+ logger.error("InterruptedException occurred while waiting for executor service to shut down: {}", e.getMessage());
+ }
+ }
+
+ private class TweetQueuer implements Runnable {
+ @Override
+ public void run() {
+ try (BufferedReader reader = new BufferedReader(new InputStreamReader(stream))) {
+ String tweetLine = reader.readLine();
Review Comment:
Agreed, making the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r848779345
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TwitterStreamAPI.java:
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+public class TwitterStreamAPI {
+ public static final String SEARCH_ENDPOINT = "Search Endpoint";
+ public static final String SAMPLE_ENDPOINT = "Sample Endpoint";
+ public static final String SAMPLE_PATH = "/2/tweets/sample/stream";
+ public static final String SEARCH_PATH = "/2/tweets/search/stream";
+
+ private static final String BEARER_TOKEN_PROPERTY_NAME = "bearer-token";
+ private static final String TWEET_FIELDS_PROPERTY_NAME = "tweet-fields";
+ private static final String USER_FIELDS_PROPERTY_NAME = "user-fields";
+ private static final String MEDIA_FIELDS_PROPERTY_NAME = "media-fields";
+ private static final String POLL_FIELDS_PROPERTY_NAME = "poll-fields";
+ private static final String PLACE_FIELDS_PROPERTY_NAME = "place-fields";
+ private static final String EXPANSIONS_PROPERTY_NAME = "expansions";
+ private static final String BACKFILL_MINUTES_PROPERTY_NAME = "backfill-minutes";
+
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Set<String> parseCommaSeparatedListPropreties(final ProcessContext context, final String propertyName) {
+ Set<String> fields = null;
+ if (context.getProperty(propertyName).isSet()) {
+ fields = new HashSet<>();
+ final String fieldsString = context.getProperty(propertyName).getValue();
+ for (final String field: fieldsString.split(",")) {
+ fields.add(field.trim());
+ }
+ }
+ return fields;
+ }
+
+ public TwitterStreamAPI(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ assert context != null;
+ assert queue != null;
+ assert logger != null;
+
+ this.queue = queue;
+ this.logger = logger;
+
+ this.tweetFields = parseCommaSeparatedListPropreties(context, TWEET_FIELDS_PROPERTY_NAME);
+ this.userFields = parseCommaSeparatedListPropreties(context, USER_FIELDS_PROPERTY_NAME);
+ this.mediaFields = parseCommaSeparatedListPropreties(context, MEDIA_FIELDS_PROPERTY_NAME);
+ this.pollFields = parseCommaSeparatedListPropreties(context, POLL_FIELDS_PROPERTY_NAME);
+ this.placeFields = parseCommaSeparatedListPropreties(context, PLACE_FIELDS_PROPERTY_NAME);
+ this.expansions = parseCommaSeparatedListPropreties(context, EXPANSIONS_PROPERTY_NAME);
+ this.backfillMinutes = context.getProperty(BACKFILL_MINUTES_PROPERTY_NAME).asInteger();
+
+ TwitterCredentialsBearer creds = new TwitterCredentialsBearer(context.getProperty(BEARER_TOKEN_PROPERTY_NAME).getValue());
+ api = new TwitterApi();
+ api.setTwitterCredentials(creds);
+
+ this.executorService = Executors.newSingleThreadExecutor();
+ }
+
+ public String getBasePath() {
+ return api.getApiClient().getBasePath();
+ }
+
+ public void setBasePath(final String path) {
+ api.getApiClient().setBasePath(path);
+ }
+
+ /**
+ * This method would be called when we would like the stream to get started. This method will spin off a thread that
+ * will continue to queue tweets on to the given queue passed in the constructor. The thread will continue
+ * to run until {@code stop} is called.
+ * @param endpoint {@code TwitterStreamAPI.SAMPLE_ENDPOINT} or {@code TwitterStreamAPI.SEARCH_ENDPOINT}
+ */
+ public void start(final String endpoint) {
+ try {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ stream = api.tweets().sampleStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else if (endpoint.equals(SEARCH_ENDPOINT)) {
+ stream = api.tweets().searchStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else {
+ throw new ProcessException("Endpoint was invalid value: " + endpoint);
+ }
+ } catch (ApiException e) {
+ logger.error("Received error {}: {}", e.getCode(), e.getMessage());
+ }
+
+ executorService.execute(new TweetQueuer());
+ }
+
+ /**
+ * This method would be called when we would like the stream to get stopped. The stream will be closed and the
+ * executorService will be shut down. If it fails to shutdown, then it will be forcefully terminated.
+ */
+ public void stop() {
+ try {
+ stream.close();
+ } catch (IOException e) {
+ logger.error("IOException occurred while closing stream: {}", e.getMessage());
Review Comment:
Making the change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r848794830
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TwitterStreamAPI.java:
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+public class TwitterStreamAPI {
+ public static final String SEARCH_ENDPOINT = "Search Endpoint";
+ public static final String SAMPLE_ENDPOINT = "Sample Endpoint";
+ public static final String SAMPLE_PATH = "/2/tweets/sample/stream";
+ public static final String SEARCH_PATH = "/2/tweets/search/stream";
+
+ private static final String BEARER_TOKEN_PROPERTY_NAME = "bearer-token";
+ private static final String TWEET_FIELDS_PROPERTY_NAME = "tweet-fields";
+ private static final String USER_FIELDS_PROPERTY_NAME = "user-fields";
+ private static final String MEDIA_FIELDS_PROPERTY_NAME = "media-fields";
+ private static final String POLL_FIELDS_PROPERTY_NAME = "poll-fields";
+ private static final String PLACE_FIELDS_PROPERTY_NAME = "place-fields";
+ private static final String EXPANSIONS_PROPERTY_NAME = "expansions";
+ private static final String BACKFILL_MINUTES_PROPERTY_NAME = "backfill-minutes";
+
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Set<String> parseCommaSeparatedListPropreties(final ProcessContext context, final String propertyName) {
+ Set<String> fields = null;
+ if (context.getProperty(propertyName).isSet()) {
+ fields = new HashSet<>();
+ final String fieldsString = context.getProperty(propertyName).getValue();
+ for (final String field: fieldsString.split(",")) {
+ fields.add(field.trim());
+ }
+ }
+ return fields;
+ }
+
+ public TwitterStreamAPI(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ assert context != null;
+ assert queue != null;
+ assert logger != null;
+
+ this.queue = queue;
+ this.logger = logger;
+
+ this.tweetFields = parseCommaSeparatedListPropreties(context, TWEET_FIELDS_PROPERTY_NAME);
+ this.userFields = parseCommaSeparatedListPropreties(context, USER_FIELDS_PROPERTY_NAME);
+ this.mediaFields = parseCommaSeparatedListPropreties(context, MEDIA_FIELDS_PROPERTY_NAME);
+ this.pollFields = parseCommaSeparatedListPropreties(context, POLL_FIELDS_PROPERTY_NAME);
+ this.placeFields = parseCommaSeparatedListPropreties(context, PLACE_FIELDS_PROPERTY_NAME);
+ this.expansions = parseCommaSeparatedListPropreties(context, EXPANSIONS_PROPERTY_NAME);
+ this.backfillMinutes = context.getProperty(BACKFILL_MINUTES_PROPERTY_NAME).asInteger();
+
+ TwitterCredentialsBearer creds = new TwitterCredentialsBearer(context.getProperty(BEARER_TOKEN_PROPERTY_NAME).getValue());
+ api = new TwitterApi();
+ api.setTwitterCredentials(creds);
+
+ this.executorService = Executors.newSingleThreadExecutor();
+ }
+
+ public String getBasePath() {
+ return api.getApiClient().getBasePath();
+ }
+
+ public void setBasePath(final String path) {
+ api.getApiClient().setBasePath(path);
+ }
+
+ /**
+ * This method would be called when we would like the stream to get started. This method will spin off a thread that
+ * will continue to queue tweets on to the given queue passed in the constructor. The thread will continue
+ * to run until {@code stop} is called.
+ * @param endpoint {@code TwitterStreamAPI.SAMPLE_ENDPOINT} or {@code TwitterStreamAPI.SEARCH_ENDPOINT}
+ */
+ public void start(final String endpoint) {
+ try {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ stream = api.tweets().sampleStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else if (endpoint.equals(SEARCH_ENDPOINT)) {
+ stream = api.tweets().searchStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else {
+ throw new ProcessException("Endpoint was invalid value: " + endpoint);
+ }
+ } catch (ApiException e) {
+ logger.error("Received error {}: {}", e.getCode(), e.getMessage());
+ }
+
+ executorService.execute(new TweetQueuer());
+ }
+
+ /**
+ * This method would be called when we would like the stream to get stopped. The stream will be closed and the
+ * executorService will be shut down. If it fails to shutdown, then it will be forcefully terminated.
+ */
+ public void stop() {
+ try {
+ stream.close();
+ } catch (IOException e) {
+ logger.error("IOException occurred while closing stream: {}", e.getMessage());
+ }
+
+ executorService.shutdown();
+ try {
+ if (!executorService.awaitTermination(3, TimeUnit.SECONDS)) {
+ executorService.shutdownNow();
+ }
+ } catch (InterruptedException e) {
+ executorService.shutdownNow();
+ logger.error("InterruptedException occurred while waiting for executor service to shut down: {}", e.getMessage());
+ }
+ }
+
+ private class TweetQueuer implements Runnable {
+ @Override
+ public void run() {
+ try (BufferedReader reader = new BufferedReader(new InputStreamReader(stream))) {
Review Comment:
I think a SSL handshake exception would occur when the API makes the HTTP request and fetches the stream of tweets. Will wrap the starting stream in a processException to try and catch this. Thanks for the comment!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r848802898
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TwitterStreamAPI.java:
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+public class TwitterStreamAPI {
+ public static final String SEARCH_ENDPOINT = "Search Endpoint";
+ public static final String SAMPLE_ENDPOINT = "Sample Endpoint";
+ public static final String SAMPLE_PATH = "/2/tweets/sample/stream";
+ public static final String SEARCH_PATH = "/2/tweets/search/stream";
+
+ private static final String BEARER_TOKEN_PROPERTY_NAME = "bearer-token";
+ private static final String TWEET_FIELDS_PROPERTY_NAME = "tweet-fields";
+ private static final String USER_FIELDS_PROPERTY_NAME = "user-fields";
+ private static final String MEDIA_FIELDS_PROPERTY_NAME = "media-fields";
+ private static final String POLL_FIELDS_PROPERTY_NAME = "poll-fields";
+ private static final String PLACE_FIELDS_PROPERTY_NAME = "place-fields";
+ private static final String EXPANSIONS_PROPERTY_NAME = "expansions";
+ private static final String BACKFILL_MINUTES_PROPERTY_NAME = "backfill-minutes";
+
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Set<String> parseCommaSeparatedListPropreties(final ProcessContext context, final String propertyName) {
+ Set<String> fields = null;
+ if (context.getProperty(propertyName).isSet()) {
+ fields = new HashSet<>();
+ final String fieldsString = context.getProperty(propertyName).getValue();
+ for (final String field: fieldsString.split(",")) {
+ fields.add(field.trim());
+ }
+ }
+ return fields;
+ }
+
+ public TwitterStreamAPI(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ assert context != null;
+ assert queue != null;
+ assert logger != null;
+
+ this.queue = queue;
+ this.logger = logger;
+
+ this.tweetFields = parseCommaSeparatedListPropreties(context, TWEET_FIELDS_PROPERTY_NAME);
+ this.userFields = parseCommaSeparatedListPropreties(context, USER_FIELDS_PROPERTY_NAME);
+ this.mediaFields = parseCommaSeparatedListPropreties(context, MEDIA_FIELDS_PROPERTY_NAME);
+ this.pollFields = parseCommaSeparatedListPropreties(context, POLL_FIELDS_PROPERTY_NAME);
+ this.placeFields = parseCommaSeparatedListPropreties(context, PLACE_FIELDS_PROPERTY_NAME);
+ this.expansions = parseCommaSeparatedListPropreties(context, EXPANSIONS_PROPERTY_NAME);
+ this.backfillMinutes = context.getProperty(BACKFILL_MINUTES_PROPERTY_NAME).asInteger();
+
+ TwitterCredentialsBearer creds = new TwitterCredentialsBearer(context.getProperty(BEARER_TOKEN_PROPERTY_NAME).getValue());
+ api = new TwitterApi();
+ api.setTwitterCredentials(creds);
+
+ this.executorService = Executors.newSingleThreadExecutor();
+ }
+
+ public String getBasePath() {
+ return api.getApiClient().getBasePath();
+ }
+
+ public void setBasePath(final String path) {
+ api.getApiClient().setBasePath(path);
+ }
+
+ /**
+ * This method would be called when we would like the stream to get started. This method will spin off a thread that
+ * will continue to queue tweets on to the given queue passed in the constructor. The thread will continue
+ * to run until {@code stop} is called.
+ * @param endpoint {@code TwitterStreamAPI.SAMPLE_ENDPOINT} or {@code TwitterStreamAPI.SEARCH_ENDPOINT}
+ */
+ public void start(final String endpoint) {
+ try {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ stream = api.tweets().sampleStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else if (endpoint.equals(SEARCH_ENDPOINT)) {
+ stream = api.tweets().searchStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else {
+ throw new ProcessException("Endpoint was invalid value: " + endpoint);
+ }
+ } catch (ApiException e) {
+ logger.error("Received error {}: {}", e.getCode(), e.getMessage());
+ }
+
+ executorService.execute(new TweetQueuer());
Review Comment:
Thanks for a suggestion, will make a check for a null stream during the start. Thanks for the comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory closed pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
exceptionfactory closed pull request #5947: NIFI-9755: Implement ConsumeTwitter
URL: https://github.com/apache/nifi/pull/5947
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r864024019
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,366 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.UUID;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
Review Comment:
Making the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r848761220
##########
nifi-assembly/NOTICE:
##########
@@ -964,6 +964,11 @@ The following binary components are provided under the Apache Software License v
The following NOTICE information applies:
Hosebird Client (hbc)
Copyright 2013 Twitter, Inc.
+
+ (ASLv2) Twitter API Client Library for Java
Review Comment:
Will make a more thorough checks of the dependency trees, thanks for the comments!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r848779569
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TwitterStreamAPI.java:
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+public class TwitterStreamAPI {
+ public static final String SEARCH_ENDPOINT = "Search Endpoint";
+ public static final String SAMPLE_ENDPOINT = "Sample Endpoint";
+ public static final String SAMPLE_PATH = "/2/tweets/sample/stream";
+ public static final String SEARCH_PATH = "/2/tweets/search/stream";
+
+ private static final String BEARER_TOKEN_PROPERTY_NAME = "bearer-token";
+ private static final String TWEET_FIELDS_PROPERTY_NAME = "tweet-fields";
+ private static final String USER_FIELDS_PROPERTY_NAME = "user-fields";
+ private static final String MEDIA_FIELDS_PROPERTY_NAME = "media-fields";
+ private static final String POLL_FIELDS_PROPERTY_NAME = "poll-fields";
+ private static final String PLACE_FIELDS_PROPERTY_NAME = "place-fields";
+ private static final String EXPANSIONS_PROPERTY_NAME = "expansions";
+ private static final String BACKFILL_MINUTES_PROPERTY_NAME = "backfill-minutes";
+
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Set<String> parseCommaSeparatedListPropreties(final ProcessContext context, final String propertyName) {
+ Set<String> fields = null;
+ if (context.getProperty(propertyName).isSet()) {
+ fields = new HashSet<>();
+ final String fieldsString = context.getProperty(propertyName).getValue();
+ for (final String field: fieldsString.split(",")) {
+ fields.add(field.trim());
+ }
+ }
+ return fields;
+ }
+
+ public TwitterStreamAPI(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ assert context != null;
+ assert queue != null;
+ assert logger != null;
+
+ this.queue = queue;
+ this.logger = logger;
+
+ this.tweetFields = parseCommaSeparatedListPropreties(context, TWEET_FIELDS_PROPERTY_NAME);
+ this.userFields = parseCommaSeparatedListPropreties(context, USER_FIELDS_PROPERTY_NAME);
+ this.mediaFields = parseCommaSeparatedListPropreties(context, MEDIA_FIELDS_PROPERTY_NAME);
+ this.pollFields = parseCommaSeparatedListPropreties(context, POLL_FIELDS_PROPERTY_NAME);
+ this.placeFields = parseCommaSeparatedListPropreties(context, PLACE_FIELDS_PROPERTY_NAME);
+ this.expansions = parseCommaSeparatedListPropreties(context, EXPANSIONS_PROPERTY_NAME);
+ this.backfillMinutes = context.getProperty(BACKFILL_MINUTES_PROPERTY_NAME).asInteger();
+
+ TwitterCredentialsBearer creds = new TwitterCredentialsBearer(context.getProperty(BEARER_TOKEN_PROPERTY_NAME).getValue());
+ api = new TwitterApi();
+ api.setTwitterCredentials(creds);
+
+ this.executorService = Executors.newSingleThreadExecutor();
+ }
+
+ public String getBasePath() {
+ return api.getApiClient().getBasePath();
+ }
+
+ public void setBasePath(final String path) {
+ api.getApiClient().setBasePath(path);
+ }
+
+ /**
+ * This method would be called when we would like the stream to get started. This method will spin off a thread that
+ * will continue to queue tweets on to the given queue passed in the constructor. The thread will continue
+ * to run until {@code stop} is called.
+ * @param endpoint {@code TwitterStreamAPI.SAMPLE_ENDPOINT} or {@code TwitterStreamAPI.SEARCH_ENDPOINT}
+ */
+ public void start(final String endpoint) {
+ try {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ stream = api.tweets().sampleStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else if (endpoint.equals(SEARCH_ENDPOINT)) {
+ stream = api.tweets().searchStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else {
+ throw new ProcessException("Endpoint was invalid value: " + endpoint);
+ }
+ } catch (ApiException e) {
+ logger.error("Received error {}: {}", e.getCode(), e.getMessage());
+ }
+
+ executorService.execute(new TweetQueuer());
+ }
+
+ /**
+ * This method would be called when we would like the stream to get stopped. The stream will be closed and the
+ * executorService will be shut down. If it fails to shutdown, then it will be forcefully terminated.
+ */
+ public void stop() {
+ try {
+ stream.close();
+ } catch (IOException e) {
+ logger.error("IOException occurred while closing stream: {}", e.getMessage());
+ }
+
+ executorService.shutdown();
+ try {
+ if (!executorService.awaitTermination(3, TimeUnit.SECONDS)) {
+ executorService.shutdownNow();
+ }
+ } catch (InterruptedException e) {
+ executorService.shutdownNow();
+ logger.error("InterruptedException occurred while waiting for executor service to shut down: {}", e.getMessage());
+ }
Review Comment:
Makes sense, making the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] MikeThomsen commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
MikeThomsen commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r846633486
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,316 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("Twitter Endpoint")
Review Comment:
Name for each property should be something like `consume-twitter-endpoint`. These are not meant to be human-friendly fields. That's what `displayName` is for.
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,316 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("Twitter Endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("Base Path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("Bearer Token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("Queue Size")
+ .displayName("Queue Size")
+ .description("The size of the queue used to queue tweets")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("Batch Size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("Tweet Fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values are: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("User Fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user " +
+ "for proper usage. Possible field values are: " +
+ "created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, " +
+ "protected, public_metrics, url, username, verified, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor MEDIA_FIELDS = new PropertyDescriptor.Builder()
+ .name("Media Fields")
+ .displayName("Media Fields")
+ .description("A comma-separated list of media fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/media " +
+ "for proper usage. Possible field values are: " +
+ "alt_text, duration_ms, height, media_key, non_public_metrics, organic_metrics, preview_image_url, " +
+ "promoted_metrics, public_metrics, type, url, width")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor POLL_FIELDS = new PropertyDescriptor.Builder()
+ .name("Poll Fields")
+ .displayName("Poll Fields")
+ .description("A comma-separated list of poll fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/poll " +
+ "for proper usage. Possible field values are: " +
+ "duration_minutes, end_datetime, id, options, voting_status")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PLACE_FIELDS = new PropertyDescriptor.Builder()
+ .name("Place Fields")
+ .displayName("Place Fields")
+ .description("A comma-separated list of place fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/place " +
+ "for proper usage. Possible field values are: " +
+ "contained_within, country, country_code, full_name, geo, id, name, place_type")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor EXPANSIONS = new PropertyDescriptor.Builder()
+ .name("Expansions")
+ .displayName("Expansions")
+ .description("A comma-separated list of expansions for objects in the returned tweet. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/expansions " +
+ "for proper usage. Possible field values are: " +
+ "author_id, referenced_tweets.id, referenced_tweets.id.author_id, entities.mentions.username, " +
+ "attachments.poll_ids, attachments.media_keys ,in_reply_to_user_id, geo.place_id")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor BACKFILL_MINUTES = new PropertyDescriptor.Builder()
+ .name("Backfill Minutes")
+ .displayName("Backfill Minutes")
+ .description("The number of minutes (up to 5 minutes) of streaming data to be requested after a " +
+ "disconnect. Only available for project with academic research access. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/integrate/" +
+ "recovery-and-redundancy-features")
+ .required(true)
+ .defaultValue("0")
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("All tweets will be routed to this relationship.")
+ .build();
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+
+ private TwitterStreamAPI api;
+
+ private volatile BlockingQueue<String> messageQueue;
+
+ private void emptyQueue() {
+ while (!messageQueue.isEmpty()) {
+ messageQueue.poll();
+ }
+ }
+
+ @Override
+ protected void init(ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptors = new ArrayList<>();
+ descriptors.add(ENDPOINT);
+ descriptors.add(BASE_PATH);
+ descriptors.add(BEARER_TOKEN);
+ descriptors.add(QUEUE_SIZE);
+ descriptors.add(BATCH_SIZE);
+ descriptors.add(TWEET_FIELDS);
+ descriptors.add(USER_FIELDS);
+ descriptors.add(MEDIA_FIELDS);
+ descriptors.add(POLL_FIELDS);
+ descriptors.add(PLACE_FIELDS);
+ descriptors.add(EXPANSIONS);
+ descriptors.add(BACKFILL_MINUTES);
+
+ this.descriptors = Collections.unmodifiableList(descriptors);
+
+ final Set<Relationship> relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ this.relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return this.descriptors;
+ }
+
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ messageQueue = new LinkedBlockingQueue<>(context.getProperty(QUEUE_SIZE).asInteger());
+
+ api = new TwitterStreamAPI(context, messageQueue, getLogger());
+ api.setBasePath(context.getProperty(BASE_PATH).getValue());
+ final String endpointName = context.getProperty(ENDPOINT).getValue();
+ if (ENDPOINT_SAMPLE.getValue().equals(endpointName)) {
+ api.start(TwitterStreamAPI.SAMPLE_ENDPOINT);
+ } else if (ENDPOINT_SEARCH.getValue().equals(endpointName)) {
+ api.start(TwitterStreamAPI.SEARCH_ENDPOINT);
+ } else {
+ throw new AssertionError("Endpoint was invalid value: " + endpointName);
+ }
+ }
+
+ @Override
+ public void onTrigger(ProcessContext context, ProcessSession session) throws ProcessException {
+ if (messageQueue.isEmpty()) {
+ context.yield();
+ return;
+ }
+
+ FlowFile flowFile = session.create();
+ flowFile = session.write(flowFile, new OutputStreamCallback() {
Review Comment:
It would be safer to use Jackson to convert the incoming tweets into `Map<String, Object>` types and then turn the current queue contents into a stringified array with Jackson.
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,316 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("Twitter Endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("Base Path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("Bearer Token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("Queue Size")
+ .displayName("Queue Size")
+ .description("The size of the queue used to queue tweets")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("Batch Size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("Tweet Fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
Review Comment:
Should have a default value
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,316 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("Twitter Endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("Base Path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("Bearer Token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("Queue Size")
+ .displayName("Queue Size")
+ .description("The size of the queue used to queue tweets")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("Batch Size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("Tweet Fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values are: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("User Fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user " +
+ "for proper usage. Possible field values are: " +
+ "created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, " +
+ "protected, public_metrics, url, username, verified, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor MEDIA_FIELDS = new PropertyDescriptor.Builder()
+ .name("Media Fields")
Review Comment:
All of the *_FIELDS should have defaults.
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TwitterStreamAPI.java:
##########
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+public class TwitterStreamAPI {
+ public static final String SEARCH_ENDPOINT = "Search Endpoint";
+ public static final String SAMPLE_ENDPOINT = "Sample Endpoint";
+ public static final String SAMPLE_PATH = "/2/tweets/sample/stream";
+ public static final String SEARCH_PATH = "/2/tweets/search/stream";
+
+ private static final String BEARER_TOKEN_PROPERTY_NAME = "Bearer Token";
+ private static final String TWEET_FIELDS_PROPERTY_NAME = "Tweet Fields";
+ private static final String USER_FIELDS_PROPERTY_NAME = "User Fields";
+ private static final String MEDIA_FIELDS_PROPERTY_NAME = "Media Fields";
+ private static final String POLL_FIELDS_PROPERTY_NAME = "Poll Fields";
+ private static final String PLACE_FIELDS_PROPERTY_NAME = "Place Fields";
+ private static final String EXPANSIONS_PROPERTY_NAME = "Expansions";
+ private static final String BACKFILL_MINUTES_PROPERTY_NAME = "Backfill Minutes";
+
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Set<String> parseCommaSeparatedListPropreties(final ProcessContext context, final String propertyName) {
+ Set<String> fields = null;
+ if (context.getProperty(propertyName).isSet()) {
+ fields = new HashSet<>();
+ final String fieldsString = context.getProperty(propertyName).getValue();
+ for (final String field: fieldsString.split(",")) {
+ fields.add(field.trim());
+ }
+ }
+ return fields;
+ }
+
+ public TwitterStreamAPI(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ assert context != null;
+ assert queue != null;
+ assert logger != null;
+
+ this.queue = queue;
+ this.logger = logger;
+
+ this.tweetFields = parseCommaSeparatedListPropreties(context, TWEET_FIELDS_PROPERTY_NAME);
+ this.userFields = parseCommaSeparatedListPropreties(context, USER_FIELDS_PROPERTY_NAME);
+ this.mediaFields = parseCommaSeparatedListPropreties(context, MEDIA_FIELDS_PROPERTY_NAME);
+ this.pollFields = parseCommaSeparatedListPropreties(context, POLL_FIELDS_PROPERTY_NAME);
+ this.placeFields = parseCommaSeparatedListPropreties(context, PLACE_FIELDS_PROPERTY_NAME);
+ this.expansions = parseCommaSeparatedListPropreties(context, EXPANSIONS_PROPERTY_NAME);
+ this.backfillMinutes = context.getProperty(BACKFILL_MINUTES_PROPERTY_NAME).asInteger();
+
+ TwitterCredentialsBearer creds = new TwitterCredentialsBearer(context.getProperty(BEARER_TOKEN_PROPERTY_NAME).getValue());
+ api = new TwitterApi();
+ api.setTwitterCredentials(creds);
+
+ this.executorService = Executors.newSingleThreadExecutor();
+ }
+
+ public String getBasePath() {
+ return api.getApiClient().getBasePath();
+ }
+
+ public void setBasePath(final String path) {
+ api.getApiClient().setBasePath(path);
+ }
+
+ /**
+ * This method would be called when we would like the stream to get started. This method will spin off a thread that
+ * will continue to queue tweets on to the given queue passed in the constructor. The thread will continue
+ * to run until {@code stop} is called.
+ * @param endpoint {@code TwitterStreamAPI.SAMPLE_ENDPOINT} or {@code TwitterStreamAPI.SEARCH_ENDPOINT}
+ */
+ public void start(final String endpoint) {
+ try {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ stream = api.tweets().sampleStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else if (endpoint.equals(SEARCH_ENDPOINT)) {
+ stream = api.tweets().searchStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else {
+ throw new AssertionError("Endpoint was invalid value: " + endpoint);
Review Comment:
Use `ProcessException`
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,316 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("Twitter Endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("Base Path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
Review Comment:
What's the expiration policy for Twitter bearer tokens and does Twitter provide easily accessed OAuth2 APIs to get a token?
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,316 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("Twitter Endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("Base Path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("Bearer Token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("Queue Size")
+ .displayName("Queue Size")
+ .description("The size of the queue used to queue tweets")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("Batch Size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("Tweet Fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values are: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("User Fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
Review Comment:
Should have a default value
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,316 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("Twitter Endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("Base Path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("Bearer Token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("Queue Size")
+ .displayName("Queue Size")
+ .description("The size of the queue used to queue tweets")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("Batch Size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("Tweet Fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values are: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("User Fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user " +
+ "for proper usage. Possible field values are: " +
+ "created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, " +
+ "protected, public_metrics, url, username, verified, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor MEDIA_FIELDS = new PropertyDescriptor.Builder()
+ .name("Media Fields")
+ .displayName("Media Fields")
+ .description("A comma-separated list of media fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/media " +
+ "for proper usage. Possible field values are: " +
+ "alt_text, duration_ms, height, media_key, non_public_metrics, organic_metrics, preview_image_url, " +
+ "promoted_metrics, public_metrics, type, url, width")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor POLL_FIELDS = new PropertyDescriptor.Builder()
+ .name("Poll Fields")
+ .displayName("Poll Fields")
+ .description("A comma-separated list of poll fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/poll " +
+ "for proper usage. Possible field values are: " +
+ "duration_minutes, end_datetime, id, options, voting_status")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PLACE_FIELDS = new PropertyDescriptor.Builder()
+ .name("Place Fields")
+ .displayName("Place Fields")
+ .description("A comma-separated list of place fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/place " +
+ "for proper usage. Possible field values are: " +
+ "contained_within, country, country_code, full_name, geo, id, name, place_type")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor EXPANSIONS = new PropertyDescriptor.Builder()
+ .name("Expansions")
+ .displayName("Expansions")
+ .description("A comma-separated list of expansions for objects in the returned tweet. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/expansions " +
+ "for proper usage. Possible field values are: " +
+ "author_id, referenced_tweets.id, referenced_tweets.id.author_id, entities.mentions.username, " +
+ "attachments.poll_ids, attachments.media_keys ,in_reply_to_user_id, geo.place_id")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor BACKFILL_MINUTES = new PropertyDescriptor.Builder()
+ .name("Backfill Minutes")
+ .displayName("Backfill Minutes")
+ .description("The number of minutes (up to 5 minutes) of streaming data to be requested after a " +
+ "disconnect. Only available for project with academic research access. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/integrate/" +
+ "recovery-and-redundancy-features")
+ .required(true)
+ .defaultValue("0")
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("All tweets will be routed to this relationship.")
+ .build();
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+
+ private TwitterStreamAPI api;
+
+ private volatile BlockingQueue<String> messageQueue;
+
+ private void emptyQueue() {
+ while (!messageQueue.isEmpty()) {
+ messageQueue.poll();
+ }
+ }
+
+ @Override
+ protected void init(ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptors = new ArrayList<>();
+ descriptors.add(ENDPOINT);
+ descriptors.add(BASE_PATH);
+ descriptors.add(BEARER_TOKEN);
+ descriptors.add(QUEUE_SIZE);
+ descriptors.add(BATCH_SIZE);
+ descriptors.add(TWEET_FIELDS);
+ descriptors.add(USER_FIELDS);
+ descriptors.add(MEDIA_FIELDS);
+ descriptors.add(POLL_FIELDS);
+ descriptors.add(PLACE_FIELDS);
+ descriptors.add(EXPANSIONS);
+ descriptors.add(BACKFILL_MINUTES);
+
+ this.descriptors = Collections.unmodifiableList(descriptors);
+
+ final Set<Relationship> relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ this.relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return this.descriptors;
+ }
+
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ messageQueue = new LinkedBlockingQueue<>(context.getProperty(QUEUE_SIZE).asInteger());
+
+ api = new TwitterStreamAPI(context, messageQueue, getLogger());
+ api.setBasePath(context.getProperty(BASE_PATH).getValue());
+ final String endpointName = context.getProperty(ENDPOINT).getValue();
+ if (ENDPOINT_SAMPLE.getValue().equals(endpointName)) {
+ api.start(TwitterStreamAPI.SAMPLE_ENDPOINT);
+ } else if (ENDPOINT_SEARCH.getValue().equals(endpointName)) {
+ api.start(TwitterStreamAPI.SEARCH_ENDPOINT);
+ } else {
+ throw new AssertionError("Endpoint was invalid value: " + endpointName);
+ }
+ }
+
+ @Override
+ public void onTrigger(ProcessContext context, ProcessSession session) throws ProcessException {
+ if (messageQueue.isEmpty()) {
+ context.yield();
+ return;
+ }
+
+ FlowFile flowFile = session.create();
+ flowFile = session.write(flowFile, new OutputStreamCallback() {
+ @Override
+ public void process(OutputStream out) throws IOException {
+ out.write("[".getBytes(StandardCharsets.UTF_8));
+ final int batchSize = context.getProperty(BATCH_SIZE).asInteger();
+ String tweet = messageQueue.poll();
+ out.write(tweet.getBytes(StandardCharsets.UTF_8));
+ int tweetCount = 1;
+ while (tweetCount < batchSize && !messageQueue.isEmpty()) {
+ out.write(",".getBytes(StandardCharsets.UTF_8));
+ tweet = messageQueue.poll();
+ out.write(tweet.getBytes(StandardCharsets.UTF_8));
+ tweetCount++;
+ }
+ out.write("]".getBytes(StandardCharsets.UTF_8));
+ }
+ });
+
+ final Map<String, String> attributes = new HashMap<>();
+ attributes.put(CoreAttributes.MIME_TYPE.key(), "application/json");
+ attributes.put(CoreAttributes.FILENAME.key(), flowFile.getAttribute(CoreAttributes.FILENAME.key()) + ".json");
+ flowFile = session.putAllAttributes(flowFile, attributes);
+
+ session.transfer(flowFile, REL_SUCCESS);
+
+ String transitUri = api.getBasePath();
+ final String endpointName = context.getProperty(ENDPOINT).getValue();
+ if (ENDPOINT_SAMPLE.getValue().equals(endpointName)) {
+ transitUri += TwitterStreamAPI.SAMPLE_PATH;
+ } else if (ENDPOINT_SEARCH.getValue().equals(endpointName)) {
+ transitUri += TwitterStreamAPI.SEARCH_PATH;
+ } else {
+ throw new AssertionError("Endpoint was invalid value: " + endpointName);
Review Comment:
Should be a `ProcessException`.
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,316 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("Twitter Endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("Base Path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("Bearer Token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("Queue Size")
+ .displayName("Queue Size")
+ .description("The size of the queue used to queue tweets")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("Batch Size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("Tweet Fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values are: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("User Fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user " +
+ "for proper usage. Possible field values are: " +
+ "created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, " +
+ "protected, public_metrics, url, username, verified, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor MEDIA_FIELDS = new PropertyDescriptor.Builder()
+ .name("Media Fields")
+ .displayName("Media Fields")
+ .description("A comma-separated list of media fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/media " +
+ "for proper usage. Possible field values are: " +
+ "alt_text, duration_ms, height, media_key, non_public_metrics, organic_metrics, preview_image_url, " +
+ "promoted_metrics, public_metrics, type, url, width")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor POLL_FIELDS = new PropertyDescriptor.Builder()
+ .name("Poll Fields")
+ .displayName("Poll Fields")
+ .description("A comma-separated list of poll fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/poll " +
+ "for proper usage. Possible field values are: " +
+ "duration_minutes, end_datetime, id, options, voting_status")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PLACE_FIELDS = new PropertyDescriptor.Builder()
+ .name("Place Fields")
+ .displayName("Place Fields")
+ .description("A comma-separated list of place fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/place " +
+ "for proper usage. Possible field values are: " +
+ "contained_within, country, country_code, full_name, geo, id, name, place_type")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor EXPANSIONS = new PropertyDescriptor.Builder()
+ .name("Expansions")
+ .displayName("Expansions")
+ .description("A comma-separated list of expansions for objects in the returned tweet. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/expansions " +
+ "for proper usage. Possible field values are: " +
+ "author_id, referenced_tweets.id, referenced_tweets.id.author_id, entities.mentions.username, " +
+ "attachments.poll_ids, attachments.media_keys ,in_reply_to_user_id, geo.place_id")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor BACKFILL_MINUTES = new PropertyDescriptor.Builder()
+ .name("Backfill Minutes")
+ .displayName("Backfill Minutes")
+ .description("The number of minutes (up to 5 minutes) of streaming data to be requested after a " +
+ "disconnect. Only available for project with academic research access. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/integrate/" +
+ "recovery-and-redundancy-features")
+ .required(true)
+ .defaultValue("0")
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("All tweets will be routed to this relationship.")
+ .build();
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+
+ private TwitterStreamAPI api;
+
+ private volatile BlockingQueue<String> messageQueue;
+
+ private void emptyQueue() {
+ while (!messageQueue.isEmpty()) {
+ messageQueue.poll();
+ }
+ }
+
+ @Override
+ protected void init(ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptors = new ArrayList<>();
+ descriptors.add(ENDPOINT);
+ descriptors.add(BASE_PATH);
+ descriptors.add(BEARER_TOKEN);
+ descriptors.add(QUEUE_SIZE);
+ descriptors.add(BATCH_SIZE);
+ descriptors.add(TWEET_FIELDS);
+ descriptors.add(USER_FIELDS);
+ descriptors.add(MEDIA_FIELDS);
+ descriptors.add(POLL_FIELDS);
+ descriptors.add(PLACE_FIELDS);
+ descriptors.add(EXPANSIONS);
+ descriptors.add(BACKFILL_MINUTES);
+
+ this.descriptors = Collections.unmodifiableList(descriptors);
+
+ final Set<Relationship> relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ this.relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return this.descriptors;
+ }
+
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ messageQueue = new LinkedBlockingQueue<>(context.getProperty(QUEUE_SIZE).asInteger());
+
+ api = new TwitterStreamAPI(context, messageQueue, getLogger());
+ api.setBasePath(context.getProperty(BASE_PATH).getValue());
+ final String endpointName = context.getProperty(ENDPOINT).getValue();
+ if (ENDPOINT_SAMPLE.getValue().equals(endpointName)) {
+ api.start(TwitterStreamAPI.SAMPLE_ENDPOINT);
+ } else if (ENDPOINT_SEARCH.getValue().equals(endpointName)) {
+ api.start(TwitterStreamAPI.SEARCH_ENDPOINT);
+ } else {
+ throw new AssertionError("Endpoint was invalid value: " + endpointName);
Review Comment:
Use `ProcessException`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r861128589
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.UUID;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("consume-twitter-endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("Maximum size of internal queue for streamed messages")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor BACKOFF_ATTEMPTS = new PropertyDescriptor.Builder()
+ .name("backoff-attempts")
+ .displayName("Backoff Attempts")
+ .description("The number of reconnection tries the processor will attempt in the event of " +
+ "a disconnection of the stream for any reason, before throwing an exception. To start a stream after " +
+ "this exception occur and the connection is fixed, please stop and restart the processor. If the value" +
+ "of this property is 0, then backoff will never occur and the processor will always need to be restarted" +
+ "if the stream fails.")
+ .required(true)
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .defaultValue("5")
+ .build();
+ public static final PropertyDescriptor BACKOFF_TIME = new PropertyDescriptor.Builder()
+ .name("backoff-time")
+ .displayName("Backoff Time")
+ .description("The duration to backoff to start attempting a new stream if" +
+ "the current one fails for any reason. Will increase by factor of 2 every time a restart fails")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("1 mins")
+ .build();
+ public static final PropertyDescriptor MAXIMUM_BACKOFF_TIME = new PropertyDescriptor.Builder()
+ .name("maximum-backoff-time")
+ .displayName("Maximum Backoff Time")
+ .description("The maximum duration to backoff to start attempting a new stream." +
+ "It is recommended that this number be much higher than the 'Backoff Time' property")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("5 mins")
+ .build();
+ public static final PropertyDescriptor CONNECT_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("connect-timeout")
+ .displayName("Connect Timeout")
+ .description("The maximum time in which client should establish a connection with the " +
+ "Twitter API before a time out. A value of 0 will mean no timeout at all")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("10 secs")
+ .build();
+ public static final PropertyDescriptor READ_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("read-timeout")
+ .displayName("Read Timeout")
+ .description("The maximum time of inactivity between receiving tweets from Twitter through " +
+ "the API before a timeout. A value of 0 will mean no timeout at all")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("10 secs")
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("tweet-fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values include: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("user-fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user " +
+ "for proper usage. Possible field values include: " +
+ "created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, " +
+ "protected, public_metrics, url, username, verified, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor MEDIA_FIELDS = new PropertyDescriptor.Builder()
+ .name("media-fields")
+ .displayName("Media Fields")
+ .description("A comma-separated list of media fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/media " +
+ "for proper usage. Possible field values include: " +
+ "alt_text, duration_ms, height, media_key, non_public_metrics, organic_metrics, preview_image_url, " +
+ "promoted_metrics, public_metrics, type, url, width")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor POLL_FIELDS = new PropertyDescriptor.Builder()
+ .name("poll-fields")
+ .displayName("Poll Fields")
+ .description("A comma-separated list of poll fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/poll " +
+ "for proper usage. Possible field values include: " +
+ "duration_minutes, end_datetime, id, options, voting_status")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PLACE_FIELDS = new PropertyDescriptor.Builder()
+ .name("place-fields")
+ .displayName("Place Fields")
+ .description("A comma-separated list of place fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/place " +
+ "for proper usage. Possible field values include: " +
+ "contained_within, country, country_code, full_name, geo, id, name, place_type")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor EXPANSIONS = new PropertyDescriptor.Builder()
+ .name("expansions")
+ .displayName("Expansions")
+ .description("A comma-separated list of expansions for objects in the returned tweet. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/expansions " +
+ "for proper usage. Possible field values include: " +
+ "author_id, referenced_tweets.id, referenced_tweets.id.author_id, entities.mentions.username, " +
+ "attachments.poll_ids, attachments.media_keys ,in_reply_to_user_id, geo.place_id")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor BACKFILL_MINUTES = new PropertyDescriptor.Builder()
+ .name("backfill-minutes")
+ .displayName("Backfill Minutes")
+ .description("The number of minutes (up to 5 minutes) of streaming data to be requested after a " +
+ "disconnect. Only available for project with academic research access. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/integrate/" +
+ "recovery-and-redundancy-features")
+ .required(true)
+ .defaultValue("0")
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("FlowFiles containing an array of one or more Tweets")
+ .build();
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+
+ private TweetStreamService tweetStreamService;
+
+ private volatile BlockingQueue<String> messageQueue;
+
+ @Override
+ protected void init(ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptors = new ArrayList<>();
+ descriptors.add(ENDPOINT);
+ descriptors.add(BASE_PATH);
+ descriptors.add(BEARER_TOKEN);
+ descriptors.add(QUEUE_SIZE);
+ descriptors.add(BATCH_SIZE);
+ descriptors.add(BACKOFF_ATTEMPTS);
+ descriptors.add(BACKOFF_TIME);
+ descriptors.add(MAXIMUM_BACKOFF_TIME);
+ descriptors.add(CONNECT_TIMEOUT);
+ descriptors.add(READ_TIMEOUT);
+ descriptors.add(TWEET_FIELDS);
+ descriptors.add(USER_FIELDS);
+ descriptors.add(MEDIA_FIELDS);
+ descriptors.add(POLL_FIELDS);
+ descriptors.add(PLACE_FIELDS);
+ descriptors.add(EXPANSIONS);
+ descriptors.add(BACKFILL_MINUTES);
Review Comment:
Making the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r861145841
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TweetStreamService.java:
##########
@@ -0,0 +1,242 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiClient;
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TweetStreamService {
+ public static final String SEARCH_ENDPOINT = "Search Endpoint";
+ public static final String SAMPLE_ENDPOINT = "Sample Endpoint";
+ private static final String SAMPLE_PATH = "/2/tweets/sample/stream";
+ private static final String SEARCH_PATH = "/2/tweets/search/stream";
+
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ScheduledExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Long backoffMultiplier;
+ private int backoffAttempts;
+ private int attemptCounter;
+ private Long backoffTime;
+ private Long maximumBackoff;
+
+ private String endpoint;
+
+ private Set<String> parseCommaSeparatedListPropreties(final ProcessContext context, final PropertyDescriptor property) {
+ Set<String> fields = null;
+ if (context.getProperty(property).isSet()) {
+ fields = new HashSet<>();
+ final String fieldsString = context.getProperty(property).getValue();
+ for (final String field: fieldsString.split(",")) {
+ fields.add(field.trim());
+ }
+ }
+ return fields;
+ }
+
+ public TweetStreamService(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ assert context != null;
+ assert queue != null;
+ assert logger != null;
+
+ this.queue = queue;
+ this.logger = logger;
+
+ this.tweetFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.TWEET_FIELDS);
+ this.userFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.USER_FIELDS);
+ this.mediaFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.MEDIA_FIELDS);
+ this.pollFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.POLL_FIELDS);
+ this.placeFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.PLACE_FIELDS);
+ this.expansions = parseCommaSeparatedListPropreties(context, ConsumeTwitter.EXPANSIONS);
+ this.backfillMinutes = context.getProperty(ConsumeTwitter.BACKFILL_MINUTES).asInteger();
+
+ this.backoffMultiplier = 1L;
+ this.backoffAttempts = context.getProperty(ConsumeTwitter.BACKOFF_ATTEMPTS).asInteger();
+ this.attemptCounter = 0;
+ this.backoffTime = context.getProperty(ConsumeTwitter.BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+ this.maximumBackoff = context.getProperty(ConsumeTwitter.MAXIMUM_BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+
+ ApiClient client = new ApiClient();
+ final int connectTimeout = context.getProperty(ConsumeTwitter.CONNECT_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ final int readTimeout = context.getProperty(ConsumeTwitter.READ_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ client = client.setConnectTimeout(connectTimeout);
+ client = client.setReadTimeout(readTimeout);
+ api = new TwitterApi(client);
+
+ TwitterCredentialsBearer creds = new TwitterCredentialsBearer(context.getProperty(ConsumeTwitter.BEARER_TOKEN).getValue());
+ api.setTwitterCredentials(creds);
+
+ this.executorService = Executors.newSingleThreadScheduledExecutor();
+ }
+
+ public String getBasePath() {
+ return api.getApiClient().getBasePath();
+ }
+
+ public void setBasePath(final String path) {
+ api.getApiClient().setBasePath(path);
+ }
+
+ public String getTransitUri(final String endpoint) {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ return api.getApiClient().getBasePath() + SAMPLE_PATH;
+ } else if (endpoint.equals(SEARCH_ENDPOINT)) {
+ return api.getApiClient().getBasePath() + SEARCH_PATH;
+ } else {
+ logger.warn("Unrecognized endpoint in getTransitUri. Returning basePath");
+ return api.getApiClient().getBasePath();
+ }
+ }
+
+ /**
+ * This method would be called when we would like the stream to get started. This method will spin off a thread that
+ * will continue to queue tweets on to the given queue passed in the constructor. The thread will continue
+ * to run until {@code stop} is called.
+ * @param endpoint {@code TwitterStreamAPI.SAMPLE_ENDPOINT} or {@code TwitterStreamAPI.SEARCH_ENDPOINT}
+ */
+ public void start(final String endpoint) {
Review Comment:
Making the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r861143162
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TweetStreamService.java:
##########
@@ -0,0 +1,242 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiClient;
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TweetStreamService {
+ public static final String SEARCH_ENDPOINT = "Search Endpoint";
+ public static final String SAMPLE_ENDPOINT = "Sample Endpoint";
+ private static final String SAMPLE_PATH = "/2/tweets/sample/stream";
+ private static final String SEARCH_PATH = "/2/tweets/search/stream";
Review Comment:
Thanks for the suggestion, putting these values in the StreamEndpoint enum
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r861175114
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TweetStreamService.java:
##########
@@ -0,0 +1,242 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiClient;
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TweetStreamService {
+ public static final String SEARCH_ENDPOINT = "Search Endpoint";
+ public static final String SAMPLE_ENDPOINT = "Sample Endpoint";
+ private static final String SAMPLE_PATH = "/2/tweets/sample/stream";
+ private static final String SEARCH_PATH = "/2/tweets/search/stream";
+
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ScheduledExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Long backoffMultiplier;
+ private int backoffAttempts;
+ private int attemptCounter;
+ private Long backoffTime;
+ private Long maximumBackoff;
+
+ private String endpoint;
+
+ private Set<String> parseCommaSeparatedListPropreties(final ProcessContext context, final PropertyDescriptor property) {
+ Set<String> fields = null;
+ if (context.getProperty(property).isSet()) {
+ fields = new HashSet<>();
+ final String fieldsString = context.getProperty(property).getValue();
+ for (final String field: fieldsString.split(",")) {
+ fields.add(field.trim());
+ }
+ }
+ return fields;
+ }
+
+ public TweetStreamService(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ assert context != null;
+ assert queue != null;
+ assert logger != null;
+
+ this.queue = queue;
+ this.logger = logger;
+
+ this.tweetFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.TWEET_FIELDS);
+ this.userFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.USER_FIELDS);
+ this.mediaFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.MEDIA_FIELDS);
+ this.pollFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.POLL_FIELDS);
+ this.placeFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.PLACE_FIELDS);
+ this.expansions = parseCommaSeparatedListPropreties(context, ConsumeTwitter.EXPANSIONS);
+ this.backfillMinutes = context.getProperty(ConsumeTwitter.BACKFILL_MINUTES).asInteger();
+
+ this.backoffMultiplier = 1L;
+ this.backoffAttempts = context.getProperty(ConsumeTwitter.BACKOFF_ATTEMPTS).asInteger();
+ this.attemptCounter = 0;
+ this.backoffTime = context.getProperty(ConsumeTwitter.BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+ this.maximumBackoff = context.getProperty(ConsumeTwitter.MAXIMUM_BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+
+ ApiClient client = new ApiClient();
+ final int connectTimeout = context.getProperty(ConsumeTwitter.CONNECT_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ final int readTimeout = context.getProperty(ConsumeTwitter.READ_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ client = client.setConnectTimeout(connectTimeout);
+ client = client.setReadTimeout(readTimeout);
+ api = new TwitterApi(client);
+
+ TwitterCredentialsBearer creds = new TwitterCredentialsBearer(context.getProperty(ConsumeTwitter.BEARER_TOKEN).getValue());
+ api.setTwitterCredentials(creds);
+
+ this.executorService = Executors.newSingleThreadScheduledExecutor();
+ }
+
+ public String getBasePath() {
+ return api.getApiClient().getBasePath();
+ }
+
+ public void setBasePath(final String path) {
+ api.getApiClient().setBasePath(path);
+ }
+
+ public String getTransitUri(final String endpoint) {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ return api.getApiClient().getBasePath() + SAMPLE_PATH;
+ } else if (endpoint.equals(SEARCH_ENDPOINT)) {
+ return api.getApiClient().getBasePath() + SEARCH_PATH;
+ } else {
+ logger.warn("Unrecognized endpoint in getTransitUri. Returning basePath");
+ return api.getApiClient().getBasePath();
+ }
+ }
+
+ /**
+ * This method would be called when we would like the stream to get started. This method will spin off a thread that
+ * will continue to queue tweets on to the given queue passed in the constructor. The thread will continue
+ * to run until {@code stop} is called.
+ * @param endpoint {@code TwitterStreamAPI.SAMPLE_ENDPOINT} or {@code TwitterStreamAPI.SEARCH_ENDPOINT}
+ */
+ public void start(final String endpoint) {
+ this.endpoint = endpoint;
+ executorService.execute(new TweetStreamStarter());
+ }
+
+ /**
+ * This method would be called when we would like the stream to get stopped. The stream will be closed and the
+ * executorService will be shut down.
+ */
+ public void stop() {
+ if (stream != null) {
+ try {
+ stream.close();
+ } catch (IOException e) {
+ logger.error("Closing response stream failed", e);
+ }
+ }
+
+ executorService.shutdownNow();
+ }
+
+ private Long calculateBackoffDelay() {
+ long backoff = backoffMultiplier * backoffTime;
+ long delay = (backoff > maximumBackoff) ? maximumBackoff : backoff;
+ return delay;
+ }
+
+ private void scheduleStartStreamWithBackoff() {
+ // use exponential(by factor of 2) backoff in scheduling the next TweetStreamStarter
+ if (attemptCounter >= backoffAttempts) {
+ final String msg = "Reached maximum attempts to backoff";
+ logger.error(msg);
+ throw new ProcessException(msg);
+ }
+ attemptCounter += 1;
+ long delay = calculateBackoffDelay();
+ backoffMultiplier *= 2;
+ logger.info("Scheduling new TweetStreamStarter after {}s delay", delay);
+ executorService.schedule(new TweetStreamStarter(), delay, TimeUnit.SECONDS);
+ }
+
+ private void resetBackoff() {
+ attemptCounter = 0;
+ backoffMultiplier = 1L;
+ }
+
+ private class TweetStreamStarter implements Runnable {
+ @Override
+ public void run() {
+ try {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ stream = api.tweets().sampleStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else {
+ stream = api.tweets().searchStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ }
+ } catch (ApiException e) {
+ stream = null;
+ logger.warn(String.format("Received ApiException, error %d: %s", e.getCode(), e.getMessage()));
+ scheduleStartStreamWithBackoff();
+ return;
+ } catch (Exception e) {
+ stream = null;
+ logger.warn(String.format("Received Exception %s: %s", e.getClass(), e.getMessage()));
+ scheduleStartStreamWithBackoff();
+ return;
+ }
+
+ if (stream == null) {
+ logger.warn("Stream is null, could not make a connection to the Twitter API");
+ scheduleStartStreamWithBackoff();
+ return;
+ } else {
+ executorService.execute(new TweetStreamHandler());
+ }
+ }
+ }
+
+ private class TweetStreamHandler implements Runnable {
+ @Override
+ public void run() {
+ try (BufferedReader reader = new BufferedReader(new InputStreamReader(stream))) {
+ String tweetRecord = reader.readLine();
+ while (tweetRecord != null) {
+ queue.offer(tweetRecord);
+
+ // reset backoff multiplier upon successful receipt of a tweet
+ resetBackoff();
+ try {
+ tweetRecord = reader.readLine();
+ } catch (IOException e) {
Review Comment:
Thanks for the catch. This was necessary before the backoff implementation where the stream may or may not return null, which would cause a failure in reading the stream; to separate this from the situation where the stream is finished. I can change it to just having the outer catch IOException
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r861176449
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TweetStreamService.java:
##########
@@ -0,0 +1,242 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiClient;
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TweetStreamService {
+ public static final String SEARCH_ENDPOINT = "Search Endpoint";
+ public static final String SAMPLE_ENDPOINT = "Sample Endpoint";
+ private static final String SAMPLE_PATH = "/2/tweets/sample/stream";
+ private static final String SEARCH_PATH = "/2/tweets/search/stream";
+
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ScheduledExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Long backoffMultiplier;
+ private int backoffAttempts;
+ private int attemptCounter;
+ private Long backoffTime;
+ private Long maximumBackoff;
+
+ private String endpoint;
+
+ private Set<String> parseCommaSeparatedListPropreties(final ProcessContext context, final PropertyDescriptor property) {
+ Set<String> fields = null;
+ if (context.getProperty(property).isSet()) {
+ fields = new HashSet<>();
+ final String fieldsString = context.getProperty(property).getValue();
+ for (final String field: fieldsString.split(",")) {
+ fields.add(field.trim());
+ }
+ }
+ return fields;
+ }
+
+ public TweetStreamService(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ assert context != null;
+ assert queue != null;
+ assert logger != null;
+
+ this.queue = queue;
+ this.logger = logger;
+
+ this.tweetFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.TWEET_FIELDS);
+ this.userFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.USER_FIELDS);
+ this.mediaFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.MEDIA_FIELDS);
+ this.pollFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.POLL_FIELDS);
+ this.placeFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.PLACE_FIELDS);
+ this.expansions = parseCommaSeparatedListPropreties(context, ConsumeTwitter.EXPANSIONS);
+ this.backfillMinutes = context.getProperty(ConsumeTwitter.BACKFILL_MINUTES).asInteger();
+
+ this.backoffMultiplier = 1L;
+ this.backoffAttempts = context.getProperty(ConsumeTwitter.BACKOFF_ATTEMPTS).asInteger();
+ this.attemptCounter = 0;
+ this.backoffTime = context.getProperty(ConsumeTwitter.BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+ this.maximumBackoff = context.getProperty(ConsumeTwitter.MAXIMUM_BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+
+ ApiClient client = new ApiClient();
+ final int connectTimeout = context.getProperty(ConsumeTwitter.CONNECT_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ final int readTimeout = context.getProperty(ConsumeTwitter.READ_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ client = client.setConnectTimeout(connectTimeout);
+ client = client.setReadTimeout(readTimeout);
+ api = new TwitterApi(client);
+
+ TwitterCredentialsBearer creds = new TwitterCredentialsBearer(context.getProperty(ConsumeTwitter.BEARER_TOKEN).getValue());
+ api.setTwitterCredentials(creds);
+
+ this.executorService = Executors.newSingleThreadScheduledExecutor();
+ }
+
+ public String getBasePath() {
+ return api.getApiClient().getBasePath();
+ }
+
+ public void setBasePath(final String path) {
+ api.getApiClient().setBasePath(path);
+ }
+
+ public String getTransitUri(final String endpoint) {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ return api.getApiClient().getBasePath() + SAMPLE_PATH;
+ } else if (endpoint.equals(SEARCH_ENDPOINT)) {
+ return api.getApiClient().getBasePath() + SEARCH_PATH;
+ } else {
+ logger.warn("Unrecognized endpoint in getTransitUri. Returning basePath");
+ return api.getApiClient().getBasePath();
+ }
+ }
+
+ /**
+ * This method would be called when we would like the stream to get started. This method will spin off a thread that
+ * will continue to queue tweets on to the given queue passed in the constructor. The thread will continue
+ * to run until {@code stop} is called.
+ * @param endpoint {@code TwitterStreamAPI.SAMPLE_ENDPOINT} or {@code TwitterStreamAPI.SEARCH_ENDPOINT}
+ */
+ public void start(final String endpoint) {
+ this.endpoint = endpoint;
+ executorService.execute(new TweetStreamStarter());
+ }
+
+ /**
+ * This method would be called when we would like the stream to get stopped. The stream will be closed and the
+ * executorService will be shut down.
+ */
+ public void stop() {
+ if (stream != null) {
+ try {
+ stream.close();
+ } catch (IOException e) {
+ logger.error("Closing response stream failed", e);
+ }
+ }
+
+ executorService.shutdownNow();
+ }
+
+ private Long calculateBackoffDelay() {
+ long backoff = backoffMultiplier * backoffTime;
+ long delay = (backoff > maximumBackoff) ? maximumBackoff : backoff;
+ return delay;
+ }
+
+ private void scheduleStartStreamWithBackoff() {
+ // use exponential(by factor of 2) backoff in scheduling the next TweetStreamStarter
+ if (attemptCounter >= backoffAttempts) {
+ final String msg = "Reached maximum attempts to backoff";
+ logger.error(msg);
+ throw new ProcessException(msg);
+ }
+ attemptCounter += 1;
+ long delay = calculateBackoffDelay();
+ backoffMultiplier *= 2;
+ logger.info("Scheduling new TweetStreamStarter after {}s delay", delay);
+ executorService.schedule(new TweetStreamStarter(), delay, TimeUnit.SECONDS);
+ }
+
+ private void resetBackoff() {
+ attemptCounter = 0;
+ backoffMultiplier = 1L;
+ }
+
+ private class TweetStreamStarter implements Runnable {
+ @Override
+ public void run() {
+ try {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ stream = api.tweets().sampleStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else {
+ stream = api.tweets().searchStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ }
+ } catch (ApiException e) {
+ stream = null;
+ logger.warn(String.format("Received ApiException, error %d: %s", e.getCode(), e.getMessage()));
+ scheduleStartStreamWithBackoff();
+ return;
+ } catch (Exception e) {
+ stream = null;
+ logger.warn(String.format("Received Exception %s: %s", e.getClass(), e.getMessage()));
+ scheduleStartStreamWithBackoff();
+ return;
+ }
+
+ if (stream == null) {
+ logger.warn("Stream is null, could not make a connection to the Twitter API");
+ scheduleStartStreamWithBackoff();
+ return;
+ } else {
+ executorService.execute(new TweetStreamHandler());
+ }
+ }
+ }
+
+ private class TweetStreamHandler implements Runnable {
+ @Override
+ public void run() {
+ try (BufferedReader reader = new BufferedReader(new InputStreamReader(stream))) {
+ String tweetRecord = reader.readLine();
+ while (tweetRecord != null) {
+ queue.offer(tweetRecord);
+
+ // reset backoff multiplier upon successful receipt of a tweet
+ resetBackoff();
+ try {
+ tweetRecord = reader.readLine();
+ } catch (IOException e) {
+ logger.info("Read Tweet failed: Stream processing completed");
+ break;
+ }
+ }
+ } catch (IOException e) {
+ logger.warn("Stream processing failed", e);
+ }
+ logger.info("TweetStreamHandler terminating");
Review Comment:
Making the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r864022718
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,366 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.UUID;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue(StreamEndpoint.SAMPLE_ENDPOINT.getEndpointName(),
+ "Sample Stream",
+ "Streams about one percent of all Tweets. " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/volume-streams/api-reference/get-tweets-sample-stream");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue(StreamEndpoint.SEARCH_ENDPOINT.getEndpointName(),
+ "Search Stream",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty. " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/get-tweets-search-stream");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("stream-endpoint")
+ .displayName("Stream Endpoint")
+ .description("Specifies which endpoint tweets should be streamed. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("Maximum size of internal queue for streamed messages")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
Review Comment:
Making the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r864021974
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,366 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.UUID;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
Review Comment:
Making the changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] MikeThomsen commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
MikeThomsen commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r847161278
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,316 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("Twitter Endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("Base Path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("Bearer Token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("Queue Size")
+ .displayName("Queue Size")
+ .description("The size of the queue used to queue tweets")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("Batch Size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("Tweet Fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values are: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("User Fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user " +
+ "for proper usage. Possible field values are: " +
+ "created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, " +
+ "protected, public_metrics, url, username, verified, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor MEDIA_FIELDS = new PropertyDescriptor.Builder()
+ .name("Media Fields")
Review Comment:
I think the goal should be to return everything by default and then let the user pare it down as they want. If the empty string does the same thing as a comma-separated list, then that works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r861151676
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TweetStreamService.java:
##########
@@ -0,0 +1,242 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiClient;
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TweetStreamService {
+ public static final String SEARCH_ENDPOINT = "Search Endpoint";
+ public static final String SAMPLE_ENDPOINT = "Sample Endpoint";
+ private static final String SAMPLE_PATH = "/2/tweets/sample/stream";
+ private static final String SEARCH_PATH = "/2/tweets/search/stream";
+
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ScheduledExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Long backoffMultiplier;
+ private int backoffAttempts;
+ private int attemptCounter;
+ private Long backoffTime;
+ private Long maximumBackoff;
+
+ private String endpoint;
+
+ private Set<String> parseCommaSeparatedListPropreties(final ProcessContext context, final PropertyDescriptor property) {
+ Set<String> fields = null;
+ if (context.getProperty(property).isSet()) {
+ fields = new HashSet<>();
+ final String fieldsString = context.getProperty(property).getValue();
+ for (final String field: fieldsString.split(",")) {
+ fields.add(field.trim());
+ }
+ }
+ return fields;
+ }
+
+ public TweetStreamService(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ assert context != null;
+ assert queue != null;
+ assert logger != null;
+
+ this.queue = queue;
+ this.logger = logger;
+
+ this.tweetFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.TWEET_FIELDS);
+ this.userFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.USER_FIELDS);
+ this.mediaFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.MEDIA_FIELDS);
+ this.pollFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.POLL_FIELDS);
+ this.placeFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.PLACE_FIELDS);
+ this.expansions = parseCommaSeparatedListPropreties(context, ConsumeTwitter.EXPANSIONS);
+ this.backfillMinutes = context.getProperty(ConsumeTwitter.BACKFILL_MINUTES).asInteger();
+
+ this.backoffMultiplier = 1L;
+ this.backoffAttempts = context.getProperty(ConsumeTwitter.BACKOFF_ATTEMPTS).asInteger();
+ this.attemptCounter = 0;
+ this.backoffTime = context.getProperty(ConsumeTwitter.BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+ this.maximumBackoff = context.getProperty(ConsumeTwitter.MAXIMUM_BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+
+ ApiClient client = new ApiClient();
+ final int connectTimeout = context.getProperty(ConsumeTwitter.CONNECT_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ final int readTimeout = context.getProperty(ConsumeTwitter.READ_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ client = client.setConnectTimeout(connectTimeout);
+ client = client.setReadTimeout(readTimeout);
+ api = new TwitterApi(client);
+
+ TwitterCredentialsBearer creds = new TwitterCredentialsBearer(context.getProperty(ConsumeTwitter.BEARER_TOKEN).getValue());
+ api.setTwitterCredentials(creds);
+
+ this.executorService = Executors.newSingleThreadScheduledExecutor();
+ }
+
+ public String getBasePath() {
+ return api.getApiClient().getBasePath();
+ }
+
+ public void setBasePath(final String path) {
+ api.getApiClient().setBasePath(path);
+ }
+
+ public String getTransitUri(final String endpoint) {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ return api.getApiClient().getBasePath() + SAMPLE_PATH;
+ } else if (endpoint.equals(SEARCH_ENDPOINT)) {
+ return api.getApiClient().getBasePath() + SEARCH_PATH;
+ } else {
+ logger.warn("Unrecognized endpoint in getTransitUri. Returning basePath");
+ return api.getApiClient().getBasePath();
+ }
+ }
+
+ /**
+ * This method would be called when we would like the stream to get started. This method will spin off a thread that
+ * will continue to queue tweets on to the given queue passed in the constructor. The thread will continue
+ * to run until {@code stop} is called.
+ * @param endpoint {@code TwitterStreamAPI.SAMPLE_ENDPOINT} or {@code TwitterStreamAPI.SEARCH_ENDPOINT}
+ */
+ public void start(final String endpoint) {
+ this.endpoint = endpoint;
+ executorService.execute(new TweetStreamStarter());
+ }
+
+ /**
+ * This method would be called when we would like the stream to get stopped. The stream will be closed and the
+ * executorService will be shut down.
+ */
+ public void stop() {
+ if (stream != null) {
+ try {
+ stream.close();
+ } catch (IOException e) {
+ logger.error("Closing response stream failed", e);
+ }
+ }
+
+ executorService.shutdownNow();
+ }
+
+ private Long calculateBackoffDelay() {
+ long backoff = backoffMultiplier * backoffTime;
+ long delay = (backoff > maximumBackoff) ? maximumBackoff : backoff;
+ return delay;
+ }
+
+ private void scheduleStartStreamWithBackoff() {
+ // use exponential(by factor of 2) backoff in scheduling the next TweetStreamStarter
+ if (attemptCounter >= backoffAttempts) {
+ final String msg = "Reached maximum attempts to backoff";
+ logger.error(msg);
+ throw new ProcessException(msg);
+ }
+ attemptCounter += 1;
+ long delay = calculateBackoffDelay();
+ backoffMultiplier *= 2;
+ logger.info("Scheduling new TweetStreamStarter after {}s delay", delay);
+ executorService.schedule(new TweetStreamStarter(), delay, TimeUnit.SECONDS);
+ }
+
+ private void resetBackoff() {
+ attemptCounter = 0;
+ backoffMultiplier = 1L;
+ }
+
+ private class TweetStreamStarter implements Runnable {
+ @Override
+ public void run() {
+ try {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ stream = api.tweets().sampleStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else {
+ stream = api.tweets().searchStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ }
+ } catch (ApiException e) {
+ stream = null;
+ logger.warn(String.format("Received ApiException, error %d: %s", e.getCode(), e.getMessage()));
+ scheduleStartStreamWithBackoff();
+ return;
+ } catch (Exception e) {
+ stream = null;
+ logger.warn(String.format("Received Exception %s: %s", e.getClass(), e.getMessage()));
Review Comment:
Making the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r861168808
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TweetStreamService.java:
##########
@@ -0,0 +1,242 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiClient;
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TweetStreamService {
+ public static final String SEARCH_ENDPOINT = "Search Endpoint";
+ public static final String SAMPLE_ENDPOINT = "Sample Endpoint";
+ private static final String SAMPLE_PATH = "/2/tweets/sample/stream";
+ private static final String SEARCH_PATH = "/2/tweets/search/stream";
+
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ScheduledExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Long backoffMultiplier;
+ private int backoffAttempts;
+ private int attemptCounter;
+ private Long backoffTime;
+ private Long maximumBackoff;
+
+ private String endpoint;
+
+ private Set<String> parseCommaSeparatedListPropreties(final ProcessContext context, final PropertyDescriptor property) {
+ Set<String> fields = null;
+ if (context.getProperty(property).isSet()) {
+ fields = new HashSet<>();
+ final String fieldsString = context.getProperty(property).getValue();
+ for (final String field: fieldsString.split(",")) {
+ fields.add(field.trim());
+ }
+ }
+ return fields;
+ }
+
+ public TweetStreamService(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ assert context != null;
+ assert queue != null;
+ assert logger != null;
+
+ this.queue = queue;
+ this.logger = logger;
+
+ this.tweetFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.TWEET_FIELDS);
+ this.userFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.USER_FIELDS);
+ this.mediaFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.MEDIA_FIELDS);
+ this.pollFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.POLL_FIELDS);
+ this.placeFields = parseCommaSeparatedListPropreties(context, ConsumeTwitter.PLACE_FIELDS);
+ this.expansions = parseCommaSeparatedListPropreties(context, ConsumeTwitter.EXPANSIONS);
+ this.backfillMinutes = context.getProperty(ConsumeTwitter.BACKFILL_MINUTES).asInteger();
+
+ this.backoffMultiplier = 1L;
+ this.backoffAttempts = context.getProperty(ConsumeTwitter.BACKOFF_ATTEMPTS).asInteger();
+ this.attemptCounter = 0;
+ this.backoffTime = context.getProperty(ConsumeTwitter.BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+ this.maximumBackoff = context.getProperty(ConsumeTwitter.MAXIMUM_BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+
+ ApiClient client = new ApiClient();
+ final int connectTimeout = context.getProperty(ConsumeTwitter.CONNECT_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ final int readTimeout = context.getProperty(ConsumeTwitter.READ_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ client = client.setConnectTimeout(connectTimeout);
+ client = client.setReadTimeout(readTimeout);
+ api = new TwitterApi(client);
+
+ TwitterCredentialsBearer creds = new TwitterCredentialsBearer(context.getProperty(ConsumeTwitter.BEARER_TOKEN).getValue());
+ api.setTwitterCredentials(creds);
+
+ this.executorService = Executors.newSingleThreadScheduledExecutor();
+ }
+
+ public String getBasePath() {
+ return api.getApiClient().getBasePath();
+ }
+
+ public void setBasePath(final String path) {
+ api.getApiClient().setBasePath(path);
+ }
+
+ public String getTransitUri(final String endpoint) {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ return api.getApiClient().getBasePath() + SAMPLE_PATH;
+ } else if (endpoint.equals(SEARCH_ENDPOINT)) {
+ return api.getApiClient().getBasePath() + SEARCH_PATH;
+ } else {
+ logger.warn("Unrecognized endpoint in getTransitUri. Returning basePath");
+ return api.getApiClient().getBasePath();
+ }
+ }
+
+ /**
+ * This method would be called when we would like the stream to get started. This method will spin off a thread that
+ * will continue to queue tweets on to the given queue passed in the constructor. The thread will continue
+ * to run until {@code stop} is called.
+ * @param endpoint {@code TwitterStreamAPI.SAMPLE_ENDPOINT} or {@code TwitterStreamAPI.SEARCH_ENDPOINT}
+ */
+ public void start(final String endpoint) {
+ this.endpoint = endpoint;
+ executorService.execute(new TweetStreamStarter());
+ }
+
+ /**
+ * This method would be called when we would like the stream to get stopped. The stream will be closed and the
+ * executorService will be shut down.
+ */
+ public void stop() {
+ if (stream != null) {
+ try {
+ stream.close();
+ } catch (IOException e) {
+ logger.error("Closing response stream failed", e);
+ }
+ }
+
+ executorService.shutdownNow();
+ }
+
+ private Long calculateBackoffDelay() {
+ long backoff = backoffMultiplier * backoffTime;
+ long delay = (backoff > maximumBackoff) ? maximumBackoff : backoff;
+ return delay;
+ }
+
+ private void scheduleStartStreamWithBackoff() {
+ // use exponential(by factor of 2) backoff in scheduling the next TweetStreamStarter
+ if (attemptCounter >= backoffAttempts) {
+ final String msg = "Reached maximum attempts to backoff";
+ logger.error(msg);
+ throw new ProcessException(msg);
+ }
+ attemptCounter += 1;
+ long delay = calculateBackoffDelay();
+ backoffMultiplier *= 2;
+ logger.info("Scheduling new TweetStreamStarter after {}s delay", delay);
+ executorService.schedule(new TweetStreamStarter(), delay, TimeUnit.SECONDS);
+ }
+
+ private void resetBackoff() {
+ attemptCounter = 0;
+ backoffMultiplier = 1L;
+ }
+
+ private class TweetStreamStarter implements Runnable {
+ @Override
+ public void run() {
+ try {
+ if (endpoint.equals(SAMPLE_ENDPOINT)) {
+ stream = api.tweets().sampleStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ } else {
+ stream = api.tweets().searchStream(expansions, tweetFields, userFields, mediaFields, placeFields, pollFields, backfillMinutes);
+ }
+ } catch (ApiException e) {
+ stream = null;
+ logger.warn(String.format("Received ApiException, error %d: %s", e.getCode(), e.getMessage()));
+ scheduleStartStreamWithBackoff();
+ return;
+ } catch (Exception e) {
+ stream = null;
+ logger.warn(String.format("Received Exception %s: %s", e.getClass(), e.getMessage()));
+ scheduleStartStreamWithBackoff();
+ return;
+ }
+
+ if (stream == null) {
+ logger.warn("Stream is null, could not make a connection to the Twitter API");
+ scheduleStartStreamWithBackoff();
+ return;
Review Comment:
Yes you're right, now that I explicitly null the stream on failure, this shouldn't happen, I will make the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r861142673
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TweetStreamService.java:
##########
@@ -0,0 +1,242 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiClient;
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TweetStreamService {
+ public static final String SEARCH_ENDPOINT = "Search Endpoint";
+ public static final String SAMPLE_ENDPOINT = "Sample Endpoint";
Review Comment:
That makes sense, making the StreamEndpoint enum in a separate class
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r861128222
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.UUID;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue("Sample Endpoint",
+ "Sample Endpoint",
+ "The endpoint that provides a stream of about 1% of tweets in real-time");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue("Search Endpoint",
+ "Search Endpoint",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("consume-twitter-endpoint")
+ .displayName("Twitter Endpoint")
+ .description("Specifies which endpoint tweets should be pulled from. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("Maximum size of internal queue for streamed messages")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor BACKOFF_ATTEMPTS = new PropertyDescriptor.Builder()
+ .name("backoff-attempts")
+ .displayName("Backoff Attempts")
+ .description("The number of reconnection tries the processor will attempt in the event of " +
+ "a disconnection of the stream for any reason, before throwing an exception. To start a stream after " +
+ "this exception occur and the connection is fixed, please stop and restart the processor. If the value" +
+ "of this property is 0, then backoff will never occur and the processor will always need to be restarted" +
+ "if the stream fails.")
+ .required(true)
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .defaultValue("5")
+ .build();
+ public static final PropertyDescriptor BACKOFF_TIME = new PropertyDescriptor.Builder()
+ .name("backoff-time")
+ .displayName("Backoff Time")
+ .description("The duration to backoff to start attempting a new stream if" +
+ "the current one fails for any reason. Will increase by factor of 2 every time a restart fails")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("1 mins")
+ .build();
+ public static final PropertyDescriptor MAXIMUM_BACKOFF_TIME = new PropertyDescriptor.Builder()
+ .name("maximum-backoff-time")
+ .displayName("Maximum Backoff Time")
+ .description("The maximum duration to backoff to start attempting a new stream." +
+ "It is recommended that this number be much higher than the 'Backoff Time' property")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("5 mins")
+ .build();
+ public static final PropertyDescriptor CONNECT_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("connect-timeout")
+ .displayName("Connect Timeout")
+ .description("The maximum time in which client should establish a connection with the " +
+ "Twitter API before a time out. A value of 0 will mean no timeout at all")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("10 secs")
+ .build();
+ public static final PropertyDescriptor READ_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("read-timeout")
+ .displayName("Read Timeout")
+ .description("The maximum time of inactivity between receiving tweets from Twitter through " +
+ "the API before a timeout. A value of 0 will mean no timeout at all")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("10 secs")
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("tweet-fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values include: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("user-fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user " +
+ "for proper usage. Possible field values include: " +
+ "created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, " +
+ "protected, public_metrics, url, username, verified, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor MEDIA_FIELDS = new PropertyDescriptor.Builder()
+ .name("media-fields")
+ .displayName("Media Fields")
+ .description("A comma-separated list of media fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/media " +
+ "for proper usage. Possible field values include: " +
+ "alt_text, duration_ms, height, media_key, non_public_metrics, organic_metrics, preview_image_url, " +
+ "promoted_metrics, public_metrics, type, url, width")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor POLL_FIELDS = new PropertyDescriptor.Builder()
+ .name("poll-fields")
+ .displayName("Poll Fields")
+ .description("A comma-separated list of poll fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/poll " +
+ "for proper usage. Possible field values include: " +
+ "duration_minutes, end_datetime, id, options, voting_status")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PLACE_FIELDS = new PropertyDescriptor.Builder()
+ .name("place-fields")
+ .displayName("Place Fields")
+ .description("A comma-separated list of place fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/place " +
+ "for proper usage. Possible field values include: " +
+ "contained_within, country, country_code, full_name, geo, id, name, place_type")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor EXPANSIONS = new PropertyDescriptor.Builder()
+ .name("expansions")
+ .displayName("Expansions")
+ .description("A comma-separated list of expansions for objects in the returned tweet. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/expansions " +
+ "for proper usage. Possible field values include: " +
+ "author_id, referenced_tweets.id, referenced_tweets.id.author_id, entities.mentions.username, " +
+ "attachments.poll_ids, attachments.media_keys ,in_reply_to_user_id, geo.place_id")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor BACKFILL_MINUTES = new PropertyDescriptor.Builder()
+ .name("backfill-minutes")
+ .displayName("Backfill Minutes")
+ .description("The number of minutes (up to 5 minutes) of streaming data to be requested after a " +
+ "disconnect. Only available for project with academic research access. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/integrate/" +
+ "recovery-and-redundancy-features")
+ .required(true)
+ .defaultValue("0")
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
Review Comment:
Making the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r864031422
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,366 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.UUID;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue(StreamEndpoint.SAMPLE_ENDPOINT.getEndpointName(),
+ "Sample Stream",
+ "Streams about one percent of all Tweets. " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/volume-streams/api-reference/get-tweets-sample-stream");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue(StreamEndpoint.SEARCH_ENDPOINT.getEndpointName(),
+ "Search Stream",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty. " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/get-tweets-search-stream");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("stream-endpoint")
+ .displayName("Stream Endpoint")
+ .description("Specifies which endpoint tweets should be streamed. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("Maximum size of internal queue for streamed messages")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
Review Comment:
Making the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r864032093
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,366 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.UUID;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue(StreamEndpoint.SAMPLE_ENDPOINT.getEndpointName(),
+ "Sample Stream",
+ "Streams about one percent of all Tweets. " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/volume-streams/api-reference/get-tweets-sample-stream");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue(StreamEndpoint.SEARCH_ENDPOINT.getEndpointName(),
+ "Search Stream",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty. " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/get-tweets-search-stream");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("stream-endpoint")
+ .displayName("Stream Endpoint")
+ .description("Specifies which endpoint tweets should be streamed. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("Maximum size of internal queue for streamed messages")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor BACKOFF_ATTEMPTS = new PropertyDescriptor.Builder()
+ .name("backoff-attempts")
+ .displayName("Backoff Attempts")
+ .description("The number of reconnection tries the processor will attempt in the event of " +
+ "a disconnection of the stream for any reason, before throwing an exception. To start a stream after " +
+ "this exception occur and the connection is fixed, please stop and restart the processor. If the value" +
+ "of this property is 0, then backoff will never occur and the processor will always need to be restarted" +
+ "if the stream fails.")
+ .required(true)
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .defaultValue("5")
+ .build();
+ public static final PropertyDescriptor BACKOFF_TIME = new PropertyDescriptor.Builder()
+ .name("backoff-time")
+ .displayName("Backoff Time")
+ .description("The duration to backoff to start attempting a new stream if" +
Review Comment:
Making the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r864046803
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/ConsumeTwitter.java:
##########
@@ -0,0 +1,366 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.PrimaryNodeOnly;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.Set;
+import java.util.List;
+import java.util.HashSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.UUID;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+@PrimaryNodeOnly
+@SupportsBatching
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@Tags({"twitter", "tweets", "social media", "status", "json"})
+@CapabilityDescription("Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search "
+ + "stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the "
+ + "tweet to be returned as part of the response. See "
+ + "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the "
+ + "Tweet object model. \n\n"
+ + "Warning: the underlying Java SDK used is still in beta as of the publishing of this processor feature.")
+@WritesAttribute(attribute = "mime.type", description = "Sets mime type to application/json")
+public class ConsumeTwitter extends AbstractProcessor {
+
+ static final AllowableValue ENDPOINT_SAMPLE = new AllowableValue(StreamEndpoint.SAMPLE_ENDPOINT.getEndpointName(),
+ "Sample Stream",
+ "Streams about one percent of all Tweets. " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/volume-streams/api-reference/get-tweets-sample-stream");
+ static final AllowableValue ENDPOINT_SEARCH = new AllowableValue(StreamEndpoint.SEARCH_ENDPOINT.getEndpointName(),
+ "Search Stream",
+ "The endpoint that provides a stream of tweets that matches the rules you added to the stream. " +
+ "If rules are not configured, then the stream will be empty. " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/get-tweets-search-stream");
+
+ public static final PropertyDescriptor ENDPOINT = new PropertyDescriptor.Builder()
+ .name("stream-endpoint")
+ .displayName("Stream Endpoint")
+ .description("Specifies which endpoint tweets should be streamed. " +
+ "Usage of search endpoint requires that rules be uploaded beforehand. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/" +
+ "post-tweets-search-stream-rules")
+ .required(true)
+ .allowableValues(ENDPOINT_SAMPLE, ENDPOINT_SEARCH)
+ .defaultValue(ENDPOINT_SAMPLE.getValue())
+ .build();
+ public static final PropertyDescriptor BASE_PATH = new PropertyDescriptor.Builder()
+ .name("base-path")
+ .displayName("Base Path")
+ .description("Specifies which base path the API client will use for HTTP requests. " +
+ "Generally should not be changed from the default https://api.twitter.com except for testing")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .defaultValue("https://api.twitter.com")
+ .build();
+ public static final PropertyDescriptor BEARER_TOKEN = new PropertyDescriptor.Builder()
+ .name("bearer-token")
+ .displayName("Bearer Token")
+ .description("The Bearer Token provided by Twitter.")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor QUEUE_SIZE = new PropertyDescriptor.Builder()
+ .name("queue-size")
+ .displayName("Queue Size")
+ .description("Maximum size of internal queue for streamed messages")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10000")
+ .build();
+ public static final PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
+ .name("batch-size")
+ .displayName("Batch Size")
+ .description("The maximum size of the number of tweets to be written to a single FlowFile." +
+ "Will write less tweets if it there are not any tweets left in queue")
+ .required(true)
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("10")
+ .build();
+ public static final PropertyDescriptor BACKOFF_ATTEMPTS = new PropertyDescriptor.Builder()
+ .name("backoff-attempts")
+ .displayName("Backoff Attempts")
+ .description("The number of reconnection tries the processor will attempt in the event of " +
+ "a disconnection of the stream for any reason, before throwing an exception. To start a stream after " +
+ "this exception occur and the connection is fixed, please stop and restart the processor. If the value" +
+ "of this property is 0, then backoff will never occur and the processor will always need to be restarted" +
+ "if the stream fails.")
+ .required(true)
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .defaultValue("5")
+ .build();
+ public static final PropertyDescriptor BACKOFF_TIME = new PropertyDescriptor.Builder()
+ .name("backoff-time")
+ .displayName("Backoff Time")
+ .description("The duration to backoff to start attempting a new stream if" +
+ "the current one fails for any reason. Will increase by factor of 2 every time a restart fails")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("1 mins")
+ .build();
+ public static final PropertyDescriptor MAXIMUM_BACKOFF_TIME = new PropertyDescriptor.Builder()
+ .name("maximum-backoff-time")
+ .displayName("Maximum Backoff Time")
+ .description("The maximum duration to backoff to start attempting a new stream." +
+ "It is recommended that this number be much higher than the 'Backoff Time' property")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("5 mins")
+ .build();
+ public static final PropertyDescriptor CONNECT_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("connect-timeout")
+ .displayName("Connect Timeout")
+ .description("The maximum time in which client should establish a connection with the " +
+ "Twitter API before a time out. A value of 0 will mean no timeout at all")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("10 secs")
+ .build();
+ public static final PropertyDescriptor READ_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("read-timeout")
+ .displayName("Read Timeout")
+ .description("The maximum time of inactivity between receiving tweets from Twitter through " +
+ "the API before a timeout. A value of 0 will mean no timeout at all")
+ .required(true)
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("10 secs")
+ .build();
+ public static final PropertyDescriptor BACKFILL_MINUTES = new PropertyDescriptor.Builder()
+ .name("backfill-minutes")
+ .displayName("Backfill Minutes")
+ .description("The number of minutes (up to 5 minutes) of streaming data to be requested after a " +
+ "disconnect. Only available for project with academic research access. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/integrate/" +
+ "recovery-and-redundancy-features")
+ .required(true)
+ .defaultValue("0")
+ .addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor TWEET_FIELDS = new PropertyDescriptor.Builder()
+ .name("tweet-fields")
+ .displayName("Tweet Fields")
+ .description("A comma-separated list of tweet fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet " +
+ "for proper usage. Possible field values include: " +
+ "attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, " +
+ "in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, " +
+ "public_metrics, referenced_tweets, reply_settings, source, text, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor USER_FIELDS = new PropertyDescriptor.Builder()
+ .name("user-fields")
+ .displayName("User Fields")
+ .description("A comma-separated list of user fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user " +
+ "for proper usage. Possible field values include: " +
+ "created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, " +
+ "protected, public_metrics, url, username, verified, withheld")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor MEDIA_FIELDS = new PropertyDescriptor.Builder()
+ .name("media-fields")
+ .displayName("Media Fields")
+ .description("A comma-separated list of media fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/media " +
+ "for proper usage. Possible field values include: " +
+ "alt_text, duration_ms, height, media_key, non_public_metrics, organic_metrics, preview_image_url, " +
+ "promoted_metrics, public_metrics, type, url, width")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor POLL_FIELDS = new PropertyDescriptor.Builder()
+ .name("poll-fields")
+ .displayName("Poll Fields")
+ .description("A comma-separated list of poll fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/poll " +
+ "for proper usage. Possible field values include: " +
+ "duration_minutes, end_datetime, id, options, voting_status")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor PLACE_FIELDS = new PropertyDescriptor.Builder()
+ .name("place-fields")
+ .displayName("Place Fields")
+ .description("A comma-separated list of place fields to be returned as part of the tweet. Refer to " +
+ "https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/place " +
+ "for proper usage. Possible field values include: " +
+ "contained_within, country, country_code, full_name, geo, id, name, place_type")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor EXPANSIONS = new PropertyDescriptor.Builder()
+ .name("expansions")
+ .displayName("Expansions")
+ .description("A comma-separated list of expansions for objects in the returned tweet. See " +
+ "https://developer.twitter.com/en/docs/twitter-api/expansions " +
+ "for proper usage. Possible field values include: " +
+ "author_id, referenced_tweets.id, referenced_tweets.id.author_id, entities.mentions.username, " +
+ "attachments.poll_ids, attachments.media_keys ,in_reply_to_user_id, geo.place_id")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final Relationship REL_SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("FlowFiles containing an array of one or more Tweets")
+ .build();
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+
+ private TweetStreamService tweetStreamService;
+
+ private volatile BlockingQueue<String> messageQueue;
+
+ @Override
+ protected void init(ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptors = new ArrayList<>();
+ descriptors.add(ENDPOINT);
+ descriptors.add(BASE_PATH);
+ descriptors.add(BEARER_TOKEN);
+ descriptors.add(QUEUE_SIZE);
+ descriptors.add(BATCH_SIZE);
+ descriptors.add(BACKOFF_ATTEMPTS);
+ descriptors.add(BACKOFF_TIME);
+ descriptors.add(MAXIMUM_BACKOFF_TIME);
+ descriptors.add(CONNECT_TIMEOUT);
+ descriptors.add(READ_TIMEOUT);
+ descriptors.add(BACKFILL_MINUTES);
+ descriptors.add(TWEET_FIELDS);
+ descriptors.add(USER_FIELDS);
+ descriptors.add(MEDIA_FIELDS);
+ descriptors.add(POLL_FIELDS);
+ descriptors.add(PLACE_FIELDS);
+ descriptors.add(EXPANSIONS);
+
+ this.descriptors = Collections.unmodifiableList(descriptors);
+
+ final Set<Relationship> relationships = new HashSet<>();
+ relationships.add(REL_SUCCESS);
+ this.relationships = Collections.unmodifiableSet(relationships);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return this.descriptors;
+ }
+
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ messageQueue = new LinkedBlockingQueue<>(context.getProperty(QUEUE_SIZE).asInteger());
+
+ tweetStreamService = new TweetStreamService(context, messageQueue, getLogger());
+ tweetStreamService.setBasePath(context.getProperty(BASE_PATH).getValue());
+ final String endpointName = context.getProperty(ENDPOINT).getValue();
+ if (ENDPOINT_SAMPLE.getValue().equals(endpointName)) {
+ tweetStreamService.start(StreamEndpoint.SAMPLE_ENDPOINT);
+ } else {
+ tweetStreamService.start(StreamEndpoint.SEARCH_ENDPOINT);
+ }
+ }
+
+ @Override
+ public void onTrigger(ProcessContext context, ProcessSession session) throws ProcessException {
+ if (messageQueue.isEmpty()) {
+ context.yield();
+ return;
+ }
+
+ FlowFile flowFile = session.create();
+ flowFile = session.write(flowFile, new OutputStreamCallback() {
+ @Override
+ public void process(OutputStream out) throws IOException {
+
+ final int batchSize = context.getProperty(BATCH_SIZE).asInteger();
+ String tweet = messageQueue.poll();
Review Comment:
Making the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r864060973
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/main/java/org/apache/nifi/processors/twitter/TweetStreamService.java:
##########
@@ -0,0 +1,222 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import com.twitter.clientlib.ApiClient;
+import com.twitter.clientlib.ApiException;
+import com.twitter.clientlib.TwitterCredentialsBearer;
+import com.twitter.clientlib.api.TwitterApi;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.HashSet;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TweetStreamService {
+ private final BlockingQueue<String> queue;
+ private final ComponentLog logger;
+
+ private final ScheduledExecutorService executorService;
+
+ private final Set<String> tweetFields;
+ private final Set<String> userFields;
+ private final Set<String> mediaFields;
+ private final Set<String> pollFields;
+ private final Set<String> placeFields;
+ private final Set<String> expansions;
+ private final int backfillMinutes;
+ private final TwitterApi api;
+ private InputStream stream;
+
+ private Long backoffMultiplier;
+ private int backoffAttempts;
+ private int attemptCounter;
+ private Long backoffTime;
+ private Long maximumBackoff;
+
+ private StreamEndpoint endpoint;
+
+ public TweetStreamService(final ProcessContext context, final BlockingQueue<String> queue, final ComponentLog logger) {
+ Objects.requireNonNull(context);
+ Objects.requireNonNull(queue);
+ Objects.requireNonNull(logger);
+
+ this.queue = queue;
+ this.logger = logger;
+
+ this.tweetFields = parseCommaSeparatedPropreties(context, ConsumeTwitter.TWEET_FIELDS);
+ this.userFields = parseCommaSeparatedPropreties(context, ConsumeTwitter.USER_FIELDS);
+ this.mediaFields = parseCommaSeparatedPropreties(context, ConsumeTwitter.MEDIA_FIELDS);
+ this.pollFields = parseCommaSeparatedPropreties(context, ConsumeTwitter.POLL_FIELDS);
+ this.placeFields = parseCommaSeparatedPropreties(context, ConsumeTwitter.PLACE_FIELDS);
+ this.expansions = parseCommaSeparatedPropreties(context, ConsumeTwitter.EXPANSIONS);
+ this.backfillMinutes = context.getProperty(ConsumeTwitter.BACKFILL_MINUTES).asInteger();
+
+ this.backoffMultiplier = 1L;
+ this.backoffAttempts = context.getProperty(ConsumeTwitter.BACKOFF_ATTEMPTS).asInteger();
+ this.attemptCounter = 0;
+ this.backoffTime = context.getProperty(ConsumeTwitter.BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+ this.maximumBackoff = context.getProperty(ConsumeTwitter.MAXIMUM_BACKOFF_TIME).asTimePeriod(TimeUnit.SECONDS);
+
+ ApiClient client = new ApiClient();
+ final int connectTimeout = context.getProperty(ConsumeTwitter.CONNECT_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ final int readTimeout = context.getProperty(ConsumeTwitter.READ_TIMEOUT).asTimePeriod(TimeUnit.MILLISECONDS).intValue();
+ client = client.setConnectTimeout(connectTimeout);
+ client = client.setReadTimeout(readTimeout);
+ api = new TwitterApi(client);
+
+ TwitterCredentialsBearer creds = new TwitterCredentialsBearer(context.getProperty(ConsumeTwitter.BEARER_TOKEN).getValue());
+ api.setTwitterCredentials(creds);
+
+ this.executorService = Executors.newSingleThreadScheduledExecutor();
+ }
+
+ public String getBasePath() {
+ return api.getApiClient().getBasePath();
+ }
+
+ public void setBasePath(final String path) {
+ api.getApiClient().setBasePath(path);
+ }
+
+ public String getTransitUri(final String endpoint) {
+ if (endpoint.equals(StreamEndpoint.SAMPLE_ENDPOINT.getEndpointName())) {
+ return api.getApiClient().getBasePath() + StreamEndpoint.SAMPLE_ENDPOINT.getPath();
+ } else if (endpoint.equals(StreamEndpoint.SEARCH_ENDPOINT.getEndpointName())) {
+ return api.getApiClient().getBasePath() + StreamEndpoint.SEARCH_ENDPOINT.getPath();
+ } else {
+ logger.warn("Unrecognized endpoint in getTransitUri. Returning basePath");
+ return api.getApiClient().getBasePath();
+ }
+ }
+
+ /**
+ * This method would be called when we would like the stream to get started. This method will spin off a thread that
+ * will continue to queue tweets on to the given queue passed in the constructor. The thread will continue
+ * to run until {@code stop} is called.
+ * @param endpoint {@code TwitterStreamAPI.SAMPLE_ENDPOINT} or {@code TwitterStreamAPI.SEARCH_ENDPOINT}
Review Comment:
Making the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r864012463
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/pom.xml:
##########
@@ -35,6 +35,21 @@
<artifactId>nifi-utils</artifactId>
<version>1.17.0-SNAPSHOT</version>
</dependency>
+ <dependency>
+ <groupId>com.twitter</groupId>
+ <artifactId>twitter-api-java-sdk</artifactId>
+ <version>1.1.2</version>
+ <exclusions>
+ <exclusion>
+ <groupId>org.json</groupId>
+ <artifactId>json</artifactId>
+ </exclusion>
Review Comment:
Thanks for the comments, removing it!
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/pom.xml:
##########
@@ -35,6 +35,21 @@
<artifactId>nifi-utils</artifactId>
<version>1.17.0-SNAPSHOT</version>
</dependency>
+ <dependency>
+ <groupId>com.twitter</groupId>
+ <artifactId>twitter-api-java-sdk</artifactId>
+ <version>1.1.2</version>
Review Comment:
Making the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #5947: NIFI-9755: Implement ConsumeTwitter
Posted by GitBox <gi...@apache.org>.
emiliosetiadarma commented on code in PR #5947:
URL: https://github.com/apache/nifi/pull/5947#discussion_r848963270
##########
nifi-nar-bundles/nifi-social-media-bundle/nifi-twitter-processors/src/test/java/org/apache/nifi/processors/twitter/TestConsumeTwitter.java:
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.twitter;
+
+import okhttp3.mockwebserver.MockResponse;
+import okhttp3.mockwebserver.MockWebServer;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+
+public class TestConsumeTwitter {
+ private MockWebServer mockWebServer;
+
+ private TestRunner runner;
+
+ private final String SAMPLE_TWEET = "{\"data\":{\"id\":\"123\",\"text\":\"This is a sample tweet and is not real!\"}}";
+
+ @BeforeEach
+ public void setRunnerAndAPI() {
+ mockWebServer = new MockWebServer();
+
+ runner = TestRunners.newTestRunner(ConsumeTwitter.class);
+
+ runner.setProperty(ConsumeTwitter.BEARER_TOKEN, "BEARER_TOKEN");
+ final String basePath = mockWebServer.url("").toString();
+ runner.setProperty(ConsumeTwitter.BASE_PATH, basePath);
+ }
+
+ @AfterEach
+ public void shutdownServerAndAPI() throws IOException {
+ mockWebServer.shutdown();
+ }
+
+ @Test
+ public void testReceiveSingleTweetInStream() {
+ MockResponse response = new MockResponse()
+ .setResponseCode(200)
+ .setBody(SAMPLE_TWEET)
+ .addHeader("Content-Type", "application/json");
+ mockWebServer.enqueue(response);
+
+
+ runner.setProperty(ConsumeTwitter.ENDPOINT, ConsumeTwitter.ENDPOINT_SAMPLE);
+ runner.setProperty(ConsumeTwitter.QUEUE_SIZE, "10000");
+ runner.setProperty(ConsumeTwitter.BATCH_SIZE, "10");
+ runner.setProperty(ConsumeTwitter.BACKFILL_MINUTES, "0");
+
+ runner.assertValid();
+
+ // the TwitterStreamAPI class spins up another thread and might not be done queueing tweets in one run of the processor
+ final int maxTries = 100;
+ int tries = 0;
+ while (tries < maxTries && runner.getFlowFilesForRelationship(ConsumeTwitter.REL_SUCCESS).size() == 0) {
+ runner.run();
+ tries++;
+ }
Review Comment:
Going to replace the loop with
` final int maxTries = 3;
final long runSchedule = 250;
runner.setRunSchedule(runSchedule);
runner.run(maxTries, false, true);
runner.stop();` To space out calls to `onTrigger.` This gives time for the one example tweet to be queued. Let me know your thoughts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org