You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by HilmiYildirim <gi...@git.apache.org> on 2015/05/19 14:38:47 UTC
[GitHub] flink pull request: Implemented TwitterSourceFilter and adapted Tw...
GitHub user HilmiYildirim opened a pull request:
https://github.com/apache/flink/pull/695
Implemented TwitterSourceFilter and adapted TwitterSource
I implemented the class TwitterFilterSource which inherits from TwitterSource. It enables to filter twitter streams, for example, by terms, languages and locations. Furthermore, you can add "followings" to the stream by adding user ids to the config. Then, the stream includes the tweets of these persons. Furthermore, it is possible to define query and post parameters.
I had to make some changes in the TwitterSource file. I had to change some variables and methods from private to protected. Furthermore, now TwitterSource inherits from RichSourceFunction instead of RichParallelSourceFunction. RichParallelSourceFunction does not make sense. With a twitter handle it is only possible to open 2 twitter streams in parallel. But if TwitterSource inherits from RichSourceFunction then only one stream will be opened and everything works fine.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/HilmiYildirim/flink TwitterFilterSource
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/695.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #695
----
commit a89d3aec62a2f5ed3ec8246148ba289f777b9dbb
Author: yildirim <yi...@gronau.neofonie.priv>
Date: 2015-05-19T12:27:27Z
Implemented TwitterSourceFilter and adapted TwitterSource
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Implemented TwitterSourceFilter and adapted Tw...
Posted by HilmiYildirim <gi...@git.apache.org>.
Github user HilmiYildirim commented on a diff in the pull request:
https://github.com/apache/flink/pull/695#discussion_r30677156
--- Diff: flink-staging/flink-streaming/flink-streaming-connectors/src/test/java/org/apache/flink/streaming/connectors/twitter/TwitterFilterSourceTest.java ---
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.twitter;
+
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.connectors.json.JSONParseFlatMap;
+import org.apache.flink.util.Collector;
+
+public class TwitterFilterSourceTest {
+
+ private static final String PATH_TO_AUTH_FILE = "/twitter.properties";
+
+ public static void main(String[] args) {
--- End diff --
I think there is a problem of overwriting com.twitter.hbc.core.Client. The stream filtering process is done by the twitter server. The filter parameters (terms, languages, ...) are sent to the twitter server to establish the twitter stream. Then, the twitter server only sends tweets to the client which fit the filtering criteria. Therefore, if I overwrite com.twitter.hbc.core.Client then I have to implement the whole filtering mechanism which leads to, that the Junit tests test the "test implementation" instead of the "real implementation".
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Implemented TwitterSourceFilter and adapted Tw...
Posted by HilmiYildirim <gi...@git.apache.org>.
Github user HilmiYildirim commented on the pull request:
https://github.com/apache/flink/pull/695#issuecomment-107551663
I think it is good.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Implemented TwitterSourceFilter and adapted Tw...
Posted by HilmiYildirim <gi...@git.apache.org>.
Github user HilmiYildirim commented on the pull request:
https://github.com/apache/flink/pull/695#issuecomment-105415577
The problem is as follows. While I was working on the commit, the interface SourceFunction has changed and was merged with the master. Now, it has the methods reachedEnd and next. Cause TwitterSource inherits from SourceFunction, the nextReachedEnd and next methods were implemented in the class. The problem os that the implementation has bugs and the old methods are not deleted which leads to that the class also has dead code. Therefore, the TwitterFilterSource class was not working and, therefore, TwitterFilterSource was also not working. Therefore, I have fixed the class TwitterSource and removed the old code. Now, it is working properly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Implemented TwitterSourceFilter and adapted Tw...
Posted by hsaputra <gi...@git.apache.org>.
Github user hsaputra commented on a diff in the pull request:
https://github.com/apache/flink/pull/695#discussion_r30620574
--- Diff: flink-staging/flink-streaming/flink-streaming-connectors/src/main/java/org/apache/flink/streaming/connectors/twitter/TwitterFilterSource.java ---
@@ -0,0 +1,284 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.twitter;
+
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.twitter.hbc.core.endpoint.Location;
+import com.twitter.hbc.core.endpoint.StatusesFilterEndpoint;
+import com.twitter.hbc.httpclient.auth.Authentication;
+
+/**
+ *
+ * An extension of {@link TwitterSource} by filter parameters. This extension
+ * enables to filter the twitter stream by user defined parameters.
+ *
+ * @author Hilmi Yildirim
--- End diff --
Could you remove @author javadoc annotation?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Implemented TwitterSourceFilter and adapted Tw...
Posted by HilmiYildirim <gi...@git.apache.org>.
Github user HilmiYildirim commented on the pull request:
https://github.com/apache/flink/pull/695#issuecomment-104243817
The problem was that the TwitterSource.next method was not implemented properly. There were also dead code. I fixed it and now it is working.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Implemented TwitterSourceFilter and adapted Tw...
Posted by HilmiYildirim <gi...@git.apache.org>.
Github user HilmiYildirim commented on the pull request:
https://github.com/apache/flink/pull/695#issuecomment-105884995
Done. I hope every thing works fine.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Implemented TwitterSourceFilter and adapted Tw...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/695#issuecomment-108290010
Thanks for your work. :smile:
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Implemented TwitterSourceFilter and adapted Tw...
Posted by HilmiYildirim <gi...@git.apache.org>.
Github user HilmiYildirim commented on a diff in the pull request:
https://github.com/apache/flink/pull/695#discussion_r30622577
--- Diff: flink-staging/flink-streaming/flink-streaming-connectors/src/test/java/org/apache/flink/streaming/connectors/twitter/TwitterFilterSourceTest.java ---
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.twitter;
+
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.connectors.json.JSONParseFlatMap;
+import org.apache.flink.util.Collector;
+
+public class TwitterFilterSourceTest {
+
+ private static final String PATH_TO_AUTH_FILE = "/twitter.properties";
+
+ public static void main(String[] args) {
--- End diff --
Unfortunately, it is not possible it implement this as Junit Tests. The stream connection needs access keys of a twitter account and the maximum number of parallel streams per twitter account is restricted to 2. If more than 2 people are executing the tests at the same time then the tests fail.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Implemented TwitterSourceFilter and adapted Tw...
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the pull request:
https://github.com/apache/flink/pull/695#issuecomment-103579307
That would be really great, thank you!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Implemented TwitterSourceFilter and adapted Tw...
Posted by HilmiYildirim <gi...@git.apache.org>.
Github user HilmiYildirim commented on the pull request:
https://github.com/apache/flink/pull/695#issuecomment-104230142
TwitterSource does not work after the last update. Now, it sends null messages. I try to fix it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Implemented TwitterSourceFilter and adapted Tw...
Posted by HilmiYildirim <gi...@git.apache.org>.
Github user HilmiYildirim commented on a diff in the pull request:
https://github.com/apache/flink/pull/695#discussion_r30621383
--- Diff: flink-staging/flink-streaming/flink-streaming-connectors/src/main/java/org/apache/flink/streaming/connectors/twitter/TwitterFilterSource.java ---
@@ -0,0 +1,284 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.twitter;
+
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.twitter.hbc.core.endpoint.Location;
+import com.twitter.hbc.core.endpoint.StatusesFilterEndpoint;
+import com.twitter.hbc.httpclient.auth.Authentication;
+
+/**
+ *
+ * An extension of {@link TwitterSource} by filter parameters. This extension
+ * enables to filter the twitter stream by user defined parameters.
+ *
+ * @author Hilmi Yildirim
--- End diff --
Yes I can remove the annotation.
There is a problem with the junit tests. For them we need a twitter account
and the access keys for the stream connection have to be in the resource
folder. This means that other people can use the access keys which can
influence the tests. I think there is no secure way to implement junit
tests.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Implemented TwitterSourceFilter and adapted Tw...
Posted by hsaputra <gi...@git.apache.org>.
Github user hsaputra commented on a diff in the pull request:
https://github.com/apache/flink/pull/695#discussion_r30724769
--- Diff: flink-staging/flink-streaming/flink-streaming-connectors/src/test/java/org/apache/flink/streaming/connectors/twitter/TwitterFilterSourceTest.java ---
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.twitter;
+
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.connectors.json.JSONParseFlatMap;
+import org.apache.flink.util.Collector;
+
+public class TwitterFilterSourceTest {
--- End diff --
Could you add JavaDoc comment on what is needed to run this test and kind of small description why it is not part of JUnit test for now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Implemented TwitterSourceFilter and adapted Tw...
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/695#discussion_r30676489
--- Diff: flink-staging/flink-streaming/flink-streaming-connectors/src/test/java/org/apache/flink/streaming/connectors/twitter/TwitterFilterSourceTest.java ---
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.twitter;
+
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.connectors.json.JSONParseFlatMap;
+import org.apache.flink.util.Collector;
+
+public class TwitterFilterSourceTest {
+
+ private static final String PATH_TO_AUTH_FILE = "/twitter.properties";
+
+ public static void main(String[] args) {
--- End diff --
I see. The source is expecting a running twitter service.
Our tests should be self-contained, without relying on external services.
For the Kafka connector, we are starting an internal testing kafka server in the same JVM and test Flink-kafka against that one.
Do you think its doable to implement a little server which is behaving like twitters APIs ?
That would be the very best way to test this reliably.
Maybe it would be sufficient to make `initializeClient()` protected, create a `TestingTwitterSource` which is overriding `initializeClient()` and creating a testing implementation of `com.twitter.hbc.core.Client` ?
But I also understand that its probably more work to implement that test compared to your contribution.
Since we didn't have tests before, I would also agree to merge your changes now and file a issue for implementing the test.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Implemented TwitterSourceFilter and adapted Tw...
Posted by hsaputra <gi...@git.apache.org>.
Github user hsaputra commented on a diff in the pull request:
https://github.com/apache/flink/pull/695#discussion_r30620688
--- Diff: flink-staging/flink-streaming/flink-streaming-connectors/src/test/java/org/apache/flink/streaming/connectors/twitter/TwitterFilterSourceTest.java ---
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.twitter;
+
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.connectors.json.JSONParseFlatMap;
+import org.apache.flink.util.Collector;
+
+public class TwitterFilterSourceTest {
+
+ private static final String PATH_TO_AUTH_FILE = "/twitter.properties";
+
+ public static void main(String[] args) {
--- End diff --
We are using JUnit for our tests so it would be great to convert this to JUnit style of test class.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Implemented TwitterSourceFilter and adapted Tw...
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the pull request:
https://github.com/apache/flink/pull/695#issuecomment-103498471
Thank you for working on the issue. I'll soon review your changes in detail.
From what I saw by scrolling over it, it looks very good (with log messages and documentation).
One thing that caught my attention was that the included test is not really a unit or integration test. Its more like an example.
Do you think there is a way to turn this into a real JUnit test?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Implemented TwitterSourceFilter and adapted Tw...
Posted by hsaputra <gi...@git.apache.org>.
Github user hsaputra commented on a diff in the pull request:
https://github.com/apache/flink/pull/695#discussion_r30724526
--- Diff: flink-staging/flink-streaming/flink-streaming-connectors/src/test/java/org/apache/flink/streaming/connectors/twitter/TwitterFilterSourceTest.java ---
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.twitter;
+
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.connectors.json.JSONParseFlatMap;
+import org.apache.flink.util.Collector;
+
+public class TwitterFilterSourceTest {
+
+ private static final String PATH_TO_AUTH_FILE = "/twitter.properties";
+
+ public static void main(String[] args) {
--- End diff --
Sounds good to me
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Implemented TwitterSourceFilter and adapted Tw...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/695#issuecomment-105845451
I tried to merge this, but it is quite unmergeable.
Before we can add this, we need you to rebase this pull request and squash all commits into one commit.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Implemented TwitterSourceFilter and adapted Tw...
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the pull request:
https://github.com/apache/flink/pull/695#issuecomment-103848061
+1 to merge like this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Implemented TwitterSourceFilter and adapted Tw...
Posted by HilmiYildirim <gi...@git.apache.org>.
Github user HilmiYildirim commented on the pull request:
https://github.com/apache/flink/pull/695#issuecomment-103597087
Unfortunately, it is not possible ti implement this as Junit Tests. The stream connection needs access keys of a twitter account and the maximum number of parallel streams per twitter account is restricted to 2. If more then 2 people are executing the test at the same time then the tests are not working.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Implemented TwitterSourceFilter and adapted Tw...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/695
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Implemented TwitterSourceFilter and adapted Tw...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/695#issuecomment-107422496
Is this good to merge now? I'm rewriting all the sources (again) and this should probably go in.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Implemented TwitterSourceFilter and adapted Tw...
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the pull request:
https://github.com/apache/flink/pull/695#issuecomment-104705751
It looks like many unrelated commits are part of the pull request. I suspect something went wrong while merging/rebasing.
Can you take a look?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Implemented TwitterSourceFilter and adapted Tw...
Posted by HilmiYildirim <gi...@git.apache.org>.
Github user HilmiYildirim commented on the pull request:
https://github.com/apache/flink/pull/695#issuecomment-104165135
I renamed the TwitterFilterSourceTest to TwitterFilterSourceExample because it is more an example instead of a test. Further, I added JavaDoc comments and I also added a twitter.properties dummy to the resource folder that you only has to fill the twitter.properties to execute the example.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Implemented TwitterSourceFilter and adapted Tw...
Posted by HilmiYildirim <gi...@git.apache.org>.
Github user HilmiYildirim commented on a diff in the pull request:
https://github.com/apache/flink/pull/695#discussion_r30622734
--- Diff: flink-staging/flink-streaming/flink-streaming-connectors/src/test/java/org/apache/flink/streaming/connectors/twitter/TwitterFilterSourceTest.java ---
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.twitter;
+
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.connectors.json.JSONParseFlatMap;
+import org.apache.flink.util.Collector;
+
+public class TwitterFilterSourceTest {
+
+ private static final String PATH_TO_AUTH_FILE = "/twitter.properties";
+
+ public static void main(String[] args) {
--- End diff --
But it would be possible to implement this as Junit tests and the tests will be skipped by default. Someone who wants to execute the tests has to use his own access keys.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Implemented TwitterSourceFilter and adapted Tw...
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/695#discussion_r30691440
--- Diff: flink-staging/flink-streaming/flink-streaming-connectors/src/test/java/org/apache/flink/streaming/connectors/twitter/TwitterFilterSourceTest.java ---
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.twitter;
+
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.connectors.json.JSONParseFlatMap;
+import org.apache.flink.util.Collector;
+
+public class TwitterFilterSourceTest {
+
+ private static final String PATH_TO_AUTH_FILE = "/twitter.properties";
+
+ public static void main(String[] args) {
--- End diff --
I agree, lets not add a test for now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Implemented TwitterSourceFilter and adapted Tw...
Posted by HilmiYildirim <gi...@git.apache.org>.
Github user HilmiYildirim commented on the pull request:
https://github.com/apache/flink/pull/695#issuecomment-103499923
It is possible to implement an integration test. For example, I can
track a term and then I has to check if all tweets contain the term. If
you want I can implement such tests.
Best Regards,
Hilmi
Am 19.05.2015 um 15:39 schrieb Robert Metzger:
>
> Thank you for working on the issue. I'll soon review your changes in
> detail.
> From what I saw by scrolling over it, it looks very good (with log
> messages and documentation).
> One thing that caught my attention was that the included test is not
> really a unit or integration test. Its more like an example.
> Do you think there is a way to turn this into a real JUnit test?
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/flink/pull/695#issuecomment-103498471>.
>
--
--
Hilmi Yildirim
Software Developer R&D
T: +49 30 24627-281
Hilmi.Yildirim@neofonie.de
http://www.neofonie.de
Besuchen Sie den Neo Tech Blog für Anwender:
http://blog.neofonie.de/
Folgen Sie uns:
https://plus.google.com/+neofonie
http://www.linkedin.com/company/neofonie-gmbh
https://www.xing.com/companies/neofoniegmbh
Neofonie GmbH | Robert-Koch-Platz 4 | 10115 Berlin
Handelsregister Berlin-Charlottenburg: HRB 67460
Geschäftsführung: Thomas Kitlitschko
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Implemented TwitterSourceFilter and adapted Tw...
Posted by hsaputra <gi...@git.apache.org>.
Github user hsaputra commented on the pull request:
https://github.com/apache/flink/pull/695#issuecomment-103963870
Add one more small request to add comments as JavaDoc for the test then it should ready to go.
Thanks again for the contribution.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---