You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by uce <gi...@git.apache.org> on 2015/12/07 14:36:30 UTC
[GitHub] flink pull request: [contrib, connector-wikiedits] Add WikipediaEd...
GitHub user uce opened a pull request:
https://github.com/apache/flink/pull/1439
[contrib, connector-wikiedits] Add WikipediaEditsSource
A non-parallel source that parses a live stream of Wikipedia edits.
Meta data about the edits is mirrored to the IRC channel `#en.wikipedia`. The source establishes a connection to this IRC channel and parses the messages into `WikipediaEditEvent` instances.
The purpose of this source is to ease the setup of demos of the `DataStream` API with live data.
The original idea is from the [Hello Samza](https://samza.apache.org/startup/hello-samza/latest/) project of [Apache Samza](http://samza.apache.org). The Samza code for this is located in the [samza-hello-samza](https://github.com/apache/samza-hello-samza) repository.
## Example
Add the following dependency to your project:
```xml
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-wikiedits</artifactId>
<version>1.0-SNAPSHOT</version>
</dependency>
```
You can use the source like regular sources:
```java
StreamExecutionEnvironment env = StreamExecutionEnvironment
.getExecutionEnvironment();
DataStream<WikipediaEditEvent> edits = env
.addSource(new WikipediaEditsSource());
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/uce/flink wikiedits
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/1439.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1439
----
commit 2c6aa8c874e8e89b12e56af7e8f415b6f4cf49ff
Author: Ufuk Celebi <uc...@apache.org>
Date: 2015-12-06T20:45:00Z
[contrib, connector-wikiedits] Add WikipediaEditsSource
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [contrib, connector-wikiedits] Add WikipediaEd...
Posted by uce <gi...@git.apache.org>.
Github user uce commented on the pull request:
https://github.com/apache/flink/pull/1439#issuecomment-164406033
Is it OK to merge this to `flink-contrib`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [contrib, connector-wikiedits] Add WikipediaEd...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/1439
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [contrib, connector-wikiedits] Add WikipediaEd...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/1439#issuecomment-164499427
Looks good to me, +1 to merge this!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [contrib, connector-wikiedits] Add WikipediaEd...
Posted by jaoki <gi...@git.apache.org>.
Github user jaoki commented on a diff in the pull request:
https://github.com/apache/flink/pull/1439#discussion_r46924797
--- Diff: flink-contrib/flink-connector-wikiedits/pom.xml ---
@@ -0,0 +1,50 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
+
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-contrib-parent</artifactId>
+ <version>1.0-SNAPSHOT</version>
+ <relativePath>..</relativePath>
+ </parent>
+
+ <artifactId>flink-connector-wikiedits</artifactId>
+ <name>flink-connector-wikiedits</name>
+
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-streaming-java</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>org.schwering</groupId>
+ <artifactId>irclib</artifactId>
+ <version>1.10</version>
--- End diff --
Very interesting stuff! Do you know where I can find irclib 1.10 source code online (like github)?
I found 2.0 beta https://github.com/java-irclib/irclib but can not find the older version.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [contrib, connector-wikiedits] Add WikipediaEd...
Posted by uce <gi...@git.apache.org>.
Github user uce commented on a diff in the pull request:
https://github.com/apache/flink/pull/1439#discussion_r47482737
--- Diff: flink-contrib/flink-connector-wikiedits/pom.xml ---
@@ -0,0 +1,50 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
+
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-contrib-parent</artifactId>
+ <version>1.0-SNAPSHOT</version>
+ <relativePath>..</relativePath>
+ </parent>
+
+ <artifactId>flink-connector-wikiedits</artifactId>
+ <name>flink-connector-wikiedits</name>
+
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-streaming-java</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>org.schwering</groupId>
+ <artifactId>irclib</artifactId>
+ <version>1.10</version>
--- End diff --
I've only used it as a Maven dependency myself.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [contrib, connector-wikiedits] Add WikipediaEd...
Posted by jaoki <gi...@git.apache.org>.
Github user jaoki commented on a diff in the pull request:
https://github.com/apache/flink/pull/1439#discussion_r46926616
--- Diff: flink-contrib/flink-connector-wikiedits/pom.xml ---
@@ -0,0 +1,50 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
+
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-contrib-parent</artifactId>
+ <version>1.0-SNAPSHOT</version>
+ <relativePath>..</relativePath>
+ </parent>
+
+ <artifactId>flink-connector-wikiedits</artifactId>
+ <name>flink-connector-wikiedits</name>
+
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-streaming-java</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>org.schwering</groupId>
+ <artifactId>irclib</artifactId>
+ <version>1.10</version>
--- End diff --
Maybe this http://sourceforge.net/p/moepii/code/HEAD/tree/trunk/ but I can't find a 1.10 tag?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---