You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@streams.apache.org by rbnks <gi...@git.apache.org> on 2015/02/17 18:02:49 UTC
[GitHub] incubator-streams pull request: Streams 279
GitHub user rbnks opened a pull request:
https://github.com/apache/incubator-streams/pull/184
Streams 279
Youtube Channel data collector. This is the youtube version of bio history.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rbnks/incubator-streams STREAMS-279
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-streams/pull/184.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #184
----
commit 83bc516f4b00d44d0d43f07a1735a697be6b29db
Author: Ryan Ebanks <rb...@users.noreply.github.com>
Date: 2015-02-03T21:07:01Z
Merge pull request #19 from apache/master
Apache Master Merge 2015/02/03
commit 1ac8c83cdb35c79f61a005b1f827ce26c562aff2
Author: Ryan Ebanks <rb...@users.noreply.github.com>
Date: 2015-02-10T22:36:44Z
Merge pull request #20 from apache/master
Merge Apache Master 2015/02/10
commit 850ae97dd2ccf15455778f8c95e49afc35e103de
Author: Ryan Ebanks <ry...@gmail.com>
Date: 2015-02-17T16:56:48Z
STREAMS-279 | Youtube Channel data. Youtubes version of biohistory
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-streams pull request: Streams 279
Posted by rbnks <gi...@git.apache.org>.
Github user rbnks commented on a diff in the pull request:
https://github.com/apache/incubator-streams/pull/184#discussion_r24846977
--- Diff: streams-contrib/streams-provider-youtube/src/main/java/com/youtube/serializer/YoutubeEventClassifier.java ---
@@ -30,6 +30,7 @@
public class YoutubeEventClassifier {
private static ObjectMapper mapper = new StreamsJacksonMapper();
private static final String VIDEO_IDENTIFIER = "\"youtube#video\"";
+ private static final String CHANNEL_IDENTIFIER = "youtube#channel";
--- End diff --
yes, because I did .contains() instead of .equals()
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-streams pull request: Streams 279
Posted by robdouglas <gi...@git.apache.org>.
Github user robdouglas commented on a diff in the pull request:
https://github.com/apache/incubator-streams/pull/184#discussion_r24832205
--- Diff: streams-contrib/streams-provider-youtube/src/main/java/com/youtube/serializer/YoutubeActivityUtil.java ---
@@ -67,6 +70,40 @@ public static void updateActivity(Video video, Activity activity, String channel
addYoutubeExtensions(activity, video);
}
+
+ /**
+ * Given a {@link com.google.api.services.youtube.model.Channel} object and an
+ * {@link org.apache.streams.pojo.json.Activity} object, fill out the appropriate details
+ *
+ * @param channel
+ * @param activity
+ * @throws org.apache.streams.exceptions.ActivitySerializerException
+ */
+ public static void updateActivity(Channel channel, Activity activity, String channelId) throws ActivitySerializerException {
+ try {
+ activity.setProvider(getProvider());
+ Actor actor = new Actor();
--- End diff --
Could you move this actor creation to a separate method?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-streams pull request: Streams 279
Posted by robdouglas <gi...@git.apache.org>.
Github user robdouglas commented on a diff in the pull request:
https://github.com/apache/incubator-streams/pull/184#discussion_r24848456
--- Diff: streams-contrib/streams-provider-youtube/src/main/java/com/youtube/serializer/YoutubeEventClassifier.java ---
@@ -30,6 +30,7 @@
public class YoutubeEventClassifier {
private static ObjectMapper mapper = new StreamsJacksonMapper();
private static final String VIDEO_IDENTIFIER = "\"youtube#video\"";
+ private static final String CHANNEL_IDENTIFIER = "youtube#channel";
--- End diff --
Ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-streams pull request: Streams 279
Posted by robdouglas <gi...@git.apache.org>.
Github user robdouglas commented on a diff in the pull request:
https://github.com/apache/incubator-streams/pull/184#discussion_r24832308
--- Diff: streams-contrib/streams-provider-youtube/src/main/java/com/youtube/serializer/YoutubeChannelDeserializer.java ---
@@ -0,0 +1,131 @@
+package com.youtube.serializer;
+
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.DeserializationContext;
+import com.fasterxml.jackson.databind.JsonDeserializer;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.google.api.client.util.DateTime;
+import com.google.api.services.youtube.model.Channel;
+import com.google.api.services.youtube.model.ChannelContentDetails;
+import com.google.api.services.youtube.model.ChannelLocalization;
+import com.google.api.services.youtube.model.ChannelSnippet;
+import com.google.api.services.youtube.model.ChannelStatistics;
+import com.google.api.services.youtube.model.ChannelTopicDetails;
+import com.google.api.services.youtube.model.Thumbnail;
+import com.google.api.services.youtube.model.ThumbnailDetails;
+import com.google.common.collect.Lists;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.List;
+
+/**
+ *
+ */
+public class YoutubeChannelDeserializer extends JsonDeserializer<Channel> {
+
+
+ @Override
+ public Channel deserialize(JsonParser jp, DeserializationContext ctxt) throws IOException, JsonProcessingException {
+ JsonNode node = jp.getCodec().readTree(jp);
+ try {
+ Channel channel = new Channel();
+ if(node.findPath("etag") != null)
+ channel.setEtag(node.get("etag").asText());
+ if(node.findPath("kind") != null)
+ channel.setKind(node.get("kind").asText());
+ channel.setId(node.get("id").asText());
+ channel.setTopicDetails(setTopicDetails(node.findValue("topicDetails")));
+ channel.setStatistics(setChannelStatistics(node.findValue("statistics")));
+ channel.setContentDetails(setContentDetails(node.findValue("contentDetails")));
+ channel.setSnippet(setChannelSnippet(node.findValue("snippet")));
+ return channel;
+ } catch (Throwable t) {
+ throw new IOException(t);
+ }
--- End diff --
In the event that there is a deserialization error, could we log the error message and return a null object? I'm not sure we want to derail the entire operation if one datum has trouble getting deserialized.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-streams pull request: Streams 279
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-streams/pull/184
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-streams pull request: Streams 279
Posted by robdouglas <gi...@git.apache.org>.
Github user robdouglas commented on a diff in the pull request:
https://github.com/apache/incubator-streams/pull/184#discussion_r24832360
--- Diff: streams-contrib/streams-provider-youtube/src/main/java/com/youtube/serializer/YoutubeEventClassifier.java ---
@@ -30,6 +30,7 @@
public class YoutubeEventClassifier {
private static ObjectMapper mapper = new StreamsJacksonMapper();
private static final String VIDEO_IDENTIFIER = "\"youtube#video\"";
+ private static final String CHANNEL_IDENTIFIER = "youtube#channel";
--- End diff --
Does this work without the escaped quotes?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-streams pull request: Streams 279
Posted by robdouglas <gi...@git.apache.org>.
Github user robdouglas commented on a diff in the pull request:
https://github.com/apache/incubator-streams/pull/184#discussion_r24848425
--- Diff: streams-contrib/streams-provider-youtube/src/main/java/com/youtube/serializer/YoutubeChannelDeserializer.java ---
@@ -0,0 +1,131 @@
+package com.youtube.serializer;
+
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.DeserializationContext;
+import com.fasterxml.jackson.databind.JsonDeserializer;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.google.api.client.util.DateTime;
+import com.google.api.services.youtube.model.Channel;
+import com.google.api.services.youtube.model.ChannelContentDetails;
+import com.google.api.services.youtube.model.ChannelLocalization;
+import com.google.api.services.youtube.model.ChannelSnippet;
+import com.google.api.services.youtube.model.ChannelStatistics;
+import com.google.api.services.youtube.model.ChannelTopicDetails;
+import com.google.api.services.youtube.model.Thumbnail;
+import com.google.api.services.youtube.model.ThumbnailDetails;
+import com.google.common.collect.Lists;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.List;
+
+/**
+ *
+ */
+public class YoutubeChannelDeserializer extends JsonDeserializer<Channel> {
+
+
+ @Override
+ public Channel deserialize(JsonParser jp, DeserializationContext ctxt) throws IOException, JsonProcessingException {
+ JsonNode node = jp.getCodec().readTree(jp);
+ try {
+ Channel channel = new Channel();
+ if(node.findPath("etag") != null)
+ channel.setEtag(node.get("etag").asText());
+ if(node.findPath("kind") != null)
+ channel.setKind(node.get("kind").asText());
+ channel.setId(node.get("id").asText());
+ channel.setTopicDetails(setTopicDetails(node.findValue("topicDetails")));
+ channel.setStatistics(setChannelStatistics(node.findValue("statistics")));
+ channel.setContentDetails(setContentDetails(node.findValue("contentDetails")));
+ channel.setSnippet(setChannelSnippet(node.findValue("snippet")));
+ return channel;
+ } catch (Throwable t) {
+ throw new IOException(t);
+ }
--- End diff --
Sounds good
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-streams pull request: Streams 279
Posted by rbnks <gi...@git.apache.org>.
Github user rbnks commented on a diff in the pull request:
https://github.com/apache/incubator-streams/pull/184#discussion_r24847322
--- Diff: streams-contrib/streams-provider-youtube/src/main/java/com/youtube/serializer/YoutubeChannelDeserializer.java ---
@@ -0,0 +1,131 @@
+package com.youtube.serializer;
+
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.DeserializationContext;
+import com.fasterxml.jackson.databind.JsonDeserializer;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.google.api.client.util.DateTime;
+import com.google.api.services.youtube.model.Channel;
+import com.google.api.services.youtube.model.ChannelContentDetails;
+import com.google.api.services.youtube.model.ChannelLocalization;
+import com.google.api.services.youtube.model.ChannelSnippet;
+import com.google.api.services.youtube.model.ChannelStatistics;
+import com.google.api.services.youtube.model.ChannelTopicDetails;
+import com.google.api.services.youtube.model.Thumbnail;
+import com.google.api.services.youtube.model.ThumbnailDetails;
+import com.google.common.collect.Lists;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.List;
+
+/**
+ *
+ */
+public class YoutubeChannelDeserializer extends JsonDeserializer<Channel> {
+
+
+ @Override
+ public Channel deserialize(JsonParser jp, DeserializationContext ctxt) throws IOException, JsonProcessingException {
+ JsonNode node = jp.getCodec().readTree(jp);
+ try {
+ Channel channel = new Channel();
+ if(node.findPath("etag") != null)
+ channel.setEtag(node.get("etag").asText());
+ if(node.findPath("kind") != null)
+ channel.setKind(node.get("kind").asText());
+ channel.setId(node.get("id").asText());
+ channel.setTopicDetails(setTopicDetails(node.findValue("topicDetails")));
+ channel.setStatistics(setChannelStatistics(node.findValue("statistics")));
+ channel.setContentDetails(setContentDetails(node.findValue("contentDetails")));
+ channel.setSnippet(setChannelSnippet(node.findValue("snippet")));
+ return channel;
+ } catch (Throwable t) {
+ throw new IOException(t);
+ }
--- End diff --
Its a advertised exception that is part of the overriden method. Any code call this method should be forced to handle this exception and choses how to respond to the exception. I believe this is fine here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-streams pull request: Streams 279
Posted by rbnks <gi...@git.apache.org>.
Github user rbnks commented on a diff in the pull request:
https://github.com/apache/incubator-streams/pull/184#discussion_r24847039
--- Diff: streams-contrib/streams-provider-youtube/src/main/java/com/youtube/processor/YoutubeTypeConverter.java ---
@@ -83,9 +91,11 @@ public YoutubeTypeConverter() {}
private Object deserializeItem(Object item) {
try {
Class klass = YoutubeEventClassifier.detectClass((String) item);
-
+ System.out.println(klass.getName());
--- End diff --
will do.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-streams pull request: Streams 279
Posted by robdouglas <gi...@git.apache.org>.
Github user robdouglas commented on the pull request:
https://github.com/apache/incubator-streams/pull/184#issuecomment-75624677
:+1:
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-streams pull request: Streams 279
Posted by robdouglas <gi...@git.apache.org>.
Github user robdouglas commented on a diff in the pull request:
https://github.com/apache/incubator-streams/pull/184#discussion_r24832039
--- Diff: streams-contrib/streams-provider-youtube/src/main/java/com/youtube/processor/YoutubeTypeConverter.java ---
@@ -83,9 +91,11 @@ public YoutubeTypeConverter() {}
private Object deserializeItem(Object item) {
try {
Class klass = YoutubeEventClassifier.detectClass((String) item);
-
+ System.out.println(klass.getName());
--- End diff --
If we're going to keep this, can we change it to a log statement?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---