You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by Yiming Liu <li...@gmail.com> on 2016/07/20 10:16:58 UTC
Feature Request: Support Embedded Structure when Parsing Streaming Message
Hi Kylin devs,
According to the doc:
http://kylin.apache.org/docs15/tutorial/cube_streaming.html, the Kafka
incoming message should be in flatten structure. But it's too limit for
JSON message. May I request the feature which supports embedded structure
raw messages, and convert them into flatten table structure?
One example:
{
"createdAt": "Jul 20, 2016 9:59:17 AM",
"id": 755703618762862600,
"text": "dejamos las tapas regionales de este #Miercoles
https://t.co/kfe0kT2Fup",
"source": "<a href=\"http://twitter.com\" rel=\"nofollow\">Twitter Web
Client</a>",
"isTruncated": false,
"inReplyToStatusId": -1,
"inReplyToUserId": -1,
"isFavorited": false,
"isRetweeted": false,
"favoriteCount": 0,
"retweetCount": 0,
"isPossiblySensitive": false,
"lang": "es",
"contributorsIDs": [],
"userMentionEntities": [],
"urlEntities": [],
"hashtagEntities": [
{
"text": "Miercoles",
"start": 37,
"end": 47
}
],
"symbolEntities": [],
"currentUserRetweetId": -1,
"user": {
"id": 4853763947,
"name": "El Metropolitano",
"screenName": "ElTWdelMetro",
"description": "Noticias, an��lisis e informaci��n para el crecimiento
de la regi��n.",
"descriptionURLEntities": [],
"isContributorsEnabled": false,
"profileImageUrl": "
http://pbs.twimg.com/profile_images/692361953377927169/cS7lUaSj_normal.png",
"profileImageUrlHttps": "
https://pbs.twimg.com/profile_images/692361953377927169/cS7lUaSj_normal.png
",
"isDefaultProfileImage": false,
"url": "http://elmetropolitano.com.ar/",
"isProtected": false,
"followersCount": 153,
"profileBackgroundColor": "F5F8FA",
"profileTextColor": "333333",
"profileLinkColor": "2B7BB9",
"profileSidebarFillColor": "DDEEF6",
"profileSidebarBorderColor": "C0DEED",
"profileUseBackgroundImage": true,
"isDefaultProfile": true,
"showAllInlineMedia": false,
"friendsCount": 169,
"createdAt": "Jan 27, 2016 3:00:21 PM",
"favouritesCount": 9,
"utcOffset": -25200,
"timeZone": "Pacific Time (US & Canada)",
"profileBackgroundImageUrl": "",
"profileBackgroundImageUrlHttps": "",
"profileBannerImageUrl": "
https://pbs.twimg.com/profile_banners/4853763947/1453907062",
"profileBackgroundTiled": false,
"lang": "es",
"statusesCount": 1000,
"isGeoEnabled": false,
"isVerified": false,
"translator": false,
"listedCount": 0,
"isFollowRequestSent": false
}
}
The flatten structure will like this:
"createdAt": "Jul 20, 2016 9:59:17 AM",
...
"user_id": 4853763947
--
With Warm regards
Yiming Liu (刘一鸣)
Re: Feature Request: Support Embedded Structure when Parsing
Streaming Message
Posted by Yiming Liu <li...@gmail.com>.
Thanks, Yang.
KYLIN-1919 filed.
2016-07-24 17:38 GMT+08:00 Li Yang <li...@apache.org>:
> Sure, pls open a JIRA.
>
> On Wed, Jul 20, 2016 at 6:16 PM, Yiming Liu <li...@gmail.com>
> wrote:
>
> > Hi Kylin devs,
> >
> > According to the doc:
> > http://kylin.apache.org/docs15/tutorial/cube_streaming.html, the Kafka
> > incoming message should be in flatten structure. But it's too limit for
> > JSON message. May I request the feature which supports embedded structure
> > raw messages, and convert them into flatten table structure?
> >
> > One example:
> > {
> > "createdAt": "Jul 20, 2016 9:59:17 AM",
> > "id": 755703618762862600,
> > "text": "dejamos las tapas regionales de este #Miercoles
> > https://t.co/kfe0kT2Fup",
> > "source": "<a href=\"http://twitter.com\" rel=\"nofollow\">Twitter Web
> > Client</a>",
> > "isTruncated": false,
> > "inReplyToStatusId": -1,
> > "inReplyToUserId": -1,
> > "isFavorited": false,
> > "isRetweeted": false,
> > "favoriteCount": 0,
> > "retweetCount": 0,
> > "isPossiblySensitive": false,
> > "lang": "es",
> > "contributorsIDs": [],
> > "userMentionEntities": [],
> > "urlEntities": [],
> > "hashtagEntities": [
> > {
> > "text": "Miercoles",
> > "start": 37,
> > "end": 47
> > }
> > ],
> > "symbolEntities": [],
> > "currentUserRetweetId": -1,
> > "user": {
> > "id": 4853763947,
> > "name": "El Metropolitano",
> > "screenName": "ElTWdelMetro",
> > "description": "Noticias, an��lisis e informaci��n para el
> crecimiento
> > de la regi��n.",
> > "descriptionURLEntities": [],
> > "isContributorsEnabled": false,
> > "profileImageUrl": "
> >
> http://pbs.twimg.com/profile_images/692361953377927169/cS7lUaSj_normal.png
> > ",
> > "profileImageUrlHttps": "
> >
> https://pbs.twimg.com/profile_images/692361953377927169/cS7lUaSj_normal.png
> > ",
> > "isDefaultProfileImage": false,
> > "url": "http://elmetropolitano.com.ar/",
> > "isProtected": false,
> > "followersCount": 153,
> > "profileBackgroundColor": "F5F8FA",
> > "profileTextColor": "333333",
> > "profileLinkColor": "2B7BB9",
> > "profileSidebarFillColor": "DDEEF6",
> > "profileSidebarBorderColor": "C0DEED",
> > "profileUseBackgroundImage": true,
> > "isDefaultProfile": true,
> > "showAllInlineMedia": false,
> > "friendsCount": 169,
> > "createdAt": "Jan 27, 2016 3:00:21 PM",
> > "favouritesCount": 9,
> > "utcOffset": -25200,
> > "timeZone": "Pacific Time (US & Canada)",
> > "profileBackgroundImageUrl": "",
> > "profileBackgroundImageUrlHttps": "",
> > "profileBannerImageUrl": "
> > https://pbs.twimg.com/profile_banners/4853763947/1453907062",
> > "profileBackgroundTiled": false,
> > "lang": "es",
> > "statusesCount": 1000,
> > "isGeoEnabled": false,
> > "isVerified": false,
> > "translator": false,
> > "listedCount": 0,
> > "isFollowRequestSent": false
> > }
> > }
> >
> > The flatten structure will like this:
> >
> > "createdAt": "Jul 20, 2016 9:59:17 AM",
> > ...
> > "user_id": 4853763947
> >
> > --
> > With Warm regards
> >
> > Yiming Liu (刘一鸣)
> >
>
--
With Warm regards
Yiming Liu (刘一鸣)
Re: Feature Request: Support Embedded Structure when Parsing
Streaming Message
Posted by Li Yang <li...@apache.org>.
Sure, pls open a JIRA.
On Wed, Jul 20, 2016 at 6:16 PM, Yiming Liu <li...@gmail.com> wrote:
> Hi Kylin devs,
>
> According to the doc:
> http://kylin.apache.org/docs15/tutorial/cube_streaming.html, the Kafka
> incoming message should be in flatten structure. But it's too limit for
> JSON message. May I request the feature which supports embedded structure
> raw messages, and convert them into flatten table structure?
>
> One example:
> {
> "createdAt": "Jul 20, 2016 9:59:17 AM",
> "id": 755703618762862600,
> "text": "dejamos las tapas regionales de este #Miercoles
> https://t.co/kfe0kT2Fup",
> "source": "<a href=\"http://twitter.com\" rel=\"nofollow\">Twitter Web
> Client</a>",
> "isTruncated": false,
> "inReplyToStatusId": -1,
> "inReplyToUserId": -1,
> "isFavorited": false,
> "isRetweeted": false,
> "favoriteCount": 0,
> "retweetCount": 0,
> "isPossiblySensitive": false,
> "lang": "es",
> "contributorsIDs": [],
> "userMentionEntities": [],
> "urlEntities": [],
> "hashtagEntities": [
> {
> "text": "Miercoles",
> "start": 37,
> "end": 47
> }
> ],
> "symbolEntities": [],
> "currentUserRetweetId": -1,
> "user": {
> "id": 4853763947,
> "name": "El Metropolitano",
> "screenName": "ElTWdelMetro",
> "description": "Noticias, an��lisis e informaci��n para el crecimiento
> de la regi��n.",
> "descriptionURLEntities": [],
> "isContributorsEnabled": false,
> "profileImageUrl": "
> http://pbs.twimg.com/profile_images/692361953377927169/cS7lUaSj_normal.png
> ",
> "profileImageUrlHttps": "
> https://pbs.twimg.com/profile_images/692361953377927169/cS7lUaSj_normal.png
> ",
> "isDefaultProfileImage": false,
> "url": "http://elmetropolitano.com.ar/",
> "isProtected": false,
> "followersCount": 153,
> "profileBackgroundColor": "F5F8FA",
> "profileTextColor": "333333",
> "profileLinkColor": "2B7BB9",
> "profileSidebarFillColor": "DDEEF6",
> "profileSidebarBorderColor": "C0DEED",
> "profileUseBackgroundImage": true,
> "isDefaultProfile": true,
> "showAllInlineMedia": false,
> "friendsCount": 169,
> "createdAt": "Jan 27, 2016 3:00:21 PM",
> "favouritesCount": 9,
> "utcOffset": -25200,
> "timeZone": "Pacific Time (US & Canada)",
> "profileBackgroundImageUrl": "",
> "profileBackgroundImageUrlHttps": "",
> "profileBannerImageUrl": "
> https://pbs.twimg.com/profile_banners/4853763947/1453907062",
> "profileBackgroundTiled": false,
> "lang": "es",
> "statusesCount": 1000,
> "isGeoEnabled": false,
> "isVerified": false,
> "translator": false,
> "listedCount": 0,
> "isFollowRequestSent": false
> }
> }
>
> The flatten structure will like this:
>
> "createdAt": "Jul 20, 2016 9:59:17 AM",
> ...
> "user_id": 4853763947
>
> --
> With Warm regards
>
> Yiming Liu (刘一鸣)
>