You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@asterixdb.apache.org by Ashok Raman <as...@gmail.com> on 2015/09/22 08:24:33 UTC
Twitter feed ingestion issue
Hi,
This is the sequence I followed with 0.8.7-Snapshot release:
-------------------------------------------------------------------------------
create dataverse feeds;
use dataverse feeds;
create type TwitterUser if not exists as open{
screen_name: string,
language: string,
friends_count: int32,
status_count: int32,
name: string,
followers_count: string
};
create type Tweet if not exists as open{
id: string,
user: TwitterUser,
latitude:double,
longitude:double,
created_at:string,
message_text:string,
topics: [string]
};
create dataset Tweets (Tweet)
primary key id;
create feed TwitterFeed if not exists using "push_twitter"
(("type-name"="Tweet"),
("consumer.key"=""),
("consumer.secret"=""),
("access.token"=""),
("access.token.secret"=""),
("topics"="allergy"));
connect feed TwitterFeed to dataset Tweets;
-------------------------------------------------------------------
While the cc.log has -
"INFO: Received message
{"nodeId":"my_asterix_node1","partition":0,"runtime-type":"STORE","dataverse":"feeds","dataset":"Tweets","value":-1,"feed":"TwitterFeed","value-type":"OUTFLOW_RATE","message-type":"FEED_REPORT"}
"
my_asterix_node1.log has
org.apache.asterix.metadata.feeds.AdapterExecutor run
SEVERE: Exception during feed ingestion java.lang.NullPointerException
java.lang.Exception: java.lang.NullPointerException
at
org.apache.asterix.external.dataset.adapter.ClientBasedFeedAdapter.st
art(ClientBasedFeedAdapter.java:126)
at
org.apache.asterix.metadata.feeds.AdapterExecutor.run(AdapterExecutor
.java:52)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
at
org.apache.asterix.external.dataset.adapter.ClientBasedFeedAdapter.st
art(ClientBasedFeedAdapter.java:118)
... 4 more
org.apache.asterix.common.exceptions.AsterixException:
java.lang.ArrayIndexOutOf
BoundsException: 6
at
org.apache.asterix.external.dataset.adapter.FeedClient.nextTuple(Feed
Client.java:106)
at
org.apache.asterix.external.dataset.adapter.ClientBasedFeedAdapter.st
art(ClientBasedFeedAdapter.java:92) at
org.apache.asterix.metadata.feeds.AdapterExecutor.run(AdapterExecutor
.java:52)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 6
at org.apache.asterix.om.base.ARecord.getValueByPos(ARecord.java:51)
at
org.apache.asterix.external.dataset.adapter.FeedClient.writeRecord(Fe
edClient.java:117)
at
org.apache.asterix.external.dataset.adapter.FeedClient.nextTuple(Feed
Client.java:87)
... 5 more
"
Thanks for any help,
Ashok
Re: Twitter feed ingestion issue
Posted by Ashok Raman <as...@gmail.com>.
Thanks, Ian, for your quick response and support, looking forward.
Best,
Ashok
On Wed, Sep 23, 2015 at 1:20 AM, Ian Maxon <im...@uci.edu> wrote:
> Hi Ashok,
> What you're seeing is definitely a bug, some of us have become aware
> of it but only relatively recently (earlier this month). Basically
> what appears to have happened is that this tutorial has existed for
> quite a long time, but there is a discrepancy between how the code
> changed in the last merge of feeds, and how the tutorial was updated.
> The source of the stack you see is because the output of the adapter
> is not matching the format of the type internally.
> We plan to fix this as soon as possible, so stay tuned...
>
> Thanks,
> -Ian
>
>
> On Mon, Sep 21, 2015 at 11:24 PM, Ashok Raman <as...@gmail.com>
> wrote:
> > Hi,
> >
> > This is the sequence I followed with 0.8.7-Snapshot release:
> >
> -------------------------------------------------------------------------------
> > create dataverse feeds;
> > use dataverse feeds;
> >
> > create type TwitterUser if not exists as open{
> > screen_name: string,
> > language: string,
> > friends_count: int32,
> > status_count: int32,
> > name: string,
> > followers_count: string
> > };
> > create type Tweet if not exists as open{
> > id: string,
> > user: TwitterUser,
> > latitude:double,
> > longitude:double,
> > created_at:string,
> > message_text:string,
> > topics: [string]
> > };
> >
> > create dataset Tweets (Tweet)
> > primary key id;
> > create feed TwitterFeed if not exists using "push_twitter"
> > (("type-name"="Tweet"),
> > ("consumer.key"=""),
> > ("consumer.secret"=""),
> > ("access.token"=""),
> > ("access.token.secret"=""),
> > ("topics"="allergy"));
> >
> > connect feed TwitterFeed to dataset Tweets;
> > -------------------------------------------------------------------
> > While the cc.log has -
> > "INFO: Received message
> >
> {"nodeId":"my_asterix_node1","partition":0,"runtime-type":"STORE","dataverse":"feeds","dataset":"Tweets","value":-1,"feed":"TwitterFeed","value-type":"OUTFLOW_RATE","message-type":"FEED_REPORT"}
> > "
> >
> > my_asterix_node1.log has
> >
> > org.apache.asterix.metadata.feeds.AdapterExecutor run
> > SEVERE: Exception during feed ingestion java.lang.NullPointerException
> > java.lang.Exception: java.lang.NullPointerException
> > at
> > org.apache.asterix.external.dataset.adapter.ClientBasedFeedAdapter.st
> > art(ClientBasedFeedAdapter.java:126)
> > at
> > org.apache.asterix.metadata.feeds.AdapterExecutor.run(AdapterExecutor
> > .java:52)
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
> > java:1145)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
> > .java:615)
> > at java.lang.Thread.run(Thread.java:745)
> > Caused by: java.lang.NullPointerException
> > at
> > org.apache.asterix.external.dataset.adapter.ClientBasedFeedAdapter.st
> > art(ClientBasedFeedAdapter.java:118)
> > ... 4 more
> > org.apache.asterix.common.exceptions.AsterixException:
> > java.lang.ArrayIndexOutOf
> > BoundsException: 6
> > at
> > org.apache.asterix.external.dataset.adapter.FeedClient.nextTuple(Feed
> > Client.java:106)
> > at
> > org.apache.asterix.external.dataset.adapter.ClientBasedFeedAdapter.st
> > art(ClientBasedFeedAdapter.java:92) at
> > org.apache.asterix.metadata.feeds.AdapterExecutor.run(AdapterExecutor
> > .java:52)
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
> > java:1145)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
> > .java:615)
> > at java.lang.Thread.run(Thread.java:745)
> > Caused by: java.lang.ArrayIndexOutOfBoundsException: 6
> > at
> org.apache.asterix.om.base.ARecord.getValueByPos(ARecord.java:51)
> > at
> > org.apache.asterix.external.dataset.adapter.FeedClient.writeRecord(Fe
> > edClient.java:117)
> > at
> > org.apache.asterix.external.dataset.adapter.FeedClient.nextTuple(Feed
> > Client.java:87)
> > ... 5 more
> > "
> >
> >
> >
> > Thanks for any help,
> > Ashok
> >
>
Re: Twitter feed ingestion issue
Posted by Ian Maxon <im...@uci.edu>.
Hi Ashok,
What you're seeing is definitely a bug, some of us have become aware
of it but only relatively recently (earlier this month). Basically
what appears to have happened is that this tutorial has existed for
quite a long time, but there is a discrepancy between how the code
changed in the last merge of feeds, and how the tutorial was updated.
The source of the stack you see is because the output of the adapter
is not matching the format of the type internally.
We plan to fix this as soon as possible, so stay tuned...
Thanks,
-Ian
On Mon, Sep 21, 2015 at 11:24 PM, Ashok Raman <as...@gmail.com> wrote:
> Hi,
>
> This is the sequence I followed with 0.8.7-Snapshot release:
> -------------------------------------------------------------------------------
> create dataverse feeds;
> use dataverse feeds;
>
> create type TwitterUser if not exists as open{
> screen_name: string,
> language: string,
> friends_count: int32,
> status_count: int32,
> name: string,
> followers_count: string
> };
> create type Tweet if not exists as open{
> id: string,
> user: TwitterUser,
> latitude:double,
> longitude:double,
> created_at:string,
> message_text:string,
> topics: [string]
> };
>
> create dataset Tweets (Tweet)
> primary key id;
> create feed TwitterFeed if not exists using "push_twitter"
> (("type-name"="Tweet"),
> ("consumer.key"=""),
> ("consumer.secret"=""),
> ("access.token"=""),
> ("access.token.secret"=""),
> ("topics"="allergy"));
>
> connect feed TwitterFeed to dataset Tweets;
> -------------------------------------------------------------------
> While the cc.log has -
> "INFO: Received message
> {"nodeId":"my_asterix_node1","partition":0,"runtime-type":"STORE","dataverse":"feeds","dataset":"Tweets","value":-1,"feed":"TwitterFeed","value-type":"OUTFLOW_RATE","message-type":"FEED_REPORT"}
> "
>
> my_asterix_node1.log has
>
> org.apache.asterix.metadata.feeds.AdapterExecutor run
> SEVERE: Exception during feed ingestion java.lang.NullPointerException
> java.lang.Exception: java.lang.NullPointerException
> at
> org.apache.asterix.external.dataset.adapter.ClientBasedFeedAdapter.st
> art(ClientBasedFeedAdapter.java:126)
> at
> org.apache.asterix.metadata.feeds.AdapterExecutor.run(AdapterExecutor
> .java:52)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
> java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
> .java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> at
> org.apache.asterix.external.dataset.adapter.ClientBasedFeedAdapter.st
> art(ClientBasedFeedAdapter.java:118)
> ... 4 more
> org.apache.asterix.common.exceptions.AsterixException:
> java.lang.ArrayIndexOutOf
> BoundsException: 6
> at
> org.apache.asterix.external.dataset.adapter.FeedClient.nextTuple(Feed
> Client.java:106)
> at
> org.apache.asterix.external.dataset.adapter.ClientBasedFeedAdapter.st
> art(ClientBasedFeedAdapter.java:92) at
> org.apache.asterix.metadata.feeds.AdapterExecutor.run(AdapterExecutor
> .java:52)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
> java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
> .java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 6
> at org.apache.asterix.om.base.ARecord.getValueByPos(ARecord.java:51)
> at
> org.apache.asterix.external.dataset.adapter.FeedClient.writeRecord(Fe
> edClient.java:117)
> at
> org.apache.asterix.external.dataset.adapter.FeedClient.nextTuple(Feed
> Client.java:87)
> ... 5 more
> "
>
>
>
> Thanks for any help,
> Ashok
>