You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@asterixdb.apache.org by Eugenia Gabrielova <ge...@gmail.com> on 2015/06/11 23:41:17 UTC
Unable to connect secondary feed to primary feed
*Summary*: I'm able to create and connect a feed from Twitter, but I am
unable to connect a secondary feed. The web UI hangs, and the feeds UI does
not detect the secondary feed. Nothing amiss in the cc.log or [node].log.
*To Replicate:*
*Step 0*
Check out this change on Asterix:
https://asterix-gerrit.ics.uci.edu/#/c/248/
Check out this change on Hyracks:
https://asterix-gerrit.ics.uci.edu/#/c/249/
Install as you normally would for feeds, including installing testlib with
managix. I can document my command line steps on request :).
Feeds console here: http://localhost:19003/
*Step 1 - Twitter Feed in Web UI - Success!*
drop dataverse feeds if exists;
create dataverse feeds;
use dataverse feeds;
create type TwitterUser if not exists
as open{
screen_name: string,
language: string,
friends_count: int32,
status_count: int32,
name: string,
followers_count: string
};
create type Tweet if not exists as open{
id: string,
user:TwitterUser,
latitude:double,
longitude:double,
created_at:string,
message_text:string
};
create dataset Tweets(Tweet) if not exists
primary key id;
create feed TwitterFeed if not exists
using "push_twitter"
(("type-name"="Tweet"),("location"="US"));
connect feed TwitterFeed to dataset Tweets;
*Step 2 - Create (but don't connect Processed Tweets) in Web UI - success!*
use dataverse feeds;
create type ProcessedTweet if not exists as open {
id: string,
user_name:string,
location:point,
created_at:string,
message_text:string,
country: string,
topics: [string]
};
create dataset ProcessedTweets(ProcessedTweet) if not exists
primary key id;
create index location_index if not exists on ProcessedTweets(location) type
rtree;
create feed ProcessedTwitterFeed if not exists
from TwitterFeed
apply function testlib#processRawTweet;
*Step 3 - Connect ProcessedTwitterFeed :( - ???*
use dataverse feeds;
connect feed ProcessedTwitterFeed to dataset ProcessedTweets;
*Observation: *Secondary feed does not appear at localhost:19003, and web
UI hangs (no return success).
Thanks for any help and insight. Please let me know if I should provide
further detail.
Re: Unable to connect secondary feed to primary feed
Posted by Eugenia Gabrielova <ge...@gmail.com>.
It was generated at
asterix-external-data/target/testlib-zip-binary-assembly.zip.
I installed it like so:
$ managix install -d feeds -l testlib -p
~/workspace/asterixdb/asterix-external-data/target/testlib-zip-binary-assembly.zip
-n feeds_asterix
No changes made after checking out the gerrit links above, just a basic mvn
install.
On Thu, Jun 11, 2015 at 3:04 PM, Raman Grover <ra...@gmail.com>
wrote:
> Could you share the library -testlib that was used ?
>
> On Thu, Jun 11, 2015 at 2:41 PM, Eugenia Gabrielova <
> genia.likes.science@gmail.com> wrote:
>
> > *Summary*: I'm able to create and connect a feed from Twitter, but I am
> > unable to connect a secondary feed. The web UI hangs, and the feeds UI
> does
> > not detect the secondary feed. Nothing amiss in the cc.log or [node].log.
> >
> > *To Replicate:*
> > *Step 0*
> > Check out this change on Asterix:
> > https://asterix-gerrit.ics.uci.edu/#/c/248/
> > Check out this change on Hyracks:
> > https://asterix-gerrit.ics.uci.edu/#/c/249/
> >
> > Install as you normally would for feeds, including installing testlib
> with
> > managix. I can document my command line steps on request :).
> >
> > Feeds console here: http://localhost:19003/
> >
> > *Step 1 - Twitter Feed in Web UI - Success!*
> > drop dataverse feeds if exists;
> > create dataverse feeds;
> > use dataverse feeds;
> >
> > create type TwitterUser if not exists
> > as open{
> > screen_name: string,
> > language: string,
> > friends_count: int32,
> > status_count: int32,
> > name: string,
> > followers_count: string
> > };
> >
> > create type Tweet if not exists as open{
> > id: string,
> > user:TwitterUser,
> > latitude:double,
> > longitude:double,
> > created_at:string,
> > message_text:string
> > };
> >
> > create dataset Tweets(Tweet) if not exists
> > primary key id;
> >
> > create feed TwitterFeed if not exists
> > using "push_twitter"
> > (("type-name"="Tweet"),("location"="US"));
> >
> > connect feed TwitterFeed to dataset Tweets;
> >
> > *Step 2 - Create (but don't connect Processed Tweets) in Web UI -
> success!*
> > use dataverse feeds;
> >
> > create type ProcessedTweet if not exists as open {
> > id: string,
> > user_name:string,
> > location:point,
> > created_at:string,
> > message_text:string,
> > country: string,
> > topics: [string]
> > };
> >
> > create dataset ProcessedTweets(ProcessedTweet) if not exists
> > primary key id;
> >
> > create index location_index if not exists on ProcessedTweets(location)
> type
> > rtree;
> >
> > create feed ProcessedTwitterFeed if not exists
> > from TwitterFeed
> > apply function testlib#processRawTweet;
> >
> > *Step 3 - Connect ProcessedTwitterFeed :( - ???*
> > use dataverse feeds;
> >
> > connect feed ProcessedTwitterFeed to dataset ProcessedTweets;
> >
> > *Observation: *Secondary feed does not appear at localhost:19003, and web
> > UI hangs (no return success).
> >
> > Thanks for any help and insight. Please let me know if I should provide
> > further detail.
> >
>
>
>
> --
> Raman
>
Re: Unable to connect secondary feed to primary feed
Posted by Ian Maxon <im...@uci.edu>.
It's just the one that is generated in asterix-external-data.
On Thu, Jun 11, 2015 at 3:04 PM, Raman Grover <ra...@gmail.com>
wrote:
> Could you share the library -testlib that was used ?
>
> On Thu, Jun 11, 2015 at 2:41 PM, Eugenia Gabrielova <
> genia.likes.science@gmail.com> wrote:
>
> > *Summary*: I'm able to create and connect a feed from Twitter, but I am
> > unable to connect a secondary feed. The web UI hangs, and the feeds UI
> does
> > not detect the secondary feed. Nothing amiss in the cc.log or [node].log.
> >
> > *To Replicate:*
> > *Step 0*
> > Check out this change on Asterix:
> > https://asterix-gerrit.ics.uci.edu/#/c/248/
> > Check out this change on Hyracks:
> > https://asterix-gerrit.ics.uci.edu/#/c/249/
> >
> > Install as you normally would for feeds, including installing testlib
> with
> > managix. I can document my command line steps on request :).
> >
> > Feeds console here: http://localhost:19003/
> >
> > *Step 1 - Twitter Feed in Web UI - Success!*
> > drop dataverse feeds if exists;
> > create dataverse feeds;
> > use dataverse feeds;
> >
> > create type TwitterUser if not exists
> > as open{
> > screen_name: string,
> > language: string,
> > friends_count: int32,
> > status_count: int32,
> > name: string,
> > followers_count: string
> > };
> >
> > create type Tweet if not exists as open{
> > id: string,
> > user:TwitterUser,
> > latitude:double,
> > longitude:double,
> > created_at:string,
> > message_text:string
> > };
> >
> > create dataset Tweets(Tweet) if not exists
> > primary key id;
> >
> > create feed TwitterFeed if not exists
> > using "push_twitter"
> > (("type-name"="Tweet"),("location"="US"));
> >
> > connect feed TwitterFeed to dataset Tweets;
> >
> > *Step 2 - Create (but don't connect Processed Tweets) in Web UI -
> success!*
> > use dataverse feeds;
> >
> > create type ProcessedTweet if not exists as open {
> > id: string,
> > user_name:string,
> > location:point,
> > created_at:string,
> > message_text:string,
> > country: string,
> > topics: [string]
> > };
> >
> > create dataset ProcessedTweets(ProcessedTweet) if not exists
> > primary key id;
> >
> > create index location_index if not exists on ProcessedTweets(location)
> type
> > rtree;
> >
> > create feed ProcessedTwitterFeed if not exists
> > from TwitterFeed
> > apply function testlib#processRawTweet;
> >
> > *Step 3 - Connect ProcessedTwitterFeed :( - ???*
> > use dataverse feeds;
> >
> > connect feed ProcessedTwitterFeed to dataset ProcessedTweets;
> >
> > *Observation: *Secondary feed does not appear at localhost:19003, and web
> > UI hangs (no return success).
> >
> > Thanks for any help and insight. Please let me know if I should provide
> > further detail.
> >
>
>
>
> --
> Raman
>
Re: Unable to connect secondary feed to primary feed
Posted by Raman Grover <ra...@gmail.com>.
Could you share the library -testlib that was used ?
On Thu, Jun 11, 2015 at 2:41 PM, Eugenia Gabrielova <
genia.likes.science@gmail.com> wrote:
> *Summary*: I'm able to create and connect a feed from Twitter, but I am
> unable to connect a secondary feed. The web UI hangs, and the feeds UI does
> not detect the secondary feed. Nothing amiss in the cc.log or [node].log.
>
> *To Replicate:*
> *Step 0*
> Check out this change on Asterix:
> https://asterix-gerrit.ics.uci.edu/#/c/248/
> Check out this change on Hyracks:
> https://asterix-gerrit.ics.uci.edu/#/c/249/
>
> Install as you normally would for feeds, including installing testlib with
> managix. I can document my command line steps on request :).
>
> Feeds console here: http://localhost:19003/
>
> *Step 1 - Twitter Feed in Web UI - Success!*
> drop dataverse feeds if exists;
> create dataverse feeds;
> use dataverse feeds;
>
> create type TwitterUser if not exists
> as open{
> screen_name: string,
> language: string,
> friends_count: int32,
> status_count: int32,
> name: string,
> followers_count: string
> };
>
> create type Tweet if not exists as open{
> id: string,
> user:TwitterUser,
> latitude:double,
> longitude:double,
> created_at:string,
> message_text:string
> };
>
> create dataset Tweets(Tweet) if not exists
> primary key id;
>
> create feed TwitterFeed if not exists
> using "push_twitter"
> (("type-name"="Tweet"),("location"="US"));
>
> connect feed TwitterFeed to dataset Tweets;
>
> *Step 2 - Create (but don't connect Processed Tweets) in Web UI - success!*
> use dataverse feeds;
>
> create type ProcessedTweet if not exists as open {
> id: string,
> user_name:string,
> location:point,
> created_at:string,
> message_text:string,
> country: string,
> topics: [string]
> };
>
> create dataset ProcessedTweets(ProcessedTweet) if not exists
> primary key id;
>
> create index location_index if not exists on ProcessedTweets(location) type
> rtree;
>
> create feed ProcessedTwitterFeed if not exists
> from TwitterFeed
> apply function testlib#processRawTweet;
>
> *Step 3 - Connect ProcessedTwitterFeed :( - ???*
> use dataverse feeds;
>
> connect feed ProcessedTwitterFeed to dataset ProcessedTweets;
>
> *Observation: *Secondary feed does not appear at localhost:19003, and web
> UI hangs (no return success).
>
> Thanks for any help and insight. Please let me know if I should provide
> further detail.
>
--
Raman
Re: Unable to connect secondary feed to primary feed
Posted by Raman Grover <ra...@gmail.com>.
Q: When we delete the dataverse, how do we actually clean up the library?
Is there a weird state between dataverse deleted but library still
installed via managix?
When deleting a dataverse, we search for the contained components that
includes datasets, types, indexes and *libraries*. For each library, we
further drill down and enumerate the contained components which may include
UDFs and adaptors. We invoke drop statements on each library component
following by a drop library statement.
Note that managix does not maintain any state information about the
installed libraries for an AsterixDB instance. This information is
maintained only as part of AsterixDB metadata.
Regards,
Raman
On Thu, Jun 11, 2015 at 3:52 PM, Ian Maxon <im...@uci.edu> wrote:
> The issue is that the 'connect feed' statement for the secondary feed
> silently fails, because the library used to process that feed isn't found.
> The primary is fine, it doesn't need the library.
>
> Q: When we delete the dataverse, how do we actually clean up the library?
> Is there a weird state between dataverse deleted but library still
> installed via managix?
>
> On Thu, Jun 11, 2015 at 3:43 PM, Eugenia Gabrielova <
> genia.likes.science@gmail.com> wrote:
>
> > Thank you for the correction about the query - I had no idea it would
> have
> > that effect. This worked!!!
> >
> > Would it be accurate to say that the UI should not have reported the
> > primary feed successfully created, and the issue with the Secondary Feed
> > was because the Primary Feed was not properly created?
> >
> > On Thu, Jun 11, 2015 at 3:31 PM, Eugenia Gabrielova <
> > genia.likes.science@gmail.com> wrote:
> >
> > > Thank you for the explanation, I'll try this now!
> > >
> > > On Thu, Jun 11, 2015 at 3:29 PM, Raman Grover <ramangrover29@gmail.com
> >
> > > wrote:
> > >
> > >> As per the steps mentioned for reproducing the issue,
> > >> step 0 involves installing the external library testlib in a dataverse
> > >> (assumed to be "feeds" dataverse). On system startup (following
> managix
> > >> install command), the library components (UDFs) get installed in the
> > stated
> > >> dataverse.
> > >>
> > >> However, subsequently as part of step 1, the dataverse is dropped
> (drop
> > >> dataverse feeds if exists;) which results in un-installing the
> contents
> > of
> > >> the dataverse including the library and thus the installed UDFs
> > thereof. At
> > >> runtime (connect feed statement), the required library is not found.
> > There
> > >> is an error-propagation issue here as this can be caught earlier and
> > thrown
> > >> at the UI.
> > >>
> > >> Before going further, it would be great if you can confirm the above
> > >> steps and that the dataverse was actually dropped post installation of
> > >> library.
> > >> Further, I tried reproducing with the above statements (but removing
> the
> > >> drop and create dataverse statements from step 2) and got the feeds
> > >> connected and running fine.
> > >> Attached is a screen shot of the recent run showing the primary and
> > >> secondary feeds in connected state together with the data rates for
> > each.
> > >>
> > >> Regards,
> > >> Raman
> > >>
> > >> On Thu, Jun 11, 2015 at 2:41 PM, Eugenia Gabrielova <
> > >> genia.likes.science@gmail.com> wrote:
> > >>
> > >>> *Summary*: I'm able to create and connect a feed from Twitter, but I
> am
> > >>> unable to connect a secondary feed. The web UI hangs, and the feeds
> UI
> > >>> does
> > >>> not detect the secondary feed. Nothing amiss in the cc.log or
> > [node].log.
> > >>>
> > >>> *To Replicate:*
> > >>> *Step 0*
> > >>> Check out this change on Asterix:
> > >>> https://asterix-gerrit.ics.uci.edu/#/c/248/
> > >>> Check out this change on Hyracks:
> > >>> https://asterix-gerrit.ics.uci.edu/#/c/249/
> > >>>
> > >>> Install as you normally would for feeds, including installing testlib
> > >>> with
> > >>> managix. I can document my command line steps on request :).
> > >>>
> > >>> Feeds console here: http://localhost:19003/
> > >>>
> > >>> *Step 1 - Twitter Feed in Web UI - Success!*
> > >>> drop dataverse feeds if exists;
> > >>> create dataverse feeds;
> > >>> use dataverse feeds;
> > >>>
> > >>> create type TwitterUser if not exists
> > >>> as open{
> > >>> screen_name: string,
> > >>> language: string,
> > >>> friends_count: int32,
> > >>> status_count: int32,
> > >>> name: string,
> > >>> followers_count: string
> > >>> };
> > >>>
> > >>> create type Tweet if not exists as open{
> > >>> id: string,
> > >>> user:TwitterUser,
> > >>> latitude:double,
> > >>> longitude:double,
> > >>> created_at:string,
> > >>> message_text:string
> > >>> };
> > >>>
> > >>> create dataset Tweets(Tweet) if not exists
> > >>> primary key id;
> > >>>
> > >>> create feed TwitterFeed if not exists
> > >>> using "push_twitter"
> > >>> (("type-name"="Tweet"),("location"="US"));
> > >>>
> > >>> connect feed TwitterFeed to dataset Tweets;
> > >>>
> > >>> *Step 2 - Create (but don't connect Processed Tweets) in Web UI -
> > >>> success!*
> > >>> use dataverse feeds;
> > >>>
> > >>> create type ProcessedTweet if not exists as open {
> > >>> id: string,
> > >>> user_name:string,
> > >>> location:point,
> > >>> created_at:string,
> > >>> message_text:string,
> > >>> country: string,
> > >>> topics: [string]
> > >>> };
> > >>>
> > >>> create dataset ProcessedTweets(ProcessedTweet) if not exists
> > >>> primary key id;
> > >>>
> > >>> create index location_index if not exists on
> ProcessedTweets(location)
> > >>> type
> > >>> rtree;
> > >>>
> > >>> create feed ProcessedTwitterFeed if not exists
> > >>> from TwitterFeed
> > >>> apply function testlib#processRawTweet;
> > >>>
> > >>> *Step 3 - Connect ProcessedTwitterFeed :( - ???*
> > >>> use dataverse feeds;
> > >>>
> > >>> connect feed ProcessedTwitterFeed to dataset ProcessedTweets;
> > >>>
> > >>> *Observation: *Secondary feed does not appear at localhost:19003, and
> > web
> > >>> UI hangs (no return success).
> > >>>
> > >>> Thanks for any help and insight. Please let me know if I should
> provide
> > >>> further detail.
> > >>>
> > >>
> > >>
> > >>
> > >> --
> > >> Raman
> > >>
> > >
> > >
> >
>
--
Raman
Re: Unable to connect secondary feed to primary feed
Posted by Ian Maxon <im...@uci.edu>.
The issue is that the 'connect feed' statement for the secondary feed
silently fails, because the library used to process that feed isn't found.
The primary is fine, it doesn't need the library.
Q: When we delete the dataverse, how do we actually clean up the library?
Is there a weird state between dataverse deleted but library still
installed via managix?
On Thu, Jun 11, 2015 at 3:43 PM, Eugenia Gabrielova <
genia.likes.science@gmail.com> wrote:
> Thank you for the correction about the query - I had no idea it would have
> that effect. This worked!!!
>
> Would it be accurate to say that the UI should not have reported the
> primary feed successfully created, and the issue with the Secondary Feed
> was because the Primary Feed was not properly created?
>
> On Thu, Jun 11, 2015 at 3:31 PM, Eugenia Gabrielova <
> genia.likes.science@gmail.com> wrote:
>
> > Thank you for the explanation, I'll try this now!
> >
> > On Thu, Jun 11, 2015 at 3:29 PM, Raman Grover <ra...@gmail.com>
> > wrote:
> >
> >> As per the steps mentioned for reproducing the issue,
> >> step 0 involves installing the external library testlib in a dataverse
> >> (assumed to be "feeds" dataverse). On system startup (following managix
> >> install command), the library components (UDFs) get installed in the
> stated
> >> dataverse.
> >>
> >> However, subsequently as part of step 1, the dataverse is dropped (drop
> >> dataverse feeds if exists;) which results in un-installing the contents
> of
> >> the dataverse including the library and thus the installed UDFs
> thereof. At
> >> runtime (connect feed statement), the required library is not found.
> There
> >> is an error-propagation issue here as this can be caught earlier and
> thrown
> >> at the UI.
> >>
> >> Before going further, it would be great if you can confirm the above
> >> steps and that the dataverse was actually dropped post installation of
> >> library.
> >> Further, I tried reproducing with the above statements (but removing the
> >> drop and create dataverse statements from step 2) and got the feeds
> >> connected and running fine.
> >> Attached is a screen shot of the recent run showing the primary and
> >> secondary feeds in connected state together with the data rates for
> each.
> >>
> >> Regards,
> >> Raman
> >>
> >> On Thu, Jun 11, 2015 at 2:41 PM, Eugenia Gabrielova <
> >> genia.likes.science@gmail.com> wrote:
> >>
> >>> *Summary*: I'm able to create and connect a feed from Twitter, but I am
> >>> unable to connect a secondary feed. The web UI hangs, and the feeds UI
> >>> does
> >>> not detect the secondary feed. Nothing amiss in the cc.log or
> [node].log.
> >>>
> >>> *To Replicate:*
> >>> *Step 0*
> >>> Check out this change on Asterix:
> >>> https://asterix-gerrit.ics.uci.edu/#/c/248/
> >>> Check out this change on Hyracks:
> >>> https://asterix-gerrit.ics.uci.edu/#/c/249/
> >>>
> >>> Install as you normally would for feeds, including installing testlib
> >>> with
> >>> managix. I can document my command line steps on request :).
> >>>
> >>> Feeds console here: http://localhost:19003/
> >>>
> >>> *Step 1 - Twitter Feed in Web UI - Success!*
> >>> drop dataverse feeds if exists;
> >>> create dataverse feeds;
> >>> use dataverse feeds;
> >>>
> >>> create type TwitterUser if not exists
> >>> as open{
> >>> screen_name: string,
> >>> language: string,
> >>> friends_count: int32,
> >>> status_count: int32,
> >>> name: string,
> >>> followers_count: string
> >>> };
> >>>
> >>> create type Tweet if not exists as open{
> >>> id: string,
> >>> user:TwitterUser,
> >>> latitude:double,
> >>> longitude:double,
> >>> created_at:string,
> >>> message_text:string
> >>> };
> >>>
> >>> create dataset Tweets(Tweet) if not exists
> >>> primary key id;
> >>>
> >>> create feed TwitterFeed if not exists
> >>> using "push_twitter"
> >>> (("type-name"="Tweet"),("location"="US"));
> >>>
> >>> connect feed TwitterFeed to dataset Tweets;
> >>>
> >>> *Step 2 - Create (but don't connect Processed Tweets) in Web UI -
> >>> success!*
> >>> use dataverse feeds;
> >>>
> >>> create type ProcessedTweet if not exists as open {
> >>> id: string,
> >>> user_name:string,
> >>> location:point,
> >>> created_at:string,
> >>> message_text:string,
> >>> country: string,
> >>> topics: [string]
> >>> };
> >>>
> >>> create dataset ProcessedTweets(ProcessedTweet) if not exists
> >>> primary key id;
> >>>
> >>> create index location_index if not exists on ProcessedTweets(location)
> >>> type
> >>> rtree;
> >>>
> >>> create feed ProcessedTwitterFeed if not exists
> >>> from TwitterFeed
> >>> apply function testlib#processRawTweet;
> >>>
> >>> *Step 3 - Connect ProcessedTwitterFeed :( - ???*
> >>> use dataverse feeds;
> >>>
> >>> connect feed ProcessedTwitterFeed to dataset ProcessedTweets;
> >>>
> >>> *Observation: *Secondary feed does not appear at localhost:19003, and
> web
> >>> UI hangs (no return success).
> >>>
> >>> Thanks for any help and insight. Please let me know if I should provide
> >>> further detail.
> >>>
> >>
> >>
> >>
> >> --
> >> Raman
> >>
> >
> >
>
Re: Unable to connect secondary feed to primary feed
Posted by Raman Grover <ra...@gmail.com>.
The primary feed was successfully created and connected because it did not
have any dependency on an external library (that got uninstalled
inadvertently). So UI reported correctly stating that the primary feed was
connected.
The issue with secondary feed was *unrelated* to the primary feed. It was
simply due to the missing components (UDF).
On Thu, Jun 11, 2015 at 3:43 PM, Eugenia Gabrielova <
genia.likes.science@gmail.com> wrote:
> Thank you for the correction about the query - I had no idea it would have
> that effect. This worked!!!
>
> Would it be accurate to say that the UI should not have reported the
> primary feed successfully created, and the issue with the Secondary Feed
> was because the Primary Feed was not properly created?
>
> On Thu, Jun 11, 2015 at 3:31 PM, Eugenia Gabrielova <
> genia.likes.science@gmail.com> wrote:
>
> > Thank you for the explanation, I'll try this now!
> >
> > On Thu, Jun 11, 2015 at 3:29 PM, Raman Grover <ra...@gmail.com>
> > wrote:
> >
> >> As per the steps mentioned for reproducing the issue,
> >> step 0 involves installing the external library testlib in a dataverse
> >> (assumed to be "feeds" dataverse). On system startup (following managix
> >> install command), the library components (UDFs) get installed in the
> stated
> >> dataverse.
> >>
> >> However, subsequently as part of step 1, the dataverse is dropped (drop
> >> dataverse feeds if exists;) which results in un-installing the contents
> of
> >> the dataverse including the library and thus the installed UDFs
> thereof. At
> >> runtime (connect feed statement), the required library is not found.
> There
> >> is an error-propagation issue here as this can be caught earlier and
> thrown
> >> at the UI.
> >>
> >> Before going further, it would be great if you can confirm the above
> >> steps and that the dataverse was actually dropped post installation of
> >> library.
> >> Further, I tried reproducing with the above statements (but removing the
> >> drop and create dataverse statements from step 2) and got the feeds
> >> connected and running fine.
> >> Attached is a screen shot of the recent run showing the primary and
> >> secondary feeds in connected state together with the data rates for
> each.
> >>
> >> Regards,
> >> Raman
> >>
> >> On Thu, Jun 11, 2015 at 2:41 PM, Eugenia Gabrielova <
> >> genia.likes.science@gmail.com> wrote:
> >>
> >>> *Summary*: I'm able to create and connect a feed from Twitter, but I am
> >>> unable to connect a secondary feed. The web UI hangs, and the feeds UI
> >>> does
> >>> not detect the secondary feed. Nothing amiss in the cc.log or
> [node].log.
> >>>
> >>> *To Replicate:*
> >>> *Step 0*
> >>> Check out this change on Asterix:
> >>> https://asterix-gerrit.ics.uci.edu/#/c/248/
> >>> Check out this change on Hyracks:
> >>> https://asterix-gerrit.ics.uci.edu/#/c/249/
> >>>
> >>> Install as you normally would for feeds, including installing testlib
> >>> with
> >>> managix. I can document my command line steps on request :).
> >>>
> >>> Feeds console here: http://localhost:19003/
> >>>
> >>> *Step 1 - Twitter Feed in Web UI - Success!*
> >>> drop dataverse feeds if exists;
> >>> create dataverse feeds;
> >>> use dataverse feeds;
> >>>
> >>> create type TwitterUser if not exists
> >>> as open{
> >>> screen_name: string,
> >>> language: string,
> >>> friends_count: int32,
> >>> status_count: int32,
> >>> name: string,
> >>> followers_count: string
> >>> };
> >>>
> >>> create type Tweet if not exists as open{
> >>> id: string,
> >>> user:TwitterUser,
> >>> latitude:double,
> >>> longitude:double,
> >>> created_at:string,
> >>> message_text:string
> >>> };
> >>>
> >>> create dataset Tweets(Tweet) if not exists
> >>> primary key id;
> >>>
> >>> create feed TwitterFeed if not exists
> >>> using "push_twitter"
> >>> (("type-name"="Tweet"),("location"="US"));
> >>>
> >>> connect feed TwitterFeed to dataset Tweets;
> >>>
> >>> *Step 2 - Create (but don't connect Processed Tweets) in Web UI -
> >>> success!*
> >>> use dataverse feeds;
> >>>
> >>> create type ProcessedTweet if not exists as open {
> >>> id: string,
> >>> user_name:string,
> >>> location:point,
> >>> created_at:string,
> >>> message_text:string,
> >>> country: string,
> >>> topics: [string]
> >>> };
> >>>
> >>> create dataset ProcessedTweets(ProcessedTweet) if not exists
> >>> primary key id;
> >>>
> >>> create index location_index if not exists on ProcessedTweets(location)
> >>> type
> >>> rtree;
> >>>
> >>> create feed ProcessedTwitterFeed if not exists
> >>> from TwitterFeed
> >>> apply function testlib#processRawTweet;
> >>>
> >>> *Step 3 - Connect ProcessedTwitterFeed :( - ???*
> >>> use dataverse feeds;
> >>>
> >>> connect feed ProcessedTwitterFeed to dataset ProcessedTweets;
> >>>
> >>> *Observation: *Secondary feed does not appear at localhost:19003, and
> web
> >>> UI hangs (no return success).
> >>>
> >>> Thanks for any help and insight. Please let me know if I should provide
> >>> further detail.
> >>>
> >>
> >>
> >>
> >> --
> >> Raman
> >>
> >
> >
>
--
Raman
Re: Unable to connect secondary feed to primary feed
Posted by Eugenia Gabrielova <ge...@gmail.com>.
Thank you for the correction about the query - I had no idea it would have
that effect. This worked!!!
Would it be accurate to say that the UI should not have reported the
primary feed successfully created, and the issue with the Secondary Feed
was because the Primary Feed was not properly created?
On Thu, Jun 11, 2015 at 3:31 PM, Eugenia Gabrielova <
genia.likes.science@gmail.com> wrote:
> Thank you for the explanation, I'll try this now!
>
> On Thu, Jun 11, 2015 at 3:29 PM, Raman Grover <ra...@gmail.com>
> wrote:
>
>> As per the steps mentioned for reproducing the issue,
>> step 0 involves installing the external library testlib in a dataverse
>> (assumed to be "feeds" dataverse). On system startup (following managix
>> install command), the library components (UDFs) get installed in the stated
>> dataverse.
>>
>> However, subsequently as part of step 1, the dataverse is dropped (drop
>> dataverse feeds if exists;) which results in un-installing the contents of
>> the dataverse including the library and thus the installed UDFs thereof. At
>> runtime (connect feed statement), the required library is not found. There
>> is an error-propagation issue here as this can be caught earlier and thrown
>> at the UI.
>>
>> Before going further, it would be great if you can confirm the above
>> steps and that the dataverse was actually dropped post installation of
>> library.
>> Further, I tried reproducing with the above statements (but removing the
>> drop and create dataverse statements from step 2) and got the feeds
>> connected and running fine.
>> Attached is a screen shot of the recent run showing the primary and
>> secondary feeds in connected state together with the data rates for each.
>>
>> Regards,
>> Raman
>>
>> On Thu, Jun 11, 2015 at 2:41 PM, Eugenia Gabrielova <
>> genia.likes.science@gmail.com> wrote:
>>
>>> *Summary*: I'm able to create and connect a feed from Twitter, but I am
>>> unable to connect a secondary feed. The web UI hangs, and the feeds UI
>>> does
>>> not detect the secondary feed. Nothing amiss in the cc.log or [node].log.
>>>
>>> *To Replicate:*
>>> *Step 0*
>>> Check out this change on Asterix:
>>> https://asterix-gerrit.ics.uci.edu/#/c/248/
>>> Check out this change on Hyracks:
>>> https://asterix-gerrit.ics.uci.edu/#/c/249/
>>>
>>> Install as you normally would for feeds, including installing testlib
>>> with
>>> managix. I can document my command line steps on request :).
>>>
>>> Feeds console here: http://localhost:19003/
>>>
>>> *Step 1 - Twitter Feed in Web UI - Success!*
>>> drop dataverse feeds if exists;
>>> create dataverse feeds;
>>> use dataverse feeds;
>>>
>>> create type TwitterUser if not exists
>>> as open{
>>> screen_name: string,
>>> language: string,
>>> friends_count: int32,
>>> status_count: int32,
>>> name: string,
>>> followers_count: string
>>> };
>>>
>>> create type Tweet if not exists as open{
>>> id: string,
>>> user:TwitterUser,
>>> latitude:double,
>>> longitude:double,
>>> created_at:string,
>>> message_text:string
>>> };
>>>
>>> create dataset Tweets(Tweet) if not exists
>>> primary key id;
>>>
>>> create feed TwitterFeed if not exists
>>> using "push_twitter"
>>> (("type-name"="Tweet"),("location"="US"));
>>>
>>> connect feed TwitterFeed to dataset Tweets;
>>>
>>> *Step 2 - Create (but don't connect Processed Tweets) in Web UI -
>>> success!*
>>> use dataverse feeds;
>>>
>>> create type ProcessedTweet if not exists as open {
>>> id: string,
>>> user_name:string,
>>> location:point,
>>> created_at:string,
>>> message_text:string,
>>> country: string,
>>> topics: [string]
>>> };
>>>
>>> create dataset ProcessedTweets(ProcessedTweet) if not exists
>>> primary key id;
>>>
>>> create index location_index if not exists on ProcessedTweets(location)
>>> type
>>> rtree;
>>>
>>> create feed ProcessedTwitterFeed if not exists
>>> from TwitterFeed
>>> apply function testlib#processRawTweet;
>>>
>>> *Step 3 - Connect ProcessedTwitterFeed :( - ???*
>>> use dataverse feeds;
>>>
>>> connect feed ProcessedTwitterFeed to dataset ProcessedTweets;
>>>
>>> *Observation: *Secondary feed does not appear at localhost:19003, and web
>>> UI hangs (no return success).
>>>
>>> Thanks for any help and insight. Please let me know if I should provide
>>> further detail.
>>>
>>
>>
>>
>> --
>> Raman
>>
>
>
Re: Unable to connect secondary feed to primary feed
Posted by Eugenia Gabrielova <ge...@gmail.com>.
Thank you for the explanation, I'll try this now!
On Thu, Jun 11, 2015 at 3:29 PM, Raman Grover <ra...@gmail.com>
wrote:
> As per the steps mentioned for reproducing the issue,
> step 0 involves installing the external library testlib in a dataverse
> (assumed to be "feeds" dataverse). On system startup (following managix
> install command), the library components (UDFs) get installed in the stated
> dataverse.
>
> However, subsequently as part of step 1, the dataverse is dropped (drop
> dataverse feeds if exists;) which results in un-installing the contents of
> the dataverse including the library and thus the installed UDFs thereof. At
> runtime (connect feed statement), the required library is not found. There
> is an error-propagation issue here as this can be caught earlier and thrown
> at the UI.
>
> Before going further, it would be great if you can confirm the above steps
> and that the dataverse was actually dropped post installation of library.
> Further, I tried reproducing with the above statements (but removing the
> drop and create dataverse statements from step 2) and got the feeds
> connected and running fine.
> Attached is a screen shot of the recent run showing the primary and
> secondary feeds in connected state together with the data rates for each.
>
> Regards,
> Raman
>
> On Thu, Jun 11, 2015 at 2:41 PM, Eugenia Gabrielova <
> genia.likes.science@gmail.com> wrote:
>
>> *Summary*: I'm able to create and connect a feed from Twitter, but I am
>> unable to connect a secondary feed. The web UI hangs, and the feeds UI
>> does
>> not detect the secondary feed. Nothing amiss in the cc.log or [node].log.
>>
>> *To Replicate:*
>> *Step 0*
>> Check out this change on Asterix:
>> https://asterix-gerrit.ics.uci.edu/#/c/248/
>> Check out this change on Hyracks:
>> https://asterix-gerrit.ics.uci.edu/#/c/249/
>>
>> Install as you normally would for feeds, including installing testlib with
>> managix. I can document my command line steps on request :).
>>
>> Feeds console here: http://localhost:19003/
>>
>> *Step 1 - Twitter Feed in Web UI - Success!*
>> drop dataverse feeds if exists;
>> create dataverse feeds;
>> use dataverse feeds;
>>
>> create type TwitterUser if not exists
>> as open{
>> screen_name: string,
>> language: string,
>> friends_count: int32,
>> status_count: int32,
>> name: string,
>> followers_count: string
>> };
>>
>> create type Tweet if not exists as open{
>> id: string,
>> user:TwitterUser,
>> latitude:double,
>> longitude:double,
>> created_at:string,
>> message_text:string
>> };
>>
>> create dataset Tweets(Tweet) if not exists
>> primary key id;
>>
>> create feed TwitterFeed if not exists
>> using "push_twitter"
>> (("type-name"="Tweet"),("location"="US"));
>>
>> connect feed TwitterFeed to dataset Tweets;
>>
>> *Step 2 - Create (but don't connect Processed Tweets) in Web UI -
>> success!*
>> use dataverse feeds;
>>
>> create type ProcessedTweet if not exists as open {
>> id: string,
>> user_name:string,
>> location:point,
>> created_at:string,
>> message_text:string,
>> country: string,
>> topics: [string]
>> };
>>
>> create dataset ProcessedTweets(ProcessedTweet) if not exists
>> primary key id;
>>
>> create index location_index if not exists on ProcessedTweets(location)
>> type
>> rtree;
>>
>> create feed ProcessedTwitterFeed if not exists
>> from TwitterFeed
>> apply function testlib#processRawTweet;
>>
>> *Step 3 - Connect ProcessedTwitterFeed :( - ???*
>> use dataverse feeds;
>>
>> connect feed ProcessedTwitterFeed to dataset ProcessedTweets;
>>
>> *Observation: *Secondary feed does not appear at localhost:19003, and web
>> UI hangs (no return success).
>>
>> Thanks for any help and insight. Please let me know if I should provide
>> further detail.
>>
>
>
>
> --
> Raman
>
Re: Unable to connect secondary feed to primary feed
Posted by Raman Grover <ra...@gmail.com>.
As per the steps mentioned for reproducing the issue,
step 0 involves installing the external library testlib in a dataverse
(assumed to be "feeds" dataverse). On system startup (following managix
install command), the library components (UDFs) get installed in the stated
dataverse.
However, subsequently as part of step 1, the dataverse is dropped (drop
dataverse feeds if exists;) which results in un-installing the contents of
the dataverse including the library and thus the installed UDFs thereof. At
runtime (connect feed statement), the required library is not found. There
is an error-propagation issue here as this can be caught earlier and thrown
at the UI.
Before going further, it would be great if you can confirm the above steps
and that the dataverse was actually dropped post installation of library.
Further, I tried reproducing with the above statements (but removing the
drop and create dataverse statements from step 2) and got the feeds
connected and running fine.
Attached is a screen shot of the recent run showing the primary and
secondary feeds in connected state together with the data rates for each.
Regards,
Raman
On Thu, Jun 11, 2015 at 2:41 PM, Eugenia Gabrielova <
genia.likes.science@gmail.com> wrote:
> *Summary*: I'm able to create and connect a feed from Twitter, but I am
> unable to connect a secondary feed. The web UI hangs, and the feeds UI does
> not detect the secondary feed. Nothing amiss in the cc.log or [node].log.
>
> *To Replicate:*
> *Step 0*
> Check out this change on Asterix:
> https://asterix-gerrit.ics.uci.edu/#/c/248/
> Check out this change on Hyracks:
> https://asterix-gerrit.ics.uci.edu/#/c/249/
>
> Install as you normally would for feeds, including installing testlib with
> managix. I can document my command line steps on request :).
>
> Feeds console here: http://localhost:19003/
>
> *Step 1 - Twitter Feed in Web UI - Success!*
> drop dataverse feeds if exists;
> create dataverse feeds;
> use dataverse feeds;
>
> create type TwitterUser if not exists
> as open{
> screen_name: string,
> language: string,
> friends_count: int32,
> status_count: int32,
> name: string,
> followers_count: string
> };
>
> create type Tweet if not exists as open{
> id: string,
> user:TwitterUser,
> latitude:double,
> longitude:double,
> created_at:string,
> message_text:string
> };
>
> create dataset Tweets(Tweet) if not exists
> primary key id;
>
> create feed TwitterFeed if not exists
> using "push_twitter"
> (("type-name"="Tweet"),("location"="US"));
>
> connect feed TwitterFeed to dataset Tweets;
>
> *Step 2 - Create (but don't connect Processed Tweets) in Web UI - success!*
> use dataverse feeds;
>
> create type ProcessedTweet if not exists as open {
> id: string,
> user_name:string,
> location:point,
> created_at:string,
> message_text:string,
> country: string,
> topics: [string]
> };
>
> create dataset ProcessedTweets(ProcessedTweet) if not exists
> primary key id;
>
> create index location_index if not exists on ProcessedTweets(location) type
> rtree;
>
> create feed ProcessedTwitterFeed if not exists
> from TwitterFeed
> apply function testlib#processRawTweet;
>
> *Step 3 - Connect ProcessedTwitterFeed :( - ???*
> use dataverse feeds;
>
> connect feed ProcessedTwitterFeed to dataset ProcessedTweets;
>
> *Observation: *Secondary feed does not appear at localhost:19003, and web
> UI hangs (no return success).
>
> Thanks for any help and insight. Please let me know if I should provide
> further detail.
>
--
Raman