Data Feeds in AsterixDB

[Bryan Glenn Marsh <br...@email.ucr.edu>]

Hi guys,

I know that there has already been a discussion/questions reagarding data
feeds and I have looked all messages pertaining as far back as May.

In one of the posts it suggests looking at
https://asterix-jenkins.ics.uci.edu/job/asterix-test-full/site/asterix-doc/feeds/tutorial.html
but
I did not find that helpful.

Let me step back and explain what I am trying to do. I want t create a
dataset and be able to insert records into this dataset through the use of
feeds.

I have tried implementing feeds with Asterix in the following two ways and
both have been unsuccessful.

The first method I tried was creating a socket adapter and try to send
packets to the feed and be able to insert records into the dataset. I based
this off of examples from here
https://github.com/apache/incubator-asterixdb/tree/master/asterix-app/src/test/resources/runtimets/queries/feeds/feeds_06


Here are the statements that I used with in the WEBUI

DDL:

drop dataverse feeds if exists;
create dataverse feeds;
use dataverse feeds;

create type Log_Type as closed {
LogId :string,
updateString:string
}

create dataset AstrixDB_Transaction_Log(Log_Type)
primary key LogId;
create index updateLogIndex on AstrixDB_Transaction_Log(LogId);

create feed socket_feed
using socket_adapter
(("sockets"="127.0.0.1:9009
"),("addressType"="IP"),("type-name"="Log_Type"),("format"="adm"));

UPDATE 1(connect feed):
use dataverse feeds;

set wait-for-completion-feed "true";

connect feed socket_feed to dataset AstrixDB_Transaction_Log;

NOTE: when I execute this command I do not get any response back saying it
was successful.

UPDATE 2 (disconnect feed):

use dataverse feeds;
disconnect feed socket_feed from dataset AstrixDB_Transaction_Log;


Query:

use dataverse feeds;

for $x in dataset AstrixDB_Transaction_Log
return $x

When I run the DDL and UPDATE I get know error, but when I try and send a
packet to 127.0.0.1:9009 the packet gets stuck, I set a timeout value to 4
seconds, but then it times out. Below I have attached some python code that
sends one adm records from the file load.txt.
I tried attaching the files, but google would not let me so please get the
files from this dropbox link
https://www.dropbox.com/s/mw2j6xsqed9a0cg/AsterixDB_FeedTest.tgz?dl=0

I thought maybe the port wasn't open, so I opened the port 9009 to allow
for traffic but that did not help either.

To run:


   1. Untar: AsterixDB_FeedTest.tar.gz
   2. cd into AsterixDB_FeedTest
   3. In AsterixDB WEBUI enter in the DDL and UPDATE statements above
   4. In the terminal enter : python feedInjector.py



The second method I tried was to insert values from a file into the feed. I
took this example from
https://github.com/apache/incubator-asterixdb/tree/master/asterix-app/src/test/resources/runtimets/queries/feeds/feeds_11

Here are the statements that I used in the WEBUI

DDL:

drop dataverse feeds if exists;
create dataverse feeds;
use dataverse feeds;

create type Log_Type as open {
LogId:string,
updateString:string
}

create dataset AstrixDB_Transaction_Log(Log_Type)
primary key LogId;
create index updateLogIndex on AstrixDB_Transaction_Log(LogId);

create feed log_feed
using file_feed
(("fs"="localfs"),("path"="127.0.0.1
:///Users/Bryan/Desktop/Masters_Project_Files/Data/DataFeedInjector/load.asm"),("format"="adm"),("type-name"="Log_Type"),("tuple-interval"="10"));

UPDATE :

use dataverse feeds;

set wait-for-completion-feed "true";

connect feed feeds.log_feed to dataset AstrixDB_Transaction_Log;

Query:

use dataverse feeds;

for $x in dataset AstrixDB_Transaction_Log
return $x

Bellow I have attached the load.adm file that I used for the loading.

When I execute the DDL statement into AsterixDB WEBUI,  it is successful,
but when I try to execute the Update statement I get the following error:

Failure in feed [AsterixException]

I am not sure what I am doing wrong.


One thing I did see happening is that when the feed does not work, 8/10
times the WEBUI becomes unresponsive yet the logs still say that the
instance is still ACTIVE.

Any help that you guys could give me would be helpful, I have been
stressing out for the last two weeks trying to figure this out. If possible
getting the first method to work would be ideal.


Thank you
Sincerely, Bryan Marsh