Change in asterixdb[master]: Introduced Local Filesystem Feed Adapter which can be used f...

["abdullah alamoudi (Code Review)" <do...@asterix-gerrit.ics.uci.edu>]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

    https://asterix-gerrit.ics.uci.edu/376

to look at the new patch set (#2).

Change subject: Introduced Local Filesystem Feed Adapter which can be used for continuous data ingestion from local filesystem. To determine what files to ingest, the user can provide a regular expression for the system to match before ingesting data.
......................................................................

Introduced Local Filesystem Feed Adapter which can be used for continuous data ingestion from local filesystem. To determine what files to ingest, the user can provide a regular expression for the system to match before ingesting data.

In this change, I have added filesystem based feed adapter. The adapter is aliased push_localfs.
It works in the following way:
When it starts, it starts ingesting available data in directories passed in the adapter
arguments. Once it is done with the existing files, it places a watch on the directories passed.
It is push based and so will only stop using a disconnect feed statement. Faulty records are
dropped and the feed will try to parse the next records.

Change-Id: I707756e3b4c9ffca4b55ec9817a08e5c16333010
---
A asterix-app/data/local-dir/data/even-more.txt
A asterix-app/data/local-dir/data/hello-world
A asterix-app/data/local-dir/data/ignored
A asterix-app/data/local-dir/data/more-records.txt
A asterix-app/data/local-dir/data/records.txt
A asterix-app/src/test/resources/runtimets/queries/external/query1/query1.1.ddl.aql
A asterix-app/src/test/resources/runtimets/queries/external/query1/query1.2.update.aql
A asterix-app/src/test/resources/runtimets/queries/external/query1/query1.3.query.aql
A asterix-app/src/test/resources/runtimets/results/external/query1/query1.1.adm
M asterix-app/src/test/resources/runtimets/testsuite.xml
M asterix-common/src/main/java/org/apache/asterix/common/parse/ITupleForwardPolicy.java
A asterix-external-data/src/main/java/org/apache/asterix/external/adapter/factory/LocalFileSystemFeedAdapterFactory.java
M asterix-external-data/src/main/java/org/apache/asterix/external/adapter/factory/NCFileSystemAdapterFactory.java
M asterix-external-data/src/main/java/org/apache/asterix/external/dataset/adapter/ClientBasedFeedAdapter.java
M asterix-external-data/src/main/java/org/apache/asterix/external/dataset/adapter/FeedClient.java
A asterix-external-data/src/main/java/org/apache/asterix/external/dataset/adapter/FileSystemWatcher.java
A asterix-external-data/src/main/java/org/apache/asterix/external/dataset/adapter/LocalFileSystemFeedAdapter.java
A asterix-external-data/src/main/java/org/apache/asterix/external/dataset/adapter/LocalFilesystemFeedClient.java
M asterix-external-data/src/main/java/org/apache/asterix/external/dataset/adapter/NCFileSystemAdapter.java
A asterix-external-data/src/main/java/org/apache/asterix/external/indexing/input/NCLocalFilesystemDirecoryInputStream.java
M asterix-metadata/src/main/java/org/apache/asterix/metadata/bootstrap/MetadataBootstrap.java
M asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/file/AbstractTupleParser.java
M asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/file/AsterixTupleParserFactory.java
M asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/file/CounterTimerTupleForwardPolicy.java
M asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/file/FrameFullTupleForwardPolicy.java
M asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/file/RateControlledTupleForwardPolicy.java
26 files changed, 1,097 insertions(+), 115 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/76/376/2
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/376
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I707756e3b4c9ffca4b55ec9817a08e5c16333010
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi <ba...@gmail.com>
Gerrit-Reviewer: Ian Maxon <im...@apache.org>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Reviewer: Murtadha Hubail <hu...@gmail.com>
Gerrit-Reviewer: Steven Jacobs <sj...@ucr.edu>