You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@accumulo.apache.org by bi...@apache.org on 2011/11/15 21:16:27 UTC
svn commit: r1202406 [1/2] - in /incubator/accumulo:
branches/1.4/docs/examples/ trunk/docs/examples/
Author: billie
Date: Tue Nov 15 20:16:26 2011
New Revision: 1202406
URL: http://svn.apache.org/viewvc?rev=1202406&view=rev
Log:
ACCUMULO-103 corresponding changes for 1.4 branch and trunk
Added:
incubator/accumulo/branches/1.4/docs/examples/README
incubator/accumulo/trunk/docs/examples/README
Modified:
incubator/accumulo/branches/1.4/docs/examples/README.aggregation
incubator/accumulo/branches/1.4/docs/examples/README.batch
incubator/accumulo/branches/1.4/docs/examples/README.bloom
incubator/accumulo/branches/1.4/docs/examples/README.bulkIngest
incubator/accumulo/branches/1.4/docs/examples/README.constraints
incubator/accumulo/branches/1.4/docs/examples/README.dirlist
incubator/accumulo/branches/1.4/docs/examples/README.filedata
incubator/accumulo/branches/1.4/docs/examples/README.filter
incubator/accumulo/branches/1.4/docs/examples/README.helloworld
incubator/accumulo/branches/1.4/docs/examples/README.mapred
incubator/accumulo/branches/1.4/docs/examples/README.shard
incubator/accumulo/trunk/docs/examples/README.aggregation
incubator/accumulo/trunk/docs/examples/README.batch
incubator/accumulo/trunk/docs/examples/README.bloom
incubator/accumulo/trunk/docs/examples/README.bulkIngest
incubator/accumulo/trunk/docs/examples/README.constraints
incubator/accumulo/trunk/docs/examples/README.dirlist
incubator/accumulo/trunk/docs/examples/README.filedata
incubator/accumulo/trunk/docs/examples/README.filter
incubator/accumulo/trunk/docs/examples/README.helloworld
incubator/accumulo/trunk/docs/examples/README.mapred
incubator/accumulo/trunk/docs/examples/README.shard
Added: incubator/accumulo/branches/1.4/docs/examples/README
URL: http://svn.apache.org/viewvc/incubator/accumulo/branches/1.4/docs/examples/README?rev=1202406&view=auto
==============================================================================
--- incubator/accumulo/branches/1.4/docs/examples/README (added)
+++ incubator/accumulo/branches/1.4/docs/examples/README Tue Nov 15 20:16:26 2011
@@ -0,0 +1,36 @@
+Title: Examples
+Notice: Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ .
+ http://www.apache.org/licenses/LICENSE-2.0
+ .
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Each README in the examples directory highlights the use of particular features of Accumulo.
+
+Before running any of the examples, the following steps must be performed.
+
+1. Install and run Accumulo via the instructions found in $ACCUMULO_HOME/README.
+Remember the instance name. It will be referred to as "instance" throughout the examples.
+A comma-separated list of zookeeper servers will be referred to as "zookeepers".
+
+2. Create an Accumulo user (see the [user manual][1]), or use the root user.
+The Accumulo user name will be referred to as "username" with password "password" throughout the examples.
+
+In all commands, you will need to replace "instance", "zookeepers", "username", and "password" with the values you set for your Accumulo instance.
+
+Commands intended to be run in bash are prefixed by '$'. These are always assumed to be run from the $ACCUMULO_HOME directory.
+
+Commands intended to be run in the Accumulo shell are prefixed by '>'.
+
+[1]: /accumulo/user_manual_1.4-incubating/Accumulo_Shell.html#User_Administration
Modified: incubator/accumulo/branches/1.4/docs/examples/README.aggregation
URL: http://svn.apache.org/viewvc/incubator/accumulo/branches/1.4/docs/examples/README.aggregation?rev=1202406&r1=1202405&r2=1202406&view=diff
==============================================================================
--- incubator/accumulo/branches/1.4/docs/examples/README.aggregation (original)
+++ incubator/accumulo/branches/1.4/docs/examples/README.aggregation Tue Nov 15 20:16:26 2011
@@ -1,32 +1,50 @@
+Title: Aggregation Example
+Notice: Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ .
+ http://www.apache.org/licenses/LICENSE-2.0
+ .
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
This is a simple aggregation example. To build this example run maven and then
copy the produced jar into the accumulo lib dir. This is already done in the
tar distribution.
-
-$ bin/accumulo shell -u user
-Enter current password for 'user'@'example': ***
-Shell - Accumulo Interactive Shell
--
-- version: 1.2-SNAPSHOT
-- instance name: example
-- instance id: 42f24290-dbc3-42db-b72d-f6647b4cb988
--
-- type 'help' for a list of available commands
--
-user@example> createtable aggtest1 -a app=org.apache.accumulo.examples.aggregation.SortedSetAggregator
-user@example aggtest1> insert foo app 1 a
-user@example aggtest1> insert foo app 1 b
-user@example aggtest1> scan
-foo app:1 [] a,b
-user@example aggtest1> insert foo app 1 z,1,foo,w
-user@example aggtest1> scan
-foo app:1 [] 1,a,b,foo,w,z
-user@example aggtest1> insert foo app 2 cat,dog,muskrat
-user@example aggtest1> insert foo app 2 mouse,bird
-user@example aggtest1> scan
-foo app:1 [] 1,a,b,foo,w,z
-foo app:2 [] bird,cat,dog,mouse,muskrat
-user@example aggtest1>
+ $ bin/accumulo shell -u username
+ Enter current password for 'username'@'instance': ***
+
+ Shell - Accumulo Interactive Shell
+ -
+ - version: 1.4.x-incubating
+ - instance name: instance
+ - instance id: 00000000-0000-0000-0000-000000000000
+ -
+ - type 'help' for a list of available commands
+ -
+ username@instance> createtable aggtest1 -a app=org.apache.accumulo.examples.aggregation.SortedSetAggregator
+ username@instance aggtest1> insert foo app 1 a
+ username@instance aggtest1> insert foo app 1 b
+ username@instance aggtest1> scan
+ foo app:1 [] a,b
+ username@instance aggtest1> insert foo app 1 z,1,foo,w
+ username@instance aggtest1> scan
+ foo app:1 [] 1,a,b,foo,w,z
+ username@instance aggtest1> insert foo app 2 cat,dog,muskrat
+ username@instance aggtest1> insert foo app 2 mouse,bird
+ username@instance aggtest1> scan
+ foo app:1 [] 1,a,b,foo,w,z
+ foo app:2 [] bird,cat,dog,mouse,muskrat
+ username@instance aggtest1>
In this example a table is created and the example set aggregator is
applied to the column family app.
Modified: incubator/accumulo/branches/1.4/docs/examples/README.batch
URL: http://svn.apache.org/viewvc/incubator/accumulo/branches/1.4/docs/examples/README.batch?rev=1202406&r1=1202405&r2=1202406&view=diff
==============================================================================
--- incubator/accumulo/branches/1.4/docs/examples/README.batch (original)
+++ incubator/accumulo/branches/1.4/docs/examples/README.batch Tue Nov 15 20:16:26 2011
@@ -1,3 +1,21 @@
+Title: Batch Writing and Scanning Example
+Notice: Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ .
+ http://www.apache.org/licenses/LICENSE-2.0
+ .
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
This is an example of how to use the batch writer and batch scanner. To compile
the example, run maven and copy the produced jar into the accumulo lib dir.
This is already done in the tar distribution.
@@ -5,26 +23,31 @@ This is already done in the tar distribu
Below are commands that add 10000 entries to accumulo and then do 100 random
queries. The write command generates random 50 byte values.
-Be sure to use the name of your instance (given as testInstance here) and the appropriate
-list of zookeeper nodes (given as localhost here).
-
-Before you run this, you must ensure that the user you are running as has the
-"exampleVis" authorization. (you can set this in the shell with "setauths -u root -s exampleVis")
-
-You must also create the table, exampleTable, ahead of time. (In the shell, use "createtable exampleTable")
-
-
- $ ./bin/accumulo org.apache.accumulo.examples.client.SequentialBatchWriter testInstance localhost root secret exampleTable 0 10000 50 20000000 500 20 exampleVis
- $ ./bin/accumulo org.apache.accumulo.examples.client.RandomBatchScanner testInstance localhost root secret exampleTable 100 0 10000 50 20 exampleVis
- 07 11:33:11,103 [client.CountingVerifyingReceiver] INFO : Generating 100 random queries...
- 07 11:33:11,112 [client.CountingVerifyingReceiver] INFO : finished
- 07 11:33:11,260 [client.CountingVerifyingReceiver] INFO : 694.44 lookups/sec 0.14 secs
-
- 07 11:33:11,260 [client.CountingVerifyingReceiver] INFO : num results : 100
-
- 07 11:33:11,364 [client.CountingVerifyingReceiver] INFO : Generating 100 random queries...
- 07 11:33:11,370 [client.CountingVerifyingReceiver] INFO : finished
- 07 11:33:11,416 [client.CountingVerifyingReceiver] INFO : 2173.91 lookups/sec 0.05 secs
+Be sure to use the name of your instance (given as instance here) and the appropriate
+list of zookeeper nodes (given as zookeepers here).
- 07 11:33:11,416 [client.CountingVerifyingReceiver] INFO : num results : 100
+Before you run this, you must ensure that the user you are running has the
+"exampleVis" authorization. (you can set this in the shell with "setauths -u username -s exampleVis")
+ $ ./bin/accumulo shell -u root
+ > setauths -u username -s exampleVis
+ > exit
+
+You must also create the table, batchtest1, ahead of time. (In the shell, use "createtable batchtest1")
+
+ $ ./bin/accumulo shell -u username
+ > createtable batchtest1
+ > exit
+ $ ./bin/accumulo org.apache.accumulo.examples.client.SequentialBatchWriter instance zookeepers username password batchtest1 0 10000 50 20000000 500 20 exampleVis
+ $ ./bin/accumulo org.apache.accumulo.examples.client.RandomBatchScanner instance zookeepers username password batchtest1 100 0 10000 50 20 exampleVis
+ 07 11:33:11,103 [client.CountingVerifyingReceiver] INFO : Generating 100 random queries...
+ 07 11:33:11,112 [client.CountingVerifyingReceiver] INFO : finished
+ 07 11:33:11,260 [client.CountingVerifyingReceiver] INFO : 694.44 lookups/sec 0.14 secs
+
+ 07 11:33:11,260 [client.CountingVerifyingReceiver] INFO : num results : 100
+
+ 07 11:33:11,364 [client.CountingVerifyingReceiver] INFO : Generating 100 random queries...
+ 07 11:33:11,370 [client.CountingVerifyingReceiver] INFO : finished
+ 07 11:33:11,416 [client.CountingVerifyingReceiver] INFO : 2173.91 lookups/sec 0.05 secs
+
+ 07 11:33:11,416 [client.CountingVerifyingReceiver] INFO : num results : 100
Modified: incubator/accumulo/branches/1.4/docs/examples/README.bloom
URL: http://svn.apache.org/viewvc/incubator/accumulo/branches/1.4/docs/examples/README.bloom?rev=1202406&r1=1202405&r2=1202406&view=diff
==============================================================================
--- incubator/accumulo/branches/1.4/docs/examples/README.bloom (original)
+++ incubator/accumulo/branches/1.4/docs/examples/README.bloom Tue Nov 15 20:16:26 2011
@@ -1,39 +1,53 @@
+Title: Bloom Filter Example
+Notice: Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ .
+ http://www.apache.org/licenses/LICENSE-2.0
+ .
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
This example shows how to create a table with bloom filters enabled. It also
shows how bloom filters increase query performance when looking for values that
do not exist in a table.
Below table named bloom_test is created and bloom filters are enabled.
- [user@localhost bin]$ ./accumulo shell -u root -p secret
+ $ ./accumulo shell -u username -p password
Shell - Accumulo Interactive Shell
- - version: 1.2-SNAPSHOT
- - instance id: c4b72d26-16b3-408d-bf3e-c15a1fb51ba0
+ - version: 1.4.x-incubating
+ - instance name: instance
+ - instance id: 00000000-0000-0000-0000-000000000000
-
- type 'help' for a list of available commands
-
- root@localhost> setauths -u root -s CV1
- root@localhost> createtable bloom_test
- root@localhost bloom_test> config -t bloom_test -s table.bloom.enabled=true
+ username@instance> setauths -u username -s exampleVis
+ username@instance> createtable bloom_test
+ username@instance bloom_test> config -t bloom_test -s table.bloom.enabled=true
+ username@instance bloom_test> exit
Below 1 million random values are inserted into accumulo. The randomly
generated rows range between 0 and 1 billion. The random number generator is
initialized with the seed 7.
- [user@localhost bin]$ ../bin/accumulo org.apache.accumulo.examples.client.RandomBatchWriter -s 7 testInstance localhost root secret bloom_test 1000000 0 1000000000 50 2000000 60000 3 CV1
+ $ ./bin/accumulo org.apache.accumulo.examples.client.RandomBatchWriter -s 7 instance zookeepers username password bloom_test 1000000 0 1000000000 50 2000000 60000 3 exampleVis
Below the table is flushed, look at the monitor page and wait for the flush to
complete.
- [user@localhost bin]$ ./accumulo shell -u root -p secret
- Shell - Accumulo Interactive Shell
- - version: 1.2-SNAPSHOT
- - instance id: c4b72d26-16b3-408d-bf3e-c15a1fb51ba0
- -
- - type 'help' for a list of available commands
- -
- root@localhost> flush -t bloom_test
+ $ ./bin/accumulo shell -u username -p password
+ username@instance> flush -t bloom_test
Flush of table bloom_test initiated...
- root@localhost> quit
+ username@instance> exit
The flush will be finished when there are no entries in memory and the
number of minor compactions goes to zero. Refresh the page to see changes to the table.
@@ -42,9 +56,9 @@ After the flush completes, 500 random qu
same seed is used to generate the queries, therefore everything is found in the
table.
- [user@localhost bin]$ ../bin/accumulo org.apache.accumulo.examples.client.RandomBatchScanner -s 7 testInstance localhost root secret bloom_test 500 0 1000000000 50 20 CV1
+ $ ./bin/accumulo org.apache.accumulo.examples.client.RandomBatchScanner -s 7 instance zookeepers username password bloom_test 500 0 1000000000 50 20 exampleVis
Generating 500 random queries...finished
- 96.19 lookups/sec 5.20 secs
+ 96.19 lookups/sec 5.20 secs
num results : 500
Generating 500 random queries...finished
102.35 lookups/sec 4.89 secs
@@ -54,7 +68,7 @@ Below another 500 queries are performed,
in nothing being found. In this case the lookups are much faster because of
the bloom filters.
- [user@localhost bin]$ ../bin/accumulo org.apache.accumulo.examples.client.RandomBatchScanner -s 8 testInstance localhost root secret bloom_test 500 0 1000000000 50 20 CV1
+ $ ../bin/accumulo org.apache.accumulo.examples.client.RandomBatchScanner -s 8 instance zookeepers username password bloom_test 500 0 1000000000 50 20 exampleVis
Generating 500 random queries...finished
2212.39 lookups/sec 0.23 secs
num results : 0
@@ -66,7 +80,7 @@ the bloom filters.
********************************************************************************
-Bloom filters can also speed up lookups for entires that exist. In accumulo
+Bloom filters can also speed up lookups for entries that exist. In accumulo
data is divided into tablets and each tablet has multiple map files. Every
lookup in accumulo goes to a specific tablet where a lookup is done on each
map file in the tablet. So if a tablet has three map files, lookup performance
@@ -77,12 +91,12 @@ identical tables were created using the
filters, the other did not. Also the major compaction ratio was increased to
prevent the files from being compacted into one file.
- * Insert 1 million entires using RandomBatchWriter with a seed of 7
- * Flush the table using the shell
- * Insert 1 million entires using RandomBatchWriter with a seed of 8
- * Flush the table using the shell
- * Insert 1 million entires using RandomBatchWriter with a seed of 9
- * Flush the table using the shell
+ * Insert 1 million entries using RandomBatchWriter with a seed of 7
+ * Flush the table using the shell
+ * Insert 1 million entries using RandomBatchWriter with a seed of 8
+ * Flush the table using the shell
+ * Insert 1 million entries using RandomBatchWriter with a seed of 9
+ * Flush the table using the shell
After following the above steps, each table will have a tablet with three map
files. Each map file will contain 1 million entries generated with a different
@@ -92,26 +106,22 @@ Below 500 lookups are done against the t
NG seed 7. Even though only one map file will likely contain entries for this
seed, all map files will be interrogated.
- [user@localhost examples]$ ../bin/accumulo org.apache.accumulo.examples.client.RandomBatchScanner -s 7 testInstance localhost root secret test 500 0 1000000000 50 20 CV1
- 05 07:52:36,789 [conf.AccumuloConfiguration] WARN : accumulo-site.xml not found on classpath
+ $ ./bin/accumulo org.apache.accumulo.examples.client.RandomBatchScanner -s 7 instance zookeepers username password bloom_test1 500 0 1000000000 50 20 exampleVis
Generating 500 random queries...finished
- 35.09 lookups/sec 14.25 secs
+ 35.09 lookups/sec 14.25 secs
num results : 500
Generating 500 random queries...finished
- 35.33 lookups/sec 14.15 secs
+ 35.33 lookups/sec 14.15 secs
num results : 500
Below the same lookups are done against the table with bloom filters. The
lookups were 2.86 times faster because only one map file was used, even though three
map files existed.
- [user@localhost examples]$ ../bin/accumulo org.apache.accumulo.examples.client.RandomBatchScanner -s 7 testInstance localhost root secret bloom_test2 500 0 1000000000 50 20 CV1
- 05 07:53:15,650 [conf.AccumuloConfiguration] WARN : accumulo-site.xml not found on classpath
+ $ ./bin/accumulo org.apache.accumulo.examples.client.RandomBatchScanner -s 7 instance zookeepers username password bloom_test2 500 0 1000000000 50 20 exampleVis
Generating 500 random queries...finished
- 99.03 lookups/sec 5.05 secs
+ 99.03 lookups/sec 5.05 secs
num results : 500
Generating 500 random queries...finished
101.15 lookups/sec 4.94 secs
num results : 500
-
-
Modified: incubator/accumulo/branches/1.4/docs/examples/README.bulkIngest
URL: http://svn.apache.org/viewvc/incubator/accumulo/branches/1.4/docs/examples/README.bulkIngest?rev=1202406&r1=1202405&r2=1202406&view=diff
==============================================================================
--- incubator/accumulo/branches/1.4/docs/examples/README.bulkIngest (original)
+++ incubator/accumulo/branches/1.4/docs/examples/README.bulkIngest Tue Nov 15 20:16:26 2011
@@ -1,16 +1,29 @@
+Title: Bulk Ingest Example
+Notice: Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ .
+ http://www.apache.org/licenses/LICENSE-2.0
+ .
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
This is an example of how to bulk ingest data into accumulo using map reduce.
-Compile using Maven. Then copy the produced jar into the accumulo lib dir.
-
+
The following commands show how to run this example. This example creates a
table called test_bulk which has two initial split points. Then 1000 rows of
test data are created in HDFS. After that the 1000 rows are ingested into
accumulo. Then we verify the 1000 rows are in accumulo. The
-first two arguments to all of the commands except for GenerateTestData is the
-accumulo instance name, and a comman-separated list of zookeepers.
-
-NOTE: You should replace "root" with your username that has the appropriate
-permissions on the table, and "secret" with that user's password.
-$ACCUMULO_HOME should also be replaced with the home directory where accumulo is stored.
+first two arguments to all of the commands except for GenerateTestData are the
+accumulo instance name, and a comma-separated list of zookeepers.
$ ./bin/accumulo org.apache.accumulo.examples.mapreduce.bulk.SetupTable instance zookeepers username password test_bulk row_00000333 row_00000666
$ ./bin/accumulo org.apache.accumulo.examples.mapreduce.bulk.GenerateTestData 0 1000 bulk/test_1.txt
@@ -18,5 +31,4 @@ $ACCUMULO_HOME should also be replaced w
$ ./bin/tool.sh lib/accumulo-examples-*[^c].jar org.apache.accumulo.examples.mapreduce.bulk.BulkIngestExample instance zookeepers username password test_bulk bulk tmp/bulkWork
$ ./bin/accumulo org.apache.accumulo.examples.mapreduce.bulk.VerifyIngest instance zookeepers username password test_bulk 0 1000
-
For a high level discussion of bulk ingest, see the docs dir.
Modified: incubator/accumulo/branches/1.4/docs/examples/README.constraints
URL: http://svn.apache.org/viewvc/incubator/accumulo/branches/1.4/docs/examples/README.constraints?rev=1202406&r1=1202405&r2=1202406&view=diff
==============================================================================
--- incubator/accumulo/branches/1.4/docs/examples/README.constraints (original)
+++ incubator/accumulo/branches/1.4/docs/examples/README.constraints Tue Nov 15 20:16:26 2011
@@ -1,31 +1,48 @@
+Title: Constraints Example
+Notice: Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ .
+ http://www.apache.org/licenses/LICENSE-2.0
+ .
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
This an example of how to create a table with constraints. Below a table is
create with two example constraints. One constraints does not allow non alpha
numeric keys. The other constraint does not allow non numeric values. Two
inserts that violate these constraints are attempted and denied. The scan at
the end shows the inserts were not allowed.
-[user@localhost bin]$ ./accumulo shell -u user -p pass
-
- Shell - Accumulo Interactive Shell
- -
- - version: 1.4.0-incubating-SNAPSHOT
- - instance name: example
- - instance id: 00000000-0000-0000-0000-000000000000
- -
- - type 'help' for a list of available commands
- -
- user@example> createtable testConstraints
- user@example testConstraints> config -t testConstraints -s table.constraint.1=org.apache.accumulo.examples.constraints.NumericValueConstraint
- user@example testConstraints> config -t testConstraints -s table.constraint.2=org.apache.accumulo.examples.constraints.AlphaNumKeyConstraint
- user@example testConstraints> insert r1 cf1 cq1 1111
- user@example testConstraints> insert r1 cf1 cq1 ABC
+ $ ./bin/accumulo shell -u username -p pass
+
+ Shell - Accumulo Interactive Shell
+ -
+ - version: 1.4.x-incubating
+ - instance name: instance
+ - instance id: 00000000-0000-0000-0000-000000000000
+ -
+ - type 'help' for a list of available commands
+ -
+ username@instance> createtable testConstraints
+ username@instance testConstraints> config -t testConstraints -s table.constraint.1=org.apache.accumulo.examples.constraints.NumericValueConstraint
+ username@instance testConstraints> config -t testConstraints -s table.constraint.2=org.apache.accumulo.examples.constraints.AlphaNumKeyConstrain
+ username@instance testConstraints> insert r1 cf1 cq1 1111
+ username@instance testConstraints> insert r1 cf1 cq1 ABC
Constraint Failures:
ConstraintViolationSummary(constrainClass:org.apache.accumulo.examples.constraints.NumericValueConstraint, violationCode:1, violationDescription:Value is not numeric, numberOfViolatingMutations:1)
- user@example testConstraints> insert r1! cf1 cq1 ABC
+ username@instance testConstraints> insert r1! cf1 cq1 ABC
Constraint Failures:
ConstraintViolationSummary(constrainClass:org.apache.accumulo.examples.constraints.NumericValueConstraint, violationCode:1, violationDescription:Value is not numeric, numberOfViolatingMutations:1)
ConstraintViolationSummary(constrainClass:org.apache.accumulo.examples.constraints.AlphaNumKeyConstraint, violationCode:1, violationDescription:Row was not alpha numeric, numberOfViolatingMutations:1)
- user@example testConstraints> scan
- r1 cf1:cq1 [] 1111
- user@example testConstraints>
-
+ username@instance testConstraints> scan
+ r1 cf1:cq1 [] 1111
+ username@instance testConstraints>
Modified: incubator/accumulo/branches/1.4/docs/examples/README.dirlist
URL: http://svn.apache.org/viewvc/incubator/accumulo/branches/1.4/docs/examples/README.dirlist?rev=1202406&r1=1202405&r2=1202406&view=diff
==============================================================================
--- incubator/accumulo/branches/1.4/docs/examples/README.dirlist (original)
+++ incubator/accumulo/branches/1.4/docs/examples/README.dirlist Tue Nov 15 20:16:26 2011
@@ -1,3 +1,21 @@
+Title: File System Archive
+Notice: Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ .
+ http://www.apache.org/licenses/LICENSE-2.0
+ .
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
This example stores filesystem information in accumulo. The example stores the information in the following three tables. More information about the table structures can be found at the end of README.dirlist.
* directory table : This table stores information about the filesystem directory structure.
@@ -15,76 +33,79 @@ This example shows how to use Accumulo t
To begin, ingest some data with Ingest.java.
- bin/accumulo org.apache.accumulo.examples.dirlist.Ingest test-instance localhost root password dirTable indexTable dataTable visibility 100000 /local/user1/workspace
+ $ ./bin/accumulo org.apache.accumulo.examples.dirlist.Ingest instance zookeepers username password dirTable indexTable dataTable visibility 100000 /local/username/workspace
-This may take some time if there are large files in the /local/user1/workspace directory. If you use 0 instead of 100000 on the command line, the ingest will run much faster, but it will not put any file data into Accumulo (the dataTable will be empty).
+This may take some time if there are large files in the /local/username/workspace directory. If you use 0 instead of 100000 on the command line, the ingest will run much faster, but it will not put any file data into Accumulo (the dataTable will be empty).
Note that running this example will create tables dirTable, indexTable, and dataTable in Accumulo that you should delete when you have completed the example.
-If you modify a file or add new files in the directory ingested (e.g. /local/user1/workspace), you can run Ingest again to add new information into the Accumulo tables.
+If you modify a file or add new files in the directory ingested (e.g. /local/username/workspace), you can run Ingest again to add new information into the Accumulo tables.
-To browse the data ingested, use Viewer.java. Be sure to give the root user the authorizations to see the data.
+To browse the data ingested, use Viewer.java. Be sure to give the "username" user the authorizations to see the data.
- bin/accumulo org.apache.accumulo.examples.dirlist.Viewer test-instance localhost root password dirTable dataTable auths /local/user1/workspace
+ $ ./bin/accumulo org.apache.accumulo.examples.dirlist.Viewer instance zookeepers username password dirTable dataTable auths /local/username/workspace
To list the contents of specific directories, use QueryUtil.java.
- bin/accumulo org.apache.accumulo.examples.dirlist.QueryUtil test-instance localhost root password dirTable auths /local/user1
- bin/accumulo org.apache.accumulo.examples.dirlist.QueryUtil test-instance localhost root password dirTable auths /local/user1/workspace
+ $ ./bin/accumulo org.apache.accumulo.examples.dirlist.QueryUtil instance zookeepers username password dirTable auths /local/username
+ $ ./bin/accumulo org.apache.accumulo.examples.dirlist.QueryUtil instance zookeepers username password dirTable auths /local/username/workspace
To perform searches on file or directory names, also use QueryUtil.java. Search terms must contain no more than one wild card and cannot contain "/".
-*Note* these queries run on the _dirindex_ table instead of the direxample table.
+*Note* these queries run on the _indexTable_ table instead of the dirTable table.
- $ ./bin/accumulo org.apache.accumulo.examples.dirlist.QueryUtil instance zookeepers username password dirindex exampleVis filename -search
- $ ./bin/accumulo org.apache.accumulo.examples.dirlist.QueryUtil instance zookeepers username password dirindex exampleVis 'filename*' -search
- $ ./bin/accumulo org.apache.accumulo.examples.dirlist.QueryUtil instance zookeepers username password dirindex exampleVis '*jar' -search
- $ ./bin/accumulo org.apache.accumulo.examples.dirlist.QueryUtil instance zookeepers username password dirindex exampleVis filename*jar -search
+ $ ./bin/accumulo org.apache.accumulo.examples.dirlist.QueryUtil instance zookeepers username password indexTable exampleVis filename -search
+ $ ./bin/accumulo org.apache.accumulo.examples.dirlist.QueryUtil instance zookeepers username password indexTable exampleVis 'filename*' -search
+ $ ./bin/accumulo org.apache.accumulo.examples.dirlist.QueryUtil instance zookeepers username password indexTable exampleVis '*jar' -search
+ $ ./bin/accumulo org.apache.accumulo.examples.dirlist.QueryUtil instance zookeepers username password indexTable exampleVis filename*jar -search
To count the number of direct children (directories and files) and descendants (children and children's descendents, directories and files), run the FileCountMR over the dirTable table.
The results can be written back to the same table.
- $ ./bin/tool.sh lib/accumulo-examples-*[^c].jar org.apache.accumulo.examples.dirlist.FileCountMR instance zookeepers username password direxample direxample exampleVis exampleVis
+ $ ./bin/tool.sh lib/accumulo-examples-*[^c].jar org.apache.accumulo.examples.dirlist.FileCountMR instance zookeepers username password dirTable dirTable exampleVis exampleVis
Alternatively, you can run FileCount.java which performs the same counts but is not a MapReduce. FileCount will be faster for small data sets.
-Directory Table
+## Directory Table
Here is a illustration of what data looks like in the directory table:
-row colf:colq [vis] value
-000 dir:exec [visibility] true
-000 dir:hidden [visibility] false
-000 dir:lastmod [visibility] 1291996886000
-000 dir:length [visibility] 1666
-001/local dir:exec [visibility] true
-001/local dir:hidden [visibility] false
-001/local dir:lastmod [visibility] 1304945270000
-001/local dir:length [visibility] 272
-002/local/Accumulo.README \x7F\xFF\xFE\xCFH\xA1\x82\x97:exec [mygroup] false
-002/local/Accumulo.README \x7F\xFF\xFE\xCFH\xA1\x82\x97:hidden [mygroup] false
-002/local/Accumulo.README \x7F\xFF\xFE\xCFH\xA1\x82\x97:lastmod [mygroup] 1308746481000
-002/local/Accumulo.README \x7F\xFF\xFE\xCFH\xA1\x82\x97:length [mygroup] 9192
-002/local/Accumulo.README \x7F\xFF\xFE\xCFH\xA1\x82\x97:md5 [mygroup] 274af6419a3c4c4a259260ac7017cbf1
+
+ row colf:colq [vis] value
+ 000 dir:exec [exampleVis] true
+ 000 dir:hidden [exampleVis] false
+ 000 dir:lastmod [exampleVis] 1291996886000
+ 000 dir:length [exampleVis] 1666
+ 001/local dir:exec [exampleVis] true
+ 001/local dir:hidden [exampleVis] false
+ 001/local dir:lastmod [exampleVis] 1304945270000
+ 001/local dir:length [exampleVis] 272
+ 002/local/Accumulo.README \x7F\xFF\xFE\xCFH\xA1\x82\x97:exec [exampleVis] false
+ 002/local/Accumulo.README \x7F\xFF\xFE\xCFH\xA1\x82\x97:hidden [exampleVis] false
+ 002/local/Accumulo.README \x7F\xFF\xFE\xCFH\xA1\x82\x97:lastmod [exampleVis] 1308746481000
+ 002/local/Accumulo.README \x7F\xFF\xFE\xCFH\xA1\x82\x97:length [exampleVis] 9192
+ 002/local/Accumulo.README \x7F\xFF\xFE\xCFH\xA1\x82\x97:md5 [exampleVis] 274af6419a3c4c4a259260ac7017cbf1
The rows are of the form depth + path, where depth is the number of slashes ("/") in the path padded to 3 digits. This is so that all the children of a directory appear as consecutive keys in Accumulo; without the depth, you would for example see all the subdirectories of /local before you saw /usr.
For directories the column family is "dir". For files the column family is Long.MAX_VALUE - lastModified in bytes rather than string format so that newer versions sort earlier.
-Index Table
+## Index Table
Here is an illustration of what data looks like in the index table:
-row colf:colq [vis]
-fAccumulo.README i:002/local/Accumulo.README [visibility]
-flocal i:001/local [visibility]
-rEMDAER.olumuccA i:002/local/Accumulo.README [visibility]
-rlacol i:001/local [visibility]
+
+ row colf:colq [vis]
+ fAccumulo.README i:002/local/Accumulo.README [exampleVis]
+ flocal i:001/local [exampleVis]
+ rEMDAER.olumuccA i:002/local/Accumulo.README [exampleVis]
+ rlacol i:001/local [exampleVis]
The values of the index table are null. The rows are of the form "f" + filename or "r" + reverse file name. This is to enable searches with wildcards at the beginning, middle, or end.
-Data Table
+## Data Table
Here is an illustration of what data looks like in the data table:
-row colf:colq [vis] value
-274af6419a3c4c4a259260ac7017cbf1 refs:e77276a2b56e5c15b540eaae32b12c69\x00filext [mygroup] README
-274af6419a3c4c4a259260ac7017cbf1 refs:e77276a2b56e5c15b540eaae32b12c69\x00name [mygroup] /local/Accumulo.README
-274af6419a3c4c4a259260ac7017cbf1 ~chunk:\x00\x0FB@\x00\x00\x00\x00 [mygroup] *******************************************************************************\x0A1. Building\x0A\x0AIn the normal tarball or RPM release of accumulo, [truncated]
-274af6419a3c4c4a259260ac7017cbf1 ~chunk:\x00\x0FB@\x00\x00\x00\x01 [mygroup]
+
+ row colf:colq [vis] value
+ 274af6419a3c4c4a259260ac7017cbf1 refs:e77276a2b56e5c15b540eaae32b12c69\x00filext [exampleVis] README
+ 274af6419a3c4c4a259260ac7017cbf1 refs:e77276a2b56e5c15b540eaae32b12c69\x00name [exampleVis] /local/Accumulo.README
+ 274af6419a3c4c4a259260ac7017cbf1 ~chunk:\x00\x0FB@\x00\x00\x00\x00 [exampleVis] *******************************************************************************\x0A1. Building\x0A\x0AIn the normal tarball or RPM release of accumulo, [truncated]
+ 274af6419a3c4c4a259260ac7017cbf1 ~chunk:\x00\x0FB@\x00\x00\x00\x01 [exampleVis]
The rows are the md5 hash of the file. Some column family : column qualifier pairs are "refs" : hash of file name + null byte + property name, in which case the value is property value. There can be multiple references to the same file which are distinguished by the hash of the file name.
Other column family : column qualifier pairs are "~chunk" : chunk size in bytes + chunk number in bytes, in which case the value is the bytes for that chunk of the file. There is an end of file data marker whose chunk number is the number of chunks for the file and whose value is empty.
Modified: incubator/accumulo/branches/1.4/docs/examples/README.filedata
URL: http://svn.apache.org/viewvc/incubator/accumulo/branches/1.4/docs/examples/README.filedata?rev=1202406&r1=1202405&r2=1202406&view=diff
==============================================================================
--- incubator/accumulo/branches/1.4/docs/examples/README.filedata (original)
+++ incubator/accumulo/branches/1.4/docs/examples/README.filedata Tue Nov 15 20:16:26 2011
@@ -1,4 +1,22 @@
-This example archives file data into a Accumulo table. Files with duplicate data are only stored once.
+Title: File System Archive (Data Only)
+Notice: Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ .
+ http://www.apache.org/licenses/LICENSE-2.0
+ .
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+This example archives file data into an Accumulo table. Files with duplicate data are only stored once.
The example has the following classes:
* ChunkCombiner - An Iterator that dedupes file data and sets their visibilities to a combined visibility based on current references to the file data.
@@ -7,7 +25,6 @@ The example has the following classes:
* FileDataIngest - Takes a list of files and archives them into Accumulo keyed on the SHA1 hashes of the files.
* FileDataQuery - Retrieves file data based on the SHA1 hash of the file.
* KeyUtil - A utility for creating and parsing null-byte separated strings into/from Text objects.
- * PeekingIterator - A Java Iterator with a peek method.
* VisibilityCombiner - A utility for merging visibilities into the form (VIS1)|(VIS2)|...
This example is coupled with the dirlist example. See README.dirlist for instructions.
\ No newline at end of file
Modified: incubator/accumulo/branches/1.4/docs/examples/README.filter
URL: http://svn.apache.org/viewvc/incubator/accumulo/branches/1.4/docs/examples/README.filter?rev=1202406&r1=1202405&r2=1202406&view=diff
==============================================================================
--- incubator/accumulo/branches/1.4/docs/examples/README.filter (original)
+++ incubator/accumulo/branches/1.4/docs/examples/README.filter Tue Nov 15 20:16:26 2011
@@ -1,3 +1,21 @@
+Title: Filter Example
+Notice: Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ .
+ http://www.apache.org/licenses/LICENSE-2.0
+ .
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
This is a simple filter example. It uses the AgeOffFilter that is provided as
part of the core package org.apache.accumulo.core.iterators.filter. Filters are used by
the FilteringIterator to select desired key/value pairs (or weed out undesired
@@ -22,9 +40,9 @@ value pair are to be delivered and false
foo a:b [] c
... wait 30 seconds ...
-
-user@example filtertest> scan
-user@example filtertest>
+
+ username@instance filtertest> scan
+ username@instance filtertest>
Note the absence of the entry inserted more than 30 seconds ago. Since the
scope was set to "scan", this means the entry is still in Accumulo, but is
@@ -57,6 +75,7 @@ After the compaction runs, the newly cre
Accumulo garbage collector will remove the old files.
To see the iterator settings for a table, use:
+
username@instance filtertest> config -t filtertest -f iterator
---------+------------------------------------------+----------------------------------------------------------
SCOPE | NAME | VALUE
Modified: incubator/accumulo/branches/1.4/docs/examples/README.helloworld
URL: http://svn.apache.org/viewvc/incubator/accumulo/branches/1.4/docs/examples/README.helloworld?rev=1202406&r1=1202405&r2=1202406&view=diff
==============================================================================
--- incubator/accumulo/branches/1.4/docs/examples/README.helloworld (original)
+++ incubator/accumulo/branches/1.4/docs/examples/README.helloworld Tue Nov 15 20:16:26 2011
@@ -1,46 +1,52 @@
-
-This tutorial uses the following Java classes, which can be found in src/examples/src/main/java/accumulo/examples/helloworld:
-
-InsertWithBatchWriter.java - Inserts 10K rows (50K entries) into accumulo as each row has 5 entries
-InsertWithOutputFormat.java - Example of inserting data in MapReduce
-ReadData.java - Reads all data between two rows
-
-NOTE: Feel free to use your own username, password, instance name, and table name.
-The ones used below are just examples.
-
-Initialize accumulo, create an instance called 'testInstance' and set the root password to 'pass':
-
- bin/accumulo init
-
-Start accumulo:
-
- bin/start-all.sh
+Title: Hello World Example
+Notice: Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ .
+ http://www.apache.org/licenses/LICENSE-2.0
+ .
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+This tutorial uses the following Java classes, which can be found in org.apache.accumulo.examples.helloworld in the accumulo-examples module:
+
+ * InsertWithBatchWriter.java - Inserts 10K rows (50K entries) into accumulo with each row having 5 entries
+ * InsertWithOutputFormat.java - Example of inserting data in MapReduce
+ * ReadData.java - Reads all data between two rows
Log into the accumulo shell:
- bin/accumulo shell -u root -p pass
+ $ ./bin/accumulo shell -u username -p password
Create a table called 'hellotable':
- root@testInstance> createtable hellotable
+ username@instance> createtable hellotable
Launch a Java program that inserts data with a BatchWriter:
- bin/accumulo org.apache.accumulo.examples.helloworld.InsertWithBatchWriter testInstance localhost hellotable root pass
-
+ $ ./bin/accumulo org.apache.accumulo.examples.helloworld.InsertWithBatchWriter instance zookeepers hellotable username password
+
Alternatively, the same data can be inserted using MapReduce writers:
- bin/accumulo org.apache.accumulo.examples.helloworld.InsertWithOutputFormat testInstance localhost hellotable root pass
-
-On accumulo status page at the URL below, you should see 50K entries
+ $ ./bin/accumulo org.apache.accumulo.examples.helloworld.InsertWithOutputFormat instance zookeepers hellotable username password
+
+On the accumulo status page at the URL below (where 'master' is replaced with the name or IP of your accumulo master), you should see 50K entries
- http://localhost:50095/
+ http://master:50095/
To view the entries, use the shell to scan the table:
- root@testInstance> table hellotable
- root@testInstance hellotable> scan
+ username@instance> table hellotable
+ username@instance hellotable> scan
You can also use a Java class to scan the table:
- bin/accumulo org.apache.accumulo.examples.helloworld.ReadData testInstance localhost hellotable root pass row_0 row_1001
+ $ ./bin/accumulo org.apache.accumulo.examples.helloworld.ReadData instance zookeepers hellotable username password row_0 row_1001
Modified: incubator/accumulo/branches/1.4/docs/examples/README.mapred
URL: http://svn.apache.org/viewvc/incubator/accumulo/branches/1.4/docs/examples/README.mapred?rev=1202406&r1=1202405&r2=1202406&view=diff
==============================================================================
--- incubator/accumulo/branches/1.4/docs/examples/README.mapred (original)
+++ incubator/accumulo/branches/1.4/docs/examples/README.mapred Tue Nov 15 20:16:26 2011
@@ -1,6 +1,20 @@
-This example uses mapreduce and accumulo to compute word counts for a
-set of documents. This is accomplished using a map only map-reduce
-job and a accumulo table with aggregators.
+Title: MapReduce Example
+Notice: Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ .
+ http://www.apache.org/licenses/LICENSE-2.0
+ .
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
This example uses mapreduce and accumulo to compute word counts for a set of
documents. This is accomplished using a map-only mapreduce job and a
@@ -9,28 +23,28 @@ accumulo table with aggregators.
To run this example you will need a directory in HDFS containing text files.
The accumulo readme will be used to show how to run this example.
- [user1@localhost accumulo]$ hadoop fs -copyFromLocal $ACCUMULO_HOME/README /user/user1/wc/Accumulo.README
- [user1@localhost accumulo]$ hadoop fs -ls /user/user1/wc
- Found 1 items
- -rw-r--r-- 2 user1 supergroup 9359 2009-07-15 17:54 /user/user1/wc/Accumulo.README
+ $ hadoop fs -copyFromLocal $ACCUMULO_HOME/README /user/username/wc/Accumulo.README
+ $ hadoop fs -ls /user/username/wc
+ Found 1 items
+ -rw-r--r-- 2 username supergroup 9359 2009-07-15 17:54 /user/username/wc/Accumulo.README
The first part of running this example is to create a table with aggregation
for the column family count.
- [user1@localhost accumulo]$ ./bin/accumulo shell -u root -p secret
- Shell - Accumulo Interactive Shell
- - version: 1.4.0-incubating-SNAPSHOT
- - instance id: 00000000-0000-0000-0000-000000000000
- -
- - type 'help' for a list of available commands
- -
- root@localhost> createtable wordCount -a count=org.apache.accumulo.core.iterators.aggregation.StringSummation
- root@localhost wordCount> quit
+ $ ./bin/accumulo shell -u username -p password
+ Shell - Accumulo Interactive Shell
+ - version: 1.4.x-incubating
+ - instance name: instance
+ - instance id: 00000000-0000-0000-0000-000000000000
+ -
+ - type 'help' for a list of available commands
+ -
+ username@instance> createtable wordCount -a count=org.apache.accumulo.core.iterators.aggregation.StringSummation
+ username@instance wordCount> quit
-
After creating the table, run the word count map reduce job.
- $ bin/tool.sh lib/accumulo-examples-*[^c].jar org.apache.accumulo.examples.mapreduce.WordCount instance zookeepers /user/user1/wc wordCount -u username -p password
+ $ bin/tool.sh lib/accumulo-examples-*[^c].jar org.apache.accumulo.examples.mapreduce.WordCount instance zookeepers /user/username/wc wordCount -u username -p password
11/02/07 18:20:11 INFO input.FileInputFormat: Total input paths to process : 1
11/02/07 18:20:12 INFO mapred.JobClient: Running job: job_201102071740_0003
@@ -47,31 +61,25 @@ After creating the table, run the word c
11/02/07 18:20:22 INFO mapred.JobClient: Map input records=255
11/02/07 18:20:22 INFO mapred.JobClient: Spilled Records=0
11/02/07 18:20:22 INFO mapred.JobClient: Map output records=1452
-
+
After the map reduce job completes, query the accumulo table to see word
counts.
- $ ./bin/accumulo shell -u root -p secret
- Shell - Accumulo Interactive Shell
- - version: 1.4.0-incubating-SNAPSHOT
- - instance id: 00000000-0000-0000-0000-000000000000
- -
- - type 'help' for a list of available commands
- -
- root@localhost> table wordCount
- root@localhost wordCount> scan -b the
- the count:20080906 [] 75
- their count:20080906 [] 2
- them count:20080906 [] 1
- then count:20080906 [] 1
- there count:20080906 [] 1
- these count:20080906 [] 3
- this count:20080906 [] 6
- through count:20080906 [] 1
- time count:20080906 [] 3
- time. count:20080906 [] 1
- to count:20080906 [] 27
- total count:20080906 [] 1
- tserver, count:20080906 [] 1
- tserver.compaction.major.concurrent.max count:20080906 [] 1
- ...
+ $ ./bin/accumulo shell -u username -p password
+ username@instance> table wordCount
+ username@instance wordCount> scan -b the
+ the count:20080906 [] 75
+ their count:20080906 [] 2
+ them count:20080906 [] 1
+ then count:20080906 [] 1
+ there count:20080906 [] 1
+ these count:20080906 [] 3
+ this count:20080906 [] 6
+ through count:20080906 [] 1
+ time count:20080906 [] 3
+ time. count:20080906 [] 1
+ to count:20080906 [] 27
+ total count:20080906 [] 1
+ tserver, count:20080906 [] 1
+ tserver.compaction.major.concurrent.max count:20080906 [] 1
+ ...
Modified: incubator/accumulo/branches/1.4/docs/examples/README.shard
URL: http://svn.apache.org/viewvc/incubator/accumulo/branches/1.4/docs/examples/README.shard?rev=1202406&r1=1202405&r2=1202406&view=diff
==============================================================================
--- incubator/accumulo/branches/1.4/docs/examples/README.shard (original)
+++ incubator/accumulo/branches/1.4/docs/examples/README.shard Tue Nov 15 20:16:26 2011
@@ -1,3 +1,21 @@
+Title: Shard Example
+Notice: Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ .
+ http://www.apache.org/licenses/LICENSE-2.0
+ .
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
Accumulo has in iterator called the intersecting iterator which supports querying a term index that is partitioned by
document, or "sharded". This example shows how to use the intersecting iterator through these four programs:
@@ -8,45 +26,37 @@ document, or "sharded". This example sho
To run these example programs, create two tables like below.
- Shell - Accumulo Interactive Shell
- -
- - version: 1.4.0-incubating-SNAPSHOT
- - instance name: test-instance
- - instance id: test-instance-id
- -
- - type 'help' for a list of available commands
- -
- foo@test-instance> createtable shard
- foo@test-instance shard> createtable doc2term
-
+ username@instance> createtable shard
+ username@instance shard> createtable doc2term
After creating the tables, index some files. The following command indexes all of the java files in the Accumulo source code.
- cd /local/user1/workspace/accumulo/
- find src -name "*.java" | xargs ./bin/accumulo org.apache.accumulo.examples.shard.Index test-instance localhost shard root password 30
+ $ cd /local/username/workspace/accumulo/
+ $ find src -name "*.java" | xargs ./bin/accumulo org.apache.accumulo.examples.shard.Index instance zookeepers shard username password 30
The following command queries the index to find all files containing 'foo' and 'bar'.
- ./bin/accumulo org.apache.accumulo.examples.shard.Query test-instance localhost shard root password foo bar
- /local/user1/workspace/accumulo/src/core/src/test/java/accumulo/core/security/ColumnVisibilityTest.java
- /local/user1/workspace/accumulo/src/core/src/test/java/accumulo/core/client/mock/MockConnectorTest.java
- /local/user1/workspace/accumulo/src/core/src/test/java/accumulo/core/security/VisibilityEvaluatorTest.java
- /local/user1/workspace/accumulo/src/server/src/main/java/accumulo/server/test/functional/RowDeleteTest.java
- /local/user1/workspace/accumulo/src/server/src/test/java/accumulo/server/logger/TestLogWriter.java
- /local/user1/workspace/accumulo/src/server/src/main/java/accumulo/server/test/functional/DeleteEverythingTest.java
- /local/user1/workspace/accumulo/src/core/src/test/java/accumulo/core/data/KeyExtentTest.java
- /local/user1/workspace/accumulo/src/server/src/test/java/accumulo/server/constraints/MetadataConstraintsTest.java
- /local/user1/workspace/accumulo/src/core/src/test/java/accumulo/core/iterators/WholeRowIteratorTest.java
- /local/user1/workspace/accumulo/src/server/src/test/java/accumulo/server/util/DefaultMapTest.java
- /local/user1/workspace/accumulo/src/server/src/test/java/accumulo/server/tabletserver/InMemoryMapTest.java
+ $ cd $ACCUMULO_HOME
+ $ ./bin/accumulo org.apache.accumulo.examples.shard.Query instance zookeepers shard username password foo bar
+ /local/username/workspace/accumulo/src/core/src/test/java/accumulo/core/security/ColumnVisibilityTest.java
+ /local/username/workspace/accumulo/src/core/src/test/java/accumulo/core/client/mock/MockConnectorTest.java
+ /local/username/workspace/accumulo/src/core/src/test/java/accumulo/core/security/VisibilityEvaluatorTest.java
+ /local/username/workspace/accumulo/src/server/src/main/java/accumulo/server/test/functional/RowDeleteTest.java
+ /local/username/workspace/accumulo/src/server/src/test/java/accumulo/server/logger/TestLogWriter.java
+ /local/username/workspace/accumulo/src/server/src/main/java/accumulo/server/test/functional/DeleteEverythingTest.java
+ /local/username/workspace/accumulo/src/core/src/test/java/accumulo/core/data/KeyExtentTest.java
+ /local/username/workspace/accumulo/src/server/src/test/java/accumulo/server/constraints/MetadataConstraintsTest.java
+ /local/username/workspace/accumulo/src/core/src/test/java/accumulo/core/iterators/WholeRowIteratorTest.java
+ /local/username/workspace/accumulo/src/server/src/test/java/accumulo/server/util/DefaultMapTest.java
+ /local/username/workspace/accumulo/src/server/src/test/java/accumulo/server/tabletserver/InMemoryMapTest.java
Inorder to run ContinuousQuery, we need to run Reverse.java to populate doc2term
- [user1@cloud1 accumulo]$ ./bin/accumulo org.apache.accumulo.examples.shard.Reverse test-instance localhost shard doc2term root password
+ $ ./bin/accumulo org.apache.accumulo.examples.shard.Reverse instance zookeepers shard doc2term username password
Below ContinuousQuery is run using 5 terms. So it selects 5 random terms from each document, then it continually randomly selects one set of 5 terms and queries. It prints the number of matching documents and the time in seconds.
- [user1@cloud1 accumulo]$ ./bin/accumulo org.apache.accumulo.examples.shard.ContinuousQuery test-instance localhost shard doc2term root password 5
+ $ ./bin/accumulo org.apache.accumulo.examples.shard.ContinuousQuery instance zookeepers shard doc2term username password 5
[public, core, class, binarycomparable, b] 2 0.081
[wordtodelete, unindexdocument, doctablename, putdelete, insert] 1 0.041
[import, columnvisibilityinterpreterfactory, illegalstateexception, cv, columnvisibility] 1 0.049
@@ -54,4 +64,3 @@ Below ContinuousQuery is run using 5 ter
[for, static, println, public, the] 55 0.211
[sleeptime, wrappingiterator, options, long, utilwaitthread] 1 0.057
[string, public, long, 0, wait] 12 0.132
-
Added: incubator/accumulo/trunk/docs/examples/README
URL: http://svn.apache.org/viewvc/incubator/accumulo/trunk/docs/examples/README?rev=1202406&view=auto
==============================================================================
--- incubator/accumulo/trunk/docs/examples/README (added)
+++ incubator/accumulo/trunk/docs/examples/README Tue Nov 15 20:16:26 2011
@@ -0,0 +1,36 @@
+Title: Examples
+Notice: Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ .
+ http://www.apache.org/licenses/LICENSE-2.0
+ .
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Each README in the examples directory highlights the use of particular features of Accumulo.
+
+Before running any of the examples, the following steps must be performed.
+
+1. Install and run Accumulo via the instructions found in $ACCUMULO_HOME/README.
+Remember the instance name. It will be referred to as "instance" throughout the examples.
+A comma-separated list of zookeeper servers will be referred to as "zookeepers".
+
+2. Create an Accumulo user (see the [user manual][1]), or use the root user.
+The Accumulo user name will be referred to as "username" with password "password" throughout the examples.
+
+In all commands, you will need to replace "instance", "zookeepers", "username", and "password" with the values you set for your Accumulo instance.
+
+Commands intended to be run in bash are prefixed by '$'. These are always assumed to be run from the $ACCUMULO_HOME directory.
+
+Commands intended to be run in the Accumulo shell are prefixed by '>'.
+
+[1]: /accumulo/user_manual_1.5-incubating/Accumulo_Shell.html#User_Administration
Modified: incubator/accumulo/trunk/docs/examples/README.aggregation
URL: http://svn.apache.org/viewvc/incubator/accumulo/trunk/docs/examples/README.aggregation?rev=1202406&r1=1202405&r2=1202406&view=diff
==============================================================================
--- incubator/accumulo/trunk/docs/examples/README.aggregation (original)
+++ incubator/accumulo/trunk/docs/examples/README.aggregation Tue Nov 15 20:16:26 2011
@@ -1,32 +1,50 @@
+Title: Aggregation Example
+Notice: Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ .
+ http://www.apache.org/licenses/LICENSE-2.0
+ .
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
This is a simple aggregation example. To build this example run maven and then
copy the produced jar into the accumulo lib dir. This is already done in the
tar distribution.
-
-$ bin/accumulo shell -u user
-Enter current password for 'user'@'example': ***
-Shell - Accumulo Interactive Shell
--
-- version: 1.2-SNAPSHOT
-- instance name: example
-- instance id: 42f24290-dbc3-42db-b72d-f6647b4cb988
--
-- type 'help' for a list of available commands
--
-user@example> createtable aggtest1 -a app=org.apache.accumulo.examples.aggregation.SortedSetAggregator
-user@example aggtest1> insert foo app 1 a
-user@example aggtest1> insert foo app 1 b
-user@example aggtest1> scan
-foo app:1 [] a,b
-user@example aggtest1> insert foo app 1 z,1,foo,w
-user@example aggtest1> scan
-foo app:1 [] 1,a,b,foo,w,z
-user@example aggtest1> insert foo app 2 cat,dog,muskrat
-user@example aggtest1> insert foo app 2 mouse,bird
-user@example aggtest1> scan
-foo app:1 [] 1,a,b,foo,w,z
-foo app:2 [] bird,cat,dog,mouse,muskrat
-user@example aggtest1>
+ $ bin/accumulo shell -u username
+ Enter current password for 'username'@'instance': ***
+
+ Shell - Accumulo Interactive Shell
+ -
+ - version: 1.5.x-incubating
+ - instance name: instance
+ - instance id: 00000000-0000-0000-0000-000000000000
+ -
+ - type 'help' for a list of available commands
+ -
+ username@instance> createtable aggtest1 -a app=org.apache.accumulo.examples.aggregation.SortedSetAggregator
+ username@instance aggtest1> insert foo app 1 a
+ username@instance aggtest1> insert foo app 1 b
+ username@instance aggtest1> scan
+ foo app:1 [] a,b
+ username@instance aggtest1> insert foo app 1 z,1,foo,w
+ username@instance aggtest1> scan
+ foo app:1 [] 1,a,b,foo,w,z
+ username@instance aggtest1> insert foo app 2 cat,dog,muskrat
+ username@instance aggtest1> insert foo app 2 mouse,bird
+ username@instance aggtest1> scan
+ foo app:1 [] 1,a,b,foo,w,z
+ foo app:2 [] bird,cat,dog,mouse,muskrat
+ username@instance aggtest1>
In this example a table is created and the example set aggregator is
applied to the column family app.
Modified: incubator/accumulo/trunk/docs/examples/README.batch
URL: http://svn.apache.org/viewvc/incubator/accumulo/trunk/docs/examples/README.batch?rev=1202406&r1=1202405&r2=1202406&view=diff
==============================================================================
--- incubator/accumulo/trunk/docs/examples/README.batch (original)
+++ incubator/accumulo/trunk/docs/examples/README.batch Tue Nov 15 20:16:26 2011
@@ -1,3 +1,21 @@
+Title: Batch Writing and Scanning Example
+Notice: Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ .
+ http://www.apache.org/licenses/LICENSE-2.0
+ .
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
This is an example of how to use the batch writer and batch scanner. To compile
the example, run maven and copy the produced jar into the accumulo lib dir.
This is already done in the tar distribution.
@@ -5,26 +23,31 @@ This is already done in the tar distribu
Below are commands that add 10000 entries to accumulo and then do 100 random
queries. The write command generates random 50 byte values.
-Be sure to use the name of your instance (given as testInstance here) and the appropriate
-list of zookeeper nodes (given as localhost here).
-
-Before you run this, you must ensure that the user you are running as has the
-"exampleVis" authorization. (you can set this in the shell with "setauths -u root -s exampleVis")
-
-You must also create the table, exampleTable, ahead of time. (In the shell, use "createtable exampleTable")
-
-
- $ ./bin/accumulo org.apache.accumulo.examples.client.SequentialBatchWriter testInstance localhost root secret exampleTable 0 10000 50 20000000 500 20 exampleVis
- $ ./bin/accumulo org.apache.accumulo.examples.client.RandomBatchScanner testInstance localhost root secret exampleTable 100 0 10000 50 20 exampleVis
- 07 11:33:11,103 [client.CountingVerifyingReceiver] INFO : Generating 100 random queries...
- 07 11:33:11,112 [client.CountingVerifyingReceiver] INFO : finished
- 07 11:33:11,260 [client.CountingVerifyingReceiver] INFO : 694.44 lookups/sec 0.14 secs
-
- 07 11:33:11,260 [client.CountingVerifyingReceiver] INFO : num results : 100
-
- 07 11:33:11,364 [client.CountingVerifyingReceiver] INFO : Generating 100 random queries...
- 07 11:33:11,370 [client.CountingVerifyingReceiver] INFO : finished
- 07 11:33:11,416 [client.CountingVerifyingReceiver] INFO : 2173.91 lookups/sec 0.05 secs
+Be sure to use the name of your instance (given as instance here) and the appropriate
+list of zookeeper nodes (given as zookeepers here).
- 07 11:33:11,416 [client.CountingVerifyingReceiver] INFO : num results : 100
+Before you run this, you must ensure that the user you are running has the
+"exampleVis" authorization. (you can set this in the shell with "setauths -u username -s exampleVis")
+ $ ./bin/accumulo shell -u root
+ > setauths -u username -s exampleVis
+ > exit
+
+You must also create the table, batchtest1, ahead of time. (In the shell, use "createtable batchtest1")
+
+ $ ./bin/accumulo shell -u username
+ > createtable batchtest1
+ > exit
+ $ ./bin/accumulo org.apache.accumulo.examples.client.SequentialBatchWriter instance zookeepers username password batchtest1 0 10000 50 20000000 500 20 exampleVis
+ $ ./bin/accumulo org.apache.accumulo.examples.client.RandomBatchScanner instance zookeepers username password batchtest1 100 0 10000 50 20 exampleVis
+ 07 11:33:11,103 [client.CountingVerifyingReceiver] INFO : Generating 100 random queries...
+ 07 11:33:11,112 [client.CountingVerifyingReceiver] INFO : finished
+ 07 11:33:11,260 [client.CountingVerifyingReceiver] INFO : 694.44 lookups/sec 0.14 secs
+
+ 07 11:33:11,260 [client.CountingVerifyingReceiver] INFO : num results : 100
+
+ 07 11:33:11,364 [client.CountingVerifyingReceiver] INFO : Generating 100 random queries...
+ 07 11:33:11,370 [client.CountingVerifyingReceiver] INFO : finished
+ 07 11:33:11,416 [client.CountingVerifyingReceiver] INFO : 2173.91 lookups/sec 0.05 secs
+
+ 07 11:33:11,416 [client.CountingVerifyingReceiver] INFO : num results : 100
Modified: incubator/accumulo/trunk/docs/examples/README.bloom
URL: http://svn.apache.org/viewvc/incubator/accumulo/trunk/docs/examples/README.bloom?rev=1202406&r1=1202405&r2=1202406&view=diff
==============================================================================
--- incubator/accumulo/trunk/docs/examples/README.bloom (original)
+++ incubator/accumulo/trunk/docs/examples/README.bloom Tue Nov 15 20:16:26 2011
@@ -1,39 +1,53 @@
+Title: Bloom Filter Example
+Notice: Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ .
+ http://www.apache.org/licenses/LICENSE-2.0
+ .
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
This example shows how to create a table with bloom filters enabled. It also
shows how bloom filters increase query performance when looking for values that
do not exist in a table.
Below table named bloom_test is created and bloom filters are enabled.
- [user@localhost bin]$ ./accumulo shell -u root -p secret
+ $ ./accumulo shell -u username -p password
Shell - Accumulo Interactive Shell
- - version: 1.2-SNAPSHOT
- - instance id: c4b72d26-16b3-408d-bf3e-c15a1fb51ba0
+ - version: 1.5.x-incubating
+ - instance name: instance
+ - instance id: 00000000-0000-0000-0000-000000000000
-
- type 'help' for a list of available commands
-
- root@localhost> setauths -u root -s CV1
- root@localhost> createtable bloom_test
- root@localhost bloom_test> config -t bloom_test -s table.bloom.enabled=true
+ username@instance> setauths -u username -s exampleVis
+ username@instance> createtable bloom_test
+ username@instance bloom_test> config -t bloom_test -s table.bloom.enabled=true
+ username@instance bloom_test> exit
Below 1 million random values are inserted into accumulo. The randomly
generated rows range between 0 and 1 billion. The random number generator is
initialized with the seed 7.
- [user@localhost bin]$ ../bin/accumulo org.apache.accumulo.examples.client.RandomBatchWriter -s 7 testInstance localhost root secret bloom_test 1000000 0 1000000000 50 2000000 60000 3 CV1
+ $ ./bin/accumulo org.apache.accumulo.examples.client.RandomBatchWriter -s 7 instance zookeepers username password bloom_test 1000000 0 1000000000 50 2000000 60000 3 exampleVis
Below the table is flushed, look at the monitor page and wait for the flush to
complete.
- [user@localhost bin]$ ./accumulo shell -u root -p secret
- Shell - Accumulo Interactive Shell
- - version: 1.2-SNAPSHOT
- - instance id: c4b72d26-16b3-408d-bf3e-c15a1fb51ba0
- -
- - type 'help' for a list of available commands
- -
- root@localhost> flush -t bloom_test
+ $ ./bin/accumulo shell -u username -p password
+ username@instance> flush -t bloom_test
Flush of table bloom_test initiated...
- root@localhost> quit
+ username@instance> exit
The flush will be finished when there are no entries in memory and the
number of minor compactions goes to zero. Refresh the page to see changes to the table.
@@ -42,9 +56,9 @@ After the flush completes, 500 random qu
same seed is used to generate the queries, therefore everything is found in the
table.
- [user@localhost bin]$ ../bin/accumulo org.apache.accumulo.examples.client.RandomBatchScanner -s 7 testInstance localhost root secret bloom_test 500 0 1000000000 50 20 CV1
+ $ ./bin/accumulo org.apache.accumulo.examples.client.RandomBatchScanner -s 7 instance zookeepers username password bloom_test 500 0 1000000000 50 20 exampleVis
Generating 500 random queries...finished
- 96.19 lookups/sec 5.20 secs
+ 96.19 lookups/sec 5.20 secs
num results : 500
Generating 500 random queries...finished
102.35 lookups/sec 4.89 secs
@@ -54,7 +68,7 @@ Below another 500 queries are performed,
in nothing being found. In this case the lookups are much faster because of
the bloom filters.
- [user@localhost bin]$ ../bin/accumulo org.apache.accumulo.examples.client.RandomBatchScanner -s 8 testInstance localhost root secret bloom_test 500 0 1000000000 50 20 CV1
+ $ ../bin/accumulo org.apache.accumulo.examples.client.RandomBatchScanner -s 8 instance zookeepers username password bloom_test 500 0 1000000000 50 20 exampleVis
Generating 500 random queries...finished
2212.39 lookups/sec 0.23 secs
num results : 0
@@ -66,7 +80,7 @@ the bloom filters.
********************************************************************************
-Bloom filters can also speed up lookups for entires that exist. In accumulo
+Bloom filters can also speed up lookups for entries that exist. In accumulo
data is divided into tablets and each tablet has multiple map files. Every
lookup in accumulo goes to a specific tablet where a lookup is done on each
map file in the tablet. So if a tablet has three map files, lookup performance
@@ -77,12 +91,12 @@ identical tables were created using the
filters, the other did not. Also the major compaction ratio was increased to
prevent the files from being compacted into one file.
- * Insert 1 million entires using RandomBatchWriter with a seed of 7
- * Flush the table using the shell
- * Insert 1 million entires using RandomBatchWriter with a seed of 8
- * Flush the table using the shell
- * Insert 1 million entires using RandomBatchWriter with a seed of 9
- * Flush the table using the shell
+ * Insert 1 million entries using RandomBatchWriter with a seed of 7
+ * Flush the table using the shell
+ * Insert 1 million entries using RandomBatchWriter with a seed of 8
+ * Flush the table using the shell
+ * Insert 1 million entries using RandomBatchWriter with a seed of 9
+ * Flush the table using the shell
After following the above steps, each table will have a tablet with three map
files. Each map file will contain 1 million entries generated with a different
@@ -92,26 +106,22 @@ Below 500 lookups are done against the t
NG seed 7. Even though only one map file will likely contain entries for this
seed, all map files will be interrogated.
- [user@localhost examples]$ ../bin/accumulo org.apache.accumulo.examples.client.RandomBatchScanner -s 7 testInstance localhost root secret test 500 0 1000000000 50 20 CV1
- 05 07:52:36,789 [conf.AccumuloConfiguration] WARN : accumulo-site.xml not found on classpath
+ $ ./bin/accumulo org.apache.accumulo.examples.client.RandomBatchScanner -s 7 instance zookeepers username password bloom_test1 500 0 1000000000 50 20 exampleVis
Generating 500 random queries...finished
- 35.09 lookups/sec 14.25 secs
+ 35.09 lookups/sec 14.25 secs
num results : 500
Generating 500 random queries...finished
- 35.33 lookups/sec 14.15 secs
+ 35.33 lookups/sec 14.15 secs
num results : 500
Below the same lookups are done against the table with bloom filters. The
lookups were 2.86 times faster because only one map file was used, even though three
map files existed.
- [user@localhost examples]$ ../bin/accumulo org.apache.accumulo.examples.client.RandomBatchScanner -s 7 testInstance localhost root secret bloom_test2 500 0 1000000000 50 20 CV1
- 05 07:53:15,650 [conf.AccumuloConfiguration] WARN : accumulo-site.xml not found on classpath
+ $ ./bin/accumulo org.apache.accumulo.examples.client.RandomBatchScanner -s 7 instance zookeepers username password bloom_test2 500 0 1000000000 50 20 exampleVis
Generating 500 random queries...finished
- 99.03 lookups/sec 5.05 secs
+ 99.03 lookups/sec 5.05 secs
num results : 500
Generating 500 random queries...finished
101.15 lookups/sec 4.94 secs
num results : 500
-
-
Modified: incubator/accumulo/trunk/docs/examples/README.bulkIngest
URL: http://svn.apache.org/viewvc/incubator/accumulo/trunk/docs/examples/README.bulkIngest?rev=1202406&r1=1202405&r2=1202406&view=diff
==============================================================================
--- incubator/accumulo/trunk/docs/examples/README.bulkIngest (original)
+++ incubator/accumulo/trunk/docs/examples/README.bulkIngest Tue Nov 15 20:16:26 2011
@@ -1,16 +1,29 @@
+Title: Bulk Ingest Example
+Notice: Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ .
+ http://www.apache.org/licenses/LICENSE-2.0
+ .
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
This is an example of how to bulk ingest data into accumulo using map reduce.
-Compile using Maven. Then copy the produced jar into the accumulo lib dir.
-
+
The following commands show how to run this example. This example creates a
table called test_bulk which has two initial split points. Then 1000 rows of
test data are created in HDFS. After that the 1000 rows are ingested into
accumulo. Then we verify the 1000 rows are in accumulo. The
-first two arguments to all of the commands except for GenerateTestData is the
-accumulo instance name, and a comman-separated list of zookeepers.
-
-NOTE: You should replace "root" with your username that has the appropriate
-permissions on the table, and "secret" with that user's password.
-$ACCUMULO_HOME should also be replaced with the home directory where accumulo is stored.
+first two arguments to all of the commands except for GenerateTestData are the
+accumulo instance name, and a comma-separated list of zookeepers.
$ ./bin/accumulo org.apache.accumulo.examples.mapreduce.bulk.SetupTable instance zookeepers username password test_bulk row_00000333 row_00000666
$ ./bin/accumulo org.apache.accumulo.examples.mapreduce.bulk.GenerateTestData 0 1000 bulk/test_1.txt
@@ -18,5 +31,4 @@ $ACCUMULO_HOME should also be replaced w
$ ./bin/tool.sh lib/accumulo-examples-*[^c].jar org.apache.accumulo.examples.mapreduce.bulk.BulkIngestExample instance zookeepers username password test_bulk bulk tmp/bulkWork
$ ./bin/accumulo org.apache.accumulo.examples.mapreduce.bulk.VerifyIngest instance zookeepers username password test_bulk 0 1000
-
For a high level discussion of bulk ingest, see the docs dir.
Modified: incubator/accumulo/trunk/docs/examples/README.constraints
URL: http://svn.apache.org/viewvc/incubator/accumulo/trunk/docs/examples/README.constraints?rev=1202406&r1=1202405&r2=1202406&view=diff
==============================================================================
--- incubator/accumulo/trunk/docs/examples/README.constraints (original)
+++ incubator/accumulo/trunk/docs/examples/README.constraints Tue Nov 15 20:16:26 2011
@@ -1,31 +1,48 @@
+Title: Constraints Example
+Notice: Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ .
+ http://www.apache.org/licenses/LICENSE-2.0
+ .
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
This an example of how to create a table with constraints. Below a table is
create with two example constraints. One constraints does not allow non alpha
numeric keys. The other constraint does not allow non numeric values. Two
inserts that violate these constraints are attempted and denied. The scan at
the end shows the inserts were not allowed.
-[user@localhost bin]$ ./accumulo shell -u user -p pass
-
- Shell - Accumulo Interactive Shell
- -
- - version: 1.5.0-incubating-SNAPSHOT
- - instance name: example
- - instance id: 00000000-0000-0000-0000-000000000000
- -
- - type 'help' for a list of available commands
- -
- user@example> createtable testConstraints
- user@example testConstraints> config -t testConstraints -s table.constraint.1=org.apache.accumulo.examples.constraints.NumericValueConstraint
- user@example testConstraints> config -t testConstraints -s table.constraint.2=org.apache.accumulo.examples.constraints.AlphaNumKeyConstraint
- user@example testConstraints> insert r1 cf1 cq1 1111
- user@example testConstraints> insert r1 cf1 cq1 ABC
+ $ ./bin/accumulo shell -u username -p pass
+
+ Shell - Accumulo Interactive Shell
+ -
+ - version: 1.5.x-incubating
+ - instance name: instance
+ - instance id: 00000000-0000-0000-0000-000000000000
+ -
+ - type 'help' for a list of available commands
+ -
+ username@instance> createtable testConstraints
+ username@instance testConstraints> config -t testConstraints -s table.constraint.1=org.apache.accumulo.examples.constraints.NumericValueConstraint
+ username@instance testConstraints> config -t testConstraints -s table.constraint.2=org.apache.accumulo.examples.constraints.AlphaNumKeyConstrain
+ username@instance testConstraints> insert r1 cf1 cq1 1111
+ username@instance testConstraints> insert r1 cf1 cq1 ABC
Constraint Failures:
ConstraintViolationSummary(constrainClass:org.apache.accumulo.examples.constraints.NumericValueConstraint, violationCode:1, violationDescription:Value is not numeric, numberOfViolatingMutations:1)
- user@example testConstraints> insert r1! cf1 cq1 ABC
+ username@instance testConstraints> insert r1! cf1 cq1 ABC
Constraint Failures:
ConstraintViolationSummary(constrainClass:org.apache.accumulo.examples.constraints.NumericValueConstraint, violationCode:1, violationDescription:Value is not numeric, numberOfViolatingMutations:1)
ConstraintViolationSummary(constrainClass:org.apache.accumulo.examples.constraints.AlphaNumKeyConstraint, violationCode:1, violationDescription:Row was not alpha numeric, numberOfViolatingMutations:1)
- user@example testConstraints> scan
- r1 cf1:cq1 [] 1111
- user@example testConstraints>
-
+ username@instance testConstraints> scan
+ r1 cf1:cq1 [] 1111
+ username@instance testConstraints>