You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@accumulo.apache.org by bi...@apache.org on 2011/10/31 22:40:45 UTC
svn commit: r1195687 [1/2] - in /incubator/accumulo:
branches/1.3/docs/examples/
site/trunk/content/accumulo/user_manual_1.3-incubating/
site/trunk/content/accumulo/user_manual_1.3-incubating/examples/
site/trunk/templates/
Author: billie
Date: Mon Oct 31 21:40:44 2011
New Revision: 1195687
URL: http://svn.apache.org/viewvc?rev=1195687&view=rev
Log:
ACCUMULO-103 for 1.3, converted to markdown and made all use the same parameters; added to site. have not yet finished checking for correctness.
Added:
incubator/accumulo/branches/1.3/docs/examples/README
incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/examples/
incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/examples.mdtext
incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/examples/aggregation.mdtext (with props)
incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/examples/batch.mdtext (with props)
incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/examples/bloom.mdtext (with props)
incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/examples/bulkIngest.mdtext (with props)
incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/examples/constraints.mdtext (with props)
incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/examples/dirlist.mdtext (with props)
incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/examples/filter.mdtext (with props)
incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/examples/helloworld.mdtext (with props)
incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/examples/mapred.mdtext (with props)
incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/examples/shard.mdtext (with props)
Modified:
incubator/accumulo/branches/1.3/docs/examples/README.aggregation
incubator/accumulo/branches/1.3/docs/examples/README.batch
incubator/accumulo/branches/1.3/docs/examples/README.bloom
incubator/accumulo/branches/1.3/docs/examples/README.bulkIngest
incubator/accumulo/branches/1.3/docs/examples/README.constraints
incubator/accumulo/branches/1.3/docs/examples/README.dirlist
incubator/accumulo/branches/1.3/docs/examples/README.filter
incubator/accumulo/branches/1.3/docs/examples/README.helloworld
incubator/accumulo/branches/1.3/docs/examples/README.mapred
incubator/accumulo/branches/1.3/docs/examples/README.shard
incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Administration.mdtext
incubator/accumulo/site/trunk/templates/sidenav.mdtext
Added: incubator/accumulo/branches/1.3/docs/examples/README
URL: http://svn.apache.org/viewvc/incubator/accumulo/branches/1.3/docs/examples/README?rev=1195687&view=auto
==============================================================================
--- incubator/accumulo/branches/1.3/docs/examples/README (added)
+++ incubator/accumulo/branches/1.3/docs/examples/README Mon Oct 31 21:40:44 2011
@@ -0,0 +1,36 @@
+Title: Examples
+Notice: Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ .
+ http://www.apache.org/licenses/LICENSE-2.0
+ .
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Each README in the examples directory highlights the use of particular features of Accumulo.
+
+Before running any of the examples, the following steps must be performed.
+
+1. Install and run Accumulo via the instructions found in $ACCUMULO_HOME/README.
+Remember the instance name. It will be referred to as "instance" throughout the examples.
+A comma-separated list of zookeeper servers will be referred to as "zookeepers".
+
+2. Create an Accumulo user (see the [user manual][1]), or use the root user.
+The Accumulo user name will be referred to as "username" with password "password" throughout the examples.
+
+In all commands, you will need to replace "instance", "zookeepers", "username", and "password" with the values you set for your Accumulo instance.
+
+Commands intended to be run in bash are prefixed by '$'. These are always assumed to be run from the $ACCUMULO_HOME directory.
+
+Commands intended to be run in the Accumulo shell are prefixed by '>'.
+
+[1]: /accumulo/user_manual_1.3-incubating/Accumulo_Shell.html#SECTION00430000000000000000
Modified: incubator/accumulo/branches/1.3/docs/examples/README.aggregation
URL: http://svn.apache.org/viewvc/incubator/accumulo/branches/1.3/docs/examples/README.aggregation?rev=1195687&r1=1195686&r2=1195687&view=diff
==============================================================================
--- incubator/accumulo/branches/1.3/docs/examples/README.aggregation (original)
+++ incubator/accumulo/branches/1.3/docs/examples/README.aggregation Mon Oct 31 21:40:44 2011
@@ -1,32 +1,50 @@
+Title: Aggregation Example
+Notice: Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ .
+ http://www.apache.org/licenses/LICENSE-2.0
+ .
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
This is a simple aggregation example. To build this example run maven and then
copy the produced jar into the accumulo lib dir. This is already done in the
tar distribution.
-
-$ bin/accumulo shell -u user
-Enter current password for 'user'@'example': ***
-Shell - Accumulo Interactive Shell
--
-- version: 1.2-SNAPSHOT
-- instance name: example
-- instance id: 42f24290-dbc3-42db-b72d-f6647b4cb988
--
-- type 'help' for a list of available commands
--
-user@example> createtable aggtest1 -a app=org.apache.accumulo.examples.aggregation.SortedSetAggregator
-user@example aggtest1> insert foo app 1 a
-user@example aggtest1> insert foo app 1 b
-user@example aggtest1> scan
-foo app:1 [] a,b
-user@example aggtest1> insert foo app 1 z,1,foo,w
-user@example aggtest1> scan
-foo app:1 [] 1,a,b,foo,w,z
-user@example aggtest1> insert foo app 2 cat,dog,muskrat
-user@example aggtest1> insert foo app 2 mouse,bird
-user@example aggtest1> scan
-foo app:1 [] 1,a,b,foo,w,z
-foo app:2 [] bird,cat,dog,mouse,muskrat
-user@example aggtest1>
+ $ bin/accumulo shell -u username
+ Enter current password for 'username'@'instance': ***
+
+ Shell - Accumulo Interactive Shell
+ -
+ - version: 1.3.x-incubating
+ - instance name: instance
+ - instance id: 00000000-0000-0000-0000-000000000000
+ -
+ - type 'help' for a list of available commands
+ -
+ username@instance> createtable aggtest1 -a app=org.apache.accumulo.examples.aggregation.SortedSetAggregator
+ username@instance aggtest1> insert foo app 1 a
+ username@instance aggtest1> insert foo app 1 b
+ username@instance aggtest1> scan
+ foo app:1 [] a,b
+ username@instance aggtest1> insert foo app 1 z,1,foo,w
+ username@instance aggtest1> scan
+ foo app:1 [] 1,a,b,foo,w,z
+ username@instance aggtest1> insert foo app 2 cat,dog,muskrat
+ username@instance aggtest1> insert foo app 2 mouse,bird
+ username@instance aggtest1> scan
+ foo app:1 [] 1,a,b,foo,w,z
+ foo app:2 [] bird,cat,dog,mouse,muskrat
+ username@instance aggtest1>
In this example a table is created and the example set aggregator is
applied to the column family app.
Modified: incubator/accumulo/branches/1.3/docs/examples/README.batch
URL: http://svn.apache.org/viewvc/incubator/accumulo/branches/1.3/docs/examples/README.batch?rev=1195687&r1=1195686&r2=1195687&view=diff
==============================================================================
--- incubator/accumulo/branches/1.3/docs/examples/README.batch (original)
+++ incubator/accumulo/branches/1.3/docs/examples/README.batch Mon Oct 31 21:40:44 2011
@@ -1,3 +1,21 @@
+Title: Batch Writing and Scanning Example
+Notice: Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ .
+ http://www.apache.org/licenses/LICENSE-2.0
+ .
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
This is an example of how to use the batch writer and batch scanner. To compile
the example, run maven and copy the produced jar into the accumulo lib dir.
This is already done in the tar distribution.
@@ -5,26 +23,31 @@ This is already done in the tar distribu
Below are commands that add 10000 entries to accumulo and then do 100 random
queries. The write command generates random 50 byte values.
-Be sure to use the name of your instance (given as testInstance here) and the appropriate
-list of zookeeper nodes (given as localhost here).
-
-Before you run this, you must ensure that the user you are running as has the
-"exampleVis" authorization. (you can set this in the shell with "setauths -u root -s exampleVis")
-
-You must also create the table, exampleTable, ahead of time. (In the shell, use "createtable exampleTable")
-
-
- $ ./bin/accumulo org.apache.accumulo.examples.client.SequentialBatchWriter testInstance localhost root secret exampleTable 0 10000 50 20000000 500 20 exampleVis
- $ ./bin/accumulo org.apache.accumulo.examples.client.RandomBatchScanner testInstance localhost root secret exampleTable 100 0 10000 50 20 exampleVis
- 07 11:33:11,103 [client.CountingVerifyingReceiver] INFO : Generating 100 random queries...
- 07 11:33:11,112 [client.CountingVerifyingReceiver] INFO : finished
- 07 11:33:11,260 [client.CountingVerifyingReceiver] INFO : 694.44 lookups/sec 0.14 secs
-
- 07 11:33:11,260 [client.CountingVerifyingReceiver] INFO : num results : 100
-
- 07 11:33:11,364 [client.CountingVerifyingReceiver] INFO : Generating 100 random queries...
- 07 11:33:11,370 [client.CountingVerifyingReceiver] INFO : finished
- 07 11:33:11,416 [client.CountingVerifyingReceiver] INFO : 2173.91 lookups/sec 0.05 secs
+Be sure to use the name of your instance (given as instance here) and the appropriate
+list of zookeeper nodes (given as zookeepers here).
- 07 11:33:11,416 [client.CountingVerifyingReceiver] INFO : num results : 100
+Before you run this, you must ensure that the user you are running has the
+"exampleVis" authorization. (you can set this in the shell with "setauths -u username -s exampleVis")
+ $ ./bin/accumulo shell -u root
+ > setauths -u username -s exampleVis
+ > exit
+
+You must also create the table, batchtest1, ahead of time. (In the shell, use "createtable batchtest1")
+
+ $ ./bin/accumulo shell -u username
+ > createtable batchtest1
+ > exit
+ $ ./bin/accumulo org.apache.accumulo.examples.client.SequentialBatchWriter instance zookeepers username password batchtest1 0 10000 50 20000000 500 20 exampleVis
+ $ ./bin/accumulo org.apache.accumulo.examples.client.RandomBatchScanner instance zookeepers username password batchtest1 100 0 10000 50 20 exampleVis
+ 07 11:33:11,103 [client.CountingVerifyingReceiver] INFO : Generating 100 random queries...
+ 07 11:33:11,112 [client.CountingVerifyingReceiver] INFO : finished
+ 07 11:33:11,260 [client.CountingVerifyingReceiver] INFO : 694.44 lookups/sec 0.14 secs
+
+ 07 11:33:11,260 [client.CountingVerifyingReceiver] INFO : num results : 100
+
+ 07 11:33:11,364 [client.CountingVerifyingReceiver] INFO : Generating 100 random queries...
+ 07 11:33:11,370 [client.CountingVerifyingReceiver] INFO : finished
+ 07 11:33:11,416 [client.CountingVerifyingReceiver] INFO : 2173.91 lookups/sec 0.05 secs
+
+ 07 11:33:11,416 [client.CountingVerifyingReceiver] INFO : num results : 100
Modified: incubator/accumulo/branches/1.3/docs/examples/README.bloom
URL: http://svn.apache.org/viewvc/incubator/accumulo/branches/1.3/docs/examples/README.bloom?rev=1195687&r1=1195686&r2=1195687&view=diff
==============================================================================
--- incubator/accumulo/branches/1.3/docs/examples/README.bloom (original)
+++ incubator/accumulo/branches/1.3/docs/examples/README.bloom Mon Oct 31 21:40:44 2011
@@ -1,47 +1,61 @@
+Title: Bloom Filter Example
+Notice: Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ .
+ http://www.apache.org/licenses/LICENSE-2.0
+ .
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
This example shows how to create a table with bloom filters enabled. It also
shows how bloom filters increase query performance when looking for values that
do not exist in a table.
Below table named bloom_test is created and bloom filters are enabled.
- [user@localhost bin]$ ./accumulo shell -u root -p secret
+ $ ./accumulo shell -u username -p password
Shell - Accumulo Interactive Shell
- - version: 1.2-SNAPSHOT
- - instance id: c4b72d26-16b3-408d-bf3e-c15a1fb51ba0
+ - version: 1.3.x-incubating
+ - instance name: instance
+ - instance id: 00000000-0000-0000-0000-000000000000
-
- type 'help' for a list of available commands
-
- root@localhost> setauths -u root -s CV1
- root@localhost> createtable bloom_test
- root@localhost bloom_test> config -t bloom_test -s table.bloom.enabled=true
+ username@instance> setauths -u username -s exampleVis
+ username@instance> createtable bloom_test
+ username@instance bloom_test> config -t bloom_test -s table.bloom.enabled=true
+ username@instance bloom_test> exit
Below 1 million random values are inserted into accumulo. The randomly
generated rows range between 0 and 1 billion. The random number generator is
initialized with the seed 7.
- [user@localhost bin]$ ../bin/accumulo org.apache.accumulo.examples.client.RandomBatchWriter -s 7 testInstance localhost root secret bloom_test 1000000 0 1000000000 50 2000000 60000 3 CV1
+ $ ./bin/accumulo org.apache.accumulo.examples.client.RandomBatchWriter -s 7 instance zookeepers username password bloom_test 1000000 0 1000000000 50 2000000 60000 3 exampleVis
Below the table is flushed, look at the monitor page and wait for the flush to
complete.
- [user@localhost bin]$ ./accumulo shell -u root -p secret
- Shell - Accumulo Interactive Shell
- - version: 1.2-SNAPSHOT
- - instance id: c4b72d26-16b3-408d-bf3e-c15a1fb51ba0
- -
- - type 'help' for a list of available commands
- -
- root@localhost> flush -t bloom_test
+ $ ./bin/accumulo shell -u username -p password
+ username@instance> flush -t bloom_test
Flush of table bloom_test initiated...
- root@localhost> quit
+ username@instance> exit
After the flush completes, 500 random queries are done against the table. The
same seed is used to generate the queries, therefore everything is found in the
table.
- [user@localhost bin]$ ../bin/accumulo org.apache.accumulo.examples.client.RandomBatchScanner -s 7 testInstance localhost root secret bloom_test 500 0 1000000000 50 20 CV1
+ $ ./bin/accumulo org.apache.accumulo.examples.client.RandomBatchScanner -s 7 instance zookeepers username password bloom_test 500 0 1000000000 50 20 exampleVis
Generating 500 random queries...finished
- 96.19 lookups/sec 5.20 secs
+ 96.19 lookups/sec 5.20 secs
num results : 500
Generating 500 random queries...finished
102.35 lookups/sec 4.89 secs
@@ -51,7 +65,7 @@ Below another 500 queries are performed,
in nothing being found. In this case the lookups are much faster because of
the bloom filters.
- [user@localhost bin]$ ../bin/accumulo org.apache.accumulo.examples.client.RandomBatchScanner -s 8 testInstance localhost root secret bloom_test 500 0 1000000000 50 20 CV1
+ $ ../bin/accumulo org.apache.accumulo.examples.client.RandomBatchScanner -s 8 instance zookeepers username password bloom_test 500 0 1000000000 50 20 exampleVis
Generating 500 random queries...finished
2212.39 lookups/sec 0.23 secs
num results : 0
@@ -63,7 +77,7 @@ the bloom filters.
********************************************************************************
-Bloom filters can also speed up lookups for entires that exist. In accumulo
+Bloom filters can also speed up lookups for entries that exist. In accumulo
data is divided into tablets and each tablet has multiple map files. Every
lookup in accumulo goes to a specific tablet where a lookup is done on each
map file in the tablet. So if a tablet has three map files, lookup performance
@@ -74,12 +88,12 @@ identical tables were created using the
filters, the other did not. Also the major compaction ratio was increased to
prevent the files from being compacted into one file.
- * Insert 1 million entires using RandomBatchWriter with a seed of 7
- * Flush the table using the shell
- * Insert 1 million entires using RandomBatchWriter with a seed of 8
- * Flush the table using the shell
- * Insert 1 million entires using RandomBatchWriter with a seed of 9
- * Flush the table using the shell
+ * Insert 1 million entries using RandomBatchWriter with a seed of 7
+ * Flush the table using the shell
+ * Insert 1 million entries using RandomBatchWriter with a seed of 8
+ * Flush the table using the shell
+ * Insert 1 million entries using RandomBatchWriter with a seed of 9
+ * Flush the table using the shell
After following the above steps, each table will have a tablet with three map
files. Each map file will contain 1 million entries generated with a different
@@ -89,26 +103,22 @@ Below 500 lookups are done against the t
NG seed 7. Even though only one map file will likely contain entries for this
seed, all map files will be interrogated.
- [user@localhost examples]$ ../bin/accumulo org.apache.accumulo.examples.client.RandomBatchScanner -s 7 testInstance localhost root secret test 500 0 1000000000 50 20 CV1
- 05 07:52:36,789 [conf.AccumuloConfiguration] WARN : accumulo-site.xml not found on classpath
+ $ ./bin/accumulo org.apache.accumulo.examples.client.RandomBatchScanner -s 7 instance zookeepers username password bloom_test1 500 0 1000000000 50 20 exampleVis
Generating 500 random queries...finished
- 35.09 lookups/sec 14.25 secs
+ 35.09 lookups/sec 14.25 secs
num results : 500
Generating 500 random queries...finished
- 35.33 lookups/sec 14.15 secs
+ 35.33 lookups/sec 14.15 secs
num results : 500
Below the same lookups are done against the table with bloom filters. The
lookups were 2.86 times faster because only one map file was used, even though three
map files existed.
- [user@localhost examples]$ ../bin/accumulo org.apache.accumulo.examples.client.RandomBatchScanner -s 7 testInstance localhost root secret bloom_test2 500 0 1000000000 50 20 CV1
- 05 07:53:15,650 [conf.AccumuloConfiguration] WARN : accumulo-site.xml not found on classpath
+ $ ./bin/accumulo org.apache.accumulo.examples.client.RandomBatchScanner -s 7 instance zookeepers username password bloom_test2 500 0 1000000000 50 20 exampleVis
Generating 500 random queries...finished
- 99.03 lookups/sec 5.05 secs
+ 99.03 lookups/sec 5.05 secs
num results : 500
Generating 500 random queries...finished
101.15 lookups/sec 4.94 secs
num results : 500
-
-
Modified: incubator/accumulo/branches/1.3/docs/examples/README.bulkIngest
URL: http://svn.apache.org/viewvc/incubator/accumulo/branches/1.3/docs/examples/README.bulkIngest?rev=1195687&r1=1195686&r2=1195687&view=diff
==============================================================================
--- incubator/accumulo/branches/1.3/docs/examples/README.bulkIngest (original)
+++ incubator/accumulo/branches/1.3/docs/examples/README.bulkIngest Mon Oct 31 21:40:44 2011
@@ -1,20 +1,34 @@
+Title: Bulk Ingest Example
+Notice: Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ .
+ http://www.apache.org/licenses/LICENSE-2.0
+ .
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
This is an example of how to bulk ingest data into accumulo using map reduce.
-Compile using Maven. Then copy the produced jar into the accumulo lib dir.
-
+
The following commands show how to run this example. This example creates a
table called test_bulk which has two initial split points. Then 1000 rows of
test data are created in HDFS. After that the 1000 rows are ingested into
accumulo. Then we verify the 1000 rows are in accumulo. The
-first two arguments to all of the commands except for GenerateTestData is the
-accumulo instance name, and a comman-separated list of zookeepers.
-
-NOTE: You should replace "root" with your username that has the appropriate
-permissions on the table, and "secret" with that user's password
+first two arguments to all of the commands except for GenerateTestData are the
+accumulo instance name, and a comma-separated list of zookeepers.
- bin/accumulo org.apache.accumulo.examples.mapreduce.bulk.SetupTable testInstance localhost root secret test_bulk row_00000333 row_00000666
- bin/accumulo org.apache.accumulo.examples.mapreduce.bulk.GenerateTestData 0 1000 bulk/test_1.txt
-
- bin/tool.sh $ACCUMULO_HOME/lib/accumulo-examples-*.jar org.apache.accumulo.examples.mapreduce.bulk.BulkIngestExample testInstance localhost root secret test_bulk bulk tmp/bulkWork
- bin/accumulo org.apache.accumulo.examples.mapreduce.bulk.VerifyIngest testInstance localhost root secret test_bulk 0 1000
+ $ ./bin/accumulo org.apache.accumulo.examples.mapreduce.bulk.SetupTable instance zookeepers username password test_bulk row_00000333 row_00000666
+ $ ./bin/accumulo org.apache.accumulo.examples.mapreduce.bulk.GenerateTestData 0 1000 bulk/test_1.txt
+
+ $ ./bin/tool.sh lib/accumulo-examples-*.jar org.apache.accumulo.examples.mapreduce.bulk.BulkIngestExample instance zookeepers username password test_bulk bulk tmp/bulkWork
+ $ ./bin/accumulo org.apache.accumulo.examples.mapreduce.bulk.VerifyIngest instance zookeepers username password test_bulk 0 1000
For a high level discussion of bulk ingest, see the docs dir.
Modified: incubator/accumulo/branches/1.3/docs/examples/README.constraints
URL: http://svn.apache.org/viewvc/incubator/accumulo/branches/1.3/docs/examples/README.constraints?rev=1195687&r1=1195686&r2=1195687&view=diff
==============================================================================
--- incubator/accumulo/branches/1.3/docs/examples/README.constraints (original)
+++ incubator/accumulo/branches/1.3/docs/examples/README.constraints Mon Oct 31 21:40:44 2011
@@ -1,31 +1,48 @@
+Title: Constraints Example
+Notice: Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ .
+ http://www.apache.org/licenses/LICENSE-2.0
+ .
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
This an example of how to create a table with constraints. Below a table is
create with two example constraints. One constraints does not allow non alpha
numeric keys. The other constraint does not allow non numeric values. Two
inserts that violate these constraints are attempted and denied. The scan at
the end shows the inserts were not allowed.
-[user@localhost bin]$ ./accumulo shell -u user -p pass
-
- Shell - Accumulo Interactive Shell
- -
- - version: 1.3.5-incubating-SNAPSHOT
- - instance name: example
- - instance id: 00000000-0000-0000-0000-000000000000
- -
- - type 'help' for a list of available commands
- -
- user@example> createtable testConstraints
- user@example testConstraints> config -t testConstraints -s table.constraint.1=org.apache.accumulo.examples.constraints.NumericValueConstraint
- user@example testConstraints> config -t testConstraints -s table.constraint.2=org.apache.accumulo.examples.constraints.AlphaNumKeyConstrain
- user@example testConstraints> insert r1 cf1 cq1 1111
- user@example testConstraints> insert r1 cf1 cq1 ABC
+ $ ./bin/accumulo shell -u username -p pass
+
+ Shell - Accumulo Interactive Shell
+ -
+ - version: 1.3.x-incubating
+ - instance name: instance
+ - instance id: 00000000-0000-0000-0000-000000000000
+ -
+ - type 'help' for a list of available commands
+ -
+ username@instance> createtable testConstraints
+ username@instance testConstraints> config -t testConstraints -s table.constraint.1=org.apache.accumulo.examples.constraints.NumericValueConstraint
+ username@instance testConstraints> config -t testConstraints -s table.constraint.2=org.apache.accumulo.examples.constraints.AlphaNumKeyConstrain
+ username@instance testConstraints> insert r1 cf1 cq1 1111
+ username@instance testConstraints> insert r1 cf1 cq1 ABC
Constraint Failures:
ConstraintViolationSummary(constrainClass:org.apache.accumulo.examples.constraints.NumericValueConstraint, violationCode:1, violationDescription:Value is not numeric, numberOfViolatingMutations:1)
- user@example testConstraints> insert r1! cf1 cq1 ABC
+ username@instance testConstraints> insert r1! cf1 cq1 ABC
Constraint Failures:
ConstraintViolationSummary(constrainClass:org.apache.accumulo.examples.constraints.NumericValueConstraint, violationCode:1, violationDescription:Value is not numeric, numberOfViolatingMutations:1)
ConstraintViolationSummary(constrainClass:org.apache.accumulo.examples.constraints.AlphaNumKeyConstraint, violationCode:1, violationDescription:Row was not alpha numeric, numberOfViolatingMutations:1)
- user@example testConstraints> scan
- r1 cf1:cq1 [] 1111
- user@example testConstraints>
-
+ username@instance testConstraints> scan
+ r1 cf1:cq1 [] 1111
+ username@instance testConstraints>
Modified: incubator/accumulo/branches/1.3/docs/examples/README.dirlist
URL: http://svn.apache.org/viewvc/incubator/accumulo/branches/1.3/docs/examples/README.dirlist?rev=1195687&r1=1195686&r2=1195687&view=diff
==============================================================================
--- incubator/accumulo/branches/1.3/docs/examples/README.dirlist (original)
+++ incubator/accumulo/branches/1.3/docs/examples/README.dirlist Mon Oct 31 21:40:44 2011
@@ -1,3 +1,21 @@
+Title: File System Archive
+Notice: Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ .
+ http://www.apache.org/licenses/LICENSE-2.0
+ .
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
This example shows how to use Accumulo to store a file system history. It has three classes:
* Ingest.java - Recursively lists the files and directories under a given path, ingests their names and file info (not the file data!) into a Accumulo table, and indexes the file names in a separate table.
@@ -9,31 +27,31 @@ This example shows how to use Accumulo t
To begin, ingest some data with Ingest.java.
- bin/accumulo org.apache.accumulo.examples.dirlist.Ingest test-instance localhost root password direxample dirindex visibility /local/user1/workspace
-
+ $ ./bin/accumulo org.apache.accumulo.examples.dirlist.Ingest instance zookeepers username password direxample dirindex exampleVis /local/user1/workspace
+
Note that running this example will create tables direxample and dirindex in Accumulo that you should delete when you have completed the example.
If you modify a file or add new files in the directory ingested (e.g. /local/user1/workspace), you can run Ingest again to add new information into the Accumulo tables.
-To browse the data ingested, use Viewer.java. Be sure to give the root user the authorizations to see the data.
+To browse the data ingested, use Viewer.java. Be sure to give the "username" user the authorizations to see the data.
- bin/accumulo org.apache.accumulo.examples.dirlist.Viewer test-instance localhost root password direxample auths /local/user1/workspace
+ $ ./bin/accumulo org.apache.accumulo.examples.dirlist.Viewer instance zookeepers username password direxample exampleVis /local/user1/workspace
To list the contents of specific directories, use QueryUtil.java.
- bin/accumulo org.apache.accumulo.examples.dirlist.QueryUtil test-instance localhost root password direxample auths /local/user1
- bin/accumulo org.apache.accumulo.examples.dirlist.QueryUtil test-instance localhost root password direxample auths /local/user1/workspace
+ $ ./bin/accumulo org.apache.accumulo.examples.dirlist.QueryUtil instance zookeepers username password direxample exampleVis /local/user1
+ $ ./bin/accumulo org.apache.accumulo.examples.dirlist.QueryUtil instance zookeepers username password direxample exampleVis /local/user1/workspace
To perform searches on file or directory names, also use QueryUtil.java. Search terms must contain no more than one wild card and cannot contain "/".
Note these queries run on the dirindex table instead of the direxample table.
- bin/accumulo org.apache.accumulo.examples.dirlist.QueryUtil test-instance localhost root password dirindex auths filename -search
- bin/accumulo org.apache.accumulo.examples.dirlist.QueryUtil test-instance localhost root password dirindex auths filename* -search
- bin/accumulo org.apache.accumulo.examples.dirlist.QueryUtil test-instance localhost root password dirindex auths *jar -search
- bin/accumulo org.apache.accumulo.examples.dirlist.QueryUtil test-instance localhost root password dirindex auths filename*jar -search
+ $ ./bin/accumulo org.apache.accumulo.examples.dirlist.QueryUtil instance zookeepers username password dirindex exampleVis filename -search
+ $ ./bin/accumulo org.apache.accumulo.examples.dirlist.QueryUtil instance zookeepers username password dirindex exampleVis filename* -search
+ $ ./bin/accumulo org.apache.accumulo.examples.dirlist.QueryUtil instance zookeepers username password dirindex exampleVis *jar -search
+ $ ./bin/accumulo org.apache.accumulo.examples.dirlist.QueryUtil instance zookeepers username password dirindex exampleVis filename*jar -search
To count the number of direct children (directories and files) and descendants (children and children's descendents, directories and files), run the FileCountMR over the direxample table.
The results can be written back to the same table.
- bin/tool.sh lib/accumulo-examples-*.jar org.apache.accumulo.examples.dirlist.FileCountMR test-instance localhost root password direxample direxample visibility visibility
+ $ ./bin/tool.sh lib/accumulo-examples-*.jar org.apache.accumulo.examples.dirlist.FileCountMR instance zookeepers username password direxample direxample exampleVis exampleVis
-Alternatively, you can also run FileCount.java.
+Alternatively, you can also run FileCount.java.
Modified: incubator/accumulo/branches/1.3/docs/examples/README.filter
URL: http://svn.apache.org/viewvc/incubator/accumulo/branches/1.3/docs/examples/README.filter?rev=1195687&r1=1195686&r2=1195687&view=diff
==============================================================================
--- incubator/accumulo/branches/1.3/docs/examples/README.filter (original)
+++ incubator/accumulo/branches/1.3/docs/examples/README.filter Mon Oct 31 21:40:44 2011
@@ -1,3 +1,21 @@
+Title: Filter Example
+Notice: Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ .
+ http://www.apache.org/licenses/LICENSE-2.0
+ .
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
This is a simple filter example. It uses the AgeOffFilter that is provided as
part of the core package org.apache.accumulo.core.iterators.filter. Filters are used by
the FilteringIterator to select desired key/value pairs (or weed out undesired
@@ -5,27 +23,27 @@ ones). Filters implement the org.apache
contains a method accept(Key k, Value v). This method returns true if the key,
value pair are to be delivered and false if they are to be ignored.
-user@example> createtable filtertest
-user@example filtertest> setiter -t filtertest -scan -p 10 -n myfilter -filter
-FilteringIterator uses Filters to accept or reject key/value pairs
-----------> entering options: <filterPriorityNumber> <ageoff|regex|filterClass>
-----------> set org.apache.accumulo.core.iterators.FilteringIterator option (<name> <value>, hit enter to skip): 0 ageoff
-----------> set org.apache.accumulo.core.iterators.FilteringIterator option (<name> <value>, hit enter to skip):
-AgeOffFilter removes entries with timestamps more than <ttl> milliseconds old
-----------> set org.apache.accumulo.core.iterators.filter.AgeOffFilter parameter currentTime, if set, use the given value as the absolute time in milliseconds as the current time of day:
-----------> set org.apache.accumulo.core.iterators.filter.AgeOffFilter parameter ttl, time to live (milliseconds): 30000
-user@example filtertest>
-
-user@example filtertest> scan
-user@example filtertest> insert foo a b c
-insert successful
-user@example filtertest> scan
-foo a:b [] c
-
+ username@instance> createtable filtertest
+ username@instance filtertest> setiter -t filtertest -scan -p 10 -n myfilter -filter
+ FilteringIterator uses Filters to accept or reject key/value pairs
+ ----------> entering options: <filterPriorityNumber> <ageoff|regex|filterClass>
+ ----------> set org.apache.accumulo.core.iterators.FilteringIterator option (<name> <value>, hit enter to skip): 0 ageoff
+ ----------> set org.apache.accumulo.core.iterators.FilteringIterator option (<name> <value>, hit enter to skip):
+ AgeOffFilter removes entries with timestamps more than <ttl> milliseconds old
+ ----------> set org.apache.accumulo.core.iterators.filter.AgeOffFilter parameter currentTime, if set, use the given value as the absolute time in milliseconds as the current time of day:
+ ----------> set org.apache.accumulo.core.iterators.filter.AgeOffFilter parameter ttl, time to live (milliseconds): 30000
+ username@instance filtertest>
+
+ username@instance filtertest> scan
+ username@instance filtertest> insert foo a b c
+ insert successful
+ username@instance filtertest> scan
+ foo a:b [] c
+
... wait 30 seconds ...
-
-user@example filtertest> scan
-user@example filtertest>
+
+ username@instance filtertest> scan
+ username@instance filtertest>
Note the absence of the entry inserted more than 30 seconds ago. Since the
scope was set to "scan", this means the entry is still in Accumulo, but is
@@ -38,31 +56,32 @@ on the "minc" and "majc" scopes you can
happen automatically as a background operation on any table that is being
actively written to, but these are the commands to force compaction:
-user@example filtertest> flush -t filtertest
-08 11:13:55,745 [shell.Shell] INFO : Flush of table filtertest initiated...
-user@example filtertest> compact -t filtertest
-08 11:14:10,800 [shell.Shell] INFO : Compaction of table filtertest scheduled for 20110208111410EST
-user@example filtertest>
+ username@instance filtertest> flush -t filtertest
+ 08 11:13:55,745 [shell.Shell] INFO : Flush of table filtertest initiated...
+ username@instance filtertest> compact -t filtertest
+ 08 11:14:10,800 [shell.Shell] INFO : Compaction of table filtertest scheduled for 20110208111410EST
+ username@instance filtertest>
After the compaction runs, the newly created files will not contain any data that should be aged off, and the
Accumulo garbage collector will remove the old files.
To see the iterator settings for a table, use:
-user@example filtertest> config -t filtertest -f iterator
----------+------------------------------------------+----------------------------------------------------------
-SCOPE | NAME | VALUE
----------+------------------------------------------+----------------------------------------------------------
-table | table.iterator.majc.vers................ | 20,org.apache.accumulo.core.iterators.VersioningIterator
-table | table.iterator.majc.vers.opt.maxVersions | 1
-table | table.iterator.minc.vers................ | 20,org.apache.accumulo.core.iterators.VersioningIterator
-table | table.iterator.minc.vers.opt.maxVersions | 1
-table | table.iterator.scan.myfilter............ | 10,org.apache.accumulo.core.iterators.FilteringIterator
-table | table.iterator.scan.myfilter.opt.0...... | org.apache.accumulo.core.iterators.filter.AgeOffFilter
-table | table.iterator.scan.myfilter.opt.0.ttl.. | 30000
-table | table.iterator.scan.vers................ | 20,org.apache.accumulo.core.iterators.VersioningIterator
-table | table.iterator.scan.vers.opt.maxVersions | 1
----------+------------------------------------------+----------------------------------------------------------
-user@example filtertest>
+
+ username@instance filtertest> config -t filtertest -f iterator
+ ---------+------------------------------------------+----------------------------------------------------------
+ SCOPE | NAME | VALUE
+ ---------+------------------------------------------+----------------------------------------------------------
+ table | table.iterator.majc.vers................ | 20,org.apache.accumulo.core.iterators.VersioningIterator
+ table | table.iterator.majc.vers.opt.maxVersions | 1
+ table | table.iterator.minc.vers................ | 20,org.apache.accumulo.core.iterators.VersioningIterator
+ table | table.iterator.minc.vers.opt.maxVersions | 1
+ table | table.iterator.scan.myfilter............ | 10,org.apache.accumulo.core.iterators.FilteringIterator
+ table | table.iterator.scan.myfilter.opt.0...... | org.apache.accumulo.core.iterators.filter.AgeOffFilter
+ table | table.iterator.scan.myfilter.opt.0.ttl.. | 30000
+ table | table.iterator.scan.vers................ | 20,org.apache.accumulo.core.iterators.VersioningIterator
+ table | table.iterator.scan.vers.opt.maxVersions | 1
+ ---------+------------------------------------------+----------------------------------------------------------
+ username@instance filtertest>
If you would like to apply multiple filters, this can be done using a single
iterator. Just continue adding entries during the
Modified: incubator/accumulo/branches/1.3/docs/examples/README.helloworld
URL: http://svn.apache.org/viewvc/incubator/accumulo/branches/1.3/docs/examples/README.helloworld?rev=1195687&r1=1195686&r2=1195687&view=diff
==============================================================================
--- incubator/accumulo/branches/1.3/docs/examples/README.helloworld (original)
+++ incubator/accumulo/branches/1.3/docs/examples/README.helloworld Mon Oct 31 21:40:44 2011
@@ -1,46 +1,52 @@
-
-This tutorial uses the following Java classes, which can be found in src/examples/src/main/java/accumulo/examples/helloworld:
-
-InsertWithBatchWriter.java - Inserts 10K rows (50K entries) into accumulo as each row has 5 entries
-InsertWithOutputFormat.java - Example of inserting data in MapReduce
-ReadData.java - Reads all data between two rows
-
-NOTE: Feel free to use your own username, password, instance name, and table name.
-The ones used below are just examples.
-
-Initialize accumulo, create an instance called 'testInstance' and set the root password to 'pass':
-
- bin/accumulo init
-
-Start accumulo:
-
- bin/start-all.sh
+Title: Hello World Example
+Notice: Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ .
+ http://www.apache.org/licenses/LICENSE-2.0
+ .
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+This tutorial uses the following Java classes, which can be found in org.apache.accumulo.examples.helloworld in the accumulo-examples module:
+
+ * InsertWithBatchWriter.java - Inserts 10K rows (50K entries) into accumulo with each row having 5 entries
+ * InsertWithOutputFormat.java - Example of inserting data in MapReduce
+ * ReadData.java - Reads all data between two rows
Log into the accumulo shell:
- bin/accumulo shell -u root -p pass
+ $ ./bin/accumulo shell -u username -p password
Create a table called 'hellotable':
- root@testInstance> createtable hellotable
+ username@instance> createtable hellotable
Launch a Java program that inserts data with a BatchWriter:
- bin/accumulo org.apache.accumulo.examples.helloworld.InsertWithBatchWriter testInstance localhost hellotable root pass
-
+ $ ./bin/accumulo org.apache.accumulo.examples.helloworld.InsertWithBatchWriter instance zookeepers hellotable username password
+
Alternatively, the same data can be inserted using MapReduce writers:
- bin/accumulo org.apache.accumulo.examples.helloworld.InsertWithOutputFormat testInstance localhost hellotable root pass
-
-On accumulo status page at the URL below, you should see 50K entries
+ $ ./bin/accumulo org.apache.accumulo.examples.helloworld.InsertWithOutputFormat instance zookeepers hellotable username password
+
+On the accumulo status page at the URL below (where 'master' is replaced with the name or IP of your accumulo master), you should see 50K entries
- http://localhost:50095/
+ http://master:50095/
To view the entries, use the shell to scan the table:
- root@testInstance> table hellotable
- root@testInstance hellotable> scan
+ username@instance> table hellotable
+ username@instance hellotable> scan
You can also use a Java class to scan the table:
- bin/accumulo org.apache.accumulo.examples.helloworld.ReadData testInstance localhost hellotable root pass row_0 row_1001
+ $ ./bin/accumulo org.apache.accumulo.examples.helloworld.ReadData instance zookeepers hellotable username password row_0 row_1001
Modified: incubator/accumulo/branches/1.3/docs/examples/README.mapred
URL: http://svn.apache.org/viewvc/incubator/accumulo/branches/1.3/docs/examples/README.mapred?rev=1195687&r1=1195686&r2=1195687&view=diff
==============================================================================
--- incubator/accumulo/branches/1.3/docs/examples/README.mapred (original)
+++ incubator/accumulo/branches/1.3/docs/examples/README.mapred Mon Oct 31 21:40:44 2011
@@ -1,3 +1,21 @@
+Title: MapReduce Example
+Notice: Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ .
+ http://www.apache.org/licenses/LICENSE-2.0
+ .
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
This example uses mapreduce and accumulo to compute word counts for a set of
documents. This is accomplished using a map only map reduce job and a
accumulo table with aggregators.
@@ -5,70 +23,63 @@ accumulo table with aggregators.
To run this example you will need a directory in HDFS containing text files.
The accumulo readme will be used to show how to run this example.
- [user1@localhost accumulo]$ hadoop fs -copyFromLocal $ACCUMULO_HOME/README /user/user1/wc/Accumulo.README
- [user1@localhost accumulo]$ hadoop fs -ls /user/user1/wc
- Found 1 items
- -rw-r--r-- 2 user1 supergroup 9359 2009-07-15 17:54 /user/user1/wc/Accumulo.README
+ $ hadoop fs -copyFromLocal $ACCUMULO_HOME/README /user/username/wc/Accumulo.README
+ $ hadoop fs -ls /user/username/wc
+ Found 1 items
+ -rw-r--r-- 2 username supergroup 9359 2009-07-15 17:54 /user/username/wc/Accumulo.README
The first part of running this example is to create a table with aggregation
for the column family count.
- [user1@localhost accumulo]$ ./bin/accumulo shell -u root -p secret
- Shell - Accumulo Interactive Shell
- - version: 1.3.5-incubating-SNAPSHOT
- - instance id: 00000000-0000-0000-0000-000000000000
- -
- - type 'help' for a list of available commands
- -
- root@localhost> createtable wordCount -a count=org.apache.accumulo.core.iterators.aggregation.StringSummation
- root@localhost wordCount> quit
+ $ ./bin/accumulo shell -u username -p password
+ Shell - Accumulo Interactive Shell
+ - version: 1.3.x-incubating
+ - instance name: instance
+ - instance id: 00000000-0000-0000-0000-000000000000
+ -
+ - type 'help' for a list of available commands
+ -
+ username@instance> createtable wordCount -a count=org.apache.accumulo.core.iterators.aggregation.StringSummation
+ username@instance wordCount> quit
-
After creating the table, run the word count map reduce job.
- [user1@localhost accumulo]$ bin/tool.sh lib/accumulo-examples-*.jar org.apache.accumulo.examples.mapreduce.WordCount testInstance localhost /user/user1/wc wordCount
-
- 11/02/07 18:20:11 INFO input.FileInputFormat: Total input paths to process : 1
- 11/02/07 18:20:12 INFO mapred.JobClient: Running job: job_201102071740_0003
- 11/02/07 18:20:13 INFO mapred.JobClient: map 0% reduce 0%
- 11/02/07 18:20:20 INFO mapred.JobClient: map 100% reduce 0%
- 11/02/07 18:20:22 INFO mapred.JobClient: Job complete: job_201102071740_0003
- 11/02/07 18:20:22 INFO mapred.JobClient: Counters: 6
- 11/02/07 18:20:22 INFO mapred.JobClient: Job Counters
- 11/02/07 18:20:22 INFO mapred.JobClient: Launched map tasks=1
- 11/02/07 18:20:22 INFO mapred.JobClient: Data-local map tasks=1
- 11/02/07 18:20:22 INFO mapred.JobClient: FileSystemCounters
- 11/02/07 18:20:22 INFO mapred.JobClient: HDFS_BYTES_READ=10487
- 11/02/07 18:20:22 INFO mapred.JobClient: Map-Reduce Framework
- 11/02/07 18:20:22 INFO mapred.JobClient: Map input records=255
- 11/02/07 18:20:22 INFO mapred.JobClient: Spilled Records=0
- 11/02/07 18:20:22 INFO mapred.JobClient: Map output records=1452
+ [user1@instance accumulo]$ bin/tool.sh lib/accumulo-examples-*.jar org.apache.accumulo.examples.mapreduce.WordCount instance zookeepers /user/user1/wc wordCount -u username -p password
+
+ 11/02/07 18:20:11 INFO input.FileInputFormat: Total input paths to process : 1
+ 11/02/07 18:20:12 INFO mapred.JobClient: Running job: job_201102071740_0003
+ 11/02/07 18:20:13 INFO mapred.JobClient: map 0% reduce 0%
+ 11/02/07 18:20:20 INFO mapred.JobClient: map 100% reduce 0%
+ 11/02/07 18:20:22 INFO mapred.JobClient: Job complete: job_201102071740_0003
+ 11/02/07 18:20:22 INFO mapred.JobClient: Counters: 6
+ 11/02/07 18:20:22 INFO mapred.JobClient: Job Counters
+ 11/02/07 18:20:22 INFO mapred.JobClient: Launched map tasks=1
+ 11/02/07 18:20:22 INFO mapred.JobClient: Data-local map tasks=1
+ 11/02/07 18:20:22 INFO mapred.JobClient: FileSystemCounters
+ 11/02/07 18:20:22 INFO mapred.JobClient: HDFS_BYTES_READ=10487
+ 11/02/07 18:20:22 INFO mapred.JobClient: Map-Reduce Framework
+ 11/02/07 18:20:22 INFO mapred.JobClient: Map input records=255
+ 11/02/07 18:20:22 INFO mapred.JobClient: Spilled Records=0
+ 11/02/07 18:20:22 INFO mapred.JobClient: Map output records=1452
-
After the map reduce job completes, query the accumulo table to see word
counts.
- [user1@localhost accumulo]$ ./bin/accumulo shell -u root -p secret
- Shell - Accumulo Interactive Shell
- - version: 1.3.5-incubating-SNAPSHOT
- - instance id: 00000000-0000-0000-0000-000000000000
- -
- - type 'help' for a list of available commands
- -
- root@localhost> table wordCount
- root@localhost wordCount> scan -b the
- the count:20080906 [] 75
- their count:20080906 [] 2
- them count:20080906 [] 1
- then count:20080906 [] 1
- there count:20080906 [] 1
- these count:20080906 [] 3
- this count:20080906 [] 6
- through count:20080906 [] 1
- time count:20080906 [] 3
- time. count:20080906 [] 1
- to count:20080906 [] 27
- total count:20080906 [] 1
- tserver, count:20080906 [] 1
- tserver.compaction.major.concurrent.max count:20080906 [] 1
- ...
+ $ ./bin/accumulo shell -u username -p password
+ username@instance> table wordCount
+ username@instance wordCount> scan -b the
+ the count:20080906 [] 75
+ their count:20080906 [] 2
+ them count:20080906 [] 1
+ then count:20080906 [] 1
+ there count:20080906 [] 1
+ these count:20080906 [] 3
+ this count:20080906 [] 6
+ through count:20080906 [] 1
+ time count:20080906 [] 3
+ time. count:20080906 [] 1
+ to count:20080906 [] 27
+ total count:20080906 [] 1
+ tserver, count:20080906 [] 1
+ tserver.compaction.major.concurrent.max count:20080906 [] 1
+ ...
Modified: incubator/accumulo/branches/1.3/docs/examples/README.shard
URL: http://svn.apache.org/viewvc/incubator/accumulo/branches/1.3/docs/examples/README.shard?rev=1195687&r1=1195686&r2=1195687&view=diff
==============================================================================
--- incubator/accumulo/branches/1.3/docs/examples/README.shard (original)
+++ incubator/accumulo/branches/1.3/docs/examples/README.shard Mon Oct 31 21:40:44 2011
@@ -1,3 +1,21 @@
+Title: Shard Example
+Notice: Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ .
+ http://www.apache.org/licenses/LICENSE-2.0
+ .
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
Accumulo has in iterator called the intersecting iterator which supports querying a term index that is partitioned by
document, or "sharded". This example shows how to use the intersecting iterator through these four programs:
@@ -8,26 +26,18 @@ document, or "sharded". This example sho
To run these example programs, create two tables like below.
- Shell - Accumulo Interactive Shell
- -
- - version: 1.3.5-incubating-SNAPSHOT
- - instance name: test-instance
- - instance id: test-instance-id
- -
- - type 'help' for a list of available commands
- -
- foo@test-instance> createtable shard
- foo@test-instance shard> createtable doc2term
-
+ username@instance> createtable shard
+ username@instance shard> createtable doc2term
After creating the tables, index some files. The following command indexes all of the java files in the Accumulo source code.
- cd /local/user1/workspace/accumulo/
- find src -name "*.java" | xargs ./bin/accumulo org.apache.accumulo.examples.shard.Index test-instance localhost shard root password 30
+ $ cd /local/user1/workspace/accumulo/
+ $ find src -name "*.java" | xargs ./bin/accumulo org.apache.accumulo.examples.shard.Index instance zookeepers shard username password 30
The following command queries the index to find all files containing 'foo' and 'bar'.
- ./bin/accumulo org.apache.accumulo.examples.shard.Query test-instance localhost shard root password foo bar
+ $ cd $ACCUMULO_HOME
+ $ ./bin/accumulo org.apache.accumulo.examples.shard.Query instance zookeepers shard username password foo bar
/local/user1/workspace/accumulo/src/core/src/test/java/accumulo/core/security/ColumnVisibilityTest.java
/local/user1/workspace/accumulo/src/core/src/test/java/accumulo/core/client/mock/MockConnectorTest.java
/local/user1/workspace/accumulo/src/core/src/test/java/accumulo/core/security/VisibilityEvaluatorTest.java
@@ -42,11 +52,11 @@ The following command queries the index
Inorder to run ContinuousQuery, we need to run Reverse.java to populate doc2term
- [user1@cloud1 accumulo]$ ./bin/accumulo org.apache.accumulo.examples.shard.Reverse test-instance localhost shard doc2term root password
+ $ ./bin/accumulo org.apache.accumulo.examples.shard.Reverse instance zookeepers shard doc2term username password
Below ContinuousQuery is run using 5 terms. So it selects 5 random terms from each document, then it continually randomly selects one set of 5 terms and queries. It prints the number of matching documents and the time in seconds.
- [user1@cloud1 accumulo]$ ./bin/accumulo org.apache.accumulo.examples.shard.ContinuousQuery test-instance localhost shard doc2term root password 5
+ $ ./bin/accumulo org.apache.accumulo.examples.shard.ContinuousQuery instance zookeepers shard doc2term username password 5
[public, core, class, binarycomparable, b] 2 0.081
[wordtodelete, unindexdocument, doctablename, putdelete, insert] 1 0.041
[import, columnvisibilityinterpreterfactory, illegalstateexception, cv, columnvisibility] 1 0.049
@@ -54,4 +64,3 @@ Below ContinuousQuery is run using 5 ter
[for, static, println, public, the] 55 0.211
[sleeptime, wrappingiterator, options, long, utilwaitthread] 1 0.057
[string, public, long, 0, wait] 12 0.132
-
Modified: incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Administration.mdtext
URL: http://svn.apache.org/viewvc/incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Administration.mdtext?rev=1195687&r1=1195686&r2=1195687&view=diff
==============================================================================
--- incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Administration.mdtext (original)
+++ incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/Administration.mdtext Mon Oct 31 21:40:44 2011
@@ -35,7 +35,7 @@ Notice: Licensed to the Apache Softwa
## <a id=SECTION001110000000000000000></a> Hardware
-Because we are running essentially two or three systems simultaneously layered across the cluster: HDFS, Accumulo, and MapReduce, it is typical for hardware to consist of 4 to 8 cores, and 8 to 32 GB RAM. This is so each running process can have at least one core and 2 - 4 GB each.
+Because we are running essentially two or three systems simultaneously layered across the cluster: HDFS, Accumulo and MapReduce, it is typical for hardware to consist of 4 to 8 cores, and 8 to 32 GB RAM. This is so each running process can have at least one core and 2 - 4 GB each.
One core running HDFS can typically keep 2 to 4 disks busy, so each machine may typically have as little as 2 x 300GB disks and as much as 4 x 1TB or 2TB disks.
@@ -57,7 +57,7 @@ Repeat this step at each machine within
## <a id=SECTION001140000000000000000></a> Dependencies
-Accumulo requires HDFS, MapReduce, and ZooKeeper to be configured and running before starting. Password-less SSH should be configured between at least the Accumulo master and TabletServer machines. It is also a good idea to run Network Time Protocol (NTP) within the cluster to ensure nodes' clocks don't get too out of sync, which can cause problems with automatically timestamped data. Accumulo will remove from the set of TabletServers those machines whose times differ too much from the master's.
+Accumulo requires HDFS and ZooKeeper to be configured and running before starting. Password-less SSH should be configured between at least the Accumulo master and TabletServer machines. It is also a good idea to run Network Time Protocol (NTP) within the cluster to ensure nodes' clocks don't get too out of sync, which can cause problems with automatically timestamped data. Accumulo will remove from the set of TabletServers those machines whose times differ too much from the master's.
## <a id=SECTION001150000000000000000></a> Configuration
@@ -108,7 +108,7 @@ $ACCUMULO_HOME/conf/accumulo-site.xml :
</property>
-This enables Accumulo to find Zookeeper. Accumulo uses ZooKeeper to coordinate settings between processes and helps finalize TabletServer failure.
+This enables Accumulo to find ZooKeeper. Accumulo uses ZooKeeper to coordinate settings between processes and helps finalize TabletServer failure.
Accumulo records all changes to tables to a write-ahead log before committing them to the table. The `walog' setting specifies the local directory on each machine to which write-ahead logs are written. This directory should exist on all machines acting as TabletServers.
@@ -132,7 +132,7 @@ The initialization script will prompt yo
### <a id=SECTION001171000000000000000></a> Starting Accumulo
-Make sure Hadoop is configured on all of the machines in the cluster, including access to a shared HDFS instance. Make sure HDFS and MapReduce are running. Make sure ZooKeeper is configured and running on at least one machine in the cluster. Start Accumulo using the bin/start-all.sh script.
+Make sure Hadoop is configured on all of the machines in the cluster, including access to a shared HDFS instance. Make sure HDFS and ZooKeeper are running. Make sure ZooKeeper is configured and running on at least one machine in the cluster. Start Accumulo using the bin/start-all.sh script.
To verify that Accumulo is running, check the Status page as described under _Monitoring_. In addition, the Shell can provide some information about the status of tables via reading the !METADATA table.
@@ -154,7 +154,7 @@ $ACCUMULO/logs/.
In the event of TabletServer failure or error on shutting Accumulo down, some mutations may not have been minor compacted to HDFS properly. In this case, Accumulo will automatically reapply such mutations from the write-ahead log either when the tablets from the failed server are reassigned by the Master, in the case of a single TabletServer failure or the next time Accumulo starts, in the event of failure during shutdown.
-Recovery is performed using the MapReduce framework and is shown on the Accumulo monitor status page. Once the MapReduce recovery is complete any tablets involved should return to an ``online" state. Until then those tablets will be unavailable to clients.
+Recovery is performed by asking the loggers to copy their write-ahead logs into HDFS. As the logs are copied, they are also sorted, so that tablets can easily find their missing updates. The copy/sort status of each file is displayed on Accumulo monitor status page. Once the recovery is complete any tablets involved should return to an ``online" state. Until then those tablets will be unavailable to clients.
The Accumulo client library is configured to retry failed mutations and in many cases clients will be able to continue processing after the recovery process without throwing an exception.
Added: incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/examples.mdtext
URL: http://svn.apache.org/viewvc/incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/examples.mdtext?rev=1195687&view=auto
==============================================================================
--- incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/examples.mdtext (added)
+++ incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/examples.mdtext Mon Oct 31 21:40:44 2011
@@ -0,0 +1,56 @@
+Title: Examples
+Notice: Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ .
+ http://www.apache.org/licenses/LICENSE-2.0
+ .
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Each README in the examples directory highlights the use of particular features of Accumulo.
+
+Before running any of the examples, the following steps must be performed.
+
+1. Install and run Accumulo via the instructions found in $ACCUMULO_HOME/README.
+Remember the instance name. It will be referred to as "instance" throughout the examples.
+A comma-separated list of zookeeper servers will be referred to as "zookeepers".
+
+2. Create an Accumulo user (see the [user manual][1]), or use the root user.
+The Accumulo user name will be referred to as "username" with password "password" throughout the examples.
+
+In all commands, you will need to replace "instance", "zookeepers", "username", and "password" with the values you set for your Accumulo instance.
+
+Commands intended to be run in bash are prefixed by '$'. These are always assumed to be run from the $ACCUMULO_HOME directory.
+
+Commands intended to be run in the Accumulo shell are prefixed by '>'.
+
+[1]: /accumulo/user_manual_1.3-incubating/Accumulo_Shell.html#SECTION00430000000000000000
+[aggregation](examples/aggregation.html)
+
+[batch](examples/batch.html)
+
+[bloom](examples/bloom.html)
+
+[bulkIngest](examples/bulkIngest.html)
+
+[constraints](examples/constraints.html)
+
+[dirlist](examples/dirlist.html)
+
+[filter](examples/filter.html)
+
+[helloworld](examples/helloworld.html)
+
+[mapred](examples/mapred.html)
+
+[shard](examples/shard.html)
+
Added: incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/examples/aggregation.mdtext
URL: http://svn.apache.org/viewvc/incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/examples/aggregation.mdtext?rev=1195687&view=auto
==============================================================================
--- incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/examples/aggregation.mdtext (added)
+++ incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/examples/aggregation.mdtext Mon Oct 31 21:40:44 2011
@@ -0,0 +1,50 @@
+Title: Aggregation Example
+Notice: Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ .
+ http://www.apache.org/licenses/LICENSE-2.0
+ .
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+This is a simple aggregation example. To build this example run maven and then
+copy the produced jar into the accumulo lib dir. This is already done in the
+tar distribution.
+
+ $ bin/accumulo shell -u username
+ Enter current password for 'username'@'instance': ***
+
+ Shell - Accumulo Interactive Shell
+ -
+ - version: 1.3.x-incubating
+ - instance name: instance
+ - instance id: 00000000-0000-0000-0000-000000000000
+ -
+ - type 'help' for a list of available commands
+ -
+ username@instance> createtable aggtest1 -a app=org.apache.accumulo.examples.aggregation.SortedSetAggregator
+ username@instance aggtest1> insert foo app 1 a
+ username@instance aggtest1> insert foo app 1 b
+ username@instance aggtest1> scan
+ foo app:1 [] a,b
+ username@instance aggtest1> insert foo app 1 z,1,foo,w
+ username@instance aggtest1> scan
+ foo app:1 [] 1,a,b,foo,w,z
+ username@instance aggtest1> insert foo app 2 cat,dog,muskrat
+ username@instance aggtest1> insert foo app 2 mouse,bird
+ username@instance aggtest1> scan
+ foo app:1 [] 1,a,b,foo,w,z
+ foo app:2 [] bird,cat,dog,mouse,muskrat
+ username@instance aggtest1>
+
+In this example a table is created and the example set aggregator is
+applied to the column family app.
Propchange: incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/examples/aggregation.mdtext
------------------------------------------------------------------------------
svn:executable = *
Added: incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/examples/batch.mdtext
URL: http://svn.apache.org/viewvc/incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/examples/batch.mdtext?rev=1195687&view=auto
==============================================================================
--- incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/examples/batch.mdtext (added)
+++ incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/examples/batch.mdtext Mon Oct 31 21:40:44 2011
@@ -0,0 +1,53 @@
+Title: Batch Writing and Scanning Example
+Notice: Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ .
+ http://www.apache.org/licenses/LICENSE-2.0
+ .
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+This is an example of how to use the batch writer and batch scanner. To compile
+the example, run maven and copy the produced jar into the accumulo lib dir.
+This is already done in the tar distribution.
+
+Below are commands that add 10000 entries to accumulo and then do 100 random
+queries. The write command generates random 50 byte values.
+
+Be sure to use the name of your instance (given as instance here) and the appropriate
+list of zookeeper nodes (given as zookeepers here).
+
+Before you run this, you must ensure that the user you are running has the
+"exampleVis" authorization. (you can set this in the shell with "setauths -u username -s exampleVis")
+
+ $ ./bin/accumulo shell -u root
+ > setauths -u username -s exampleVis
+ > exit
+
+You must also create the table, batchtest1, ahead of time. (In the shell, use "createtable batchtest1")
+
+ $ ./bin/accumulo shell -u username
+ > createtable batchtest1
+ > exit
+ $ ./bin/accumulo org.apache.accumulo.examples.client.SequentialBatchWriter instance zookeepers username password batchtest1 0 10000 50 20000000 500 20 exampleVis
+ $ ./bin/accumulo org.apache.accumulo.examples.client.RandomBatchScanner instance zookeepers username password batchtest1 100 0 10000 50 20 exampleVis
+ 07 11:33:11,103 [client.CountingVerifyingReceiver] INFO : Generating 100 random queries...
+ 07 11:33:11,112 [client.CountingVerifyingReceiver] INFO : finished
+ 07 11:33:11,260 [client.CountingVerifyingReceiver] INFO : 694.44 lookups/sec 0.14 secs
+
+ 07 11:33:11,260 [client.CountingVerifyingReceiver] INFO : num results : 100
+
+ 07 11:33:11,364 [client.CountingVerifyingReceiver] INFO : Generating 100 random queries...
+ 07 11:33:11,370 [client.CountingVerifyingReceiver] INFO : finished
+ 07 11:33:11,416 [client.CountingVerifyingReceiver] INFO : 2173.91 lookups/sec 0.05 secs
+
+ 07 11:33:11,416 [client.CountingVerifyingReceiver] INFO : num results : 100
Propchange: incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/examples/batch.mdtext
------------------------------------------------------------------------------
svn:executable = *
Added: incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/examples/bloom.mdtext
URL: http://svn.apache.org/viewvc/incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/examples/bloom.mdtext?rev=1195687&view=auto
==============================================================================
--- incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/examples/bloom.mdtext (added)
+++ incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/examples/bloom.mdtext Mon Oct 31 21:40:44 2011
@@ -0,0 +1,124 @@
+Title: Bloom Filter Example
+Notice: Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ .
+ http://www.apache.org/licenses/LICENSE-2.0
+ .
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+This example shows how to create a table with bloom filters enabled. It also
+shows how bloom filters increase query performance when looking for values that
+do not exist in a table.
+
+Below table named bloom_test is created and bloom filters are enabled.
+
+ $ ./accumulo shell -u username -p password
+ Shell - Accumulo Interactive Shell
+ - version: 1.3.x-incubating
+ - instance name: instance
+ - instance id: 00000000-0000-0000-0000-000000000000
+ -
+ - type 'help' for a list of available commands
+ -
+ username@instance> setauths -u username -s exampleVis
+ username@instance> createtable bloom_test
+ username@instance bloom_test> config -t bloom_test -s table.bloom.enabled=true
+ username@instance bloom_test> exit
+
+Below 1 million random values are inserted into accumulo. The randomly
+generated rows range between 0 and 1 billion. The random number generator is
+initialized with the seed 7.
+
+ $ ./bin/accumulo org.apache.accumulo.examples.client.RandomBatchWriter -s 7 instance zookeepers username password bloom_test 1000000 0 1000000000 50 2000000 60000 3 exampleVis
+
+Below the table is flushed, look at the monitor page and wait for the flush to
+complete.
+
+ $ ./bin/accumulo shell -u username -p password
+ username@instance> flush -t bloom_test
+ Flush of table bloom_test initiated...
+ username@instance> exit
+
+After the flush completes, 500 random queries are done against the table. The
+same seed is used to generate the queries, therefore everything is found in the
+table.
+
+ $ ./bin/accumulo org.apache.accumulo.examples.client.RandomBatchScanner -s 7 instance zookeepers username password bloom_test 500 0 1000000000 50 20 exampleVis
+ Generating 500 random queries...finished
+ 96.19 lookups/sec 5.20 secs
+ num results : 500
+ Generating 500 random queries...finished
+ 102.35 lookups/sec 4.89 secs
+ num results : 500
+
+Below another 500 queries are performed, using a different seed which results
+in nothing being found. In this case the lookups are much faster because of
+the bloom filters.
+
+ $ ../bin/accumulo org.apache.accumulo.examples.client.RandomBatchScanner -s 8 instance zookeepers username password bloom_test 500 0 1000000000 50 20 exampleVis
+ Generating 500 random queries...finished
+ 2212.39 lookups/sec 0.23 secs
+ num results : 0
+ Did not find 500 rows
+ Generating 500 random queries...finished
+ 4464.29 lookups/sec 0.11 secs
+ num results : 0
+ Did not find 500 rows
+
+********************************************************************************
+
+Bloom filters can also speed up lookups for entries that exist. In accumulo
+data is divided into tablets and each tablet has multiple map files. Every
+lookup in accumulo goes to a specific tablet where a lookup is done on each
+map file in the tablet. So if a tablet has three map files, lookup performance
+can be three times slower than a tablet with one map file. However if the map
+files contain unique sets of data, then bloom filters can help eliminate map
+files that do not contain the row being looked up. To illustrate this two
+identical tables were created using the following process. One table had bloom
+filters, the other did not. Also the major compaction ratio was increased to
+prevent the files from being compacted into one file.
+
+ * Insert 1 million entries using RandomBatchWriter with a seed of 7
+ * Flush the table using the shell
+ * Insert 1 million entries using RandomBatchWriter with a seed of 8
+ * Flush the table using the shell
+ * Insert 1 million entries using RandomBatchWriter with a seed of 9
+ * Flush the table using the shell
+
+After following the above steps, each table will have a tablet with three map
+files. Each map file will contain 1 million entries generated with a different
+seed.
+
+Below 500 lookups are done against the table without bloom filters using random
+NG seed 7. Even though only one map file will likely contain entries for this
+seed, all map files will be interrogated.
+
+ $ ./bin/accumulo org.apache.accumulo.examples.client.RandomBatchScanner -s 7 instance zookeepers username password bloom_test1 500 0 1000000000 50 20 exampleVis
+ Generating 500 random queries...finished
+ 35.09 lookups/sec 14.25 secs
+ num results : 500
+ Generating 500 random queries...finished
+ 35.33 lookups/sec 14.15 secs
+ num results : 500
+
+Below the same lookups are done against the table with bloom filters. The
+lookups were 2.86 times faster because only one map file was used, even though three
+map files existed.
+
+ $ ./bin/accumulo org.apache.accumulo.examples.client.RandomBatchScanner -s 7 instance zookeepers username password bloom_test2 500 0 1000000000 50 20 exampleVis
+ Generating 500 random queries...finished
+ 99.03 lookups/sec 5.05 secs
+ num results : 500
+ Generating 500 random queries...finished
+ 101.15 lookups/sec 4.94 secs
+ num results : 500
Propchange: incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/examples/bloom.mdtext
------------------------------------------------------------------------------
svn:executable = *
Added: incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/examples/bulkIngest.mdtext
URL: http://svn.apache.org/viewvc/incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/examples/bulkIngest.mdtext?rev=1195687&view=auto
==============================================================================
--- incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/examples/bulkIngest.mdtext (added)
+++ incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/examples/bulkIngest.mdtext Mon Oct 31 21:40:44 2011
@@ -0,0 +1,34 @@
+Title: Bulk Ingest Example
+Notice: Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ .
+ http://www.apache.org/licenses/LICENSE-2.0
+ .
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+This is an example of how to bulk ingest data into accumulo using map reduce.
+
+The following commands show how to run this example. This example creates a
+table called test_bulk which has two initial split points. Then 1000 rows of
+test data are created in HDFS. After that the 1000 rows are ingested into
+accumulo. Then we verify the 1000 rows are in accumulo. The
+first two arguments to all of the commands except for GenerateTestData are the
+accumulo instance name, and a comma-separated list of zookeepers.
+
+ $ ./bin/accumulo org.apache.accumulo.examples.mapreduce.bulk.SetupTable instance zookeepers username password test_bulk row_00000333 row_00000666
+ $ ./bin/accumulo org.apache.accumulo.examples.mapreduce.bulk.GenerateTestData 0 1000 bulk/test_1.txt
+
+ $ ./bin/tool.sh lib/accumulo-examples-*.jar org.apache.accumulo.examples.mapreduce.bulk.BulkIngestExample instance zookeepers username password test_bulk bulk tmp/bulkWork
+ $ ./bin/accumulo org.apache.accumulo.examples.mapreduce.bulk.VerifyIngest instance zookeepers username password test_bulk 0 1000
+
+For a high level discussion of bulk ingest, see the docs dir.
Propchange: incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/examples/bulkIngest.mdtext
------------------------------------------------------------------------------
svn:executable = *
Added: incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/examples/constraints.mdtext
URL: http://svn.apache.org/viewvc/incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/examples/constraints.mdtext?rev=1195687&view=auto
==============================================================================
--- incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/examples/constraints.mdtext (added)
+++ incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/examples/constraints.mdtext Mon Oct 31 21:40:44 2011
@@ -0,0 +1,48 @@
+Title: Constraints Example
+Notice: Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ .
+ http://www.apache.org/licenses/LICENSE-2.0
+ .
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+This an example of how to create a table with constraints. Below a table is
+create with two example constraints. One constraints does not allow non alpha
+numeric keys. The other constraint does not allow non numeric values. Two
+inserts that violate these constraints are attempted and denied. The scan at
+the end shows the inserts were not allowed.
+
+ $ ./bin/accumulo shell -u username -p pass
+
+ Shell - Accumulo Interactive Shell
+ -
+ - version: 1.3.x-incubating
+ - instance name: instance
+ - instance id: 00000000-0000-0000-0000-000000000000
+ -
+ - type 'help' for a list of available commands
+ -
+ username@instance> createtable testConstraints
+ username@instance testConstraints> config -t testConstraints -s table.constraint.1=org.apache.accumulo.examples.constraints.NumericValueConstraint
+ username@instance testConstraints> config -t testConstraints -s table.constraint.2=org.apache.accumulo.examples.constraints.AlphaNumKeyConstrain
+ username@instance testConstraints> insert r1 cf1 cq1 1111
+ username@instance testConstraints> insert r1 cf1 cq1 ABC
+ Constraint Failures:
+ ConstraintViolationSummary(constrainClass:org.apache.accumulo.examples.constraints.NumericValueConstraint, violationCode:1, violationDescription:Value is not numeric, numberOfViolatingMutations:1)
+ username@instance testConstraints> insert r1! cf1 cq1 ABC
+ Constraint Failures:
+ ConstraintViolationSummary(constrainClass:org.apache.accumulo.examples.constraints.NumericValueConstraint, violationCode:1, violationDescription:Value is not numeric, numberOfViolatingMutations:1)
+ ConstraintViolationSummary(constrainClass:org.apache.accumulo.examples.constraints.AlphaNumKeyConstraint, violationCode:1, violationDescription:Row was not alpha numeric, numberOfViolatingMutations:1)
+ username@instance testConstraints> scan
+ r1 cf1:cq1 [] 1111
+ username@instance testConstraints>
Propchange: incubator/accumulo/site/trunk/content/accumulo/user_manual_1.3-incubating/examples/constraints.mdtext
------------------------------------------------------------------------------
svn:executable = *