You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@accumulo.apache.org by ct...@apache.org on 2020/01/10 04:41:54 UTC
[accumulo-website] branch master updated: Proxy client code updates
(#216)
This is an automated email from the ASF dual-hosted git repository.
ctubbsii pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/accumulo-website.git
The following commit(s) were added to refs/heads/master by this push:
new 0badae8 Proxy client code updates (#216)
0badae8 is described below
commit 0badae87caced1722ba99c1473dcc7719702cb3e
Author: Mike Walch <mw...@apache.org>
AuthorDate: Thu Jan 9 23:41:45 2020 -0500
Proxy client code updates (#216)
* Moved code from proxy docs on website to the accumulo-proxy repo
* Updated names for client scripts
---
_docs-2/development/proxy.md | 368 +++----------------------------
_posts/blog/2019-12-16-accumulo-proxy.md | 6 +-
2 files changed, 31 insertions(+), 343 deletions(-)
diff --git a/_docs-2/development/proxy.md b/_docs-2/development/proxy.md
index 9bd5a1c..f663331 100644
--- a/_docs-2/development/proxy.md
+++ b/_docs-2/development/proxy.md
@@ -4,7 +4,7 @@ category: development
order: 7
---
-The proxy API allows the interaction with Accumulo with languages other than Java.
+The [Accumulo Proxy] allows the interaction with Accumulo with languages other than Java.
A proxy server is provided in the codebase and a client can further be generated.
The proxy API can also be used instead of the traditional [AccumuloClient] class to
provide a single TCP port in which clients can be securely routed through a firewall,
@@ -18,361 +18,49 @@ DataNodes. A proxy client only needs the ability to communicate with the proxy s
## Running the Proxy Server
-The proxy server is included in the Accumulo tarball distribution and can be run using
-the `accumulo` command. The proxy is configured using `accumulo-client.properties` and
-`proxy.properties` which can be found at `conf/templates/proxy.properties`. Create a
-copy of this file and edit it for your environment:
+To run [Accumulo Proxy] server, first clone the repository:
- cp ./conf/templates/proxy.properties ./conf/
- vim ./conf/proxy.properties
-
-At the very least, you need to configure the following properties:
-
- port=42424
- protocolFactory=org.apache.thrift.protocol.TCompactProtocol$Factory
- tokenClass=org.apache.accumulo.core.client.security.tokens.PasswordToken
-
-After `proxy.properties` is configured, the proxy server can be started using the
-`accumulo` command:
-
- accumulo proxy -p /path/to/proxy.properties -c /path/to/accumulo-client.properties &
-
-Logs will be output to the directory set by `ACCUMULO_LOG_DIR` in `accumulo-env.sh`.
-
-## Prerequisites for Proxy Clients
-
-Before you can run a proxy client, you will need the following:
-
-1. Proxy client code generated for your language
-2. Thrift library installed for your language
-
-These requirements are described in detail below.
-
-### Proxy client code
-
-The Accumulo tarball distribution ships with pre-generated client code for Python, Ruby, and C++ in
-`lib/proxy`.
-
-If you want to write a proxy in another language, you will need to install Thrift and generate
-client code for you language using `lib/proxy/thrift/proxy.thrift`. See the [Thrift Tutorial][tutorial]
-to how generate source from a thrift file.
-
-### Thrift library
-
-Language-specific Thrift libraries can be installed using an OS or language package manager (i.e gem, pip, etc).
-For example, `pip install thrift` will install Python-specific thrift libraries on your machine.
-
-## Proxy Client Examples
-
-The following examples show proxy clients written in Java, Ruby, and Python.
-
-### Java
-
-After initiating a connection to the Proxy (see Apache Thrift's documentation for examples
-of connecting to a Thrift service), the methods on the proxy client will be available. The
-first thing to do is log in:
-
-```java
-Map password = new HashMap<String,String>();
-password.put("password", "secret");
-ByteBuffer token = client.login("root", password);
+```bash
+git clone https://github.com/apache/accumulo-proxy
```
-Once logged in, the token returned will be used for most subsequent calls to the client.
-Let's create a table, add some data, scan the table, and delete it.
+Next, follow the instructions in the Proxy [README.md] or use [Uno] to run the proxy.
-First, create a table.
+To run the Proxy using [Uno], configure `uno.conf` to start the Proxy by setting the
+configuration below:
-```java
-client.createTable(token, "myTable", true, TimeType.MILLIS);
```
-
-Next, add some data:
-
-```java
-// first, create a writer on the server
-String writer = client.createWriter(token, "myTable", new WriterOptions());
-
-//rowid
-ByteBuffer rowid = ByteBuffer.wrap("UUID".getBytes());
-
-//mutation like class
-ColumnUpdate cu = new ColumnUpdate();
-cu.setColFamily("MyFamily".getBytes());
-cu.setColQualifier("MyQualifier".getBytes());
-cu.setColVisibility("VisLabel".getBytes());
-cu.setValue("Some Value.".getBytes());
-
-List<ColumnUpdate> updates = new ArrayList<ColumnUpdate>();
-updates.add(cu);
-
-// build column updates
-Map<ByteBuffer, List<ColumnUpdate>> cellsToUpdate = new HashMap<ByteBuffer, List<ColumnUpdate>>();
-cellsToUpdate.put(rowid, updates);
-
-// send updates to the server
-client.updateAndFlush(writer, "myTable", cellsToUpdate);
-
-client.closeWriter(writer);
+export POST_RUN_PLUGINS="accumulo-proxy"
+export PROXY_REPO=/path/to/accumulo-proxy
```
-Scan for the data and batch the return of the results on the server:
-
-```java
-String scanner = client.createScanner(token, "myTable", new ScanOptions());
-ScanResult results = client.nextK(scanner, 100);
-
-for(KeyValue keyValue : results.getResultsIterator()) {
- // do something with results
-}
+## Proxy Client Examples
-client.closeScanner(scanner);
-```
+The following examples show proxy clients written in Java, Ruby, and Python.
### Ruby
-The example ruby code below can be run using the following command (the -I option is needed for ruby 1.9.x):
-
- ruby -I . client.rb <host of server>
-
-**Warning:** The script will connect to Accumulo, create a table, and add some rows to it.
-
-```ruby
-require 'rubygems'
-require 'thrift'
-require 'accumulo_proxy'
-
-server = ARGV[0] || 'localhost'
-
-socket = Thrift::Socket.new(server, 42424, 9001)
-transport = Thrift::FramedTransport.new(socket)
-proto = Thrift::CompactProtocol.new(transport)
-proxy = Accumulo::AccumuloProxy::Client.new(proto)
-
-# open up the connect
-transport.open()
-
-# Test if the server is up
-login = proxy.login('root', {'password' => 'secret'})
-
-# print out a table list
-puts "List of tables: #{proxy.listTables(login).inspect}"
-
-testtable = "rubytest"
-proxy.createTable(login, testtable, true, Accumulo::TimeType::MILLIS) unless proxy.tableExists(login,testtable)
-
-update1 = Accumulo::ColumnUpdate.new({'colFamily' => "cf1", 'colQualifier' => "cq1", 'value'=> "a"})
-update2 = Accumulo::ColumnUpdate.new({'colFamily' => "cf2", 'colQualifier' => "cq2", 'value'=> "b"})
-proxy.updateAndFlush(login,testtable,{'row1' => [update1,update2]})
-
-cookie = proxy.createScanner(login,testtable,nil)
-result = proxy.nextK(cookie,10)
-result.results.each{ |keyvalue| puts "Key: #{keyvalue.key.inspect} Value: #{keyvalue.value}" }
-```
+The [Accumulo Proxy] repo has an example [ruby client] along with [instructions][rubyinstruct] on how
+to run it.
### Python
-The example python client code below (if saved to TestClient.py) can be run using the following command:
+The [Accumulo Proxy] repo has two example Python scripts that can be run using these [instructions][pythoninstruct]:
+ * [basic client] - creates a table, writes data to it, and then reads it
+ * [namespace client] - shows how to manage Accumulo namespaces.
- PYTHONPATH=/path/to/accumulo-{{ page.latest_release }}/lib/proxy/gen-py python TestClient.py
-
-As a warning, this script will create a table in your Accumulo instance and add a few cells to it.
-
-```python
-#! /usr/bin/env python
-
-import sys
-
-from thrift import Thrift
-from thrift.transport import TSocket
-from thrift.transport import TTransport
-from thrift.protocol import TCompactProtocol
-
-from accumulo import AccumuloProxy
-from accumulo.ttypes import *
-
-transport = TSocket.TSocket('localhost', 42424)
-transport = TTransport.TFramedTransport(transport)
-protocol = TCompactProtocol.TCompactProtocol(transport)
-client = AccumuloProxy.Client(protocol)
-transport.open()
-
-login = client.login('root', {'password':'secret'})
-
-print client.listTables(login)
-
-testtable = "pythontest"
-if not client.tableExists(login, testtable):
- client.createTable(login, testtable, True, TimeType.MILLIS)
-
-row1 = {'a':[ColumnUpdate('a','a',value='value1'), ColumnUpdate('b','b',value='value2')]}
-client.updateAndFlush(login, testtable, row1)
-
-cookie = client.createScanner(login, testtable, None)
-for entry in client.nextK(cookie, 10).results:
- print entry
-```
-
-The example code below shows proxy client code for managing namespaces:
-
-```python
-#! /usr/bin/env python
-
-from thrift.protocol import TCompactProtocol
-from thrift.transport import TSocket, TTransport
-
-from proxy import AccumuloProxy
-from proxy.ttypes import NamespacePermission, IteratorSetting, IteratorScope, AccumuloException
-
-def main():
- transport = TSocket.TSocket('localhost', 42424)
- transport = TTransport.TFramedTransport(transport)
- protocol = TCompactProtocol.TCompactProtocol(transport)
- client = AccumuloProxy.Client(protocol)
- transport.open()
- login = client.login('root', {'password': 'password'})
-
- client.createLocalUser(login, 'user1', 'password1')
-
- print client.listNamespaces(login)
-
- # create a namespace and give the user1 all permissions
- print 'creating namespace testing'
- client.createNamespace(login, 'testing')
- assert client.namespaceExists(login, 'testing')
- print client.listNamespaces(login)
-
- print 'testing namespace renaming'
- client.renameNamespace(login, 'testing', 'testing2')
- assert not client.namespaceExists(login, 'testing')
- assert client.namespaceExists(login, 'testing2')
- client.renameNamespace(login, 'testing2', 'testing')
- assert not client.namespaceExists(login, 'testing2')
- assert client.namespaceExists(login, 'testing')
-
- print 'granting all namespace permissions to user1'
- for k, v in NamespacePermission._VALUES_TO_NAMES.iteritems():
- client.grantNamespacePermission(login, 'user1', 'testing', k)
-
- # make sure the last operation worked
- for k, v in NamespacePermission._VALUES_TO_NAMES.iteritems():
- assert client.hasNamespacePermission(login, 'user1', 'testing', k), \
- 'user1 does\'nt have namespace permission %s' % v
-
- print 'default namespace: ' + client.defaultNamespace()
- print 'system namespace: ' + client.systemNamespace()
-
- # grab the namespace properties
- print 'retrieving namespace properties'
- props = client.getNamespaceProperties(login, 'testing')
- assert props and props['table.compaction.major.ratio'] == '3'
-
- # update a property and verify it is good
- print 'setting namespace property table.compaction.major.ratio = 4'
- client.setNamespaceProperty(login, 'testing', 'table.compaction.major.ratio', '4')
- props = client.getNamespaceProperties(login, 'testing')
- assert props and props['table.compaction.major.ratio'] == '4'
-
- print 'retrieving namespace ID map'
- nsids = client.namespaceIdMap(login)
- assert nsids and 'accumulo' in nsids
-
- print 'attaching debug iterator to namespace testing'
- setting = IteratorSetting(priority=40, name='DebugTheThings',
- iteratorClass='org.apache.accumulo.core.iterators.DebugIterator', properties={})
- client.attachNamespaceIterator(login, 'testing', setting, [IteratorScope.SCAN])
- setting = client.getNamespaceIteratorSetting(login, 'testing', 'DebugTheThings', IteratorScope.SCAN)
- assert setting and setting.name == 'DebugTheThings'
-
- # make sure the iterator is in the list
- iters = client.listNamespaceIterators(login, 'testing')
- found = False
- for name, scopes in iters.iteritems():
- if name == 'DebugTheThings':
- found = True
- break
- assert found
-
- print 'checking for iterator conflicts'
-
- # this next statment should be fine since we are on a different scope
- client.checkNamespaceIteratorConflicts(login, 'testing', setting, [IteratorScope.MINC])
-
- # this time it should throw an exception since we have already added the iterator with this scope
- try:
- client.checkNamespaceIteratorConflicts(login, 'testing', setting, [IteratorScope.SCAN, IteratorScope.MINC])
- except AccumuloException:
- pass
- else:
- assert False, 'There should have been a namespace iterator conflict!'
-
- print 'removing debug iterator from namespace testing'
- client.removeNamespaceIterator(login, 'testing', 'DebugTheThings', [IteratorScope.SCAN])
-
- # make sure the iterator is NOT in the list anymore
- iters = client.listNamespaceIterators(login, 'testing')
- found = False
- for name, scopes in iters.iteritems():
- if name == 'DebugTheThings':
- found = True
- break
- assert not found
-
- print 'adding max mutation size namespace constraint'
- constraintid = client.addNamespaceConstraint(login, 'testing',
- 'org.apache.accumulo.test.constraints.MaxMutationSize')
-
- print 'make sure constraint was added'
- constraints = client.listNamespaceConstraints(login, 'testing')
- found = False
- for name, cid in constraints.iteritems():
- if cid == constraintid and name == 'org.apache.accumulo.test.constraints.MaxMutationSize':
- found = True
- break
- assert found
-
- print 'remove max mutation size namespace constraint'
- client.removeNamespaceConstraint(login, 'testing', constraintid)
-
- print 'make sure constraint was removed'
- constraints = client.listNamespaceConstraints(login, 'testing')
- found = False
- for name, cid in constraints.iteritems():
- if cid == constraintid and name == 'org.apache.accumulo.test.constraints.MaxMutationSize':
- found = True
- break
- assert not found
-
- print 'test a namespace class load of the VersioningIterator'
- res = client.testNamespaceClassLoad(login, 'testing', 'org.apache.accumulo.core.iterators.user.VersioningIterator',
- 'org.apache.accumulo.core.iterators.SortedKeyValueIterator')
- assert res
-
- print 'test a bad namespace class load of the VersioningIterator'
- res = client.testNamespaceClassLoad(login, 'testing', 'org.apache.accumulo.core.iterators.user.VersioningIterator',
- 'dummy')
- assert not res
-
- # revoke the permissions
- print 'revoking namespace permissions for user1'
- for k, v in NamespacePermission._VALUES_TO_NAMES.iteritems():
- client.revokeNamespacePermission(login, 'user1', 'testing', k)
-
- # make sure the last operation worked
- for k, v in NamespacePermission._VALUES_TO_NAMES.iteritems():
- assert not client.hasNamespacePermission(login, 'user1', 'testing', k), \
- 'user1 does\'nt have namespace permission %s' % v
-
- print 'deleting namespace testing'
- client.deleteNamespace(login, 'testing')
- assert not client.namespaceExists(login, 'testing')
-
- print 'deleting user1'
- client.dropLocalUser(login, 'user1')
-
-if __name__ == "__main__":
- main()
-```
+### Java
+Users may want to write a [Java client] to the proxy to restrict access to the cluster.
+
+[Accumulo Proxy]: https://github.com/apache/accumulo-proxy/
+[Uno]: https://github.com/apache/fluo-uno/
+[README.md]: https://github.com/apache/accumulo-proxy/blob/master/README.md
+[Java client]: https://github.com/apache/accumulo-proxy/docs/java_client.md
+[ruby client]: https://github.com/apache/accumulo-proxy/src/main/ruby/client.rb
+[pythoninstruct]: https://github.com/apache/accumulo-proxy/#create-an-accumulo-client-using-python
+[rubyinstruct]: https://github.com/apache/accumulo-proxy/#create-an-accumulo-client-using-ruby
+[basic client]: https://github.com/apache/accumulo-proxy/blob/master/src/main/python/basic_client.py
+[namespace client]: https://github.com/apache/accumulo-proxy/blob/master/src/main/python/namespace_client.py
[AccumuloClient]: {% jurl org.apache.accumulo.core.client.AccumuloClient %}
[tutorial]: https://thrift.apache.org/tutorial/
diff --git a/_posts/blog/2019-12-16-accumulo-proxy.md b/_posts/blog/2019-12-16-accumulo-proxy.md
index f6e8b89..e9d2f15 100644
--- a/_posts/blog/2019-12-16-accumulo-proxy.md
+++ b/_posts/blog/2019-12-16-accumulo-proxy.md
@@ -44,10 +44,10 @@ cd accumulo-client/
pipenv --python 2.7
pipenv install thrift
pipenv install -e /path/to/accumulo-proxy/src/main/python
-cp /path/to/accumulo-proxy/src/main/python/example.py .
+cp /path/to/accumulo-proxy/src/main/python/basic_client.py .
# Edit credentials if needed
-vim example.py
-pipenv run python2 example.py
+vim basic_client.py
+pipenv run python2 basic_client.py
```
Verify that the table was created or data was written using `uno ashell` or the Accumulo monitor.