You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@accumulo.apache.org by mw...@apache.org on 2018/11/28 19:13:45 UTC
[accumulo-website] branch master updated: Updated MapReduce docs to
include new configuration (#135)
This is an automated email from the ASF dual-hosted git repository.
mwalch pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/accumulo-website.git
The following commit(s) were added to refs/heads/master by this push:
new 2bab009 Updated MapReduce docs to include new configuration (#135)
2bab009 is described below
commit 2bab009eda7c4786a1b01f2c2e223c91072c30fb
Author: Mike Walch <mw...@apache.org>
AuthorDate: Wed Nov 28 14:13:41 2018 -0500
Updated MapReduce docs to include new configuration (#135)
---
_docs-2/administration/upgrading.md | 1 +
_docs-2/development/mapreduce.md | 15 +++++++++++++++
2 files changed, 16 insertions(+)
diff --git a/_docs-2/administration/upgrading.md b/_docs-2/administration/upgrading.md
index 4ad5a00..18c5714 100644
--- a/_docs-2/administration/upgrading.md
+++ b/_docs-2/administration/upgrading.md
@@ -42,6 +42,7 @@ Below are some changes in 2.0 that you should be aware of:
- `log4j-service.properties` for all Accumulo services (except monitor)
- `logj4-monitor.properties` for Accumulo monitor
- `log4j.properties` for Accumulo clients and commands
+* [New Hadoop configuration is required]({% durl development/mapreduce#configuration %}) when reading or writing to Accumulo using MapReduce.
* Run the command `accumulo shell` to access the shell using configuration in `conf/accumulo-client.properties`
When your Accumulo 2.0 installation is properly configured, stop Accumulo 1.8/9 and start Accumulo 2.0:
diff --git a/_docs-2/development/mapreduce.md b/_docs-2/development/mapreduce.md
index 0adb649..ee6a93a 100644
--- a/_docs-2/development/mapreduce.md
+++ b/_docs-2/development/mapreduce.md
@@ -13,6 +13,21 @@ can be set via these two format classes to do the following:
* Restrict the scan to a range of rows
* Restrict the input to a subset of available columns
+## Configuration
+
+Since 2.0.0, Accumulo no longer has the same versions of dependencies (i.e Guava, etc) as Hadoop.
+When launching a MapReduce job that reads or writes to Accumulo, you should build a shaded jar
+with all of your dependencies and complete the following steps so YARN only includes Hadoop code
+(and not all of Hadoop dependencies) when running your MapReduce job:
+
+1. Set `export HADOOP_USE_CLIENT_CLASSLOADER=true` in your environment before submitting
+ your job with `yarn` command.
+
+2. Set the following in your Job configuration.
+ ```java
+ job.getConfiguration().set("mapreduce.job.classloader", "true");
+ ```
+
## Mapper and Reducer classes
To read from an Accumulo table create a Mapper with the following class