You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by "Josh Elser (JIRA)" <ji...@apache.org> on 2015/03/05 19:50:39 UTC
[jira] [Created] (ACCUMULO-3647) Improve bulk-import sanity check
with HDFS permissions enabled
Josh Elser created ACCUMULO-3647:
------------------------------------
Summary: Improve bulk-import sanity check with HDFS permissions enabled
Key: ACCUMULO-3647
URL: https://issues.apache.org/jira/browse/ACCUMULO-3647
Project: Accumulo
Issue Type: Improvement
Components: tserver
Reporter: Josh Elser
Fix For: 1.7.0
If HDFS permissions are enabled, the ownership of the directories used to bulk import files into Accumulo are important. For example:
{code:java,title=BulkImport.java}
@Override
public Repo<Master> call(final long tid, final Master master) throws Exception {
ExecutorService executor = getThreadPool(master);
final AccumuloConfiguration conf = master.getConfiguration();
VolumeManager fs = master.getFileSystem();
List<FileStatus> files = new ArrayList<FileStatus>();
for (FileStatus entry : fs.listStatus(new Path(bulk))) {
files.add(entry);
}
log.debug("tid " + tid + " importing " + files.size() + " files");
Path writable = new Path(this.errorDir, ".iswritable");
if (!fs.createNewFile(writable)) {
// Maybe this is a re-try... clear the flag and try again
fs.delete(writable);
if (!fs.createNewFile(writable))
throw new ThriftTableOperationException(tableId, null, TableOperation.BULK_IMPORT, TableOperationExceptionType.BULK_BAD_ERROR_DIRECTORY,
"Unable to write to " + this.errorDir);
}
{code}
{{fs.createNewFile(writable)}} will fail if the Accumulo user cannot write to that directory
{noformat}
org.apache.hadoop.security.AccessControlException: Permission denied: user=accumulo, access=WRITE, inode="/tmp/org.apache.accumulo.test.BulkImportVolumeIT/err":admin:hdfs:drwxr-xr-x
{noformat}
We should also be checking the permissions of the directories so that a good error message can be returned instead of what a user would presently see:
{noformat}
org.apache.accumulo.core.client.AccumuloException: Internal error processing waitForFateOperation
at org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:71)
at org.apache.accumulo.core.master.thrift.FateService$Client.recv_waitForFateOperation(FateService.java:174)
at org.apache.accumulo.core.master.thrift.FateService$Client.waitForFateOperation(FateService.java:159)
at org.apache.accumulo.core.client.impl.TableOperationsImpl.waitForFateOperation(TableOperationsImpl.java:280)
at org.apache.accumulo.core.client.impl.TableOperationsImpl.doFateOperation(TableOperationsImpl.java:322)
at org.apache.accumulo.core.client.impl.TableOperationsImpl.doFateOperation(TableOperationsImpl.java:308)
at org.apache.accumulo.core.client.impl.TableOperationsImpl.doTableFateOperation(TableOperationsImpl.java:1630)
at org.apache.accumulo.core.client.impl.TableOperationsImpl.importDirectory(TableOperationsImpl.java:1218)
at org.apache.accumulo.test.BulkImportVolumeIT.testBulkImportFailure(BulkImportVolumeIT.java:90)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)