You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by "Jordi (JIRA)" <ji...@apache.org> on 2016/08/10 14:50:20 UTC
[jira] [Comment Edited] (SQOOP-2997) --password-file option
triggers FileSystemClosed exception at end of Oozie action
[ https://issues.apache.org/jira/browse/SQOOP-2997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15415380#comment-15415380 ]
Jordi edited comment on SQOOP-2997 at 8/10/16 2:49 PM:
-------------------------------------------------------
This is my fix proposal, already tested in my project.
org.apache.sqoop.util.password.CryptoFileLoader
Current sqoop-1.4.6 code:
{code}
@Override
public String loadPassword(String p, Configuration configuration) throws IOException {
LOG.debug("Fetching password from specified path: " + p);
Path path = new Path(p);
FileSystem fs = path.getFileSystem(configuration);
byte [] encrypted;
try {
verifyPath(fs, path);
encrypted = readBytes(fs, path);
} finally {
fs.close();
}
{code}
Fix proposal:
@Override
public String loadPassword(String p, Configuration configuration) throws IOException {
LOG.debug("Fetching password from specified path: " + p);
Path path = new Path(p);
FileSystem fs = path.getFileSystem(configuration);
// Not closing FileSystem object because of SQOOP-2997
verifyPath(fs, path);
byte [] encrypted = readBytes(fs, path);
...
was (Author: jordirodri):
This is my fix proposal, already tested in my project.
org.apache.sqoop.util.password.CryptoFileLoader
Current sqoop-1.4.6 code:
@Override
public String loadPassword(String p, Configuration configuration) throws IOException {
LOG.debug("Fetching password from specified path: " + p);
Path path = new Path(p);
FileSystem fs = path.getFileSystem(configuration);
byte [] encrypted;
try {
verifyPath(fs, path);
encrypted = readBytes(fs, path);
} finally {
fs.close();
}
Fix proposal:
@Override
public String loadPassword(String p, Configuration configuration) throws IOException {
LOG.debug("Fetching password from specified path: " + p);
Path path = new Path(p);
FileSystem fs = path.getFileSystem(configuration);
// Not closing FileSystem object because of SQOOP-2997
verifyPath(fs, path);
byte [] encrypted = readBytes(fs, path);
...
> --password-file option triggers FileSystemClosed exception at end of Oozie action
> ---------------------------------------------------------------------------------
>
> Key: SQOOP-2997
> URL: https://issues.apache.org/jira/browse/SQOOP-2997
> Project: Sqoop
> Issue Type: Bug
> Affects Versions: 1.4.6
> Environment: Java 1.8 + CDH5.5.1
> Reporter: Jordi
>
> When using the --password-file option triggers FileSystemClosed exception at end of Oozie action.
> This error was fixed for FilePasswordLoader class for sqoop-1.4.3, but it is also happening at CryptoFileLoader which extends from the fixed one.
> https://issues.apache.org/jira/browse/SQOOP-1226
> In this case we are using --password-file option with an encrypted file so we need to use CryptoFileLoder.
> Error LOG:
> Job commit failed: java.io.IOException: Filesystem closed
> at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:837)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1720)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1662)
> at org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:404)
> at org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:400)
> at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:400)
> at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:343)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:917)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:898)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:795)
> at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.touchz(CommitterEventHandler.java:265)
> at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:271)
> at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:237)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Average Map Time 57sec
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)