You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Sergey Bahchissaraitsev (JIRA)" <ji...@apache.org> on 2017/06/19 16:23:00 UTC
[jira] [Commented] (HADOOP-5732) Add SFTP FileSystem
[ https://issues.apache.org/jira/browse/HADOOP-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16054270#comment-16054270 ]
Sergey Bahchissaraitsev commented on HADOOP-5732:
-------------------------------------------------
I have both configured on Hadoop 2.8.0:
{code:java}
fs.AbstractFileSystem.sftp.impl=org.apache.hadoop.fs.sftp.SFTPFileSystem
fs.sftp.impl=org.apache.hadoop.fs.sftp.SFTPFileSystem
{code}
I get the following error:
{code:java}
org.apache.hadoop.mapred.YarnChild: Exception running child : java.io.IOException: Seek not supported
at org.apache.hadoop.fs.sftp.SFTPInputStream.seek(SFTPInputStream.java:66)
at org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:65)
at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:114)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.initialize(PigRecordReader.java:181)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:548)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:786)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:169)
{code}
By looking at the code, it looks like the seek method is unsupported. Not sure what I am missing:
https://github.com/apache/hadoop/blob/release-2.8.0-RC3/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPInputStream.java#L66
> Add SFTP FileSystem
> -------------------
>
> Key: HADOOP-5732
> URL: https://issues.apache.org/jira/browse/HADOOP-5732
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs
> Environment: Any environment
> Reporter: Íñigo Goiri
> Assignee: ramtin
> Priority: Minor
> Fix For: 2.8.0, 3.0.0-alpha1
>
> Attachments: HADOOP-5732.008.patch, HADOOP-5732.009.patch, HADOOP-5732.010.patch, HADOOP-5732.patch, HADOOP-5732.patch, HADOOP-5732.patch, HADOOP-5732.patch, HADOOP-5732.patch, ivy-for-hadoop-7532.patch, ivy-for-hadoop-7532.patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
> I have implemented a FileSystem that supports SFTP. It uses JSch (http://www.jcraft.com/jsch/) in order to manage SFTP.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org