RE: "sqoop import-mainframe" fails with " No sequential datasetsretrieved"

["Asokan, M" <ma...@syncsort.com>]

Dima,
   The mainframe connection manager was meant to connect to mainframes only.  It expects the FTP server output in the same format as the z/OS FTP server.  It parses the file listing to identify sequential datasets.  You can either emulate this with your FTP server or modify the mainframe connection manager code to support non-mainframe systems.

________________________________
From: Dima Fadeyev [dfadeyev@pragsis.com]
Sent: Tuesday, February 03, 2015 12:21 PM
To: user@sqoop.apache.org
Subject: Re: "sqoop import-mainframe" fails with " No sequential datasetsretrieved"

Hello, Asokan,

Thanks a lot for your reply. I can't test against a mainframe right now. It is not possible to "emulate" a mainframe with a general ftp server?

Best regards,

[cid:part1.02070705.02020507@pragsis.com]
El 03/02/15 a las 17:38, Asokan, M escribió:
Hi Dima,
   First, please make sure that you are indeed connecting to a mainframe host.  Secondly, you need to specify the fully qualified name of the dataset like:
user1234.empleados

Hope it helps.

From: Dima Fadeyev [mailto:dfadeyev@pragsis.com]
Sent: Tuesday, February 03, 2015 9:39 AM
To: user@sqoop.apache.org<ma...@sqoop.apache.org>
Subject: "sqoop import-mainframe" fails with " No sequential datasets retrieved"

Hello, everone,

I'm trying to test "import-mainframe" sqoop command added with this patch: https://issues.apache.org/jira/browse/SQOOP-1272. It is not yet available in any sqoop releases, so I've compiled sqoop from trunk.

As far as I understand, I could test import-mainframe using a normal ftp server. So, I've installed one, created a directory 'empleados' and copied some text files in it. I was hoping sqoop would download the files from my ftp server.

Here is the command I launch and the result:

sqoop import-mainframe --connect 192.168.10.210:21 --dataset empleados --username user1234 --password pass1234 --verbose

5/02/03 15:30:10 INFO mainframe.MainframeDatasetInputFormat: Datasets to transfer from: empleados
15/02/03 15:30:10 DEBUG db.DBConfiguration: Fetching password from job credentials store
220 (vsFTPd 2.2.2)
15/02/03 15:30:10 INFO util.MainframeFTPClientUtils: Connected to 192.168.10.210 on 21
USER *******
331 Please specify the password.
PASS *******
230 Login successful.
TYPE A
200 Switching to ASCII mode.
CWD 'empleados'
250 Directory successfully changed.
PASV
227 Entering Passive Mode (192,168,10,210,251,170).
LIST
150 Here comes the directory listing.
226 Directory send OK.
NOOP
200 NOOP ok.
QUIT
221 Goodbye.
15/02/03 15:30:10 INFO mapreduce.JobSubmitter: Cleaning up the staging area /user/root/.staging/job_1422968174081_0008
15/02/03 15:30:10 WARN security.UserGroupInformation: PriviledgedActionException as:root (auth:SIMPLE) cause:java.io.IOException: No sequential datasets retrieved from empleados
15/02/03 15:30:10 DEBUG util.ClassLoaderStack: Restoring classloader: sun.misc.Launcher$AppClassLoader@530f243b<ma...@530f243b>
15/02/03 15:30:10 ERROR tool.ImportTool: Encountered IOException running import job: java.io.IOException: No sequential datasets retrieved from empleados
at org.apache.sqoop.mapreduce.mainframe.MainframeDatasetInputFormat.getSplits(MainframeDatasetInputFormat.java:65)
at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:493)
at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:510)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:394)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1295)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1292)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1292)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1313)
at org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:186)
at org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:159)
at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:247)
at org.apache.sqoop.manager.MainframeManager.importTable(MainframeManager.java:97)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:497)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)

I could check with tcpdump that my ftp sends a list of files from directory 'empleados' to sqoop. However sqoop somehow sees an empty list.

If anyone has sqoop import-mainframe working, please help me resolve this.

Thanks in advance

--
[X]

AVISO CONFIDENCIAL
Este correo y la información contenida o adjunta al mismo es privada y confidencial y va dirigida exclusivamente a su destinatario. Pragsis informa a quien pueda haber recibido este correo por error que contiene información confidencial cuyo uso, copia, reproducción o distribución está expresamente prohibida. Si no es Vd. el destinatario del mismo y recibe este correo por error, le rogamos lo ponga en conocimiento del emisor y proceda a su eliminación sin copiarlo, imprimirlo o utilizarlo de ningún modo.
CONFIDENTIALITY WARNING.
This message and the information contained in or attached to it are private and confidential and intended exclusively for the addressee. Pragsis informs to whom it may receive it in error that it contains privileged information and its use, copy, reproduction or distribution is prohibited. If you are not an intended recipient of this E-mail, please notify the sender, delete it and do not read, a ct upon, print, disclose, copy, retain or redistribute any portion of this E-mail.

________________________________



ATTENTION: -----

The information contained in this message (including any files transmitted with this message) may contain proprietary, trade secret or other confidential and/or legally privileged information. Any pricing information contained in this message or in any files transmitted with this message is always confidential and cannot be shared with any third parties without prior written approval from Syncsort. This message is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any use, disclosure, copying or distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or Syncsort and destroy all copies of this message in your possession, custody or control.


AVISO CONFIDENCIAL
Este correo y la información contenida o adjunta al mismo es privada y confidencial y va dirigida exclusivamente a su destinatario. Pragsis informa a quien pueda haber recibido este correo por error que contiene información confidencial cuyo uso, copia, reproducción o distribución está expresamente prohibida. Si no es Vd. el destinatario del mismo y recibe este correo por error, le rogamos lo ponga en conocimiento del emisor y proceda a su eliminación sin copiarlo, imprimirlo o utilizarlo de ningún modo.
CONFIDENTIALITY WARNING.
This message and the information contained in or attached to it are private and confidential and intended exclusively for the addressee. Pragsis informs to whom it may receive it in error that it contains privileged information and its use, copy, reproduction or distribution is prohibited. If you are not an intended recipient of this E-mail, please notify the sender, delete it and do not read, act upon, print, disclose, copy, retain or redistribute any portion of this E-mail.