You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by SRINIVAS SURASANI <va...@gmail.com> on 2011/10/15 10:51:23 UTC
Question on mianframe files
Hi,
I'm downloading mainframe files using FTP in binary mode on to local file
system. These files are now seen as EBCDIC. The information about these
files are
(a) fixed in length ( each
field in record has fixed length).
(b)each record is of some KB
( This KB is fixed for each record).
Now here I'm able to convert this EBCDIC files to ASCII files with in unix
file system using the following command.
dd if <INPUTFILENAME> of <OUTPUTFILENAME> conv=ascii,unblock cbs=150
150 being the record size.
So, here I want this conversion to be done in Hadoop to leverage the use of
parallel processing. I was wondering is there any record reader available
for this kind of files and also about how to convert Packed Decimals COMP(3)
files to ASCII.
Any suggestions on how this can be done.
Srinivas
Re: Question on mianframe files
Posted by Michel Segel <mi...@hotmail.com>.
Ok...
Your best bet is to bite the bullet and forego using dd. What end up doing is staging the files twice ... Raw ebcdic, converted ASCII then move to HDFS...
You would be better off passing in the data definition of the record and convert the files on the fly... Pretty straight forward...
Not sure why you would want to do this in a m/r unless you're going to do more processing than a simple conversion...
Sent from a remote device. Please excuse any typos...
Mike Segel
On Oct 15, 2011, at 10:30 AM, SRINIVAS SURASANI <va...@gmail.com> wrote:
> Yes, we have files which are only EBCDIC and files containing EBCDIC +
> PACKED DECIMALS. As a first step started working on only EBCDIC files. dd
> command working fine but the intention is to do this conversion with in HDFS
> to leverage parallel processing.
>
> On Sat, Oct 15, 2011 at 10:58 AM, Michel Segel <mi...@hotmail.com>wrote:
>
>> You may not want to do this...
>> Does the data contain and packed or zoned decimals?
>> If so, the dd conversion will corrupt your data.
>>
>>
>> Sent from a remote device. Please excuse any typos...
>>
>> Mike Segel
>>
>> On Oct 15, 2011, at 3:51 AM, SRINIVAS SURASANI <va...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I'm downloading mainframe files using FTP in binary mode on to local file
>>> system. These files are now seen as EBCDIC. The information about these
>>> files are
>>> (a) fixed in length ( each
>>> field in record has fixed length).
>>> (b)each record is of some
>> KB
>>> ( This KB is fixed for each record).
>>>
>>> Now here I'm able to convert this EBCDIC files to ASCII files with in
>> unix
>>> file system using the following command.
>>> dd if <INPUTFILENAME> of <OUTPUTFILENAME> conv=ascii,unblock cbs=150
>>> 150 being the record size.
>>>
>>> So, here I want this conversion to be done in Hadoop to leverage the use
>> of
>>> parallel processing. I was wondering is there any record reader available
>>> for this kind of files and also about how to convert Packed Decimals
>> COMP(3)
>>> files to ASCII.
>>>
>>> Any suggestions on how this can be done.
>>>
>>>
>>> Srinivas
>>
Re: Question on mianframe files
Posted by SRINIVAS SURASANI <va...@gmail.com>.
Yes, we have files which are only EBCDIC and files containing EBCDIC +
PACKED DECIMALS. As a first step started working on only EBCDIC files. dd
command working fine but the intention is to do this conversion with in HDFS
to leverage parallel processing.
On Sat, Oct 15, 2011 at 10:58 AM, Michel Segel <mi...@hotmail.com>wrote:
> You may not want to do this...
> Does the data contain and packed or zoned decimals?
> If so, the dd conversion will corrupt your data.
>
>
> Sent from a remote device. Please excuse any typos...
>
> Mike Segel
>
> On Oct 15, 2011, at 3:51 AM, SRINIVAS SURASANI <va...@gmail.com> wrote:
>
> > Hi,
> >
> > I'm downloading mainframe files using FTP in binary mode on to local file
> > system. These files are now seen as EBCDIC. The information about these
> > files are
> > (a) fixed in length ( each
> > field in record has fixed length).
> > (b)each record is of some
> KB
> > ( This KB is fixed for each record).
> >
> > Now here I'm able to convert this EBCDIC files to ASCII files with in
> unix
> > file system using the following command.
> > dd if <INPUTFILENAME> of <OUTPUTFILENAME> conv=ascii,unblock cbs=150
> > 150 being the record size.
> >
> > So, here I want this conversion to be done in Hadoop to leverage the use
> of
> > parallel processing. I was wondering is there any record reader available
> > for this kind of files and also about how to convert Packed Decimals
> COMP(3)
> > files to ASCII.
> >
> > Any suggestions on how this can be done.
> >
> >
> > Srinivas
>
Re: Question on mianframe files
Posted by Michel Segel <mi...@hotmail.com>.
You may not want to do this...
Does the data contain and packed or zoned decimals?
If so, the dd conversion will corrupt your data.
Sent from a remote device. Please excuse any typos...
Mike Segel
On Oct 15, 2011, at 3:51 AM, SRINIVAS SURASANI <va...@gmail.com> wrote:
> Hi,
>
> I'm downloading mainframe files using FTP in binary mode on to local file
> system. These files are now seen as EBCDIC. The information about these
> files are
> (a) fixed in length ( each
> field in record has fixed length).
> (b)each record is of some KB
> ( This KB is fixed for each record).
>
> Now here I'm able to convert this EBCDIC files to ASCII files with in unix
> file system using the following command.
> dd if <INPUTFILENAME> of <OUTPUTFILENAME> conv=ascii,unblock cbs=150
> 150 being the record size.
>
> So, here I want this conversion to be done in Hadoop to leverage the use of
> parallel processing. I was wondering is there any record reader available
> for this kind of files and also about how to convert Packed Decimals COMP(3)
> files to ASCII.
>
> Any suggestions on how this can be done.
>
>
> Srinivas