You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@sqoop.apache.org by Mark Roddy <ma...@gmail.com> on 2011/10/21 00:12:53 UTC
Escape Char
I'm moving free form data out of a RDBMS that has a lot of \n, \r\n,
and \t characters.
I used "--escaped-by \\" (extra \ cause of bash), but I'm a little
confused about what to do with this data now. I can't seem to find
any tools that will honor the '\' escape char. TextInputFormat does
not seem to.
I'm working on replacing an existing in house tool w/sqoop that
replace newlines with the literal string '\n'. I'd be happy to do as
such but I don't see any way of doing so.
I'm sure I'm not the first person to run into this so I appreciate any
suggestions.
-Mark
Re: Escape Char
Posted by Arvind Prabhakar <ar...@apache.org>.
+1
On Fri, Oct 21, 2011 at 11:03 AM, Mark Roddy <ma...@gmail.com> wrote:
> Seeing as I'm now depending on this behavior, I nominate that that bug
> be upgraded to feature :-)
>
> -Mark
>
> On Fri, Oct 21, 2011 at 1:38 PM, arvind@cloudera.com
> <ar...@cloudera.com> wrote:
>> Glad it worked Mark!
>>> And it looks like you don't have to do a hive import to use it.
>> That sounds like a bug to me :)
>> Arvind
>>
>> On Fri, Oct 21, 2011 at 9:41 AM, Mark Roddy <ma...@gmail.com> wrote:
>>>
>>> Thanks for the help Arvind. The hive-drop-import-delims worked. And
>>> it looks like you don't have to do a hive import to use it.
>>>
>>> -Mark
>>>
>>>
>>> On Fri, Oct 21, 2011 at 11:43 AM, Arvind Prabhakar <ar...@apache.org>
>>> wrote:
>>> > One work around worth trying is to use the "--hive-drop-import-delims"
>>> > option and do a hive import. With this option set, Sqoop will remove
>>> > any new lines or ^A characters which are the default delimiters used
>>> > for Hive. After the import is done, you could copy the file out of
>>> > Hive directly and use it in your application.
>>> >
>>> > Arvind
>>> >
>>> > On Fri, Oct 21, 2011 at 7:05 AM, Mark Roddy <ma...@gmail.com> wrote:
>>> >> I used "--escaped-by \\" due to bash, so that "\" would be the escape
>>> >> character used. That works fine, I end up with \n and \t characters
>>> >> escaped by '\'.
>>> >>
>>> >>
>>> >> To put the problem more concretely, I have a singe record from the db
>>> >> with a field containing the following value:
>>> >> "foo
>>> >> bar baz
>>> >> biz"
>>> >>
>>> >> Sqoop will spit out:
>>> >> "foo\
>>> >> bar baz\
>>> >> biz"
>>> >>
>>> >>
>>> >> No if I run a map reduce job on this with the TextInputFormat, the
>>> >> record will be terminated after "foo" not after "biz". I did a little
>>> >> digging and TextInputFormat uses LineRecordReader, which uses
>>> >> LineReader which looking at the source, clearly does not honor the
>>> >> escape char. Is there a tool/input format/etc that will read from
>>> >> HDFS and honor this? It does not seem that M/R can do it out of the
>>> >> box. I can't find a way to get Pig. I assume there must be something
>>> >> that will honor the escape, but can not find anything.
>>> >>
>>> >>
>>> >>
>>> >> On Fri, Oct 21, 2011 at 5:26 AM, Alexander C.H. Lorenz
>>> >> <wg...@googlemail.com> wrote:
>>> >>> Hi Mark,
>>> >>> --escaped-by \/ (backslash - slash) tells bash to escape the next
>>> >>> character.
>>> >>> (if I understood you right)
>>> >>> - Alex
>>> >>> On Fri, Oct 21, 2011 at 12:12 AM, Mark Roddy <ma...@gmail.com>
>>> >>> wrote:
>>> >>>>
>>> >>>> I'm moving free form data out of a RDBMS that has a lot of \n, \r\n,
>>> >>>> and \t characters.
>>> >>>>
>>> >>>> I used "--escaped-by \\" (extra \ cause of bash), but I'm a little
>>> >>>> confused about what to do with this data now. I can't seem to find
>>> >>>> any tools that will honor the '\' escape char. TextInputFormat does
>>> >>>> not seem to.
>>> >>>>
>>> >>>> I'm working on replacing an existing in house tool w/sqoop that
>>> >>>> replace newlines with the literal string '\n'. I'd be happy to do as
>>> >>>> such but I don't see any way of doing so.
>>> >>>>
>>> >>>> I'm sure I'm not the first person to run into this so I appreciate
>>> >>>> any
>>> >>>> suggestions.
>>> >>>>
>>> >>>> -Mark
>>> >>>
>>> >>>
>>> >>>
>>> >>> --
>>> >>> Alexander Lorenz
>>> >>> http://mapredit.blogspot.com
>>> >>>
>>> >>>
>>> >>
>>> >
>>
>>
>
Re: Escape Char
Posted by Mark Roddy <ma...@gmail.com>.
Seeing as I'm now depending on this behavior, I nominate that that bug
be upgraded to feature :-)
-Mark
On Fri, Oct 21, 2011 at 1:38 PM, arvind@cloudera.com
<ar...@cloudera.com> wrote:
> Glad it worked Mark!
>> And it looks like you don't have to do a hive import to use it.
> That sounds like a bug to me :)
> Arvind
>
> On Fri, Oct 21, 2011 at 9:41 AM, Mark Roddy <ma...@gmail.com> wrote:
>>
>> Thanks for the help Arvind. The hive-drop-import-delims worked. And
>> it looks like you don't have to do a hive import to use it.
>>
>> -Mark
>>
>>
>> On Fri, Oct 21, 2011 at 11:43 AM, Arvind Prabhakar <ar...@apache.org>
>> wrote:
>> > One work around worth trying is to use the "--hive-drop-import-delims"
>> > option and do a hive import. With this option set, Sqoop will remove
>> > any new lines or ^A characters which are the default delimiters used
>> > for Hive. After the import is done, you could copy the file out of
>> > Hive directly and use it in your application.
>> >
>> > Arvind
>> >
>> > On Fri, Oct 21, 2011 at 7:05 AM, Mark Roddy <ma...@gmail.com> wrote:
>> >> I used "--escaped-by \\" due to bash, so that "\" would be the escape
>> >> character used. That works fine, I end up with \n and \t characters
>> >> escaped by '\'.
>> >>
>> >>
>> >> To put the problem more concretely, I have a singe record from the db
>> >> with a field containing the following value:
>> >> "foo
>> >> bar baz
>> >> biz"
>> >>
>> >> Sqoop will spit out:
>> >> "foo\
>> >> bar baz\
>> >> biz"
>> >>
>> >>
>> >> No if I run a map reduce job on this with the TextInputFormat, the
>> >> record will be terminated after "foo" not after "biz". I did a little
>> >> digging and TextInputFormat uses LineRecordReader, which uses
>> >> LineReader which looking at the source, clearly does not honor the
>> >> escape char. Is there a tool/input format/etc that will read from
>> >> HDFS and honor this? It does not seem that M/R can do it out of the
>> >> box. I can't find a way to get Pig. I assume there must be something
>> >> that will honor the escape, but can not find anything.
>> >>
>> >>
>> >>
>> >> On Fri, Oct 21, 2011 at 5:26 AM, Alexander C.H. Lorenz
>> >> <wg...@googlemail.com> wrote:
>> >>> Hi Mark,
>> >>> --escaped-by \/ (backslash - slash) tells bash to escape the next
>> >>> character.
>> >>> (if I understood you right)
>> >>> - Alex
>> >>> On Fri, Oct 21, 2011 at 12:12 AM, Mark Roddy <ma...@gmail.com>
>> >>> wrote:
>> >>>>
>> >>>> I'm moving free form data out of a RDBMS that has a lot of \n, \r\n,
>> >>>> and \t characters.
>> >>>>
>> >>>> I used "--escaped-by \\" (extra \ cause of bash), but I'm a little
>> >>>> confused about what to do with this data now. I can't seem to find
>> >>>> any tools that will honor the '\' escape char. TextInputFormat does
>> >>>> not seem to.
>> >>>>
>> >>>> I'm working on replacing an existing in house tool w/sqoop that
>> >>>> replace newlines with the literal string '\n'. I'd be happy to do as
>> >>>> such but I don't see any way of doing so.
>> >>>>
>> >>>> I'm sure I'm not the first person to run into this so I appreciate
>> >>>> any
>> >>>> suggestions.
>> >>>>
>> >>>> -Mark
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Alexander Lorenz
>> >>> http://mapredit.blogspot.com
>> >>>
>> >>>
>> >>
>> >
>
>
Re: Escape Char
Posted by "arvind@cloudera.com" <ar...@cloudera.com>.
Glad it worked Mark!
> And it looks like you don't have to do a hive import to use it.
That sounds like a bug to me :)
Arvind
On Fri, Oct 21, 2011 at 9:41 AM, Mark Roddy <ma...@gmail.com> wrote:
> Thanks for the help Arvind. The hive-drop-import-delims worked. And
> it looks like you don't have to do a hive import to use it.
>
> -Mark
>
>
> On Fri, Oct 21, 2011 at 11:43 AM, Arvind Prabhakar <ar...@apache.org>
> wrote:
> > One work around worth trying is to use the "--hive-drop-import-delims"
> > option and do a hive import. With this option set, Sqoop will remove
> > any new lines or ^A characters which are the default delimiters used
> > for Hive. After the import is done, you could copy the file out of
> > Hive directly and use it in your application.
> >
> > Arvind
> >
> > On Fri, Oct 21, 2011 at 7:05 AM, Mark Roddy <ma...@gmail.com> wrote:
> >> I used "--escaped-by \\" due to bash, so that "\" would be the escape
> >> character used. That works fine, I end up with \n and \t characters
> >> escaped by '\'.
> >>
> >>
> >> To put the problem more concretely, I have a singe record from the db
> >> with a field containing the following value:
> >> "foo
> >> bar baz
> >> biz"
> >>
> >> Sqoop will spit out:
> >> "foo\
> >> bar baz\
> >> biz"
> >>
> >>
> >> No if I run a map reduce job on this with the TextInputFormat, the
> >> record will be terminated after "foo" not after "biz". I did a little
> >> digging and TextInputFormat uses LineRecordReader, which uses
> >> LineReader which looking at the source, clearly does not honor the
> >> escape char. Is there a tool/input format/etc that will read from
> >> HDFS and honor this? It does not seem that M/R can do it out of the
> >> box. I can't find a way to get Pig. I assume there must be something
> >> that will honor the escape, but can not find anything.
> >>
> >>
> >>
> >> On Fri, Oct 21, 2011 at 5:26 AM, Alexander C.H. Lorenz
> >> <wg...@googlemail.com> wrote:
> >>> Hi Mark,
> >>> --escaped-by \/ (backslash - slash) tells bash to escape the next
> character.
> >>> (if I understood you right)
> >>> - Alex
> >>> On Fri, Oct 21, 2011 at 12:12 AM, Mark Roddy <ma...@gmail.com>
> wrote:
> >>>>
> >>>> I'm moving free form data out of a RDBMS that has a lot of \n, \r\n,
> >>>> and \t characters.
> >>>>
> >>>> I used "--escaped-by \\" (extra \ cause of bash), but I'm a little
> >>>> confused about what to do with this data now. I can't seem to find
> >>>> any tools that will honor the '\' escape char. TextInputFormat does
> >>>> not seem to.
> >>>>
> >>>> I'm working on replacing an existing in house tool w/sqoop that
> >>>> replace newlines with the literal string '\n'. I'd be happy to do as
> >>>> such but I don't see any way of doing so.
> >>>>
> >>>> I'm sure I'm not the first person to run into this so I appreciate any
> >>>> suggestions.
> >>>>
> >>>> -Mark
> >>>
> >>>
> >>>
> >>> --
> >>> Alexander Lorenz
> >>> http://mapredit.blogspot.com
> >>>
> >>>
> >>
> >
>
Re: Escape Char
Posted by Mark Roddy <ma...@gmail.com>.
Thanks for the help Arvind. The hive-drop-import-delims worked. And
it looks like you don't have to do a hive import to use it.
-Mark
On Fri, Oct 21, 2011 at 11:43 AM, Arvind Prabhakar <ar...@apache.org> wrote:
> One work around worth trying is to use the "--hive-drop-import-delims"
> option and do a hive import. With this option set, Sqoop will remove
> any new lines or ^A characters which are the default delimiters used
> for Hive. After the import is done, you could copy the file out of
> Hive directly and use it in your application.
>
> Arvind
>
> On Fri, Oct 21, 2011 at 7:05 AM, Mark Roddy <ma...@gmail.com> wrote:
>> I used "--escaped-by \\" due to bash, so that "\" would be the escape
>> character used. That works fine, I end up with \n and \t characters
>> escaped by '\'.
>>
>>
>> To put the problem more concretely, I have a singe record from the db
>> with a field containing the following value:
>> "foo
>> bar baz
>> biz"
>>
>> Sqoop will spit out:
>> "foo\
>> bar baz\
>> biz"
>>
>>
>> No if I run a map reduce job on this with the TextInputFormat, the
>> record will be terminated after "foo" not after "biz". I did a little
>> digging and TextInputFormat uses LineRecordReader, which uses
>> LineReader which looking at the source, clearly does not honor the
>> escape char. Is there a tool/input format/etc that will read from
>> HDFS and honor this? It does not seem that M/R can do it out of the
>> box. I can't find a way to get Pig. I assume there must be something
>> that will honor the escape, but can not find anything.
>>
>>
>>
>> On Fri, Oct 21, 2011 at 5:26 AM, Alexander C.H. Lorenz
>> <wg...@googlemail.com> wrote:
>>> Hi Mark,
>>> --escaped-by \/ (backslash - slash) tells bash to escape the next character.
>>> (if I understood you right)
>>> - Alex
>>> On Fri, Oct 21, 2011 at 12:12 AM, Mark Roddy <ma...@gmail.com> wrote:
>>>>
>>>> I'm moving free form data out of a RDBMS that has a lot of \n, \r\n,
>>>> and \t characters.
>>>>
>>>> I used "--escaped-by \\" (extra \ cause of bash), but I'm a little
>>>> confused about what to do with this data now. I can't seem to find
>>>> any tools that will honor the '\' escape char. TextInputFormat does
>>>> not seem to.
>>>>
>>>> I'm working on replacing an existing in house tool w/sqoop that
>>>> replace newlines with the literal string '\n'. I'd be happy to do as
>>>> such but I don't see any way of doing so.
>>>>
>>>> I'm sure I'm not the first person to run into this so I appreciate any
>>>> suggestions.
>>>>
>>>> -Mark
>>>
>>>
>>>
>>> --
>>> Alexander Lorenz
>>> http://mapredit.blogspot.com
>>>
>>>
>>
>
Re: Escape Char
Posted by Arvind Prabhakar <ar...@apache.org>.
One work around worth trying is to use the "--hive-drop-import-delims"
option and do a hive import. With this option set, Sqoop will remove
any new lines or ^A characters which are the default delimiters used
for Hive. After the import is done, you could copy the file out of
Hive directly and use it in your application.
Arvind
On Fri, Oct 21, 2011 at 7:05 AM, Mark Roddy <ma...@gmail.com> wrote:
> I used "--escaped-by \\" due to bash, so that "\" would be the escape
> character used. That works fine, I end up with \n and \t characters
> escaped by '\'.
>
>
> To put the problem more concretely, I have a singe record from the db
> with a field containing the following value:
> "foo
> bar baz
> biz"
>
> Sqoop will spit out:
> "foo\
> bar baz\
> biz"
>
>
> No if I run a map reduce job on this with the TextInputFormat, the
> record will be terminated after "foo" not after "biz". I did a little
> digging and TextInputFormat uses LineRecordReader, which uses
> LineReader which looking at the source, clearly does not honor the
> escape char. Is there a tool/input format/etc that will read from
> HDFS and honor this? It does not seem that M/R can do it out of the
> box. I can't find a way to get Pig. I assume there must be something
> that will honor the escape, but can not find anything.
>
>
>
> On Fri, Oct 21, 2011 at 5:26 AM, Alexander C.H. Lorenz
> <wg...@googlemail.com> wrote:
>> Hi Mark,
>> --escaped-by \/ (backslash - slash) tells bash to escape the next character.
>> (if I understood you right)
>> - Alex
>> On Fri, Oct 21, 2011 at 12:12 AM, Mark Roddy <ma...@gmail.com> wrote:
>>>
>>> I'm moving free form data out of a RDBMS that has a lot of \n, \r\n,
>>> and \t characters.
>>>
>>> I used "--escaped-by \\" (extra \ cause of bash), but I'm a little
>>> confused about what to do with this data now. I can't seem to find
>>> any tools that will honor the '\' escape char. TextInputFormat does
>>> not seem to.
>>>
>>> I'm working on replacing an existing in house tool w/sqoop that
>>> replace newlines with the literal string '\n'. I'd be happy to do as
>>> such but I don't see any way of doing so.
>>>
>>> I'm sure I'm not the first person to run into this so I appreciate any
>>> suggestions.
>>>
>>> -Mark
>>
>>
>>
>> --
>> Alexander Lorenz
>> http://mapredit.blogspot.com
>>
>>
>
Re: Escape Char
Posted by Mark Roddy <ma...@gmail.com>.
I used "--escaped-by \\" due to bash, so that "\" would be the escape
character used. That works fine, I end up with \n and \t characters
escaped by '\'.
To put the problem more concretely, I have a singe record from the db
with a field containing the following value:
"foo
bar baz
biz"
Sqoop will spit out:
"foo\
bar baz\
biz"
No if I run a map reduce job on this with the TextInputFormat, the
record will be terminated after "foo" not after "biz". I did a little
digging and TextInputFormat uses LineRecordReader, which uses
LineReader which looking at the source, clearly does not honor the
escape char. Is there a tool/input format/etc that will read from
HDFS and honor this? It does not seem that M/R can do it out of the
box. I can't find a way to get Pig. I assume there must be something
that will honor the escape, but can not find anything.
On Fri, Oct 21, 2011 at 5:26 AM, Alexander C.H. Lorenz
<wg...@googlemail.com> wrote:
> Hi Mark,
> --escaped-by \/ (backslash - slash) tells bash to escape the next character.
> (if I understood you right)
> - Alex
> On Fri, Oct 21, 2011 at 12:12 AM, Mark Roddy <ma...@gmail.com> wrote:
>>
>> I'm moving free form data out of a RDBMS that has a lot of \n, \r\n,
>> and \t characters.
>>
>> I used "--escaped-by \\" (extra \ cause of bash), but I'm a little
>> confused about what to do with this data now. I can't seem to find
>> any tools that will honor the '\' escape char. TextInputFormat does
>> not seem to.
>>
>> I'm working on replacing an existing in house tool w/sqoop that
>> replace newlines with the literal string '\n'. I'd be happy to do as
>> such but I don't see any way of doing so.
>>
>> I'm sure I'm not the first person to run into this so I appreciate any
>> suggestions.
>>
>> -Mark
>
>
>
> --
> Alexander Lorenz
> http://mapredit.blogspot.com
>
>
Re: Escape Char
Posted by "Alexander C.H. Lorenz" <wg...@googlemail.com>.
Hi Mark,
--escaped-by \/ (backslash - slash) tells bash to escape the next character.
(if I understood you right)
- Alex
On Fri, Oct 21, 2011 at 12:12 AM, Mark Roddy <ma...@gmail.com> wrote:
> I'm moving free form data out of a RDBMS that has a lot of \n, \r\n,
> and \t characters.
>
> I used "--escaped-by \\" (extra \ cause of bash), but I'm a little
> confused about what to do with this data now. I can't seem to find
> any tools that will honor the '\' escape char. TextInputFormat does
> not seem to.
>
> I'm working on replacing an existing in house tool w/sqoop that
> replace newlines with the literal string '\n'. I'd be happy to do as
> such but I don't see any way of doing so.
>
> I'm sure I'm not the first person to run into this so I appreciate any
> suggestions.
>
> -Mark
>
--
Alexander Lorenz
http://mapredit.blogspot.com
Re: Escape Char
Posted by Ken Krugler <kk...@transpac.com>.
Hi Mark,
On Oct 21, 2011, at 12:12am, Mark Roddy wrote:
> I'm moving free form data out of a RDBMS that has a lot of \n, \r\n,
> and \t characters.
>
> I used "--escaped-by \\" (extra \ cause of bash), but I'm a little
> confused about what to do with this data now. I can't seem to find
> any tools that will honor the '\' escape char. TextInputFormat does
> not seem to.
>
> I'm working on replacing an existing in house tool w/sqoop that
> replace newlines with the literal string '\n'. I'd be happy to do as
> such but I don't see any way of doing so.
>
> I'm sure I'm not the first person to run into this so I appreciate any
> suggestions.
We use CSVParser (au.com.bytecode.opencsv.CSVParser) to parse escaped text that we're importing using Sqoop.
We're using a custom Cascading Function to do this processing during our workflow.
-- Ken
--------------------------
Ken Krugler
+1 530-210-6378
http://bixolabs.com
custom big data solutions & training
Hadoop, Cascading, Mahout & Solr