sc.textFile can't recognize '\004'

[anny9699 <an...@gmail.com>]

Hi,

I need to parse a file which is separated by a series of separators. I used
SparkContext.textFile and I met two problems:

1) One of the separators is '\004', which could be recognized by python or R
or Hive, however Spark seems can't recognize this one and returns a symbol
looking like '?'. Also this symbol is not a question mark and I don't know
how to parse.

2) Some of the separator are composed of several Chars, like "} =>". If I
use str.split(Array('}', '=>')), it will separate the string but with many
white spaces included in the middle. Is there a good way that I could
separate by String instead of by Array of Chars? 

Thanks a lot!



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/sc-textFile-can-t-recognize-004-tp8059.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: sc.textFile can't recognize '\004'

[anny9699 <an...@gmail.com>]

Thanks a lot Sean! It works now for me now~~



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/sc-textFile-can-t-recognize-004-tp8059p8071.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: sc.textFile can't recognize '\004'

[Sean Owen <so...@cloudera.com>]

These are actually Scala / Java questions.

On Sat, Jun 21, 2014 at 1:08 AM, anny9699 <an...@gmail.com> wrote:
> 1) One of the separators is '\004', which could be recognized by python or R
> or Hive, however Spark seems can't recognize this one and returns a symbol
> looking like '?'. Also this symbol is not a question mark and I don't know
> how to parse.

(The \004 octal syntax appears deprecated, but it works.) It's not
turned into ?, it is just how the shell shows non-printing characters.

scala> val c = '\004'
warning: there were 1 deprecation warning(s); re-run with -deprecation
for details
c: Char = ?

scala> c.toInt
res2: Int = 4

Which is all correct. Is it presenting any problem?

> 2) Some of the separator are composed of several Chars, like "} =>". If I
> use str.split(Array('}', '=>')), it will separate the string but with many
> white spaces included in the middle. Is there a good way that I could
> separate by String instead of by Array of Chars?

Your example doesn't compile but I assume the argument should be an
array of the 3 chars. String.split will return an empty match between
tokens. If you don't want them, you can
str.split(...).filterNot(_.isEmpty)