You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Jerry OELoo <oy...@gmail.com> on 2015/08/19 13:10:07 UTC
Spark return key value pair
Hi.
I want to parse a file and return a key-value pair with pySpark, but
result is strange to me.
the test.sql is a big fie and each line is usename and password, with
# between them, I use below mapper2 to map data, and in my
understanding, i in words.take(10) should be a tuple, but the result
is that i is username or password, this is strange for me to
understand, Thanks for you help.
def mapper2(line):
words = line.split('#')
return (words[0].strip(), words[1].strip())
def main2(sc):
lines = sc.textFile("hdfs://master:9000/spark/test.sql")
words = lines.flatMap(mapper2)
for i in words.take(10):
msg = i + ":" + "\n"
--
Rejoice,I Desire!
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark return key value pair
Posted by Robin East <ro...@xense.co.uk>.
Dawid is right, if you did words.count it would be twice the number of input lines. You can use map like this:
words = lines.map(mapper2)
for i in words.take(10):
msg = i[0] + ":” + i[1] + "\n”
-------------------------------------------------------------------------------
Robin East
Spark GraphX in Action Michael Malak and Robin East
Manning Publications Co.
http://www.manning.com/malak/ <http://www.manning.com/malak/>
> On 19 Aug 2015, at 12:19, Dawid Wysakowicz <wy...@gmail.com> wrote:
>
> I am not 100% sure but probably flatMap unwinds the tuples. Try with map instead.
>
> 2015-08-19 13:10 GMT+02:00 Jerry OELoo <oyljerry@gmail.com <ma...@gmail.com>>:
> Hi.
> I want to parse a file and return a key-value pair with pySpark, but
> result is strange to me.
> the test.sql is a big fie and each line is usename and password, with
> # between them, I use below mapper2 to map data, and in my
> understanding, i in words.take(10) should be a tuple, but the result
> is that i is username or password, this is strange for me to
> understand, Thanks for you help.
>
> def mapper2(line):
>
> words = line.split('#')
> return (words[0].strip(), words[1].strip())
>
> def main2(sc):
>
> lines = sc.textFile("hdfs://master:9000/spark/test.sql")
> words = lines.flatMap(mapper2)
>
> for i in words.take(10):
> msg = i + ":" + "\n"
>
>
> --
> Rejoice,I Desire!
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org <ma...@spark.apache.org>
> For additional commands, e-mail: user-help@spark.apache.org <ma...@spark.apache.org>
>
>
Re: Spark return key value pair
Posted by Dawid Wysakowicz <wy...@gmail.com>.
I am not 100% sure but probably flatMap unwinds the tuples. Try with map
instead.
2015-08-19 13:10 GMT+02:00 Jerry OELoo <oy...@gmail.com>:
> Hi.
> I want to parse a file and return a key-value pair with pySpark, but
> result is strange to me.
> the test.sql is a big fie and each line is usename and password, with
> # between them, I use below mapper2 to map data, and in my
> understanding, i in words.take(10) should be a tuple, but the result
> is that i is username or password, this is strange for me to
> understand, Thanks for you help.
>
> def mapper2(line):
>
> words = line.split('#')
> return (words[0].strip(), words[1].strip())
>
> def main2(sc):
>
> lines = sc.textFile("hdfs://master:9000/spark/test.sql")
> words = lines.flatMap(mapper2)
>
> for i in words.take(10):
> msg = i + ":" + "\n"
>
>
> --
> Rejoice,I Desire!
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>