You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GenTang <gi...@git.apache.org> on 2015/01/07 03:14:31 UTC

[GitHub] spark pull request: The improvement of python converter for hbase(...

GitHub user GenTang opened a pull request:

    https://github.com/apache/spark/pull/3920

    The improvement of python converter for hbase(examples)

    Hi,
    
    Following the discussion in http://apache-spark-developers-list.1001551.n3.nabble.com/python-converter-in-HBaseConverter-scala-spark-examples-td10001.html. I made some modification in three files in package examples:
    1. HBaseConverters.scala: the new converter will converts all the records in an hbase results into a single string
    2. hbase_input.py: as the value string may contain several records, we can use ast package to convert the string into dict
    3. HBaseTest.scala: as the package examples use hbase 0.98.7 the original constructor HTableDescriptor is deprecated. The updation to new constructor is made


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/GenTang/spark master

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/3920.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3920
    
----
commit 3253b61e4a740233154c54458fa68dbec5a2b004
Author: GenTang <ge...@gmail.com>
Date:   2015-01-07T02:04:13Z

    the modification accompanying the improvement of pythonconverter

commit ceb31c5e1619e56b91a8d03b4edd09c7c17813ce
Author: GenTang <ge...@gmail.com>
Date:   2015-01-07T02:05:24Z

    the modification for adapting updation of hbase

commit 5cbbcfc3e4034f824ae94e84374b4f505ee15a63
Author: GenTang <ge...@gmail.com>
Date:   2015-01-07T02:06:00Z

    the improvement of pythonconverter

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5090][examples] The improvement of pyth...

Posted by GenTang <gi...@git.apache.org>.
Github user GenTang commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3920#discussion_r23414030
  
    --- Diff: examples/src/main/python/hbase_inputformat.py ---
    @@ -16,6 +16,7 @@
     #
     
     import sys
    +import ast
     
     from pyspark import SparkContext
     
    --- End diff --
    
    Great idea. I will modify the sample data to show multiple records in a columnFamily and columnFamily:qualifier 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5090][examples] The improvement of pyth...

Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3920#discussion_r24184139
  
    --- Diff: examples/src/main/python/hbase_inputformat.py ---
    @@ -72,6 +75,9 @@
             keyConverter=keyConv,
             valueConverter=valueConv,
             conf=conf)
    +    # hbase_rdd is a RDD[dict]
    --- End diff --
    
    Python people may be not familiar with `RDD[dict]`, how about just `RDD of dict`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5090][examples] The improvement of pyth...

Posted by GenTang <gi...@git.apache.org>.
Github user GenTang commented on the pull request:

    https://github.com/apache/spark/pull/3920#issuecomment-71121501
  
    @MLnick I change the sample data to show that we can have several records in one columnFamily. 
    
    In fact, in HBase 0.96 and newer, the default maximum number of versions(timestamp) has been changed to 1 and in order to show all the versions in a HBase table, we need to add *{"hbase.mapreduce.scan.maxversions":"num"}* in the conf. For me, it is rather a topic of HBase not Spark. That's why I don't show that we can have several versions of records in `Result`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5090][examples] The improvement of pyth...

Posted by GenTang <gi...@git.apache.org>.
Github user GenTang commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3920#discussion_r24188212
  
    --- Diff: examples/src/main/python/hbase_inputformat.py ---
    @@ -72,6 +75,9 @@
             keyConverter=keyConv,
             valueConverter=valueConv,
             conf=conf)
    +    # hbase_rdd is a RDD[dict]
    --- End diff --
    
    I am sorry that I misunderstood your meaning. 
    I thought that you mean we should change the RDD......


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5090][examples] The improvement of pyth...

Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3920#discussion_r25145679
  
    --- Diff: examples/src/main/scala/org/apache/spark/examples/pythonconverters/HBaseConverters.scala ---
    @@ -18,20 +18,34 @@
     package org.apache.spark.examples.pythonconverters
     
     import scala.collection.JavaConversions._
    +import scala.util.parsing.json._
     
     import org.apache.spark.api.python.Converter
     import org.apache.hadoop.hbase.client.{Put, Result}
     import org.apache.hadoop.hbase.io.ImmutableBytesWritable
     import org.apache.hadoop.hbase.util.Bytes
    +import org.apache.hadoop.hbase.KeyValue.Type
    +import org.apache.hadoop.hbase.CellUtil
     
     /**
    - * Implementation of [[org.apache.spark.api.python.Converter]] that converts an
    - * HBase Result to a String
    + * Implementation of [[org.apache.spark.api.python.Converter]] that converts all
    + * the records in an HBase Result to a String
      */
     class HBaseResultToStringConverter extends Converter[Any, String] {
       override def convert(obj: Any): String = {
    +    import collection.JavaConverters._
         val result = obj.asInstanceOf[Result]
    -    Bytes.toStringBinary(result.value())
    +    val output = result.listCells.asScala.map(cell =>
    +        Map(
    +          "row" -> Bytes.toStringBinary(CellUtil.cloneRow(cell)),
    +          "columnFamily" -> Bytes.toStringBinary(CellUtil.cloneFamily(cell)),
    +          "qualifier" -> Bytes.toStringBinary(CellUtil.cloneQualifier(cell)),
    +          "timestamp" -> cell.getTimestamp.toString,
    +          "type" -> Type.codeToType(cell.getTypeByte).toString,
    +          "value" -> Bytes.toStringBinary(CellUtil.cloneValue(cell))
    +        )
    +    )
    +    output.map(JSONObject(_).toString()).mkString("\n")
    --- End diff --
    
    How about  `JSONObject(output).toString()`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5090][examples] The improvement of pyth...

Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3920#discussion_r24719439
  
    --- Diff: examples/src/main/scala/org/apache/spark/examples/pythonconverters/HBaseConverters.scala ---
    @@ -18,20 +18,34 @@
     package org.apache.spark.examples.pythonconverters
     
     import scala.collection.JavaConversions._
    +import scala.util.parsing.json._
     
     import org.apache.spark.api.python.Converter
     import org.apache.hadoop.hbase.client.{Put, Result}
     import org.apache.hadoop.hbase.io.ImmutableBytesWritable
     import org.apache.hadoop.hbase.util.Bytes
    +import org.apache.hadoop.hbase.KeyValue.Type
    +import org.apache.hadoop.hbase.CellUtil
     
     /**
    - * Implementation of [[org.apache.spark.api.python.Converter]] that converts an
    - * HBase Result to a String
    + * Implementation of [[org.apache.spark.api.python.Converter]] that converts all
    + * the records in an HBase Result to an Array[String]
      */
    -class HBaseResultToStringConverter extends Converter[Any, String] {
    -  override def convert(obj: Any): String = {
    +class HBaseResultToStringConverter extends Converter[Any, Array[String]] {
    --- End diff --
    
    I'm wondering that does changing the type from String to Array[String] really work? Currently, PySpark does not support deserialize a RDD of (Any, Array[String]), only (String, String) or (Array[Byte], Array[Byte]) are supported.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5090][examples] The improvement of pyth...

Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:

    https://github.com/apache/spark/pull/3920#issuecomment-69404136
  
    OK to test.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5090][examples] The improvement of pyth...

Posted by MLnick <gi...@git.apache.org>.
Github user MLnick commented on the pull request:

    https://github.com/apache/spark/pull/3920#issuecomment-71085426
  
    @GenTang overall looks fine to me - just noted a few minor comments / questions, and in particular it would be good to expand on the sample data in the example just to show multiple records per `Result`.
    
    I will give it a final once over after you've had a look at the above comments.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5090][examples] The improvement of pyth...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/3920#issuecomment-96769849
  
    Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5090] The improvement of python convert...

Posted by GenTang <gi...@git.apache.org>.
Github user GenTang commented on the pull request:

    https://github.com/apache/spark/pull/3920#issuecomment-68969946
  
    @MLnick 
    Hi Nick Pentreath, please review. Thanks ^^


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5090][examples] The improvement of pyth...

Posted by MLnick <gi...@git.apache.org>.
Github user MLnick commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3920#discussion_r23403597
  
    --- Diff: examples/src/main/python/hbase_inputformat.py ---
    @@ -16,6 +16,7 @@
     #
     
     import sys
    +import ast
     
     from pyspark import SparkContext
     
    --- End diff --
    
    If we're expanding the example to illustrate dealing with `Result` that contains multiple records, it would probably be a good idea to expand the sample data illustrated in the comments below to make use of that feature (i.e. have the sample data have multiple records (column families?).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5090][examples] The improvement of pyth...

Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3920#discussion_r25180080
  
    --- Diff: examples/src/main/scala/org/apache/spark/examples/pythonconverters/HBaseConverters.scala ---
    @@ -18,20 +18,34 @@
     package org.apache.spark.examples.pythonconverters
     
     import scala.collection.JavaConversions._
    +import scala.util.parsing.json._
     
     import org.apache.spark.api.python.Converter
     import org.apache.hadoop.hbase.client.{Put, Result}
     import org.apache.hadoop.hbase.io.ImmutableBytesWritable
     import org.apache.hadoop.hbase.util.Bytes
    +import org.apache.hadoop.hbase.KeyValue.Type
    +import org.apache.hadoop.hbase.CellUtil
     
     /**
    - * Implementation of [[org.apache.spark.api.python.Converter]] that converts an
    - * HBase Result to a String
    + * Implementation of [[org.apache.spark.api.python.Converter]] that converts all
    + * the records in an HBase Result to a String
      */
     class HBaseResultToStringConverter extends Converter[Any, String] {
       override def convert(obj: Any): String = {
    +    import collection.JavaConverters._
         val result = obj.asInstanceOf[Result]
    -    Bytes.toStringBinary(result.value())
    +    val output = result.listCells.asScala.map(cell =>
    +        Map(
    +          "row" -> Bytes.toStringBinary(CellUtil.cloneRow(cell)),
    +          "columnFamily" -> Bytes.toStringBinary(CellUtil.cloneFamily(cell)),
    +          "qualifier" -> Bytes.toStringBinary(CellUtil.cloneQualifier(cell)),
    +          "timestamp" -> cell.getTimestamp.toString,
    +          "type" -> Type.codeToType(cell.getTypeByte).toString,
    +          "value" -> Bytes.toStringBinary(CellUtil.cloneValue(cell))
    +        )
    +    )
    +    output.map(JSONObject(_).toString()).mkString("\n")
    --- End diff --
    
    That make sense. JSON will escape the `\n` in String, so it's safe to have `\n` as separator.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5090][examples] The improvement of pyth...

Posted by GenTang <gi...@git.apache.org>.
Github user GenTang commented on the pull request:

    https://github.com/apache/spark/pull/3920#issuecomment-77252529
  
    Hi, I am sorry to bother you all.
    But is this pull request OK for merging, please?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5090][examples] The improvement of pyth...

Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3920#discussion_r24187753
  
    --- Diff: examples/src/main/python/hbase_inputformat.py ---
    @@ -72,6 +75,9 @@
             keyConverter=keyConv,
             valueConverter=valueConv,
             conf=conf)
    +    # hbase_rdd is a RDD[dict]
    --- End diff --
    
    I think they should have the same meaning, `RDD[dict]` is only a notation in Scala, `RDD of dict` is plain English. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5090][examples] The improvement of pyth...

Posted by GenTang <gi...@git.apache.org>.
Github user GenTang commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3920#discussion_r24495658
  
    --- Diff: examples/src/main/scala/org/apache/spark/examples/pythonconverters/HBaseConverters.scala ---
    @@ -23,15 +23,27 @@ import org.apache.spark.api.python.Converter
     import org.apache.hadoop.hbase.client.{Put, Result}
     import org.apache.hadoop.hbase.io.ImmutableBytesWritable
     import org.apache.hadoop.hbase.util.Bytes
    +import org.apache.hadoop.hbase.KeyValue.Type
    +import org.apache.hadoop.hbase.CellUtil
     
     /**
    - * Implementation of [[org.apache.spark.api.python.Converter]] that converts an
    - * HBase Result to a String
    + * Implementation of [[org.apache.spark.api.python.Converter]] that converts all
    + * the records in an HBase Result to a String
      */
     class HBaseResultToStringConverter extends Converter[Any, String] {
       override def convert(obj: Any): String = {
    +    import collection.JavaConverters._
         val result = obj.asInstanceOf[Result]
    -    Bytes.toStringBinary(result.value())
    +    val output = result.listCells.asScala.map(cell =>
    +        "{'columnFamily':'%s','qualifier':'%s','timestamp':'%s','type':'%s','value':'%s'}".format(
    --- End diff --
    
    @davies 
    Hi, Sorry for the delay.
    Right now, for returning a string in json format is done.
    However, I am thinking let the converter return a `Array[String]`, since there are several records for one row. Here what we are doing now is that we convert the `Buffer[String]` to `string` in the converter and then in python, we reconvert the 'string' to 'list'(or 'tuple'). 
    If the converter return directly Array[String], there are two advantages:
    1. We don't need to do conversion from 'string' to 'list' in python.
    2. Right now, we should add a specific character to aggregate the `Buffer[String]` to a `String`(I am thinking "\n"). Therefore it is safer if we return `Array[String]`
    
    How do you think about this? Thanks 



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5090][examples] The improvement of pyth...

Posted by MLnick <gi...@git.apache.org>.
Github user MLnick commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3920#discussion_r23403050
  
    --- Diff: examples/src/main/scala/org/apache/spark/examples/pythonconverters/HBaseConverters.scala ---
    @@ -23,15 +23,27 @@ import org.apache.spark.api.python.Converter
     import org.apache.hadoop.hbase.client.{Put, Result}
     import org.apache.hadoop.hbase.io.ImmutableBytesWritable
     import org.apache.hadoop.hbase.util.Bytes
    +import org.apache.hadoop.hbase.KeyValue.Type
    +import org.apache.hadoop.hbase.CellUtil
     
     /**
    - * Implementation of [[org.apache.spark.api.python.Converter]] that converts an
    - * HBase Result to a String
    + * Implementation of [[org.apache.spark.api.python.Converter]] that converts all
    + * the records in an HBase Result to a String
      */
     class HBaseResultToStringConverter extends Converter[Any, String] {
       override def convert(obj: Any): String = {
    +    import collection.JavaConverters._
         val result = obj.asInstanceOf[Result]
    -    Bytes.toStringBinary(result.value())
    +    val output = result.listCells.asScala.map(cell =>
    +        "{'columnFamliy':'%s','qualifier':'%s','timestamp':'%s','type':'%s','value':'%s'}".format(
    --- End diff --
    
    minor spelling error of `columnFamily` here


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5090][examples] The improvement of pyth...

Posted by GenTang <gi...@git.apache.org>.
Github user GenTang commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3920#discussion_r25182187
  
    --- Diff: examples/src/main/scala/org/apache/spark/examples/pythonconverters/HBaseConverters.scala ---
    @@ -18,20 +18,34 @@
     package org.apache.spark.examples.pythonconverters
     
     import scala.collection.JavaConversions._
    +import scala.util.parsing.json._
     
     import org.apache.spark.api.python.Converter
     import org.apache.hadoop.hbase.client.{Put, Result}
     import org.apache.hadoop.hbase.io.ImmutableBytesWritable
     import org.apache.hadoop.hbase.util.Bytes
    +import org.apache.hadoop.hbase.KeyValue.Type
    +import org.apache.hadoop.hbase.CellUtil
     
     /**
    - * Implementation of [[org.apache.spark.api.python.Converter]] that converts an
    - * HBase Result to a String
    + * Implementation of [[org.apache.spark.api.python.Converter]] that converts all
    + * the records in an HBase Result to a String
      */
     class HBaseResultToStringConverter extends Converter[Any, String] {
       override def convert(obj: Any): String = {
    +    import collection.JavaConverters._
         val result = obj.asInstanceOf[Result]
    -    Bytes.toStringBinary(result.value())
    +    val output = result.listCells.asScala.map(cell =>
    +        Map(
    +          "row" -> Bytes.toStringBinary(CellUtil.cloneRow(cell)),
    +          "columnFamily" -> Bytes.toStringBinary(CellUtil.cloneFamily(cell)),
    +          "qualifier" -> Bytes.toStringBinary(CellUtil.cloneQualifier(cell)),
    +          "timestamp" -> cell.getTimestamp.toString,
    +          "type" -> Type.codeToType(cell.getTypeByte).toString,
    +          "value" -> Bytes.toStringBinary(CellUtil.cloneValue(cell))
    +        )
    +    )
    +    output.map(JSONObject(_).toString()).mkString("\n")
    --- End diff --
    
    Great! In fact, HBase itself will escape `\n` too. That's why I choose `\n` at the first place.
    Thanks!



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5090][examples] The improvement of pyth...

Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:

    https://github.com/apache/spark/pull/3920#issuecomment-75582752
  
    @GenTang  This PR looks good to me now, thanks
    
    @JoshRosen I think it's ready to go.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5090][examples] The improvement of pyth...

Posted by GenTang <gi...@git.apache.org>.
Github user GenTang commented on the pull request:

    https://github.com/apache/spark/pull/3920#issuecomment-73889175
  
    @davies @MLnick 
    Perhaps it is not a good place to discuss this, but I tried the script hbase_outputformat.py in spark 1.2.0 and it caused `java.lang.IncompatibleClassChangeError: Implementing class`. I guess that it is caused by the conflict between dependency and version of java.
    Do you have any idea about this error? Or do we need create a jira for this?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5090][examples] The improvement of pyth...

Posted by GenTang <gi...@git.apache.org>.
Github user GenTang commented on the pull request:

    https://github.com/apache/spark/pull/3920#issuecomment-74393987
  
    @davies 
    Now we return the string in json format. 
    Therefore the specific characters such as `'` or `"` don't cause problem any more.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5090] The improvement of python convert...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/3920#issuecomment-68969694
  
    Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5090][examples] The improvement of pyth...

Posted by GenTang <gi...@git.apache.org>.
Github user GenTang commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3920#discussion_r23413908
  
    --- Diff: examples/src/main/scala/org/apache/spark/examples/pythonconverters/HBaseConverters.scala ---
    @@ -23,15 +23,27 @@ import org.apache.spark.api.python.Converter
     import org.apache.hadoop.hbase.client.{Put, Result}
     import org.apache.hadoop.hbase.io.ImmutableBytesWritable
     import org.apache.hadoop.hbase.util.Bytes
    +import org.apache.hadoop.hbase.KeyValue.Type
    +import org.apache.hadoop.hbase.CellUtil
     
     /**
    - * Implementation of [[org.apache.spark.api.python.Converter]] that converts an
    - * HBase Result to a String
    + * Implementation of [[org.apache.spark.api.python.Converter]] that converts all
    + * the records in an HBase Result to a String
      */
     class HBaseResultToStringConverter extends Converter[Any, String] {
       override def convert(obj: Any): String = {
    +    import collection.JavaConverters._
         val result = obj.asInstanceOf[Result]
    -    Bytes.toStringBinary(result.value())
    +    val output = result.listCells.asScala.map(cell =>
    +        "{'columnFamliy':'%s','qualifier':'%s','timestamp':'%s','type':'%s','value':'%s'}".format(
    +          Bytes.toStringBinary(CellUtil.cloneFamily(cell)),
    +          Bytes.toStringBinary(CellUtil.cloneQualifier(cell)),
    +          cell.getTimestamp.toString,
    +          Type.codeToType(cell.getTypeByte),
    +          Bytes.toStringBinary(CellUtil.cloneValue(cell))
    +        )
    +    )   
    +    output.mkString(" ")
    --- End diff --
    
    In fact, in HBase a A *{row, columnFamily:qualifier, version}* tuple exactly specifies a cell (a record) in HBase. So there are usually several records per column family in a row. 
    In fact, HBase `Result` contains all the information about a HBase table and `Result.listCells` returns all the cells as a java list of `hbase.Cell` which contains all the information about a cell.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5090][examples] The improvement of pyth...

Posted by GenTang <gi...@git.apache.org>.
Github user GenTang commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3920#discussion_r24187737
  
    --- Diff: examples/src/main/scala/org/apache/spark/examples/pythonconverters/HBaseConverters.scala ---
    @@ -23,15 +23,27 @@ import org.apache.spark.api.python.Converter
     import org.apache.hadoop.hbase.client.{Put, Result}
     import org.apache.hadoop.hbase.io.ImmutableBytesWritable
     import org.apache.hadoop.hbase.util.Bytes
    +import org.apache.hadoop.hbase.KeyValue.Type
    +import org.apache.hadoop.hbase.CellUtil
     
     /**
    - * Implementation of [[org.apache.spark.api.python.Converter]] that converts an
    - * HBase Result to a String
    + * Implementation of [[org.apache.spark.api.python.Converter]] that converts all
    + * the records in an HBase Result to a String
      */
     class HBaseResultToStringConverter extends Converter[Any, String] {
       override def convert(obj: Any): String = {
    +    import collection.JavaConverters._
         val result = obj.asInstanceOf[Result]
    -    Bytes.toStringBinary(result.value())
    +    val output = result.listCells.asScala.map(cell =>
    +        "{'columnFamily':'%s','qualifier':'%s','timestamp':'%s','type':'%s','value':'%s'}".format(
    --- End diff --
    
    Aha,... I didn't consider that. 
    You are right. It is not safe to use ast.literal_val.
    For your suggestion, you mean that we transform string into json in python, or the converter returns a json? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5090][examples] The improvement of pyth...

Posted by GenTang <gi...@git.apache.org>.
Github user GenTang commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3920#discussion_r24727321
  
    --- Diff: examples/src/main/scala/org/apache/spark/examples/pythonconverters/HBaseConverters.scala ---
    @@ -18,20 +18,34 @@
     package org.apache.spark.examples.pythonconverters
     
     import scala.collection.JavaConversions._
    +import scala.util.parsing.json._
     
     import org.apache.spark.api.python.Converter
     import org.apache.hadoop.hbase.client.{Put, Result}
     import org.apache.hadoop.hbase.io.ImmutableBytesWritable
     import org.apache.hadoop.hbase.util.Bytes
    +import org.apache.hadoop.hbase.KeyValue.Type
    +import org.apache.hadoop.hbase.CellUtil
     
     /**
    - * Implementation of [[org.apache.spark.api.python.Converter]] that converts an
    - * HBase Result to a String
    + * Implementation of [[org.apache.spark.api.python.Converter]] that converts all
    + * the records in an HBase Result to an Array[String]
      */
    -class HBaseResultToStringConverter extends Converter[Any, String] {
    -  override def convert(obj: Any): String = {
    +class HBaseResultToStringConverter extends Converter[Any, Array[String]] {
    --- End diff --
    
    You are right.
    Before, I test the code in local mode where the code passed. However, in cluster mode, it doesn't


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5090][examples] The improvement of pyth...

Posted by GenTang <gi...@git.apache.org>.
Github user GenTang commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3920#discussion_r24185498
  
    --- Diff: examples/src/main/python/hbase_inputformat.py ---
    @@ -72,6 +75,9 @@
             keyConverter=keyConv,
             valueConverter=valueConv,
             conf=conf)
    +    # hbase_rdd is a RDD[dict]
    --- End diff --
    
    In fact, every record in dict is a record of HBase. 
    Therefore by using RDD[dict], the element of RDD corresponds a record in HBase. If we use RDD of dict, it is impossible to tell a certain value belongs to which row, which column, etc.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5090][examples] The improvement of pyth...

Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3920#discussion_r24184650
  
    --- Diff: examples/src/main/scala/org/apache/spark/examples/pythonconverters/HBaseConverters.scala ---
    @@ -23,15 +23,27 @@ import org.apache.spark.api.python.Converter
     import org.apache.hadoop.hbase.client.{Put, Result}
     import org.apache.hadoop.hbase.io.ImmutableBytesWritable
     import org.apache.hadoop.hbase.util.Bytes
    +import org.apache.hadoop.hbase.KeyValue.Type
    +import org.apache.hadoop.hbase.CellUtil
     
     /**
    - * Implementation of [[org.apache.spark.api.python.Converter]] that converts an
    - * HBase Result to a String
    + * Implementation of [[org.apache.spark.api.python.Converter]] that converts all
    + * the records in an HBase Result to a String
      */
     class HBaseResultToStringConverter extends Converter[Any, String] {
       override def convert(obj: Any): String = {
    +    import collection.JavaConverters._
         val result = obj.asInstanceOf[Result]
    -    Bytes.toStringBinary(result.value())
    +    val output = result.listCells.asScala.map(cell =>
    +        "{'columnFamily':'%s','qualifier':'%s','timestamp':'%s','type':'%s','value':'%s'}".format(
    --- End diff --
    
    Is it possible that `cell` will have `'` in it? Then the string will be broken (can not be parsed by ast. literal_eval).
    
    Is it any easy and safe to do the serialization? Json?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5090][examples] The improvement of pyth...

Posted by MLnick <gi...@git.apache.org>.
Github user MLnick commented on the pull request:

    https://github.com/apache/spark/pull/3920#issuecomment-71476341
  
    @GenTang ok, thanks for updating the example data. I checked out the PR and tested quickly locally, the new example and data works for me. Looks good.
    
    @davies  +1 to merge.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5090][examples] The improvement of pyth...

Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3920#discussion_r24188162
  
    --- Diff: examples/src/main/scala/org/apache/spark/examples/pythonconverters/HBaseConverters.scala ---
    @@ -23,15 +23,27 @@ import org.apache.spark.api.python.Converter
     import org.apache.hadoop.hbase.client.{Put, Result}
     import org.apache.hadoop.hbase.io.ImmutableBytesWritable
     import org.apache.hadoop.hbase.util.Bytes
    +import org.apache.hadoop.hbase.KeyValue.Type
    +import org.apache.hadoop.hbase.CellUtil
     
     /**
    - * Implementation of [[org.apache.spark.api.python.Converter]] that converts an
    - * HBase Result to a String
    + * Implementation of [[org.apache.spark.api.python.Converter]] that converts all
    + * the records in an HBase Result to a String
      */
     class HBaseResultToStringConverter extends Converter[Any, String] {
       override def convert(obj: Any): String = {
    +    import collection.JavaConverters._
         val result = obj.asInstanceOf[Result]
    -    Bytes.toStringBinary(result.value())
    +    val output = result.listCells.asScala.map(cell =>
    +        "{'columnFamily':'%s','qualifier':'%s','timestamp':'%s','type':'%s','value':'%s'}".format(
    --- End diff --
    
    Also, we can bypassing the decoding in Python, then user could get a DataFrame by call sqlCtx.JsonRDD()


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5090][examples] The improvement of pyth...

Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3920#discussion_r24187984
  
    --- Diff: examples/src/main/scala/org/apache/spark/examples/pythonconverters/HBaseConverters.scala ---
    @@ -23,15 +23,27 @@ import org.apache.spark.api.python.Converter
     import org.apache.hadoop.hbase.client.{Put, Result}
     import org.apache.hadoop.hbase.io.ImmutableBytesWritable
     import org.apache.hadoop.hbase.util.Bytes
    +import org.apache.hadoop.hbase.KeyValue.Type
    +import org.apache.hadoop.hbase.CellUtil
     
     /**
    - * Implementation of [[org.apache.spark.api.python.Converter]] that converts an
    - * HBase Result to a String
    + * Implementation of [[org.apache.spark.api.python.Converter]] that converts all
    + * the records in an HBase Result to a String
      */
     class HBaseResultToStringConverter extends Converter[Any, String] {
       override def convert(obj: Any): String = {
    +    import collection.JavaConverters._
         val result = obj.asInstanceOf[Result]
    -    Bytes.toStringBinary(result.value())
    +    val output = result.listCells.asScala.map(cell =>
    +        "{'columnFamily':'%s','qualifier':'%s','timestamp':'%s','type':'%s','value':'%s'}".format(
    --- End diff --
    
    We could the converter still return a string, but in json format, then we could decode them in Python using simplejson. Json library will do the escaping for us.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5090][examples] The improvement of pyth...

Posted by GenTang <gi...@git.apache.org>.
Github user GenTang commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3920#discussion_r24188381
  
    --- Diff: examples/src/main/scala/org/apache/spark/examples/pythonconverters/HBaseConverters.scala ---
    @@ -23,15 +23,27 @@ import org.apache.spark.api.python.Converter
     import org.apache.hadoop.hbase.client.{Put, Result}
     import org.apache.hadoop.hbase.io.ImmutableBytesWritable
     import org.apache.hadoop.hbase.util.Bytes
    +import org.apache.hadoop.hbase.KeyValue.Type
    +import org.apache.hadoop.hbase.CellUtil
     
     /**
    - * Implementation of [[org.apache.spark.api.python.Converter]] that converts an
    - * HBase Result to a String
    + * Implementation of [[org.apache.spark.api.python.Converter]] that converts all
    + * the records in an HBase Result to a String
      */
     class HBaseResultToStringConverter extends Converter[Any, String] {
       override def convert(obj: Any): String = {
    +    import collection.JavaConverters._
         val result = obj.asInstanceOf[Result]
    -    Bytes.toStringBinary(result.value())
    +    val output = result.listCells.asScala.map(cell =>
    +        "{'columnFamily':'%s','qualifier':'%s','timestamp':'%s','type':'%s','value':'%s'}".format(
    --- End diff --
    
    Great, I will do that.Thanks


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5090][examples] The improvement of pyth...

Posted by GenTang <gi...@git.apache.org>.
Github user GenTang commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3920#discussion_r24184482
  
    --- Diff: examples/src/main/scala/org/apache/spark/examples/HBaseTest.scala ---
    @@ -36,7 +36,7 @@ object HBaseTest {
         // Initialize hBase table if necessary
         val admin = new HBaseAdmin(conf)
         if (!admin.isTableAvailable(args(0))) {
    -      val tableDesc = new HTableDescriptor(args(0))
    +      val tableDesc = new HTableDescriptor(TableName.valueOf(args(0)))
    --- End diff --
    
    Yes. The old one is deprecated


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5090][examples] The improvement of pyth...

Posted by GenTang <gi...@git.apache.org>.
Github user GenTang commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3920#discussion_r25155799
  
    --- Diff: examples/src/main/scala/org/apache/spark/examples/pythonconverters/HBaseConverters.scala ---
    @@ -18,20 +18,34 @@
     package org.apache.spark.examples.pythonconverters
     
     import scala.collection.JavaConversions._
    +import scala.util.parsing.json._
     
     import org.apache.spark.api.python.Converter
     import org.apache.hadoop.hbase.client.{Put, Result}
     import org.apache.hadoop.hbase.io.ImmutableBytesWritable
     import org.apache.hadoop.hbase.util.Bytes
    +import org.apache.hadoop.hbase.KeyValue.Type
    +import org.apache.hadoop.hbase.CellUtil
     
     /**
    - * Implementation of [[org.apache.spark.api.python.Converter]] that converts an
    - * HBase Result to a String
    + * Implementation of [[org.apache.spark.api.python.Converter]] that converts all
    + * the records in an HBase Result to a String
      */
     class HBaseResultToStringConverter extends Converter[Any, String] {
       override def convert(obj: Any): String = {
    +    import collection.JavaConverters._
         val result = obj.asInstanceOf[Result]
    -    Bytes.toStringBinary(result.value())
    +    val output = result.listCells.asScala.map(cell =>
    +        Map(
    +          "row" -> Bytes.toStringBinary(CellUtil.cloneRow(cell)),
    +          "columnFamily" -> Bytes.toStringBinary(CellUtil.cloneFamily(cell)),
    +          "qualifier" -> Bytes.toStringBinary(CellUtil.cloneQualifier(cell)),
    +          "timestamp" -> cell.getTimestamp.toString,
    +          "type" -> Type.codeToType(cell.getTypeByte).toString,
    +          "value" -> Bytes.toStringBinary(CellUtil.cloneValue(cell))
    +        )
    +    )
    +    output.map(JSONObject(_).toString()).mkString("\n")
    --- End diff --
    
    Output is a `Buffer[Map[String, String]]`, since there are several records in an HBase Result.  
    However `JSONObject` has the only constructor `JSONObject(obj: Map[String, Any])`. So `JSONObject(output).toString()` will cause compilation failure. ^^


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5090][examples] The improvement of pyth...

Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3920#discussion_r24719372
  
    --- Diff: examples/src/main/python/hbase_inputformat.py ---
    @@ -16,6 +16,7 @@
     #
     
     import sys
    +import simplejson as json
    --- End diff --
    
    simplejson is not builtin in Python 2.6, we should use json.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5090][examples] The improvement of pyth...

Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3920#discussion_r24184037
  
    --- Diff: examples/src/main/scala/org/apache/spark/examples/HBaseTest.scala ---
    @@ -36,7 +36,7 @@ object HBaseTest {
         // Initialize hBase table if necessary
         val admin = new HBaseAdmin(conf)
         if (!admin.isTableAvailable(args(0))) {
    -      val tableDesc = new HTableDescriptor(args(0))
    +      val tableDesc = new HTableDescriptor(TableName.valueOf(args(0)))
    --- End diff --
    
    Is this an API change in HBase?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5090][examples] The improvement of pyth...

Posted by MLnick <gi...@git.apache.org>.
Github user MLnick commented on the pull request:

    https://github.com/apache/spark/pull/3920#issuecomment-70049746
  
    @GenTang  Sorry for the delay, have been travelling. I will try to get to this in the next few days.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-5090][examples] The improvement of pyth...

Posted by MLnick <gi...@git.apache.org>.
Github user MLnick commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3920#discussion_r23403193
  
    --- Diff: examples/src/main/scala/org/apache/spark/examples/pythonconverters/HBaseConverters.scala ---
    @@ -23,15 +23,27 @@ import org.apache.spark.api.python.Converter
     import org.apache.hadoop.hbase.client.{Put, Result}
     import org.apache.hadoop.hbase.io.ImmutableBytesWritable
     import org.apache.hadoop.hbase.util.Bytes
    +import org.apache.hadoop.hbase.KeyValue.Type
    +import org.apache.hadoop.hbase.CellUtil
     
     /**
    - * Implementation of [[org.apache.spark.api.python.Converter]] that converts an
    - * HBase Result to a String
    + * Implementation of [[org.apache.spark.api.python.Converter]] that converts all
    + * the records in an HBase Result to a String
      */
     class HBaseResultToStringConverter extends Converter[Any, String] {
       override def convert(obj: Any): String = {
    +    import collection.JavaConverters._
         val result = obj.asInstanceOf[Result]
    -    Bytes.toStringBinary(result.value())
    +    val output = result.listCells.asScala.map(cell =>
    +        "{'columnFamliy':'%s','qualifier':'%s','timestamp':'%s','type':'%s','value':'%s'}".format(
    +          Bytes.toStringBinary(CellUtil.cloneFamily(cell)),
    +          Bytes.toStringBinary(CellUtil.cloneQualifier(cell)),
    +          cell.getTimestamp.toString,
    +          Type.codeToType(cell.getTypeByte),
    +          Bytes.toStringBinary(CellUtil.cloneValue(cell))
    +        )
    +    )   
    +    output.mkString(" ")
    --- End diff --
    
    I'm not too familiar with HBase `Result` - are multiple rows in a result, separated by whitespace (in this case)? or is the idea one record per column family in a row?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org