[GitHub] [hbase-connectors] meszibalu commented on pull request #105: HBASE-27397 Spark-hbase support for 'startWith' predicate

[GitBox <gi...@apache.org>]

meszibalu commented on PR #105:
URL: https://github.com/apache/hbase-connectors/pull/105#issuecomment-1323672919

   I also added some debug information here: [HBaseTableScanRDD.scala:90](https://github.com/apache/hbase-connectors/blob/master/spark/hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark/datasources/HBaseTableScanRDD.scala#L90)
   
   ```scala
   if (log.isDebugEnabled) {
       logDebug(s"Partitions: ${ps.size}");
       ps.foreach(x => logDebug(x.toString))
   }
   ```
   
   Then I added some test data with `hbase shell`:
   
   ```
   create 'spark', 'f', SPLITS => ['m']
   
   put 'spark', 'a1', 'f:value', 'a1'
   put 'spark', 'a2', 'f:value', 'a2'
   put 'spark', 'b1', 'f:value', 'b1'
   put 'spark', 'b2', 'f:value', 'b2'
   put 'spark', 'c1', 'f:value', 'c1'
   put 'spark', 'c2', 'f:value', 'c2'
   put 'spark', 'k1', 'f:value', 'k1'
   put 'spark', 'k2', 'f:value', 'k2'
   put 'spark', 'l1', 'f:value', 'l1'
   put 'spark', 'l2', 'f:value', 'l2'
   put 'spark', 'm1', 'f:value', 'm1'
   put 'spark', 'm2', 'f:value', 'm2'
   put 'spark', 'n1', 'f:value', 'n1'
   put 'spark', 'n2', 'f:value', 'n2'
   put 'spark', 'o1', 'f:value', 'o1'
   put 'spark', 'o2', 'f:value', 'o2'
   put 'spark', 'p1', 'f:value', 'p1'
   put 'spark', 'p2', 'f:value', 'p2'
   ```
   
   And run the following spark code:
   
   ```scala
   org.apache.log4j.LogManager.getLogger("org.apache.hadoop.hbase").setLevel(org.apache.log4j.Level.DEBUG)
   import org.apache.spark.sql.functions.col
   
   val sql = spark.sqlContext
   
   val df = sql.read.format("org.apache.hadoop.hbase.spark").option("hbase.columns.mapping", "key STRING :key, value STRING f:value").option("hbase.table", "spark").option("hbase.spark.use.hbasecontext", false).load()
   // this is good, uses 2 regions, dumps everything
   df.filter(col("key").startsWith("")).show()
   // this uses 1 region, but it does not return any row
   df.filter(col("key").startsWith("a")).show()
   // this uses 1 region and it is ok
   df.filter(col("key").equalTo("a1")).show()
   ```
   
   So `startsWith("a")` does not work for me.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org