[jira] [Created] (PHOENIX-1430) Spark queries against tables with VARCHAR ARRAY columns fail

["Robert Roland (JIRA)" <ji...@apache.org>]

Robert Roland created PHOENIX-1430:
--------------------------------------

             Summary: Spark queries against tables with VARCHAR ARRAY columns fail
                 Key: PHOENIX-1430
                 URL: https://issues.apache.org/jira/browse/PHOENIX-1430
             Project: Phoenix
          Issue Type: Bug
    Affects Versions: 4.1
            Reporter: Robert Roland


Running Phoenix 4.1 against HDP 2.2 Preview, I'm unable to execute queries in Spark against tables that contain VARCHAR ARRAY columns. Given the error, I think it's likely to affect any array column.

Given the following table schema:

{noformat}
CREATE TABLE ARRAY_TEST_TABLE (
  ID BIGINT NOT NULL,
  STRING_ARRAY VARCHAR[]
  CONSTRAINT pk PRIMARY KEY (ID));
{noformat}

I am unable to execute a query via Spark, using the PhoenixInputFormat:

{noformat}
val phoenixConf = new PhoenixPigConfiguration(new Configuration())

phoenixConf.setSelectStatement("SELECT ID, STRING_ARRAY FROM ARRAY_TEST_TABLE")
phoenixConf.setSelectColumns("ID,STRING_ARRAY")
phoenixConf.setSchemaType(SchemaType.QUERY)
phoenixConf.configure("sandbox.hortonworks.com:2181:/hbase-unsecure", "ARRAY_TEST_TABLE", 100)
val phoenixRDD = sc.newAPIHadoopRDD(phoenixConf.getConfiguration,
  classOf[PhoenixInputFormat],
  classOf[NullWritable],
  classOf[PhoenixRecord])

val count = phoenixRDD.count()
{noformat}

I get the following error:

{noformat}
  java.lang.RuntimeException: java.sql.SQLException: org.apache.phoenix.schema.IllegalDataException: Unsupported sql type: VARCHAR ARRAY
  at org.apache.phoenix.pig.hadoop.PhoenixInputFormat.getQueryPlan(PhoenixInputFormat.java:162)
  at org.apache.phoenix.pig.hadoop.PhoenixInputFormat.getSplits(PhoenixInputFormat.java:88)
  at org.apache.spark.rdd.NewHadoopRDD.getPartitions(NewHadoopRDD.scala:94)
  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)
  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202)
  at scala.Option.getOrElse(Option.scala:120)
  at org.apache.spark.rdd.RDD.partitions(RDD.scala:202)
  at org.apache.spark.SparkContext.runJob(SparkContext.scala:1135)
  at org.apache.spark.rdd.RDD.count(RDD.scala:904)
  at com.simplymeasured.spark.PhoenixRDDTest$$anonfun$4.apply$mcV$sp(PhoenixRDDTest.scala:147)
  ...
  Cause: java.sql.SQLException: org.apache.phoenix.schema.IllegalDataException: Unsupported sql type: VARCHAR ARRAY
  at org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:947)
  at org.apache.phoenix.query.ConnectionQueryServicesImpl.getTable(ConnectionQueryServicesImpl.java:1171)
  at org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:315)
  at org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:284)
  at org.apache.phoenix.compile.FromCompiler$BaseColumnResolver.createTableRef(FromCompiler.java:289)
  at org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.<init>(FromCompiler.java:210)
  at org.apache.phoenix.compile.FromCompiler.getResolverForQuery(FromCompiler.java:158)
  at org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:300)
  at org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:290)
  at org.apache.phoenix.jdbc.PhoenixStatement.compileQuery(PhoenixStatement.java:926)
  ...
  Cause: org.apache.phoenix.schema.IllegalDataException: Unsupported sql type: VARCHAR ARRAY
  at org.apache.phoenix.schema.PDataType.fromSqlTypeName(PDataType.java:6977)
  at org.apache.phoenix.schema.PColumnImpl.createFromProto(PColumnImpl.java:195)
  at org.apache.phoenix.schema.PTableImpl.createFromProto(PTableImpl.java:848)
  at org.apache.phoenix.coprocessor.MetaDataProtocol$MetaDataMutationResult.constructFromProto(MetaDataProtocol.java:158)
  at org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:939)
  at org.apache.phoenix.query.ConnectionQueryServicesImpl.getTable(ConnectionQueryServicesImpl.java:1171)
  at org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:315)
  at org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:284)
  at org.apache.phoenix.compile.FromCompiler$BaseColumnResolver.createTableRef(FromCompiler.java:289)
  at org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.<init>(FromCompiler.java:210)
{noformat}

Using sqlline to investigate the column's type, it looks like it's considered "VARCHAR ARRAY" instead of "VARCHAR_ARRAY": (truncated for brevity)

{noformat}
0: jdbc:phoenix:localhost:2181:/hbase-unsecur> !columns ARRAY_TEST_TABLE
+------------+-------------+------------+-------------+------------+------------+
| TABLE_CAT  | TABLE_SCHEM | TABLE_NAME | COLUMN_NAME | DATA_TYPE  | TYPE_NAME  |
+------------+-------------+------------+-------------+------------+------------+
| null       | null        | ARRAY_TEST_TABLE | ID          | -5         | BIGINT     |
| null       | null        | ARRAY_TEST_TABLE | STRING_ARRAY | 2003       | VARCHAR ARRAY |
+------------+-------------+------------+-------------+------------+------------+
{noformat}

The PDataType class defines VARCHAR_ARRAY as such:

{noformat}
VARCHAR_ARRAY("VARCHAR_ARRAY", PDataType.ARRAY_TYPE_BASE + PDataType.VARCHAR.getSqlType(), PhoenixArray.class, null) { ... }
{noformat}

The first parameter there being the sqlTypeName, which is "VARCHAR_ARRAY" but it appears to try and look it up as "VARCHAR ARRAY" (space instead of underscore)

I'm not sure if the fix here is to change those values, or if it's deep inside MetaDataEndpointImpl where the ProtoBuf returned to the client is implemented when a getTable call occurs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)