You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by GitBox <gi...@apache.org> on 2021/03/17 07:30:03 UTC
[GitHub] [iotdb] FlowerBirds opened a new issue #2861: read empty dataframe when one column result by spark iotdb connector
FlowerBirds opened a new issue #2861:
URL: https://github.com/apache/iotdb/issues/2861
**环境**
- Iotdb 0.9.3 & 0.11.2
- spark 2.3.3
- spark-iotdb-connector 0.11.2
**问题**
spark读取iotdb数据后,返回一个空的DataFrame对象
**分析**
0.9.3中执行带聚合函数的sql,如`select count(root.ln.wf01.wt01.status) from root`,返回为两列,一列是时间列,一列是值,但是在0.11.2中返回的是一列,但是spark-iotdb-connector中并未处理这种,在生成schame的时候,从2开始遍历,导致只有一列的数据返回空schema,代码如下:
spark-iotdb-connector\src\main\scala\org\apache\iotdb\spark\db\Converter.scala
```
def toSparkSchema(options: IoTDBOptions): StructType = {
Class.forName("org.apache.iotdb.jdbc.IoTDBDriver")
val sqlConn: Connection = DriverManager.getConnection(options.url, options.user, options.password)
val sqlStatement: Statement = sqlConn.createStatement()
val hasResultSet: Boolean = sqlStatement.execute(options.sql)
val fields = new ListBuffer[StructField]()
if (hasResultSet) {
val resultSet: ResultSet = sqlStatement.getResultSet
val resultSetMetaData: ResultSetMetaData = resultSet.getMetaData
val printTimestamp = !resultSet.asInstanceOf[IoTDBJDBCResultSet].isIgnoreTimeStamp
if (printTimestamp) {
fields += StructField(SQLConstant.TIMESTAMP_STR, LongType, nullable = false)
}
val colCount = resultSetMetaData.getColumnCount
for (i <- 2 to colCount) {
.....
}
StructType(fields.toList)
}
else {
StructType(fields)
}
}
```
需要加个startCol变量,来确定是从2开始遍历还是1,如下:
```
var startCol = 1
if (printTimestamp) {
fields += StructField(SQLConstant.TIMESTAMP_STR, LongType, nullable = false)
startCol = 2
}
val colCount = resultSetMetaData.getColumnCount
for (i <- startCol to colCount) {
.....
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] jixuan1989 commented on issue #2861: read empty dataframe when one column result by spark iotdb connector
Posted by GitBox <gi...@apache.org>.
jixuan1989 commented on issue #2861:
URL: https://github.com/apache/iotdb/issues/2861#issuecomment-846976270
> How to find the TsFile-connector project for Spark and Hive on github?

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] Ymick commented on issue #2861: read empty dataframe when one column result by spark iotdb connector
Posted by GitBox <gi...@apache.org>.
Ymick commented on issue #2861:
URL: https://github.com/apache/iotdb/issues/2861#issuecomment-846942660
How to find the TsFile-connector project for Spark and Hive on github?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] github-actions[bot] commented on issue #2861: read empty dataframe when one column result by spark iotdb connector
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #2861:
URL: https://github.com/apache/iotdb/issues/2861#issuecomment-800876217
Hi, this is your first issue in IoTDB project. Thanks for your report. Welcome to join the community!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org