You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "daikon12 (via GitHub)" <gi...@apache.org> on 2023/10/18 03:54:17 UTC
[PR] [fix](load) fix npe when array is null [doris-spark-connector]
daikon12 opened a new pull request, #147:
URL: https://github.com/apache/doris-spark-connector/pull/147
## Problem Summary:
Currently, when a value of the array type is null, rowColumnValue will encounter a null pointer
Describe the overview of changes.
I suggest that when processing arrays, consider the case of null and return null.
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](load) fix npe when array is null [doris-spark-connector]
Posted by "wolfboys (via GitHub)" <gi...@apache.org>.
wolfboys commented on code in PR #147:
URL: https://github.com/apache/doris-spark-connector/pull/147#discussion_r1363166877
##########
spark-doris-connector/src/main/scala/org/apache/doris/spark/sql/SchemaUtils.scala:
##########
@@ -166,13 +166,16 @@ private[spark] object SchemaUtils {
case dt: DecimalType => row.getDecimal(ordinal, dt.precision, dt.scale)
case at: ArrayType =>
val arrayData = row.getArray(ordinal)
- var i = 0
- val buffer = mutable.Buffer[Any]()
- while (i < arrayData.numElements()) {
- if (arrayData.isNullAt(i)) buffer += null else buffer += rowColumnValue(arrayData, i, at.elementType)
- i += 1
- }
- s"[${buffer.mkString(",")}]"
+ val result: String = Option(arrayData)
Review Comment:
๐Thank you for your contribution! ๐ It is necessary to consider the case where the array is empty (size = 0). The code can be further improved!
```
if (arrayData == null) DataUtil.NULL_VALUE else {
if (arrayData.numElements() == 0) "[]" else {
(0 until arrayData.numElements()).map(i => {
if (arrayData.isNullAt(i)) null else rowColumnValue(data, i, at.elementType)
}).mkString("[", ",", "]")
}
}
```
##########
spark-doris-connector/src/main/scala/org/apache/doris/spark/sql/SchemaUtils.scala:
##########
@@ -166,13 +166,16 @@ private[spark] object SchemaUtils {
case dt: DecimalType => row.getDecimal(ordinal, dt.precision, dt.scale)
case at: ArrayType =>
val arrayData = row.getArray(ordinal)
- var i = 0
- val buffer = mutable.Buffer[Any]()
- while (i < arrayData.numElements()) {
- if (arrayData.isNullAt(i)) buffer += null else buffer += rowColumnValue(arrayData, i, at.elementType)
- i += 1
- }
- s"[${buffer.mkString(",")}]"
+ val result: String = Option(arrayData)
Review Comment:
๐Thank you for your contribution! ๐ It is necessary to consider the case where the array is empty (size = 0). The code can be further improved!
```
if (arrayData == null) DataUtil.NULL_VALUE else {
if (arrayData.numElements() == 0) "[]" else {
(0 until arrayData.numElements()).map(i => {
if (arrayData.isNullAt(i)) null else rowColumnValue(data, i, at.elementType)
}).mkString("[", ",", "]")
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](load) fix npe when array is null [doris-spark-connector]
Posted by "daikon12 (via GitHub)" <gi...@apache.org>.
daikon12 commented on code in PR #147:
URL: https://github.com/apache/doris-spark-connector/pull/147#discussion_r1363269325
##########
spark-doris-connector/src/main/scala/org/apache/doris/spark/sql/SchemaUtils.scala:
##########
@@ -166,13 +166,16 @@ private[spark] object SchemaUtils {
case dt: DecimalType => row.getDecimal(ordinal, dt.precision, dt.scale)
case at: ArrayType =>
val arrayData = row.getArray(ordinal)
- var i = 0
- val buffer = mutable.Buffer[Any]()
- while (i < arrayData.numElements()) {
- if (arrayData.isNullAt(i)) buffer += null else buffer += rowColumnValue(arrayData, i, at.elementType)
- i += 1
- }
- s"[${buffer.mkString(",")}]"
+ val result: String = Option(arrayData)
Review Comment:
@wolfboys Thank you for reminding me. You are right. I will improve it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](load) fix npe when array is null [doris-spark-connector]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba merged PR #147:
URL: https://github.com/apache/doris-spark-connector/pull/147
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org