You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sedona.apache.org by "prantogg (via GitHub)" <gi...@apache.org> on 2024/04/24 22:23:15 UTC
[PR] [SEDONA-543] Fixes RS_Union_aggr throwing referenceRaster is null error when run on cluster [sedona]
prantogg opened a new pull request, #1364:
URL: https://github.com/apache/sedona/pull/1364
## Did you read the Contributor Guide?
- Yes, I have read the [Contributor Rules](https://sedona.apache.org/latest-snapshot/community/rule/) and [Contributor Development Guide](https://sedona.apache.org/latest-snapshot/community/develop/)
## Is this PR related to a JIRA ticket?
- Yes, the URL of the associated JIRA ticket is https://issues.apache.org/jira/browse/SEDONA-543. The PR name follows the format `[SEDONA-XXX] my subject`.
## What changes were proposed in this PR?
- In RS_Union_aggr, moves class level members like `referenceRaster` to data buffer.
## How was this patch tested?
- Passes existing tests
## Did this PR include necessary documentation updates?
- No, this PR does not affect any public API so no need to change the documentation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [SEDONA-543] Fixes RS_Union_aggr throwing referenceRaster is null error when run on cluster [sedona]
Posted by "prantogg (via GitHub)" <gi...@apache.org>.
prantogg commented on code in PR #1364:
URL: https://github.com/apache/sedona/pull/1364#discussion_r1581447327
##########
spark/common/src/main/scala/org/apache/sedona/sql/utils/RasterSerializer.scala:
##########
Review Comment:
Sorry I missed this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [SEDONA-543] Fixes RS_Union_aggr throwing referenceRaster is null error when run on cluster [sedona]
Posted by "Kontinuation (via GitHub)" <gi...@apache.org>.
Kontinuation commented on code in PR #1364:
URL: https://github.com/apache/sedona/pull/1364#discussion_r1578756097
##########
spark/common/src/main/scala/org/apache/sedona/sql/utils/RasterSerializer.scala:
##########
@@ -42,4 +43,18 @@ object RasterSerializer {
def deserialize(value: Array[Byte]): GridCoverage2D = {
Serde.deserialize(value);
}
+
+ def serializeSampleDimension(sampleDimension: GridSampleDimension): Array[Byte] = {
+ val bos = new ByteArrayOutputStream()
+ val oos = new ObjectOutputStream(bos)
+ oos.writeObject(sampleDimension)
+ oos.flush()
+ bos.toByteArray
+ }
+
+ def deserializeSampleDimension(bytes: Array[Byte]): GridSampleDimension = {
+ val bis = new ByteArrayInputStream(bytes)
+ val ois = new ObjectInputStream(bis)
+ ois.readObject().asInstanceOf[GridSampleDimension]
+ }
Review Comment:
We have a `GridSampleDimensionSerializer` implemented using Kryo. I recommend using that instead of the Java serializer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [SEDONA-543] Fixes RS_Union_aggr throwing referenceRaster is null error when run on cluster [sedona]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu merged PR #1364:
URL: https://github.com/apache/sedona/pull/1364
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [SEDONA-543] Fixes RS_Union_aggr throwing referenceRaster is null error when run on cluster [sedona]
Posted by "prantogg (via GitHub)" <gi...@apache.org>.
prantogg commented on code in PR #1364:
URL: https://github.com/apache/sedona/pull/1364#discussion_r1581449304
##########
spark/common/src/main/scala/org/apache/sedona/sql/utils/RasterSerializer.scala:
##########
Review Comment:
This is fixed now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [SEDONA-543] Fixes RS_Union_aggr throwing referenceRaster is null error when run on cluster [sedona]
Posted by "prantogg (via GitHub)" <gi...@apache.org>.
prantogg commented on code in PR #1364:
URL: https://github.com/apache/sedona/pull/1364#discussion_r1581484447
##########
common/src/main/java/org/apache/sedona/common/raster/serde/Serde.java:
##########
@@ -176,4 +178,19 @@ public static GridCoverage2D deserialize(byte[] bytes) throws IOException, Class
return state.restore();
}
}
+
+ public static byte[] serializeGridSampleDimension(GridSampleDimension sampleDimension) {
Review Comment:
This is as per Kristin's recommendation to implement new serialization/deserialization methods for GridSampleDimension that use `kryos.get()` to retrieve `Kryo` objects. `kryos.get()` avoids repeatedly initializing new Kryo instances.
I should have used the existing `GridSampleDimensionSerializer` for read and write into output buffer. This is added now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [SEDONA-543] Fixes RS_Union_aggr throwing referenceRaster is null error when run on cluster [sedona]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on code in PR #1364:
URL: https://github.com/apache/sedona/pull/1364#discussion_r1581451491
##########
common/src/main/java/org/apache/sedona/common/raster/serde/Serde.java:
##########
@@ -176,4 +178,19 @@ public static GridCoverage2D deserialize(byte[] bytes) throws IOException, Class
return state.restore();
}
}
+
+ public static byte[] serializeGridSampleDimension(GridSampleDimension sampleDimension) {
Review Comment:
This was already implemented in Sedona. Why implement it again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [SEDONA-543] Fixes RS_Union_aggr throwing referenceRaster is null error when run on cluster [sedona]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on code in PR #1364:
URL: https://github.com/apache/sedona/pull/1364#discussion_r1581443174
##########
spark/common/src/main/scala/org/apache/sedona/sql/utils/RasterSerializer.scala:
##########
Review Comment:
Please don't commit this kine of empty change.
##########
spark/common/src/main/scala/org/apache/sedona/sql/utils/RasterSerializer.scala:
##########
Review Comment:
Please don't commit this kind of empty change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org