You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@griffin.apache.org by GitBox <gi...@apache.org> on 2022/01/18 12:58:39 UTC
[GitHub] [griffin] lipzhu opened a new pull request #598: [GRIFFIN-369] Bug fix for avro format in Spark 2.3.x environment
lipzhu opened a new pull request #598:
URL: https://github.com/apache/griffin/pull/598
**What changes were proposed in this pull request?**
Built in Avro format is released in Spark 2.4.0,https://issues.apache.org/jira/browse/SPARK-24768
For Griffin, we still need to convert the Avro to com.databricks.spark.avro in Spark 2.3.x environment.
**Does this PR introduce any user-facing change?**
No.
**How was this patch tested?**
Unit Tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@griffin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [griffin] chitralverma closed pull request #598: [GRIFFIN-369] Bug fix for avro format in Spark 2.3.x environment
Posted by GitBox <gi...@apache.org>.
chitralverma closed pull request #598:
URL: https://github.com/apache/griffin/pull/598
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@griffin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [griffin] chitralverma commented on a change in pull request #598: [GRIFFIN-369] Bug fix for avro format in Spark 2.3.x environment
Posted by GitBox <gi...@apache.org>.
chitralverma commented on a change in pull request #598:
URL: https://github.com/apache/griffin/pull/598#discussion_r790411689
##########
File path: measure/src/main/scala/org/apache/griffin/measure/datasource/connector/batch/FileBasedDataConnector.scala
##########
@@ -79,7 +79,8 @@ case class FileBasedDataConnector(
SupportedFormats.contains(format),
s"Invalid format '$format' specified. Must be one of ${SupportedFormats.mkString("['", "', '", "']")}")
- if (format.equalsIgnoreCase("avro") && sparkSession.version < "2.3.0") {
+ // built-in AVRO data source implementation is released in spark 2.4.0
Review comment:
Suggestion:
```suggestion
// Use old implementation for AVRO format if current spark version is not 2.4.x and above
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@griffin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [griffin] chitralverma commented on a change in pull request #598: [GRIFFIN-369] Bug fix for avro format in Spark 2.3.x environment
Posted by GitBox <gi...@apache.org>.
chitralverma commented on a change in pull request #598:
URL: https://github.com/apache/griffin/pull/598#discussion_r790412145
##########
File path: measure/src/main/scala/org/apache/griffin/measure/datasource/connector/batch/FileBasedDataConnector.scala
##########
@@ -79,7 +79,8 @@ case class FileBasedDataConnector(
SupportedFormats.contains(format),
s"Invalid format '$format' specified. Must be one of ${SupportedFormats.mkString("['", "', '", "']")}")
- if (format.equalsIgnoreCase("avro") && sparkSession.version < "2.3.0") {
+ // Use old implementation for AVRO format if current spark version is not 2.4.x and above
+ if (format.equalsIgnoreCase("avro") && sparkSession.version < "2.4.0") {
Review comment:
```suggestion
if ("avro".equalsIgnoreCase(format) && sparkSession.version < "2.4.0") {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@griffin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [griffin] asfgit closed pull request #598: [GRIFFIN-369] Bug fix for avro format in Spark 2.3.x environment
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #598:
URL: https://github.com/apache/griffin/pull/598
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@griffin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org