You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "parisni (via GitHub)" <gi...@apache.org> on 2023/07/10 14:04:14 UTC
[GitHub] [hudi] parisni opened a new pull request, #8683: [HUDI-5533] Support spark columns comments
parisni opened a new pull request, #8683:
URL: https://github.com/apache/hudi/pull/8683
### Change Logs
fixes #7531 ie: show comments within spark schemas
### Impact
_Describe any public API or user-facing feature change or any performance impact._
### Risk level (write none, low medium or high below)
None
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change_
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [X] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] parisni commented on pull request #8683: [HUDI-5533] Support spark columns comments
Posted by "parisni (via GitHub)" <gi...@apache.org>.
parisni commented on PR #8683:
URL: https://github.com/apache/hudi/pull/8683#issuecomment-1646829280
@danny0405 just learned [here why the spark data source stuff shall be kept within hms](https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=147427331#content/view/147427331)
> With the above, all of Spark SQL queries will use Hudi DataSource and hence end up using the custom FileIndex for doing the listing.
Then there is a need to update the datasource code to be aware of comments within hive sync.
@prashantwason can you confirm this part is still applicable and as a result `hoodie.datasource.hive_sync.sync_as_datasource` is highly encouraged ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] parisni commented on pull request #8683: [HUDI-5533] Support spark columns comments
Posted by "parisni (via GitHub)" <gi...@apache.org>.
parisni commented on PR #8683:
URL: https://github.com/apache/hudi/pull/8683#issuecomment-1587806101
while `spark.read.format("hudi").load("path")` and `spark.table("database.table")` returns the comments within the schema, `spark.sql("desc database.schema")` won't. So I have to investigate and add a fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8683: [HUDI-5533] Support spark columns comments
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8683:
URL: https://github.com/apache/hudi/pull/8683#issuecomment-1543040480
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7bdb94998ee2853e15de0b4ce6c20735f43a0f5c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17006",
"triggerID" : "7bdb94998ee2853e15de0b4ce6c20735f43a0f5c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7bdb94998ee2853e15de0b4ce6c20735f43a0f5c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17006)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] parisni commented on a diff in pull request #8683: [HUDI-5533] Support spark columns comments
Posted by "parisni (via GitHub)" <gi...@apache.org>.
parisni commented on code in PR #8683:
URL: https://github.com/apache/hudi/pull/8683#discussion_r1293890701
##########
hudi-sync/hudi-sync-common/src/main/java/org/apache/hudi/sync/common/util/Parquet2SparkSchemaUtils.java:
##########
@@ -19,40 +19,61 @@
package org.apache.hudi.sync.common.util;
import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.sync.common.model.FieldSchema;
+
import org.apache.parquet.schema.GroupType;
import org.apache.parquet.schema.OriginalType;
import org.apache.parquet.schema.PrimitiveType;
import org.apache.parquet.schema.Type;
+import java.util.Arrays;
+import java.util.List;
+import java.util.regex.Matcher;
+
import static org.apache.parquet.schema.Type.Repetition.OPTIONAL;
/**
* Convert the parquet schema to spark schema' json string.
* This code is refer to org.apache.spark.sql.execution.datasources.parquet.ParquetToSparkSchemaConverter
* in spark project.
+ *
+ * Currently, due to limitations in the FieldSchema only the first level of fields get comments
+ * extracted, if exist.
*/
public class Parquet2SparkSchemaUtils {
- public static String convertToSparkSchemaJson(GroupType parquetSchema) {
+ public static String convertToSparkSchemaJson(GroupType parquetSchema, List<FieldSchema> fieldSchema) {
String fieldsJsonString = parquetSchema.getFields().stream().map(field -> {
switch (field.getRepetition()) {
case OPTIONAL:
return "{\"name\":\"" + field.getName() + "\",\"type\":" + convertFieldType(field)
- + ",\"nullable\":true,\"metadata\":{}}";
+ + ",\"nullable\":true,\"metadata\":{" + getComment(field.getName(), fieldSchema) + "}}";
case REQUIRED:
return "{\"name\":\"" + field.getName() + "\",\"type\":" + convertFieldType(field)
- + ",\"nullable\":false,\"metadata\":{}}";
+ + ",\"nullable\":false,\"metadata\":{" + getComment(field.getName(), fieldSchema) + "}}";
case REPEATED:
String arrayType = arrayType(field, false);
return "{\"name\":\"" + field.getName() + "\",\"type\":" + arrayType
- + ",\"nullable\":false,\"metadata\":{}}";
+ + ",\"nullable\":false,\"metadata\":{" + getComment(field.getName(), fieldSchema) + "}}";
default:
throw new UnsupportedOperationException("Unsupport convert " + field + " to spark sql type");
}
}).reduce((a, b) -> a + "," + b).orElse("");
return "{\"type\":\"struct\",\"fields\":[" + fieldsJsonString + "]}";
}
+ private static String getComment(String name, List<FieldSchema> fromStorage) {
+ return fromStorage.stream()
+ .filter(f -> name.equals(f.getName()))
+ .filter(f -> f.getComment().isPresent())
+ .map(f -> "\"comment\":\"" + escapeQuote(f.getComment().get()) + "\"")
+ .findFirst().orElse("");
+ }
+
+ private static String escapeQuote(String s) {
+ return s.replaceAll("\"", Matcher.quoteReplacement("\\\""));
+ }
Review Comment:
see UT. I added a double quote in the column comment, and for some hive sync ddl exectutors, this is needed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5533] Support spark columns comments [hudi]
Posted by "parisni (via GitHub)" <gi...@apache.org>.
parisni closed pull request #8683: [HUDI-5533] Support spark columns comments
URL: https://github.com/apache/hudi/pull/8683
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8683: [HUDI-5533] Support spark columns comments
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8683:
URL: https://github.com/apache/hudi/pull/8683#discussion_r1286883188
##########
hudi-sync/hudi-sync-common/src/main/java/org/apache/hudi/sync/common/util/Parquet2SparkSchemaUtils.java:
##########
@@ -19,40 +19,61 @@
package org.apache.hudi.sync.common.util;
import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.sync.common.model.FieldSchema;
+
import org.apache.parquet.schema.GroupType;
import org.apache.parquet.schema.OriginalType;
import org.apache.parquet.schema.PrimitiveType;
import org.apache.parquet.schema.Type;
+import java.util.Arrays;
+import java.util.List;
+import java.util.regex.Matcher;
+
import static org.apache.parquet.schema.Type.Repetition.OPTIONAL;
/**
* Convert the parquet schema to spark schema' json string.
* This code is refer to org.apache.spark.sql.execution.datasources.parquet.ParquetToSparkSchemaConverter
* in spark project.
+ *
+ * Currently, due to limitations in the FieldSchema only the first level of fields get comments
+ * extracted, if exist.
*/
public class Parquet2SparkSchemaUtils {
- public static String convertToSparkSchemaJson(GroupType parquetSchema) {
+ public static String convertToSparkSchemaJson(GroupType parquetSchema, List<FieldSchema> fieldSchema) {
String fieldsJsonString = parquetSchema.getFields().stream().map(field -> {
switch (field.getRepetition()) {
case OPTIONAL:
return "{\"name\":\"" + field.getName() + "\",\"type\":" + convertFieldType(field)
- + ",\"nullable\":true,\"metadata\":{}}";
+ + ",\"nullable\":true,\"metadata\":{" + getComment(field.getName(), fieldSchema) + "}}";
case REQUIRED:
return "{\"name\":\"" + field.getName() + "\",\"type\":" + convertFieldType(field)
- + ",\"nullable\":false,\"metadata\":{}}";
+ + ",\"nullable\":false,\"metadata\":{" + getComment(field.getName(), fieldSchema) + "}}";
case REPEATED:
String arrayType = arrayType(field, false);
return "{\"name\":\"" + field.getName() + "\",\"type\":" + arrayType
- + ",\"nullable\":false,\"metadata\":{}}";
+ + ",\"nullable\":false,\"metadata\":{" + getComment(field.getName(), fieldSchema) + "}}";
default:
throw new UnsupportedOperationException("Unsupport convert " + field + " to spark sql type");
}
}).reduce((a, b) -> a + "," + b).orElse("");
return "{\"type\":\"struct\",\"fields\":[" + fieldsJsonString + "]}";
}
+ private static String getComment(String name, List<FieldSchema> fromStorage) {
+ return fromStorage.stream()
+ .filter(f -> name.equals(f.getName()))
+ .filter(f -> f.getComment().isPresent())
+ .map(f -> "\"comment\":\"" + escapeQuote(f.getComment().get()) + "\"")
+ .findFirst().orElse("");
+ }
+
+ private static String escapeQuote(String s) {
+ return s.replaceAll("\"", Matcher.quoteReplacement("\\\""));
+ }
Review Comment:
Not sure why we need a escape.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8683: [HUDI-5533] Support spark columns comments
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8683:
URL: https://github.com/apache/hudi/pull/8683#issuecomment-1656979785
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7bdb94998ee2853e15de0b4ce6c20735f43a0f5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17006",
"triggerID" : "7bdb94998ee2853e15de0b4ce6c20735f43a0f5c",
"triggerType" : "PUSH"
}, {
"hash" : "8d6893fd9daf07c30524474cf9a4d39c66a37cba",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8d6893fd9daf07c30524474cf9a4d39c66a37cba",
"triggerType" : "PUSH"
}, {
"hash" : "de250fbbcf1d16ba358dd08270eab5e11a5e3740",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "de250fbbcf1d16ba358dd08270eab5e11a5e3740",
"triggerType" : "PUSH"
}, {
"hash" : "f41404ad3c5c399ee0243fe0ea9a5ee70b74f896",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18214",
"triggerID" : "f41404ad3c5c399ee0243fe0ea9a5ee70b74f896",
"triggerType" : "PUSH"
}, {
"hash" : "e216e514efb89b1ef1a810111cac8175fc0c8941",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18449",
"triggerID" : "e216e514efb89b1ef1a810111cac8175fc0c8941",
"triggerType" : "PUSH"
}, {
"hash" : "a8b972a6d624afb07e197ffef99473f644e3305f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18451",
"triggerID" : "a8b972a6d624afb07e197ffef99473f644e3305f",
"triggerType" : "PUSH"
}, {
"hash" : "89409c24fcccd59f3bdae47bef696c1f3400e650",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18890",
"triggerID" : "89409c24fcccd59f3bdae47bef696c1f3400e650",
"triggerType" : "PUSH"
}, {
"hash" : "b92e5eb595620867378ffe2b35b1bd03ce42cb01",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18892",
"triggerID" : "b92e5eb595620867378ffe2b35b1bd03ce42cb01",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8d6893fd9daf07c30524474cf9a4d39c66a37cba UNKNOWN
* de250fbbcf1d16ba358dd08270eab5e11a5e3740 UNKNOWN
* b92e5eb595620867378ffe2b35b1bd03ce42cb01 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18892)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #8683: [HUDI-5533] Support spark columns comments
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #8683:
URL: https://github.com/apache/hudi/pull/8683#issuecomment-1669234208
Thanks @parisni , would find some time for the review ~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8683: [HUDI-5533] Support spark columns comments
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8683:
URL: https://github.com/apache/hudi/pull/8683#discussion_r1286882926
##########
hudi-sync/hudi-sync-common/src/main/java/org/apache/hudi/sync/common/util/Parquet2SparkSchemaUtils.java:
##########
@@ -19,40 +19,61 @@
package org.apache.hudi.sync.common.util;
import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.sync.common.model.FieldSchema;
+
import org.apache.parquet.schema.GroupType;
import org.apache.parquet.schema.OriginalType;
import org.apache.parquet.schema.PrimitiveType;
import org.apache.parquet.schema.Type;
+import java.util.Arrays;
+import java.util.List;
+import java.util.regex.Matcher;
+
import static org.apache.parquet.schema.Type.Repetition.OPTIONAL;
/**
* Convert the parquet schema to spark schema' json string.
* This code is refer to org.apache.spark.sql.execution.datasources.parquet.ParquetToSparkSchemaConverter
* in spark project.
+ *
+ * Currently, due to limitations in the FieldSchema only the first level of fields get comments
+ * extracted, if exist.
*/
public class Parquet2SparkSchemaUtils {
- public static String convertToSparkSchemaJson(GroupType parquetSchema) {
+ public static String convertToSparkSchemaJson(GroupType parquetSchema, List<FieldSchema> fieldSchema) {
String fieldsJsonString = parquetSchema.getFields().stream().map(field -> {
switch (field.getRepetition()) {
case OPTIONAL:
return "{\"name\":\"" + field.getName() + "\",\"type\":" + convertFieldType(field)
- + ",\"nullable\":true,\"metadata\":{}}";
+ + ",\"nullable\":true,\"metadata\":{" + getComment(field.getName(), fieldSchema) + "}}";
case REQUIRED:
return "{\"name\":\"" + field.getName() + "\",\"type\":" + convertFieldType(field)
- + ",\"nullable\":false,\"metadata\":{}}";
+ + ",\"nullable\":false,\"metadata\":{" + getComment(field.getName(), fieldSchema) + "}}";
case REPEATED:
String arrayType = arrayType(field, false);
return "{\"name\":\"" + field.getName() + "\",\"type\":" + arrayType
- + ",\"nullable\":false,\"metadata\":{}}";
+ + ",\"nullable\":false,\"metadata\":{" + getComment(field.getName(), fieldSchema) + "}}";
default:
throw new UnsupportedOperationException("Unsupport convert " + field + " to spark sql type");
}
}).reduce((a, b) -> a + "," + b).orElse("");
return "{\"type\":\"struct\",\"fields\":[" + fieldsJsonString + "]}";
}
+ private static String getComment(String name, List<FieldSchema> fromStorage) {
+ return fromStorage.stream()
+ .filter(f -> name.equals(f.getName()))
+ .filter(f -> f.getComment().isPresent())
Review Comment:
So we only match from the first level for the fields by name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8683: [HUDI-5533] Support spark columns comments
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8683:
URL: https://github.com/apache/hudi/pull/8683#discussion_r1286878550
##########
hudi-sync/hudi-adb-sync/src/main/java/org/apache/hudi/sync/adb/AdbSyncTool.java:
##########
@@ -212,8 +213,9 @@ private void syncSchema(String tableName, boolean tableExists, boolean useRealTi
Map<String, String> tableProperties = ConfigUtils.toMap(config.getString(ADB_SYNC_TABLE_PROPERTIES));
Map<String, String> serdeProperties = ConfigUtils.toMap(config.getString(ADB_SYNC_SERDE_PROPERTIES));
if (config.getBoolean(ADB_SYNC_SYNC_AS_SPARK_DATA_SOURCE_TABLE)) {
+ List<FieldSchema> fromStorage = syncClient.getStorageFieldSchemas();
Map<String, String> sparkTableProperties = SparkDataSourceTableUtils.getSparkTableProperties(config.getSplitStrings(META_SYNC_PARTITION_FIELDS),
Review Comment:
`fromStorage` -> `fieldSchema`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] parisni closed pull request #8683: [HUDI-5533] Support spark columns comments
Posted by "parisni (via GitHub)" <gi...@apache.org>.
parisni closed pull request #8683: [HUDI-5533] Support spark columns comments
URL: https://github.com/apache/hudi/pull/8683
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] parisni commented on a diff in pull request #8683: [HUDI-5533] Support spark columns comments
Posted by "parisni (via GitHub)" <gi...@apache.org>.
parisni commented on code in PR #8683:
URL: https://github.com/apache/hudi/pull/8683#discussion_r1193657663
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/avro/SchemaConverters.scala:
##########
@@ -59,32 +59,32 @@ private[sql] object SchemaConverters {
private val unionFieldMemberPrefix = "member"
private def toSqlTypeHelper(avroSchema: Schema, existingRecordNames: Set[String]): SchemaType = {
- avroSchema.getType match {
- case INT => avroSchema.getLogicalType match {
- case _: Date => SchemaType(DateType, nullable = false)
- case _ => SchemaType(IntegerType, nullable = false)
+ (avroSchema.getType, Option(avroSchema.getDoc)) match {
Review Comment:
sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] parisni commented on pull request #8683: [HUDI-5533] Support spark columns comments
Posted by "parisni (via GitHub)" <gi...@apache.org>.
parisni commented on PR #8683:
URL: https://github.com/apache/hudi/pull/8683#issuecomment-1546895581
> Does Option.ofNullable work correctly here?
Scala Option does not have ofNullable (java Optinonal do have). BTW `Option(value)` is equivalent as what you suggest, and I have corrected
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8683: [HUDI-5533] Support spark columns comments
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8683:
URL: https://github.com/apache/hudi/pull/8683#issuecomment-1656901567
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7bdb94998ee2853e15de0b4ce6c20735f43a0f5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17006",
"triggerID" : "7bdb94998ee2853e15de0b4ce6c20735f43a0f5c",
"triggerType" : "PUSH"
}, {
"hash" : "8d6893fd9daf07c30524474cf9a4d39c66a37cba",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8d6893fd9daf07c30524474cf9a4d39c66a37cba",
"triggerType" : "PUSH"
}, {
"hash" : "de250fbbcf1d16ba358dd08270eab5e11a5e3740",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "de250fbbcf1d16ba358dd08270eab5e11a5e3740",
"triggerType" : "PUSH"
}, {
"hash" : "f41404ad3c5c399ee0243fe0ea9a5ee70b74f896",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18214",
"triggerID" : "f41404ad3c5c399ee0243fe0ea9a5ee70b74f896",
"triggerType" : "PUSH"
}, {
"hash" : "e216e514efb89b1ef1a810111cac8175fc0c8941",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18449",
"triggerID" : "e216e514efb89b1ef1a810111cac8175fc0c8941",
"triggerType" : "PUSH"
}, {
"hash" : "a8b972a6d624afb07e197ffef99473f644e3305f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18451",
"triggerID" : "a8b972a6d624afb07e197ffef99473f644e3305f",
"triggerType" : "PUSH"
}, {
"hash" : "89409c24fcccd59f3bdae47bef696c1f3400e650",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "89409c24fcccd59f3bdae47bef696c1f3400e650",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8d6893fd9daf07c30524474cf9a4d39c66a37cba UNKNOWN
* de250fbbcf1d16ba358dd08270eab5e11a5e3740 UNKNOWN
* a8b972a6d624afb07e197ffef99473f644e3305f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18451)
* 89409c24fcccd59f3bdae47bef696c1f3400e650 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] parisni commented on a diff in pull request #8683: [HUDI-5533] Support spark columns comments
Posted by "parisni (via GitHub)" <gi...@apache.org>.
parisni commented on code in PR #8683:
URL: https://github.com/apache/hudi/pull/8683#discussion_r1293890017
##########
hudi-sync/hudi-sync-common/src/main/java/org/apache/hudi/sync/common/util/Parquet2SparkSchemaUtils.java:
##########
@@ -19,40 +19,61 @@
package org.apache.hudi.sync.common.util;
import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.sync.common.model.FieldSchema;
+
import org.apache.parquet.schema.GroupType;
import org.apache.parquet.schema.OriginalType;
import org.apache.parquet.schema.PrimitiveType;
import org.apache.parquet.schema.Type;
+import java.util.Arrays;
+import java.util.List;
+import java.util.regex.Matcher;
+
import static org.apache.parquet.schema.Type.Repetition.OPTIONAL;
/**
* Convert the parquet schema to spark schema' json string.
* This code is refer to org.apache.spark.sql.execution.datasources.parquet.ParquetToSparkSchemaConverter
* in spark project.
+ *
+ * Currently, due to limitations in the FieldSchema only the first level of fields get comments
+ * extracted, if exist.
*/
public class Parquet2SparkSchemaUtils {
- public static String convertToSparkSchemaJson(GroupType parquetSchema) {
+ public static String convertToSparkSchemaJson(GroupType parquetSchema, List<FieldSchema> fieldSchema) {
String fieldsJsonString = parquetSchema.getFields().stream().map(field -> {
switch (field.getRepetition()) {
case OPTIONAL:
return "{\"name\":\"" + field.getName() + "\",\"type\":" + convertFieldType(field)
- + ",\"nullable\":true,\"metadata\":{}}";
+ + ",\"nullable\":true,\"metadata\":{" + getComment(field.getName(), fieldSchema) + "}}";
case REQUIRED:
return "{\"name\":\"" + field.getName() + "\",\"type\":" + convertFieldType(field)
- + ",\"nullable\":false,\"metadata\":{}}";
+ + ",\"nullable\":false,\"metadata\":{" + getComment(field.getName(), fieldSchema) + "}}";
case REPEATED:
String arrayType = arrayType(field, false);
return "{\"name\":\"" + field.getName() + "\",\"type\":" + arrayType
- + ",\"nullable\":false,\"metadata\":{}}";
+ + ",\"nullable\":false,\"metadata\":{" + getComment(field.getName(), fieldSchema) + "}}";
default:
throw new UnsupportedOperationException("Unsupport convert " + field + " to spark sql type");
}
}).reduce((a, b) -> a + "," + b).orElse("");
return "{\"type\":\"struct\",\"fields\":[" + fieldsJsonString + "]}";
}
+ private static String getComment(String name, List<FieldSchema> fromStorage) {
+ return fromStorage.stream()
+ .filter(f -> name.equals(f.getName()))
+ .filter(f -> f.getComment().isPresent())
Review Comment:
Yes that's the current limitation of comments support. It's a flatten list of fields.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8683: [HUDI-5533] Support spark columns comments
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8683:
URL: https://github.com/apache/hudi/pull/8683#issuecomment-1542931756
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7bdb94998ee2853e15de0b4ce6c20735f43a0f5c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7bdb94998ee2853e15de0b4ce6c20735f43a0f5c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7bdb94998ee2853e15de0b4ce6c20735f43a0f5c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8683: [HUDI-5533] Support spark columns comments
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8683:
URL: https://github.com/apache/hudi/pull/8683#discussion_r1193637452
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/avro/SchemaConverters.scala:
##########
@@ -59,32 +59,32 @@ private[sql] object SchemaConverters {
private val unionFieldMemberPrefix = "member"
private def toSqlTypeHelper(avroSchema: Schema, existingRecordNames: Set[String]): SchemaType = {
- avroSchema.getType match {
- case INT => avroSchema.getLogicalType match {
- case _: Date => SchemaType(DateType, nullable = false)
- case _ => SchemaType(IntegerType, nullable = false)
+ (avroSchema.getType, Option(avroSchema.getDoc)) match {
Review Comment:
Can you write a test case for it, especially for creating table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] parisni commented on pull request #8683: [HUDI-5533] Support spark columns comments
Posted by "parisni (via GitHub)" <gi...@apache.org>.
parisni commented on PR #8683:
URL: https://github.com/apache/hudi/pull/8683#issuecomment-1656905343
@danny0405 implemented hive sync datasource, so to clarify about what brings this PR :
- spark read hudi from path (recursive support for comments)
- spark read hudi from a hive metastore with `hoodie.datasource.hive_sync.sync_as_datasource=true` (default) : support for first level of comments
#4960 provided:
- spark read hudi from hive metastore with `hoodie.datasource.hive_sync.sync_as_datasource=false` : support for first level of comments
- query engine (athena, trino..) access to comments from hive metastore
#8740 provided:
- spark read hudi from glue metastore with `hoodie.datasource.hive_sync.sync_as_datasource=false` : support for first level of comments
- query engine (athena, trino..) access to comments from glue metastore
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8683: [HUDI-5533] Support spark columns comments
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8683:
URL: https://github.com/apache/hudi/pull/8683#issuecomment-1629066942
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7bdb94998ee2853e15de0b4ce6c20735f43a0f5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17006",
"triggerID" : "7bdb94998ee2853e15de0b4ce6c20735f43a0f5c",
"triggerType" : "PUSH"
}, {
"hash" : "8d6893fd9daf07c30524474cf9a4d39c66a37cba",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8d6893fd9daf07c30524474cf9a4d39c66a37cba",
"triggerType" : "PUSH"
}, {
"hash" : "de250fbbcf1d16ba358dd08270eab5e11a5e3740",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "de250fbbcf1d16ba358dd08270eab5e11a5e3740",
"triggerType" : "PUSH"
}, {
"hash" : "f41404ad3c5c399ee0243fe0ea9a5ee70b74f896",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18214",
"triggerID" : "f41404ad3c5c399ee0243fe0ea9a5ee70b74f896",
"triggerType" : "PUSH"
}, {
"hash" : "e216e514efb89b1ef1a810111cac8175fc0c8941",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e216e514efb89b1ef1a810111cac8175fc0c8941",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8d6893fd9daf07c30524474cf9a4d39c66a37cba UNKNOWN
* de250fbbcf1d16ba358dd08270eab5e11a5e3740 UNKNOWN
* f41404ad3c5c399ee0243fe0ea9a5ee70b74f896 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18214)
* e216e514efb89b1ef1a810111cac8175fc0c8941 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8683: [HUDI-5533] Support spark columns comments
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8683:
URL: https://github.com/apache/hudi/pull/8683#discussion_r1286881465
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/ddl/QueryBasedDDLExecutor.java:
##########
@@ -220,5 +221,16 @@ private List<String> constructChangePartitions(String tableName, List<String> pa
}
return changePartitions;
}
+
+ /**
+ * SQL statement should be be escaped in order to consider anti-slash
+ *
Review Comment:
Every sentence should end up with `.`. Every new paragraph should start with `<p>`, remove the params comments if there are no comments at all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] parisni commented on a diff in pull request #8683: [HUDI-5533] Support spark columns comments
Posted by "parisni (via GitHub)" <gi...@apache.org>.
parisni commented on code in PR #8683:
URL: https://github.com/apache/hudi/pull/8683#discussion_r1293892201
##########
hudi-sync/hudi-sync-common/src/main/java/org/apache/hudi/sync/common/util/Parquet2SparkSchemaUtils.java:
##########
@@ -133,7 +154,7 @@ private static String convertPrimitiveType(PrimitiveType field) {
private static String convertGroupField(GroupType field) {
if (field.getOriginalType() == null) {
- return convertToSparkSchemaJson(field);
+ return convertToSparkSchemaJson(field, Arrays.asList());
}
Review Comment:
yes, currently in hive sync we don't have the composition data type `List<FieldSchema>` is flat
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] parisni commented on a diff in pull request #8683: [HUDI-5533] Support spark columns comments
Posted by "parisni (via GitHub)" <gi...@apache.org>.
parisni commented on code in PR #8683:
URL: https://github.com/apache/hudi/pull/8683#discussion_r1293889135
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/ddl/JDBCExecutor.java:
##########
@@ -65,7 +65,7 @@ public void runSQL(String s) {
try {
stmt = connection.createStatement();
LOG.info("Executing SQL " + s);
- stmt.execute(s);
+ stmt.execute(escapeAntiSlash(s));
Review Comment:
yes, both JDBC and HMS generate sql and antislash should be doubled escaped otherwise it is lost.
Antislash is used to escape double quotes in the comments DDL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8683: [HUDI-5533] Support spark columns comments
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8683:
URL: https://github.com/apache/hudi/pull/8683#issuecomment-1656905336
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7bdb94998ee2853e15de0b4ce6c20735f43a0f5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17006",
"triggerID" : "7bdb94998ee2853e15de0b4ce6c20735f43a0f5c",
"triggerType" : "PUSH"
}, {
"hash" : "8d6893fd9daf07c30524474cf9a4d39c66a37cba",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8d6893fd9daf07c30524474cf9a4d39c66a37cba",
"triggerType" : "PUSH"
}, {
"hash" : "de250fbbcf1d16ba358dd08270eab5e11a5e3740",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "de250fbbcf1d16ba358dd08270eab5e11a5e3740",
"triggerType" : "PUSH"
}, {
"hash" : "f41404ad3c5c399ee0243fe0ea9a5ee70b74f896",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18214",
"triggerID" : "f41404ad3c5c399ee0243fe0ea9a5ee70b74f896",
"triggerType" : "PUSH"
}, {
"hash" : "e216e514efb89b1ef1a810111cac8175fc0c8941",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18449",
"triggerID" : "e216e514efb89b1ef1a810111cac8175fc0c8941",
"triggerType" : "PUSH"
}, {
"hash" : "a8b972a6d624afb07e197ffef99473f644e3305f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18451",
"triggerID" : "a8b972a6d624afb07e197ffef99473f644e3305f",
"triggerType" : "PUSH"
}, {
"hash" : "89409c24fcccd59f3bdae47bef696c1f3400e650",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18890",
"triggerID" : "89409c24fcccd59f3bdae47bef696c1f3400e650",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8d6893fd9daf07c30524474cf9a4d39c66a37cba UNKNOWN
* de250fbbcf1d16ba358dd08270eab5e11a5e3740 UNKNOWN
* 89409c24fcccd59f3bdae47bef696c1f3400e650 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18890)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8683: [HUDI-5533] Support spark columns comments
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8683:
URL: https://github.com/apache/hudi/pull/8683#issuecomment-1629143254
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7bdb94998ee2853e15de0b4ce6c20735f43a0f5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17006",
"triggerID" : "7bdb94998ee2853e15de0b4ce6c20735f43a0f5c",
"triggerType" : "PUSH"
}, {
"hash" : "8d6893fd9daf07c30524474cf9a4d39c66a37cba",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8d6893fd9daf07c30524474cf9a4d39c66a37cba",
"triggerType" : "PUSH"
}, {
"hash" : "de250fbbcf1d16ba358dd08270eab5e11a5e3740",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "de250fbbcf1d16ba358dd08270eab5e11a5e3740",
"triggerType" : "PUSH"
}, {
"hash" : "f41404ad3c5c399ee0243fe0ea9a5ee70b74f896",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18214",
"triggerID" : "f41404ad3c5c399ee0243fe0ea9a5ee70b74f896",
"triggerType" : "PUSH"
}, {
"hash" : "e216e514efb89b1ef1a810111cac8175fc0c8941",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18449",
"triggerID" : "e216e514efb89b1ef1a810111cac8175fc0c8941",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8d6893fd9daf07c30524474cf9a4d39c66a37cba UNKNOWN
* de250fbbcf1d16ba358dd08270eab5e11a5e3740 UNKNOWN
* f41404ad3c5c399ee0243fe0ea9a5ee70b74f896 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18214)
* e216e514efb89b1ef1a810111cac8175fc0c8941 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18449)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8683: [HUDI-5533] Support spark columns comments
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8683:
URL: https://github.com/apache/hudi/pull/8683#discussion_r1294178078
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/catalog/TableOptionProperties.java:
##########
@@ -184,7 +184,9 @@ public static Map<String, String> translateFlinkTableProperties2Spark(
partitionKeys,
sparkVersion,
4000,
- messageType);
+ messageType,
+ // flink does not support comment yet
+ Arrays.asList());
Review Comment:
Collections.emptyList() ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8683: [HUDI-5533] Support spark columns comments
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8683:
URL: https://github.com/apache/hudi/pull/8683#issuecomment-1629175341
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7bdb94998ee2853e15de0b4ce6c20735f43a0f5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17006",
"triggerID" : "7bdb94998ee2853e15de0b4ce6c20735f43a0f5c",
"triggerType" : "PUSH"
}, {
"hash" : "8d6893fd9daf07c30524474cf9a4d39c66a37cba",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8d6893fd9daf07c30524474cf9a4d39c66a37cba",
"triggerType" : "PUSH"
}, {
"hash" : "de250fbbcf1d16ba358dd08270eab5e11a5e3740",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "de250fbbcf1d16ba358dd08270eab5e11a5e3740",
"triggerType" : "PUSH"
}, {
"hash" : "f41404ad3c5c399ee0243fe0ea9a5ee70b74f896",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18214",
"triggerID" : "f41404ad3c5c399ee0243fe0ea9a5ee70b74f896",
"triggerType" : "PUSH"
}, {
"hash" : "e216e514efb89b1ef1a810111cac8175fc0c8941",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18449",
"triggerID" : "e216e514efb89b1ef1a810111cac8175fc0c8941",
"triggerType" : "PUSH"
}, {
"hash" : "a8b972a6d624afb07e197ffef99473f644e3305f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18451",
"triggerID" : "a8b972a6d624afb07e197ffef99473f644e3305f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8d6893fd9daf07c30524474cf9a4d39c66a37cba UNKNOWN
* de250fbbcf1d16ba358dd08270eab5e11a5e3740 UNKNOWN
* e216e514efb89b1ef1a810111cac8175fc0c8941 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18449)
* a8b972a6d624afb07e197ffef99473f644e3305f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18451)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8683: [HUDI-5533] Support spark columns comments
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8683:
URL: https://github.com/apache/hudi/pull/8683#issuecomment-1629447846
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7bdb94998ee2853e15de0b4ce6c20735f43a0f5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17006",
"triggerID" : "7bdb94998ee2853e15de0b4ce6c20735f43a0f5c",
"triggerType" : "PUSH"
}, {
"hash" : "8d6893fd9daf07c30524474cf9a4d39c66a37cba",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8d6893fd9daf07c30524474cf9a4d39c66a37cba",
"triggerType" : "PUSH"
}, {
"hash" : "de250fbbcf1d16ba358dd08270eab5e11a5e3740",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "de250fbbcf1d16ba358dd08270eab5e11a5e3740",
"triggerType" : "PUSH"
}, {
"hash" : "f41404ad3c5c399ee0243fe0ea9a5ee70b74f896",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18214",
"triggerID" : "f41404ad3c5c399ee0243fe0ea9a5ee70b74f896",
"triggerType" : "PUSH"
}, {
"hash" : "e216e514efb89b1ef1a810111cac8175fc0c8941",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18449",
"triggerID" : "e216e514efb89b1ef1a810111cac8175fc0c8941",
"triggerType" : "PUSH"
}, {
"hash" : "a8b972a6d624afb07e197ffef99473f644e3305f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18451",
"triggerID" : "a8b972a6d624afb07e197ffef99473f644e3305f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8d6893fd9daf07c30524474cf9a4d39c66a37cba UNKNOWN
* de250fbbcf1d16ba358dd08270eab5e11a5e3740 UNKNOWN
* a8b972a6d624afb07e197ffef99473f644e3305f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18451)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8683: [HUDI-5533] Support spark columns comments
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8683:
URL: https://github.com/apache/hudi/pull/8683#discussion_r1286883771
##########
hudi-sync/hudi-sync-common/src/main/java/org/apache/hudi/sync/common/util/Parquet2SparkSchemaUtils.java:
##########
@@ -133,7 +154,7 @@ private static String convertPrimitiveType(PrimitiveType field) {
private static String convertGroupField(GroupType field) {
if (field.getOriginalType() == null) {
- return convertToSparkSchemaJson(field);
+ return convertToSparkSchemaJson(field, Arrays.asList());
}
Review Comment:
Does this mean composition data type is not supported?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8683: [HUDI-5533] Support spark columns comments
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8683:
URL: https://github.com/apache/hudi/pull/8683#discussion_r1193277450
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/avro/SchemaConverters.scala:
##########
@@ -59,32 +59,32 @@ private[sql] object SchemaConverters {
private val unionFieldMemberPrefix = "member"
private def toSqlTypeHelper(avroSchema: Schema, existingRecordNames: Set[String]): SchemaType = {
- avroSchema.getType match {
- case INT => avroSchema.getLogicalType match {
- case _: Date => SchemaType(DateType, nullable = false)
- case _ => SchemaType(IntegerType, nullable = false)
+ (avroSchema.getType, Option(avroSchema.getDoc)) match {
Review Comment:
The conversion tool is copied from Spark: https://github.com/apache/spark/blob/dd4db21cb69a9a9c3715360673a76e6f150303d4/connector/avro/src/main/scala/org/apache/spark/sql/avro/SchemaConverters.scala#L58, just noticed that Spark also does not support keeping comments from Avro fields while doing the converison.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] parisni commented on a diff in pull request #8683: [HUDI-5533] Support spark columns comments
Posted by "parisni (via GitHub)" <gi...@apache.org>.
parisni commented on code in PR #8683:
URL: https://github.com/apache/hudi/pull/8683#discussion_r1198994530
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/avro/SchemaConverters.scala:
##########
@@ -59,32 +59,32 @@ private[sql] object SchemaConverters {
private val unionFieldMemberPrefix = "member"
private def toSqlTypeHelper(avroSchema: Schema, existingRecordNames: Set[String]): SchemaType = {
- avroSchema.getType match {
- case INT => avroSchema.getLogicalType match {
- case _: Date => SchemaType(DateType, nullable = false)
- case _ => SchemaType(IntegerType, nullable = false)
+ (avroSchema.getType, Option(avroSchema.getDoc)) match {
Review Comment:
@danny0405 added a test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] parisni commented on a diff in pull request #8683: [HUDI-5533] Support spark columns comments
Posted by "parisni (via GitHub)" <gi...@apache.org>.
parisni commented on code in PR #8683:
URL: https://github.com/apache/hudi/pull/8683#discussion_r1200536081
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/avro/SchemaConverters.scala:
##########
@@ -59,32 +59,32 @@ private[sql] object SchemaConverters {
private val unionFieldMemberPrefix = "member"
private def toSqlTypeHelper(avroSchema: Schema, existingRecordNames: Set[String]): SchemaType = {
- avroSchema.getType match {
- case INT => avroSchema.getLogicalType match {
- case _: Date => SchemaType(DateType, nullable = false)
- case _ => SchemaType(IntegerType, nullable = false)
+ (avroSchema.getType, Option(avroSchema.getDoc)) match {
Review Comment:
> Kind of feel there is no need to change each match for every data types
Any columns including nested columns also may have comments, so I don't see why we should'nt look after all avro content for doc.
> which invokes toSqlTypeHelper firstly then fix the comment separately.
This would lead to walk thought the avro schema two times, and also lead to complex merge of results. Am i missing something ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8683: [HUDI-5533] Support spark columns comments
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8683:
URL: https://github.com/apache/hudi/pull/8683#discussion_r1192910419
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/avro/SchemaConverters.scala:
##########
@@ -58,33 +58,37 @@ private[sql] object SchemaConverters {
private val unionFieldMemberPrefix = "member"
+ private def extractDoc(doc: String): Option[String] = {
+ if(doc == null) Option.empty
+ else Option(doc)
Review Comment:
Does `Option.ofNullable` work correctly here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8683: [HUDI-5533] Support spark columns comments
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8683:
URL: https://github.com/apache/hudi/pull/8683#discussion_r1199543824
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/avro/SchemaConverters.scala:
##########
@@ -59,32 +59,32 @@ private[sql] object SchemaConverters {
private val unionFieldMemberPrefix = "member"
private def toSqlTypeHelper(avroSchema: Schema, existingRecordNames: Set[String]): SchemaType = {
- avroSchema.getType match {
- case INT => avroSchema.getLogicalType match {
- case _: Date => SchemaType(DateType, nullable = false)
- case _ => SchemaType(IntegerType, nullable = false)
+ (avroSchema.getType, Option(avroSchema.getDoc)) match {
Review Comment:
Kind of feel there is no need to change each match for every data types, can we write another method similiar with `toSqlTypeHelper` which invokes `toSqlTypeHelper` firstly then fix the comment separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8683: [HUDI-5533] Support spark columns comments
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8683:
URL: https://github.com/apache/hudi/pull/8683#issuecomment-1554650588
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7bdb94998ee2853e15de0b4ce6c20735f43a0f5c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17006",
"triggerID" : "7bdb94998ee2853e15de0b4ce6c20735f43a0f5c",
"triggerType" : "PUSH"
}, {
"hash" : "8d6893fd9daf07c30524474cf9a4d39c66a37cba",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8d6893fd9daf07c30524474cf9a4d39c66a37cba",
"triggerType" : "PUSH"
}, {
"hash" : "de250fbbcf1d16ba358dd08270eab5e11a5e3740",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "de250fbbcf1d16ba358dd08270eab5e11a5e3740",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7bdb94998ee2853e15de0b4ce6c20735f43a0f5c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17006)
* 8d6893fd9daf07c30524474cf9a4d39c66a37cba UNKNOWN
* de250fbbcf1d16ba358dd08270eab5e11a5e3740 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] parisni commented on a diff in pull request #8683: [HUDI-5533] Support spark columns comments
Posted by "parisni (via GitHub)" <gi...@apache.org>.
parisni commented on code in PR #8683:
URL: https://github.com/apache/hudi/pull/8683#discussion_r1193606161
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/avro/SchemaConverters.scala:
##########
@@ -59,32 +59,32 @@ private[sql] object SchemaConverters {
private val unionFieldMemberPrefix = "member"
private def toSqlTypeHelper(avroSchema: Schema, existingRecordNames: Set[String]): SchemaType = {
- avroSchema.getType match {
- case INT => avroSchema.getLogicalType match {
- case _: Date => SchemaType(DateType, nullable = false)
- case _ => SchemaType(IntegerType, nullable = false)
+ (avroSchema.getType, Option(avroSchema.getDoc)) match {
Review Comment:
Likely spark also have this limitation when retrieving schema from avro. But spark don't usually infer spark schema from avro. Hudi does, and that's the reason of the patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] parisni commented on pull request #8683: [HUDI-5533] Support spark columns comments
Posted by "parisni (via GitHub)" <gi...@apache.org>.
parisni commented on PR #8683:
URL: https://github.com/apache/hudi/pull/8683#issuecomment-1629076136
@danny0405 The PR is ready.
The way to activate comment for all engines is as below:
```
hoodie.datasource.hive_sync.sync_comment=true
hoodie.datasource.hive_sync.sync_as_datasource=false
```
To recap:
- `hoodie.datasource.hive_sync.sync_comment`: ads the comments in the metastore (hive, glue ...)
- the current PR fixes missing comment when reading the hudi table from path (not metastore) with spark
- `hoodie.datasource.hive_sync.sync_as_datasource`: allow spark to get comments from the metastore
The reason spark datasource should be disabled is our current table property builder rely on parquet types which does [not know about comments](https://github.com/apache/hudi/blob/a9225bd9093e8be8f9ccdd07b0bc542994ffc4c6/hudi-sync/hudi-sync-common/src/main/java/org/apache/hudi/sync/common/util/Parquet2SparkSchemaUtils.java#L36). It would be painful to modify and spark table properties are useless and leads to errors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8683: [HUDI-5533] Support spark columns comments
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8683:
URL: https://github.com/apache/hudi/pull/8683#issuecomment-1629160171
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7bdb94998ee2853e15de0b4ce6c20735f43a0f5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17006",
"triggerID" : "7bdb94998ee2853e15de0b4ce6c20735f43a0f5c",
"triggerType" : "PUSH"
}, {
"hash" : "8d6893fd9daf07c30524474cf9a4d39c66a37cba",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8d6893fd9daf07c30524474cf9a4d39c66a37cba",
"triggerType" : "PUSH"
}, {
"hash" : "de250fbbcf1d16ba358dd08270eab5e11a5e3740",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "de250fbbcf1d16ba358dd08270eab5e11a5e3740",
"triggerType" : "PUSH"
}, {
"hash" : "f41404ad3c5c399ee0243fe0ea9a5ee70b74f896",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18214",
"triggerID" : "f41404ad3c5c399ee0243fe0ea9a5ee70b74f896",
"triggerType" : "PUSH"
}, {
"hash" : "e216e514efb89b1ef1a810111cac8175fc0c8941",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18449",
"triggerID" : "e216e514efb89b1ef1a810111cac8175fc0c8941",
"triggerType" : "PUSH"
}, {
"hash" : "a8b972a6d624afb07e197ffef99473f644e3305f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a8b972a6d624afb07e197ffef99473f644e3305f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8d6893fd9daf07c30524474cf9a4d39c66a37cba UNKNOWN
* de250fbbcf1d16ba358dd08270eab5e11a5e3740 UNKNOWN
* f41404ad3c5c399ee0243fe0ea9a5ee70b74f896 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18214)
* e216e514efb89b1ef1a810111cac8175fc0c8941 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18449)
* a8b972a6d624afb07e197ffef99473f644e3305f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8683: [HUDI-5533] Support spark columns comments
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8683:
URL: https://github.com/apache/hudi/pull/8683#issuecomment-1614058101
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7bdb94998ee2853e15de0b4ce6c20735f43a0f5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17006",
"triggerID" : "7bdb94998ee2853e15de0b4ce6c20735f43a0f5c",
"triggerType" : "PUSH"
}, {
"hash" : "8d6893fd9daf07c30524474cf9a4d39c66a37cba",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8d6893fd9daf07c30524474cf9a4d39c66a37cba",
"triggerType" : "PUSH"
}, {
"hash" : "de250fbbcf1d16ba358dd08270eab5e11a5e3740",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "de250fbbcf1d16ba358dd08270eab5e11a5e3740",
"triggerType" : "PUSH"
}, {
"hash" : "f41404ad3c5c399ee0243fe0ea9a5ee70b74f896",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18214",
"triggerID" : "f41404ad3c5c399ee0243fe0ea9a5ee70b74f896",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8d6893fd9daf07c30524474cf9a4d39c66a37cba UNKNOWN
* de250fbbcf1d16ba358dd08270eab5e11a5e3740 UNKNOWN
* f41404ad3c5c399ee0243fe0ea9a5ee70b74f896 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18214)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8683: [HUDI-5533] Support spark columns comments
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8683:
URL: https://github.com/apache/hudi/pull/8683#issuecomment-1656932683
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7bdb94998ee2853e15de0b4ce6c20735f43a0f5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17006",
"triggerID" : "7bdb94998ee2853e15de0b4ce6c20735f43a0f5c",
"triggerType" : "PUSH"
}, {
"hash" : "8d6893fd9daf07c30524474cf9a4d39c66a37cba",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8d6893fd9daf07c30524474cf9a4d39c66a37cba",
"triggerType" : "PUSH"
}, {
"hash" : "de250fbbcf1d16ba358dd08270eab5e11a5e3740",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "de250fbbcf1d16ba358dd08270eab5e11a5e3740",
"triggerType" : "PUSH"
}, {
"hash" : "f41404ad3c5c399ee0243fe0ea9a5ee70b74f896",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18214",
"triggerID" : "f41404ad3c5c399ee0243fe0ea9a5ee70b74f896",
"triggerType" : "PUSH"
}, {
"hash" : "e216e514efb89b1ef1a810111cac8175fc0c8941",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18449",
"triggerID" : "e216e514efb89b1ef1a810111cac8175fc0c8941",
"triggerType" : "PUSH"
}, {
"hash" : "a8b972a6d624afb07e197ffef99473f644e3305f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18451",
"triggerID" : "a8b972a6d624afb07e197ffef99473f644e3305f",
"triggerType" : "PUSH"
}, {
"hash" : "89409c24fcccd59f3bdae47bef696c1f3400e650",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18890",
"triggerID" : "89409c24fcccd59f3bdae47bef696c1f3400e650",
"triggerType" : "PUSH"
}, {
"hash" : "b92e5eb595620867378ffe2b35b1bd03ce42cb01",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b92e5eb595620867378ffe2b35b1bd03ce42cb01",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8d6893fd9daf07c30524474cf9a4d39c66a37cba UNKNOWN
* de250fbbcf1d16ba358dd08270eab5e11a5e3740 UNKNOWN
* 89409c24fcccd59f3bdae47bef696c1f3400e650 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18890)
* b92e5eb595620867378ffe2b35b1bd03ce42cb01 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8683: [HUDI-5533] Support spark columns comments
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8683:
URL: https://github.com/apache/hudi/pull/8683#discussion_r1286879370
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/ddl/JDBCExecutor.java:
##########
@@ -65,7 +65,7 @@ public void runSQL(String s) {
try {
stmt = connection.createStatement();
LOG.info("Executing SQL " + s);
- stmt.execute(s);
+ stmt.execute(escapeAntiSlash(s));
Review Comment:
Is the escape related with this change ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8683: [HUDI-5533] Support spark columns comments
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8683:
URL: https://github.com/apache/hudi/pull/8683#issuecomment-1613762346
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7bdb94998ee2853e15de0b4ce6c20735f43a0f5c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17006",
"triggerID" : "7bdb94998ee2853e15de0b4ce6c20735f43a0f5c",
"triggerType" : "PUSH"
}, {
"hash" : "8d6893fd9daf07c30524474cf9a4d39c66a37cba",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8d6893fd9daf07c30524474cf9a4d39c66a37cba",
"triggerType" : "PUSH"
}, {
"hash" : "de250fbbcf1d16ba358dd08270eab5e11a5e3740",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "de250fbbcf1d16ba358dd08270eab5e11a5e3740",
"triggerType" : "PUSH"
}, {
"hash" : "f41404ad3c5c399ee0243fe0ea9a5ee70b74f896",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f41404ad3c5c399ee0243fe0ea9a5ee70b74f896",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7bdb94998ee2853e15de0b4ce6c20735f43a0f5c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17006)
* 8d6893fd9daf07c30524474cf9a4d39c66a37cba UNKNOWN
* de250fbbcf1d16ba358dd08270eab5e11a5e3740 UNKNOWN
* f41404ad3c5c399ee0243fe0ea9a5ee70b74f896 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8683: [HUDI-5533] Support spark columns comments
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8683:
URL: https://github.com/apache/hudi/pull/8683#issuecomment-1613772156
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7bdb94998ee2853e15de0b4ce6c20735f43a0f5c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17006",
"triggerID" : "7bdb94998ee2853e15de0b4ce6c20735f43a0f5c",
"triggerType" : "PUSH"
}, {
"hash" : "8d6893fd9daf07c30524474cf9a4d39c66a37cba",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8d6893fd9daf07c30524474cf9a4d39c66a37cba",
"triggerType" : "PUSH"
}, {
"hash" : "de250fbbcf1d16ba358dd08270eab5e11a5e3740",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "de250fbbcf1d16ba358dd08270eab5e11a5e3740",
"triggerType" : "PUSH"
}, {
"hash" : "f41404ad3c5c399ee0243fe0ea9a5ee70b74f896",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18214",
"triggerID" : "f41404ad3c5c399ee0243fe0ea9a5ee70b74f896",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7bdb94998ee2853e15de0b4ce6c20735f43a0f5c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17006)
* 8d6893fd9daf07c30524474cf9a4d39c66a37cba UNKNOWN
* de250fbbcf1d16ba358dd08270eab5e11a5e3740 UNKNOWN
* f41404ad3c5c399ee0243fe0ea9a5ee70b74f896 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18214)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] parisni commented on pull request #8683: [HUDI-5533] Support spark columns comments
Posted by "parisni (via GitHub)" <gi...@apache.org>.
parisni commented on PR #8683:
URL: https://github.com/apache/hudi/pull/8683#issuecomment-1613914438
I have investigated a bit, and here my current understanding:
Reading hudi table w/ spark has two path:
1. if `spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog` (which is what hudi recommend in the documentation), then hudi [will rely on the `HiveSessionCatalog` to get the schema](https://github.com/apache/hudi/blob/dc3aa399ffc4875abba7be5833ebabca222eb6ff/hudi-spark-datasource/hudi-spark3.2plus-common/src/main/scala/org/apache/spark/sql/hudi/catalog/HoodieCatalog.scala#L101-L109). Then if it's a hive metastore implementation, spark will try to get the schema as case sensitive and thus not get it from the hive schema (which is case insensitive), and fall back fetching the table properties `spark.sql.sources.schema` instead. If it's a glue metastore likely the same happens. BTW, [our hive_sync service currently don't propagate the comments](https://github.com/apache/hudi/blob/dc3aa399ffc4875abba7be5833ebabca222eb6ff/hudi-sync/hudi-sync-common/src/main/java/org/apache/hudi/sync/common/util/SparkDataSourceTableUtils.java#L44-L97) in the `spark.sql.source
s.schema` and that's why in this case `spark.sql("desc table")` or `spark.table("table").schema` won't return the comments. This behavior can currently be avoided by setting `hoodie.datasource.hive_sync.sync_as_datasource=false`, which forces spark to grab the information from hive (by letting the spark properties empty in the hms), but in a case insensitive way. I'm not sure what are the consequences of relying on hive only.
2. if `spark.sql.catalog.spark_catalog` is not set or if reading hudi table by path `spark.read.format("hudi").load("path")`, then spark uses the path updated in this PR, by mean get the schema information from the hudi avro file. Except when using `spark.sql("desc table")` b/c spark fallbacks to `hiveSessionCatalog` in this case.
So right now, using this PR and setting both`hoodie.datasource.hive_sync.sync_comment=true` and `hoodie.datasource.hive_sync.sync_as_datasource=false`, one will get the comments in any case (by identifier or by path). However not setting the spark datasource informations within the HMS might have some bad effects (if not, why making so much efforts to maintains two schemas within the hms?).
To fix this we could:
1. make hive_sync [populate the comments in the properties](https://github.com/apache/hudi/blob/dc3aa399ffc4875abba7be5833ebabca222eb6ff/hudi-sync/hudi-sync-common/src/main/java/org/apache/hudi/sync/common/util/SparkDataSourceTableUtils.java#L44-L97)
2. make `HoodieCatalog` not use anymore the `HiveSessionCatalog` to get the schema, but use the hudi avro in place and skip the HMS for this.
I would go for `1` b/c it keeps the current logic intact, and also cover the case `spark.sql("desc tablename")` when `spark.sql.catalog.spark_catalog` is not set.
Thought @danny0405 @bhasudha @yihua ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8683: [HUDI-5533] Support spark columns comments
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8683:
URL: https://github.com/apache/hudi/pull/8683#issuecomment-1656934700
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7bdb94998ee2853e15de0b4ce6c20735f43a0f5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17006",
"triggerID" : "7bdb94998ee2853e15de0b4ce6c20735f43a0f5c",
"triggerType" : "PUSH"
}, {
"hash" : "8d6893fd9daf07c30524474cf9a4d39c66a37cba",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8d6893fd9daf07c30524474cf9a4d39c66a37cba",
"triggerType" : "PUSH"
}, {
"hash" : "de250fbbcf1d16ba358dd08270eab5e11a5e3740",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "de250fbbcf1d16ba358dd08270eab5e11a5e3740",
"triggerType" : "PUSH"
}, {
"hash" : "f41404ad3c5c399ee0243fe0ea9a5ee70b74f896",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18214",
"triggerID" : "f41404ad3c5c399ee0243fe0ea9a5ee70b74f896",
"triggerType" : "PUSH"
}, {
"hash" : "e216e514efb89b1ef1a810111cac8175fc0c8941",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18449",
"triggerID" : "e216e514efb89b1ef1a810111cac8175fc0c8941",
"triggerType" : "PUSH"
}, {
"hash" : "a8b972a6d624afb07e197ffef99473f644e3305f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18451",
"triggerID" : "a8b972a6d624afb07e197ffef99473f644e3305f",
"triggerType" : "PUSH"
}, {
"hash" : "89409c24fcccd59f3bdae47bef696c1f3400e650",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18890",
"triggerID" : "89409c24fcccd59f3bdae47bef696c1f3400e650",
"triggerType" : "PUSH"
}, {
"hash" : "b92e5eb595620867378ffe2b35b1bd03ce42cb01",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18892",
"triggerID" : "b92e5eb595620867378ffe2b35b1bd03ce42cb01",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8d6893fd9daf07c30524474cf9a4d39c66a37cba UNKNOWN
* de250fbbcf1d16ba358dd08270eab5e11a5e3740 UNKNOWN
* 89409c24fcccd59f3bdae47bef696c1f3400e650 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18890)
* b92e5eb595620867378ffe2b35b1bd03ce42cb01 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18892)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8683: [HUDI-5533] Support spark columns comments
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8683:
URL: https://github.com/apache/hudi/pull/8683#issuecomment-1546904147
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7bdb94998ee2853e15de0b4ce6c20735f43a0f5c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17006",
"triggerID" : "7bdb94998ee2853e15de0b4ce6c20735f43a0f5c",
"triggerType" : "PUSH"
}, {
"hash" : "8d6893fd9daf07c30524474cf9a4d39c66a37cba",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8d6893fd9daf07c30524474cf9a4d39c66a37cba",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7bdb94998ee2853e15de0b4ce6c20735f43a0f5c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17006)
* 8d6893fd9daf07c30524474cf9a4d39c66a37cba UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8683: [HUDI-5533] Support spark columns comments
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8683:
URL: https://github.com/apache/hudi/pull/8683#issuecomment-1542935815
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7bdb94998ee2853e15de0b4ce6c20735f43a0f5c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17006",
"triggerID" : "7bdb94998ee2853e15de0b4ce6c20735f43a0f5c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7bdb94998ee2853e15de0b4ce6c20735f43a0f5c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17006)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org